This archive copy of closed topic from Arch Linux forums
For more information and lightweight paged version check this page

#1 2013-05-05 16:13:55

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

THIS GUIDE IS SLIGHTLY OUTDATED AND I'M NOT MAINTAINING IT ANYMORE
PLEASE CHECK OUT ALEX'S BLOG FOR AN UP TO DATE GUIDE

This is intented for people who wish to pass-through a GPU to a virtual machine using the KVM hypervisor, QEMU and vfio-pci

NOTE: AMD RADEON 5xxx, 6xxx, 7xxx, 2xx and NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx, 7xx 9xx have been reported working with this, passing though an intel IGD is not supported YET

NOTE: THIS IS EXPERIMENTAL SO IT MIGHT NOT WORK ON YOUR SYSTEM

This is the result on my radeon 6950:

qCrwPFP.jpg

This is the result on my geforce 470 gtx:

2WrnYi4.jpg

Recommended reads

Alex Williamson's blog and FAQ for detailed technical information
OVMF whitepaper.
GPU passthrough database by noctavian

At this moment there are two ways to achieve this using KVM: legacy VGA via Seabios or non-VGA (UEFI) via OVMF

The main practical advantage of using OVMF is that on intel systems you no longer need the i915 VGA arbiter patch which also disables DRI on the host

Requirements

  • AMD-VI/VT-D enabled and working (On intel systems both your MB and you CPU must support it, to find out if you CPU has VT-D support go here: http://ark.intel.com/, you also need to boot with intel_iommu=on to enable it)

  • At least 2 GPU's, one primary boot device and the card you wish to pass-through

  • Qemu=>2.0

  • Aditional kernel patches might be required if you're using an Intel CPU: ACS override patch and i915 VGA arbiter patch, you can find a kernel package with these patches included on AUR: linux-vfio

Aditional requirements for OVMF:

If you're building your kernel manually this option is required:

CONFIG_VFIO_PCI_VGA=y

Preparing your System

On intel system the i915 VGA arbiter patch is required if:

  1. You're using the IGD on your host

  2. You're doing VGA assignment

NOTE: Enabling this will disable DRI!

If you wish to avoid this, then then you should try OVMF

To enable it add this kernel parameter to your bootloader:

i915.enable_hd_vgaarb=1

Next, we need to prepare the GPU for vfio:

  1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

    Example, blacklisting the opensource radeon module:

    echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 
  2. Use pci-stub

    In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub

    NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this

    mkinitcpio -p linux-mainline

    lspci

    07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

    lspci -n

    07:00.0 0300: 1002:6719 <-- radeon 6950 07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

    Now add this kernel parameter to your bootloader:

    pci-stub.ids=1002:6719,1002:aa80

    dmesg | grep pci-stub

    [ 2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.096160] pci-stub 0000:07:00.0: claimed by stub [ 2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.096174] pci-stub 0000:07:00.1: claimed by stub [ 2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

Setting up vfio and kvm modules

*********************************

This is only required if you get this message:

vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

If your board doesn't enable interrupt remapping, you need to add this to your bootloader:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

*********************************

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Binding a device to vfio-pci

We'll use this script to make life easier:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

Bind the GPU and GPU audio:

vfio-bind 0000:07:00.0 0000:07:00.1

Systemd service:

[Unit] Description=Binds devices to vfio-pci After=syslog.target [Service] EnvironmentFile=-/etc/vfio-pci.cfg Type=oneshot RemainAfterExit=yes ExecStart=-/usr/bin/vfio-bind $DEVICES [Install] WantedBy=multi-user.target

cat /etc/vfio-pci.cfg

DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1"

Testing if its working out

Seabios:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=07:00.0,x-vga=on -device vfio-pci,host=07:00.1 \
-vga none

OVMF:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
-device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1 \
-vga none

Note:

  • kvm=off will hide the kvm hypervisor signature, this is required for NVIDIA cards, since its driver will refuse to work on an hypervisor and result in Code 43 on windows (unless you're using a QUADRO)

  • x-vga=on is required for vga assignment

  • -vga none disables the default vga device on QEMU, it is also required for vga assignment

You should see a black qemu window on your main display, and seabios/ovmf ouput on your monitor from your passthru'd card saying it cant find anything to boot

If you're using an intel cpu and nothing happens try this:

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this

At this point you're ready to create your vm, you could use libvirt + virt-manager to create your vm using a nice graphical interface, there are
also some interesting examples in this thread, otherwise if you prefer to use the command line continue reading

DISK:

Creating a disk image:

Create a 30Gb raw image:

dd if=/dev/zero of=windows.img bs=1M seek=30000 count=0

Using a VIRTIO controller:

We create our virtio controller for qemu:

 -device virtio-scsi-pci,id=scsi

Attaching our hdd:

-drive file=/home/nbhs/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk

Attaching a cdrom:

-drive file=/home/nbhs/windows.iso,id=isocd,if=none -device scsi-cd,drive=isocd

To install windows, you'll need to download the virtio drivers iso from here

Attaching the virtio iso:

-drive file=/home/nbhs/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

Windows will complain it cant find a disk to install to, just click browse, find the folder for your os/arch, select "RED HAT VirtIO pass-through controller" then click next.

For more info check out the Archlinux wiki's QEMU section

Using a physical disk or partition:

-drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk

You might need to change the bus depending on the controller you use

If you wish to use a physical partition and be able to read its contents later on, you can follow these guides:

http://fds-team.de/cms/articles/2013-12 … yer-u.html
https://wiki.archlinux.org/index.php/QE … disk_image


Passing a sata disk controller to qemu:

If you dual boot like me you can pass an entire sata controller (in ahci mode) and all the drives attached, and seabios will boot from them.

lspci

00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)

Make sure all volumes are unmounted, then bind the controller to vfio:

vfio-bind 0000:00:11.0

Now we pass the controller to the vm:

-device vfio-pci,host=00:11.0

USB:

Its also possible to passthrough a usb controller, in my case an ASMEDIA USB3 Controller:

lspci

04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

Bind them to vfio:

vfio-bind 0000:04:00.0 0000:05:00.0 0000:06:00.0

Pass them to the vm:

-device vfio-pci,host=04:00.0 \ -device vfio-pci,host=05:00.0 \ -device vfio-pci,host=06:00.0

Or a specific device:

lsusb

... Bus 004 Device 010: ID 045e:00e1 Microsoft Corp. Wireless Laser Mouse 6000 Reciever <-- mouse Bus 004 Device 011: ID 045e:074b Microsoft Corp. <-- keyboard ...

We pass them to the vm:

-usb -usbdevice host:045e:00e1 -usbdevice host:045e:074b

NETWORK:

Please see https://wiki.archlinux.org/index.php/QEMU#Networking

AUDIO EMULATION:

To hear the sound from the vm on your host speakers you'll need to add this lines:

-soundhw hda

You might need to start qemu like this:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

alsa:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64...

Note: these are the settings i found that work great on my system, if you get crackling/skipping audio you might want to try different settings

To see the available drivers and audio options:

qemu-system-x86_64 -audio-help

LOADING YOUR CARD ROM FROM A DUMP/FILE:

modify this line:

-device vfio-pci,host=07:00.0,...... \

change it to this:

-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \

ISSUES:

PERFORMANCE IMPROVEMENTS:

Last edited by nbhs (2015-07-11 16:23:26)

Offline

#2 2013-05-06 02:24:16

crondog
Member
Registered: 2011-04-21
Posts: 108

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nice work. I currently use xen to achieve a similar thing since the upstream qemu never worked smile

I hope to give this a try soon so i can get rid of xen since i only ever run 1 vm and its a bit of a waste

Offline

#3 2013-05-10 00:30:06

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDITED:

The single most important thing i found to improve the vm performance on my amd board is to disable nested page tables:

echo "options kvm-amd npt=0" > /etc/modprobe.d/kvm-amd.conf

These are some other things i did to improve the performance:

  • -cpu host

  • Using hugetlbfs instead of transparent huge pages

  • Set Preemption Model to Voluntary on kernel

  • Set Timer frequency  1000HZ on kernel

Im using -cpu Opteron_G4 instead of -cpu host, this shouldnt affect performance though.

KL5cJaX.jpg

NOTE: i use 6 cores (2-7) for the vm, 2 (0-1) are reserved for the host using cpuset cgroups
NOTE: Most of these settings affect vcpu performance, gpu performance remains about the same.

Last edited by nbhs (2014-01-20 22:02:03)

Offline

#4 2013-05-10 19:16:52

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for the very detailed post, nbhs! I've been struggling to get Xen working in either a primary or secondary passthrough situation over the past week, and I was about to start trying with KVM. This guide will sure come in handy.

I have a fairly similar configuration to you:
ASUS M5A99X EVO (990X/SB950)
AMD FX-8350 (4.8 GHz)
AMD Radeon HD 5750 (for Linux host)
AMD Radeon HD 7870 (for Windows guest)

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, I actually went to all the bother to send ASUS a detailed and helpful support request detailing the AMD IOMMU spec and the chipset documentation, telling them how they could fix their firmware. My mistake, as the support guy continually misunderstands me due to the language barrier (and because he probably just wants me to go away). So I'm almost certainly wasting my time trying to get the actual responsible party to fix their broken firmware.

I'll keep you posted on any issues that I run into as I give KVM a try.

Offline

#5 2013-05-11 01:51:58

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, I actually went to all the bother to send ASUS a detailed and helpful support request detailing the AMD IOMMU spec and the chipset documentation, telling them how they could fix their firmware. My mistake, as the support guy continually misunderstands me due to the language barrier (and because he probably just wants me to go away). So I'm almost certainly wasting my time trying to get the actual responsible party to fix their broken firmware.

I'll keep you posted on any issues that I run into as I give KVM a try.

Yeah so did i, unfortunately asus doesnt support linux, and their tech support are bunch of monkeys behind a keyboard, they remind me of this http://www.youtube.com/watch?v=C2Ph8zwpNyI, ill never buy a product from them ever again.

Last edited by nbhs (2013-05-11 01:54:31)

Offline

#6 2013-05-16 00:12:13

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've run into two problems so far.

First problem: I had to stop using the proprietary ATI graphics driver on the host, as running the VM and X at the same time would hang the system. Unfortunately, the fglrx module thinks it's a great idea to grab all the cards in the system and remap their IRQs (as shown by dmesg and /proc/interrupts) despite them actually being assigned to pci-stub. Apparently it just scans the PCI bus looking for ATI cards and then configures all of them, no matter which device the module was actually loaded for. Brilliant.

I thought I might be able to get around this, as I was able to hack the PCI ID for the 7870 out of the fglrx kernel module by modifying /usr/share/ati/build_mod/fglrxko_pci_ids.h and rebuilding. After I did this, the fglrx kernel module ignored the secondary card like I wanted it to. But the fglrx X driver would then refuse to let X start, since it was doing its own PCI bus scan, in which it recognized the secondary card and realized that the kernel module hadn't configured it. And to my great disgruntlement, the fglrx X driver ignores useful options like "AutoAddGPU" "false", plus I can't just hack their PCI scan to ignore the card because they've gone and duplicated a bunch of external code (like the PCI bus scanning). I basically can't change its behavior because it's closed source.

I suppose an older version of fglrx might work (pre-7000 series, so it wouldn't recognize the card). Hacking the kernel to misreport the PCI ID would probably work too. For now, though, just using the open source radeon driver is an acceptable solution.

Second problem: despite the new patches to qemu and linux, the card is apparently not being reset between guest runs. If I run the VM twice, the host will hang, gradually getting slower for a few seconds until it freezes, almost like it's experiencing an interrupt storm or something; eventually even Magic SysRq doesn't work. Suspending the host appears to be enough to reset the card, so for now, I'll just suspend/wake between VM runs. I plan to try the eject-before-shutdown method too.

Everything else seems to be working though. Performance is impressive! Now I need to buy a USB switch for input devices; or maybe I'll just solder/glue one out of spare parts. Those things are ridiculously overpriced for what amounts to a 4-pole, double-throw switch with USB ports on the ends.

(I was going to attach some screenshots to this post, but Print Screen made the guest freeze. So maybe I'll post those later.)

Offline

#7 2013-05-16 00:17:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the report!, I had no idea about fglrx, i havent used them since i had a Radeon 9800xt smile, and opensource is good enough for my 5450. You can try ejecting the card on windows just like you would eject a pendrive, method No3 is about the automatization of this process before shutting down/restarting the guest. You could also fill a bug report on qemu-devel about the card not reseting properly, in my case i can hear the gpu fan speeding up after a reset. I also re-uploaded linux-mainline based on 3.9.2 and qemu-git packages with reset patches, they're working fine on my pc.

EDIT: Could you try to load your gpu bios by file and see if it still crash? i remember a couple of times my vm crashed after a reset ( before i accidentally physically destroyed the bios chip lol, now it wont work without loading a rom file smile ) and after i manually loaded the rom, didnt happen again, you can do this by appending ',rombar=0, romfile=/path/to/the/rom.bin'

ex:

-device vfio-pci,host=07:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1,rombar=0,romfile=/home/nbhs/vgabios-6950.rom

Or use my script, and modify VBIOS="" to VBIOS="/path/to/rom"

You can find yours here: http://www.techpowerup.com/vgabios/

Last edited by nbhs (2013-05-16 01:43:08)

Offline

#8 2013-05-16 13:51:55

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried to get it working for me too, but with no success. sad

My system:
Radeon 7950
i5 3470 (non-K) incl Intel GPU for Host
AsRock Z77 Extreme4
8GB Ram

First thing I was trying was the old way, with pci-assign. I was able to first boot with VNC, installing ATI drivers. After reboot the screen woke up at login promt and the Perfomance was ok/good (I wasn't trying too much. Windows Performance Index at 7.5, so it should have worked wink). But at reboot of VM the host crashed....

So I was looking around the internet and found this thread. Tried many hours to get it working, but I don't get any picture to the screen. No Seabios Texts and nothing in Windows after installing ATI drivers with vnc (and removing VNC again) like it was working with pci-assign.

Have tried Qemu-git from AUR, Qemu-1.5-rc1, seabios from your link, stock kernel (3.9.2-1) and kernel 3.10-rc1.
Did not change anything wink.

The only thing I havn't tried till now is to dump the VGA bios and put it in. Will do this today, when I´m at home.

IOMMU is enabled. logfiles look good (for me big_smile). The only thing is "DMAR: No ATSR found". You have a Idea about this?
Full logfiles I can add later. 

Bind everything to vfio-pci seems to work. Also there isn't any error message at KVM-Start
I also tried all with pci-stub entries at grub and without, same with the interrupt remapping entry.

Do you have any Idea? big_smile

Thanks
apo

Last edited by apoapo (2013-05-16 14:07:21)

Offline

#9 2013-05-16 19:32:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

Tried to get it working for me too, but with no success. sad

My system:
Radeon 7950
i5 3470 (non-K) incl Intel GPU for Host
AsRock Z77 Extreme4
8GB Ram

First thing I was trying was the old way, with pci-assign. I was able to first boot with VNC, installing ATI drivers. After reboot the screen woke up at login promt and the Perfomance was ok/good (I wasn't trying too much. Windows Performance Index at 7.5, so it should have worked wink). But at reboot of VM the host crashed....

So I was looking around the internet and found this thread. Tried many hours to get it working, but I don't get any picture to the screen. No Seabios Texts and nothing in Windows after installing ATI drivers with vnc (and removing VNC again) like it was working with pci-assign.

Have tried Qemu-git from AUR, Qemu-1.5-rc1, seabios from your link, stock kernel (3.9.2-1) and kernel 3.10-rc1.
Did not change anything wink.

The only thing I havn't tried till now is to dump the VGA bios and put it in. Will do this today, when I´m at home.

IOMMU is enabled. logfiles look good (for me big_smile). The only thing is "DMAR: No ATSR found". You have a Idea about this?
Full logfiles I can add later. 

Bind everything to vfio-pci seems to work. Also there isn't any error message at KVM-Start
I also tried all with pci-stub entries at grub and without, same with the interrupt remapping entry.

Do you have any Idea? big_smile

Thanks
apo

Stock kernel wont work, you need to rebuild it to include:

 --- VFIO Non-Privileged userspace driver framework <*> VFIO support for PCI devices [*] VFIO PCI support for VGA devices 

Please use the linux-mainline, and qemu-git packages that i posted on "RADEON RELATED PROBLEMS", it has this enabled by default.

If nothing appears on screen try:

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

Also remember when launching qemu you must use q35 machine mode, vga none,  and "x-vga=on" on your gpu

ex:

qemu-system-x86_64 --enable-kvm -M q35 -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

Last edited by nbhs (2013-05-16 19:46:34)

Offline

#10 2013-05-16 21:35:47

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah. I forgot to say this.
I changed the configuration of the kernel wink. I enabled the 4 vfio things in the kernels I used.
All things should be like you said. Also tried reloading kvm_intel module ...
I will take a look in the next hours and post some Outputs and try the linux-mainline ....

Thx for helping smile

EDIT: Some outputs wink

$ dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 00000000bd969280 000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.144709] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.144714] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.144785] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.404802] DMAR: No ATSR found [ 0.404822] IOMMU 0 0xfed90000: using Queued invalidation [ 0.404823] IOMMU 1 0xfed91000: using Queued invalidation [ 0.404824] IOMMU: Setting RMRR: [ 0.404832] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.405981] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd568000 - 0xbd59afff] [ 0.405997] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd568000 - 0xbd59afff] [ 0.406011] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd568000 - 0xbd59afff] [ 0.406019] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.406025] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
ls -l /sys/bus/pci/drivers/vfio-pci/ insgesamt 0 lrwxrwxrwx 1 root root 0 17. Mai 00:53 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 17. Mai 00:53 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 17. Mai 00:53 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 --w------- 1 root root 4096 17. Mai 00:53 bind lrwxrwxrwx 1 root root 0 17. Mai 00:53 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 17. Mai 00:53 new_id --w------- 1 root root 4096 17. Mai 00:53 remove_id --w------- 1 root root 4096 17. Mai 00:48 uevent --w------- 1 root root 4096 17. Mai 00:53 unbind
lspci 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti PRO [Radeon HD 7950] 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]
Starting KVM: dmesg [ 511.383247] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 511.383256] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
qemu-system-x86_64 --enable-kvm -M q35 -m 2096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Last edited by apoapo (2013-05-16 22:59:45)

Offline

#11 2013-05-17 00:32:44

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things smile.

Offline

#12 2013-05-17 00:57:17

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things smile.

NIce! have you tried running windows with it yet? does it work if you dont load the rom file by hand?

Offline

#13 2013-05-17 01:45:06

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
apoapo wrote:

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things smile.

NIce! have you tried running windows with it yet? does it work if you dont load the rom file by hand?

Hi,

I tried it again and I captured this error message if I use the PCI-e card es Primary Card:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow

This was, when I got some seabios picture.

After some reboots I wanted to try it again, I was spammed with errors sad. Dont have them to show you now...  will look for it.


Now I'm With the onboard as primary Card and I found this in dmesg:

[ 0.401391] DMAR: No ATSR found [ 0.401412] IOMMU 0 0xfed90000: using Queued invalidation [ 0.401413] IOMMU 1 0xfed91000: using Queued invalidation [ 0.401415] IOMMU: Setting RMRR: [ 0.401422] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.402590] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd568000 - 0xbd59afff] [ 0.402606] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd568000 - 0xbd59afff] [ 0.402619] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd568000 - 0xbd59afff] [ 0.402628] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.402635] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.402717] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.402782] ------------[ cut here ]------------ [ 0.402786] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x83/0x90() [ 0.402787] Hardware name: To Be Filled By O.E.M. [ 0.402788] Modules linked in: [ 0.402790] Pid: 1, comm: swapper/0 Not tainted 3.9.2-1-mainline #1 [ 0.402791] Call Trace: [ 0.402796] [<ffffffff81058090>] warn_slowpath_common+0x70/0xa0 [ 0.402797] [<ffffffff8105817a>] warn_slowpath_null+0x1a/0x20 [ 0.402799] [<ffffffff8129ea43>] pci_find_upstream_pcie_bridge+0x83/0x90 [ 0.402802] [<ffffffff813b3c7b>] intel_iommu_add_device+0x4b/0x1f0 [ 0.402805] [<ffffffff813ab630>] ? bus_set_iommu+0x50/0x50 [ 0.402806] [<ffffffff813ab65a>] add_iommu_group+0x2a/0x50 [ 0.402809] [<ffffffff8135bf63>] bus_for_each_dev+0x63/0xa0 [ 0.402811] [<ffffffff813ab622>] bus_set_iommu+0x42/0x50 [ 0.402814] [<ffffffff819130b6>] intel_iommu_init+0xa83/0xb7b [ 0.402816] [<ffffffff818dd10b>] ? memblock_find_dma_reserve+0x124/0x124 [ 0.402818] [<ffffffff818dd11d>] pci_iommu_init+0x12/0x3c [ 0.402820] [<ffffffff8100210a>] do_one_initcall+0x10a/0x160 [ 0.402823] [<ffffffff818d5037>] kernel_init_freeable+0x15b/0x1dc [ 0.402824] [<ffffffff818d4881>] ? do_early_param+0x88/0x88 [ 0.402827] [<ffffffff814b55f0>] ? rest_init+0x90/0x90 [ 0.402828] [<ffffffff814b55fe>] kernel_init+0xe/0x190 [ 0.402830] [<ffffffff814dad6c>] ret_from_fork+0x7c/0xb0 [ 0.402832] [<ffffffff814b55f0>] ? rest_init+0x90/0x9

Offline

#14 2013-05-17 01:55:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Pease take a look at http://qemu.11.n7.nabble.com/VFIO-VGA-t … 01483.html, @Knut Omang this guy had it working using 2 discrete cards but he had to disable the integrated intel gpu

Last edited by nbhs (2013-05-17 01:57:12)

Offline

#15 2013-05-17 02:31:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You should set you primary boot device as IGP instead of pcie, vfio-vga wont work on you primary vga device.

lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
--w------- 1 root root 4096 17. Mai 00:53 bind
lrwxrwxrwx 1 root root    0 17. Mai 00:53 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 17. Mai 00:53 new_id
--w------- 1 root root 4096 17. Mai 00:53 remove_id
--w------- 1 root root 4096 17. Mai 00:48 uevent
--w------- 1 root root 4096 17. Mai 00:53 unbind

Looking back at it, i dont think your pcie root port should be owned by vfio, here's mine:

lrwxrwxrwx 1 root root 0 may 16 23:31 0000:07:00.0 -> ../../../../devices/pci0000:00/0000:00:0b.0/0000:07:00.0 lrwxrwxrwx 1 root root 0 may 16 23:31 0000:07:00.1 -> ../../../../devices/pci0000:00/0000:00:0b.0/0000:07:00.1 --w------- 1 root root 4096 may 16 23:31 bind lrwxrwxrwx 1 root root 0 may 16 23:31 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 may 16 16:58 new_id --w------- 1 root root 4096 may 16 23:31 remove_id --w------- 1 root root 4096 may 16 16:58 uevent --w------- 1 root root 4096 may 16 23:31 unbind

Anyway you could try playing around with intel_iommu kernel options like intel_iommu=on,igfx_off,pass-through, if it doesnt work you should fill a bug report on qemu-devel

Last edited by nbhs (2013-05-17 02:43:50)

Offline

#16 2013-05-17 02:38:56

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The Root-Port is in the same iommu-group with the graphicscard. I was wondering myself too

Offline

#17 2013-05-17 02:45:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried setting intel_iommu=on,igfx_off or   intel_iommu=on,igfx_on with igd as primary?

Offline

#18 2013-05-17 03:05:29

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Have you tried setting intel_iommu=on,igfx_off or   intel_iommu=on,igfx_on with igd as primary?

No change till now.
Will go to sleep now... its 5:00 in the morning here big_smile

Thx for your help smile. Will follow the conversation you linked me + try again with a second graphic card.
You will hear from me wink


PS:
some logoutput I don't have all the time at starting kvm (with intel GPU as primary):

[ 109.801109] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 109.827474] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 109.827481] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 [ 110.087849] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

Last edited by apoapo (2013-05-31 13:27:14)

Offline

#19 2013-05-17 04:43:26

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Could you try to load your gpu bios by file and see if it still crash? i remember a couple of times my vm crashed after a reset ( before i accidentally physically destroyed the bios chip lol, now it wont work without loading a rom file smile ) and after i manually loaded the rom, didnt happen again, you can do this by appending ',rombar=0, romfile=/path/to/the/rom.bin'

I finally had time to give this a try, and it doesn't seem to make a difference. The VM works fine the first time, but the second time, the host freezes up as the guest boots. I tried a ROM image obtained via sysfs and one from techpowerup.com (different version for the same card); no visible difference.

What I may try next is swapping the cards. This isn't the configuration I'll want to use in the long term, but it will be interesting to see if the 5750 behaves any differently from the 7870 when passed through.

Incidentally, secondary passthrough (-vga cirrus or -vga std) doesn't cause any additional problems, which is nice, although it doesn't fix anything either.

Offline

#20 2013-05-17 06:14:26

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:

What I may try next is swapping the cards. This isn't the configuration I'll want to use in the long term, but it will be interesting to see if the 5750 behaves any differently from the 7870 when passed through.

So I just gave this a try: 7870 as primary, 5750 as passthru. Started out without the ROM override. First boot, the guest works great. Second boot, the VM won't start, but I get this line in dmesg:

vfio-pci 0000:02:00.0: Invalid ROM contents

This didn't happen previously. Overriding the ROM on the second boot seems to get past this problem, but then the host freezes up, just like before.

Offline

#21 2013-05-17 06:18:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried the "eject" method yet?

I'm still wondering why my particular card doesnt have this issue, i used to get performance degradation without reset patches but my host never crashed.

Are you sure the kernel and qemu are patched?

Last edited by nbhs (2013-05-17 06:24:34)

Offline

#22 2013-05-17 06:30:57

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Have you tried the "eject" method yet?

I'm still wondering why my particular card doesnt have this issue, i used to get performance degradation without reset patches but my host never crashed.

That's on my list of things to do, but first I'm going to reply to the qemu-devel mailing list.

I want to figure out why the pci reset from software isn't happening (presumably it is not happening, since suspending the host makes everything work properly). I don't know if it's possible to divine from the DEBUG_VFIO output why the reset attempt failed, or if the hardware is just silently ignoring the request.

nbhs wrote:

Are you sure the kernel and qemu are patched?

Yes.

Last edited by jgott (2013-05-17 06:31:22)

Offline

#23 2013-05-17 06:53:45

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oops. It seems that I forgot to change the pci-stub vendor and device IDs when I swapped cards. X wasn't running, but the radeon module might have been loaded. So maybe the card-swapping results were invalid. I don't want to switch everything over again though, what a pain. mad

Offline

#24 2013-05-17 21:18:58

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I put in a second graphicscard and tested it again.
The good thing is. I had no problems to pass it to the VM... have seen the bios of the Card + seabios on the screen + Windows was booting (was booting to system repair, but I think that's because of other storage type)

The bad thing is, that both graphicscards are in the same iommu-group + 2 PCI-Controller 01.0 and 01.1. Think that is because my 2 PCIe Ports share share the 16 lanes, that are normally used by the first port.
So I think I will be abled to pass the card to the VM, but can't really use the primary card for Linux and I have to wait to get the Intel as Primary working ...

Any other thoughts about this?

Offline

#25 2013-05-17 21:49:15

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

I put in a second graphicscard and tested it again.
The good thing is. I had no problems to pass it to the VM... have seen the bios of the Card + seabios on the screen + Windows was booting (was booting to system repair, but I think that's because of other storage type)

The bad thing is, that both graphicscards are in the same iommu-group + 2 PCI-Controller 01.0 and 01.1. Think that is because my 2 PCIe Ports share share the 16 lanes, that are normally used by the first port.
So I think I will be abled to pass the card to the VM, but can't really use the primary card for Linux and I have to wait to get the Intel as Primary working ...

Any other thoughts about this?

Unfortunately, no, your best bet at this point is fill a bug @ qemu-devel, if you're using my script to bind the devices, you could try doing it manually, only binding the gpu and the gpu_snd, but i dont think vfio will work if all the devices of the same group are not binded to it.

Offline

#26 2013-05-17 21:58:13

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Unfortunately, no, your best bet at this point is fill a bug @ qemu-devel, if you're using my script to bind the devices, you could try doing it manually, only binding the gpu and the gpu_snd, but i dont think vfio will work if all the devices of the same group are not binded to it.

Yeah will try this again and will then see what I can do wink.Or I have to buy a PCIe 1x oder PCI VGA card for host xD

Offline

#27 2013-05-17 22:05:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is how you can manually bind it:

echo "0000:07:00.0" > /sys/bus/pci/devices/0000\:07\:00.0/driver/unbind echo 1002 6719 > /sys/bus/pci/drivers/vfio-pci/new_id

Last edited by nbhs (2013-05-17 22:05:30)

Offline

#28 2013-05-17 22:12:03

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

This is how you can manually bind it:

echo "0000:07:00.0" > /sys/bus/pci/devices/0000\:07\:00.0/driver/unbind echo 1002 6719 > /sys/bus/pci/drivers/vfio-pci/new_id

Yepp thx, know this smile.

Tried to get VGA passthrough working some times ago with different hardware, XEN and kvm with pci-assign wink

Last edited by apoapo (2013-05-17 22:12:59)

Offline

#29 2013-05-17 22:16:33

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well if pci-assign worked for you in the past, and it crashed after a guest reboot, you can try the eject method under radeon related problems, that worked for me when using pci-assign

Offline

#30 2013-05-18 01:39:11

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems to work really well with pci-assign.

Now I have the Intel GPU as host-GPU. Running with X without any problems.
I'm able to reboot the VM with ejecting the card before reboot / shutdown.
Perfomance seems good and till now i havn't got freezes.

I can't use q35. If i do, the system says, the graphic card could not be started, so I use this configuration for now:

qemu-system-x86_64 --enable-kvm -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -drive file=/var/lib/libvirt/images/win-test.img 

Offline

#31 2013-05-18 01:49:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

It seems to work really well with pci-assign.

Now I have the Intel GPU as host-GPU. Running with X without any problems.
I'm able to reboot the VM with ejecting the card before reboot / shutdown.
Perfomance seems good and till now i havn't got freezes.

I can't use q35. If i do, the system says, the graphic card could not be started, so I use this configuration for now:

qemu-system-x86_64 --enable-kvm -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -drive file=/var/lib/libvirt/images/win-test.img 

Great!

You can do the ejecting automatically using a tool called deveject, i detailed this process under radeon related problems

Offline

#32 2013-05-18 03:12:15

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It is running really great! big_smile

The only thing is, that running the scripts automatically with gpedit.msc isn't working. If i try to use this, the system will freeze at next start, so I think the scripts will not be executed.
If I eject the vga card manually I can reboot (shutdown and start again) as often as I want. When rebooting, the System starts without vga-card, because they are still ejected, I think.

But the performance is stunning!

Tried a game (Project Cars), where CPU and GPU are utilized very much at same settings.
Windows 7 nativ: 58fps
Windows 7 VM: 60fps
I think the 2 fps come from the clean install/less programms running wink

Also testing with very high Settings of the game (With downsampling 9 =>5760x3600 Pixel resolution), I get ~same fps like native!

Offline

#33 2013-05-18 03:16:47

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Gratz, the script should work tho, run cmd.exe as admin and execute

 deveject.exe -EjectId:"PCI\VEN_1002&DEV_6719&SUBSYS_31221682&REV_00\4&184E0D10&0&00E0"

change pci\ven.... to match yours, its works the same as manually ejecting so there's prolly a typo somewhere

Last edited by nbhs (2013-05-18 03:17:55)

Offline

#34 2013-05-18 03:21:39

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Gratz, the script should work tho, run cmd.exe as admin and execute

 deveject.exe -EjectId:"PCI\VEN_1002&DEV_6719&SUBSYS_31221682&REV_00\4&184E0D10&0&00E0"

change pci\ven.... to match yours, its works the same as manually ejecting so there's prolly a typo somewhere

yeah. running deveject.exe manually is running fine. Only if I want to run it automatically at shutdown (with dpedit.msc script) it isn't working.

Offline

#35 2013-05-18 03:24:37

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

copy deveject.exe to C:\Windows\System32\deveject.exe and try again, you need to put the full path under "Script name" and the rest under "Script Parameters" --> 
-EjectId:"PCI\VEN_1002&DEV_6719&SUBSYS_31221682&REV_00\4&184E0D10&0&00E0"

Offline

#36 2013-05-18 03:26:01

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tried it again... is working now.
Did the same, I did before... think it could be a typo wink
Thanks a lot.

Offline

#37 2013-05-18 03:26:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

np, and gratz

Offline

#38 2013-05-19 07:19:50

shadey
Member
Registered: 2013-05-19
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

It seems to work really well with pci-assign.

Now I have the Intel GPU as host-GPU. Running with X without any problems.
I'm able to reboot the VM with ejecting the card before reboot / shutdown.
Perfomance seems good and till now i havn't got freezes.

I can't use q35. If i do, the system says, the graphic card could not be started, so I use this configuration for now:

qemu-system-x86_64 --enable-kvm -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -drive file=/var/lib/libvirt/images/win-test.img 

Hi there!
We have the same motherboard and GPU and almost the same CPU. (I have 3770) It looks like we are both trying to achieve the same thing too! I want to use my Intel IGP as the primary Linux desktop with the Radeon passed through for windows gaming.

After following this thread and your conversation I have encountered the same problems that you had. I noticed that vfio grabbed the PCI Express root and I could not get the BIOS display on the pass'd thru card. I see that you were able to get that working with PCI Express as the Primary VGA device on the host hardware. I have seen this behavior elsewhere so I think it might be a hardware limitation on our boards. I have been playing with many different VM systems lately and I saw the same behaviour with VMware ESXi 5.0.0 - if I didn't have primary VGA set to PCI Express AND disabled intel IGP in the host bios then the passed through display would be blank. Before VMWare I tried VGA pass through on Virtualbox and I gave up on that because of the blank screen thing and kernel crashes (I was not aware trying the Primary VGA bios setting at that stage). My best success before KVM has been in Xen using qemu-upstream.

Thank you for your configuration, I used a similar command line and managed to boot straight into my installed Win7 partition that I had created using Xen the day before, passthrough working and all. Now that I understand more the inner workings of QEMU I realised that I don't even need Xen to achieve my goals. I think KVM is a neat way to accomplish what we are trying to do.

Are you planning to use virtio drivers for network, disk etc? Would be interested to hear if you have success with it.

Dear OP: Thank you for this thread! It is how I discovered Archlinux and now I am converted.

I am using qemu-1.5.0-rc3 built with the "seabios" branch from seabios.git

Last edited by shadey (2013-05-19 07:20:57)

Offline

#39 2013-05-19 09:13:06

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shadey wrote:

Dear OP: Thank you for this thread! It is how I discovered Archlinux and now I am converted.

Sweet i'm glad this is useful

Offline

#40 2013-05-19 10:41:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Even though this is a kvm thread, i wanna share some of my experiences with XEN, and how to get PRIMARY vga passthrough ( gfx_passthru = 1 ) on radeon cards.

Sooooo, this is basically what im doing, i'm passing through my secondary radeon ( 6950 ) card ( although you can also pass your primary card), as a primary vga (without cirrus emulated card), my usb3 ports, and my sata controller with my 2TB disk connected to it, using a DUET image ( see http://www.rodsbooks.com/bios2uefi/ ), booting windows server 2008 on UEFI Mode straight from the controller ( no emulated disk controller ), with a hacked xen to enable intremap on my broken Asus crosshair v formula ivrs table, using the XM toolstack.

This is the package i use, xen-4.1.5 based on xen-4.2.2 from AUR: xen.tar.gz

I commented out my hack @ line 37 because this might break working systems.

Once extracted you'll find a file called vgabios-pt.bin. That is the rom file for my 6950, you have to dump your card's rom or d/l it from http://www.techpowerup.com/vgabios/, and replace that file with yours.

AFTER you copy your rom, build xen, install it then enable the needed services:

systemctl enable xenstored systemctl enable xend

Now you need to actually boot xen so we create a new file called 05_xen @ /etc/grub.d

nano -w /etc/grub.d/05_xen

The fist 2 lines:

#!/bin/sh exec tail -n +3 $0

For the rest the easiest way and the only one i'll describe here is to copy an entry from you /boot/grub/grub.cfg file:

cat /boot/grub/grub.cfg

Now copy the first menuentry to 05_xen:

menuentry 'Arch GNU/Linux, with Linux.... { .... .... }

Add this line on top of "echo    'Loading Linux core repo kernel .":

multiboot /xen-4.1.5.gz

Modify "linux    /vmlinuz-linux" to "module    /vmlinuz-linux" and "initrd    /initramfs-linux" to "module    /initramfs-linux"

Make 05_xen executable

chmod +x /etc/grub.d/05_xen

Edit /etc/mkinitcpio.conf and include xen-pciback

MODULES="xen-pciback"

If you wish to bind a device on boot to pciback modify 05_xen like this:

module /vmlinuz-linux root=/dev/mapper/system-root ro xen-pciback.passthrough=1 xen-pciback.permissive=1 xen-pciback.hide=(07:00.0)(07:00.1)

Update grub.cfg

grub-mkconfig -o /boot/grub/grub.cfg

Reboot and check everything is works

xm info | grep virt_caps

virt_caps : hvm hvm_directio

After this point if everything is working i suggest you read the xen documentation on how to create a vm, this is my configuration:

kernel = "hvmloader" builder='hvm' memory = 8192 name = "hvm" cpus = [ "2", "3", "4", "5", "6", "7" ] vcpus = 6 disk = [ 'file:/root/xen/images/duet.img,hda,w' ] pci = [ '00:11.0', '04:00.0', '05:00.0', '06:00.0', '0000:07:00.*' ] vif = [ 'bridge=br0' ] gfx_passthru = 1 localtime = 1 xen_platform_pci = 1 pci_power_mgmt = 1 pci_msitranslate = 0 viridian = 1

How i got windows to boot from my sata controller using qemu-dm and DUET

Notice this line:

disk = [ 'file:/root/xen/images/duet.img,hda,w' ]

This is the duet image i've been writing about, its a 200MB image containing DUET and REFIND ( http://www.rodsbooks.com/refind )( also included is a efi driver dump of the controller for RAID mode (sb950), so you can boot on RAID but i havent experimented with it much )

Duet image here: Duet.img.zip

The boot sequence is like this ROMBIOS-->DUET-->WINDOWS.

Duet contains drivers for the controller in ahci mode, it finds the windows efi partition and boots from it.

Radeon xen primary passthrough on xen:

dEGPaNI.jpg

Duet with REFIND booting windows ( with gfx_passthu = 0 to take the screenshot ):

S1bLj7P.png

Last edited by nbhs (2013-05-21 08:04:18)

Offline

#41 2013-05-19 20:40:33

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

This is the package i use, xen-4.1.5 based on xen-4.2.2 from AUR: xen.tar.gz

Did the Xen people break VGA passthrough in recent versions? I got the sense that they had because I couldn't ever get 4.2 or 4.3 unstable to work completely (several different problems kept coming up), and I see that you're using 4.1.

I know they had a recent security advisory that resulted in tightening up some things and making matters worse for AMD IOMMU users with broken BIOSes. Apparently the switch to xl broke some things too (see the first entry on this page). And the gfx_passthrough option isn't yet supported for the new qemu device model (the default in 4.3, I think).

My sense is that some of the work done early on to make VGA passthrough work in Xen hasn't kept up with many of the recent changes in the project.

Offline

#42 2013-05-19 21:01:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:
nbhs wrote:

This is the package i use, xen-4.1.5 based on xen-4.2.2 from AUR: xen.tar.gz

Did the Xen people break VGA passthrough in recent versions? I got the sense that they had because I couldn't ever get 4.2 or 4.3 unstable to work completely (several different problems kept coming up), and I see that you're using 4.1.

I know they had a recent security advisory that resulted in tightening up some things and making matters worse for AMD IOMMU users with broken BIOSes. Apparently the switch to xl broke some things too (see the first entry on this page). And the gfx_passthrough option isn't yet supported for the new qemu device model (the default in 4.3, I think).

My sense is that some of the work done early on to make VGA passthrough work in Xen hasn't kept up with many of the recent changes in the project.

Im using xen-4.1 not because xen-4.2+ broke anything, simply because the xl toolstack doesnt support resseting passthru'd devices, what you're refering about not working on broken ivrs tables is this: XSA-36
which is also applied on xen 4.1.5, so i had to hack the xen code to make it work, which can aslo be applied (well it has to be modified a little) to xen-4.2+ ( i tried, it worked)

Included in my build i attached said fix, i have no idea if it breaks working builds, but im sure it works in our boards.

xm dmesg

... (XEN) IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI 0-23 (XEN) IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI 24-55 (XEN) Enabling APIC mode: Flat. Using 2 I/O APICs (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 4414.882 MHz processor. (XEN) Initing memory sharing. (XEN) AMD-Vi: IOMMU 0 Enabled. (XEN) AMD-Vi: Enabling per-device vector maps (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled ...

The other 2 patches included, is a patch to load the bios from your card from here: http://markmail.org/message/7gb7djbmlaxruaai, which is important if you wish to passthrough a host secondary vga device as primary and this patch: http://old-list-archives.xenproject.org … 00705.html, modified by Greg Wettstein (http://lists.xen.org/archives/html/xen- … 00009.html) which allows to passthrough radeon cards as primary devices on xen with a pvops kernel.

Last edited by nbhs (2013-05-19 21:42:05)

Offline

#43 2013-05-20 17:44:19

jelly
Trusted User (TU)
From: /dev/null
Registered: 2008-06-10
Posts: 711

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It would be nice if you converted your first post about the KVM pci passthrough to a wiki article. Anyway nice work, sadly I don't have a VT-d enabled computer sad

Offline

#44 2013-05-20 21:45:15

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jelly wrote:

It would be nice if you converted your first post about the KVM pci passthrough to a wiki article. Anyway nice work, sadly I don't have a VT-d enabled computer sad

I'm not sure since this is experimental, and the success rate depends on a lot of variables, so i believe a thread where users can share experiences and solutions is a better alternative for now.

Offline

#45 2013-05-21 20:27:13

dariocaruso
Member
Registered: 2013-05-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!

I just read about vfio driver, and i'm looking if it's possible to use the secondary (or different) gpu AND plug separate displays to have a multi-seat server, with virtualized different indipendent guests. Can i realize this with vfio driver?

Offline

#46 2013-05-21 22:23:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dariocaruso wrote:

Hi!

I just read about vfio driver, and i'm looking if it's possible to use the secondary (or different) gpu AND plug separate displays to have a multi-seat server, with virtualized differentin dependentguests. Can i realize this with vfio driver?

Well yeah, this is exactly what we're doing, 1 gpu per OS ( or more ), my host linux has 1 GPU and 1 monitor and my windows VM has 1 GPU and 1 monitor, each independent, each with a different usb keyboard and mouse

Offline

#47 2013-05-21 22:57:56

dariocaruso
Member
Registered: 2013-05-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
dariocaruso wrote:

Hi!

I just read about vfio driver, and i'm looking if it's possible to use the secondary (or different) gpu AND plug separate displays to have a multi-seat server, with virtualized differentin dependentguests. Can i realize this with vfio driver?

Well yeah, this is exactly what we're doing, 1 gpu per OS ( or more ), my host linux has 1 GPU and 1 monitor and my windows VM has 1 GPU and 1 monitor, each independent, each with a different usb keyboard and mouse


it seems exactly what i'm looking for years! I want to put in the office a good one server and plug 3 or 4 workstation for administrate differents vm's! Please someone tell me if really i can use autocad with no lag on 2-3 guests at same time (of coourse with enough ram, gpu and all). For years followed and tested spice, but it was not good, like i want, to work confortably with cad/cam programs. Really a good chance to get an usable/0 lag application server for architects! I will start to build mine soon!

Offline

#48 2013-05-21 23:00:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dariocaruso wrote:
nbhs wrote:
dariocaruso wrote:

Hi!

I just read about vfio driver, and i'm looking if it's possible to use the secondary (or different) gpu AND plug separate displays to have a multi-seat server, with virtualized differentin dependentguests. Can i realize this with vfio driver?

Well yeah, this is exactly what we're doing, 1 gpu per OS ( or more ), my host linux has 1 GPU and 1 monitor and my windows VM has 1 GPU and 1 monitor, each independent, each with a different usb keyboard and mouse


it seems exactly what i'm looking for years! I want to put in the office a good one server and plug 3 or 4 workstation for administrate differents vm's! Please someone tell me if really i can use autocad with no lag on 2-3 guests at same time (of coourse with enough ram, gpu and all). For years followed and tested spice, but it was not good, like i want, to work confortably with cad/cam programs. Really a good chance to get an usable/0 lag application server for architects! I will start to build mine soon!

Well, i'm playing metro last light and bioshock infinite @ max settings with no lag, while i watch tv on my arch host, everything works like a dream.

Offline

#49 2013-05-22 03:11:01

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nbhs: This is a great tutorial - thanks!

@dariocaruso: VGA passthrough doesn't exactly replace Spice. The way it works is that you have multiple VGA cards in your PC which can support multiple VMs (one per VGA card). However, you may have a problem using this one PC for say 2-3 people since they would have to sit together very close, because of the limitations of the DVI or HDMI cables as well as the USB cables, all of which have a limitted length they support. There are extenders, though.

Other than that you should get good performance for CAD/CAM and two architects sitting opposite or next to each other (because of the length limit of the cables) should be able to work on one PC.

Offline

#50 2013-05-22 03:15:59

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:
nbhs wrote:

Are you sure the kernel and qemu are patched?

Yes.

I figured out what was wrong! Turns out, it was all my fault. I built my own packages, and I was pulling in the wrong branch from the qemu-vfio git repository. When troubleshooting, I had tried the linux-mainline package posted here, but not the qemu one. Oops. big_smile

VGA reset now works perfectly every time! No more system crashes! Awesome!

Windows Experience Index:
olfKcGz.png

Some useful information I've gleaned:

Input devices:
Basic usability is achievable by using '-usbdevice tablet -vga none' (without '-nographic', so the qemu window will still open) and making the qemu window fullscreen so it catches cursor and keyboard events.

Individual USB device passthrough is problematic for input devices, since at some point you need to tell the host to un-passthru the devices, and your input devices are in the guest. It might be possible to make this work (ssh from guest to host and unbind the input devices), but all the ways I could think of were pretty ugly.

I've had good results passing through an entire USB3 controller (on the motherboard but not part of the chipset). A PCI add-on card would probably work great too, if I had the slots for it. Passing through one of the chipset's USB2 controllers seemed to result in more headers/ports being assigned than I intended, and it was trickier to determine the PCI device->USB root hub->physical port mapping. Your results may vary.

Audio:
Emulated audio was stuttery. My best results were with QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_TIMER_PERIOD=0. AC97 was better than HD Audio for me, although I had to manually download Realtek's AC97 driver and force it in Add New Hardware (Intel's AC97 driver caused a guest BSoD). I think the PCI IDs are slightly off and Windows doesn't recognize the AC97 device automatically.

I'm going to pass through a real sound card device for better audio (and lower CPU usage). Unfortunately, the chipset audio device is tied to other important devices, and the second video card covers up the only PCI slot I have, so I need to buy a PCI Express one. hmm

Disk:
Use the Windows virtio drivers unless you want to pass through a real disk controller. This will increase performance quite a bit.

The default driver is virtio-blk, but the virtio-scsi driver is the only one where I could get Windows to enable TRIM. In Defrag, Windows shows a virtio-scsi disk as a 'thin provisioned disk', which essentially means it knows it's a VM disk device and will pass TRIM commands (intended for VM disk images, but should work for real SSDs as well).

TRIM isn't strictly necessary for VM images or SSDs, but without it, images will take up more space than is necessary and SSD performance will degrade over time.

My current qemu invocation:
In case this is useful to anyone. It's a little ugly right now, as I'm still in the process of figuring out the best way to configure the devices.

QEMU_AUDIO_DRV=alsa \ QEMU_AUDIO_TIMER_PERIOD=0 \ qemu-system-x86_64 -enable-kvm -name Windows8 \ -M q35 -nodefconfig -readconfig /pool/KVM/Windows8/q35-chipset.cfg \ -m 4096 -balloon none -mem-path /dev/hugepages \ -rtc base=localtime \ -cpu host -smp 8,sockets=1,cores=4,threads=2 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device vfio-pci,host=02:00.0,bus=ich9-pcie-port-1,addr=0.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=ich9-pcie-port-1,addr=0.1 \ -device virtio-scsi-pci,id=scsi \ -drive id=ssd,if=none,format=raw,discard=on,cache=none,file=/dev/CorsairVG/Windows \ -drive if=virtio,format=raw,discard=on,file=/pool/KVM/Windows8/WinData.ntfs.img \ -drive id=cdrom,media=cdrom,format=raw,file=/dev/null \ -device scsi-hd,drive=ssd \ -device ide-cd,bus=ide.0,drive=cdrom \ -device vfio-pci,host=06:00.0,bus=ich9-pcie-port-2 \ -boot order=cd,menu=on \ -net nic,model=virtio,macaddr=00:55:aa:00:00:01 -net bridge,br=vm_br \ -soundhw hda,ac97 \ -usbdevice tablet \ >$LOG 2>$LOG

Offline

#51 2013-05-22 03:27:31

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Powerhouse

Thanks a lot man!

@ Jgott

Great!

Audio: I managed to fix all the audio scraching/stutter by disabling nested pages and rebuilding my kernel ( see my second post on first page about performance numbers ), with audio passthrough using QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128, and i also had a problem with my headset but its fixed now with the new kernel config.

DISK: To tell you the truth i havent even tried virtio-blk since i just pass my second sata controller instead of using the emulated one, so it boots directly from the connected disk.

I also had some timing issues playing bioshock infinite, i solved it by following this guide: https://access.redhat.com/site/document … ement.html

EDIT: BTW im using cpuset cgroup and taskset to pin the 6 qemu threads to each vcpu, so in total there's 7 threads + 1 if audio passthrough is enabled,  the main qemu thread + audio thread is running on cpus [0-1] while the vcpus threads run on cpus [2-7]

Last edited by nbhs (2013-05-22 04:46:13)

Offline

#52 2013-05-22 03:42:24

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I managed to fix all the audio scraching/stutter by disabling nested pages and rebuilding my kernel ( see my second post on first page about performance numbers )

Interesting that nested pages makes audio performance worse; I never would have guessed. I'll have to give this a try.

nbhs wrote:

EDIT: BTW im using cpuset cgroup and taskset to pin the 6 qemu threads to each vcpu, so in total there's 7 threads + 1 if audio passthrough is enabled,  the main qemu thread + audio thread is running on cpus [0-1] while the vcpus threads run on cpus [2-7]

I wasn't setting CPU affinity before, but suddenly I can see why this matters, since qemu emulated devices (e.g. audio) are competing with the VM guest for CPU time. I'll have to try this, too.

Thanks for the tips. I should have looked at the performance tuning parts sooner. tongue

Offline

#53 2013-05-22 03:51:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I sugest you use cgroups and move all the (movable) host threads to a couple of cores, and dedicate the rest to the vm, that way, even if your building a kernel on the host it wont interfere with the vm, you can create the cgroups manually or you can also use cset

https://code.google.com/p/cpuset/
https://rt.wiki.kernel.org/index.php/Cp … y/tutorial

You might need to patch the lastest version of cset (1.5.6)

EDIT: Patch here:

cset.py.patch

My VM script using CSET:

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) QEMU=/usr/bin/qemu-system-x86_64 CSET=/usr/bin/cset VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm" MEM="8192" MEM_PATH="/dev/hugepages" CPU="host" BIOS="/usr/share/qemu/bios.bin" NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" USB_DEVICES="tablet" PCI_DEVICES="00:11.0 04:00.0 05:00.0 06:00.0" SND=true SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" GPU="07:00.0" GPU_SND="07:00.1" VBIOS="/usr/local/share/qemu/vgabios-6950.bin" HDD="" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1,rombar=0,romfile=$VBIOS" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEMORY+="-m $MEM" if [ -n "$MEM_PATH" ]; then MEMORY+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEMORY+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi sync && echo 1 > /proc/sys/vm/drop_caches & wait HUGEPAGES_NR="$(($MEM/$(($(grep Hugepagesize /proc/meminfo | awk '{print $2}')/1024))))" echo $HUGEPAGES_NR > /proc/sys/vm/nr_hugepages if [ $HUGEPAGES_NR -eq $(cat /proc/sys/vm/nr_hugepages) ]; then $CSET shield -c $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) set -m $QEMU $CONFIG $BIOS $CPU $MEMORY $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then $CSET shield --shield --pid $t taskset -pc $i $t let i++ fi fi done break fi done ) & fg %1 wait $CSET shield --reset echo 0 > /proc/sys/vm/nr_hugepages echo "All Done!!" else echo "Cannot reserve memory!" fi

Last edited by nbhs (2013-05-22 04:27:31)

Offline

#54 2013-05-22 14:53:30

GreatEmerald
Member
Registered: 2013-05-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I attempted to get this working as well. I have an integrated Intel HD Graphics 2500 (for the host) and a dedicated NVIDIA GeForce GTX 660 (for the guest).

I installed QEMU 1.5, kernel 3.9 (the VFIO components are modularised) and set intel_iommu=on in the boot options. Then I disabled everything that used NVIDIA (blacklisted, set libGL and libglx to point to Mesa, removed the paths from ldconfig.d, set xorg.conf.d to load the intel drivers). Then executed this:

echo 1 | sudo tee /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts

Then used your script to bind the two parts of the graphics card to vfio. This had an effect:

[ 5517.777364] VFIO - User Level meta-driver version: 0.3 [ 5692.736558] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO0: (unknown bits 0x00008000) [ 5692.736567] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO0: ch 1 [DRM] subc 0 mthd 0x0000 data 0x00000000 [ 5695.738853] nouveau E[ DRM] failed to idle channel 0xcccc0000 [DRM] [ 5695.741961] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO2: (unknown bits 0x00008000) [ 5695.741965] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO2: ch 0 [DRM] subc 0 mthd 0x0000 data 0x00000000 [ 5698.744410] nouveau E[ DRM] failed to idle channel 0xcccc0001 [DRM] [ 5698.748200] [TTM] Finalizing pool allocator [ 5698.748203] [TTM] Finalizing DMA pool allocator [ 5698.748217] [TTM] Zone kernel: Used memory at exit: 0 kiB [ 5698.748219] [TTM] Zone dma32: Used memory at exit: 0 kiB

I also added the systemd unit file and enabled it, but haven't restarted just yet.

Finally, I executed this:

pkexec qemu-system-x86_64 --enable-kvm -M q35 -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -bios /home/dainius/seabios/out/bios.bin

The reason why it's using pkexec is that using sudo makes it fail due to missing display, kdesu eats all the stdout/stderr (even when passed the -t option), and qemu-system seemingly ignores the -D option, which leaves me with only pkexec that can actually give me a GUI and show debugging information at the same time (that required me to fiddle with policy settings, since you need to explicitly add a policy that enables org.freedesktop.policykit.exec.allow_gui for each GUI application you want to pkexec).

The output of the command is:

Warning: default mac address being used, creating potential for address conflict qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset

And dmesg gives me:

[11151.655299] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [11151.717650] vfio_ecap_init: 0000:01:00.1 hiding ecap 0x0@0x100

I get the QEMU window, showing "This VM has no graphic display device". When I connect my monitor to the GTX 660's DVI ports, I see no output at all (the screen goes into sleep mode).

So, I'm out of ideas right now. Maybe I should give pci-assign a try as well, although I'm not yet sure how it works.

I'm also not entirely sure how the whole passthrough is supposed to work as well. Does SeaBIOS see my graphics card as something it needs to initialise and use, so in essence the graphics card is supposed to think that it's being initialised in a clean boot process? But doesn't it generally need something to be connected to the card during that process, so that it could get EDID data and such? I have only one monitor here, so the only way to do so would be to connect the monitor before launching the virtual machine...

Offline

#55 2013-05-22 16:32:06

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,291

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can you elaborate on exactly what you're doing with audio? In the first post you PCI-passthrough both your video card and the associated audio device. But you also mention

AUDIO PASSTHROUGH:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Isn't the second method emulation rather than passthrough?

And in any case, is the second method what you're talking about wrt cpu pinning in post 51?


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#56 2013-05-22 20:36:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GreatEmerald wrote:

Then used your script to bind the two parts of the graphics card to vfio. This had an effect:

[ 5517.777364] VFIO - User Level meta-driver version: 0.3 [ 5692.736558] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO0: (unknown bits 0x00008000) [ 5692.736567] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO0: ch 1 [DRM] subc 0 mthd 0x0000 data 0x00000000 [ 5695.738853] nouveau E[ DRM] failed to idle channel 0xcccc0000 [DRM] [ 5695.741961] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO2: (unknown bits 0x00008000) [ 5695.741965] nouveau E[ PFIFO][0000:01:00.0] SUBFIFO2: ch 0 [DRM] subc 0 mthd 0x0000 data 0x00000000 [ 5698.744410] nouveau E[ DRM] failed to idle channel 0xcccc0001 [DRM] [ 5698.748200] [TTM] Finalizing pool allocator [ 5698.748203] [TTM] Finalizing DMA pool allocator [ 5698.748217] [TTM] Zone kernel: Used memory at exit: 0 kiB [ 5698.748219] [TTM] Zone dma32: Used memory at exit: 0 kiB

You need to block nouveau modules as well

Also try this:

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

Also make sure seabios is patched as described on my guide, or use the seabios-git package i provided.

Other users reported it not working when using the IGP as primary device, you might want to try with 2 discrete cards and IGP off.

EDIT: This guy got some 660 cards working,unfortunately his page is down so i'll link webcache http://webcache.googleusercontent.com/s … e&ie=UTF-8

Last edited by nbhs (2013-05-22 21:00:42)

Offline

#57 2013-05-22 20:44:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alphaniner wrote:

Can you elaborate on exactly what you're doing with audio? In the first post you PCI-passthrough both your video card and the associated audio device. But you also mention

AUDIO PASSTHROUGH:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Isn't the second method emulation rather than passthrough?

And in any case, is the second method what you're talking about wrt cpu pinning in post 51?

Yes this is audio emulation, i called it passthrough because it pass the audio from the vm to the host, it creates an emulated sound card inside the vm, but im also using a usb headset connected to my usb3 ports which are passed thru to my vm.

Last edited by nbhs (2013-05-22 21:40:46)

Offline

#58 2013-05-22 21:01:32

GreatEmerald
Member
Registered: 2013-05-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You need to block nouveau modules as well

Also try this:

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

Also make sure seabios is patched as described on my guide, or use the seabios-git package i provided.

Other users reported it not working when using the IGP as primary device, you might want to try with 2 discrete cards and IGP off.

Well, nouveau never was loaded. It's not on lsmod. So I don't know why it's mentioned in dmesg. Just to make sure, I added it to blacklist as well, but I'll have to restart to try it again.

And yes, I did load kvm_intel like that, I just forgot to mention it.

As for SeaBIOS, the patch actually isn't needed, because it's been merged upstream already: http://code.coreboot.org/p/seabios/sour … 97fbab821/ And yes, I am using SeaBIOS built from from git.

As for two dedicated cards, unfortunately I don't have such a luxury... I could try reversing the cards, so that the host uses NVIDIA and the guest uses Intel, but that's about it.

Offline

#59 2013-05-22 21:03:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It hasnt been merged here git.qemu.org/seabios.git, which is the one we need to use, so please use my package.

Offline

#60 2013-05-22 21:52:52

GreatEmerald
Member
Registered: 2013-05-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

It hasnt been merged here git.qemu.org/seabios.git, which is the one we need to use, so please use my package.

The QEMU SeaBIOS repository is an exact clone of the upstream SeaBIOS repository, just lagging behind (you can diff the git logs to make sure). So you can literally get the same result by fetching upstream and then doing a `git reset --hard f465e1ec1b01826100c92b890487a2ab09bfe2c1`. So unless the new commits actually break QEMU (unlikely, given that it hasn't broken it in a single commit up to now, unless upstream git is prone to breaking before it gets fixed up down the road), there really is no reason not to use the upstream SeaBIOS repository.

I suppose I can reset and cherry-pick the needed commit if you are so inclined. I just highly doubt there will be any difference.

Offline

#61 2013-05-22 21:59:52

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GreatEmerald wrote:
nbhs wrote:

It hasnt been merged here git.qemu.org/seabios.git, which is the one we need to use, so please use my package.

The QEMU SeaBIOS repository is an exact clone of the upstream SeaBIOS repository, just lagging behind (you can diff the git logs to make sure). So you can literally get the same result by fetching upstream and then doing a `git reset --hard f465e1ec1b01826100c92b890487a2ab09bfe2c1`. So unless the new commits actually break QEMU (unlikely, given that it hasn't broken it in a single commit up to now, unless upstream git is prone to breaking before it gets fixed up down the road), there really is no reason not to use the upstream SeaBIOS repository.

I suppose I can reset and cherry-pick the needed commit if you are so inclined. I just highly doubt there will be any difference.

Yes i know, thing is upstream seabios does break my system, i cant boot windows with it, you should be able to get output on your card tho, but that might be broken too.

Last edited by nbhs (2013-05-22 22:02:34)

Offline

#62 2013-05-24 00:46:06

Kamek
Member
Registered: 2013-05-24
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to share a single GPU between the host and the guest ? I only have one GTX 670 in my rig, but I'm sick and tired of rebooting every time I want to play a game.

Offline

#63 2013-05-24 00:59:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kamek wrote:

Is there a way to share a single GPU between the host and the guest ? I only have one GTX 670 in my rig, but I'm sick and tired of rebooting every time I want to play a game.

Not using this method, if you had a radeon card you could use kvm pci-assign and pass it as secondary but on NV cards this is the only reliable method (unless you're willing to try xen, and its counteless unreliable hacks)

Offline

#64 2013-05-24 18:37:10

darkspider
Member
Registered: 2013-05-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey nbhs,
I followed your guide and got everything running, but I have an issue with the pci-passthrough of an USB Controller. My board has an Etron USB 3.0 controller I never use, so I wanted to pass it to my Win7 guest.
Here is the output of lspci:

03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

So I use "vfio-bind 0000:03:00.0 0000:05:00.0" and add "-device vfio-pci,host=03:00.0,bus=pcie.0 -device vfio-pci,host=05:00.0,bus=pcie.0" to the start command of qemu and plug my mouse and keyboard into the 2 USB 3.0 ports.
But both aren't even powered (Razer-logo doesn't fade in and out and keyboard keys aren't lit) and the guest doesn't recognize them, same happens if I use any other USB Controller that lspci spits out. The mainboard I am using is an Gigabyte 990XA-UD3, which seems to have a quite good implementation of IOMMU. So, if I pass my mouse or my keyboard using the usb-function of qemu, it works, but of course it isn't available for the host anymore and I'd like to use a KVM switch to switch between the guest and the host.
Any suggestions on how I can fix the problem?

Offline

#65 2013-05-24 21:20:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

darkspider wrote:

Hey nbhs,
I followed your guide and got everything running, but I have an issue with the pci-passthrough of an USB Controller. My board has an Etron USB 3.0 controller I never use, so I wanted to pass it to my Win7 guest.
Here is the output of lspci:

03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

So I use "vfio-bind 0000:03:00.0 0000:05:00.0" and add "-device vfio-pci,host=03:00.0,bus=pcie.0 -device vfio-pci,host=05:00.0,bus=pcie.0" to the start command of qemu and plug my mouse and keyboard into the 2 USB 3.0 ports.
But both aren't even powered (Razer-logo doesn't fade in and out and keyboard keys aren't lit) and the guest doesn't recognize them, same happens if I use any other USB Controller that lspci spits out. The mainboard I am using is an Gigabyte 990XA-UD3, which seems to have a quite good implementation of IOMMU. So, if I pass my mouse or my keyboard using the usb-function of qemu, it works, but of course it isn't available for the host anymore and I'd like to use a KVM switch to switch between the guest and the host.
Any suggestions on how I can fix the problem?

Great man what card did you manage to pass through?

Make sure every device on /sys/bus/pci/drivers/vfio-pci/0000\:03\:00.0/iommu_group/devices/ and  /sys/bus/pci/drivers/vfio-pci/0000\:05\:00.0/iommu_group/devices/ are passed through

Or can try passing through usb2 ports, but they normally come in pairs, so you have to pass both for example:

lspci

00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller

You might have to pass it through as a single multifunction device.

Last edited by nbhs (2013-05-24 21:27:05)

Offline

#66 2013-05-24 21:45:53

darkspider
Member
Registered: 2013-05-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank your for your quick reply and your great tutorial! smile
I managed to pass through a Nvidia GeForce GTX 460 while the host is running on an old Quadro FX 1700 (nouveau) I had laying around.
I saw a YouTube Video by Ubisoft, who benchmarked a 2 VM-system running on 2 GTX 460s with the only difference that they were using XEN. So I thought I'd try this out, but XEN didn't really seem to be what I was looking for and then I found your guide and yeah, it worked (except the USB part, which I'll try later) smile.

So, let's come to the USB part:
I tried your second example before in which I exactly did what you described, but it would turn out the same way as my USB3 controller approach. Only thing I didn't try is to "pass it through as a single multifunction device". Can you please explain how this works?
Thank you smile

Edit: Just remembered this: your systemd service contains an error (or typo):

ExecStart=-/usr/bin/vfio-pci $DEVICES

  should say vfio-bind, because the script in /usr/bin is named vfio-bind in your tutorial smile

Last edited by darkspider (2013-05-24 21:53:33)

Offline

#67 2013-05-24 21:53:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

darkspider wrote:

Thank your for your quick reply and your great tutorial! smile
I managed to pass through a Nvidia GeForce GTX 460 while the host is running on an old Quadro FX 1700 (nouveau) I had laying around.
I saw a YouTube Video by Ubisoft, who benchmarked a 2 VM-system running on 2 GTX 460s with the only difference that they were using XEN. So I thought I'd try this out, but XEN didn't really seem to be what I was looking for and then I found your guide and yeah, it worked (except the USB part, which I'll try later) smile.

So, let's come to the USB part:
I tried your second example before in which I exactly did what you described, but it would turn out the same way as my USB3 controller approach. Only thing I didn't try is to "pass it through as a single multifunction device". Can you please explain how this works?
Thank you smile

Yeah for example for my usb 2 ports i would do something like this:

qemu-system-x86_64 -device vfio-pci,host=00:13.0,multifunction=on,addr=13.0 \ -device vfio-pci,host=00:13.2,addr=13.2

EDIT: Thanks didnt notice the typo ill correct it.

Last edited by nbhs (2013-05-24 21:58:46)

Offline

#68 2013-05-24 22:08:14

darkspider
Member
Registered: 2013-05-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Typing this from my Win7 VM, pass through of USB2 ports works, the Etron controller doesn't want to work. "multifunction" won't do the trick for it, but it works like a charm for a pair of USB2 ports. Thanks a lot!
My last step is now getting audio to work (haven't tried it before, I was happy enough to see something on my screen big_smile )

Offline

#69 2013-05-24 22:14:34

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Perhaps this "etron" controller needs an option rom to work, if you manage to dump it/find it somewhere (its most likely included in your mobo bios, you could extract it from there) you can pass it to the vm with

-option-rom /path/to/rom

Audio is painful, see my second post on first page on how to optimize for my amd cpu and also my first post in this page,  i had to disable nested pages even for a usb headset  to work without stuttering/scratchy sound.

Last edited by nbhs (2013-05-24 22:23:29)

Offline

#70 2013-05-24 22:57:42

darkspider
Member
Registered: 2013-05-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guess what? Audio's working perfectly with

 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

and

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

just like how you mentioned in your first post, no stuttering, no scratching, everything is perfectly fine smile!

I have extracted a recent version of my BIOS, but I can't find any files that may be related to the Etron USB 3.0 Controller (there are some "GROUP ROM"s, like tgroup.bin, t1group.bin up to "t4" and ggroup, ffgroup) and I honestly never did something like this (extracting a BIOS) before, also Google doesn't deliver anything about the controller's ROM (just many forum entries about issues with this controller in normal use i.e. driver issues)

Offline

#71 2013-05-24 23:04:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

darkspider wrote:

Guess what? Audio's working perfectly with

 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

and

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

just like how you mentioned in your first post, no stuttering, no scratching, everything is perfectly fine smile!

I have extracted a recent version of my BIOS, but I can't find any files that may be related to the Etron USB 3.0 Controller (there are some "GROUP ROM"s, like tgroup.bin, t1group.bin up to "t4" and ggroup, ffgroup) and I honestly never did something like this (extracting a BIOS) before, also Google doesn't deliver anything about the controller's ROM (just many forum entries about issues with this controller in normal use i.e. driver issues)

Well if you encounter, audio problems playing games, take a look at my second post on first page, like you i had no problems with audio, until i started playing metro last light, and then all kind of problems started to arrise, like scratchy audio, stuttering in sound and fps, i solved all that doing the steps i mentioned, and rebuilding my kernel with the config file i attached

Offline

#72 2013-05-25 13:29:22

GreatEmerald
Member
Registered: 2013-05-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I tried a few other things, like blacklisting nouveau, setting pci_stub to claim the graphics card and cherry-picking SeaBIOS. Still no go. I get this in dmesg:

[19144.183320] VFIO - User Level meta-driver version: 0.3 [19433.567127] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [19433.567278] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [19433.641853] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002) [19433.641964] vfio_ecap_init: 0000:01:00.1 hiding ecap 0x0@0x100

And this from the console:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset

I also tried the commands suggested at the FrankD page. But it resulted in QEMU eating one core or outright dying with this message followed by a backtrace:

KVM internal error. Suberror: 1 emulation failure

The card still shows no output either way.

Offline

#73 2013-05-25 16:44:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GreatEmerald wrote:

So I tried a few other things, like blacklisting nouveau, setting pci_stub to claim the graphics card and cherry-picking SeaBIOS. Still no go. I get this in dmesg:

[19144.183320] VFIO - User Level meta-driver version: 0.3 [19433.567127] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [19433.567278] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [19433.641853] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002) [19433.641964] vfio_ecap_init: 0000:01:00.1 hiding ecap 0x0@0x100

And this from the console:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset

I also tried the commands suggested at the FrankD page. But it resulted in QEMU eating one core or outright dying with this message followed by a backtrace:

KVM internal error. Suberror: 1 emulation failure

The card still shows no output either way.

You should fill a bug report @qemu-devel see: http://lists.gnu.org/archive/html/qemu- … 00432.html

Offline

#74 2013-05-25 17:05:32

darkspider
Member
Registered: 2013-05-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs, thanks for your help, you're very kind. smile
Could you elaborate on your kernel config? What's the purpose of using a 1000HZ timer frequency and a voluntary Preemption Model? I've seen some posts of people, who recommend this settings for a Desktop setup, because it reduces latency, is this the reason?
Since I'm pretty much new to this kind of virtualization, I wonder if those kernel settings have an impact on the usability of the host system. So, would you recommend booting the modified kernel on a daily basis for productive work, or just if virtualization is needed? I'd say there isn't that much difference to the "vanilla" Arch kernel, so there's is nothing that would really speak against using the modified kernel even if I don't intend to use virtualization every time, right?

Offline

#75 2013-05-25 17:19:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

darkspider wrote:

nbhs, thanks for your help, you're very kind. smile
Could you elaborate on your kernel config? What's the purpose of using a 1000HZ timer frequency and a voluntary Preemption Model? I've seen some posts of people, who recommend this settings for a Desktop setup, because it reduces latency, is this the reason?
Since I'm pretty much new to this kind of virtualization, I wonder if those kernel settings have an impact on the usability of the host system. So, would you recommend booting the modified kernel on a daily basis for productive work, or just if virtualization is needed? I'd say there isn't that much difference to the "vanilla" Arch kernel, so there's is nothing that would really speak against using the modified kernel even if I don't intend to use virtualization every time, right?

Im using 1000hz instead of dynticks on my system because running the vm on cpus != of 0 would result in my vm hanging specially gaming, and im using voluntary preempt because without it my vm feels sluggish with npt disabled.

And yeah you shouldnt notice any issue on your host using my kernel config.

Last edited by nbhs (2013-05-25 17:21:19)

Offline

#76 2013-05-26 02:24:21

apocolypse600
Member
Registered: 2013-05-26
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Firstly, cheers for the tutorial. That must have taken awhile to write, and the effort is appreciated.

I've been looking into setting up a VGA passthrough setup under Xen for my next build for awhile now. I'm trying to work out what the advantages of this setup are over a setup under Xen. From what I've seen thrown around in different places, the main advantages are:

  • a less hacky setup for the passthrough, requiring less patches (once all this stuff gets released)

  • better support for NVIDIA cards? I've seen a few people say this, but nothing official to confirm it.

  • KVM receives more support than Xen, mainly due to Red Hat

  • Xen has a (relatively small) impact on the speed of the host, whereas KVM has no impact

  • not having to mess around with xl vs xm under Xen

Am I missing anything? I've seen a lot of people getting excited about using KVM over Xen for VGA passthrough, but not many people saying why they are so excited.

Offline

#77 2013-05-26 02:39:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:

Firstly, cheers for the tutorial. That must have taken awhile to write, and the effort is appreciated.

I've been looking into setting up a VGA passthrough setup under Xen for my next build for awhile now. I'm trying to work out what the advantages of this setup are over a setup under Xen. From what I've seen thrown around in different places, the main advantages are:

  • a less hacky setup for the passthrough, requiring less patches (once all this stuff gets released)

  • better support for NVIDIA cards? I've seen a few people say this, but nothing official to confirm it.

  • KVM receives more support than Xen, mainly due to Red Hat

  • Xen has a (relatively small) impact on the speed of the host, whereas KVM has no impact

  • not having to mess around with xl vs xm under Xen

Am I missing anything? I've seen a lot of people getting excited about using KVM over Xen for VGA passthrough, but not many people saying why they are so excited.

Yes, xen only officialy supports quadro cards, everything else requires hacks, that might or might not work, its not in the main xen tree, and its not being maintained/developed, while effort (still experimental) on kvm/qemu is being made to support
non-professional gpus out of the box.

Also, a little outdated but still relevant:

http://chucknology.com/2012/02/02/kvm-i … en-is-not/

EDIT: Forgot to add, all the hacks needed to get vga passthrough on xen, only work on the old "qemu-dm" (qemu 0.10.2) code they ship with, which is like 3-4 years old if i'm not mistaken, so things like audio passthrough wont work on windows x64 for example.

Last edited by nbhs (2013-05-26 03:06:07)

Offline

#78 2013-05-26 03:18:22

apocolypse600
Member
Registered: 2013-05-26
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
apocolypse600 wrote:

Firstly, cheers for the tutorial. That must have taken awhile to write, and the effort is appreciated.

I've been looking into setting up a VGA passthrough setup under Xen for my next build for awhile now. I'm trying to work out what the advantages of this setup are over a setup under Xen. From what I've seen thrown around in different places, the main advantages are:

  • a less hacky setup for the passthrough, requiring less patches (once all this stuff gets released)

  • better support for NVIDIA cards? I've seen a few people say this, but nothing official to confirm it.

  • KVM receives more support than Xen, mainly due to Red Hat

  • Xen has a (relatively small) impact on the speed of the host, whereas KVM has no impact

  • not having to mess around with xl vs xm under Xen

Am I missing anything? I've seen a lot of people getting excited about using KVM over Xen for VGA passthrough, but not many people saying why they are so excited.

Yes, xen only officialy supports quadro cards, everything else requires hacks, that might or might not work, its not in the main xen tree, and its not being maintained/developed, while effort (still experimental) on kvm/qemu is being made to support
non-professional gpus out of the box.

Also, a little outdated but still relevant:

http://chucknology.com/2012/02/02/kvm-i … en-is-not/

EDIT: Forgot to add, all the hacks needed to get vga passthrough on xen, only work on the old "qemu-dm" (qemu 0.10.2) code they ship with, which is like 3-4 years old if i'm not mistaken, so things like audio passthrough wont work on windows x64 for example.

Yeah, I remember reading about the old version of qemu, and I think I may have read that article before (the diagrams looks familiar). The comments on the article are really hostile though, but as far as I understand, it's a pretty accurate portrayal of KVM vs Xen.

Offline

#79 2013-05-26 15:20:33

shadey
Member
Registered: 2013-05-19
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:

Audio:
Emulated audio was stuttery. My best results were with QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_TIMER_PERIOD=0. AC97 was better than HD Audio for me, although I had to manually download Realtek's AC97 driver and force it in Add New Hardware (Intel's AC97 driver caused a guest BSoD). I think the PCI IDs are slightly off and Windows doesn't recognize the AC97 device automatically.

I'm going to pass through a real sound card device for better audio (and lower CPU usage). Unfortunately, the chipset audio device is tied to other important devices, and the second video card covers up the only PCI slot I have, so I need to buy a PCI Express one. hmm

I would suggest for going for a "native PCIe" sound card like the Soundblaster Z, most of the newer Auzentech stuff, ASUS "Phoebus" (not the other ASUS stuff) or any other brand that uses a PCIe chipset and not a PCIe to PCI bridge. I have spent a long time trying to get my PCI sound card passed through to a VM and have now given up on it.

With the card in my mobo PCI slot (ASMedia PCIe to PCI bridge in the same IOMMU group as all the non-intel stuff on the board) the sound card didn't play nice with intel_iommu=on, hard locking the system whenever it was initialised. I then put it in a PLX 8111 PCIe to PCI card in a 1x slot and got drivers fully installed on the passed through guest but no sound in the guest and it was doing weird stuff to my system. The PLX 8111/8112 bridge is generally what is used by those older "non native" PCIe sound cards.

Passing through the onboard sound was no problem but now I am looking to get a proper PCIe sound card.

Offline

#80 2013-05-26 16:01:36

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I try to get vga-passthrough to work on my system at the moment, but the second graphics card doesn't output anything and qemu freezes.

My setup is…
hardware:
Mainboard: AsRock 990FX Extreme4
CPU: AMD FX-8350
GPU0 (host): Gigabyte GeForce GTX 650
GPU1 (guest): Sapphire Radeon HD 5670
RAM: G.SKILL F3-10666CL9-8GBXL

software:
linux-mainline (pkg from the original post updated to 3.9.4)
qemu (pkg from original post)
seabios-git (pkg from original post adapted to new VCS syntax/rel1.7.286gf465e1e-1)

lspci output

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) 00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E) 00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GTX 650] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK107 HDMI Audio Controller (rev a1) 02:00.0 IDE interface: Marvell Technology Group Ltd. 88SE91A0 SATA 6Gb/s Controller (rev 12) 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 05:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood XT [Radeon HD 5670] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] 08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 01)

dmesg output

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.9.4-1-mainline (andy@desk) (gcc version 4.8.0 20130502 (prerelease) (GCC) ) #1 SMP PREEMPT Sun May 26 16:09:27 CEST 2013 [ 0.000000] Command line: ../vmlinuz-linux-mainline ../vmlinuz-linux-mainline root=/dev/disk/by-id/ata-ADATA_SP900_02B20081500200001033-part3 rootflags=subvol=root quiet ro security=tomoyo irqpoll initrd=../initramfs-linux-mainline.img [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009cfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000009df04fff] usable [ 0.000000] BIOS-e820: [mem 0x000000009df05000-0x000000009e4e2fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000009e4e3000-0x000000009e8d5fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000009e8d6000-0x000000009ee28fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000009ee29000-0x000000009ee29fff] usable [ 0.000000] BIOS-e820: [mem 0x000000009ee2a000-0x000000009f02ffff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000009f030000-0x000000009f452fff] usable [ 0.000000] BIOS-e820: [mem 0x000000009f453000-0x000000009f7f3fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000009f7f4000-0x000000009f7fffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec20000-0x00000000fec20fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed61000-0x00000000fed70fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fef00000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100001000-0x000000025effffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./990FX Extreme4, BIOS P2.00 10/15/2012 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] No AGP bridge found [ 0.000000] e820: last_pfn = 0x25f000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF write-through [ 0.000000] C0000-D1FFF write-protect [ 0.000000] D2000-EAFFF uncachable [ 0.000000] EB000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask FFFF80000000 write-back [ 0.000000] 1 base 000080000000 mask FFFFE0000000 write-back [ 0.000000] 2 base 00009F800000 mask FFFFFF800000 uncachable [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] TOM2: 000000025f000000 aka 9712M [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: update [mem 0x9f800000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0x9f800 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd9e0-0x000fd9ef] mapped at [ffff8800000fd9e0] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01b0e000, 0x01b0efff] PGTABLE [ 0.000000] BRK [0x01b0f000, 0x01b0ffff] PGTABLE [ 0.000000] BRK [0x01b10000, 0x01b10fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x25ee00000-0x25effffff] [ 0.000000] [mem 0x25ee00000-0x25effffff] page 2M [ 0.000000] BRK [0x01b11000, 0x01b11fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x25c000000-0x25edfffff] [ 0.000000] [mem 0x25c000000-0x25edfffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x200000000-0x25bffffff] [ 0.000000] [mem 0x200000000-0x23fffffff] page 1G [ 0.000000] [mem 0x240000000-0x25bffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0x9df04fff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0x9ddfffff] page 2M [ 0.000000] [mem 0x9de00000-0x9df04fff] page 4k [ 0.000000] init_memory_mapping: [mem 0x9ee29000-0x9ee29fff] [ 0.000000] [mem 0x9ee29000-0x9ee29fff] page 4k [ 0.000000] BRK [0x01b12000, 0x01b12fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x9f030000-0x9f452fff] [ 0.000000] [mem 0x9f030000-0x9f1fffff] page 4k [ 0.000000] [mem 0x9f200000-0x9f3fffff] page 2M [ 0.000000] [mem 0x9f400000-0x9f452fff] page 4k [ 0.000000] init_memory_mapping: [mem 0x9f7f4000-0x9f7fffff] [ 0.000000] [mem 0x9f7f4000-0x9f7fffff] page 4k [ 0.000000] init_memory_mapping: [mem 0x100001000-0x1ffffffff] [ 0.000000] [mem 0x100001000-0x1001fffff] page 4k [ 0.000000] [mem 0x100200000-0x13fffffff] page 2M [ 0.000000] [mem 0x140000000-0x1ffffffff] page 1G [ 0.000000] RAMDISK: [mem 0x7fa42000-0x7fffffff] [ 0.000000] ACPI: RSDP 00000000000f0490 00024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 000000009e8c5078 0006C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 000000009e8cb600 0010C (v05 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130117/tbfadt-599) [ 0.000000] ACPI: DSDT 000000009e8c5178 06483 (v02 ALASKA A M I 00000000 INTL 20051117) [ 0.000000] ACPI: FACS 000000009e8cf080 00040 [ 0.000000] ACPI: APIC 000000009e8cb710 0009E (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 000000009e8cb7b0 00044 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: MCFG 000000009e8cb7f8 0003C (v01 ALASKA A M I 01072009 MSFT 00010013) [ 0.000000] ACPI: AAFT 000000009e8cb838 0008A (v01 ALASKA OEMAAFT 01072009 MSFT 00000097) [ 0.000000] ACPI: HPET 000000009e8cb8c8 00038 (v01 ALASKA A M I 01072009 AMI 00000005) [ 0.000000] ACPI: SSDT 000000009e8cb900 01714 (v01 AMD POWERNOW 00000001 AMD 00000001) [ 0.000000] ACPI: IVRS 000000009e8cd018 000E8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: BGRT 000000009e8cd100 00038 (v00 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000025effffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x25effffff] [ 0.000000] NODE_DATA [mem 0x25eff4000-0x25eff8fff] [ 0.000000] [ffffea0000000000-ffffea00097fffff] PMD -> [ffff880256600000-ffff88025e5fffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x25effffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x9df04fff] [ 0.000000] node 0: [mem 0x9ee29000-0x9ee29fff] [ 0.000000] node 0: [mem 0x9f030000-0x9f452fff] [ 0.000000] node 0: [mem 0x9f7f4000-0x9f7fffff] [ 0.000000] node 0: [mem 0x100001000-0x25effffff] [ 0.000000] On node 0 totalpages: 2085584 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 10061 pages used for memmap [ 0.000000] DMA32 zone: 643893 pages, LIFO batch:31 [ 0.000000] Normal zone: 22464 pages used for memmap [ 0.000000] Normal zone: 1437695 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x808 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x17] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI 24-55 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ2 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x43538210 base: 0xfed00000 [ 0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs [ 0.000000] nr_irqs_gsi: 72 [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000 [ 0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000 [ 0.000000] PM: Registered nosave memory: 000000009df05000 - 000000009e4e3000 [ 0.000000] PM: Registered nosave memory: 000000009e4e3000 - 000000009e8d6000 [ 0.000000] PM: Registered nosave memory: 000000009e8d6000 - 000000009ee29000 [ 0.000000] PM: Registered nosave memory: 000000009ee2a000 - 000000009f030000 [ 0.000000] PM: Registered nosave memory: 000000009f453000 - 000000009f7f4000 [ 0.000000] PM: Registered nosave memory: 000000009f800000 - 00000000fec00000 [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000 [ 0.000000] PM: Registered nosave memory: 00000000fec01000 - 00000000fec10000 [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fec11000 [ 0.000000] PM: Registered nosave memory: 00000000fec11000 - 00000000fec20000 [ 0.000000] PM: Registered nosave memory: 00000000fec20000 - 00000000fec21000 [ 0.000000] PM: Registered nosave memory: 00000000fec21000 - 00000000fed00000 [ 0.000000] PM: Registered nosave memory: 00000000fed00000 - 00000000fed01000 [ 0.000000] PM: Registered nosave memory: 00000000fed01000 - 00000000fed61000 [ 0.000000] PM: Registered nosave memory: 00000000fed61000 - 00000000fed71000 [ 0.000000] PM: Registered nosave memory: 00000000fed71000 - 00000000fed80000 [ 0.000000] PM: Registered nosave memory: 00000000fed80000 - 00000000fed90000 [ 0.000000] PM: Registered nosave memory: 00000000fed90000 - 00000000fef00000 [ 0.000000] PM: Registered nosave memory: 00000000fef00000 - 0000000100000000 [ 0.000000] PM: Registered nosave memory: 0000000100000000 - 0000000100001000 [ 0.000000] e820: [mem 0x9f800000-0xfebfffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:8 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88025ec00000 s85632 r8192 d20864 u262144 [ 0.000000] pcpu-alloc: s85632 r8192 d20864 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2052974 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: ../vmlinuz-linux-mainline ../vmlinuz-linux-mainline root=/dev/disk/by-id/ata-ADATA_SP900_02B20081500200001033-part3 rootflags=subvol=root quiet ro security=tomoyo irqpoll initrd=../initramfs-linux-mainline.img [ 0.000000] Misrouted IRQ fixup and polling support enabled [ 0.000000] This may significantly impact system performance [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] __ex_table already sorted, skipping sort [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Node 0: aperture @ 94000000 size 32 MB [ 0.000000] Aperture pointing to e820 RAM. Ignoring. [ 0.000000] Your BIOS doesn't leave a aperture memory hole [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.000000] This costs you 64 MB of RAM [ 0.000000] Mapping aperture over 65536 KB of RAM @ 94000000 [ 0.000000] PM: Registered nosave memory: 0000000094000000 - 0000000098000000 [ 0.000000] Memory: 8061176k/9945088k available (5131k kernel code, 1602752k absent, 281160k reserved, 3816k data, 1092k init) [ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] Dump stacks of tasks blocking RCU-preempt GP. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8. [ 0.000000] NR_IRQS:4352 nr_irqs:1288 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] allocated 33554432 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.001000] tsc: Detected 3991.652 MHz processor [ 0.000003] Calibrating delay loop (skipped), value calculated using timer frequency.. 7983.30 BogoMIPS (lpj=3991652) [ 0.000005] pid_max: default: 32768 minimum: 301 [ 0.000033] Security Framework initialized [ 0.000043] TOMOYO Linux initialized [ 0.000049] AppArmor: AppArmor disabled by boot time parameter [ 0.000515] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.003115] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.004338] Mount-cache hash table entries: 256 [ 0.004525] Initializing cgroup subsys cpuacct [ 0.004527] Initializing cgroup subsys memory [ 0.004536] Initializing cgroup subsys devices [ 0.004537] Initializing cgroup subsys freezer [ 0.004539] Initializing cgroup subsys net_cls [ 0.004540] Initializing cgroup subsys blkio [ 0.004563] tseg: 009f800000 [ 0.004565] CPU: Physical Processor ID: 0 [ 0.004565] CPU: Processor Core ID: 0 [ 0.004567] mce: CPU supports 7 MCE banks [ 0.004573] LVT offset 1 assigned for vector 0xf9 [ 0.004579] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512 Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512 tlb_flushall_shift: 5 [ 0.004655] Freeing SMP alternatives: 20k freed [ 0.005255] ACPI: Core revision 20130117 [ 0.007025] ACPI: All ACPI Tables successfully acquired [ 0.012122] ftrace: allocating 19520 entries in 77 pages [ 0.092544] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.102565] smpboot: CPU0: AMD FX(tm)-8350 Eight-Core Processor (fam: 15, model: 02, stepping: 00) [ 0.204459] Performance Events: [ 0.204461] perf: AMD core performance counters detected [ 0.204462] perf: AMD northbridge performance counters detected [ 0.204465] AMD PMU driver. [ 0.204466] ... version: 0 [ 0.204467] ... bit width: 48 [ 0.204468] ... generic registers: 10 [ 0.204469] ... value mask: 0000ffffffffffff [ 0.204470] ... max period: 00007fffffffffff [ 0.204470] ... fixed-purpose events: 0 [ 0.204471] ... event mask: 00000000000003ff [ 0.214526] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.216529] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK [ 0.319834] Brought up 8 CPUs [ 0.319838] smpboot: Total of 8 processors activated (63866.43 BogoMIPS) [ 0.329769] devtmpfs: initialized [ 0.331259] PM: Registering ACPI NVS region [mem 0x9e4e3000-0x9e8d5fff] (4141056 bytes) [ 0.331313] PM: Registering ACPI NVS region [mem 0x9ee2a000-0x9f02ffff] (2121728 bytes) [ 0.331974] RTC time: 15:36:07, date: 05/26/13 [ 0.332012] NET: Registered protocol family 16 [ 0.332170] ACPI: bus type PCI registered [ 0.332220] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.332222] PCI: not using MMCONFIG [ 0.332223] PCI: Using configuration type 1 for base access [ 0.332224] PCI: Using configuration type 1 for extended access [ 0.332950] bio: create slab <bio-0> at 0 [ 0.333007] ACPI: Added _OSI(Module Device) [ 0.333008] ACPI: Added _OSI(Processor Device) [ 0.333010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333786] ACPI: EC: Look up EC in DSDT [ 0.334658] ACPI: Executed 3 blocks of module-level executable AML code [ 0.337829] ACPI: Interpreter enabled [ 0.337836] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130117/hwxface-568) [ 0.337840] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130117/hwxface-568) [ 0.337858] ACPI: (supports S0 S3 S4 S5) [ 0.337860] ACPI: Using IOAPIC for interrupt routing [ 0.337991] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.338027] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.351694] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.356496] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356733] PCI host bridge to bus 0000:00 [ 0.356736] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.356738] pci_bus 0000:00: root bus resource [io 0x0000-0x03af] [ 0.356740] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7] [ 0.356741] pci_bus 0000:00: root bus resource [io 0x03b0-0x03df] [ 0.356743] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.356744] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.356746] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff] [ 0.356747] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xffffffff] [ 0.356757] pci 0000:00:00.0: [1002:5a14] type 00 class 0x060000 [ 0.356832] pci 0000:00:00.2: [1002:5a23] type 00 class 0x080600 [ 0.356911] pci 0000:00:02.0: [1002:5a16] type 01 class 0x060400 [ 0.356941] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 0.356960] pci 0000:00:02.0: System wakeup disabled by ACPI [ 0.357004] pci 0000:00:04.0: [1002:5a18] type 01 class 0x060400 [ 0.357033] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold [ 0.357052] pci 0000:00:04.0: System wakeup disabled by ACPI [ 0.357080] pci 0000:00:05.0: [1002:5a19] type 01 class 0x060400 [ 0.357109] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold [ 0.357128] pci 0000:00:05.0: System wakeup disabled by ACPI [ 0.357155] pci 0000:00:06.0: [1002:5a1a] type 01 class 0x060400 [ 0.357184] pci 0000:00:06.0: PME# supported from D0 D3hot D3cold [ 0.357204] pci 0000:00:06.0: System wakeup disabled by ACPI [ 0.357234] pci 0000:00:0a.0: [1002:5a1d] type 01 class 0x060400 [ 0.357263] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold [ 0.357285] pci 0000:00:0a.0: System wakeup disabled by ACPI [ 0.357315] pci 0000:00:0b.0: [1002:5a1f] type 01 class 0x060400 [ 0.357344] pci 0000:00:0b.0: PME# supported from D0 D3hot D3cold [ 0.357369] pci 0000:00:0b.0: System wakeup disabled by ACPI [ 0.357406] pci 0000:00:11.0: [1002:4391] type 00 class 0x010601 [ 0.357420] pci 0000:00:11.0: reg 10: [io 0xf040-0xf047] [ 0.357428] pci 0000:00:11.0: reg 14: [io 0xf030-0xf033] [ 0.357435] pci 0000:00:11.0: reg 18: [io 0xf020-0xf027] [ 0.357442] pci 0000:00:11.0: reg 1c: [io 0xf010-0xf013] [ 0.357449] pci 0000:00:11.0: reg 20: [io 0xf000-0xf00f] [ 0.357456] pci 0000:00:11.0: reg 24: [mem 0xfe70b000-0xfe70b3ff] [ 0.357545] pci 0000:00:12.0: [1002:4397] type 00 class 0x0c0310 [ 0.357555] pci 0000:00:12.0: reg 10: [mem 0xfe70a000-0xfe70afff] [ 0.357621] pci 0000:00:12.0: System wakeup disabled by ACPI [ 0.357653] pci 0000:00:12.2: [1002:4396] type 00 class 0x0c0320 [ 0.357667] pci 0000:00:12.2: reg 10: [mem 0xfe709000-0xfe7090ff] [ 0.357728] pci 0000:00:12.2: supports D1 D2 [ 0.357729] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot [ 0.357761] pci 0000:00:12.2: System wakeup disabled by ACPI [ 0.357793] pci 0000:00:13.0: [1002:4397] type 00 class 0x0c0310 [ 0.357803] pci 0000:00:13.0: reg 10: [mem 0xfe708000-0xfe708fff] [ 0.357871] pci 0000:00:13.0: System wakeup disabled by ACPI [ 0.357917] pci 0000:00:13.2: [1002:4396] type 00 class 0x0c0320 [ 0.357931] pci 0000:00:13.2: reg 10: [mem 0xfe707000-0xfe7070ff] [ 0.357991] pci 0000:00:13.2: supports D1 D2 [ 0.357993] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot [ 0.358027] pci 0000:00:13.2: System wakeup disabled by ACPI [ 0.358059] pci 0000:00:14.0: [1002:4385] type 00 class 0x0c0500 [ 0.358149] pci 0000:00:14.2: [1002:4383] type 00 class 0x040300 [ 0.358165] pci 0000:00:14.2: reg 10: [mem 0xfe700000-0xfe703fff 64bit] [ 0.358236] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold [ 0.358262] pci 0000:00:14.2: System wakeup disabled by ACPI [ 0.358292] pci 0000:00:14.3: [1002:439d] type 00 class 0x060100 [ 0.358383] pci 0000:00:14.4: [1002:4384] type 01 class 0x060401 [ 0.358432] pci 0000:00:14.4: System wakeup disabled by ACPI [ 0.358458] pci 0000:00:14.5: [1002:4399] type 00 class 0x0c0310 [ 0.358468] pci 0000:00:14.5: reg 10: [mem 0xfe706000-0xfe706fff] [ 0.358541] pci 0000:00:14.5: System wakeup disabled by ACPI [ 0.358574] pci 0000:00:15.0: [1002:43a0] type 01 class 0x060400 [ 0.358628] pci 0000:00:15.0: supports D1 D2 [ 0.358670] pci 0000:00:15.0: System wakeup disabled by ACPI [ 0.358703] pci 0000:00:16.0: [1002:4397] type 00 class 0x0c0310 [ 0.358713] pci 0000:00:16.0: reg 10: [mem 0xfe705000-0xfe705fff] [ 0.358783] pci 0000:00:16.0: System wakeup disabled by ACPI [ 0.358814] pci 0000:00:16.2: [1002:4396] type 00 class 0x0c0320 [ 0.358828] pci 0000:00:16.2: reg 10: [mem 0xfe704000-0xfe7040ff] [ 0.358894] pci 0000:00:16.2: supports D1 D2 [ 0.358896] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot [ 0.358932] pci 0000:00:16.2: System wakeup disabled by ACPI [ 0.358963] pci 0000:00:18.0: [1022:1600] type 00 class 0x060000 [ 0.359026] pci 0000:00:18.1: [1022:1601] type 00 class 0x060000 [ 0.359086] pci 0000:00:18.2: [1022:1602] type 00 class 0x060000 [ 0.359146] pci 0000:00:18.3: [1022:1603] type 00 class 0x060000 [ 0.359209] pci 0000:00:18.4: [1022:1604] type 00 class 0x060000 [ 0.359268] pci 0000:00:18.5: [1022:1605] type 00 class 0x060000 [ 0.359372] pci 0000:01:00.0: [10de:0fc6] type 00 class 0x030000 [ 0.359381] pci 0000:01:00.0: reg 10: [mem 0xfd000000-0xfdffffff] [ 0.359391] pci 0000:01:00.0: reg 14: [mem 0xa0000000-0xafffffff 64bit pref] [ 0.359401] pci 0000:01:00.0: reg 1c: [mem 0xb0000000-0xb1ffffff 64bit pref] [ 0.359408] pci 0000:01:00.0: reg 24: [io 0xe000-0xe07f] [ 0.359414] pci 0000:01:00.0: reg 30: [mem 0xfe000000-0xfe07ffff pref] [ 0.359516] pci 0000:01:00.1: [10de:0e1b] type 00 class 0x040300 [ 0.359525] pci 0000:01:00.1: reg 10: [mem 0xfe080000-0xfe083fff] [ 0.360912] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.360916] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.360918] pci 0000:00:02.0: bridge window [mem 0xfd000000-0xfe0fffff] [ 0.360921] pci 0000:00:02.0: bridge window [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.360964] pci 0000:02:00.0: [1b4b:91a0] type 00 class 0x01018f [ 0.360973] pci 0000:02:00.0: reg 10: [io 0xd090-0xd097] [ 0.360979] pci 0000:02:00.0: reg 14: [io 0xd080-0xd083] [ 0.360985] pci 0000:02:00.0: reg 18: [io 0xd070-0xd077] [ 0.360991] pci 0000:02:00.0: reg 1c: [io 0xd060-0xd063] [ 0.360997] pci 0000:02:00.0: reg 20: [io 0xd050-0xd05f] [ 0.361003] pci 0000:02:00.0: reg 24: [mem 0xfe615000-0xfe6157ff] [ 0.361010] pci 0000:02:00.0: reg 30: [mem 0xfe600000-0xfe60ffff pref] [ 0.361037] pci 0000:02:00.0: PME# supported from D3hot [ 0.362910] pci 0000:00:04.0: PCI bridge to [bus 02] [ 0.362914] pci 0000:00:04.0: bridge window [io 0xd000-0xdfff] [ 0.362916] pci 0000:00:04.0: bridge window [mem 0xfe600000-0xfe6fffff] [ 0.362961] pci 0000:03:00.0: [1b6f:7023] type 00 class 0x0c0330 [ 0.362975] pci 0000:03:00.0: reg 10: [mem 0xfe500000-0xfe507fff 64bit] [ 0.363039] pci 0000:03:00.0: supports D1 D2 [ 0.363041] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.364915] pci 0000:00:05.0: PCI bridge to [bus 03] [ 0.364919] pci 0000:00:05.0: bridge window [mem 0xfe500000-0xfe5fffff] [ 0.364964] pci 0000:04:00.0: [1b6f:7023] type 00 class 0x0c0330 [ 0.364979] pci 0000:04:00.0: reg 10: [mem 0xfe400000-0xfe407fff 64bit] [ 0.365043] pci 0000:04:00.0: supports D1 D2 [ 0.365045] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.366918] pci 0000:00:06.0: PCI bridge to [bus 04] [ 0.366922] pci 0000:00:06.0: bridge window [mem 0xfe400000-0xfe4fffff] [ 0.366974] pci 0000:05:00.0: [14e4:16b1] type 00 class 0x020000 [ 0.366991] pci 0000:05:00.0: reg 10: [mem 0xd0010000-0xd001ffff 64bit pref] [ 0.367005] pci 0000:05:00.0: reg 18: [mem 0xd0000000-0xd000ffff 64bit pref] [ 0.367029] pci 0000:05:00.0: reg 30: [mem 0xfe300000-0xfe3007ff pref] [ 0.367076] pci 0000:05:00.0: PME# supported from D0 D3hot D3cold [ 0.368923] pci 0000:00:0a.0: PCI bridge to [bus 05] [ 0.368927] pci 0000:00:0a.0: bridge window [mem 0xfe300000-0xfe3fffff] [ 0.368930] pci 0000:00:0a.0: bridge window [mem 0xd0000000-0xd00fffff 64bit pref] [ 0.368974] pci 0000:06:00.0: [1002:68d8] type 00 class 0x030000 [ 0.368987] pci 0000:06:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.368997] pci 0000:06:00.0: reg 18: [mem 0xfe220000-0xfe23ffff 64bit] [ 0.369004] pci 0000:06:00.0: reg 20: [io 0xc000-0xc0ff] [ 0.369017] pci 0000:06:00.0: reg 30: [mem 0xfe200000-0xfe21ffff pref] [ 0.369049] pci 0000:06:00.0: supports D1 D2 [ 0.369092] pci 0000:06:00.1: [1002:aa60] type 00 class 0x040300 [ 0.369105] pci 0000:06:00.1: reg 10: [mem 0xfe240000-0xfe243fff 64bit] [ 0.369164] pci 0000:06:00.1: supports D1 D2 [ 0.370925] pci 0000:00:0b.0: PCI bridge to [bus 06] [ 0.370928] pci 0000:00:0b.0: bridge window [io 0xc000-0xcfff] [ 0.370930] pci 0000:00:0b.0: bridge window [mem 0xfe200000-0xfe2fffff] [ 0.370933] pci 0000:00:0b.0: bridge window [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.370995] pci 0000:00:14.4: PCI bridge to [bus 07] (subtractive decode) [ 0.371002] pci 0000:00:14.4: bridge window [io 0x0000-0x03af] (subtractive decode) [ 0.371004] pci 0000:00:14.4: bridge window [io 0x03e0-0x0cf7] (subtractive decode) [ 0.371005] pci 0000:00:14.4: bridge window [io 0x03b0-0x03df] (subtractive decode) [ 0.371007] pci 0000:00:14.4: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.371008] pci 0000:00:14.4: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.371009] pci 0000:00:14.4: bridge window [mem 0x000c0000-0x000dffff] (subtractive decode) [ 0.371011] pci 0000:00:14.4: bridge window [mem 0xa0000000-0xffffffff] (subtractive decode) [ 0.371077] pci 0000:08:00.0: [10ec:8168] type 00 class 0x020000 [ 0.371094] pci 0000:08:00.0: reg 10: [io 0xb000-0xb0ff] [ 0.371123] pci 0000:08:00.0: reg 18: [mem 0xfe100000-0xfe100fff 64bit] [ 0.371156] pci 0000:08:00.0: reg 30: [mem 0xfffe0000-0xffffffff pref] [ 0.371229] pci 0000:08:00.0: supports D1 D2 [ 0.371231] pci 0000:08:00.0: PME# supported from D1 D2 D3hot D3cold [ 0.371285] pci 0000:08:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.371293] pci 0000:00:15.0: PCI bridge to [bus 08] [ 0.371298] pci 0000:00:15.0: bridge window [io 0xb000-0xbfff] [ 0.371300] pci 0000:00:15.0: bridge window [mem 0xfe100000-0xfe1fffff] [ 0.371321] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM [ 0.371323] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08) [ 0.371833] ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11 14 15) *0 [ 0.371891] ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11 14 15) *0 [ 0.371958] ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11 14 15) *0 [ 0.372018] ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11 14 15) *0 [ 0.372067] ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11 14 15) *0 [ 0.372106] ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11 14 15) *0 [ 0.372144] ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11 14 15) *0 [ 0.372182] ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11 14 15) *0 [ 0.372282] acpi root: \_SB_.PCI0 notify handler is installed [ 0.372311] Found 1 acpi root devices [ 0.372397] ACPI: No dock devices found. [ 0.372458] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.372461] vgaarb: device added: PCI:0000:06:00.0,decodes=io+mem,owns=none,locks=none [ 0.372463] vgaarb: loaded [ 0.372463] vgaarb: bridge control possible 0000:06:00.0 [ 0.372464] vgaarb: bridge control possible 0000:01:00.0 [ 0.372498] PCI: Using ACPI for IRQ routing [ 0.378656] PCI: pci_cache_line_size set to 64 bytes [ 0.378718] e820: reserve RAM buffer [mem 0x0009d000-0x0009ffff] [ 0.378720] e820: reserve RAM buffer [mem 0x9df05000-0x9fffffff] [ 0.378721] e820: reserve RAM buffer [mem 0x9ee2a000-0x9fffffff] [ 0.378723] e820: reserve RAM buffer [mem 0x9f453000-0x9fffffff] [ 0.378724] e820: reserve RAM buffer [mem 0x9f800000-0x9fffffff] [ 0.378725] e820: reserve RAM buffer [mem 0x25f000000-0x25fffffff] [ 0.378797] NetLabel: Initializing [ 0.378799] NetLabel: domain hash size = 128 [ 0.378799] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.378808] NetLabel: unlabeled traffic allowed by default [ 0.378836] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.378839] hpet0: 3 comparators, 32-bit 14.318180 MHz counter [ 0.380868] Switching to clocksource hpet [ 0.385544] pnp: PnP ACPI init [ 0.385553] ACPI: bus type PNP registered [ 0.385630] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved [ 0.385633] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.385730] system 00:01: [mem 0xfec20000-0xfec200ff] could not be reserved [ 0.385733] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.385799] system 00:02: [mem 0xfeb20000-0xfeb23fff] could not be reserved [ 0.385801] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386080] system 00:03: [io 0x040b] has been reserved [ 0.386082] system 00:03: [io 0x04d6] has been reserved [ 0.386084] system 00:03: [io 0x0c00-0x0c01] has been reserved [ 0.386086] system 00:03: [io 0x0c14] has been reserved [ 0.386087] system 00:03: [io 0x0c50-0x0c51] has been reserved [ 0.386089] system 00:03: [io 0x0c52] has been reserved [ 0.386090] system 00:03: [io 0x0c6c] has been reserved [ 0.386092] system 00:03: [io 0x0c6f] has been reserved [ 0.386093] system 00:03: [io 0x0cd0-0x0cd1] has been reserved [ 0.386095] system 00:03: [io 0x0cd2-0x0cd3] has been reserved [ 0.386097] system 00:03: [io 0x0cd4-0x0cd5] has been reserved [ 0.386098] system 00:03: [io 0x0cd6-0x0cd7] has been reserved [ 0.386100] system 00:03: [io 0x0cd8-0x0cdf] has been reserved [ 0.386101] system 00:03: [io 0x0800-0x089f] has been reserved [ 0.386103] system 00:03: [io 0x0b20-0x0b3f] has been reserved [ 0.386105] system 00:03: [io 0x0900-0x090f] has been reserved [ 0.386106] system 00:03: [io 0x0910-0x091f] has been reserved [ 0.386108] system 00:03: [io 0xfe00-0xfefe] has been reserved [ 0.386110] system 00:03: [mem 0xfec00000-0xfec00fff] could not be reserved [ 0.386112] system 00:03: [mem 0xfee00000-0xfee00fff] has been reserved [ 0.386114] system 00:03: [mem 0xfed80000-0xfed8ffff] has been reserved [ 0.386116] system 00:03: [mem 0xfed61000-0xfed70fff] has been reserved [ 0.386117] system 00:03: [mem 0xfec10000-0xfec10fff] has been reserved [ 0.386119] system 00:03: [mem 0xfed00000-0xfed00fff] has been reserved [ 0.386121] system 00:03: [mem 0xffc00000-0xffffffff] has been reserved [ 0.386123] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386199] system 00:04: [io 0x0290-0x029f] has been reserved [ 0.386202] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386411] pnp 00:05: [dma 4] [ 0.386429] pnp 00:05: Plug and Play ACPI device, IDs PNP0200 (active) [ 0.386458] pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.386478] pnp 00:07: Plug and Play ACPI device, IDs PNP0800 (active) [ 0.386527] system 00:08: [io 0x04d0-0x04d1] has been reserved [ 0.386530] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386560] pnp 00:09: Plug and Play ACPI device, IDs PNP0c04 (active) [ 0.386592] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386819] pnp 00:0b: Plug and Play ACPI device, IDs PNP0103 (active) [ 0.386822] pnp: PnP ACPI: found 12 devices [ 0.386823] ACPI: bus type PNP unregistered [ 0.393157] pci 0000:08:00.0: no compatible bridge window for [mem 0xfffe0000-0xffffffff pref] [ 0.393190] pci 0000:00:15.0: BAR 15: assigned [mem 0xb2000000-0xb20fffff pref] [ 0.393192] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.393194] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.393197] pci 0000:00:02.0: bridge window [mem 0xfd000000-0xfe0fffff] [ 0.393199] pci 0000:00:02.0: bridge window [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.393202] pci 0000:00:04.0: PCI bridge to [bus 02] [ 0.393204] pci 0000:00:04.0: bridge window [io 0xd000-0xdfff] [ 0.393206] pci 0000:00:04.0: bridge window [mem 0xfe600000-0xfe6fffff] [ 0.393210] pci 0000:00:05.0: PCI bridge to [bus 03] [ 0.393212] pci 0000:00:05.0: bridge window [mem 0xfe500000-0xfe5fffff] [ 0.393215] pci 0000:00:06.0: PCI bridge to [bus 04] [ 0.393218] pci 0000:00:06.0: bridge window [mem 0xfe400000-0xfe4fffff] [ 0.393221] pci 0000:00:0a.0: PCI bridge to [bus 05] [ 0.393223] pci 0000:00:0a.0: bridge window [mem 0xfe300000-0xfe3fffff] [ 0.393225] pci 0000:00:0a.0: bridge window [mem 0xd0000000-0xd00fffff 64bit pref] [ 0.393228] pci 0000:00:0b.0: PCI bridge to [bus 06] [ 0.393230] pci 0000:00:0b.0: bridge window [io 0xc000-0xcfff] [ 0.393232] pci 0000:00:0b.0: bridge window [mem 0xfe200000-0xfe2fffff] [ 0.393234] pci 0000:00:0b.0: bridge window [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.393237] pci 0000:00:14.4: PCI bridge to [bus 07] [ 0.393246] pci 0000:08:00.0: BAR 6: assigned [mem 0xb2000000-0xb201ffff pref] [ 0.393248] pci 0000:00:15.0: PCI bridge to [bus 08] [ 0.393250] pci 0000:00:15.0: bridge window [io 0xb000-0xbfff] [ 0.393253] pci 0000:00:15.0: bridge window [mem 0xfe100000-0xfe1fffff] [ 0.393255] pci 0000:00:15.0: bridge window [mem 0xb2000000-0xb20fffff pref] [ 0.393627] pci_bus 0000:00: resource 4 [io 0x0000-0x03af] [ 0.393629] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7] [ 0.393631] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df] [ 0.393632] pci_bus 0000:00: resource 7 [io 0x0d00-0xffff] [ 0.393634] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff] [ 0.393635] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff] [ 0.393637] pci_bus 0000:00: resource 10 [mem 0xa0000000-0xffffffff] [ 0.393638] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.393640] pci_bus 0000:01: resource 1 [mem 0xfd000000-0xfe0fffff] [ 0.393641] pci_bus 0000:01: resource 2 [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.393643] pci_bus 0000:02: resource 0 [io 0xd000-0xdfff] [ 0.393644] pci_bus 0000:02: resource 1 [mem 0xfe600000-0xfe6fffff] [ 0.393646] pci_bus 0000:03: resource 1 [mem 0xfe500000-0xfe5fffff] [ 0.393647] pci_bus 0000:04: resource 1 [mem 0xfe400000-0xfe4fffff] [ 0.393648] pci_bus 0000:05: resource 1 [mem 0xfe300000-0xfe3fffff] [ 0.393650] pci_bus 0000:05: resource 2 [mem 0xd0000000-0xd00fffff 64bit pref] [ 0.393651] pci_bus 0000:06: resource 0 [io 0xc000-0xcfff] [ 0.393653] pci_bus 0000:06: resource 1 [mem 0xfe200000-0xfe2fffff] [ 0.393654] pci_bus 0000:06: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.393656] pci_bus 0000:07: resource 4 [io 0x0000-0x03af] [ 0.393657] pci_bus 0000:07: resource 5 [io 0x03e0-0x0cf7] [ 0.393658] pci_bus 0000:07: resource 6 [io 0x03b0-0x03df] [ 0.393660] pci_bus 0000:07: resource 7 [io 0x0d00-0xffff] [ 0.393661] pci_bus 0000:07: resource 8 [mem 0x000a0000-0x000bffff] [ 0.393662] pci_bus 0000:07: resource 9 [mem 0x000c0000-0x000dffff] [ 0.393664] pci_bus 0000:07: resource 10 [mem 0xa0000000-0xffffffff] [ 0.393665] pci_bus 0000:08: resource 0 [io 0xb000-0xbfff] [ 0.393667] pci_bus 0000:08: resource 1 [mem 0xfe100000-0xfe1fffff] [ 0.393668] pci_bus 0000:08: resource 2 [mem 0xb2000000-0xb20fffff pref] [ 0.393691] NET: Registered protocol family 2 [ 0.393879] TCP established hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.394117] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.394334] TCP: Hash tables configured (established 65536 bind 65536) [ 0.394374] TCP: reno registered [ 0.394384] UDP hash table entries: 4096 (order: 5, 131072 bytes) [ 0.394429] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes) [ 0.394512] NET: Registered protocol family 1 [ 0.650568] pci 0000:01:00.0: Boot video device [ 0.650681] PCI: CLS 64 bytes, default 64 [ 0.650716] Unpacking initramfs... [ 1.139954] Freeing initrd memory: 5880k freed [ 1.141944] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.141945] AMD-Vi: Interrupt remapping enabled [ 1.141966] pci 0000:00:00.2: irq 72 for MSI/MSI-X [ 1.153516] AMD-Vi: Lazy IO/TLB flushing enabled [ 1.228038] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.228040] software IO TLB [mem 0x99f05000-0x9df05000] (64MB) mapped at [ffff880099f05000-ffff88009df04fff] [ 1.228355] LVT offset 0 assigned for vector 0x400 [ 1.228391] perf: AMD IBS detected (0x000000ff) [ 1.228419] Scanning for low memory corruption every 60 seconds [ 1.228591] audit: initializing netlink socket (disabled) [ 1.228610] type=2000 audit(1369582568.031:1): initialized [ 1.237950] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.239493] VFS: Disk quotas dquot_6.5.2 [ 1.239528] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.239669] msgmni has been set to 15756 [ 1.239941] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 1.239972] io scheduler noop registered [ 1.239975] io scheduler deadline registered [ 1.239980] io scheduler cfq registered (default) [ 1.240206] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.240219] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.240300] GHES: HEST is not enabled! [ 1.240341] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 1.240723] Linux agpgart interface v0.103 [ 1.240807] VFIO - User Level meta-driver version: 0.3 [ 1.241050] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 1.243505] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.243528] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.243677] mousedev: PS/2 mouse device common for all mice [ 1.243724] rtc_cmos 00:06: RTC can wake from S4 [ 1.243815] rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0 [ 1.243834] rtc_cmos 00:06: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 1.243841] cpuidle: using governor ladder [ 1.243842] cpuidle: using governor menu [ 1.243844] EFI Variables Facility v0.08 2004-May-17 [ 1.243888] drop_monitor: Initializing network drop monitor service [ 1.243937] TCP: cubic registered [ 1.244017] NET: Registered protocol family 10 [ 1.244133] NET: Registered protocol family 17 [ 1.244140] Key type dns_resolver registered [ 1.244656] PM: Hibernation image not present or could not be loaded. [ 1.244664] registered taskstats version 1 [ 1.245341] Magic number: 13:120:640 [ 1.245398] rtc_cmos 00:06: setting system clock to 2013-05-26 15:36:08 UTC (1369582568) [ 1.245984] Freeing unused kernel memory: 1092k freed [ 1.246231] Write protecting the kernel read-only data: 8192k [ 1.248312] Freeing unused kernel memory: 1004k freed [ 1.249495] Freeing unused kernel memory: 420k freed [ 1.254966] systemd-udevd[80]: starting version 204 [ 1.257148] [drm] Initialized drm 1.1.0 20060810 [ 1.263456] nvidia: module license 'NVIDIA' taints kernel. [ 1.263459] Disabling lock debugging due to kernel taint [ 1.274456] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.274458] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:06:00.0 [ 1.274672] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0 [ 1.274680] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 319.17 Thu Apr 25 22:45:49 PDT 2013 [ 1.292640] SCSI subsystem initialized [ 1.294690] ACPI: bus type ATA registered [ 1.294771] libata version 3.00 loaded. [ 1.295287] ACPI: bus type USB registered [ 1.295398] usbcore: registered new interface driver usbfs [ 1.295454] usbcore: registered new interface driver hub [ 1.295496] usbcore: registered new device driver usb [ 1.296456] ahci 0000:00:11.0: version 3.0 [ 1.296663] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.296710] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 1.296713] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 1.296855] ehci-pci: EHCI PCI platform driver [ 1.297028] xhci_hcd 0000:03:00.0: xHCI Host Controller [ 1.297037] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 1 [ 1.297135] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.297172] xhci_hcd 0000:03:00.0: irq 73 for MSI/MSI-X [ 1.297491] xHCI xhci_add_endpoint called for root hub [ 1.297493] xHCI xhci_check_bandwidth called for root hub [ 1.297558] hub 1-0:1.0: USB hub found [ 1.297566] hub 1-0:1.0: 2 ports detected [ 1.297716] scsi0 : pata_marvell [ 1.297970] xhci_hcd 0000:03:00.0: xHCI Host Controller [ 1.297974] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 2 [ 1.298501] scsi1 : pata_marvell [ 1.298667] xHCI xhci_add_endpoint called for root hub [ 1.298669] xHCI xhci_check_bandwidth called for root hub [ 1.298711] ata1: PATA max UDMA/100 cmd 0xd090 ctl 0xd080 bmdma 0xd050 irq 44 [ 1.298713] ata2: PATA max UDMA/133 cmd 0xd070 ctl 0xd060 bmdma 0xd058 irq 44 [ 1.298786] hub 2-0:1.0: USB hub found [ 1.298793] hub 2-0:1.0: 2 ports detected [ 1.299597] scsi2 : ahci [ 1.299734] scsi3 : ahci [ 1.299852] scsi4 : ahci [ 1.299993] scsi5 : ahci [ 1.300114] scsi6 : ahci [ 1.300255] scsi7 : ahci [ 1.300342] ata3: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b100 irq 19 [ 1.300345] ata4: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b180 irq 19 [ 1.300347] ata5: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b200 irq 19 [ 1.300349] ata6: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b280 irq 19 [ 1.300351] ata7: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b300 irq 19 [ 1.300353] ata8: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b380 irq 19 [ 1.300456] ehci-pci 0000:00:12.2: EHCI Host Controller [ 1.300466] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 3 [ 1.300482] QUIRK: Enable AMD PLL fix [ 1.300483] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 1.300499] ehci-pci 0000:00:12.2: debug port 1 [ 1.300562] ehci-pci 0000:00:12.2: irq 17, io mem 0xfe709000 [ 1.300685] xhci_hcd 0000:04:00.0: xHCI Host Controller [ 1.300693] xhci_hcd 0000:04:00.0: new USB bus registered, assigned bus number 4 [ 1.300818] xhci_hcd 0000:04:00.0: irq 74 for MSI/MSI-X [ 1.301029] xHCI xhci_add_endpoint called for root hub [ 1.301030] xHCI xhci_check_bandwidth called for root hub [ 1.301066] hub 4-0:1.0: USB hub found [ 1.301071] hub 4-0:1.0: 2 ports detected [ 1.301226] xhci_hcd 0000:04:00.0: xHCI Host Controller [ 1.301230] xhci_hcd 0000:04:00.0: new USB bus registered, assigned bus number 5 [ 1.301351] xHCI xhci_add_endpoint called for root hub [ 1.301353] xHCI xhci_check_bandwidth called for root hub [ 1.301414] hub 5-0:1.0: USB hub found [ 1.301419] hub 5-0:1.0: 2 ports detected [ 1.306554] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00 [ 1.306685] hub 3-0:1.0: USB hub found [ 1.306688] hub 3-0:1.0: 5 ports detected [ 1.306946] ehci-pci 0000:00:13.2: EHCI Host Controller [ 1.306958] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 6 [ 1.306961] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 1.306971] ehci-pci 0000:00:13.2: debug port 1 [ 1.306993] ehci-pci 0000:00:13.2: irq 17, io mem 0xfe707000 [ 1.312551] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00 [ 1.312661] hub 6-0:1.0: USB hub found [ 1.312664] hub 6-0:1.0: 5 ports detected [ 1.312828] ehci-pci 0000:00:16.2: EHCI Host Controller [ 1.312832] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 7 [ 1.312835] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 1.312844] ehci-pci 0000:00:16.2: debug port 1 [ 1.312865] ehci-pci 0000:00:16.2: irq 17, io mem 0xfe704000 [ 1.318563] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00 [ 1.318664] hub 7-0:1.0: USB hub found [ 1.318667] hub 7-0:1.0: 4 ports detected [ 1.318841] ohci_hcd 0000:00:12.0: OHCI Host Controller [ 1.318852] ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 8 [ 1.318879] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfe70a000 [ 1.373777] hub 8-0:1.0: USB hub found [ 1.373785] hub 8-0:1.0: 5 ports detected [ 1.373948] ohci_hcd 0000:00:13.0: OHCI Host Controller [ 1.373952] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 9 [ 1.373964] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfe708000 [ 1.428882] hub 9-0:1.0: USB hub found [ 1.428887] hub 9-0:1.0: 5 ports detected [ 1.429038] ohci_hcd 0000:00:14.5: OHCI Host Controller [ 1.429043] ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 10 [ 1.429056] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfe706000 [ 1.483961] hub 10-0:1.0: USB hub found [ 1.483967] hub 10-0:1.0: 2 ports detected [ 1.484113] ohci_hcd 0000:00:16.0: OHCI Host Controller [ 1.484117] ohci_hcd 0000:00:16.0: new USB bus registered, assigned bus number 11 [ 1.484128] ohci_hcd 0000:00:16.0: irq 18, io mem 0xfe705000 [ 1.504185] ata1.01: ATA-7: ST3320820AS, 3.AAC, max UDMA/133 [ 1.504188] ata1.01: 625142448 sectors, multi 0: LBA48 NCQ (depth 0/32) [ 1.539050] hub 11-0:1.0: USB hub found [ 1.539055] hub 11-0:1.0: 4 ports detected [ 1.579276] ata1.01: configured for UDMA/100 [ 1.579412] scsi 0:0:1:0: Direct-Access ATA ST3320820AS 3.AA PQ: 0 ANSI: 5 [ 1.760363] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.760386] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.760409] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.760431] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.760452] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.761129] ata3.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133 [ 1.761131] ata3.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.761342] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 1.761968] ata3.00: configured for UDMA/133 [ 1.762044] scsi 2:0:0:0: Direct-Access ATA WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5 [ 1.764131] ata8.00: ATAPI: HL-DT-ST DVDRAM GH20NS10, EL00, max UDMA/100 [ 1.766537] ata5.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133 [ 1.766540] ata5.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.766866] ata4.00: ATA-7: SAMSUNG HD103SI, 1AG01118, max UDMA7 [ 1.766869] ata4.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.767196] ata6.00: ATA-8: WDC WD20EARX-00PASB0, 51.0AB51, max UDMA/133 [ 1.767198] ata6.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.768130] ata8.00: configured for UDMA/100 [ 1.771688] ata7.00: ATA-8: ADATA SP900, 5.0.2a, max UDMA/133 [ 1.771690] ata7.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.772685] ata5.00: configured for UDMA/133 [ 1.773361] ata4.00: configured for UDMA/133 [ 1.773455] scsi 3:0:0:0: Direct-Access ATA SAMSUNG HD103SI 1AG0 PQ: 0 ANSI: 5 [ 1.773632] scsi 4:0:0:0: Direct-Access ATA SAMSUNG HD204UI 1AQ1 PQ: 0 ANSI: 5 [ 1.774193] ata6.00: configured for UDMA/133 [ 1.774274] scsi 5:0:0:0: Direct-Access ATA WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5 [ 1.782015] ata7.00: configured for UDMA/133 [ 1.782103] scsi 6:0:0:0: Direct-Access ATA ADATA SP900 5.0. PQ: 0 ANSI: 5 [ 1.785353] scsi 7:0:0:0: CD-ROM HL-DT-ST DVDRAM GH20NS10 EL00 PQ: 0 ANSI: 5 [ 1.789538] sd 0:0:1:0: [sda] 625142448 512-byte logical blocks: (320 GB/298 GiB) [ 1.789610] sd 0:0:1:0: [sda] Write Protect is off [ 1.789613] sd 0:0:1:0: [sda] Mode Sense: 00 3a 00 00 [ 1.789659] sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789723] sd 2:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 1.789789] sd 3:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 1.789815] sd 2:0:0:0: [sdb] Write Protect is off [ 1.789818] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 1.789861] sd 4:0:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [ 1.789864] sd 3:0:0:0: [sdc] Write Protect is off [ 1.789866] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 1.789868] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789893] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789902] sd 5:0:0:0: [sde] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [ 1.789905] sd 5:0:0:0: [sde] 4096-byte physical blocks [ 1.789923] sd 4:0:0:0: [sdd] Write Protect is off [ 1.789926] sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00 [ 1.789928] sd 6:0:0:0: [sdf] 250069680 512-byte logical blocks: (128 GB/119 GiB) [ 1.789954] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789972] sd 5:0:0:0: [sde] Write Protect is off [ 1.789974] sd 6:0:0:0: [sdf] Write Protect is off [ 1.789975] sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00 [ 1.789979] sd 6:0:0:0: [sdf] Mode Sense: 00 3a 00 00 [ 1.789991] sd 6:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789999] sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.791154] sdf: sdf1 sdf2 sdf3 [ 1.791424] sd 6:0:0:0: [sdf] Attached SCSI disk [ 1.793671] sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray [ 1.793673] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 1.793847] sr 7:0:0:0: Attached scsi CD-ROM sr0 [ 1.807557] sdc: sdc1 sdc2 sdc3 sdc4 [ 1.807974] sd 3:0:0:0: [sdc] Attached SCSI disk [ 1.810863] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000001000 flags=0x0050] [ 1.824834] sdd: sdd1 sdd2 [ 1.825116] sd 4:0:0:0: [sdd] Attached SCSI disk [ 1.836967] sdb: sdb1 [ 1.837233] sd 2:0:0:0: [sdb] Attached SCSI disk [ 1.837476] raid6: sse2x1 6937 MB/s [ 1.854501] raid6: sse2x2 11195 MB/s [ 1.871543] raid6: sse2x4 13304 MB/s [ 1.871544] raid6: using algorithm sse2x4 (13304 MB/s) [ 1.871545] raid6: using ssse3x2 recovery algorithm [ 1.871754] xor: automatically using best checksumming function: [ 1.881535] avx : 4852.000 MB/sec [ 1.884902] Btrfs loaded [ 1.886377] device fsid 33e09c2d-5913-4ece-bbc2-069d55f1fecc devid 1 transid 69987 /dev/sdf3 [ 1.986737] usb 8-1: new full-speed USB device number 2 using ohci_hcd [ 2.138080] hub 8-1:1.0: USB hub found [ 2.140045] hub 8-1:1.0: 4 ports detected [ 2.230163] tsc: Refined TSC clocksource calibration: 3991.443 MHz [ 2.230171] Switching to clocksource tsc [ 2.252034] sde: sde1 [ 2.252274] sd 5:0:0:0: [sde] Attached SCSI disk [ 2.432558] usb 8-1.1: new low-speed USB device number 3 using ohci_hcd [ 2.546585] hidraw: raw HID events driver (C) Jiri Kosina [ 2.575860] usbcore: registered new interface driver usbhid [ 2.575861] usbhid: USB HID core driver [ 2.576268] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.1/8-1.1:1.0/input/input0 [ 2.576310] hid-generic 0003:04D9:1605.0001: input,hidraw0: USB HID v1.10 Keyboard [ USB Keyboard] on usb-0000:00:12.0-1.1/input0 [ 2.581867] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.1/8-1.1:1.1/input/input1 [ 2.581934] hid-generic 0003:04D9:1605.0002: input,hidraw1: USB HID v1.10 Device [ USB Keyboard] on usb-0000:00:12.0-1.1/input1 [ 2.614877] usb 8-1.2: new full-speed USB device number 4 using ohci_hcd [ 2.715202] input: USB USB Mouse as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.2/8-1.2:1.0/input/input2 [ 2.715264] hid-generic 0003:1D57:3108.0003: input,hidraw2: USB HID v1.11 Mouse [USB USB Mouse] on usb-0000:00:12.0-1.2/input0 [ 2.783174] usb 8-1.3: new low-speed USB device number 5 using ohci_hcd [ 2.896435] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.3/8-1.3:1.0/input/input3 [ 2.896492] hid-generic 0003:10D5:55A4.0004: input,hidraw3: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:12.0-1.3/input0 [ 2.898397] usbhid 8-1.3:1.1: couldn't find an input interrupt endpoint [ 31.962370] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 31.962439] ata1.01: failed command: READ DMA [ 31.962503] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 31.962616] ata1.01: status: { DRDY } [ 33.598405] device fsid 33e09c2d-5913-4ece-bbc2-069d55f1fecc devid 1 transid 69987 /dev/sdf3 [ 33.598725] btrfs: disk space caching is enabled [ 33.606689] Btrfs detected SSD devices, enabling SSD mode [ 37.020231] ata1: link is slow to respond, please be patient (ready=0) [ 42.027021] ata1: device not ready (errno=-16), forcing hardreset [ 42.027026] ata1: soft resetting link [ 47.338344] ata1: link is slow to respond, please be patient (ready=0) [ 52.089695] ata1: SRST failed (errno=-16) [ 52.089762] ata1: soft resetting link [ 52.363688] ata1.01: configured for UDMA/100 [ 52.363692] ata1.01: device reported invalid CHS sector 0 [ 52.363700] ata1: EH complete [ 63.742321] systemd[1]: systemd 204 running in system mode. (+PAM -LIBWRAP -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ) [ 63.742886] systemd[1]: Set hostname to <desk>. [ 63.784765] systemd[1]: Starting Forward Password Requests to Wall Directory Watch. [ 63.784814] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 63.784824] systemd[1]: Expecting device sys-subsystem-net-devices-enp8s0.device... [ 63.784831] systemd[1]: Expecting device sys-subsystem-net-devices-enp5s0.device... [ 63.784838] systemd[1]: Starting Remote File Systems. [ 63.784845] systemd[1]: Reached target Remote File Systems. [ 63.784853] systemd[1]: Starting /dev/initctl Compatibility Named Pipe. [ 63.784871] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe. [ 63.784877] systemd[1]: Starting Delayed Shutdown Socket. [ 63.784893] systemd[1]: Listening on Delayed Shutdown Socket. [ 63.784899] systemd[1]: Starting LVM2 metadata daemon socket. [ 63.784918] systemd[1]: Listening on LVM2 metadata daemon socket. [ 63.784923] systemd[1]: Starting Device-mapper event daemon FIFOs. [ 63.784943] systemd[1]: Listening on Device-mapper event daemon FIFOs. [ 63.784950] systemd[1]: Starting Journal Socket. [ 63.784984] systemd[1]: Listening on Journal Socket. [ 63.785795] systemd[1]: Starting Apply Kernel Variables... [ 63.786316] systemd[1]: Starting Create static device nodes in /dev... [ 63.786626] systemd[1]: Starting Journal Service... [ 63.786985] systemd[1]: Started Journal Service. [ 63.787009] systemd[1]: Starting Setup Virtual Console... [ 63.787420] systemd[1]: Mounting Debug File System... [ 63.788914] systemd[1]: Starting Load Kernel Modules... [ 63.789374] systemd[1]: Starting udev Kernel Socket. [ 63.789393] systemd[1]: Listening on udev Kernel Socket. [ 63.789436] systemd[1]: Starting udev Control Socket. [ 63.789454] systemd[1]: Listening on udev Control Socket. [ 63.789488] systemd[1]: Starting udev Coldplug all Devices... [ 63.789937] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. [ 63.790037] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ 63.791799] systemd[1]: Started Set Up Additional Binary Formats. [ 63.791816] systemd[1]: Mounting POSIX Message Queue File System... [ 63.792117] systemd[1]: Mounting Huge Pages File System... [ 63.792472] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch. [ 63.792509] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 63.792520] systemd[1]: Starting Swap. [ 63.792527] systemd[1]: Reached target Swap. [ 63.792537] systemd[1]: Mounting Temporary Directory... [ 63.792927] systemd[1]: Starting File System Check on Root Device... [ 63.793307] systemd[1]: Expecting device dev-disk-by\x2duuid-c17eadf9\x2deac3\x2d4b44\x2d8eae\x2d970173be0269.device... [ 63.793318] systemd[1]: Expecting device dev-disk-by\x2duuid-99463033\x2d334a\x2d4947\x2d9965\x2d2491be857342.device... [ 63.793329] systemd[1]: Expecting device dev-disk-by\x2duuid-0d0b891c\x2da74e\x2d4216\x2daed0\x2db4c5683b3f4f.device... [ 63.793339] systemd[1]: Expecting device dev-mapper-home.device... [ 63.793346] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dSAMSUNG_HD103SI_S1VSJ9CZ618796\x2dpart4.device... [ 63.793353] systemd[1]: Expecting device dev-mapper-files0.device... [ 63.793363] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dSAMSUNG_HD204UI_S2H7J1CB903523\x2dpart2.device... [ 63.793370] systemd[1]: Expecting device dev-mapper-files1.device... [ 63.793379] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dWDC_WD20EARX\x2d00PASB0_WD\x2dWCAZAH511243\x2dpart1.device... [ 63.793386] systemd[1]: Expecting device dev-mapper-files2.device... [ 63.793393] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dWDC_WD10EADS\x2d00L5B1_WD\x2dWCAU4C887789\x2dpart1.device... [ 63.793400] systemd[1]: Expecting device dev-mapper-files3.device... [ 63.793929] systemd[1]: Started Apply Kernel Variables. [ 63.819258] btrfs: use lzo compression [ 63.819261] btrfs: use ssd allocation scheme [ 63.819273] btrfs: disk space caching is enabled [ 63.819427] systemd-udevd[211]: starting version 204 [ 63.820248] microcode: CPU0: patch_level=0x06000817 [ 63.829635] microcode: CPU1: patch_level=0x06000817 [ 63.829649] microcode: CPU2: patch_level=0x06000817 [ 63.829657] microcode: CPU3: patch_level=0x06000817 [ 63.829666] microcode: CPU4: patch_level=0x06000817 [ 63.829673] microcode: CPU5: patch_level=0x06000817 [ 63.829680] microcode: CPU6: patch_level=0x06000817 [ 63.829689] microcode: CPU7: patch_level=0x06000817 [ 63.829757] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 63.829815] systemd-modules-load[192]: Inserted module 'microcode' [ 63.837429] ACPI: acpi_idle registered with cpuidle [ 63.842104] acpi-cpufreq: overriding BIOS provided _PSD data [ 63.843068] systemd-modules-load[192]: Inserted module 'acpi_cpufreq' [ 63.843107] systemd-modules-load[192]: Module 'cpufreq_ondemand' is builtin [ 63.843591] RPC: Registered named UNIX socket transport module. [ 63.843593] RPC: Registered udp transport module. [ 63.843594] RPC: Registered tcp transport module. [ 63.843595] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 63.846582] fuse init (API version 7.21) [ 63.846702] systemd-modules-load[192]: Inserted module 'fuse' [ 63.852743] device-mapper: uevent: version 1.0.3 [ 63.852965] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com [ 63.853006] systemd-modules-load[192]: Inserted module 'dm_mod' [ 63.858862] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 63.861605] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input4 [ 63.861616] ACPI: Power Button [PWRB] [ 63.861713] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input5 [ 63.861719] ACPI: Power Button [PWRF] [ 63.865563] FS-Cache: Loaded [ 63.873987] FS-Cache: Netfs 'nfs' registered for caching [ 63.906386] wmi: Mapper loaded [ 63.937443] pps_core: LinuxPPS API ver. 1 registered [ 63.937449] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 63.942978] PTP clock support registered [ 63.951959] tg3.c:v3.130 (February 14, 2013) [ 63.962462] tg3 0000:05:00.0 eth0: Tigon3 [partno(BCM57781) rev 57785100] (PCI Express) MAC address bc:5f:f4:96:14:9d [ 63.962474] tg3 0000:05:00.0 eth0: attached PHY is 57765 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 63.962482] tg3 0000:05:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1] [ 63.962490] tg3 0000:05:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 63.975019] MCE: In-kernel MCE decoding enabled. [ 63.983574] EDAC MC: Ver: 3.0.0 [ 63.990404] input: HDA ATI SB Line as /devices/pci0000:00/0000:00:14.2/sound/card0/input6 [ 63.991242] AMD64 EDAC driver v3.4.0 [ 63.992089] input: HDA ATI SB Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input7 [ 63.992127] input: HDA ATI SB Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input8 [ 63.992162] input: HDA ATI SB Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card0/input9 [ 63.992235] input: HDA ATI SB Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card0/input10 [ 63.992266] input: HDA ATI SB Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card0/input11 [ 63.992305] input: HDA ATI SB Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card0/input12 [ 63.992383] input: HDA ATI SB Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card0/input13 [ 63.993624] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 [ 63.993897] hda_intel: Disabling MSI [ 63.993908] hda-intel 0000:01:00.1: Handle VGA-switcheroo audio client [ 63.994711] EDAC amd64: DRAM ECC disabled. [ 63.994733] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. Either enable ECC checking or force module loading by setting 'ecc_enable_override'. (Note that use of the override may cause unknown side effects.) [ 63.997086] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05 [ 63.997120] sp5100_tco: PCI Revision ID: 0x42 [ 63.997149] sp5100_tco: Using 0xfed80b00 for watchdog MMIO address [ 63.997159] sp5100_tco: Last reboot was not triggered by watchdog. [ 63.997271] sp5100_tco: initialized (0xffffc900121feb00). heartbeat=60 sec (nowayout=0) [ 64.000212] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 64.001029] r8169 0000:08:00.0: irq 75 for MSI/MSI-X [ 64.004497] r8169 0000:08:00.0 eth1: RTL8168b/8111b at 0xffffc9001285e000, 74:ea:3a:80:f2:f5, XID 18000000 IRQ 75 [ 64.004506] r8169 0000:08:00.0 eth1: jumbo features [frames: 4080 bytes, tx checksumming: ko] [ 64.034976] kvm: Nested Virtualization enabled [ 64.034983] kvm: Nested Paging enabled [ 64.061792] device fsid 33e09c2d-5913-4ece-bbc2-069d55f1fecc devid 1 transid 69987 /dev/sdf3 [ 64.079895] systemd-udevd[241]: renamed network interface eth0 to enp5s0 [ 64.116024] systemd-udevd[241]: renamed network interface eth1 to enp8s0 [ 64.168124] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.0/0000:01:00.1/sound/card1/input14 [ 64.168301] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.0/0000:01:00.1/sound/card1/input15 [ 64.168450] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.0/0000:01:00.1/sound/card1/input16 [ 64.169170] hda-intel 0000:06:00.1: Handle VGA-switcheroo audio client [ 64.169277] snd_hda_intel 0000:06:00.1: irq 76 for MSI/MSI-X [ 64.174041] input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:0b.0/0000:06:00.1/sound/card2/input17 [ 64.211834] EXT4-fs (sdf1): mounting ext2 file system using the ext4 subsystem [ 64.212705] EXT4-fs (sdf1): mounted filesystem without journal. Opts: (null) [ 64.495480] EXT4-fs (sdc3): mounted filesystem with ordered data mode. Opts: (null) [ 71.464558] bio: create slab <bio-1> at 1 [ 71.623112] bio: create slab <bio-1> at 1 [ 72.290453] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled [ 72.292244] XFS (dm-0): Mounting Filesystem [ 72.471942] XFS (dm-0): Ending clean mount [ 79.849647] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.124275] ata1.01: limiting speed to UDMA/66:PIO4 [ 83.124284] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 83.124375] ata1.01: failed command: READ DMA [ 83.124449] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 83.124571] ata1.01: status: { DRDY } [ 83.375580] XFS (dm-2): Mounting Filesystem [ 83.528100] XFS (dm-2): Ending clean mount [ 87.645654] device fsid 374ec396-e838-46a8-9298-a0c110be2283 devid 1 transid 106056 /dev/dm-3 [ 87.666262] device fsid 374ec396-e838-46a8-9298-a0c110be2283 devid 1 transid 106056 /dev/mapper/home [ 87.666893] btrfs: use lzo compression [ 87.666899] btrfs: use ssd allocation scheme [ 87.666904] btrfs: disk space caching is enabled [ 88.182170] ata1: link is slow to respond, please be patient (ready=0) [ 93.188960] ata1: device not ready (errno=-16), forcing hardreset [ 93.188971] ata1: soft resetting link [ 95.178654] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.215897] systemd-journald[189]: File /var/log/journal/e3c0520ae85a403a8970565f63db0dad/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 95.234368] w83627ehf: Found NCT6775F chip at 0x290 [ 95.255172] ip_tables: (C) 2000-2006 Netfilter Core Team [ 95.278686] tg3 0000:05:00.0: irq 76 for MSI/MSI-X [ 95.278698] tg3 0000:05:00.0: irq 77 for MSI/MSI-X [ 95.278706] tg3 0000:05:00.0: irq 78 for MSI/MSI-X [ 95.278713] tg3 0000:05:00.0: irq 79 for MSI/MSI-X [ 95.278720] tg3 0000:05:00.0: irq 80 for MSI/MSI-X [ 95.431968] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready [ 95.436203] r8169 0000:08:00.0 enp8s0: link down [ 95.436248] IPv6: ADDRCONF(NETDEV_UP): enp8s0: link is not ready [ 95.440355] systemd-logind[735]: Watching system buttons on /dev/input/event5 (Power Button) [ 95.440440] systemd-logind[735]: Watching system buttons on /dev/input/event4 (Power Button) [ 98.502285] ata1: link is slow to respond, please be patient (ready=0) [ 98.824473] tg3 0000:05:00.0 enp5s0: Link is up at 1000 Mbps, full duplex [ 98.824497] tg3 0000:05:00.0 enp5s0: Flow control is on for TX and on for RX [ 98.824504] tg3 0000:05:00.0 enp5s0: EEE is disabled [ 98.824530] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready [ 99.007095] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 99.024418] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 99.024454] NFSD: starting 90-second grace period (net ffffffff81891f00) [ 102.463962] Calling /sbin/tomoyo-init to load policy. Please wait. [ 102.610329] TOMOYO: 2.5.0 [ 102.610437] Mandatory Access Control activated. [ 102.661726] systemd-journald[189]: Failed to set ACL on /var/log/journal/e3c0520ae85a403a8970565f63db0dad/user-1000.journal, ignoring: Invalid argument [ 103.256650] ata1: SRST failed (errno=-16) [ 103.256663] ata1: soft resetting link [ 103.540336] ata1.01: configured for UDMA/66 [ 103.540343] ata1.01: device reported invalid CHS sector 0 [ 103.540350] ata1: EH complete [ 104.770584] EXT4-fs (sdc3): re-mounted. Opts: commit=0 [ 104.827244] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro,commit=0 [ 104.894490] EXT4-fs (dm-4): re-mounted. Opts: errors=remount-ro,commit=0 [ 134.158287] ata1.01: limiting speed to UDMA/33:PIO4 [ 134.158296] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 134.158304] ata1.01: failed command: READ DMA [ 134.158313] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 134.158318] ata1.01: status: { DRDY } [ 139.216274] ata1: link is slow to respond, please be patient (ready=0) [ 144.223140] ata1: device not ready (errno=-16), forcing hardreset [ 144.223152] ata1: soft resetting link [ 149.534523] ata1: link is slow to respond, please be patient (ready=0) [ 154.285986] ata1: SRST failed (errno=-16) [ 154.285999] ata1: soft resetting link [ 154.566812] ata1.01: configured for UDMA/33 [ 154.566826] ata1.01: device reported invalid CHS sector 0 [ 154.566853] ata1: EH complete [ 185.320926] ata1.01: limiting speed to PIO4 [ 185.320936] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 185.320943] ata1.01: failed command: READ DMA [ 185.320952] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 185.320958] ata1.01: status: { DRDY } [ 190.378863] ata1: link is slow to respond, please be patient (ready=0) [ 195.385674] ata1: device not ready (errno=-16), forcing hardreset [ 195.385686] ata1: soft resetting link [ 200.697124] ata1: link is slow to respond, please be patient (ready=0) [ 205.448488] ata1: SRST failed (errno=-16) [ 205.448501] ata1: soft resetting link [ 205.726753] ata1.01: configured for PIO4 [ 205.726767] ata1.01: device reported invalid CHS sector 0 [ 205.726792] ata1: EH complete [ 205.794930] sda: sda1 sda2 sda3 < sda5 sda6 > [ 205.795831] sda: detected capacity change from 0 to 320072933376 [ 205.796047] sd 0:0:1:0: [sda] Attached SCSI disk [ 251.299287] vfio-pci 0000:06:00.0: enabling device (0000 -> 0003) [ 255.133480] usb 8-1: USB disconnect, device number 2 [ 255.133489] usb 8-1.1: USB disconnect, device number 3 [ 255.157518] usb 8-1.2: USB disconnect, device number 4 [ 255.166979] usb 8-1.3: USB disconnect, device number 5 [ 255.826679] usb 8-2: new full-speed USB device number 6 using ohci_hcd [ 255.978095] hub 8-2:1.0: USB hub found [ 255.980003] hub 8-2:1.0: 4 ports detected [ 256.270515] usb 8-2.1: new low-speed USB device number 7 using ohci_hcd [ 256.396482] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.1/8-2.1:1.0/input/input18 [ 256.396793] hid-generic 0003:04D9:1605.0005: input,hidraw0: USB HID v1.10 Keyboard [ USB Keyboard] on usb-0000:00:12.0-2.1/input0 [ 256.418909] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.1/8-2.1:1.1/input/input19 [ 256.419159] hid-generic 0003:04D9:1605.0006: input,hidraw1: USB HID v1.10 Device [ USB Keyboard] on usb-0000:00:12.0-2.1/input1 [ 256.486896] usb 8-2.2: new full-speed USB device number 8 using ohci_hcd [ 256.587496] input: USB USB Mouse as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.2/8-2.2:1.0/input/input20 [ 256.587765] hid-generic 0003:1D57:3108.0007: input,hidraw2: USB HID v1.11 Mouse [USB USB Mouse] on usb-0000:00:12.0-2.2/input0 [ 257.604866] usb 8-2.3: new low-speed USB device number 9 using ohci_hcd [ 257.717260] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.3/8-2.3:1.0/input/input21 [ 257.717481] hid-generic 0003:10D5:55A4.0008: input,hidraw3: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:12.0-2.3/input0 [ 257.719123] usbhid 8-2.3:1.1: couldn't find an input interrupt endpoint

I read the line " [    0.000000] Please enable the IOMMU option in the BIOS setup", but I already did that.
The input and usb/usbhid spam is there because of my "KVMSwicther".

what I'm executing

/usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -vga none -nographic -bios /usr/share/qemu/bios.bin -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 1024 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -device vfio-pci,host=06:00.1,addr=0.1,bus=root.1 -boot menu=on -monitor stdio

Any ideas? What should I test/change? Is there some sort of debugging mode for qemu?

Last edited by andy123 (2013-05-26 16:34:20)


i'm sorry for my poor english wirting skills…

Offline

#81 2013-05-26 16:05:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

whats the output of this:

zcat /proc/config.gz | grep VFIO

Offline

#82 2013-05-26 16:07:37

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zcat /proc/config.gz | grep VFIO returns this

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

i'm sorry for my poor english wirting skills…

Offline

#83 2013-05-26 16:10:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you using my packages from the "radeon related problems" section?

Last edited by nbhs (2013-05-26 16:23:19)

Offline

#84 2013-05-26 16:18:25

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, the qemu 1.5 and linux-mainline packages. But I tried with linux-ck from aur with VFIO enabled before and it didn't work, too.


i'm sorry for my poor english wirting skills…

Offline

#85 2013-05-26 16:20:04

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

disable these lines "security=tomoyo irqpoll" "and add "iommu=pt" to you grub cfg

EDIT: use my seabios-git package please, unmodiffied

Last edited by nbhs (2013-05-26 16:23:33)

Offline

#86 2013-05-26 16:38:59

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I applied both (except I use syslinux instead of grub), but still same result. I think I'll enabe DEBUG_VFIO as detailed here: https://lists.gnu.org/archive/html/qemu … 00432.html next, does this make sense?
edit: I think the second graphics card's spins up when I try to start qemu, but it's really quiet, so I'm not 100% sure.

Last edited by andy123 (2013-05-26 16:41:16)


i'm sorry for my poor english wirting skills…

Offline

#87 2013-05-26 16:42:19

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

I applied both (except I use syslinux instead of grub), but still same result. I think I'll enabe DEBUG_VFIO as detailed here: https://lists.gnu.org/archive/html/qemu … 00432.html next, does this make sense?

Sure but you should report it on qemu-devel, seeing your mobo is very similar to mine i dont see any reason it shouldnt work, have you try to disable the nvidia and nouveau modules? changing pcie ports? laoding you rom by file? swapping cards? you could also pass it through using pci-assign instead of vfio-pci like https://bbs.archlinux.org/viewtopic.php … 3#p1274453

EDIT: you can find your card rom here http://www.techpowerup.com/vgabios/ or dump it via sysfs

Last edited by nbhs (2013-05-26 16:49:13)

Offline

#88 2013-05-26 16:54:37

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Sure but you should report it on qemu-devel, seeing your mobo is very similar to mine i dont see any reason it shouldnt work, have you try to disable the nvidia and nouveau modules? changing pcie ports? laoding you rom by file? swapping cards? you could also pass it through using pci-assign instead of vfio-pci like https://bbs.archlinux.org/viewtopic.php … 3#p1274453

I disabled the radeon module (because I try to forward the Radeon card to the guest) and loading the graphics card rom (dumped via "echo 1 > cat /sys/bus/pci/devices/0000:06:00.0/rom ; cat  /sys/bus/pci/devices/0000:06:00.0/rom > vgabios-5670.rom) with the additional parameters "rombar=0,romfile=/root/vgabios-5670.rom," to the -device directive. I'll try pci-assign now and swap the cards later (both physical and "logical" (try to pass the GeForce through)). If it still doesn't work after that, I'll take a break, recompile the kernel and report to qemu-devel.

PS: the radeon hd 5670 doesn't support triple monitors, or does it?


i'm sorry for my poor english wirting skills…

Offline

#89 2013-05-26 17:04:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Sure but you should report it on qemu-devel, seeing your mobo is very similar to mine i dont see any reason it shouldnt work, have you try to disable the nvidia and nouveau modules? changing pcie ports? laoding you rom by file? swapping cards? you could also pass it through using pci-assign instead of vfio-pci like https://bbs.archlinux.org/viewtopic.php … 3#p1274453

I disabled the radeon module (because I try to forward the Radeon card to the guest) and loading the graphics card rom (dumped via "echo 1 > cat /sys/bus/pci/devices/0000:06:00.0/rom ; cat  /sys/bus/pci/devices/0000:06:00.0/rom > vgabios-5670.rom) with the additional parameters "rombar=0,romfile=/root/vgabios-5670.rom," to the -device directive. I'll try pci-assign now and swap the cards later (both physical and "logical" (try to pass the GeForce through)). If it still doesn't work after that, I'll take a break, recompile the kernel and report to qemu-devel.

What im saying is trying to disable every driver: radeon, nvidia, nouveau so we can discard that as a factor even though you're only passing through the radeon.

PS: the radeon hd 5670 doesn't support triple monitors, or does it?

No idea

Offline

#90 2013-05-26 17:08:26

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

What im saying is trying to disable every driver: radeon, nvidia, nouveau so we can discard that as a factor even though you're only passing through the radeon.

So how would I start qemu then? Do I still have terminal without nouveau or nvidia or do I have to use ssh?


i'm sorry for my poor english wirting skills…

Offline

#91 2013-05-26 17:10:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, you'll have a terminal

if you use pci-assign you'll need to bind the gpu and the gpu snd to pci-stub instead of vfio-pci

Offline

#92 2013-05-26 22:44:34

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so I'm on 3.10 rc2 now and with vfio-pci the radeon card displays the seabios menu but as soon as a bootloader (grub, isolinux, the windows installer cd thing) trys to display something the screengoes black (but seems to stay on). Furthermore, qemu is telling me this:

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

With the nvidia card passed through it says something about not being able to reset it (and some other strange things happend, like only output on the first launch, but I want the nvidia for the host anyway…).

edit: I think I read somewhere in this thread, that windows didn't boot for you with a newer seabios version and something about out of sync repositories, could you tell me which version you're using (I'm on rel-1.7.2-86-gf465e1e)?
edit2: ok I looked at http://git.qemu.org/?p=seabios.git;a=summary and the last change was a few months ago, so we should have the same seabios version (I used your pkgbuild, but if they had changed the repository…)

Last edited by andy123 (2013-05-26 23:06:57)


i'm sorry for my poor english wirting skills…

Offline

#93 2013-05-27 00:19:06

reacocard
Member
Registered: 2008-05-25
Posts: 20
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the guide! Worked great on my system, better than the Xen passthrough I had been using. I still haven't managed to make the emulated audio work without distortion, but as I had the same problem with Xen I bought a usb soundcard to pass through, which works very well.

I do have a small tip for improving the guide - rather than blacklisting modules, you can use 'softdep' instead to make pci-stub load before radeon/nouveau/etc. To use it, add this to a .conf file under /etc/modprobe:

softdep radeon pre: pci-stub

Replace radeon with your driver. Now add pci-stub to MODULES in /etc/mkinitcpio.conf and run mkinitcpio to regenerate the image. That's it!

I needed this instead of blacklisting because I actually have two radeon cards, one for the host and one for the VM, so just outright blacklisting radeon would have left me with no drivers for the host's card.

Last edited by reacocard (2013-05-27 00:22:45)

Offline

#94 2013-05-27 00:33:19

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Ok, so I'm on 3.10 rc2 now and with vfio-pci the radeon card displays the seabios menu but as soon as a bootloader (grub, isolinux, the windows installer cd thing) trys to display something the screengoes black (but seems to stay on). Furthermore, qemu is telling me this:

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

This might happen if you're mixing packages, for example qemu 1.5 from "radeon related packages" and a not patched kernel, since my qemu 1.5  package contains code to do a pcie bus reset, you'll have to patch kernel 3.10-rc2 or use my patched linux-mainline build, this code is not yet on the main qemu/kernel tree.

the code is from

git://github.com/awilliam/linux-vfio.git vfio-vga-reset git://github.com/awilliam/qemu-vfio.git vfio-vga-reset 

read http://qemu.11.n7.nabble.com/VFIO-VGA-t … 01483.html

With the nvidia card passed through it says something about not being able to reset it (and some other strange things happend, like only output on the first launch, but I want the nvidia for the host anyway…).

Same with my nvidia card, but your problem on second launch might be related with what i mentioned before

You can use this patch for 3.10-rc2 in combination with my qemu 1.5 package:

vfio-vga-reset.patch.tar.gz

Last edited by nbhs (2013-05-27 00:51:39)

Offline

#95 2013-05-27 00:35:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reacocard wrote:

Thanks for the guide! Worked great on my system, better than the Xen passthrough I had been using. I still haven't managed to make the emulated audio work without distortion, but as I had the same problem with Xen I bought a usb soundcard to pass through, which works very well.

I do have a small tip for improving the guide - rather than blacklisting modules, you can use 'softdep' instead to make pci-stub load before radeon/nouveau/etc. To use it, add this to a .conf file under /etc/modprobe:

softdep radeon pre: pci-stub

Replace radeon with your driver. Now add pci-stub to MODULES in /etc/mkinitcpio.conf and run mkinitcpio to regenerate the image. That's it!

I needed this instead of blacklisting because I actually have two radeon cards, one for the host and one for the VM, so just outright blacklisting radeon would have left me with no drivers for the host's card.

NIce!, can you mention your setup?

Im using 2 radeon cards too, and im using pci-stub, i mentioned it in the guide.

softdep radeon pre: pci-stub

wont that bind both cards to pci-stub?

EDIT: Nvm i think i understand now, tnx for the info

Last edited by nbhs (2013-05-27 00:45:08)

Offline

#96 2013-05-27 00:52:15

reacocard
Member
Registered: 2008-05-25
Posts: 20
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
reacocard wrote:

Thanks for the guide! Worked great on my system, better than the Xen passthrough I had been using. I still haven't managed to make the emulated audio work without distortion, but as I had the same problem with Xen I bought a usb soundcard to pass through, which works very well.

I do have a small tip for improving the guide - rather than blacklisting modules, you can use 'softdep' instead to make pci-stub load before radeon/nouveau/etc. To use it, add this to a .conf file under /etc/modprobe:

softdep radeon pre: pci-stub

Replace radeon with your driver. Now add pci-stub to MODULES in /etc/mkinitcpio.conf and run mkinitcpio to regenerate the image. That's it!

I needed this instead of blacklisting because I actually have two radeon cards, one for the host and one for the VM, so just outright blacklisting radeon would have left me with no drivers for the host's card.

NIce!, can you mention your setup?

AMD FX-4170 CPU
Gigabyte GA-990FXA-UD3 motherboard
16GB DDR3-1600 RAM
Radeon 6850 - Host
Radeon 7770 - VM
This usb soundcard for the guest: http://www.amazon.com/gp/product/B001MS … UTF8&psc=1 (it actually has a bug where it sounds awful by default, but if you set the output sampling rate to 48khz instead of 44.1khz it becomes perfect)
3 ssds (various kinds and sizes), one for host, one for guest os, one for guest game storage.

Using the patched kernel/qemu/seabios from your first post.

Here's the script I use to launch the VM: https://dl.dropboxusercontent.com/u/147 … /gaming.sh

Im using 2 radeon cards too, and im using pci-stub, i mentioned it in the guide.

How did you avoid having both blacklisted? It's not clear from your guide what, if anything, had to be done. I just copied over this trick from my prior xen setup and didn't try blacklisting at all.

softdep radeon pre: pci-stub

wont that bind both cards to pci-stub?

No, because the two cards are different and have different pci ids, so pci-stub will only bind the one whose id was given to it in pci-stub.ids. This would presumably break if your cards were identical however.

Last edited by reacocard (2013-05-27 00:56:03)

Offline

#97 2013-05-27 00:57:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How did you avoid having both blacklisted? It's not clear from your guide what, if anything, had to be done. I just copied over this trick from my prior xen setup and didn't try blacklisting at all.

first i added pci-stub to mkinitcpio.conf

then i binded my card to pci-stub like this

lspci

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350 Series] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]

lspci -n

01:00.0 0300: 1002:68f9 01:00.1 0403: 1002:aa68 07:00.0 0300: 1002:6719 <-- this one i want to passthrough 07:00.1 0403: 1002:aa80 <-- this one i want to passthrough

so i added this line to grub

pci-stub.ids=1002:6719,1002:aa80

My card gets binded to pci-stub at boot

Last edited by nbhs (2013-05-27 00:57:55)

Offline

#98 2013-05-27 01:02:06

reacocard
Member
Registered: 2008-05-25
Posts: 20
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

How did you avoid having both blacklisted? It's not clear from your guide what, if anything, had to be done. I just copied over this trick from my prior xen setup and didn't try blacklisting at all.

first i added pci-stub to mkinitcpio.conf

then i binded my card to pci-stub like this

lspci

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350 Series] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]

lspci -n

01:00.0 0300: 1002:68f9 01:00.1 0403: 1002:aa68 07:00.0 0300: 1002:6719 <-- this one i want to passthrough 07:00.1 0403: 1002:aa80 <-- this one i want to passthrough

so i added this line to grub

pci-stub.ids=1002:6719,1002:aa80

Ah, and your radeon isn't in mkinitcpio.conf, so it loads later and only claims the cards pci-stub hasn't? Basically the same effect as my method then, just without the explicit dependency. I needed the dependencies under xen because I was also passing through a USB controller and the USB modules were all getting loaded in the initrd before I could load xen-pciback (the xen equivalent to pci-stub), so I found softdep as a workaround. It's still useful here though, because I now also can have radeon in initrd so I get early KMS. smile

Offline

#99 2013-05-27 01:05:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yup radeon module isnt on the initramfs but it works even if it is.

Offline

#100 2013-05-27 12:59:11

ezhik
Member
Registered: 2013-05-27
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

since qemu 1.5 released
is it possible to make vfio based vga passthru without patching (either qemu or its seabios)?

Offline

#101 2013-05-27 22:20:59

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so I made progress. I'm able to boot with "-vga cirrus", start X in an archlinux vm and run gl apps (like unigine heaven) on the radeon card.
However, I'm stuck with installing Windows, because it doesn't recognise drives to install it to…
I tried passing through my second SATA controller, but that thing is really buggy (even on the host) and produces messages like this.

[ 1235.907697] vfio-pci 0000:02:00.0: Invalid ROM contents [ 1251.533814] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533825] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533829] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533834] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070] [ 1251.533838] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070]

Any ideas?


i'm sorry for my poor english wirting skills…

Offline

#102 2013-05-27 23:57:55

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Ok, so I made progress. I'm able to boot with "-vga cirrus", start X in an archlinux vm and run gl apps (like unigine heaven) on the radeon card.
However, I'm stuck with installing Windows, because it doesn't recognise drives to install it to…
I tried passing through my second SATA controller, but that thing is really buggy (even on the host) and produces messages like this.

[ 1235.907697] vfio-pci 0000:02:00.0: Invalid ROM contents [ 1251.533814] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533825] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533829] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533834] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070] [ 1251.533838] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070]

Any ideas?

try using the amd sata controller for passthrough, i had no problems passing it through

i use the asmedia controller for linux and the amd controller for windows vm

Last edited by nbhs (2013-05-27 23:58:30)

Offline

#103 2013-05-28 00:14:00

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

vfio-pci 0000:02:00.0: Invalid ROM contents

This used to happen on my radeon 6950 the solution was to load the rom by file

Offline

#104 2013-05-28 00:25:09

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
andy123 wrote:

Ok, so I made progress. I'm able to boot with "-vga cirrus", start X in an archlinux vm and run gl apps (like unigine heaven) on the radeon card.
However, I'm stuck with installing Windows, because it doesn't recognise drives to install it to…
I tried passing through my second SATA controller, but that thing is really buggy (even on the host) and produces messages like this.

[ 1235.907697] vfio-pci 0000:02:00.0: Invalid ROM contents [ 1251.533814] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533825] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533829] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533834] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070] [ 1251.533838] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070]

Any ideas?

try using the amd sata controller for passthrough, i had no problems passing it through

i use the asmedia controller for linux and the amd controller for windows vm

Yeah, that's sadly not an option… The amd controller has 6 ports, which I have 4 hdds, 1 ssd and a dvd drive attached to.
The strange thing is, Windows doesn't recognise the emulated qemu controller. Also the q35 chipset only boots scsi disks, not ide for some reason. I'm trying some stuff with a different emulated chipset at the moment and see how it turns out and report back later.

Also 02:00.0 is my 2nd sata controller. I think I'll open another thread about that thing… the uefi has options to be able to boot from it and the read performance in linux is really bad.


i'm sorry for my poor english wirting skills…

Offline

#105 2013-05-28 00:37:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
andy123 wrote:

Ok, so I made progress. I'm able to boot with "-vga cirrus", start X in an archlinux vm and run gl apps (like unigine heaven) on the radeon card.
However, I'm stuck with installing Windows, because it doesn't recognise drives to install it to…
I tried passing through my second SATA controller, but that thing is really buggy (even on the host) and produces messages like this.

[ 1235.907697] vfio-pci 0000:02:00.0: Invalid ROM contents [ 1251.533814] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533825] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533829] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000000 flags=0x0070] [ 1251.533834] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070] [ 1251.533838] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000000040 flags=0x0070]

Any ideas?

try using the amd sata controller for passthrough, i had no problems passing it through

i use the asmedia controller for linux and the amd controller for windows vm

Yeah, that's sadly not an option… The amd controller has 6 ports, which I have 4 hdds, 1 ssd and a dvd drive attached to.
The strange thing is, Windows doesn't recognise the emulated qemu controller. Also the q35 chipset only boots scsi disks, not ide for some reason. I'm trying some stuff with a different emulated chipset at the moment and see how it turns out and report back later.

Also 02:00.0 is my 2nd sata controller. I think I'll open another thread about that thing… the uefi has options to be able to boot from it and the read performance in linux is really bad.

Its on the guide, the default emulated q35 sata controller doesnt work, you should re-read that part, for best performance you should use virtio disk and drivers.

EDIT: see this post by jgott https://bbs.archlinux.org/viewtopic.php … 9#p1276259 on virtio disks

Last edited by nbhs (2013-05-28 00:48:10)

Offline

#106 2013-05-28 01:05:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ezhik wrote:

since qemu 1.5 released
is it possible to make vfio based vga passthru without patching (either qemu or its seabios)?

Probably not.

Offline

#107 2013-05-28 03:16:25

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Its on the guide, the default emulated q35 sata controller doesnt work, you should re-read that part, for best performance you should use virtio disk and drivers.

EDIT: see this post by jgott https://bbs.archlinux.org/viewtopic.php … 9#p1276259 on virtio disks

Thanks, I somehow forgot about that part/overlooked it. I'm now in windows, but I still can't boot with "-vga none", so I have two graphics cards and Windows picks the wrong one. The same setup works perfect in a (arch) linux vm. How would I go on about convincing windows to use the right graphics card?


i'm sorry for my poor english wirting skills…

Offline

#108 2013-05-28 03:23:47

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Its on the guide, the default emulated q35 sata controller doesnt work, you should re-read that part, for best performance you should use virtio disk and drivers.

EDIT: see this post by jgott https://bbs.archlinux.org/viewtopic.php … 9#p1276259 on virtio disks

Thanks, I somehow forgot about that part/overlooked it. I'm now in windows, but I still can't boot with "-vga none", so I have two graphics cards and Windows picks the wrong one. The same setup works perfect in a (arch) linux vm. How would I go on about convincing windows to use the right graphics card?

Right click your desktop go to properties-->screen resolution there you can choose the primary display,  on my vm using -vga cirrus wont work with x-vga=on, so you might have to disable that line.

To check if your device is working go to device manager and make sure you gpu doesnt have a yellow marking saying the device cannot start

Last edited by nbhs (2013-05-28 03:24:59)

Offline

#109 2013-05-28 07:50:12

ezhik
Member
Registered: 2013-05-27
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
ezhik wrote:

since qemu 1.5 released
is it possible to make vfio based vga passthru without patching (either qemu or its seabios)?

Probably not.


could you make experiment with vanilla qemu 1.5 (with internal seabios) and kernel 3.10rc3 (with you patch for 3.10rc2) ?
also how you figure out your motherboard have broken VT-d? Is there any test available?

In my case (asus p8h77-v-le + 3770 + radeon 7750) I cannot see seabios, but able to get working framebuffer (from the ubuntu live cd), while X server is not working and crashing in strange way (inside pthread lib)

Offline

#110 2013-05-28 10:13:05

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ezhik wrote:
nbhs wrote:
ezhik wrote:

since qemu 1.5 released
is it possible to make vfio based vga passthru without patching (either qemu or its seabios)?

Probably not.


could you make experiment with vanilla qemu 1.5 (with internal seabios) and kernel 3.10rc3 (with you patch for 3.10rc2) ?
also how you figure out your motherboard have broken VT-d? Is there any test available?

In my case (asus p8h77-v-le + 3770 + radeon 7750) I cannot see seabios, but able to get working framebuffer (from the ubuntu live cd), while X server is not working and crashing in strange way (inside pthread lib)

Internal seabios wont work, it doesnt have the required commit, it needs to be patched. Even if it worked out of the box you would still need to patch the kernel and qemu, otherwise resetting the vm using radeon cards results in the host freezing and/or performance degradation.

Passing radeon cards as secondary devices is quite easy, you can use virt-manager to do it and any version of kernel and qemu, but you will experience the symptoms above (performance degradation/freezing) which can be resolved using the "eject method" or suspending the host between vm runs.

also how you figure out your motherboard have broken VT-d? Is there any test available?

I'm afraid i have no experience using vt-d, but i suppose doing a simple pci passthrough would be one.

Last edited by nbhs (2013-05-28 12:20:39)

Offline

#111 2013-05-28 14:10:09

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Right click your desktop go to properties-->screen resolution there you can choose the primary display,  on my vm using -vga cirrus wont work with x-vga=on, so you might have to disable that line.

To check if your device is working go to device manager and make sure you gpu doesnt have a yellow marking saying the device cannot start

I can't choose primary display and there is a yellow marking in the device manager. I tried it with x-vga=on and without and also loading a rom file, but the emulated vga-bios doesn't go away then/the vm doesn't boot at all.
The properties dialog in the device manager says something about error "Code 12" and conflicts (I would copy&past the whole message, but it's in german and I forgot how to change the language in windows). I googled it but only found things like this and that wasn't very helpful.


i'm sorry for my poor english wirting skills…

Offline

#112 2013-05-28 14:16:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Right click your desktop go to properties-->screen resolution there you can choose the primary display,  on my vm using -vga cirrus wont work with x-vga=on, so you might have to disable that line.

To check if your device is working go to device manager and make sure you gpu doesnt have a yellow marking saying the device cannot start

I can't choose primary display and there is a yellow marking in the device manager. I tried it with x-vga=on and without and also loading a rom file, but the emulated vga-bios doesn't go away then/the vm doesn't boot at all.
The properties dialog in the device manager says something about error "Code 12" and conflicts (I would copy&past the whole message, but it's in german and I forgot how to change the language in windows). I googled it but only found things like this and that wasn't very helpful.

OK so forget about vfio, pass it through as secondary using pci-assign

First bind your card to pci-stub, you can follow those steps on the guide, dont bind your card to vfio-pci, launch qemu without q35 and use

-device pci-assign,host=06:00.0 -device pci-assign,host=06:00.1

This will passthrough your card as secondary vga device, though you might have problems rebooting the vm, use the eject method from my guide or the suspend method to solve this.

Last edited by nbhs (2013-05-28 14:17:48)

Offline

#113 2013-05-28 15:27:46

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

OK so forget about vfio, pass it through as secondary using pci-assign

First bind your card to pci-stub, you can follow those steps on the guide, dont bind your card to vfio-pci, launch qemu without q35 and use

-device pci-assign,host=06:00.0 -device pci-assign,host=06:00.1

This will passthrough your card as secondary vga device, though you might have problems rebooting the vm, use the eject method from my guide or the suspend method to solve this.

Wow, just wow. Is windows trying to troll me, or whats up here? Now it bluescreens while loading atikmpag.sys. How comes all of this works in a Linux vm and windows fails so hard? That's what annoys me most about this operating system, you start doing something only half crazy and it's like "No, I can't do that sam!".

edit: I rebootet the host and it works now… yeah, no idea. I'm kind of happy.
Thanks for the tutorial and help nbhs, I'd never managed to get this working without you^^

edit2: ok… the bsod is back, so I'll try the whole "eject before shutdown" stuff now

Last edited by andy123 (2013-05-28 16:56:21)


i'm sorry for my poor english wirting skills…

Offline

#114 2013-05-28 17:15:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

OK so forget about vfio, pass it through as secondary using pci-assign

First bind your card to pci-stub, you can follow those steps on the guide, dont bind your card to vfio-pci, launch qemu without q35 and use

-device pci-assign,host=06:00.0 -device pci-assign,host=06:00.1

This will passthrough your card as secondary vga device, though you might have problems rebooting the vm, use the eject method from my guide or the suspend method to solve this.

Wow, just wow. Is windows trying to troll me, or whats up here? Now it bluescreens while loading atikmpag.sys. How comes all of this works in a Linux vm and windows fails so hard? That's what annoys me most about this operating system, you start doing something only half crazy and it's like "No, I can't do that sam!".

edit: I rebootet the host and it works now… yeah, no idea. I'm kind of happy.
Thanks for the tutorial and help nbhs, I'd never managed to get this working without you^^

edit2: ok… the bsod is back, so I'll try the whole "eject before shutdown" stuff now

Try doing a clean windows install, and using my seabios package. Can you show me how you're launching qemu?

Last edited by nbhs (2013-05-28 17:15:36)

Offline

#115 2013-05-28 17:38:09

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Try doing a clean windows install, and using my seabios package. Can you show me how you're launching qemu?

I use your package and this is a clean install, but I'm using pci-assign, so according to your tutorial, deveject is the way to go (if you don't use vfio), or did I misunderstand that?

/usr/bin/qemu-system-x86_64 -bios /usr/share/qemu/bios.bin -enable-kvm -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 4096 -device pci-assign,host=06:00.0 -device pci-assign,host=06:00.1 -boot menu=on -monitor stdio -usb -device usb-host,hostbus=6 -hda win7_64.img -boot c

I'll try virtio after this runs stable.


i'm sorry for my poor english wirting skills…

Offline

#116 2013-05-28 17:40:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Try doing a clean windows install, and using my seabios package. Can you show me how you're launching qemu?

I use your package and this is a clean install, but I'm using pci-assign, so according to your tutorial, deveject is the way to go (if you don't use vfio), or did I misunderstand that?

/usr/bin/qemu-system-x86_64 -bios /usr/share/qemu/bios.bin -enable-kvm -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 4096 -device pci-assign,host=06:00.0 -device pci-assign,host=06:00.1 -boot menu=on -monitor stdio -usb -device usb-host,hostbus=6 -hda win7_64.img -boot c

I'll try virtio after this runs stable.

Yes deveject is used to automate the ejecting, but you can eject the card from windows just like you would eject a pendrive.

EDIT: make sure ejecting is done on the vm first run.

Last edited by nbhs (2013-05-28 17:46:26)

Offline

#117 2013-05-28 19:30:35

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Yes deveject is used to automate the ejecting, but you can eject the card from windows just like you would eject a pendrive.

EDIT: make sure ejecting is done on the vm first run.

deveject works perfect, the vm switches from the cirrus card to the radeon on bootup and back und shutdown big_smile
I'm still a bit unclear on the virtio stuff though, I downloaded the drivers, but how do I install them (srsly, windows is confusing…)?


i'm sorry for my poor english wirting skills…

Offline

#118 2013-05-28 19:36:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Yes deveject is used to automate the ejecting, but you can eject the card from windows just like you would eject a pendrive.

EDIT: make sure ejecting is done on the vm first run.

deveject works perfect, the vm switches from the cirrus card to the radeon on bootup and back und shutdown big_smile
I'm still a bit unclear on the virtio stuff though, I downloaded the drivers, but how do I install them (srsly, windows is confusing…)?

Take a look at https://bbs.archlinux.org/viewtopic.php … 9#p1276259

You need to attach the virtio controller, then attach your disk image to it, since you're using regular pci-assign and not using q35, you can use virt-manager to do all this.

Offline

#119 2013-05-28 21:16:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:
nbhs wrote:
apoapo wrote:

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things smile.

NIce! have you tried running windows with it yet? does it work if you dont load the rom file by hand?

Hi,

I tried it again and I captured this error message if I use the PCI-e card es Primary Card:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow

This was, when I got some seabios picture.

After some reboots I wanted to try it again, I was spammed with errors sad. Dont have them to show you now...  will look for it.


Now I'm With the onboard as primary Card and I found this in dmesg:

[ 0.401391] DMAR: No ATSR found [ 0.401412] IOMMU 0 0xfed90000: using Queued invalidation [ 0.401413] IOMMU 1 0xfed91000: using Queued invalidation [ 0.401415] IOMMU: Setting RMRR: [ 0.401422] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.402590] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd568000 - 0xbd59afff] [ 0.402606] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd568000 - 0xbd59afff] [ 0.402619] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd568000 - 0xbd59afff] [ 0.402628] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.402635] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.402717] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.402782] ------------[ cut here ]------------ [ 0.402786] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x83/0x90() [ 0.402787] Hardware name: To Be Filled By O.E.M. [ 0.402788] Modules linked in: [ 0.402790] Pid: 1, comm: swapper/0 Not tainted 3.9.2-1-mainline #1 [ 0.402791] Call Trace: [ 0.402796] [<ffffffff81058090>] warn_slowpath_common+0x70/0xa0 [ 0.402797] [<ffffffff8105817a>] warn_slowpath_null+0x1a/0x20 [ 0.402799] [<ffffffff8129ea43>] pci_find_upstream_pcie_bridge+0x83/0x90 [ 0.402802] [<ffffffff813b3c7b>] intel_iommu_add_device+0x4b/0x1f0 [ 0.402805] [<ffffffff813ab630>] ? bus_set_iommu+0x50/0x50 [ 0.402806] [<ffffffff813ab65a>] add_iommu_group+0x2a/0x50 [ 0.402809] [<ffffffff8135bf63>] bus_for_each_dev+0x63/0xa0 [ 0.402811] [<ffffffff813ab622>] bus_set_iommu+0x42/0x50 [ 0.402814] [<ffffffff819130b6>] intel_iommu_init+0xa83/0xb7b [ 0.402816] [<ffffffff818dd10b>] ? memblock_find_dma_reserve+0x124/0x124 [ 0.402818] [<ffffffff818dd11d>] pci_iommu_init+0x12/0x3c [ 0.402820] [<ffffffff8100210a>] do_one_initcall+0x10a/0x160 [ 0.402823] [<ffffffff818d5037>] kernel_init_freeable+0x15b/0x1dc [ 0.402824] [<ffffffff818d4881>] ? do_early_param+0x88/0x88 [ 0.402827] [<ffffffff814b55f0>] ? rest_init+0x90/0x90 [ 0.402828] [<ffffffff814b55fe>] kernel_init+0xe/0x190 [ 0.402830] [<ffffffff814dad6c>] ret_from_fork+0x7c/0xb0 [ 0.402832] [<ffffffff814b55f0>] ? rest_init+0x90/0x9

I think this bug is related to this: https://bugzilla.kernel.org/show_bug.cgi?id=44881

Here's the solution: http://www.spinics.net/lists/linux-pci/msg22136.html

Offline

#120 2013-05-29 16:48:05

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First of all: nbhs - BIG THANKS for sharing your knowledge!

I am running a setup with VGA-Passthrough myself. The setup consists of a Athlon II 640 on some Asus M99XEVO (990x/950sb) paired with a HD6850 for DomU and 4GB RAM - the dom0 got its own Matrox Millenium II smile.
But i am wondering why flash videos and games tend to stutter (even sound through hdmi) and  so on. It seems like my HD6850 has some lags (?). Does anybody experience the same problems? I was trying to setup a small howto in german, but beeing this far from a enjoyable solution, i tend to experiement some more before writing some stuff. *edit* Chipset is Fx440 and disks are already virtio (image file, drivers from http://alt.fedoraproject.org/pub/alt/vi … images/bin ).

Second question: what´s your 3Dmark score under xen? Any noticeable differences in performance?

Greetings

Last edited by Diggo (2013-05-29 16:53:20)

Offline

#121 2013-05-29 18:46:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:

First of all: nbhs - BIG THANKS for sharing your knowledge!

I am running a setup with VGA-Passthrough myself. The setup consists of a Athlon II 640 on some Asus M99XEVO (990x/950sb) paired with a HD6850 for DomU and 4GB RAM - the dom0 got its own Matrox Millenium II smile.
But i am wondering why flash videos and games tend to stutter (even sound through hdmi) and  so on. It seems like my HD6850 has some lags (?). Does anybody experience the same problems? I was trying to setup a small howto in german, but beeing this far from a enjoyable solution, i tend to experiement some more before writing some stuff. *edit* Chipset is Fx440 and disks are already virtio (image file, drivers from http://alt.fedoraproject.org/pub/alt/vi … images/bin ).

Second question: what´s your 3Dmark score under xen? Any noticeable differences in performance?

Greetings

The redhat dev recommends using q35 chipset and the gpu behind a pcie root port, i dont use hdmi, so i have no experience with it, i havent experienced stutter in flash but i did experience some in games, i solved this by disabling nested page tables and tweaking the kernel config (see my second post on first page).

Xen 3dmark vs kvm is about the same with the above tweaks.

Last edited by nbhs (2013-05-29 18:53:30)

Offline

#122 2013-05-29 19:06:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Hi, I try to get vga-passthrough to work on my system at the moment, but the second graphics card doesn't output anything and qemu freezes.

My setup is…
hardware:
Mainboard: AsRock 990FX Extreme4
CPU: AMD FX-8350
GPU0 (host): Gigabyte GeForce GTX 650
GPU1 (guest): Sapphire Radeon HD 5670
RAM: G.SKILL F3-10666CL9-8GBXL

software:
linux-mainline (pkg from the original post updated to 3.9.4)
qemu (pkg from original post)
seabios-git (pkg from original post adapted to new VCS syntax/rel1.7.286gf465e1e-1)

lspci output

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) 00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E) 00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GTX 650] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK107 HDMI Audio Controller (rev a1) 02:00.0 IDE interface: Marvell Technology Group Ltd. 88SE91A0 SATA 6Gb/s Controller (rev 12) 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 05:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood XT [Radeon HD 5670] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] 08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 01)

dmesg output

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 3.9.4-1-mainline (andy@desk) (gcc version 4.8.0 20130502 (prerelease) (GCC) ) #1 SMP PREEMPT Sun May 26 16:09:27 CEST 2013 [ 0.000000] Command line: ../vmlinuz-linux-mainline ../vmlinuz-linux-mainline root=/dev/disk/by-id/ata-ADATA_SP900_02B20081500200001033-part3 rootflags=subvol=root quiet ro security=tomoyo irqpoll initrd=../initramfs-linux-mainline.img [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009cfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000009df04fff] usable [ 0.000000] BIOS-e820: [mem 0x000000009df05000-0x000000009e4e2fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000009e4e3000-0x000000009e8d5fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000009e8d6000-0x000000009ee28fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000009ee29000-0x000000009ee29fff] usable [ 0.000000] BIOS-e820: [mem 0x000000009ee2a000-0x000000009f02ffff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000009f030000-0x000000009f452fff] usable [ 0.000000] BIOS-e820: [mem 0x000000009f453000-0x000000009f7f3fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000009f7f4000-0x000000009f7fffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec20000-0x00000000fec20fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed61000-0x00000000fed70fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fef00000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100001000-0x000000025effffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./990FX Extreme4, BIOS P2.00 10/15/2012 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] No AGP bridge found [ 0.000000] e820: last_pfn = 0x25f000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF write-through [ 0.000000] C0000-D1FFF write-protect [ 0.000000] D2000-EAFFF uncachable [ 0.000000] EB000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask FFFF80000000 write-back [ 0.000000] 1 base 000080000000 mask FFFFE0000000 write-back [ 0.000000] 2 base 00009F800000 mask FFFFFF800000 uncachable [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] TOM2: 000000025f000000 aka 9712M [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: update [mem 0x9f800000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0x9f800 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd9e0-0x000fd9ef] mapped at [ffff8800000fd9e0] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01b0e000, 0x01b0efff] PGTABLE [ 0.000000] BRK [0x01b0f000, 0x01b0ffff] PGTABLE [ 0.000000] BRK [0x01b10000, 0x01b10fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x25ee00000-0x25effffff] [ 0.000000] [mem 0x25ee00000-0x25effffff] page 2M [ 0.000000] BRK [0x01b11000, 0x01b11fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x25c000000-0x25edfffff] [ 0.000000] [mem 0x25c000000-0x25edfffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x200000000-0x25bffffff] [ 0.000000] [mem 0x200000000-0x23fffffff] page 1G [ 0.000000] [mem 0x240000000-0x25bffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0x9df04fff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0x9ddfffff] page 2M [ 0.000000] [mem 0x9de00000-0x9df04fff] page 4k [ 0.000000] init_memory_mapping: [mem 0x9ee29000-0x9ee29fff] [ 0.000000] [mem 0x9ee29000-0x9ee29fff] page 4k [ 0.000000] BRK [0x01b12000, 0x01b12fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x9f030000-0x9f452fff] [ 0.000000] [mem 0x9f030000-0x9f1fffff] page 4k [ 0.000000] [mem 0x9f200000-0x9f3fffff] page 2M [ 0.000000] [mem 0x9f400000-0x9f452fff] page 4k [ 0.000000] init_memory_mapping: [mem 0x9f7f4000-0x9f7fffff] [ 0.000000] [mem 0x9f7f4000-0x9f7fffff] page 4k [ 0.000000] init_memory_mapping: [mem 0x100001000-0x1ffffffff] [ 0.000000] [mem 0x100001000-0x1001fffff] page 4k [ 0.000000] [mem 0x100200000-0x13fffffff] page 2M [ 0.000000] [mem 0x140000000-0x1ffffffff] page 1G [ 0.000000] RAMDISK: [mem 0x7fa42000-0x7fffffff] [ 0.000000] ACPI: RSDP 00000000000f0490 00024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 000000009e8c5078 0006C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 000000009e8cb600 0010C (v05 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130117/tbfadt-599) [ 0.000000] ACPI: DSDT 000000009e8c5178 06483 (v02 ALASKA A M I 00000000 INTL 20051117) [ 0.000000] ACPI: FACS 000000009e8cf080 00040 [ 0.000000] ACPI: APIC 000000009e8cb710 0009E (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 000000009e8cb7b0 00044 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: MCFG 000000009e8cb7f8 0003C (v01 ALASKA A M I 01072009 MSFT 00010013) [ 0.000000] ACPI: AAFT 000000009e8cb838 0008A (v01 ALASKA OEMAAFT 01072009 MSFT 00000097) [ 0.000000] ACPI: HPET 000000009e8cb8c8 00038 (v01 ALASKA A M I 01072009 AMI 00000005) [ 0.000000] ACPI: SSDT 000000009e8cb900 01714 (v01 AMD POWERNOW 00000001 AMD 00000001) [ 0.000000] ACPI: IVRS 000000009e8cd018 000E8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: BGRT 000000009e8cd100 00038 (v00 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000025effffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x25effffff] [ 0.000000] NODE_DATA [mem 0x25eff4000-0x25eff8fff] [ 0.000000] [ffffea0000000000-ffffea00097fffff] PMD -> [ffff880256600000-ffff88025e5fffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x25effffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x9df04fff] [ 0.000000] node 0: [mem 0x9ee29000-0x9ee29fff] [ 0.000000] node 0: [mem 0x9f030000-0x9f452fff] [ 0.000000] node 0: [mem 0x9f7f4000-0x9f7fffff] [ 0.000000] node 0: [mem 0x100001000-0x25effffff] [ 0.000000] On node 0 totalpages: 2085584 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 10061 pages used for memmap [ 0.000000] DMA32 zone: 643893 pages, LIFO batch:31 [ 0.000000] Normal zone: 22464 pages used for memmap [ 0.000000] Normal zone: 1437695 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x808 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x17] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 9, version 33, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec20000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 10, version 33, address 0xfec20000, GSI 24-55 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ2 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x43538210 base: 0xfed00000 [ 0.000000] smpboot: Allowing 8 CPUs, 0 hotplug CPUs [ 0.000000] nr_irqs_gsi: 72 [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000 [ 0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000 [ 0.000000] PM: Registered nosave memory: 000000009df05000 - 000000009e4e3000 [ 0.000000] PM: Registered nosave memory: 000000009e4e3000 - 000000009e8d6000 [ 0.000000] PM: Registered nosave memory: 000000009e8d6000 - 000000009ee29000 [ 0.000000] PM: Registered nosave memory: 000000009ee2a000 - 000000009f030000 [ 0.000000] PM: Registered nosave memory: 000000009f453000 - 000000009f7f4000 [ 0.000000] PM: Registered nosave memory: 000000009f800000 - 00000000fec00000 [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000 [ 0.000000] PM: Registered nosave memory: 00000000fec01000 - 00000000fec10000 [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fec11000 [ 0.000000] PM: Registered nosave memory: 00000000fec11000 - 00000000fec20000 [ 0.000000] PM: Registered nosave memory: 00000000fec20000 - 00000000fec21000 [ 0.000000] PM: Registered nosave memory: 00000000fec21000 - 00000000fed00000 [ 0.000000] PM: Registered nosave memory: 00000000fed00000 - 00000000fed01000 [ 0.000000] PM: Registered nosave memory: 00000000fed01000 - 00000000fed61000 [ 0.000000] PM: Registered nosave memory: 00000000fed61000 - 00000000fed71000 [ 0.000000] PM: Registered nosave memory: 00000000fed71000 - 00000000fed80000 [ 0.000000] PM: Registered nosave memory: 00000000fed80000 - 00000000fed90000 [ 0.000000] PM: Registered nosave memory: 00000000fed90000 - 00000000fef00000 [ 0.000000] PM: Registered nosave memory: 00000000fef00000 - 0000000100000000 [ 0.000000] PM: Registered nosave memory: 0000000100000000 - 0000000100001000 [ 0.000000] e820: [mem 0x9f800000-0xfebfffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:8 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88025ec00000 s85632 r8192 d20864 u262144 [ 0.000000] pcpu-alloc: s85632 r8192 d20864 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2052974 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: ../vmlinuz-linux-mainline ../vmlinuz-linux-mainline root=/dev/disk/by-id/ata-ADATA_SP900_02B20081500200001033-part3 rootflags=subvol=root quiet ro security=tomoyo irqpoll initrd=../initramfs-linux-mainline.img [ 0.000000] Misrouted IRQ fixup and polling support enabled [ 0.000000] This may significantly impact system performance [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] __ex_table already sorted, skipping sort [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Node 0: aperture @ 94000000 size 32 MB [ 0.000000] Aperture pointing to e820 RAM. Ignoring. [ 0.000000] Your BIOS doesn't leave a aperture memory hole [ 0.000000] Please enable the IOMMU option in the BIOS setup [ 0.000000] This costs you 64 MB of RAM [ 0.000000] Mapping aperture over 65536 KB of RAM @ 94000000 [ 0.000000] PM: Registered nosave memory: 0000000094000000 - 0000000098000000 [ 0.000000] Memory: 8061176k/9945088k available (5131k kernel code, 1602752k absent, 281160k reserved, 3816k data, 1092k init) [ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] Dump stacks of tasks blocking RCU-preempt GP. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8. [ 0.000000] NR_IRQS:4352 nr_irqs:1288 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] allocated 33554432 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.001000] tsc: Detected 3991.652 MHz processor [ 0.000003] Calibrating delay loop (skipped), value calculated using timer frequency.. 7983.30 BogoMIPS (lpj=3991652) [ 0.000005] pid_max: default: 32768 minimum: 301 [ 0.000033] Security Framework initialized [ 0.000043] TOMOYO Linux initialized [ 0.000049] AppArmor: AppArmor disabled by boot time parameter [ 0.000515] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.003115] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.004338] Mount-cache hash table entries: 256 [ 0.004525] Initializing cgroup subsys cpuacct [ 0.004527] Initializing cgroup subsys memory [ 0.004536] Initializing cgroup subsys devices [ 0.004537] Initializing cgroup subsys freezer [ 0.004539] Initializing cgroup subsys net_cls [ 0.004540] Initializing cgroup subsys blkio [ 0.004563] tseg: 009f800000 [ 0.004565] CPU: Physical Processor ID: 0 [ 0.004565] CPU: Processor Core ID: 0 [ 0.004567] mce: CPU supports 7 MCE banks [ 0.004573] LVT offset 1 assigned for vector 0xf9 [ 0.004579] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512 Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512 tlb_flushall_shift: 5 [ 0.004655] Freeing SMP alternatives: 20k freed [ 0.005255] ACPI: Core revision 20130117 [ 0.007025] ACPI: All ACPI Tables successfully acquired [ 0.012122] ftrace: allocating 19520 entries in 77 pages [ 0.092544] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.102565] smpboot: CPU0: AMD FX(tm)-8350 Eight-Core Processor (fam: 15, model: 02, stepping: 00) [ 0.204459] Performance Events: [ 0.204461] perf: AMD core performance counters detected [ 0.204462] perf: AMD northbridge performance counters detected [ 0.204465] AMD PMU driver. [ 0.204466] ... version: 0 [ 0.204467] ... bit width: 48 [ 0.204468] ... generic registers: 10 [ 0.204469] ... value mask: 0000ffffffffffff [ 0.204470] ... max period: 00007fffffffffff [ 0.204470] ... fixed-purpose events: 0 [ 0.204471] ... event mask: 00000000000003ff [ 0.214526] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.216529] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK [ 0.319834] Brought up 8 CPUs [ 0.319838] smpboot: Total of 8 processors activated (63866.43 BogoMIPS) [ 0.329769] devtmpfs: initialized [ 0.331259] PM: Registering ACPI NVS region [mem 0x9e4e3000-0x9e8d5fff] (4141056 bytes) [ 0.331313] PM: Registering ACPI NVS region [mem 0x9ee2a000-0x9f02ffff] (2121728 bytes) [ 0.331974] RTC time: 15:36:07, date: 05/26/13 [ 0.332012] NET: Registered protocol family 16 [ 0.332170] ACPI: bus type PCI registered [ 0.332220] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.332222] PCI: not using MMCONFIG [ 0.332223] PCI: Using configuration type 1 for base access [ 0.332224] PCI: Using configuration type 1 for extended access [ 0.332950] bio: create slab <bio-0> at 0 [ 0.333007] ACPI: Added _OSI(Module Device) [ 0.333008] ACPI: Added _OSI(Processor Device) [ 0.333010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333786] ACPI: EC: Look up EC in DSDT [ 0.334658] ACPI: Executed 3 blocks of module-level executable AML code [ 0.337829] ACPI: Interpreter enabled [ 0.337836] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130117/hwxface-568) [ 0.337840] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130117/hwxface-568) [ 0.337858] ACPI: (supports S0 S3 S4 S5) [ 0.337860] ACPI: Using IOAPIC for interrupt routing [ 0.337991] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.338027] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.351694] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.356496] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356733] PCI host bridge to bus 0000:00 [ 0.356736] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.356738] pci_bus 0000:00: root bus resource [io 0x0000-0x03af] [ 0.356740] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7] [ 0.356741] pci_bus 0000:00: root bus resource [io 0x03b0-0x03df] [ 0.356743] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.356744] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.356746] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff] [ 0.356747] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xffffffff] [ 0.356757] pci 0000:00:00.0: [1002:5a14] type 00 class 0x060000 [ 0.356832] pci 0000:00:00.2: [1002:5a23] type 00 class 0x080600 [ 0.356911] pci 0000:00:02.0: [1002:5a16] type 01 class 0x060400 [ 0.356941] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 0.356960] pci 0000:00:02.0: System wakeup disabled by ACPI [ 0.357004] pci 0000:00:04.0: [1002:5a18] type 01 class 0x060400 [ 0.357033] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold [ 0.357052] pci 0000:00:04.0: System wakeup disabled by ACPI [ 0.357080] pci 0000:00:05.0: [1002:5a19] type 01 class 0x060400 [ 0.357109] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold [ 0.357128] pci 0000:00:05.0: System wakeup disabled by ACPI [ 0.357155] pci 0000:00:06.0: [1002:5a1a] type 01 class 0x060400 [ 0.357184] pci 0000:00:06.0: PME# supported from D0 D3hot D3cold [ 0.357204] pci 0000:00:06.0: System wakeup disabled by ACPI [ 0.357234] pci 0000:00:0a.0: [1002:5a1d] type 01 class 0x060400 [ 0.357263] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold [ 0.357285] pci 0000:00:0a.0: System wakeup disabled by ACPI [ 0.357315] pci 0000:00:0b.0: [1002:5a1f] type 01 class 0x060400 [ 0.357344] pci 0000:00:0b.0: PME# supported from D0 D3hot D3cold [ 0.357369] pci 0000:00:0b.0: System wakeup disabled by ACPI [ 0.357406] pci 0000:00:11.0: [1002:4391] type 00 class 0x010601 [ 0.357420] pci 0000:00:11.0: reg 10: [io 0xf040-0xf047] [ 0.357428] pci 0000:00:11.0: reg 14: [io 0xf030-0xf033] [ 0.357435] pci 0000:00:11.0: reg 18: [io 0xf020-0xf027] [ 0.357442] pci 0000:00:11.0: reg 1c: [io 0xf010-0xf013] [ 0.357449] pci 0000:00:11.0: reg 20: [io 0xf000-0xf00f] [ 0.357456] pci 0000:00:11.0: reg 24: [mem 0xfe70b000-0xfe70b3ff] [ 0.357545] pci 0000:00:12.0: [1002:4397] type 00 class 0x0c0310 [ 0.357555] pci 0000:00:12.0: reg 10: [mem 0xfe70a000-0xfe70afff] [ 0.357621] pci 0000:00:12.0: System wakeup disabled by ACPI [ 0.357653] pci 0000:00:12.2: [1002:4396] type 00 class 0x0c0320 [ 0.357667] pci 0000:00:12.2: reg 10: [mem 0xfe709000-0xfe7090ff] [ 0.357728] pci 0000:00:12.2: supports D1 D2 [ 0.357729] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot [ 0.357761] pci 0000:00:12.2: System wakeup disabled by ACPI [ 0.357793] pci 0000:00:13.0: [1002:4397] type 00 class 0x0c0310 [ 0.357803] pci 0000:00:13.0: reg 10: [mem 0xfe708000-0xfe708fff] [ 0.357871] pci 0000:00:13.0: System wakeup disabled by ACPI [ 0.357917] pci 0000:00:13.2: [1002:4396] type 00 class 0x0c0320 [ 0.357931] pci 0000:00:13.2: reg 10: [mem 0xfe707000-0xfe7070ff] [ 0.357991] pci 0000:00:13.2: supports D1 D2 [ 0.357993] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot [ 0.358027] pci 0000:00:13.2: System wakeup disabled by ACPI [ 0.358059] pci 0000:00:14.0: [1002:4385] type 00 class 0x0c0500 [ 0.358149] pci 0000:00:14.2: [1002:4383] type 00 class 0x040300 [ 0.358165] pci 0000:00:14.2: reg 10: [mem 0xfe700000-0xfe703fff 64bit] [ 0.358236] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold [ 0.358262] pci 0000:00:14.2: System wakeup disabled by ACPI [ 0.358292] pci 0000:00:14.3: [1002:439d] type 00 class 0x060100 [ 0.358383] pci 0000:00:14.4: [1002:4384] type 01 class 0x060401 [ 0.358432] pci 0000:00:14.4: System wakeup disabled by ACPI [ 0.358458] pci 0000:00:14.5: [1002:4399] type 00 class 0x0c0310 [ 0.358468] pci 0000:00:14.5: reg 10: [mem 0xfe706000-0xfe706fff] [ 0.358541] pci 0000:00:14.5: System wakeup disabled by ACPI [ 0.358574] pci 0000:00:15.0: [1002:43a0] type 01 class 0x060400 [ 0.358628] pci 0000:00:15.0: supports D1 D2 [ 0.358670] pci 0000:00:15.0: System wakeup disabled by ACPI [ 0.358703] pci 0000:00:16.0: [1002:4397] type 00 class 0x0c0310 [ 0.358713] pci 0000:00:16.0: reg 10: [mem 0xfe705000-0xfe705fff] [ 0.358783] pci 0000:00:16.0: System wakeup disabled by ACPI [ 0.358814] pci 0000:00:16.2: [1002:4396] type 00 class 0x0c0320 [ 0.358828] pci 0000:00:16.2: reg 10: [mem 0xfe704000-0xfe7040ff] [ 0.358894] pci 0000:00:16.2: supports D1 D2 [ 0.358896] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot [ 0.358932] pci 0000:00:16.2: System wakeup disabled by ACPI [ 0.358963] pci 0000:00:18.0: [1022:1600] type 00 class 0x060000 [ 0.359026] pci 0000:00:18.1: [1022:1601] type 00 class 0x060000 [ 0.359086] pci 0000:00:18.2: [1022:1602] type 00 class 0x060000 [ 0.359146] pci 0000:00:18.3: [1022:1603] type 00 class 0x060000 [ 0.359209] pci 0000:00:18.4: [1022:1604] type 00 class 0x060000 [ 0.359268] pci 0000:00:18.5: [1022:1605] type 00 class 0x060000 [ 0.359372] pci 0000:01:00.0: [10de:0fc6] type 00 class 0x030000 [ 0.359381] pci 0000:01:00.0: reg 10: [mem 0xfd000000-0xfdffffff] [ 0.359391] pci 0000:01:00.0: reg 14: [mem 0xa0000000-0xafffffff 64bit pref] [ 0.359401] pci 0000:01:00.0: reg 1c: [mem 0xb0000000-0xb1ffffff 64bit pref] [ 0.359408] pci 0000:01:00.0: reg 24: [io 0xe000-0xe07f] [ 0.359414] pci 0000:01:00.0: reg 30: [mem 0xfe000000-0xfe07ffff pref] [ 0.359516] pci 0000:01:00.1: [10de:0e1b] type 00 class 0x040300 [ 0.359525] pci 0000:01:00.1: reg 10: [mem 0xfe080000-0xfe083fff] [ 0.360912] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.360916] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.360918] pci 0000:00:02.0: bridge window [mem 0xfd000000-0xfe0fffff] [ 0.360921] pci 0000:00:02.0: bridge window [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.360964] pci 0000:02:00.0: [1b4b:91a0] type 00 class 0x01018f [ 0.360973] pci 0000:02:00.0: reg 10: [io 0xd090-0xd097] [ 0.360979] pci 0000:02:00.0: reg 14: [io 0xd080-0xd083] [ 0.360985] pci 0000:02:00.0: reg 18: [io 0xd070-0xd077] [ 0.360991] pci 0000:02:00.0: reg 1c: [io 0xd060-0xd063] [ 0.360997] pci 0000:02:00.0: reg 20: [io 0xd050-0xd05f] [ 0.361003] pci 0000:02:00.0: reg 24: [mem 0xfe615000-0xfe6157ff] [ 0.361010] pci 0000:02:00.0: reg 30: [mem 0xfe600000-0xfe60ffff pref] [ 0.361037] pci 0000:02:00.0: PME# supported from D3hot [ 0.362910] pci 0000:00:04.0: PCI bridge to [bus 02] [ 0.362914] pci 0000:00:04.0: bridge window [io 0xd000-0xdfff] [ 0.362916] pci 0000:00:04.0: bridge window [mem 0xfe600000-0xfe6fffff] [ 0.362961] pci 0000:03:00.0: [1b6f:7023] type 00 class 0x0c0330 [ 0.362975] pci 0000:03:00.0: reg 10: [mem 0xfe500000-0xfe507fff 64bit] [ 0.363039] pci 0000:03:00.0: supports D1 D2 [ 0.363041] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.364915] pci 0000:00:05.0: PCI bridge to [bus 03] [ 0.364919] pci 0000:00:05.0: bridge window [mem 0xfe500000-0xfe5fffff] [ 0.364964] pci 0000:04:00.0: [1b6f:7023] type 00 class 0x0c0330 [ 0.364979] pci 0000:04:00.0: reg 10: [mem 0xfe400000-0xfe407fff 64bit] [ 0.365043] pci 0000:04:00.0: supports D1 D2 [ 0.365045] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.366918] pci 0000:00:06.0: PCI bridge to [bus 04] [ 0.366922] pci 0000:00:06.0: bridge window [mem 0xfe400000-0xfe4fffff] [ 0.366974] pci 0000:05:00.0: [14e4:16b1] type 00 class 0x020000 [ 0.366991] pci 0000:05:00.0: reg 10: [mem 0xd0010000-0xd001ffff 64bit pref] [ 0.367005] pci 0000:05:00.0: reg 18: [mem 0xd0000000-0xd000ffff 64bit pref] [ 0.367029] pci 0000:05:00.0: reg 30: [mem 0xfe300000-0xfe3007ff pref] [ 0.367076] pci 0000:05:00.0: PME# supported from D0 D3hot D3cold [ 0.368923] pci 0000:00:0a.0: PCI bridge to [bus 05] [ 0.368927] pci 0000:00:0a.0: bridge window [mem 0xfe300000-0xfe3fffff] [ 0.368930] pci 0000:00:0a.0: bridge window [mem 0xd0000000-0xd00fffff 64bit pref] [ 0.368974] pci 0000:06:00.0: [1002:68d8] type 00 class 0x030000 [ 0.368987] pci 0000:06:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.368997] pci 0000:06:00.0: reg 18: [mem 0xfe220000-0xfe23ffff 64bit] [ 0.369004] pci 0000:06:00.0: reg 20: [io 0xc000-0xc0ff] [ 0.369017] pci 0000:06:00.0: reg 30: [mem 0xfe200000-0xfe21ffff pref] [ 0.369049] pci 0000:06:00.0: supports D1 D2 [ 0.369092] pci 0000:06:00.1: [1002:aa60] type 00 class 0x040300 [ 0.369105] pci 0000:06:00.1: reg 10: [mem 0xfe240000-0xfe243fff 64bit] [ 0.369164] pci 0000:06:00.1: supports D1 D2 [ 0.370925] pci 0000:00:0b.0: PCI bridge to [bus 06] [ 0.370928] pci 0000:00:0b.0: bridge window [io 0xc000-0xcfff] [ 0.370930] pci 0000:00:0b.0: bridge window [mem 0xfe200000-0xfe2fffff] [ 0.370933] pci 0000:00:0b.0: bridge window [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.370995] pci 0000:00:14.4: PCI bridge to [bus 07] (subtractive decode) [ 0.371002] pci 0000:00:14.4: bridge window [io 0x0000-0x03af] (subtractive decode) [ 0.371004] pci 0000:00:14.4: bridge window [io 0x03e0-0x0cf7] (subtractive decode) [ 0.371005] pci 0000:00:14.4: bridge window [io 0x03b0-0x03df] (subtractive decode) [ 0.371007] pci 0000:00:14.4: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.371008] pci 0000:00:14.4: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.371009] pci 0000:00:14.4: bridge window [mem 0x000c0000-0x000dffff] (subtractive decode) [ 0.371011] pci 0000:00:14.4: bridge window [mem 0xa0000000-0xffffffff] (subtractive decode) [ 0.371077] pci 0000:08:00.0: [10ec:8168] type 00 class 0x020000 [ 0.371094] pci 0000:08:00.0: reg 10: [io 0xb000-0xb0ff] [ 0.371123] pci 0000:08:00.0: reg 18: [mem 0xfe100000-0xfe100fff 64bit] [ 0.371156] pci 0000:08:00.0: reg 30: [mem 0xfffe0000-0xffffffff pref] [ 0.371229] pci 0000:08:00.0: supports D1 D2 [ 0.371231] pci 0000:08:00.0: PME# supported from D1 D2 D3hot D3cold [ 0.371285] pci 0000:08:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' [ 0.371293] pci 0000:00:15.0: PCI bridge to [bus 08] [ 0.371298] pci 0000:00:15.0: bridge window [io 0xb000-0xbfff] [ 0.371300] pci 0000:00:15.0: bridge window [mem 0xfe100000-0xfe1fffff] [ 0.371321] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM [ 0.371323] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08) [ 0.371833] ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11 14 15) *0 [ 0.371891] ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11 14 15) *0 [ 0.371958] ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11 14 15) *0 [ 0.372018] ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11 14 15) *0 [ 0.372067] ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11 14 15) *0 [ 0.372106] ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11 14 15) *0 [ 0.372144] ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11 14 15) *0 [ 0.372182] ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11 14 15) *0 [ 0.372282] acpi root: \_SB_.PCI0 notify handler is installed [ 0.372311] Found 1 acpi root devices [ 0.372397] ACPI: No dock devices found. [ 0.372458] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.372461] vgaarb: device added: PCI:0000:06:00.0,decodes=io+mem,owns=none,locks=none [ 0.372463] vgaarb: loaded [ 0.372463] vgaarb: bridge control possible 0000:06:00.0 [ 0.372464] vgaarb: bridge control possible 0000:01:00.0 [ 0.372498] PCI: Using ACPI for IRQ routing [ 0.378656] PCI: pci_cache_line_size set to 64 bytes [ 0.378718] e820: reserve RAM buffer [mem 0x0009d000-0x0009ffff] [ 0.378720] e820: reserve RAM buffer [mem 0x9df05000-0x9fffffff] [ 0.378721] e820: reserve RAM buffer [mem 0x9ee2a000-0x9fffffff] [ 0.378723] e820: reserve RAM buffer [mem 0x9f453000-0x9fffffff] [ 0.378724] e820: reserve RAM buffer [mem 0x9f800000-0x9fffffff] [ 0.378725] e820: reserve RAM buffer [mem 0x25f000000-0x25fffffff] [ 0.378797] NetLabel: Initializing [ 0.378799] NetLabel: domain hash size = 128 [ 0.378799] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.378808] NetLabel: unlabeled traffic allowed by default [ 0.378836] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.378839] hpet0: 3 comparators, 32-bit 14.318180 MHz counter [ 0.380868] Switching to clocksource hpet [ 0.385544] pnp: PnP ACPI init [ 0.385553] ACPI: bus type PNP registered [ 0.385630] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved [ 0.385633] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.385730] system 00:01: [mem 0xfec20000-0xfec200ff] could not be reserved [ 0.385733] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.385799] system 00:02: [mem 0xfeb20000-0xfeb23fff] could not be reserved [ 0.385801] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386080] system 00:03: [io 0x040b] has been reserved [ 0.386082] system 00:03: [io 0x04d6] has been reserved [ 0.386084] system 00:03: [io 0x0c00-0x0c01] has been reserved [ 0.386086] system 00:03: [io 0x0c14] has been reserved [ 0.386087] system 00:03: [io 0x0c50-0x0c51] has been reserved [ 0.386089] system 00:03: [io 0x0c52] has been reserved [ 0.386090] system 00:03: [io 0x0c6c] has been reserved [ 0.386092] system 00:03: [io 0x0c6f] has been reserved [ 0.386093] system 00:03: [io 0x0cd0-0x0cd1] has been reserved [ 0.386095] system 00:03: [io 0x0cd2-0x0cd3] has been reserved [ 0.386097] system 00:03: [io 0x0cd4-0x0cd5] has been reserved [ 0.386098] system 00:03: [io 0x0cd6-0x0cd7] has been reserved [ 0.386100] system 00:03: [io 0x0cd8-0x0cdf] has been reserved [ 0.386101] system 00:03: [io 0x0800-0x089f] has been reserved [ 0.386103] system 00:03: [io 0x0b20-0x0b3f] has been reserved [ 0.386105] system 00:03: [io 0x0900-0x090f] has been reserved [ 0.386106] system 00:03: [io 0x0910-0x091f] has been reserved [ 0.386108] system 00:03: [io 0xfe00-0xfefe] has been reserved [ 0.386110] system 00:03: [mem 0xfec00000-0xfec00fff] could not be reserved [ 0.386112] system 00:03: [mem 0xfee00000-0xfee00fff] has been reserved [ 0.386114] system 00:03: [mem 0xfed80000-0xfed8ffff] has been reserved [ 0.386116] system 00:03: [mem 0xfed61000-0xfed70fff] has been reserved [ 0.386117] system 00:03: [mem 0xfec10000-0xfec10fff] has been reserved [ 0.386119] system 00:03: [mem 0xfed00000-0xfed00fff] has been reserved [ 0.386121] system 00:03: [mem 0xffc00000-0xffffffff] has been reserved [ 0.386123] system 00:03: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386199] system 00:04: [io 0x0290-0x029f] has been reserved [ 0.386202] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386411] pnp 00:05: [dma 4] [ 0.386429] pnp 00:05: Plug and Play ACPI device, IDs PNP0200 (active) [ 0.386458] pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.386478] pnp 00:07: Plug and Play ACPI device, IDs PNP0800 (active) [ 0.386527] system 00:08: [io 0x04d0-0x04d1] has been reserved [ 0.386530] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386560] pnp 00:09: Plug and Play ACPI device, IDs PNP0c04 (active) [ 0.386592] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.386819] pnp 00:0b: Plug and Play ACPI device, IDs PNP0103 (active) [ 0.386822] pnp: PnP ACPI: found 12 devices [ 0.386823] ACPI: bus type PNP unregistered [ 0.393157] pci 0000:08:00.0: no compatible bridge window for [mem 0xfffe0000-0xffffffff pref] [ 0.393190] pci 0000:00:15.0: BAR 15: assigned [mem 0xb2000000-0xb20fffff pref] [ 0.393192] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.393194] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.393197] pci 0000:00:02.0: bridge window [mem 0xfd000000-0xfe0fffff] [ 0.393199] pci 0000:00:02.0: bridge window [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.393202] pci 0000:00:04.0: PCI bridge to [bus 02] [ 0.393204] pci 0000:00:04.0: bridge window [io 0xd000-0xdfff] [ 0.393206] pci 0000:00:04.0: bridge window [mem 0xfe600000-0xfe6fffff] [ 0.393210] pci 0000:00:05.0: PCI bridge to [bus 03] [ 0.393212] pci 0000:00:05.0: bridge window [mem 0xfe500000-0xfe5fffff] [ 0.393215] pci 0000:00:06.0: PCI bridge to [bus 04] [ 0.393218] pci 0000:00:06.0: bridge window [mem 0xfe400000-0xfe4fffff] [ 0.393221] pci 0000:00:0a.0: PCI bridge to [bus 05] [ 0.393223] pci 0000:00:0a.0: bridge window [mem 0xfe300000-0xfe3fffff] [ 0.393225] pci 0000:00:0a.0: bridge window [mem 0xd0000000-0xd00fffff 64bit pref] [ 0.393228] pci 0000:00:0b.0: PCI bridge to [bus 06] [ 0.393230] pci 0000:00:0b.0: bridge window [io 0xc000-0xcfff] [ 0.393232] pci 0000:00:0b.0: bridge window [mem 0xfe200000-0xfe2fffff] [ 0.393234] pci 0000:00:0b.0: bridge window [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.393237] pci 0000:00:14.4: PCI bridge to [bus 07] [ 0.393246] pci 0000:08:00.0: BAR 6: assigned [mem 0xb2000000-0xb201ffff pref] [ 0.393248] pci 0000:00:15.0: PCI bridge to [bus 08] [ 0.393250] pci 0000:00:15.0: bridge window [io 0xb000-0xbfff] [ 0.393253] pci 0000:00:15.0: bridge window [mem 0xfe100000-0xfe1fffff] [ 0.393255] pci 0000:00:15.0: bridge window [mem 0xb2000000-0xb20fffff pref] [ 0.393627] pci_bus 0000:00: resource 4 [io 0x0000-0x03af] [ 0.393629] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7] [ 0.393631] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df] [ 0.393632] pci_bus 0000:00: resource 7 [io 0x0d00-0xffff] [ 0.393634] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff] [ 0.393635] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff] [ 0.393637] pci_bus 0000:00: resource 10 [mem 0xa0000000-0xffffffff] [ 0.393638] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.393640] pci_bus 0000:01: resource 1 [mem 0xfd000000-0xfe0fffff] [ 0.393641] pci_bus 0000:01: resource 2 [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.393643] pci_bus 0000:02: resource 0 [io 0xd000-0xdfff] [ 0.393644] pci_bus 0000:02: resource 1 [mem 0xfe600000-0xfe6fffff] [ 0.393646] pci_bus 0000:03: resource 1 [mem 0xfe500000-0xfe5fffff] [ 0.393647] pci_bus 0000:04: resource 1 [mem 0xfe400000-0xfe4fffff] [ 0.393648] pci_bus 0000:05: resource 1 [mem 0xfe300000-0xfe3fffff] [ 0.393650] pci_bus 0000:05: resource 2 [mem 0xd0000000-0xd00fffff 64bit pref] [ 0.393651] pci_bus 0000:06: resource 0 [io 0xc000-0xcfff] [ 0.393653] pci_bus 0000:06: resource 1 [mem 0xfe200000-0xfe2fffff] [ 0.393654] pci_bus 0000:06: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.393656] pci_bus 0000:07: resource 4 [io 0x0000-0x03af] [ 0.393657] pci_bus 0000:07: resource 5 [io 0x03e0-0x0cf7] [ 0.393658] pci_bus 0000:07: resource 6 [io 0x03b0-0x03df] [ 0.393660] pci_bus 0000:07: resource 7 [io 0x0d00-0xffff] [ 0.393661] pci_bus 0000:07: resource 8 [mem 0x000a0000-0x000bffff] [ 0.393662] pci_bus 0000:07: resource 9 [mem 0x000c0000-0x000dffff] [ 0.393664] pci_bus 0000:07: resource 10 [mem 0xa0000000-0xffffffff] [ 0.393665] pci_bus 0000:08: resource 0 [io 0xb000-0xbfff] [ 0.393667] pci_bus 0000:08: resource 1 [mem 0xfe100000-0xfe1fffff] [ 0.393668] pci_bus 0000:08: resource 2 [mem 0xb2000000-0xb20fffff pref] [ 0.393691] NET: Registered protocol family 2 [ 0.393879] TCP established hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.394117] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.394334] TCP: Hash tables configured (established 65536 bind 65536) [ 0.394374] TCP: reno registered [ 0.394384] UDP hash table entries: 4096 (order: 5, 131072 bytes) [ 0.394429] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes) [ 0.394512] NET: Registered protocol family 1 [ 0.650568] pci 0000:01:00.0: Boot video device [ 0.650681] PCI: CLS 64 bytes, default 64 [ 0.650716] Unpacking initramfs... [ 1.139954] Freeing initrd memory: 5880k freed [ 1.141944] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.141945] AMD-Vi: Interrupt remapping enabled [ 1.141966] pci 0000:00:00.2: irq 72 for MSI/MSI-X [ 1.153516] AMD-Vi: Lazy IO/TLB flushing enabled [ 1.228038] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.228040] software IO TLB [mem 0x99f05000-0x9df05000] (64MB) mapped at [ffff880099f05000-ffff88009df04fff] [ 1.228355] LVT offset 0 assigned for vector 0x400 [ 1.228391] perf: AMD IBS detected (0x000000ff) [ 1.228419] Scanning for low memory corruption every 60 seconds [ 1.228591] audit: initializing netlink socket (disabled) [ 1.228610] type=2000 audit(1369582568.031:1): initialized [ 1.237950] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.239493] VFS: Disk quotas dquot_6.5.2 [ 1.239528] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.239669] msgmni has been set to 15756 [ 1.239941] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 1.239972] io scheduler noop registered [ 1.239975] io scheduler deadline registered [ 1.239980] io scheduler cfq registered (default) [ 1.240206] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.240219] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.240300] GHES: HEST is not enabled! [ 1.240341] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 1.240723] Linux agpgart interface v0.103 [ 1.240807] VFIO - User Level meta-driver version: 0.3 [ 1.241050] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 1.243505] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.243528] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.243677] mousedev: PS/2 mouse device common for all mice [ 1.243724] rtc_cmos 00:06: RTC can wake from S4 [ 1.243815] rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0 [ 1.243834] rtc_cmos 00:06: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 1.243841] cpuidle: using governor ladder [ 1.243842] cpuidle: using governor menu [ 1.243844] EFI Variables Facility v0.08 2004-May-17 [ 1.243888] drop_monitor: Initializing network drop monitor service [ 1.243937] TCP: cubic registered [ 1.244017] NET: Registered protocol family 10 [ 1.244133] NET: Registered protocol family 17 [ 1.244140] Key type dns_resolver registered [ 1.244656] PM: Hibernation image not present or could not be loaded. [ 1.244664] registered taskstats version 1 [ 1.245341] Magic number: 13:120:640 [ 1.245398] rtc_cmos 00:06: setting system clock to 2013-05-26 15:36:08 UTC (1369582568) [ 1.245984] Freeing unused kernel memory: 1092k freed [ 1.246231] Write protecting the kernel read-only data: 8192k [ 1.248312] Freeing unused kernel memory: 1004k freed [ 1.249495] Freeing unused kernel memory: 420k freed [ 1.254966] systemd-udevd[80]: starting version 204 [ 1.257148] [drm] Initialized drm 1.1.0 20060810 [ 1.263456] nvidia: module license 'NVIDIA' taints kernel. [ 1.263459] Disabling lock debugging due to kernel taint [ 1.274456] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.274458] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:06:00.0 [ 1.274672] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0 [ 1.274680] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 319.17 Thu Apr 25 22:45:49 PDT 2013 [ 1.292640] SCSI subsystem initialized [ 1.294690] ACPI: bus type ATA registered [ 1.294771] libata version 3.00 loaded. [ 1.295287] ACPI: bus type USB registered [ 1.295398] usbcore: registered new interface driver usbfs [ 1.295454] usbcore: registered new interface driver hub [ 1.295496] usbcore: registered new device driver usb [ 1.296456] ahci 0000:00:11.0: version 3.0 [ 1.296663] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.296710] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 1.296713] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 1.296855] ehci-pci: EHCI PCI platform driver [ 1.297028] xhci_hcd 0000:03:00.0: xHCI Host Controller [ 1.297037] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 1 [ 1.297135] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.297172] xhci_hcd 0000:03:00.0: irq 73 for MSI/MSI-X [ 1.297491] xHCI xhci_add_endpoint called for root hub [ 1.297493] xHCI xhci_check_bandwidth called for root hub [ 1.297558] hub 1-0:1.0: USB hub found [ 1.297566] hub 1-0:1.0: 2 ports detected [ 1.297716] scsi0 : pata_marvell [ 1.297970] xhci_hcd 0000:03:00.0: xHCI Host Controller [ 1.297974] xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 2 [ 1.298501] scsi1 : pata_marvell [ 1.298667] xHCI xhci_add_endpoint called for root hub [ 1.298669] xHCI xhci_check_bandwidth called for root hub [ 1.298711] ata1: PATA max UDMA/100 cmd 0xd090 ctl 0xd080 bmdma 0xd050 irq 44 [ 1.298713] ata2: PATA max UDMA/133 cmd 0xd070 ctl 0xd060 bmdma 0xd058 irq 44 [ 1.298786] hub 2-0:1.0: USB hub found [ 1.298793] hub 2-0:1.0: 2 ports detected [ 1.299597] scsi2 : ahci [ 1.299734] scsi3 : ahci [ 1.299852] scsi4 : ahci [ 1.299993] scsi5 : ahci [ 1.300114] scsi6 : ahci [ 1.300255] scsi7 : ahci [ 1.300342] ata3: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b100 irq 19 [ 1.300345] ata4: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b180 irq 19 [ 1.300347] ata5: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b200 irq 19 [ 1.300349] ata6: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b280 irq 19 [ 1.300351] ata7: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b300 irq 19 [ 1.300353] ata8: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b380 irq 19 [ 1.300456] ehci-pci 0000:00:12.2: EHCI Host Controller [ 1.300466] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 3 [ 1.300482] QUIRK: Enable AMD PLL fix [ 1.300483] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 1.300499] ehci-pci 0000:00:12.2: debug port 1 [ 1.300562] ehci-pci 0000:00:12.2: irq 17, io mem 0xfe709000 [ 1.300685] xhci_hcd 0000:04:00.0: xHCI Host Controller [ 1.300693] xhci_hcd 0000:04:00.0: new USB bus registered, assigned bus number 4 [ 1.300818] xhci_hcd 0000:04:00.0: irq 74 for MSI/MSI-X [ 1.301029] xHCI xhci_add_endpoint called for root hub [ 1.301030] xHCI xhci_check_bandwidth called for root hub [ 1.301066] hub 4-0:1.0: USB hub found [ 1.301071] hub 4-0:1.0: 2 ports detected [ 1.301226] xhci_hcd 0000:04:00.0: xHCI Host Controller [ 1.301230] xhci_hcd 0000:04:00.0: new USB bus registered, assigned bus number 5 [ 1.301351] xHCI xhci_add_endpoint called for root hub [ 1.301353] xHCI xhci_check_bandwidth called for root hub [ 1.301414] hub 5-0:1.0: USB hub found [ 1.301419] hub 5-0:1.0: 2 ports detected [ 1.306554] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00 [ 1.306685] hub 3-0:1.0: USB hub found [ 1.306688] hub 3-0:1.0: 5 ports detected [ 1.306946] ehci-pci 0000:00:13.2: EHCI Host Controller [ 1.306958] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 6 [ 1.306961] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 1.306971] ehci-pci 0000:00:13.2: debug port 1 [ 1.306993] ehci-pci 0000:00:13.2: irq 17, io mem 0xfe707000 [ 1.312551] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00 [ 1.312661] hub 6-0:1.0: USB hub found [ 1.312664] hub 6-0:1.0: 5 ports detected [ 1.312828] ehci-pci 0000:00:16.2: EHCI Host Controller [ 1.312832] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 7 [ 1.312835] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 1.312844] ehci-pci 0000:00:16.2: debug port 1 [ 1.312865] ehci-pci 0000:00:16.2: irq 17, io mem 0xfe704000 [ 1.318563] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00 [ 1.318664] hub 7-0:1.0: USB hub found [ 1.318667] hub 7-0:1.0: 4 ports detected [ 1.318841] ohci_hcd 0000:00:12.0: OHCI Host Controller [ 1.318852] ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 8 [ 1.318879] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfe70a000 [ 1.373777] hub 8-0:1.0: USB hub found [ 1.373785] hub 8-0:1.0: 5 ports detected [ 1.373948] ohci_hcd 0000:00:13.0: OHCI Host Controller [ 1.373952] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 9 [ 1.373964] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfe708000 [ 1.428882] hub 9-0:1.0: USB hub found [ 1.428887] hub 9-0:1.0: 5 ports detected [ 1.429038] ohci_hcd 0000:00:14.5: OHCI Host Controller [ 1.429043] ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 10 [ 1.429056] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfe706000 [ 1.483961] hub 10-0:1.0: USB hub found [ 1.483967] hub 10-0:1.0: 2 ports detected [ 1.484113] ohci_hcd 0000:00:16.0: OHCI Host Controller [ 1.484117] ohci_hcd 0000:00:16.0: new USB bus registered, assigned bus number 11 [ 1.484128] ohci_hcd 0000:00:16.0: irq 18, io mem 0xfe705000 [ 1.504185] ata1.01: ATA-7: ST3320820AS, 3.AAC, max UDMA/133 [ 1.504188] ata1.01: 625142448 sectors, multi 0: LBA48 NCQ (depth 0/32) [ 1.539050] hub 11-0:1.0: USB hub found [ 1.539055] hub 11-0:1.0: 4 ports detected [ 1.579276] ata1.01: configured for UDMA/100 [ 1.579412] scsi 0:0:1:0: Direct-Access ATA ST3320820AS 3.AA PQ: 0 ANSI: 5 [ 1.760363] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.760386] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.760409] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.760431] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.760452] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.761129] ata3.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133 [ 1.761131] ata3.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.761342] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 1.761968] ata3.00: configured for UDMA/133 [ 1.762044] scsi 2:0:0:0: Direct-Access ATA WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5 [ 1.764131] ata8.00: ATAPI: HL-DT-ST DVDRAM GH20NS10, EL00, max UDMA/100 [ 1.766537] ata5.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133 [ 1.766540] ata5.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.766866] ata4.00: ATA-7: SAMSUNG HD103SI, 1AG01118, max UDMA7 [ 1.766869] ata4.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.767196] ata6.00: ATA-8: WDC WD20EARX-00PASB0, 51.0AB51, max UDMA/133 [ 1.767198] ata6.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.768130] ata8.00: configured for UDMA/100 [ 1.771688] ata7.00: ATA-8: ADATA SP900, 5.0.2a, max UDMA/133 [ 1.771690] ata7.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.772685] ata5.00: configured for UDMA/133 [ 1.773361] ata4.00: configured for UDMA/133 [ 1.773455] scsi 3:0:0:0: Direct-Access ATA SAMSUNG HD103SI 1AG0 PQ: 0 ANSI: 5 [ 1.773632] scsi 4:0:0:0: Direct-Access ATA SAMSUNG HD204UI 1AQ1 PQ: 0 ANSI: 5 [ 1.774193] ata6.00: configured for UDMA/133 [ 1.774274] scsi 5:0:0:0: Direct-Access ATA WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5 [ 1.782015] ata7.00: configured for UDMA/133 [ 1.782103] scsi 6:0:0:0: Direct-Access ATA ADATA SP900 5.0. PQ: 0 ANSI: 5 [ 1.785353] scsi 7:0:0:0: CD-ROM HL-DT-ST DVDRAM GH20NS10 EL00 PQ: 0 ANSI: 5 [ 1.789538] sd 0:0:1:0: [sda] 625142448 512-byte logical blocks: (320 GB/298 GiB) [ 1.789610] sd 0:0:1:0: [sda] Write Protect is off [ 1.789613] sd 0:0:1:0: [sda] Mode Sense: 00 3a 00 00 [ 1.789659] sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789723] sd 2:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 1.789789] sd 3:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 1.789815] sd 2:0:0:0: [sdb] Write Protect is off [ 1.789818] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 1.789861] sd 4:0:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [ 1.789864] sd 3:0:0:0: [sdc] Write Protect is off [ 1.789866] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 1.789868] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789893] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789902] sd 5:0:0:0: [sde] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [ 1.789905] sd 5:0:0:0: [sde] 4096-byte physical blocks [ 1.789923] sd 4:0:0:0: [sdd] Write Protect is off [ 1.789926] sd 4:0:0:0: [sdd] Mode Sense: 00 3a 00 00 [ 1.789928] sd 6:0:0:0: [sdf] 250069680 512-byte logical blocks: (128 GB/119 GiB) [ 1.789954] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789972] sd 5:0:0:0: [sde] Write Protect is off [ 1.789974] sd 6:0:0:0: [sdf] Write Protect is off [ 1.789975] sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00 [ 1.789979] sd 6:0:0:0: [sdf] Mode Sense: 00 3a 00 00 [ 1.789991] sd 6:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.789999] sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.791154] sdf: sdf1 sdf2 sdf3 [ 1.791424] sd 6:0:0:0: [sdf] Attached SCSI disk [ 1.793671] sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray [ 1.793673] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 1.793847] sr 7:0:0:0: Attached scsi CD-ROM sr0 [ 1.807557] sdc: sdc1 sdc2 sdc3 sdc4 [ 1.807974] sd 3:0:0:0: [sdc] Attached SCSI disk [ 1.810863] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x0000000000001000 flags=0x0050] [ 1.824834] sdd: sdd1 sdd2 [ 1.825116] sd 4:0:0:0: [sdd] Attached SCSI disk [ 1.836967] sdb: sdb1 [ 1.837233] sd 2:0:0:0: [sdb] Attached SCSI disk [ 1.837476] raid6: sse2x1 6937 MB/s [ 1.854501] raid6: sse2x2 11195 MB/s [ 1.871543] raid6: sse2x4 13304 MB/s [ 1.871544] raid6: using algorithm sse2x4 (13304 MB/s) [ 1.871545] raid6: using ssse3x2 recovery algorithm [ 1.871754] xor: automatically using best checksumming function: [ 1.881535] avx : 4852.000 MB/sec [ 1.884902] Btrfs loaded [ 1.886377] device fsid 33e09c2d-5913-4ece-bbc2-069d55f1fecc devid 1 transid 69987 /dev/sdf3 [ 1.986737] usb 8-1: new full-speed USB device number 2 using ohci_hcd [ 2.138080] hub 8-1:1.0: USB hub found [ 2.140045] hub 8-1:1.0: 4 ports detected [ 2.230163] tsc: Refined TSC clocksource calibration: 3991.443 MHz [ 2.230171] Switching to clocksource tsc [ 2.252034] sde: sde1 [ 2.252274] sd 5:0:0:0: [sde] Attached SCSI disk [ 2.432558] usb 8-1.1: new low-speed USB device number 3 using ohci_hcd [ 2.546585] hidraw: raw HID events driver (C) Jiri Kosina [ 2.575860] usbcore: registered new interface driver usbhid [ 2.575861] usbhid: USB HID core driver [ 2.576268] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.1/8-1.1:1.0/input/input0 [ 2.576310] hid-generic 0003:04D9:1605.0001: input,hidraw0: USB HID v1.10 Keyboard [ USB Keyboard] on usb-0000:00:12.0-1.1/input0 [ 2.581867] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.1/8-1.1:1.1/input/input1 [ 2.581934] hid-generic 0003:04D9:1605.0002: input,hidraw1: USB HID v1.10 Device [ USB Keyboard] on usb-0000:00:12.0-1.1/input1 [ 2.614877] usb 8-1.2: new full-speed USB device number 4 using ohci_hcd [ 2.715202] input: USB USB Mouse as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.2/8-1.2:1.0/input/input2 [ 2.715264] hid-generic 0003:1D57:3108.0003: input,hidraw2: USB HID v1.11 Mouse [USB USB Mouse] on usb-0000:00:12.0-1.2/input0 [ 2.783174] usb 8-1.3: new low-speed USB device number 5 using ohci_hcd [ 2.896435] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:12.0/usb8/8-1/8-1.3/8-1.3:1.0/input/input3 [ 2.896492] hid-generic 0003:10D5:55A4.0004: input,hidraw3: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:12.0-1.3/input0 [ 2.898397] usbhid 8-1.3:1.1: couldn't find an input interrupt endpoint [ 31.962370] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 31.962439] ata1.01: failed command: READ DMA [ 31.962503] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 31.962616] ata1.01: status: { DRDY } [ 33.598405] device fsid 33e09c2d-5913-4ece-bbc2-069d55f1fecc devid 1 transid 69987 /dev/sdf3 [ 33.598725] btrfs: disk space caching is enabled [ 33.606689] Btrfs detected SSD devices, enabling SSD mode [ 37.020231] ata1: link is slow to respond, please be patient (ready=0) [ 42.027021] ata1: device not ready (errno=-16), forcing hardreset [ 42.027026] ata1: soft resetting link [ 47.338344] ata1: link is slow to respond, please be patient (ready=0) [ 52.089695] ata1: SRST failed (errno=-16) [ 52.089762] ata1: soft resetting link [ 52.363688] ata1.01: configured for UDMA/100 [ 52.363692] ata1.01: device reported invalid CHS sector 0 [ 52.363700] ata1: EH complete [ 63.742321] systemd[1]: systemd 204 running in system mode. (+PAM -LIBWRAP -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ) [ 63.742886] systemd[1]: Set hostname to <desk>. [ 63.784765] systemd[1]: Starting Forward Password Requests to Wall Directory Watch. [ 63.784814] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 63.784824] systemd[1]: Expecting device sys-subsystem-net-devices-enp8s0.device... [ 63.784831] systemd[1]: Expecting device sys-subsystem-net-devices-enp5s0.device... [ 63.784838] systemd[1]: Starting Remote File Systems. [ 63.784845] systemd[1]: Reached target Remote File Systems. [ 63.784853] systemd[1]: Starting /dev/initctl Compatibility Named Pipe. [ 63.784871] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe. [ 63.784877] systemd[1]: Starting Delayed Shutdown Socket. [ 63.784893] systemd[1]: Listening on Delayed Shutdown Socket. [ 63.784899] systemd[1]: Starting LVM2 metadata daemon socket. [ 63.784918] systemd[1]: Listening on LVM2 metadata daemon socket. [ 63.784923] systemd[1]: Starting Device-mapper event daemon FIFOs. [ 63.784943] systemd[1]: Listening on Device-mapper event daemon FIFOs. [ 63.784950] systemd[1]: Starting Journal Socket. [ 63.784984] systemd[1]: Listening on Journal Socket. [ 63.785795] systemd[1]: Starting Apply Kernel Variables... [ 63.786316] systemd[1]: Starting Create static device nodes in /dev... [ 63.786626] systemd[1]: Starting Journal Service... [ 63.786985] systemd[1]: Started Journal Service. [ 63.787009] systemd[1]: Starting Setup Virtual Console... [ 63.787420] systemd[1]: Mounting Debug File System... [ 63.788914] systemd[1]: Starting Load Kernel Modules... [ 63.789374] systemd[1]: Starting udev Kernel Socket. [ 63.789393] systemd[1]: Listening on udev Kernel Socket. [ 63.789436] systemd[1]: Starting udev Control Socket. [ 63.789454] systemd[1]: Listening on udev Control Socket. [ 63.789488] systemd[1]: Starting udev Coldplug all Devices... [ 63.789937] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. [ 63.790037] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ 63.791799] systemd[1]: Started Set Up Additional Binary Formats. [ 63.791816] systemd[1]: Mounting POSIX Message Queue File System... [ 63.792117] systemd[1]: Mounting Huge Pages File System... [ 63.792472] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch. [ 63.792509] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 63.792520] systemd[1]: Starting Swap. [ 63.792527] systemd[1]: Reached target Swap. [ 63.792537] systemd[1]: Mounting Temporary Directory... [ 63.792927] systemd[1]: Starting File System Check on Root Device... [ 63.793307] systemd[1]: Expecting device dev-disk-by\x2duuid-c17eadf9\x2deac3\x2d4b44\x2d8eae\x2d970173be0269.device... [ 63.793318] systemd[1]: Expecting device dev-disk-by\x2duuid-99463033\x2d334a\x2d4947\x2d9965\x2d2491be857342.device... [ 63.793329] systemd[1]: Expecting device dev-disk-by\x2duuid-0d0b891c\x2da74e\x2d4216\x2daed0\x2db4c5683b3f4f.device... [ 63.793339] systemd[1]: Expecting device dev-mapper-home.device... [ 63.793346] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dSAMSUNG_HD103SI_S1VSJ9CZ618796\x2dpart4.device... [ 63.793353] systemd[1]: Expecting device dev-mapper-files0.device... [ 63.793363] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dSAMSUNG_HD204UI_S2H7J1CB903523\x2dpart2.device... [ 63.793370] systemd[1]: Expecting device dev-mapper-files1.device... [ 63.793379] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dWDC_WD20EARX\x2d00PASB0_WD\x2dWCAZAH511243\x2dpart1.device... [ 63.793386] systemd[1]: Expecting device dev-mapper-files2.device... [ 63.793393] systemd[1]: Expecting device dev-disk-by\x2did-ata\x2dWDC_WD10EADS\x2d00L5B1_WD\x2dWCAU4C887789\x2dpart1.device... [ 63.793400] systemd[1]: Expecting device dev-mapper-files3.device... [ 63.793929] systemd[1]: Started Apply Kernel Variables. [ 63.819258] btrfs: use lzo compression [ 63.819261] btrfs: use ssd allocation scheme [ 63.819273] btrfs: disk space caching is enabled [ 63.819427] systemd-udevd[211]: starting version 204 [ 63.820248] microcode: CPU0: patch_level=0x06000817 [ 63.829635] microcode: CPU1: patch_level=0x06000817 [ 63.829649] microcode: CPU2: patch_level=0x06000817 [ 63.829657] microcode: CPU3: patch_level=0x06000817 [ 63.829666] microcode: CPU4: patch_level=0x06000817 [ 63.829673] microcode: CPU5: patch_level=0x06000817 [ 63.829680] microcode: CPU6: patch_level=0x06000817 [ 63.829689] microcode: CPU7: patch_level=0x06000817 [ 63.829757] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 63.829815] systemd-modules-load[192]: Inserted module 'microcode' [ 63.837429] ACPI: acpi_idle registered with cpuidle [ 63.842104] acpi-cpufreq: overriding BIOS provided _PSD data [ 63.843068] systemd-modules-load[192]: Inserted module 'acpi_cpufreq' [ 63.843107] systemd-modules-load[192]: Module 'cpufreq_ondemand' is builtin [ 63.843591] RPC: Registered named UNIX socket transport module. [ 63.843593] RPC: Registered udp transport module. [ 63.843594] RPC: Registered tcp transport module. [ 63.843595] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 63.846582] fuse init (API version 7.21) [ 63.846702] systemd-modules-load[192]: Inserted module 'fuse' [ 63.852743] device-mapper: uevent: version 1.0.3 [ 63.852965] device-mapper: ioctl: 4.24.0-ioctl (2013-01-15) initialised: dm-devel@redhat.com [ 63.853006] systemd-modules-load[192]: Inserted module 'dm_mod' [ 63.858862] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 63.861605] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input4 [ 63.861616] ACPI: Power Button [PWRB] [ 63.861713] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input5 [ 63.861719] ACPI: Power Button [PWRF] [ 63.865563] FS-Cache: Loaded [ 63.873987] FS-Cache: Netfs 'nfs' registered for caching [ 63.906386] wmi: Mapper loaded [ 63.937443] pps_core: LinuxPPS API ver. 1 registered [ 63.937449] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 63.942978] PTP clock support registered [ 63.951959] tg3.c:v3.130 (February 14, 2013) [ 63.962462] tg3 0000:05:00.0 eth0: Tigon3 [partno(BCM57781) rev 57785100] (PCI Express) MAC address bc:5f:f4:96:14:9d [ 63.962474] tg3 0000:05:00.0 eth0: attached PHY is 57765 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 63.962482] tg3 0000:05:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1] [ 63.962490] tg3 0000:05:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 63.975019] MCE: In-kernel MCE decoding enabled. [ 63.983574] EDAC MC: Ver: 3.0.0 [ 63.990404] input: HDA ATI SB Line as /devices/pci0000:00/0000:00:14.2/sound/card0/input6 [ 63.991242] AMD64 EDAC driver v3.4.0 [ 63.992089] input: HDA ATI SB Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input7 [ 63.992127] input: HDA ATI SB Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input8 [ 63.992162] input: HDA ATI SB Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card0/input9 [ 63.992235] input: HDA ATI SB Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card0/input10 [ 63.992266] input: HDA ATI SB Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card0/input11 [ 63.992305] input: HDA ATI SB Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card0/input12 [ 63.992383] input: HDA ATI SB Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card0/input13 [ 63.993624] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 [ 63.993897] hda_intel: Disabling MSI [ 63.993908] hda-intel 0000:01:00.1: Handle VGA-switcheroo audio client [ 63.994711] EDAC amd64: DRAM ECC disabled. [ 63.994733] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. Either enable ECC checking or force module loading by setting 'ecc_enable_override'. (Note that use of the override may cause unknown side effects.) [ 63.997086] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05 [ 63.997120] sp5100_tco: PCI Revision ID: 0x42 [ 63.997149] sp5100_tco: Using 0xfed80b00 for watchdog MMIO address [ 63.997159] sp5100_tco: Last reboot was not triggered by watchdog. [ 63.997271] sp5100_tco: initialized (0xffffc900121feb00). heartbeat=60 sec (nowayout=0) [ 64.000212] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 64.001029] r8169 0000:08:00.0: irq 75 for MSI/MSI-X [ 64.004497] r8169 0000:08:00.0 eth1: RTL8168b/8111b at 0xffffc9001285e000, 74:ea:3a:80:f2:f5, XID 18000000 IRQ 75 [ 64.004506] r8169 0000:08:00.0 eth1: jumbo features [frames: 4080 bytes, tx checksumming: ko] [ 64.034976] kvm: Nested Virtualization enabled [ 64.034983] kvm: Nested Paging enabled [ 64.061792] device fsid 33e09c2d-5913-4ece-bbc2-069d55f1fecc devid 1 transid 69987 /dev/sdf3 [ 64.079895] systemd-udevd[241]: renamed network interface eth0 to enp5s0 [ 64.116024] systemd-udevd[241]: renamed network interface eth1 to enp8s0 [ 64.168124] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.0/0000:01:00.1/sound/card1/input14 [ 64.168301] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.0/0000:01:00.1/sound/card1/input15 [ 64.168450] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.0/0000:01:00.1/sound/card1/input16 [ 64.169170] hda-intel 0000:06:00.1: Handle VGA-switcheroo audio client [ 64.169277] snd_hda_intel 0000:06:00.1: irq 76 for MSI/MSI-X [ 64.174041] input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:0b.0/0000:06:00.1/sound/card2/input17 [ 64.211834] EXT4-fs (sdf1): mounting ext2 file system using the ext4 subsystem [ 64.212705] EXT4-fs (sdf1): mounted filesystem without journal. Opts: (null) [ 64.495480] EXT4-fs (sdc3): mounted filesystem with ordered data mode. Opts: (null) [ 71.464558] bio: create slab <bio-1> at 1 [ 71.623112] bio: create slab <bio-1> at 1 [ 72.290453] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled [ 72.292244] XFS (dm-0): Mounting Filesystem [ 72.471942] XFS (dm-0): Ending clean mount [ 79.849647] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.124275] ata1.01: limiting speed to UDMA/66:PIO4 [ 83.124284] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 83.124375] ata1.01: failed command: READ DMA [ 83.124449] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 83.124571] ata1.01: status: { DRDY } [ 83.375580] XFS (dm-2): Mounting Filesystem [ 83.528100] XFS (dm-2): Ending clean mount [ 87.645654] device fsid 374ec396-e838-46a8-9298-a0c110be2283 devid 1 transid 106056 /dev/dm-3 [ 87.666262] device fsid 374ec396-e838-46a8-9298-a0c110be2283 devid 1 transid 106056 /dev/mapper/home [ 87.666893] btrfs: use lzo compression [ 87.666899] btrfs: use ssd allocation scheme [ 87.666904] btrfs: disk space caching is enabled [ 88.182170] ata1: link is slow to respond, please be patient (ready=0) [ 93.188960] ata1: device not ready (errno=-16), forcing hardreset [ 93.188971] ata1: soft resetting link [ 95.178654] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.215897] systemd-journald[189]: File /var/log/journal/e3c0520ae85a403a8970565f63db0dad/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 95.234368] w83627ehf: Found NCT6775F chip at 0x290 [ 95.255172] ip_tables: (C) 2000-2006 Netfilter Core Team [ 95.278686] tg3 0000:05:00.0: irq 76 for MSI/MSI-X [ 95.278698] tg3 0000:05:00.0: irq 77 for MSI/MSI-X [ 95.278706] tg3 0000:05:00.0: irq 78 for MSI/MSI-X [ 95.278713] tg3 0000:05:00.0: irq 79 for MSI/MSI-X [ 95.278720] tg3 0000:05:00.0: irq 80 for MSI/MSI-X [ 95.431968] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready [ 95.436203] r8169 0000:08:00.0 enp8s0: link down [ 95.436248] IPv6: ADDRCONF(NETDEV_UP): enp8s0: link is not ready [ 95.440355] systemd-logind[735]: Watching system buttons on /dev/input/event5 (Power Button) [ 95.440440] systemd-logind[735]: Watching system buttons on /dev/input/event4 (Power Button) [ 98.502285] ata1: link is slow to respond, please be patient (ready=0) [ 98.824473] tg3 0000:05:00.0 enp5s0: Link is up at 1000 Mbps, full duplex [ 98.824497] tg3 0000:05:00.0 enp5s0: Flow control is on for TX and on for RX [ 98.824504] tg3 0000:05:00.0 enp5s0: EEE is disabled [ 98.824530] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready [ 99.007095] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 99.024418] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory [ 99.024454] NFSD: starting 90-second grace period (net ffffffff81891f00) [ 102.463962] Calling /sbin/tomoyo-init to load policy. Please wait. [ 102.610329] TOMOYO: 2.5.0 [ 102.610437] Mandatory Access Control activated. [ 102.661726] systemd-journald[189]: Failed to set ACL on /var/log/journal/e3c0520ae85a403a8970565f63db0dad/user-1000.journal, ignoring: Invalid argument [ 103.256650] ata1: SRST failed (errno=-16) [ 103.256663] ata1: soft resetting link [ 103.540336] ata1.01: configured for UDMA/66 [ 103.540343] ata1.01: device reported invalid CHS sector 0 [ 103.540350] ata1: EH complete [ 104.770584] EXT4-fs (sdc3): re-mounted. Opts: commit=0 [ 104.827244] EXT4-fs (dm-1): re-mounted. Opts: errors=remount-ro,commit=0 [ 104.894490] EXT4-fs (dm-4): re-mounted. Opts: errors=remount-ro,commit=0 [ 134.158287] ata1.01: limiting speed to UDMA/33:PIO4 [ 134.158296] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 134.158304] ata1.01: failed command: READ DMA [ 134.158313] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 134.158318] ata1.01: status: { DRDY } [ 139.216274] ata1: link is slow to respond, please be patient (ready=0) [ 144.223140] ata1: device not ready (errno=-16), forcing hardreset [ 144.223152] ata1: soft resetting link [ 149.534523] ata1: link is slow to respond, please be patient (ready=0) [ 154.285986] ata1: SRST failed (errno=-16) [ 154.285999] ata1: soft resetting link [ 154.566812] ata1.01: configured for UDMA/33 [ 154.566826] ata1.01: device reported invalid CHS sector 0 [ 154.566853] ata1: EH complete [ 185.320926] ata1.01: limiting speed to PIO4 [ 185.320936] ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [ 185.320943] ata1.01: failed command: READ DMA [ 185.320952] ata1.01: cmd c8/00:08:00:00:00/00:00:00:00:00/f0 tag 0 dma 4096 in res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) [ 185.320958] ata1.01: status: { DRDY } [ 190.378863] ata1: link is slow to respond, please be patient (ready=0) [ 195.385674] ata1: device not ready (errno=-16), forcing hardreset [ 195.385686] ata1: soft resetting link [ 200.697124] ata1: link is slow to respond, please be patient (ready=0) [ 205.448488] ata1: SRST failed (errno=-16) [ 205.448501] ata1: soft resetting link [ 205.726753] ata1.01: configured for PIO4 [ 205.726767] ata1.01: device reported invalid CHS sector 0 [ 205.726792] ata1: EH complete [ 205.794930] sda: sda1 sda2 sda3 < sda5 sda6 > [ 205.795831] sda: detected capacity change from 0 to 320072933376 [ 205.796047] sd 0:0:1:0: [sda] Attached SCSI disk [ 251.299287] vfio-pci 0000:06:00.0: enabling device (0000 -> 0003) [ 255.133480] usb 8-1: USB disconnect, device number 2 [ 255.133489] usb 8-1.1: USB disconnect, device number 3 [ 255.157518] usb 8-1.2: USB disconnect, device number 4 [ 255.166979] usb 8-1.3: USB disconnect, device number 5 [ 255.826679] usb 8-2: new full-speed USB device number 6 using ohci_hcd [ 255.978095] hub 8-2:1.0: USB hub found [ 255.980003] hub 8-2:1.0: 4 ports detected [ 256.270515] usb 8-2.1: new low-speed USB device number 7 using ohci_hcd [ 256.396482] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.1/8-2.1:1.0/input/input18 [ 256.396793] hid-generic 0003:04D9:1605.0005: input,hidraw0: USB HID v1.10 Keyboard [ USB Keyboard] on usb-0000:00:12.0-2.1/input0 [ 256.418909] input: USB Keyboard as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.1/8-2.1:1.1/input/input19 [ 256.419159] hid-generic 0003:04D9:1605.0006: input,hidraw1: USB HID v1.10 Device [ USB Keyboard] on usb-0000:00:12.0-2.1/input1 [ 256.486896] usb 8-2.2: new full-speed USB device number 8 using ohci_hcd [ 256.587496] input: USB USB Mouse as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.2/8-2.2:1.0/input/input20 [ 256.587765] hid-generic 0003:1D57:3108.0007: input,hidraw2: USB HID v1.11 Mouse [USB USB Mouse] on usb-0000:00:12.0-2.2/input0 [ 257.604866] usb 8-2.3: new low-speed USB device number 9 using ohci_hcd [ 257.717260] input: No brand 4 Port KVMSwicther as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2.3/8-2.3:1.0/input/input21 [ 257.717481] hid-generic 0003:10D5:55A4.0008: input,hidraw3: USB HID v1.10 Keyboard [No brand 4 Port KVMSwicther] on usb-0000:00:12.0-2.3/input0 [ 257.719123] usbhid 8-2.3:1.1: couldn't find an input interrupt endpoint

I read the line " [    0.000000] Please enable the IOMMU option in the BIOS setup", but I already did that.
The input and usb/usbhid spam is there because of my "KVMSwicther".

what I'm executing

/usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -vga none -nographic -bios /usr/share/qemu/bios.bin -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 1024 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -device vfio-pci,host=06:00.1,addr=0.1,bus=root.1 -boot menu=on -monitor stdio

Any ideas? What should I test/change? Is there some sort of debugging mode for qemu?

Following the conversation on qemu-devel, it looks like there might be some problems using the nvidia binary driver and vfio passthrough check out:

http://lists.gnu.org/archive/html/qemu- … 04296.html

And a proposed solution:

http://lists.gnu.org/archive/html/qemu- … 04300.html

Offline

#123 2013-05-29 21:52:13

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Following the conversation on qemu-devel, it looks like there might be some problems using the nvidia binary driver and vfio passthrough check out:

http://lists.gnu.org/archive/html/qemu- … 04296.html

And a proposed solution:

http://lists.gnu.org/archive/html/qemu- … 04300.html

Did I mention that you are a genius? No, srsly… with this patch, it finally works with "-M q35 -vga none" and vfio-pci.


i'm sorry for my poor english wirting skills…

Offline

#124 2013-05-30 15:16:06

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Following the conversation on qemu-devel, it looks like there might be some problems using the nvidia binary driver and vfio passthrough check out:

http://lists.gnu.org/archive/html/qemu- … 04296.html

And a proposed solution:

http://lists.gnu.org/archive/html/qemu- … 04300.html

Did I mention that you are a genius? No, srsly… with this patch, it finally works with "-M q35 -vga none" and vfio-pci.

Nice, though all credit goes to the guy that found the solution not me

Last edited by nbhs (2013-05-30 15:16:51)

Offline

#125 2013-05-30 18:46:52

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

The redhat dev recommends using q35 chipset and the gpu behind a pcie root port, i dont use hdmi, so i have no experience with it, i havent experienced stutter in flash but i did experience some in games, i solved this by disabling nested page tables and tweaking the kernel config (see my second post on first page).

Xen 3dmark vs kvm is about the same with the above tweaks.

Okay, i´ll try out Q35 when i´ll have more time. I was going to try the NPT Flag when i first found this thread and today i had some spare time to test ist. Just to give an Impression how much this did improve my results (maybe for some other thread readers):

Unigine Heaven Benchmark 4.0:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:

1. Run (with running, but stuttering config)
FPS: 9.5
Core: 239
Min FPS: 4.9
Max FPS: 16.5
Impression: Overall Stuttering, stuttering in sound too

2. Run, now NPT disabled:
FPS: 39.2
Score: 988
Min FPS: 7.8
Max FPS: 83.0
Impression: Seems smooth, but sometimes frames seem to get into a jam: then it looks like someone is playing a movie in fast forward for some microseconds.

3. Run, NPT disabled and now Governor set to performance:
FPS: 62.8
Score: 1582
Min FPS: 8.7
Max FPS: 103.8
Impression: Really smooth, good FPS smile

Still running on stock kernel and so on, but still i am impressed by these improvements. Thanks for the good advices with nested page tables and the governor smile
Sadly flash is still not running good. Looks better, but still stuttering. At least the sound seems to play continously.

Last edited by Diggo (2013-05-30 18:47:20)

Offline

#126 2013-05-30 18:54:58

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:
nbhs wrote:

The redhat dev recommends using q35 chipset and the gpu behind a pcie root port, i dont use hdmi, so i have no experience with it, i havent experienced stutter in flash but i did experience some in games, i solved this by disabling nested page tables and tweaking the kernel config (see my second post on first page).

Xen 3dmark vs kvm is about the same with the above tweaks.

Okay, i´ll try out Q35 when i´ll have more time. I was going to try the NPT Flag when i first found this thread and today i had some spare time to test ist. Just to give an Impression how much this did improve my results (maybe for some other thread readers):

Unigine Heaven Benchmark 4.0:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:

1. Run (with running, but stuttering config)
FPS: 9.5
Core: 239
Min FPS: 4.9
Max FPS: 16.5
Impression: Overall Stuttering, stuttering in sound too

2. Run, now NPT disabled:
FPS: 39.2
Score: 988
Min FPS: 7.8
Max FPS: 83.0
Impression: Seems smooth, but sometimes frames seem to get into a jam: then it looks like someone is playing a movie in fast forward for some microseconds.

3. Run, NPT disabled and now Governor set to performance:
FPS: 62.8
Score: 1582
Min FPS: 8.7
Max FPS: 103.8
Impression: Really smooth, good FPS smile

Still running on stock kernel and so on, but still i am impressed by these improvements. Thanks for the good advices with nested page tables and the governor smile
Sadly flash is still not running good. Looks better, but still stuttering. At least the sound seems to play continously.

Try using my kernel config and disabling dynticks on your bootloader (nohz=off)  + q35 and gpu behind pcie root port, i got a massive boost using that.

Offline

#127 2013-06-01 00:46:27

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Hi. tested the patches.
The warning output with the trace is gone, but I have not seen any changed behavior wink.
Edit: But I have only tested with intel GPU as host.

Did also a benchmark test with pci-assign and no changes to my kvm-start:

qemu-system-x86_64 --enable-kvm -m 6G -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -drive file=/var/lib/libvirt/images/win-test.img \ -hdb /dev/sda3

Unigine Heaven Benchmark 4.0. settings like diggo:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:
Hardware is a AMD HD 7950

FPS: 117,5
Score: 2959
Min FPS: 9,2
Max FPS: 194,3

Last edited by apoapo (2013-06-01 01:10:18)

Offline

#128 2013-06-01 02:11:19

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:
nbhs wrote:

Hi. tested the patches.
The warning output with the trace is gone, but I have not seen any changed behavior wink.
Edit: But I have only tested with intel GPU as host.

Did also a benchmark test with pci-assign and no changes to my kvm-start:

qemu-system-x86_64 --enable-kvm -m 6G -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -drive file=/var/lib/libvirt/images/win-test.img \ -hdb /dev/sda3

Unigine Heaven Benchmark 4.0. settings like diggo:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:
Hardware is a AMD HD 7950

FPS: 117,5
Score: 2959
Min FPS: 9,2
Max FPS: 194,3

Too bad it didnt work sad im gonna try unigine and see what i get.

Offline

#129 2013-06-01 22:50:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well i ran the benchmark

Unigine 4.0 on radeon 6950

resolution 1980x1080

Low settings & tessellation off: ~1680
Extreme settings & tessellation normal: ~660

Last edited by nbhs (2013-06-01 22:51:57)

Offline

#130 2013-06-02 10:00:31

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:

Firstly, cheers for the tutorial. That must have taken awhile to write, and the effort is appreciated.

I've been looking into setting up a VGA passthrough setup under Xen for my next build for awhile now. I'm trying to work out what the advantages of this setup are over a setup under Xen. From what I've seen thrown around in different places, the main advantages are:

  • a less hacky setup for the passthrough, requiring less patches (once all this stuff gets released)

  • better support for NVIDIA cards? I've seen a few people say this, but nothing official to confirm it.

  • KVM receives more support than Xen, mainly due to Red Hat

  • Xen has a (relatively small) impact on the speed of the host, whereas KVM has no impact

  • not having to mess around with xl vs xm under Xen

Am I missing anything? I've seen a lot of people getting excited about using KVM over Xen for VGA passthrough, but not many people saying why they are so excited.

I'm doing VGA passthrough on Xen 4.1 (Linux Mint 14 / Ubuntu 12.10) and here some observations:

  • Works well with a number of AMD (ATI) as well as some Nvidia Quadro video cards doing secondary passthrough - no patches, no nothing!

  • I've been using standard Xen packages from the Ubuntu / Linux Mint repos

  • Xen doesn't play well with Nvidia cards under dom0 - only Nouveau driver, no proprietary drivers (there may be a work-around, but I haven't found/tested it yet)

  • The latest Xen versions in the Ubuntu 12.10 and 13 repos break PCI passthrough - "force version" provides a work-around

  • Except for some expensive Quadro etc. cards, VGA passthrough using Nvidia cards usually require patches

  • I've seen reports that AMD CPUs may have issues with CPU power management/frequency settings under Xen - this may already be solved

  • Xen and specifically VGA passthrough under Xen is better documented (KVM is quite new to VGA passthrough)

  • Regarding performance, I wouldn't pay attention to the Phoronix benchmarks (they are unclear at best and inaccurate at worst) - I believe the OP gave some comparisons between KVM and Xen. My experiences with Xen are very good with regard to performance.

  • I'm using xm and will wait until xl provides the features I'm using now (specifically pci reattach)

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Offline

#131 2013-06-04 04:05:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

Last edited by nbhs (2013-06-04 04:12:30)

Offline

#132 2013-06-05 02:39:17

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

Thanks! I'm learning every day.

Offline

#133 2013-06-08 06:12:44

dalingrin
Member
Registered: 2009-03-18
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

I can confirm that KVM without vfio has VGA reset issues with Radeon on cards. However, Xen ~4.2 does not have issues with reset on my Radeon 7970. I used Xen VGA pass through for the better part of a year as my daily driver for games without issue.

Offline

#134 2013-06-08 07:56:40

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dalingrin wrote:
nbhs wrote:
powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

I can confirm that KVM without vfio has VGA reset issues with Radeon on cards. However, Xen ~4.2 does not have issues with reset on my Radeon 7970. I used Xen VGA pass through for the better part of a year as my daily driver for games without issue.

Using the xl or the xm toolkit?

I didnt have this issue on xen using the xm toolkit but i experienced performance degradation using xl

Last edited by nbhs (2013-06-08 08:06:27)

Offline

#135 2013-06-08 21:11:43

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! I'm also interested in passing thru my VGA to VM. But I get an error 12 (Not enough system resources) and can't solve this problem. My HW is: ASRock Z77 Extreme 6 (BIOS 2.70), Intel Core i5 3470, GPUS: Intel HD 2500 (host), AMD Sapphire Radeon 6750 (68bf), ATI Radeon 4850 (9442).

I use kernel version 3.9.4 with applied kvm polarity patch (to properly run OSX) and

 CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y

(i tried to do everything exactly as written in this article with the same result)

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipv6.disable=1 intel_iommu=on iommu=group_mf pci-stub.ids=1002:68bf,1002:9442 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0"

mkinitcpio.conf MODULES:

MODULES="pci-stub kvm kvm-intel i915 tun bridge fuse"

QEMU Win7 conf:

sudo qemu-system-x86_64 --enable-kvm -M q35 -m 2048 -cpu host \ -boot menu=on \ -uuid 91848507-02ce-7d89-1ed4-40039265bbf9 \ -smp 2,sockets=1,cores=2,threads=1 \ -vga qxl \ -spice port=5902,disable-ticketing \ -device virtio-serial \ -chardev spicevmc,id=vdagent,name=vdagent \ -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \ -readconfig /etc/qemu/ich9-ehci-uhci.cfg \ -chardev spicevmc,name=usbredir,id=usbredirchardev1 \ -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \ -chardev spicevmc,name=usbredir,id=usbredirchardev2 \ -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \ -chardev spicevmc,name=usbredir,id=usbredirchardev3 \ -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \ -net user \ -device ahci,bus=pcie.0,id=ahci \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=01.0,multifunction=on,x-vga=on \ -drive file='/home/splinter/Qemu/Win7/win7.img',if=virtio

(also I tried with -bios "/path/to/patched-bios/" and -vga none options - got "VM has no graphics card" message without any output to the secondary display)

dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.018497] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.018502] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.018572] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.326767] IOMMU 0 0xfed90000: using Queued invalidation [ 0.326768] IOMMU 1 0xfed91000: using Queued invalidation [ 0.326770] IOMMU: Setting RMRR: [ 0.326779] IOMMU: Setting identity map for device 0000:00:02.0 [0xaf800000 - 0xbf9fffff] [ 0.327898] IOMMU: Setting identity map for device 0000:00:1d.0 [0xad6f5000 - 0xad721fff] [ 0.327915] IOMMU: Setting identity map for device 0000:00:1a.0 [0xad6f5000 - 0xad721fff] [ 0.327929] IOMMU: Setting identity map for device 0000:00:14.0 [0xad6f5000 - 0xad721fff] [ 0.327938] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.327944] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] dmesg | grep vfio [ 16.812467] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) dmesg | grep DMAR [ 0.000000] ACPI: DMAR 00000000adafee70 000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.326743] DMAR: No ATSR found [ 0.765236] dmar: DMAR:[DMA Read] Request device [0c:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear

Thx for your help! smile

Last edited by mastersplinter777 (2013-06-08 21:47:11)

Offline

#136 2013-06-09 05:03:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mastersplinter777 wrote:

Hi! I'm also interested in passing thru my VGA to VM. But I get an error 12 (Not enough system resources) and can't solve this problem. My HW is: ASRock Z77 Extreme 6 (BIOS 2.70), Intel Core i5 3470, GPUS: Intel HD 2500 (host), AMD Sapphire Radeon 6750 (68bf), ATI Radeon 4850 (9442).

I use kernel version 3.9.4 with applied kvm polarity patch (to properly run OSX) and

 CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y

(i tried to do everything exactly as written in this article with the same result)

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipv6.disable=1 intel_iommu=on iommu=group_mf pci-stub.ids=1002:68bf,1002:9442 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0"

mkinitcpio.conf MODULES:

MODULES="pci-stub kvm kvm-intel i915 tun bridge fuse"

QEMU Win7 conf:

sudo qemu-system-x86_64 --enable-kvm -M q35 -m 2048 -cpu host \ -boot menu=on \ -uuid 91848507-02ce-7d89-1ed4-40039265bbf9 \ -smp 2,sockets=1,cores=2,threads=1 \ -vga qxl \ -spice port=5902,disable-ticketing \ -device virtio-serial \ -chardev spicevmc,id=vdagent,name=vdagent \ -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \ -readconfig /etc/qemu/ich9-ehci-uhci.cfg \ -chardev spicevmc,name=usbredir,id=usbredirchardev1 \ -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \ -chardev spicevmc,name=usbredir,id=usbredirchardev2 \ -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \ -chardev spicevmc,name=usbredir,id=usbredirchardev3 \ -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \ -net user \ -device ahci,bus=pcie.0,id=ahci \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=01.0,multifunction=on,x-vga=on \ -drive file='/home/splinter/Qemu/Win7/win7.img',if=virtio

(also I tried with -bios "/path/to/patched-bios/" and -vga none options - got "VM has no graphics card" message without any output to the secondary display)

dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.018497] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.018502] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.018572] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.326767] IOMMU 0 0xfed90000: using Queued invalidation [ 0.326768] IOMMU 1 0xfed91000: using Queued invalidation [ 0.326770] IOMMU: Setting RMRR: [ 0.326779] IOMMU: Setting identity map for device 0000:00:02.0 [0xaf800000 - 0xbf9fffff] [ 0.327898] IOMMU: Setting identity map for device 0000:00:1d.0 [0xad6f5000 - 0xad721fff] [ 0.327915] IOMMU: Setting identity map for device 0000:00:1a.0 [0xad6f5000 - 0xad721fff] [ 0.327929] IOMMU: Setting identity map for device 0000:00:14.0 [0xad6f5000 - 0xad721fff] [ 0.327938] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.327944] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] dmesg | grep vfio [ 16.812467] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) dmesg | grep DMAR [ 0.000000] ACPI: DMAR 00000000adafee70 000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.326743] DMAR: No ATSR found [ 0.765236] dmar: DMAR:[DMA Read] Request device [0c:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear

Thx for your help! smile

A lot of people have reported problems using the intel igp as the host boot device you might want try to disable it and use one of your radeon cards for the host

Last edited by nbhs (2013-06-09 05:06:06)

Offline

#137 2013-06-09 10:51:23

thelatios
Member
Registered: 2013-06-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

Thanks a lot for your guide, it helped a lot!

Even though VGA passthrough with VFIO is working (I got seabios output on my passthroughed card), a detail is bothering me.

I found this in my dmesg output:

[ 0.277175] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [ 0.277207] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [ 0.277237] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table [ 0.277268] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

I guess I have to add something in my kernel command line like you did (ivrs_ioapic[9]=...), but I'm kind of confused on how you got the right values to put there. Could you give me a hint?

My motherboard is a Asus Sabertooth 990FX R2.0 with a AMD Athlon FX 8350 CPU.

Thanks again for the great guide!

Last edited by thelatios (2013-06-09 10:52:28)

Offline

#138 2013-06-09 11:39:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thelatios wrote:

Hello,

Thanks a lot for your guide, it helped a lot!

Even though VGA passthrough with VFIO is working (I got seabios output on my passthroughed card), a detail is bothering me.

I found this in my dmesg output:

[ 0.277175] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [ 0.277207] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [ 0.277237] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table [ 0.277268] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

I guess I have to add something in my kernel command line like you did (ivrs_ioapic[9]=...), but I'm kind of confused on how you got the right values to put there. Could you give me a hint?

My motherboard is a Asus Sabertooth 990FX R2.0 with a AMD Athlon FX 8350 CPU.

Thanks again for the great guide!

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

Would you mind sharing which card you passed thru'?

Last edited by nbhs (2013-06-09 11:40:24)

Offline

#139 2013-06-09 12:44:04

thelatios
Member
Registered: 2013-06-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

Would you mind sharing which card you passed thru'?

Thanks for the patch. Do you know why 00:14.0 and 00:00.1 are the correct devices for those parameters?
On my motherboard,
- 00:14.0 is SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
- 00:00.1 doesn't exist in lspci...

The passed thru' card is a Radeon HD5450.

Thanks!

Offline

#140 2013-06-09 13:10:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thelatios wrote:
nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

Would you mind sharing which card you passed thru'?

Thanks for the patch. Do you know why 00:14.0 and 00:00.1 are the correct devices for those parameters?
On my motherboard,
- 00:14.0 is SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
- 00:00.1 doesn't exist in lspci...

The passed thru' card is a Radeon HD5450.

Thanks!

Yes they wont show up on lspci but those are the addresses of the sb ioapic and nb ioapic

Offline

#141 2013-06-10 01:56:49

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For anyone else who's gotten video passthrough to work but is annoyed by stuttery or inconsistent video/audio performance, I may have some useful information.

Passing through real devices to virtual machines provides essentially full throughput (average performance over time), but the problem is that the worst-case latency can be very bad for important things like interrupt handlers and deferred procedure calls. (Interrupt handlers allow hardware to quickly inform its driver of something; DPCs are used by Windows drivers to delay time-consuming work that would otherwise be done in the int handler). If a hardware interrupt or DPC is delayed for some reason, then the hardware driver in question basically just has to wait it out until the delay is over, and then maybe try to catch up afterward. But for latency-sensitive things like audio or video, a delay of even a few milliseconds can result in a break in the audio stream or a long video frame. For storage hardware, it can result in longer access times, and therefore poor random access performance.

On real hardware, interrupt or DPC latency are almost always consistently good (unless you have a hardware or driver problem) because the kernel gives these tasks very high priority. Within a virtual machine, though, things can get weird because interrupts have to get handled by the host kernel before being passed to the guest kernel, and because interrupt handlers and DPCs that need to be handled immediately by the guest kernel may get preempted by unimportant tasks on the host.

Here are some tools for measuring DPC and interrupt latency in Windows that should allow you to get an idea if this is a problem for you:
LatencyMon
DPC Latency Checker
Windows Performance Analyzer (surprisingly powerful)

The single most effective thing I was able to do to make the guest's latency more consistent was to run the qemu process under one of Linux's realtime scheduler classes. This will probably make the biggest difference if you haven't dedicated CPU cores exclusively to the guest operating system, because the host and guest are in competition for the same CPU resources. Under normal conditions, the scheduler may allow processes on the host to preempt the guest and doesn't make any guarantees about how long the guest might have to wait to get a time slice.

Of the two realtime classes available, round-robin is preferable to fifo. Exactly how to run the qemu process with a realtime scheduling class is complicated by a number of things, including whether your VM is running in its own cgroup, so I can't give instructions that are universally applicable. Because I happen to run my VM as a systemd service, it involved adding this to the service file:

CPUSchedulingPolicy=rr

Another tweak that may be effective if your guest OS is Windows 8 is to turn off dynamic ticks. Linux has had dynamic ticks for a while (called tickless operation, NOHZ, etc.) but it's new in Windows 8 because of Microsoft's shift in focus toward mobile devices. Unfortunately, it's turned on by default. It's primarily a power optimization, and supposedly it shouldn't make any noticeable impact on the system's latency, but it has the potential to cause issues, so I recommend disabling it.

Also, be sure to read nbhs's performance tuning advice in the first two posts of this thread. In particular, if you're using your own kernel configuration, definitely try turning on kernel preemption, increasing the tick frequency, and turning off dynamic ticks. These host kernel tweaks all have the potential to improve latency for the guest.

Last edited by jgott (2013-06-10 02:02:02)

Offline

#142 2013-06-16 03:58:28

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Intel users having problems with vfio grabbing a lot of devices like https://bbs.archlinux.org/viewtopic.php … 8#p1274378, might want to check this out: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Offline

#143 2013-06-16 06:59:57

apocolypse600
Member
Registered: 2013-05-26
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Where does everyone keep up to date with news about VGA passthrough? I follow this thread, http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and http://www.overclock.net/t/1205216/guid … al-machine , although those are both aimed at passthrough via Xen rather than KVM. I also follow the xen-users, xen-devel and kvm general mailing lists. I notice that nbhs sometimes posts from a different mailing list (like the post directly before this one), but I can't find an easy way to navigate that list. Mailing lists aren't really my thing though, so that could be why. The KVM irc on freenode is pretty quiet most of the time, but I usually lurk there aswell.

It looks like Xen 4.3 is coming out very soon (wiki says release date of tomorrow, and they are definitely into the RC phase), and according to http://wiki.xen.org/wiki/Xen_Roadmap/4.3 it's defaulting to upstream Qemu (although it has partial in brackets next to it, so I'm not entirely sure what that means), so it'll be interesting to see how well it works. Although in the Linux Mint thread I posted, they were reporting issues doing the passthrough with newer versions of Xen 4.2, so who knows what a new version might bring.

I like the idea of using a NVIDIA card over an AMD card, so that if for some reason I can't get the passthrough to work, I can fall back on the propriety NVIDIA drivers under Linux, and just continue to use wine for games like I have been doing. That's why I was leaning towards KVM over Xen for my first stab at this. Also maybe it's the cynic in me, but I'm not sure many projects that the Linux Foundation has worked with have ever lasted all that long, so I'm assuming KVM (with the backing of Redhat) is a better option for support in the long run. No offence meant to the Linux Foundation by that, just an observation. Maybe I'm just still bitter about MeeGo.

I haven't actually bought any hardware yet, as I was hoping for passthrough techniques to mature a little before I try and get it to work. I tested using Synergy to share a keyboard and mouse between a laptop and a desktop, and that worked really well, so I imagine that should work find for sharing the keyboard and mouse with the virtual machine host and guest. I'm not entirely sure about audio yet, but I'd rather not have to passthrough a USB controller and switch my USB headset from one USB port to another when switching from host to guest, but I could probably live with it. Can anyone see any immediate problems with using Synergy to share the keyboard and mouse, either integrated graphics (I've seen some reports of issues with this in this thread) or my current GTS 250 to serve the Linux host and feeding a newer GPU through to a Windows guest (Probably Windows 7, but I do still have an install disk for XP)? I'd probably pass a hard-drive through directly. It's a bit hard to make comments without the exact hardware specifics I know, I just want to make sure I'm not missing something glaring when starting to plan for this build.

Offline

#144 2013-06-18 04:12:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:

Where does everyone keep up to date with news about VGA passthrough? I follow this thread, http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and http://www.overclock.net/t/1205216/guid … al-machine , although those are both aimed at passthrough via Xen rather than KVM. I also follow the xen-users, xen-devel and kvm general mailing lists. I notice that nbhs sometimes posts from a different mailing list (like the post directly before this one), but I can't find an easy way to navigate that list. Mailing lists aren't really my thing though, so that could be why. The KVM irc on freenode is pretty quiet most of the time, but I usually lurk there aswell.

It looks like Xen 4.3 is coming out very soon (wiki says release date of tomorrow, and they are definitely into the RC phase), and according to http://wiki.xen.org/wiki/Xen_Roadmap/4.3 it's defaulting to upstream Qemu (although it has partial in brackets next to it, so I'm not entirely sure what that means), so it'll be interesting to see how well it works. Although in the Linux Mint thread I posted, they were reporting issues doing the passthrough with newer versions of Xen 4.2, so who knows what a new version might bring.

I like the idea of using a NVIDIA card over an AMD card, so that if for some reason I can't get the passthrough to work, I can fall back on the propriety NVIDIA drivers under Linux, and just continue to use wine for games like I have been doing. That's why I was leaning towards KVM over Xen for my first stab at this. Also maybe it's the cynic in me, but I'm not sure many projects that the Linux Foundation has worked with have ever lasted all that long, so I'm assuming KVM (with the backing of Redhat) is a better option for support in the long run. No offence meant to the Linux Foundation by that, just an observation. Maybe I'm just still bitter about MeeGo.

I haven't actually bought any hardware yet, as I was hoping for passthrough techniques to mature a little before I try and get it to work. I tested using Synergy to share a keyboard and mouse between a laptop and a desktop, and that worked really well, so I imagine that should work find for sharing the keyboard and mouse with the virtual machine host and guest. I'm not entirely sure about audio yet, but I'd rather not have to passthrough a USB controller and switch my USB headset from one USB port to another when switching from host to guest, but I could probably live with it. Can anyone see any immediate problems with using Synergy to share the keyboard and mouse, either integrated graphics (I've seen some reports of issues with this in this thread) or my current GTS 250 to serve the Linux host and feeding a newer GPU through to a Windows guest (Probably Windows 7, but I do still have an install disk for XP)? I'd probably pass a hard-drive through directly. It's a bit hard to make comments without the exact hardware specifics I know, I just want to make sure I'm not missing something glaring when starting to plan for this build.

Well i read the mailing lists you can also subscribe if you want.
I dont think the xen devs are really interested in getting vga passthrough to work on non pro cards (quadro) atm, i havent seen any real activity since 4.0.
Right now most success reports have been from AMD users like me, you should expect this to change in the coming months since pci-assign is deprecated on kvm in favor of vfio-pci

Offline

#145 2013-06-20 20:39:01

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this post! i was looking for this.

The default kernel settings in 3.9.6 are like this

CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

Do i need to recompile it or is it enough to load that stuff as modules?

Seabios and Qemu conflict because Qemu already comes with seabios, what should i do in that regard? is it ok to just use the seabios version that came with qemu?

Last edited by rabcor (2013-06-20 20:39:30)

Offline

#146 2013-06-20 22:17:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Thanks for this post! i was looking for this.

The default kernel settings in 3.9.6 are like this

CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

Do i need to recompile it or is it enough to load that stuff as modules?

Seabios and Qemu conflict because Qemu already comes with seabios, what should i do in that regard? is it ok to just use the seabios version that came with qemu?

Yes its ok.

You'll need a patched seabios, this is the packages i currently use:

linux-mainline.zip
qemu.zip
seabios.zip

Offline

#147 2013-06-21 01:24:05

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't build that qemu package, breaks on compilation.

Since the latest version of qemu (1.6) includes seabios wouldn't it be best to just patch that included seabios with this before compilation? (but i don't know how to do that, since i can't find the seabios files in there)

Last edited by rabcor (2013-06-21 03:38:32)

Offline

#148 2013-06-21 15:17:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I can't build that qemu package, breaks on compilation.

Since the latest version of qemu (1.6) includes seabios wouldn't it be best to just patch that included seabios with this before compilation? (but i don't know how to do that, since i can't find the seabios files in there)

Have you updated your system? i just tried to build it ant it seems there was a problem with bluez dependency but it builds ok, i uploaded it again just in case

qemu.zip

I suppose you can patch the included seabios but on arch seabios comes as its own package, so you might break something

Offline

#149 2013-06-21 18:15:21

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

failed again, same error, and yes i updated my system only yesterday.

In file included from /usr/include/libfdt.h:55:0, from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28: /usr/include/fdt.h:58:2: error: unknown type name ‘fdt32_t’ fdt32_t magic; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup magic word FDT_MAGIC */ ^ /usr/include/fdt.h:59:2: error: unknown type name ‘fdt32_t’ fdt32_t totalsize; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup total size of DT block */ ^ /usr/include/fdt.h:60:2: error: unknown type name ‘fdt32_t’ fdt32_t off_dt_struct; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup offset to structure */ ^ /usr/include/fdt.h:61:2: error: unknown type name ‘fdt32_t’ fdt32_t off_dt_strings; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup offset to strings */ ^ /usr/include/fdt.h:62:2: error: unknown type name ‘fdt32_t’ fdt32_t off_mem_rsvmap; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup offset to memory reserve map */ ^ /usr/include/fdt.h:63:2: error: unknown type name ‘fdt32_t’ fdt32_t version; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup format version */ ^ /usr/include/fdt.h:64:2: error: unknown type name ‘fdt32_t’ fdt32_t last_comp_version; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup last compatible version */ ^ /usr/include/fdt.h:67:2: error: unknown type name ‘fdt32_t’ fdt32_t boot_cpuid_phys; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Which physical CPU id we're ^ /usr/include/fdt.h:70:2: error: unknown type name ‘fdt32_t’ fdt32_t size_dt_strings; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup size of the strings block */ ^ /usr/include/fdt.h:73:2: error: unknown type name ‘fdt32_t’ fdt32_t size_dt_struct; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup size of the structure block */ ^ /usr/include/fdt.h:77:2: error: unknown type name ‘fdt64_t’ fdt64_t address; ^ /usr/include/fdt.h:78:2: error: unknown type name ‘fdt64_t’ fdt64_t size; ^ /usr/include/fdt.h:82:2: error: unknown type name ‘fdt32_t’ fdt32_t tag; ^ /usr/include/fdt.h:87:2: error: unknown type name ‘fdt32_t’ fdt32_t tag; ^ /usr/include/fdt.h:88:2: error: unknown type name ‘fdt32_t’ fdt32_t len; ^ /usr/include/fdt.h:89:2: error: unknown type name ‘fdt32_t’ fdt32_t nameoff; ^ In file included from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28:0: /usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u32’: /usr/include/libfdt.h:921:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u64’: /usr/include/libfdt.h:956:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ /usr/include/libfdt.h: In function ‘fdt_property_u32’: /usr/include/libfdt.h:1032:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_property_u64’: /usr/include/libfdt.h:1037:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ /usr/include/libfdt.h: In function ‘fdt_setprop_u32’: /usr/include/libfdt.h:1193:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_setprop_u64’: /usr/include/libfdt.h:1228:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ /usr/include/libfdt.h: In function ‘fdt_appendprop_u32’: /usr/include/libfdt.h:1335:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_appendprop_u64’: /usr/include/libfdt.h:1370:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ make[1]: 1.sh topic_1.html topic_2.html topic_3.html topic_4.html topic_5.html [device_tree.o] Error 1 make: 1.sh topic_1.html topic_2.html topic_3.html topic_4.html topic_5.html [subdir-i386-softmmu] Error 2 ==> ERROR: A failure occurred in build().

if that's the case with seabios, can't the latest qemu version somehow be modified to not include seabios so it can be installed seperately like in older versions? (i would try to patch the seabios in qemu and hope nothing breaks in arch, but i don't know how i can find the right file, all i could find was an automated config script of some sort, but i'm at a loss of how i could force qemu to use an independantly installed version of seabios rather than installing it on its own, i figure it can't be that hard though, shouldn't it be possible to just copy old code and replace some of the new code to do that? or is it maybe more complicated than that?)

Last edited by rabcor (2013-06-22 04:10:29)

Offline

#150 2013-06-22 04:10:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

failed again, same error, and yes i updated my system only yesterday.

In file included from /usr/include/libfdt.h:55:0, from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28: /usr/include/fdt.h:58:2: error: unknown type name ‘fdt32_t’ fdt32_t magic; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup magic word FDT_MAGIC */ ^ /usr/include/fdt.h:59:2: error: unknown type name ‘fdt32_t’ fdt32_t totalsize; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup total size of DT block */ ^ /usr/include/fdt.h:60:2: error: unknown type name ‘fdt32_t’ fdt32_t off_dt_struct; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup offset to structure */ ^ /usr/include/fdt.h:61:2: error: unknown type name ‘fdt32_t’ fdt32_t off_dt_strings; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup offset to strings */ ^ /usr/include/fdt.h:62:2: error: unknown type name ‘fdt32_t’ fdt32_t off_mem_rsvmap; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup offset to memory reserve map */ ^ /usr/include/fdt.h:63:2: error: unknown type name ‘fdt32_t’ fdt32_t version; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup format version */ ^ /usr/include/fdt.h:64:2: error: unknown type name ‘fdt32_t’ fdt32_t last_comp_version; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup last compatible version */ ^ /usr/include/fdt.h:67:2: error: unknown type name ‘fdt32_t’ fdt32_t boot_cpuid_phys; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Which physical CPU id we're ^ /usr/include/fdt.h:70:2: error: unknown type name ‘fdt32_t’ fdt32_t size_dt_strings; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup size of the strings block */ ^ /usr/include/fdt.h:73:2: error: unknown type name ‘fdt32_t’ fdt32_t size_dt_struct; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup size of the structure block */ ^ /usr/include/fdt.h:77:2: error: unknown type name ‘fdt64_t’ fdt64_t address; ^ /usr/include/fdt.h:78:2: error: unknown type name ‘fdt64_t’ fdt64_t size; ^ /usr/include/fdt.h:82:2: error: unknown type name ‘fdt32_t’ fdt32_t tag; ^ /usr/include/fdt.h:87:2: error: unknown type name ‘fdt32_t’ fdt32_t tag; ^ /usr/include/fdt.h:88:2: error: unknown type name ‘fdt32_t’ fdt32_t len; ^ /usr/include/fdt.h:89:2: error: unknown type name ‘fdt32_t’ fdt32_t nameoff; ^ In file included from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28:0: /usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u32’: /usr/include/libfdt.h:921:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u64’: /usr/include/libfdt.h:956:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ /usr/include/libfdt.h: In function ‘fdt_property_u32’: /usr/include/libfdt.h:1032:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_property_u64’: /usr/include/libfdt.h:1037:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ /usr/include/libfdt.h: In function ‘fdt_setprop_u32’: /usr/include/libfdt.h:1193:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_setprop_u64’: /usr/include/libfdt.h:1228:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ /usr/include/libfdt.h: In function ‘fdt_appendprop_u32’: /usr/include/libfdt.h:1335:2: error: unknown type name ‘fdt32_t’ fdt32_t tmp = cpu_to_fdt32(val); ^ /usr/include/libfdt.h: In function ‘fdt_appendprop_u64’: /usr/include/libfdt.h:1370:2: error: unknown type name ‘fdt64_t’ fdt64_t tmp = cpu_to_fdt64(val); ^ make[1]: 1.sh topic_1.html topic_2.html topic_3.html topic_4.html topic_5.html [device_tree.o] Error 1 make: 1.sh topic_1.html topic_2.html topic_3.html topic_4.html topic_5.html [subdir-i386-softmmu] Error 2 ==> ERROR: A failure occurred in build().

if that's the case with seabios, can't the latest qemu version somehow be modified to not include seabios so it can be installed seperately like in older versions?


I managed to reproduce your error by installing dtc-git try this patched build:

qemu.zip

Offline

#151 2013-06-22 04:49:59

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your help, it unfortunately however gave me the exact same error again. so i tried replacing my dtc with dtc-git, and again got the exact same error again. I uninstalled dtc and it compiles just fine.(thanks to you i knew dtc was the problem)

I initially tried to install qemu-git from aur like you suggested in your guide, which then lead me to the conflict with seabios (which i assume started with v 1.6 of qemu). one of the dependencies for qemu-git was dtc, that's how i got it. (i didn't install dtc-git, but just dtc. I in fact don't know the difference between git and not git, do you have a link to somewhere that can explain that to me? i try to avoid git builds because i just don't know what they are nor if to handle them differently than any other packages)

I like this fileswap thing you're using, so much better to download from than that filesend thing, since you don't have to wait for a download button smile

Anyway, i'll try to use that last qemu build you sent me (without dtc installed) and the seabios files you sent as-well, tomorrow i'll try to learn how to use qemu to begin with since i never have (only virtualbox, vmware and microsofts thing) and after that we'll see how my attempt to passthrough will go! i'll get back to you when i get that far. Before i try though, i have a quick question or 2, do i need two monitors to do this? will it run windows 8? (i want to use it for it's performance over 7, otherwise i hate it's very existance, i've heard reports of it not working with qemu)

Last edited by rabcor (2013-06-22 05:49:26)

Offline

#152 2013-06-22 06:35:44

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Thank you for your help, it unfortunately however gave me the exact same error again. so i tried replacing my dtc with dtc-git, and again got the exact same error again. I uninstalled dtc and it compiles just fine.(thanks to you i knew dtc was the problem)

I initially tried to install qemu-git from aur like you suggested in your guide, which then lead me to the conflict with seabios (which i assume started with v 1.6 of qemu). one of the dependencies for qemu-git was dtc, that's how i got it. (i didn't install dtc-git, but just dtc. I in fact don't know the difference between git and not git, do you have a link to somewhere that can explain that to me? i try to avoid git builds because i just don't know what they are nor if to handle them differently than any other packages)

I like this fileswap thing you're using, so much better to download from than that filesend thing, since you don't have to wait for a download button smile

Anyway, i'll try to use that last qemu build you sent me (without dtc installed) and the seabios files you sent as-well, tomorrow i'll try to learn how to use qemu to begin with since i never have (only virtualbox, vmware and microsofts thing) and after that we'll see how my attempt to passthrough will go! i'll get back to you when i get that far. Before i try though, i have a quick question or 2, do i need two monitors to do this? will it run windows 8? (i want to use it for it's performance over 7, otherwise i hate it's very existance, i've heard reports of it not working with qemu)

yes it will run windows 8, im using it atm

Offline

#153 2013-06-23 05:23:48

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't know how to be able to tell if my board supports Interrupt Remapping

but when i try to run vfio-bind it says

# vfio-bind 0000:01:00.0 ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

if i run

# ls /sys/bus/pci/devices/0000\:01\:00.0/

(yes it requests "\"s like as if there are spaces between, i don't know why)
it doesn't give me any iommu directory so i assume this is because i didn't know how to add

vfio_iommu_type1.allow_unsafe_interrupts=1

to my grub.cfg, i did some googling but didn't find out how, i tried to add it myself in a place that looked about right but it did nothing

# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux root=UUID=876f6b61-7387-489e-b980-d42deb194c34 ro quiet

where am i supposed to add the line?

Offline

#154 2013-06-23 07:14:59

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I don't know how to be able to tell if my board supports Interrupt Remapping

but when i try to run vfio-bind it says

# vfio-bind 0000:01:00.0 ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

if i run

# ls /sys/bus/pci/devices/0000\:01\:00.0/

(yes it requests "\"s like as if there are spaces between, i don't know why)
it doesn't give me any iommu directory so i assume this is because i didn't know how to add

vfio_iommu_type1.allow_unsafe_interrupts=1

to my grub.cfg, i did some googling but didn't find out how, i tried to add it myself in a place that looked about right but it did nothing

# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux root=UUID=876f6b61-7387-489e-b980-d42deb194c34 ro quiet

where am i supposed to add the line?

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Offline

#155 2013-06-23 20:04:29

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I had already done that, and the kvm thing too. think the problem might be that my proprietary nvidia drivers are running? (should i uninstall them?)

Offline

#156 2013-06-23 20:11:38

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I had already done that, and the kvm thing too. think the problem might be that my proprietary nvidia drivers are running? (should i uninstall them?)

There's a patch for them, but using nouveau works too.


i'm sorry for my poor english wirting skills…

Offline

#157 2013-06-23 22:23:48

dogblue
Member
Registered: 2013-06-23
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: This issue was fixed by blacklisting the nouveau module.

Hi all,

I have spent most of the last couple of days trying to get this to work, with no luck so far.
As soon as I launch the virtual machine my entire computer will freeze and require a cold reboot.

Hardware is ASRock 970 Extreme3 motherboard, AMD FX CPU, NVidia GT 520 GPU (Primary, nouveau), AMD HD 7750 GPU (Secondary, passthrough).

I am using the AUR/ABS packages that nbhs posted a page or two ago.

I get an error on boot regarding the ivrs_ioapic lines

/bin/init: eval: line 1: ivrs_ioapic[9]=00:14.0: not found /bin/init: eval: line 1: ivrs_ioapic[10]=00:00.1: not found

I've attached as much additional information to this post as I can in the hope that someone will be able to offer a suggestion as to how I should proceed, as I am completely out of ideas.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=8ff5d0cb-77d4-403f-9387-e04405a329e7 ro iommu=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 acpi_sleep=s3_bios,s3_mode pci-stub.ids=1002:683f,1002:aab0 vfio_iommu_type1.allow_unsafe_interrupts=1

lspci -v (abridged output)

01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 352f Flags: bus master, fast devsel, latency 0, IRQ 24 Memory at fd000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (64-bit, prefetchable) [size=128M] Memory at d8000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at fe000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: nouveau Kernel modules: nouveau 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750] (prog-if 00 [VGA controller]) Subsystem: Hightech Information System Ltd. Device 2318 Flags: fast devsel, IRQ 10 Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at fe200000 (64-bit, non-prefetchable) [disabled] [size=256K] I/O ports at d000 [disabled] [size=256] Expansion ROM at fe240000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Kernel driver in use: pci-stub Kernel modules: radeon 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: Hightech Information System Ltd. Device aab0 Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at fe260000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: pci-stub Kernel modules: snd_hda_intel

I am attempting to start the VM with the following script. The machine displays a black qemu window and then halts (the whole computer, not just the VM) as soon as I run it.

#!/bin/sh #DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1" DEVICES="0000:02:00.0 0000:02:00.1" ./vfio-bind $DEVICES qemu-system-x86_64 --enable-kvm -M q35 -m 512 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

I would really appreciate any help that you all can offer. I actually purchased this machine a few days ago specifically to follow this guide, so I'm really looking forward to getting it all working!

Thanks in advance.

Last edited by dogblue (2013-06-24 21:35:23)

Offline

#158 2013-06-24 03:16:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dogblue wrote:

Hi all,

I have spent most of the last couple of days trying to get this to work, with no luck so far.
As soon as I launch the virtual machine my entire computer will freeze and require a cold reboot.

Hardware is ASRock 970 Extreme3 motherboard, AMD FX CPU, NVidia GT 520 GPU (Primary, nouveau), AMD HD 7750 GPU (Secondary, passthrough).

I am using the AUR/ABS packages that nbhs posted a page or two ago.

I get an error on boot regarding the ivrs_ioapic lines

/bin/init: eval: line 1: ivrs_ioapic[9]=00:14.0: not found /bin/init: eval: line 1: ivrs_ioapic[10]=00:00.1: not found

I've attached as much additional information to this post as I can in the hope that someone will be able to offer a suggestion as to how I should proceed, as I am completely out of ideas.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=8ff5d0cb-77d4-403f-9387-e04405a329e7 ro iommu=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 acpi_sleep=s3_bios,s3_mode pci-stub.ids=1002:683f,1002:aab0 vfio_iommu_type1.allow_unsafe_interrupts=1

lspci -v (abridged output)

01:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 520] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 352f Flags: bus master, fast devsel, latency 0, IRQ 24 Memory at fd000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (64-bit, prefetchable) [size=128M] Memory at d8000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at fe000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: nouveau Kernel modules: nouveau 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750] (prog-if 00 [VGA controller]) Subsystem: Hightech Information System Ltd. Device 2318 Flags: fast devsel, IRQ 10 Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at fe200000 (64-bit, non-prefetchable) [disabled] [size=256K] I/O ports at d000 [disabled] [size=256] Expansion ROM at fe240000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Kernel driver in use: pci-stub Kernel modules: radeon 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: Hightech Information System Ltd. Device aab0 Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at fe260000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: pci-stub Kernel modules: snd_hda_intel

I am attempting to start the VM with the following script. The machine displays a black qemu window and then halts (the whole computer, not just the VM) as soon as I run it.

#!/bin/sh #DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1" DEVICES="0000:02:00.0 0000:02:00.1" ./vfio-bind $DEVICES qemu-system-x86_64 --enable-kvm -M q35 -m 512 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

I would really appreciate any help that you all can offer. I actually purchased this machine a few days ago specifically to follow this guide, so I'm really looking forward to getting it all working!

Thanks in advance.

Those eval errors are normal dont worry about it. If you're overriding the ivrs table to enable intremap you dont need this line

vfio_iommu_type1.allow_unsafe_interrupts=1

Could you post your dmesg output?

Have you blacklisted the radeon or fglrx driver first? Also if you're using nvidia proprietary drivers you'll need to patch it (see andy123 post above). I would suggest uninstalling/blacklisting all the drivers first, and try to bind the devices with the script manually, then launch qemu.

Last edited by nbhs (2013-06-24 03:18:09)

Offline

#159 2013-06-24 03:20:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I had already done that, and the kvm thing too. think the problem might be that my proprietary nvidia drivers are running? (should i uninstall them?)

Just to make sure can you build/rebuild the linux-mainline package i posted with:

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

Could you be a little more specific about your hardware?, If your primary display is nvidia, and you're using nvidia proprietary drivers see andy123 post above.

Offline

#160 2013-06-24 12:16:28

dogblue
Member
Registered: 2013-06-23
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi nbhs, thanks for the suggestions (and the guide!)

nbhs wrote:

Those eval errors are normal dont worry about it. If you're overriding the ivrs table to enable intremap you dont need this line

vfio_iommu_type1.allow_unsafe_interrupts=1

Understood, I've removed that kernel option and tried again.

nbhs wrote:

Could you post your dmesg output?

Absolutely. http://pastebin.com/iqmLCsqq

nbhs wrote:

Have you blacklisted the radeon or fglrx driver first? Also if you're using nvidia proprietary drivers you'll need to patch it (see andy123 post above).

I have blacklisted the radeon driver as well as added the id's to pci-stub. I am also using the nouveau driver for the nvidia card, so that shouldn't be an issue at least.

I would suggest uninstalling/blacklisting all the drivers first, and try to bind the devices with the script manually, then launch qemu.

As far as I can tell the radeon module is blacklisted, and fglrx and the binary nvidia drivers have never been installed.

Here is what the drivers look like just prior to running the qemu command (after binding the devices with the vfio-bind script)

# ls /sys/bus/pci/drivers/vfio-pci/ 0000:02:00.0 0000:02:00.1 bind module new_id remove_id uevent unbind # ls /sys/bus/pci/drivers/nouveau/ 0000:01:00.0 bind module new_id remove_id uevent unbind # cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 # cat /etc/modprobe.d/blacklist.conf blacklist radeon

I was also able to retrieve the following output by running 'cat /proc/kmsg' during the crash

<6>[ 996.181525] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) <6>[ 996.207159] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 <6>[ 998.065072] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem <6>[ 998.065076] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:02:00.0 <3>[ 998.095398] ata3: illegal qc_active transition (00000001->97b282a8) <3>[ 998.095540] ata3.00: exception Emask 0x2 SAct 0x0 SErr 0xd0000000 action 0x6 frozen <3>[ 998.095567] ata3: SError: { } <3>[ 998.095580] ata3.00: failed command: FLUSH CACHE EXT <3>[ 998.095607] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 <3>[ 998.095607] res 40/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation) <3>[ 998.095637] ata3.00: status: { DRDY } <6>[ 998.095653] ata3: hard resetting link <4>[ 998.103575] Clocksource tsc unstable (delta = -147227892866 ns)

Thanks again for your help!

Last edited by dogblue (2013-06-24 12:18:50)

Offline

#161 2013-06-24 15:16:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dogblue wrote:

Hi nbhs, thanks for the suggestions (and the guide!)

nbhs wrote:

Those eval errors are normal dont worry about it. If you're overriding the ivrs table to enable intremap you dont need this line

vfio_iommu_type1.allow_unsafe_interrupts=1

Understood, I've removed that kernel option and tried again.

nbhs wrote:

Could you post your dmesg output?

Absolutely. http://pastebin.com/iqmLCsqq

nbhs wrote:

Have you blacklisted the radeon or fglrx driver first? Also if you're using nvidia proprietary drivers you'll need to patch it (see andy123 post above).

I have blacklisted the radeon driver as well as added the id's to pci-stub. I am also using the nouveau driver for the nvidia card, so that shouldn't be an issue at least.

I would suggest uninstalling/blacklisting all the drivers first, and try to bind the devices with the script manually, then launch qemu.

As far as I can tell the radeon module is blacklisted, and fglrx and the binary nvidia drivers have never been installed.

Here is what the drivers look like just prior to running the qemu command (after binding the devices with the vfio-bind script)

# ls /sys/bus/pci/drivers/vfio-pci/ 0000:02:00.0 0000:02:00.1 bind module new_id remove_id uevent unbind # ls /sys/bus/pci/drivers/nouveau/ 0000:01:00.0 bind module new_id remove_id uevent unbind # cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 # cat /etc/modprobe.d/blacklist.conf blacklist radeon

I was also able to retrieve the following output by running 'cat /proc/kmsg' during the crash

<6>[ 996.181525] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) <6>[ 996.207159] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 <6>[ 998.065072] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem <6>[ 998.065076] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:02:00.0 <3>[ 998.095398] ata3: illegal qc_active transition (00000001->97b282a8) <3>[ 998.095540] ata3.00: exception Emask 0x2 SAct 0x0 SErr 0xd0000000 action 0x6 frozen <3>[ 998.095567] ata3: SError: { } <3>[ 998.095580] ata3.00: failed command: FLUSH CACHE EXT <3>[ 998.095607] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 <3>[ 998.095607] res 40/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation) <3>[ 998.095637] ata3.00: status: { DRDY } <6>[ 998.095653] ata3: hard resetting link <4>[ 998.103575] Clocksource tsc unstable (delta = -147227892866 ns)

Thanks again for your help!


Have you tried to blacklist the nouveau module and start without X? take a look at this post, it seems to describe your problem: http://www.mail-archive.com/qemu-devel@ … 73628.html,

2) If the 'nouveau.ko' driver is loaded it is even more strange. As soon
as I start qemu all my SATA links get a hard reset and kernel freezes.
No SysRQs are working anymore and only reboot helps. If needed I can
look if I can get some dumps from this freeze because it writes nothing
more to the disks.

It seems he solved the problem using patched nvidia proprietary drivers as described here: http://www.mail-archive.com/qemu-devel@ … 74066.html

Offline

#162 2013-06-24 21:34:22

dogblue
Member
Registered: 2013-06-23
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Have you tried to blacklist the nouveau module and start without X? take a look at this post, it seems to describe your problem: http://www.mail-archive.com/qemu-devel@ … 73628.html,

That was it!
Blacklisting nouveau fixed it. No more crashes on starting the VM.

Thank you so much for all your help!!

Offline

#163 2013-06-25 03:20:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dogblue wrote:
nbhs wrote:

Have you tried to blacklist the nouveau module and start without X? take a look at this post, it seems to describe your problem: http://www.mail-archive.com/qemu-devel@ … 73628.html,

That was it!
Blacklisting nouveau fixed it. No more crashes on starting the VM.

Thank you so much for all your help!!

NIce! have you tried the patched nvidia drivers yet?

Offline

#164 2013-06-25 11:16:30

blackout23
Member
Registered: 2011-11-16
Posts: 765

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just wanted to thank you for this great thread and all the effort you put into it!

I have looked at VGA-Passthrough with Xen in the past but this Wiki article here has diminished my hopes a bit. What makes me wonder is why it says "400 Series and 600 Series should work" after some hacking and only on Windows XP. Since I have a GTX 580 OC I wonder why the 500 series was excluded in this wiki. Maybe this means I can't use it with your KVM tutorial, too?

My rig currently consists of a Intel i7 2600K @ 4.5 Ghz, 8 GB RAM, ASUS P8Z86-V Pro and GTX 580 @ 900 Mhz. If no one has ever tried a 500 series card I might have to do that and report back.

Last edited by blackout23 (2013-06-25 11:30:39)

Offline

#165 2013-06-25 11:31:54

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Just to make sure can you build/rebuild the linux-mainline package i posted with:...

Could you be a little more specific about your hardware?, If your primary display is nvidia, and you're using nvidia proprietary drivers see andy123 post above.

Yosh! i got it right!

My hardware setup is intel igpu for linux and nvidia gtx 670 for passthrough, my motherboard supports this and it's turned on, and i assume my i7 does as-well.

I don't know if it was setting these settings to Y or if it was setting

CONFIG_INTEL_IOMMU_DEFAULT_ON=y

that did it. but it worked and vfio-bind worked on 0000:01:00.0 and 1

#lspci ... 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ...

now my next problem is

# qemu-system-x86_64 --enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci, host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Gives me

qemu-system-x86_64: -cpu host: could not open disk image -smp: No such file or directory

Last edited by rabcor (2013-06-26 05:45:54)

Offline

#166 2013-06-25 15:32:17

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
nbhs wrote:

Just to make sure can you build/rebuild the linux-mainline package i posted with:...

Could you be a little more specific about your hardware?, If your primary display is nvidia, and you're using nvidia proprietary drivers see andy123 post above.

Yosh! i got it right!

My hardware setup is intel igpu for linux and nvidia gtx 670 for passthrough, my motherboard supports this and it's turned on, and i assume my i7 does as-well.

I don't know if it was setting these settings to Y or if it was setting

CONFIG_INTEL_IOMMU_DEFAULT_ON=y

that did it. but it worked and vfio-bind worked on 0000:01:00.0 and 1

#lspci ... 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ...

now my next problem is

# qemu-system-x86_64 --enable-kvm -M q35 -m 1024 -cpu host \ -smp 6, sockets=1, cores=6, threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci, host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Gives me

qemu-system-x86_64: -cpu host: could not open disk image -smp: No such file or directory

i think it is this line

-smp 6, sockets=1, cores=6, threads=1

to:

-smp 6,sockets=1,cores=6,threads=1 

You should know that, from previous users, using the igp as primary might not work

Last edited by nbhs (2013-06-25 15:33:59)

Offline

#167 2013-06-25 15:36:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blackout23 wrote:

I just wanted to thank you for this great thread and all the effort you put into it!

I have looked at VGA-Passthrough with Xen in the past but this Wiki article here has diminished my hopes a bit. What makes me wonder is why it says "400 Series and 600 Series should work" after some hacking and only on Windows XP. Since I have a GTX 580 OC I wonder why the 500 series was excluded in this wiki. Maybe this means I can't use it with your KVM tutorial, too?

My rig currently consists of a Intel i7 2600K @ 4.5 Ghz, 8 GB RAM, ASUS P8Z86-V Pro and GTX 580 @ 900 Mhz. If no one has ever tried a 500 series card I might have to do that and report back.

I think it means no one has tried it yet since this is experimental.

Offline

#168 2013-06-25 16:05:11

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

i think it is this line

-smp 6, sockets=1, cores=6, threads=1

to:

-smp 6,sockets=1,cores=6,threads=1 

You should know that, from previous users, using the igp as primary might not work

I tried that, it made no difference, i got the same error.

Even if it may not work with igp as primary, i'm sure it should be possible to make it past an error like that (if we can figure out the cause). I have no idea how to troubleshoot this, i don't even know what the error means to tell the truth. But if it's caused by the igp i will probably just buy another gpu later for this.

Last edited by rabcor (2013-06-25 18:05:21)

Offline

#169 2013-06-26 00:15:02

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
My kernel line used:
/kernel-genkernel-x86_64-3.9.6-gentoo root=/dev/sda2 ro radeon.audio=1 debug kvm-intel.nested=1 iommu=1 pci-stub.ids=1002:68f9,1002:aa68 ivrs_ioapic[9]=00:14.0
(You don't say how/why you use ivrs_ioapic, isn't it?).
I have an intel card (for my host), and an ATI 5450 for windows. I have uninstall all the radeon kernel module, but I have:
-device vfio-pci,host=01:00.0,addr=00.0,multifunction=on: PCI: slot 0 function 0 not available for vfio-pci, in use by i440FX
If you can help me, cheer,

Offline

#170 2013-06-26 03:01:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
nbhs wrote:

i think it is this line

-smp 6, sockets=1, cores=6, threads=1

to:

-smp 6,sockets=1,cores=6,threads=1 

You should know that, from previous users, using the igp as primary might not work

I tried that, it made no difference, i got the same error.

Even if it may not work with igp as primary, i'm sure it should be possible to make it past an error like that (if we can figure out the cause). I have no idea how to troubleshoot this, i don't even know what the error means to tell the truth. But if it's caused by the igp i will probably just buy another gpu later for this.

I think its this typo i overlooked before

--enable-kvm

change to:

-enable-kvm

Right now AMD mobo/cpu users have like 90% chance of getting this to work, even mixing nvidia/amd cards and with broken ivrs tables, remember this is work in progress, so i wouldn't waste money yet only to find it doesn't work, but im prety sure it will in the near future.

Last edited by nbhs (2013-06-26 03:24:47)

Offline

#171 2013-06-26 03:07:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:

Hello,
My kernel line used:
/kernel-genkernel-x86_64-3.9.6-gentoo root=/dev/sda2 ro radeon.audio=1 debug kvm-intel.nested=1 iommu=1 pci-stub.ids=1002:68f9,1002:aa68 ivrs_ioapic[9]=00:14.0
(You don't say how/why you use ivrs_ioapic, isn't it?).

If you're using an intel board you shouldnt be using ivrs_ioapic override, this is a workarround for amd boards

alpha_one_x86 wrote:

I have an intel card (for my host), and an ATI 5450 for windows. I have uninstall all the radeon kernel module, but I have:
-device vfio-pci,host=01:00.0,addr=00.0,multifunction=on: PCI: slot 0 function 0 not available for vfio-pci, in use by i440FX
If you can help me, cheer,

Yeah change this:

-device vfio-pci,host=01:00.0,addr=00.0,multifunction=on

to this:

-device vfio-pci,host=01:00.0,multifunction=on

Offline

#172 2013-06-26 05:53:08

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I think its this typo i overlooked before

--enable-kvm

change to:

-enable-kvm

Right now AMD mobo/cpu users have like 90% chance of getting this to work, even mixing nvidia/amd cards and with broken ivrs tables, remember this is work in progress, so i wouldn't waste money yet only to find it doesn't work, but im prety sure it will in the near future.

still nothing, i figured it seems to think that  "-cpu host" should be a disk image (which it cannot find) and that "-smp" should be a directory, which it also cannot find

if i remove "-cpu host \ -smp ...." it will give me the same error

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci, host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -m 1024: could not open disk image -bios: No such file or directory

Offline

#173 2013-06-26 08:58:00

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I have:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio_dma_map(0x7fc3e881f800, 0xfffe0000, 0x20000, 0x7fc3e0a00000) = -12 (Cannot allocate memory)
qemu-system-x86_64: vfio_dma_map(0x7fc3e881f800, 0xc8000, 0x18000, 0x7fc3e0808000) = -12 (Cannot allocate memory)
I have checked all into the web (right problem, iommu, ...)
I'm with intel haswell + linux 3.9.
What I should test?

Offline

#174 2013-06-26 09:11:50

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio_dma_map(0x7fc3e881f800, 0xfffe0000, 0x20000, 0x7fc3e0a00000) = -12 (Cannot allocate memory)
qemu-system-x86_64: vfio_dma_map(0x7fc3e881f800, 0xc8000, 0x18000, 0x7fc3e0808000) = -12 (Cannot allocate memory)
[…]
What I should test?

Sounds to me like mem-path/hugepages issues. You should probably read some things about hugepages, remove the mem-path option or lower the memory (-m parameter).


i'm sorry for my poor english wirting skills…

Offline

#175 2013-06-26 09:19:51

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

remove the mem-path option

How? I have not this options.
Already checked for the rest, tested with 512, I can't do less.

Offline

#176 2013-06-26 10:14:16

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I missed the "vfio_dma_map", which indicates some things, but I'm not the expert here. I managed to set this up and did some strange stuff with (1GB) hugepages, so I thought it might be that.
Maybe nbhs has some ideas.


i'm sorry for my poor english wirting skills…

Offline

#177 2013-06-26 13:00:46

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fixed by kdesu, now I have:
qemu-system-x86_64: virtio_ioport_write: unexpected address 0x13 value 0x1
Add crash into amd officiel windows drivers. Blue screen at windows boot.

Last edited by alpha_one_x86 (2013-06-26 13:01:03)

Offline

#178 2013-06-26 13:22:10

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,291

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@rabcor

You still have a rogue space after the first -device vfio-pci.

# qemu-system-x86_64 ... -device vfio-pci, host=01:00.0,...

I put a space in a working command and got a "could not open disk image" error, so that could be it.

Last edited by alphaniner (2013-06-26 13:22:39)


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#179 2013-06-26 13:22:31

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:

Add crash into amd officiel windows drivers. Blue screen at windows boot.

Sounds like you're not using a vfio-vga-reset patched qemu+kernel. Is it a fresh windows install or are you trying to migrate an old one?


i'm sorry for my poor english wirting skills…

Offline

#180 2013-06-26 13:26:07

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
alpha_one_x86 wrote:

Add crash into amd officiel windows drivers. Blue screen at windows boot.

Sounds like you're not using a vfio-vga-reset patched qemu+kernel. Is it a fresh windows install or are you trying to migrate an old one?

Near to be fresh. I will try the patch, and after the clean install.

EDIT: This repository is temporarily unavailable.
The backend storage is temporarily offline. Usually this means the storage server is undergoing maintenance. Please contact support if the problem persists.

Some body can send me the patch?

Last edited by alpha_one_x86 (2013-06-26 13:31:41)

Offline

#181 2013-06-27 04:45:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:
andy123 wrote:
alpha_one_x86 wrote:

Add crash into amd officiel windows drivers. Blue screen at windows boot.

Sounds like you're not using a vfio-vga-reset patched qemu+kernel. Is it a fresh windows install or are you trying to migrate an old one?

Near to be fresh. I will try the patch, and after the clean install.

EDIT: This repository is temporarily unavailable.
The backend storage is temporarily offline. Usually this means the storage server is undergoing maintenance. Please contact support if the problem persists.

Some body can send me the patch?

On page 6 i posted some builds for arch that contains the patches

Offline

#182 2013-06-27 07:23:18

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:

Where does everyone keep up to date with news about VGA passthrough? I follow this thread, http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and http://www.overclock.net/t/1205216/guid … al-machine , although those are both aimed at passthrough via Xen rather than KVM. I also follow the xen-users, xen-devel and kvm general mailing lists. I notice that nbhs sometimes posts from a different mailing list (like the post directly before this one), but I can't find an easy way to navigate that list. Mailing lists aren't really my thing though, so that could be why. The KVM irc on freenode is pretty quiet most of the time, but I usually lurk there aswell.

...

Have been busy moving to a new appartment, so sorry for the late reply.

About the Linux Mint forum thread on VGA passthrough: I'm not an expert in Linux/virtualization/etc., but I usually get things done somehow. My favorite distro is Linux Mint and since there wasn't any VGA passthrough guide for Linux Mint, I simply wrote one. If you're looking for the latest developments in Xen, then that thread will probably not help you much. It doesn't cover KVM at all, at least not now. I hope to be able to update it for LM15 and Xen 4.2 with xl toolstack.
My aim was to get VGA passthrough working without the need for kernel/Xen patches, etc., just using standard packages. I definitely chose the hardware to support the software (Xen), not vice versa.

This thread here is my main source for info on KVM, and for getting a general perspective on KVM versus Xen. But I haven't tried KVM, yet. One reason for not trying KVM in the first place was the lack of documentation on the Internet, but nbhs is doing a good job here. Today I would definitely give KVM a try.

I second the advice of nbhs about choosing the right video card!

Last edited by powerhouse (2013-06-27 07:24:23)

Offline

#183 2013-06-27 08:37:41

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

On page 6 i posted some builds for arch that contains the patches

I have gentoo + custom kernel build, then I need a patch more than all.

Offline

#184 2013-06-27 10:01:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:
nbhs wrote:

On page 6 i posted some builds for arch that contains the patches

I have gentoo + custom kernel build, then I need a patch more than all.

Yeah i know, but the zip packages contains the needed patches

Offline

#185 2013-06-27 10:17:17

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I prefer do my self the patching, but well, you speak well about: message of 2013-06-21 00:17:12, no? I can't dl: Error 102 (net::ERR_CONNECTION_REFUSED): The server refused the connection.

Offline

#186 2013-06-27 10:26:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:

I prefer do my self the patching, but well, you speak well about: message of 2013-06-21 00:17:12, no? I can't dl: Error 102 (net::ERR_CONNECTION_REFUSED): The server refused the connection.

kernel and qemu patches here: http://www.fileswap.com/dl/LFDgdLJbyj/

Offline

#187 2013-06-27 11:03:47

dogblue
Member
Registered: 2013-06-23
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

NIce! have you tried the patched nvidia drivers yet?

Yep, no trouble there at all actually, or at least no trouble that wasn't due to my own unfamiliarity with Arch :)

I have a couple of minor issues on the windows side (BSOD installing certain drivers), but nothing that can't be resolved, I think.
I can also confirm that the Etron EJ168 USB controller mentioned earlier in this thread is a pain to get going in Win 7 (had to test-sign drivers, etc), but it worked out of the box for me in Win 8.

I'm going to get the rest of these drivers installed then work my way through the optimizations that you posted on the first page. I'll report back with results and/or any more questions.

Offline

#188 2013-06-27 11:36:22

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On linux 3.10 RC7, I have to do manual patching, after I have the error:

drivers/pci/pci.c:3261:2: error: implicit declaration of function 'pci_hp_reset_slot' [-Werror=implicit-function-declaration]

Then I have put into the header:
int pci_hp_reset_slot(struct hotplug_slot *hotplug, int probe);
After it:

*drivers/vfio/pci/vfio_pci.c:85:10: error: 'irq_type' undeclared (first use in this function)

unknow resolution, can you help me?

*drivers/vfio/pci/vfio_pci.c:268:3: error: duplicate (or overlapping) case value *drivers/vfio/pci/vfio_pci.c:257:3: error: this is the first entry overlapping that value

Fixed the switch.

*drivers/vfio/pci/vfio_pci.c:416:18: error: 'VFIO_DEVICE_PCI_BUS_RESET' undeclared (first use in this function)

resolved by define into the header.
Do you think release patch for 3.10 final?
After I will try patch qemu.

Last edited by alpha_one_x86 (2013-06-27 11:36:54)

Offline

#189 2013-06-27 18:13:40

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:

On linux 3.10 RC7, I have to do manual patching, after I have the error:

drivers/pci/pci.c:3261:2: error: implicit declaration of function 'pci_hp_reset_slot' [-Werror=implicit-function-declaration]

Then I have put into the header:
int pci_hp_reset_slot(struct hotplug_slot *hotplug, int probe);
After it:

*drivers/vfio/pci/vfio_pci.c:85:10: error: 'irq_type' undeclared (first use in this function)

unknow resolution, can you help me?

*drivers/vfio/pci/vfio_pci.c:268:3: error: duplicate (or overlapping) case value *drivers/vfio/pci/vfio_pci.c:257:3: error: this is the first entry overlapping that value

Fixed the switch.

*drivers/vfio/pci/vfio_pci.c:416:18: error: 'VFIO_DEVICE_PCI_BUS_RESET' undeclared (first use in this function)

resolved by define into the header.
Do you think release patch for 3.10 final?
After I will try patch qemu.

You need to build pcie hotplug static instead of as a module, i suggest you use kernel 3.9

Last edited by nbhs (2013-06-27 18:30:56)

Offline

#190 2013-06-27 19:07:38

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need the kernel 3.10 to my haswell.
PCIe hotplug already in static.

Offline

#191 2013-06-27 19:09:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For 3.10-rc7 try this patch kernel-vfio-vga-reset.patch.tar.gz

Offline

#192 2013-06-28 08:28:10

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well same, under windows (with kernel + qemu patched):
http://files.first-world.info/temp/ati-crash.png
And in console:

qemu-system-x86_64: virtio_ioport_write: unexpected address 0x13 value 0x1

Offline

#193 2013-06-28 16:09:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:

Well same, under windows (with kernel + qemu patched):
http://files.first-world.info/temp/ati-crash.png
And in console:

qemu-system-x86_64: virtio_ioport_write: unexpected address 0x13 value 0x1

Could you please post your qemu cmdline

Offline

#194 2013-06-28 22:00:48

perfectsine
Member
Registered: 2013-06-28
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs,

Do foresee any problems with doing multiple VM's and passing a different VGA adapter to each?

-perfectsine

Offline

#195 2013-06-29 02:56:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

perfectsine wrote:

nbhs,

Do foresee any problems with doing multiple VM's and passing a different VGA adapter to each?

-perfectsine

I havent tried since i only have 2 cards ( and my brother's gtx 470) but i dont think its a problem if you got it working with one

Last edited by nbhs (2013-06-29 02:58:09)

Offline

#196 2013-06-30 11:55:25

Chais
Member
Registered: 2011-08-08
Posts: 28

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot for this huge tutorial. I can't say I have fully understood it. I'm quite new to the deeper magic of virtualisation.
My idea is build a gaming system that can run a virtual Windows for those games unavailable for linux.
I'd like to do a primary vga passthrough, as I don't want to buy two potent graphics cards. Since only host or guest will be needing it ata a time that shouldn't be a problem. Just I have no idea how to do it.
If that's impossible for some reason, plan B would be to have have an Intel igp as primary vga for Arch, pass the nvidia through to Windows or, if the VM isn't running, use it for the host with bumblebee. How would I do that?

Offline

#197 2013-07-01 19:07:45

alpha_one_x86
Member
Registered: 2013-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/usr/bin/qemu-system-x86_64 -drive file=hdd.img,if=virtio,cache=none,aio=native -k fr -alt-grab -m 512 -vga vmware -net nic,vlan=0,model=virtio -net user -rtc base=localtime -smp 4,cores=4,sockets=1 -boot once=c -cpu kvm64,+nx -enable-kvm -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on

With x-vga=on I have artifact on my second screen of my dual screen (and black screen for qemu), with it do the BsD.

Offline

#198 2013-07-02 03:26:33

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alpha_one_x86 wrote:
/usr/bin/qemu-system-x86_64 -drive file=hdd.img,if=virtio,cache=none,aio=native -k fr -alt-grab -m 512 -vga vmware -net nic,vlan=0,model=virtio -net user -rtc base=localtime -smp 4,cores=4,sockets=1 -boot once=c -cpu kvm64,+nx -enable-kvm -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on

With x-vga=on I have artifact on my second screen of my dual screen (and black screen for qemu), with it do the BsD.

You should probably send the redhat dev a bug report (alex williamson), see: http://lists.nongnu.org/archive/html/qe … 00432.html to submit a bug report

Offline

#199 2013-07-03 13:23:51

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm going to try this with my Gigabyte 990FXA-UD5, a Nvidia GT-430 (host), and some older Geforce 7xxx which currently rotting in the cellar. (If I can use it for that: great)
The display panel has two inputs, so I'm planning to switch video between host and guest by simply pressing the source button.

But I wonder if it's possible to use a single keyboard and mouse for both host and guest? Not at the same time of course, more like in usual KVM VNC window.
And if so, how's the handling of the focus (vm or host)?
Or would it be better to get a separate PCIe USB controller and use a KVM switch?

Sorry for the maybe stupid question.

Offline

#200 2013-07-03 13:35:00

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

But I wonder if it's possible to use a single keyboard and mouse for both host and guest? Not at the same time of course, more like in usual KVM VNC window.
And if so, how's the handling of the focus (vm or host)?
Or would it be better to get a separate PCIe USB controller and use a KVM switch?

Hi,
I use a setup with KVM switch and it's not that complicated, assuming you have multiple usb controllers on your mainboard (are there boards that don't?). You simply have to pass one through to the guest as a PCI device. The host will ignore the input on those, since it doesn't load a driver for it and it'll appear as a normal usb controller to the guest.
I don't thik I get what you mean by "usual KVM VNC window", so no idea about that one.
I hope this helps…


i'm sorry for my poor english wirting skills…

Offline

#201 2013-07-03 14:04:15

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Hi,
I use a setup with KVM switch and it's not that complicated, assuming you have multiple usb controllers on your mainboard (are there boards that don't?). You simply have to pass one through to the guest as a PCI device. The host will ignore the input on those, since it doesn't load a driver for it and it'll appear as a normal usb controller to the guest.
I don't thik I get what you mean by "usual KVM VNC window", so no idea about that one.
I hope this helps…

With the KVM VNC window stuff I mean the classical way of having a window with the VM running in it, and switching the input between host and guest by moving the mouse over the window border, or pressing ctrl+alt - something like that.
But forget about it. A KVM switch is the way to go I guess.

Your idea of picking one of the motherboard's USB controllers is great, just a matter of finding out if the 8 ports on the backpanel all use the same controller or multiple
Thanks!

Offline

#202 2013-07-03 14:14:00

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
andy123 wrote:

Hi,
I use a setup with KVM switch and it's not that complicated, assuming you have multiple usb controllers on your mainboard (are there boards that don't?). You simply have to pass one through to the guest as a PCI device. The host will ignore the input on those, since it doesn't load a driver for it and it'll appear as a normal usb controller to the guest.
I don't thik I get what you mean by "usual KVM VNC window", so no idea about that one.
I hope this helps…

With the KVM VNC window stuff I mean the classical way of having a window with the VM running in it, and switching the input between host and guest by moving the mouse over the window border, or pressing ctrl+alt - something like that.
But forget about it. A KVM switch is the way to go I guess.

Your idea of picking one of the motherboard's USB controllers is great, just a matter of finding out if the 8 ports on the backpanel all use the same controller or multiple
Thanks!

Ah, now I get it. I did that first, trying to get this set up. I still had the emulated graphics card enabled, but the other one takes over at some point and there is this black window you can click into and use ctrl+alt to get out of. I can't really tell what's more "elegant" or better in general. I'm not a qemu expert, so this may be possible without an emulated graphics card.
KVM switches aren't cheap though and I know I wouldn't have bought one for this.


i'm sorry for my poor english wirting skills…

Offline

#203 2013-07-04 00:37:14

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alphaniner wrote:

@rabcor

You still have a rogue space after the first -device vfio-pci.

# qemu-system-x86_64 ... -device vfio-pci, host=01:00.0,...

I put a space in a working command and got a "could not open disk image" error, so that could be it.

i'm  pretty sure this should be correct, but it doesn't seem to be starting up sad

qemu-system-x86_64 -enable-kvm -M q35 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

tidied up version

qemu-system-x86_64 -enable-kvm -M q35 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

This stuff is driving me nuts!

it gives this

qemu-system-x86_64: -cpu host: could not open disk image -smp: No such file or directory

if i move the backslashes (\) it gives me different results, for example "-cpu host \-smp 6..."

qemu-system-x86_64: -smp 6,sockets=1,cores=6,threads=1: could not open disk image -bios: No such file or directory

i tried messing around with it a bit but i couldn't find any way to make it work.

I took a break from this for a while, but my windows installation on the dual boot got wrecked by some random virus which apparently infected me on a day i didn't even use the computer. So i wanna see how far i get with this before i go into reinstalling windows, i'd like to get rid of the dual boot anyways. I'm tired now but tomorrow if nobody has any suggestions for fixing the code i will try reinstalling qemu and seabios. (speaking of which what happened to the qemu source from the OP?)

Last edited by rabcor (2013-07-04 01:15:58)

Offline

#204 2013-07-04 03:11:44

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
alphaniner wrote:

@rabcor

You still have a rogue space after the first -device vfio-pci.

# qemu-system-x86_64 ... -device vfio-pci, host=01:00.0,...

I put a space in a working command and got a "could not open disk image" error, so that could be it.

i'm  pretty sure this should be correct, but it doesn't seem to be starting up sad

qemu-system-x86_64 -enable-kvm -M q35 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

tidied up version

qemu-system-x86_64 -enable-kvm -M q35 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

This stuff is driving me nuts!

it gives this

qemu-system-x86_64: -cpu host: could not open disk image -smp: No such file or directory

if i move the backslashes (\) it gives me different results, for example "-cpu host \-smp 6..."

qemu-system-x86_64: -smp 6,sockets=1,cores=6,threads=1: could not open disk image -bios: No such file or directory

i tried messing around with it a bit but i couldn't find any way to make it work.

I took a break from this for a while, but my windows installation on the dual boot got wrecked by some random virus which apparently infected me on a day i didn't even use the computer. So i wanna see how far i get with this before i go into reinstalling windows, i'd like to get rid of the dual boot anyways. I'm tired now but tomorrow if nobody has any suggestions for fixing the code i will try reinstalling qemu and seabios. (speaking of which what happened to the qemu source from the OP?)

Try:

-cpu qemu64

and remove the -smp line

If that worked you might have built qemu without kvm support

Last edited by nbhs (2013-07-04 03:20:39)

Offline

#205 2013-07-04 06:58:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
andy123 wrote:

Hi,
I use a setup with KVM switch and it's not that complicated, assuming you have multiple usb controllers on your mainboard (are there boards that don't?). You simply have to pass one through to the guest as a PCI device. The host will ignore the input on those, since it doesn't load a driver for it and it'll appear as a normal usb controller to the guest.
I don't thik I get what you mean by "usual KVM VNC window", so no idea about that one.
I hope this helps…

With the KVM VNC window stuff I mean the classical way of having a window with the VM running in it, and switching the input between host and guest by moving the mouse over the window border, or pressing ctrl+alt - something like that.
But forget about it. A KVM switch is the way to go I guess.

Your idea of picking one of the motherboard's USB controllers is great, just a matter of finding out if the 8 ports on the backpanel all use the same controller or multiple
Thanks!

I think its still possible to capture the mouse and keyboard on the qemu window, i havent tried though, i do recommend a kvm switch, i dont have one unfortunately, i do have 4 frontal usb ports, of which 2 are usb3, so i just switch my keyboard and mouse by changing from usb2 (host) to usb3(guest)

Last edited by nbhs (2013-07-04 07:00:10)

Offline

#206 2013-07-05 02:37:28

apocolypse600
Member
Registered: 2013-05-26
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

I'm going to try this with my Gigabyte 990FXA-UD5, a Nvidia GT-430 (host), and some older Geforce 7xxx which currently rotting in the cellar. (If I can use it for that: great)
The display panel has two inputs, so I'm planning to switch video between host and guest by simply pressing the source button.

But I wonder if it's possible to use a single keyboard and mouse for both host and guest? Not at the same time of course, more like in usual KVM VNC window.
And if so, how's the handling of the focus (vm or host)?
Or would it be better to get a separate PCIe USB controller and use a KVM switch?

Sorry for the maybe stupid question.

Another option is to use Synergy. I'm using it at the moment to share a keyboard and mouse between a desktop and laptop and it works perfectly. A few people have reported using it for sharing the keyboard and mouse between host and guest in a setup like this aswell. It's in the arch repositories under 'synergy'. It supports clipboard sharing and allows you to lock focus to a computer with a keypress, so as far as I can tell, it should work pretty well (it's what I plan to use when I eventually try to set this all up).

Offline

#207 2013-07-05 13:18:27

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Try:

-cpu qemu64

and remove the -smp line

If that worked you might have built qemu without kvm support

#qemu-system-x86_64 -enable-kvm -M q35 -cpu qemu64 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -cpu qemu64: could not open disk image -bios: No such file or directory

I'm looking into reinstalling qemu but like i said, you removed the link from your original post and i don't have any different source to build from than my current one except the git one, which is v 1.6 and has seabios intergrated, and i don't know how to patch the intergrated seabios, i couldn't find out how (but otherwise that is what i would do).

i tried removing the backslashes and the VM will start if i run it from X.

qemu-system-x86_64 -enable-kvm -M q35 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

if not running from x (terminal) then i get

Could not initialize SDL(No available video device) - exiting

instead of

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

Last edited by rabcor (2013-07-05 13:28:33)

Offline

#208 2013-07-05 22:12:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
nbhs wrote:

Try:

-cpu qemu64

and remove the -smp line

If that worked you might have built qemu without kvm support

#qemu-system-x86_64 -enable-kvm -M q35 -cpu qemu64 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -cpu qemu64: could not open disk image -bios: No such file or directory

I'm looking into reinstalling qemu but like i said, you removed the link from your original post and i don't have any different source to build from than my current one except the git one, which is v 1.6 and has seabios intergrated, and i don't know how to patch the intergrated seabios, i couldn't find out how (but otherwise that is what i would do).

i tried removing the backslashes and the VM will start if i run it from X.

qemu-system-x86_64 -enable-kvm -M q35 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

if not running from x (terminal) then i get

Could not initialize SDL(No available video device) - exiting

instead of

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

Qemu by default creates a sdl window, if you want to lauch qemu without X launch qemu with this option

-nographic

Are you using a patched kernel with vfio-reset patches?

EDIT: i re-uploaded qemu and linux-mainline builds on the main page

qemu 1.5.1 http://www.fileswap.com/dl/wzFxm2ltkJ
linux-mainline 3.10 http://www.fileswap.com/dl/9HpoqHLeBJ

Last edited by nbhs (2013-07-05 22:31:27)

Offline

#209 2013-07-06 13:18:23

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

the qemu link says the file is not available.

If i use the -nographic option in terminal it just gives me the same error as in X.

I'm trying to passthrough an nvidia card.

my kernel has these options

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
INTEL_IOMMU_DEFAULT_ON=y

Since i don't have radeon i shouldn't need the vga-reset patches right? i've got nvidia GTX 600 series card.

Offline

#210 2013-07-06 19:03:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

the qemu link says the file is not available.

If i use the -nographic option in terminal it just gives me the same error as in X.

I'm trying to passthrough an nvidia card.

my kernel has these options

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
INTEL_IOMMU_DEFAULT_ON=y

Since i don't have radeon i shouldn't need the vga-reset patches right? i've got nvidia GTX 600 series card.

Yeah sorry here's the qemu build: http://www.fileswap.com/dl/qDG531EyVy

In my experience using a 470gtx i didnt need the patches im not sure about the gtx 600

You should launch qemu like this:

qemu-system-x86_64 ... -vga none -nographic ...

Last edited by nbhs (2013-07-06 19:05:23)

Offline

#211 2013-07-07 11:08:14

Silencement
Member
From: France
Registered: 2012-08-06
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I'm currently using Xen 4.2 and a Windows 7 domU with VGA passthrough, and everything works perfectly. I saw your guide and thought I would try KVM.

I have installed Linux Mint (didn't really have the time to perform a full install of Arch), built kernel 3.10 with :

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

I also built qemu 1.5 and seabios 1.7.2 with the patch you provided, and added this to grub :

intel_iommu=on pci-stub.ids=1002:6738,1002:aa88 vfio_iommu_type1.allow_unsafe_interrupts=1

I can start a Windows VM but can't get VGA passthrough to work :

  • with -vga none, i have "qemu 1.5.0 monitor" in VNC and the VM does not start

  • with -vga cirrus or -vga std, and pci-assign, I have "the device cannot start" in W7 device manager

  • with vfio-pci, I have "the device cannot find enough free resources that it can use"

Here is how I start the VM :

qemu-system-x86_64 -enable-kvm \ -M q35 -nodefconfig -readconfig q35.cfg \ -m 4096 \ -cpu host -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/seabios/bios.bin \ -vga none \ -device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1,addr=0.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=ich9-pcie-port-1,addr=0.1 \ -device ahci,id=drives \ -drive id=hda,format=raw,file=/dev/LVM/test \ -device ide-hd,drive=hda

And my PCI devices :

# lspci | grep AMD 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [Radeon HD 6870] 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Barts HDMI Audio [Radeon HD 6800 Series]

Any help would be appreciated !

Also, I have a question : would it be possible to VGA passthrough the Intel integrated graphics ? I know it's partially possible with Xen (works fine in Windows domU but black screen in Linux domU), is it possible with KVM ?


Derpy Hooves is best pony.

Offline

#212 2013-07-09 10:57:17

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:
teekay wrote:

I'm going to try this with my Gigabyte 990FXA-UD5, a Nvidia GT-430 (host), and some older Geforce 7xxx which currently rotting in the cellar. (If I can use it for that: great)
The display panel has two inputs, so I'm planning to switch video between host and guest by simply pressing the source button.

But I wonder if it's possible to use a single keyboard and mouse for both host and guest? Not at the same time of course, more like in usual KVM VNC window.
And if so, how's the handling of the focus (vm or host)?
Or would it be better to get a separate PCIe USB controller and use a KVM switch?

Sorry for the maybe stupid question.

Another option is to use Synergy. I'm using it at the moment to share a keyboard and mouse between a desktop and laptop and it works perfectly. A few people have reported using it for sharing the keyboard and mouse between host and guest in a setup like this aswell. It's in the arch repositories under 'synergy'. It supports clipboard sharing and allows you to lock focus to a computer with a keypress, so as far as I can tell, it should work pretty well (it's what I plan to use when I eventually try to set this all up).

This looks interesting for multi-screen, indeed. Thanks!

nbhs wrote:

I think its still possible to capture the mouse and keyboard on the qemu window, i havent tried though, i do recommend a kvm switch, i dont have one unfortunately, i do have 4 frontal usb ports, of which 2 are usb3, so i just switch my keyboard and mouse by changing from usb2 (host) to usb3(guest)

I went for a KVM switch - an el cheapo Aten CS682. Only issue so far is the crappy Etron USB3 host controller on the 990FXA-UD5 - the driver causes BSOD in Windows when trying to install it. Works fine in linux, though.
Other than that, everything works great so far. Thanks a lot for the HowTo!

Offline

#213 2013-07-09 19:14:11

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
apocolypse600 wrote:
teekay wrote:

I'm going to try this with my Gigabyte 990FXA-UD5, a Nvidia GT-430 (host), and some older Geforce 7xxx which currently rotting in the cellar. (If I can use it for that: great)
The display panel has two inputs, so I'm planning to switch video between host and guest by simply pressing the source button.

But I wonder if it's possible to use a single keyboard and mouse for both host and guest? Not at the same time of course, more like in usual KVM VNC window.
And if so, how's the handling of the focus (vm or host)?
Or would it be better to get a separate PCIe USB controller and use a KVM switch?

Sorry for the maybe stupid question.

Another option is to use Synergy. I'm using it at the moment to share a keyboard and mouse between a desktop and laptop and it works perfectly. A few people have reported using it for sharing the keyboard and mouse between host and guest in a setup like this aswell. It's in the arch repositories under 'synergy'. It supports clipboard sharing and allows you to lock focus to a computer with a keypress, so as far as I can tell, it should work pretty well (it's what I plan to use when I eventually try to set this all up).

This looks interesting for multi-screen, indeed. Thanks!

nbhs wrote:

I think its still possible to capture the mouse and keyboard on the qemu window, i havent tried though, i do recommend a kvm switch, i dont have one unfortunately, i do have 4 frontal usb ports, of which 2 are usb3, so i just switch my keyboard and mouse by changing from usb2 (host) to usb3(guest)

I went for a KVM switch - an el cheapo Aten CS682. Only issue so far is the crappy Etron USB3 host controller on the 990FXA-UD5 - the driver causes BSOD in Windows when trying to install it. Works fine in linux, though.
Other than that, everything works great so far. Thanks a lot for the HowTo!

I think i read some post mentioning it working out of the box with windows 8

EDIT: do you mind if i ask what card did you got it working with?

Last edited by nbhs (2013-07-09 19:52:29)

Offline

#214 2013-07-09 19:21:01

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Silencement wrote:

Hi,

I'm currently using Xen 4.2 and a Windows 7 domU with VGA passthrough, and everything works perfectly. I saw your guide and thought I would try KVM.

I have installed Linux Mint (didn't really have the time to perform a full install of Arch), built kernel 3.10 with :

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

I also built qemu 1.5 and seabios 1.7.2 with the patch you provided, and added this to grub :

intel_iommu=on pci-stub.ids=1002:6738,1002:aa88 vfio_iommu_type1.allow_unsafe_interrupts=1

I can start a Windows VM but can't get VGA passthrough to work :

  • with -vga none, i have "qemu 1.5.0 monitor" in VNC and the VM does not start

  • with -vga cirrus or -vga std, and pci-assign, I have "the device cannot start" in W7 device manager

  • with vfio-pci, I have "the device cannot find enough free resources that it can use"

Here is how I start the VM :

qemu-system-x86_64 -enable-kvm \ -M q35 -nodefconfig -readconfig q35.cfg \ -m 4096 \ -cpu host -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/seabios/bios.bin \ -vga none \ -device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1,addr=0.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=ich9-pcie-port-1,addr=0.1 \ -device ahci,id=drives \ -drive id=hda,format=raw,file=/dev/LVM/test \ -device ide-hd,drive=hda

And my PCI devices :

# lspci | grep AMD 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [Radeon HD 6870] 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Barts HDMI Audio [Radeon HD 6800 Series]

Any help would be appreciated !

Also, I have a question : would it be possible to VGA passthrough the Intel integrated graphics ? I know it's partially possible with Xen (works fine in Windows domU but black screen in Linux domU), is it possible with KVM ?

Right now there seems to be some problems using this guide on intel boards, the redhat dev mentioned that he got it working on his intel setup, and some users on the mailing list also reported success but with the igp disabled, but for most people on this thread the only solution was doing a secondary passthrough, im confident this will get worked out eventually. You can do it with standard mint packages for it you dont need vfio and/or a patched kernel (for secondary ), and no, its not possible right now to passthrough the igp with kvm.

EDIT: here's the mailing list discussion http://lists.gnu.org/archive/html/qemu- … 01722.html, and this patch might be of use http://lists.gnu.org/archive/html/qemu- … 01722.html

Last edited by nbhs (2013-07-09 19:27:40)

Offline

#215 2013-07-09 20:50:52

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
teekay wrote:

I went for a KVM switch - an el cheapo Aten CS682. Only issue so far is the crappy Etron USB3 host controller on the 990FXA-UD5 - the driver causes BSOD in Windows when trying to install it. Works fine in linux, though.
Other than that, everything works great so far. Thanks a lot for the HowTo!

I think i read some post mentioning it working out of the box with windows 8

EDIT: do you mind if i ask what card did you got it working with?

Sure thing. It's a GT430 for the host, and a 8500GT for the guest. Had to use pci-stub so the host nvidia driver ignores it, and I'm using the patched nvidia-driver.

Last edited by teekay (2013-07-09 20:51:33)

Offline

#216 2013-07-09 22:38:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
nbhs wrote:
teekay wrote:

I went for a KVM switch - an el cheapo Aten CS682. Only issue so far is the crappy Etron USB3 host controller on the 990FXA-UD5 - the driver causes BSOD in Windows when trying to install it. Works fine in linux, though.
Other than that, everything works great so far. Thanks a lot for the HowTo!

I think i read some post mentioning it working out of the box with windows 8

EDIT: do you mind if i ask what card did you got it working with?

Sure thing. It's a GT430 for the host, and a 8500GT for the guest. Had to use pci-stub so the host nvidia driver ignores it, and I'm using the patched nvidia-driver.

Thanks a lot, its good to know a lot nvidia cards are working

Last edited by nbhs (2013-07-09 22:45:59)

Offline

#217 2013-07-10 05:36:19

Eldak
Member
Registered: 2013-07-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great writeup!  I've gotten most the way through this, but I'm running into a BSoD on Windows 7 (0xA5 -- The BIOS in this system is not fully ACPI compliant.)

I am using the latest SeaBIOS from the git repository, I've checked the "make menuconfig" and it does appear to be compiling this with ACPI support.  I have a Gigabyte 990XA-UD3 motherboard, which from what I can tell another use has gotten this to work.

=== EDIT ===
It looks like I was using the wrong SeaBIOS version (1.7.3 instead of 1.7.2 with the patch).  Windows 7 now attempts to install without throwing the ACPI Error.  However, during the Windows 7 installation, it finishes the "Copying Windows files" stage, but gets stuck on starting the "Expanding Windows files".   It does this using raw .img disk images and LVMs.  Any suggestions?

Last edited by Eldak (2013-07-10 06:55:52)

Offline

#218 2013-07-10 07:34:59

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eldak wrote:

=== EDIT ===
It looks like I was using the wrong SeaBIOS version (1.7.3 instead of 1.7.2 with the patch).  Windows 7 now attempts to install without throwing the ACPI Error.  However, during the Windows 7 installation, it finishes the "Copying Windows files" stage, but gets stuck on starting the "Expanding Windows files".   It does this using raw .img disk images and LVMs.  Any suggestions?

Are you using an IDE bus for the disk or the legacy AHCI SATA as explained by nbhs in the howto? The IDE bus is _dead slow_ when installing Windows. I haven't tried SATA, so I can't say if it's better there. (I did the install in VirtualBox which takes only some minutes, and used VBoxManage clonehd to convert the .vdi to .raw, then created a 2GB fake qcow2 image and attached both in Qemu, so the system boots from the cloned raw image with IDE, and detects the VirtIO SCSI bus with the fake image, then installed virtio drivers for it.)

Offline

#219 2013-07-10 15:00:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eldak wrote:

Great writeup!  I've gotten most the way through this, but I'm running into a BSoD on Windows 7 (0xA5 -- The BIOS in this system is not fully ACPI compliant.)

I am using the latest SeaBIOS from the git repository, I've checked the "make menuconfig" and it does appear to be compiling this with ACPI support.  I have a Gigabyte 990XA-UD3 motherboard, which from what I can tell another use has gotten this to work.

I get BSOD'S too using the lastest seabios from git with windows 7, this does not happen on windows 8, but that's why i recommend using the stable 1.7.2 branch + the patch i provided.


Eldak wrote:

=== EDIT ===
It looks like I was using the wrong SeaBIOS version (1.7.3 instead of 1.7.2 with the patch).  Windows 7 now attempts to install without throwing the ACPI Error.  However, during the Windows 7 installation, it finishes the "Copying Windows files" stage, but gets stuck on starting the "Expanding Windows files".   It does this using raw .img disk images and LVMs.  Any suggestions?

Perhaps the image is corrupt?

Offline

#220 2013-07-10 19:50:05

Eldak
Member
Registered: 2013-07-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed Windows 7 to an LVM using QEMU/virt-manager, then tried to start it up using the command given on page 1 of this thread to start the VM, enable the passthrough, etc, and then Windows blue screens.  I am able to boot the LVM up from a normal libvirt session, so the image is not corrupt.

Offline

#221 2013-07-10 22:28:28

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eldak wrote:

I installed Windows 7 to an LVM using QEMU/virt-manager, then tried to start it up using the command given on page 1 of this thread to start the VM, enable the passthrough, etc, and then Windows blue screens.  I am able to boot the LVM up from a normal libvirt session, so the image is not corrupt.

Could you provide:

-whats your setup
-what card are you trying to passthrough
-which version of seabios, qemu, kernel are you using
-cmdline to launch qemu
-bsod info

Offline

#222 2013-07-11 03:38:54

Eldak
Member
Registered: 2013-07-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BSOD 0x7B
SeaBIOS: 1.7.2 + Patch provided here
QEMU 1.5.1
Kernel 3.10
ATI Radeon HD6870

Motherboard is a Gigabyte GA-990XA-UD3 with BIOS F13.

Everything works when I attach the LVM to a VM within virt-manager, but BSOD's when starting with the following command line:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/patrick/seabios-1.7.2/seabios-1.7.2/out/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/media/patrick/Storage/data/Operating\ Systems/Windows\ 7/Win7Pro.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd -drive file=/dev/mapper/kvmspace-win7,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -boot menu=on &

Looking up the BSOD, It's referencing invalid boot device:  http://msdn.microsoft.com/en-us/library … s.85).aspx

The passthrough, etc, is working as intended as everything is displaying on my second video card that's being passed through.  I should also that, that with this configuration, command, etc, when I create a "new" win7 LVM and attempt to install Windows 7, the installation freezes before it starts to extract windows files.

Last edited by Eldak (2013-07-11 03:59:54)

Offline

#223 2013-07-11 04:39:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eldak wrote:

BSOD 0x7B
SeaBIOS: 1.7.2 + Patch provided here
QEMU 1.5.1
Kernel 3.10
ATI Radeon HD6870

Motherboard is a Gigabyte GA-990XA-UD3 with BIOS F13.

Everything works when I attach the LVM to a VM within virt-manager, but BSOD's when starting with the following command line:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/patrick/seabios-1.7.2/seabios-1.7.2/out/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/media/patrick/Storage/data/Operating\ Systems/Windows\ 7/Win7Pro.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd -drive file=/dev/mapper/kvmspace-win7,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -boot menu=on &

Looking up the BSOD, It's referencing invalid boot device:  http://msdn.microsoft.com/en-us/library … s.85).aspx

The passthrough, etc, is working as intended as everything is displaying on my second video card that's being passed through.  I should also that, that with this configuration, command, etc, when I create a "new" win7 LVM and attempt to install Windows 7, the installation freezes before it starts to extract windows files.

Did you install windows with virt manager using an ahci or ide controller? if you installed windows with the ide controller (which i believe virt-manager uses by default) you'll need to change some registry keys to make it boot on ahci, see this post: http://www.neowin.net/forum/topic/98737 … g-windows/

Also, If you're using vanilla kernel + qemu you might experience crashes, i posted some solutions on the guide, the most relieable one is using a patched kernel + patched qemu

Last edited by nbhs (2013-07-11 04:45:25)

Offline

#224 2013-07-11 05:25:36

Eldak
Member
Registered: 2013-07-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Did you install windows with virt manager using an ahci or ide controller? if you installed windows with the ide controller (which i believe virt-manager uses by default) you'll need to change some registry keys to make it boot on ahci, see this post: http://www.neowin.net/forum/topic/98737 … g-windows/

Also, If you're using vanilla kernel + qemu you might experience crashes, i posted some solutions on the guide, the most relieable one is using a patched kernel + patched qemu

That helped it get past the BSoD, Thanks!  Now, it's acting like it's freezing up, but it's not frozen.  The Starting Windows screen, with the fading in/out brightness to the Windows logo, just sitting their.  I'll do some more research on this tomorrow.

Offline

#225 2013-07-11 10:16:24

naythuwin
Member
Registered: 2013-07-11
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[abcd@unknow ~]$ qemu-system-x86_64 -enable-kvm -m 2048 -cpu host -smp 2 -device vfio-pci,host=05:00.0,x-vga=on -bios /usr/share/qemu/bios.bin -vga none
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=0000a19e EDX=000003d4
ESI=00009e48 EDI=0000823c EBP=0000c58a ESP=00008232
EIP=0009ffff EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =0000 00000000 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000fd808 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

cpu : intel core i7
mobo : foxconn h61ap
kernel : Linux abcdef 3.10.0-1-mainline #1 SMP PREEMPT Wed Jul 10 23:38:07 MMT 2013 x86_64 GNU/Linux
graphic : onboard intel and pny geforce 8400gs pci 512M low profile

lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
05:00.0 VGA compatible controller: NVIDIA Corporation G98 [GeForce 8400 GS Rev. 2] (rev a1)

secondery passthrough is work for me but primary is not
help me!

Last edited by naythuwin (2013-07-11 10:24:57)

Offline

#226 2013-07-11 20:00:35

argon
Member
Registered: 2013-07-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello and thank you for your great tutorial.

I did everything I needed to do (what you mentioned), dmesg does not show any errors, however my second monitor
is not turning on.

I was trying two approaches: at first I had assigned my second GPU (nvidia 7600GT) to a machine, and then installed windows7.
After that I did everything to make this hardware work - installation of drivers, but in hardware manager in windows, there is
an info: "bla bla does not work, because it has problems" or something similar. I thought, maybe using command line I'd be more
lucky, and I tried with

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -nographic -boot c -drive file=/var/lib/libvirt/images/Windows7.img,if=virtio

Windows is starting (I hear how HDD works), but my screen is still dead. I haven't tried any qemu/kernel patching yet.
Do you have any ideas what should I do?

EDIT: I actually managed to make this work. I'll describe my case later.

Last edited by argon (2013-07-13 13:35:49)

Offline

#227 2013-07-13 16:30:17

argon
Member
Registered: 2013-07-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, I'll try to be quick here. What helped me was in fact pointing out to qemu where seabios has its
roms. After I've compiled seabios given on the first page, I added

-L /path/to/seabios/out/ -L /usr/share/qemu/bios.bin

to the command line and my GPU responded and now I am able to use it without any problems.
However I'm not sure whether mobo BIOS update I did, was helpful - probably.

I also have to say, I am also using patched linux-mainline and qemu.
My hardware:
MB: ASUS M4A89TD PRO USB3.0
Host GPU: ASUS NVIDIA GTX460
Guest GPU: GIGABYTE NVIDIA 7600GT

Last edited by argon (2013-07-13 16:33:54)

Offline

#228 2013-07-13 17:38:37

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things smile.

Hello,

Could you please describe how did you get vbios and how you specified vbios for qemu?

Thank you in advance!

Offline

#229 2013-07-13 18:58:28

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

argon wrote:

Ok, I'll try to be quick here. What helped me was in fact pointing out to qemu where seabios has its
roms. After I've compiled seabios given on the first page, I added

-L /path/to/seabios/out/ -L /usr/share/qemu/bios.bin

to the command line and my GPU responded and now I am able to use it without any problems.
However I'm not sure whether mobo BIOS update I did, was helpful - probably.

I also have to say, I am also using patched linux-mainline and qemu.
My hardware:
MB: ASUS M4A89TD PRO USB3.0
Host GPU: ASUS NVIDIA GTX460
Guest GPU: GIGABYTE NVIDIA 7600GT

Sorry for the late reply i was kinda busy working, so you're saying it didnt work with the -bios command? my seabios package should install it to /usr/share/qemu/bios.bin by default

Offline

#230 2013-07-13 19:00:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

naythuwin wrote:

[abcd@unknow ~]$ qemu-system-x86_64 -enable-kvm -m 2048 -cpu host -smp 2 -device vfio-pci,host=05:00.0,x-vga=on -bios /usr/share/qemu/bios.bin -vga none
KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=0000a19e EDX=000003d4
ESI=00009e48 EDI=0000823c EBP=0000c58a ESP=00008232
EIP=0009ffff EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =0000 00000000 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000fd808 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

cpu : intel core i7
mobo : foxconn h61ap
kernel : Linux abcdef 3.10.0-1-mainline #1 SMP PREEMPT Wed Jul 10 23:38:07 MMT 2013 x86_64 GNU/Linux
graphic : onboard intel and pny geforce 8400gs pci 512M low profile

lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
05:00.0 VGA compatible controller: NVIDIA Corporation G98 [GeForce 8400 GS Rev. 2] (rev a1)

secondery passthrough is work for me but primary is not
help me!

Does this only happens when using vfio-pci? You should also use the q35 model and put your gpu behind a root port like the redhat dev recommended

Offline

#231 2013-07-13 19:03:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
apoapo wrote:

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things smile.

Hello,

Could you please describe how did you get vbios and how you specified vbios for qemu?

Thank you in advance!

myweb here's the discussion of some user that got it working with an intel board: http://lists.gnu.org/archive/html/qemu- … 01722.html, also like i mentioned before this patch might be of use http://lists.gnu.org/archive/html/qemu- … 01722.html you could also try doing a bug report on qemu-devel or kvm-devel

Last edited by nbhs (2013-07-13 19:08:17)

Offline

#232 2013-07-13 20:22:30

argon
Member
Registered: 2013-07-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
argon wrote:

Ok, I'll try to be quick here. What helped me was in fact pointing out to qemu where seabios has its
roms. After I've compiled seabios given on the first page, I added

-L /path/to/seabios/out/ -L /usr/share/qemu/bios.bin

to the command line and my GPU responded and now I am able to use it without any problems.
However I'm not sure whether mobo BIOS update I did, was helpful - probably.

I also have to say, I am also using patched linux-mainline and qemu.
My hardware:
MB: ASUS M4A89TD PRO USB3.0
Host GPU: ASUS NVIDIA GTX460
Guest GPU: GIGABYTE NVIDIA 7600GT

Sorry for the late reply i was kinda busy working, so you're saying it didnt work with the -bios command? my seabios package should install it to /usr/share/qemu/bios.bin by default

No problem, I really enjoyed digging in the issue by myself smile. And no, it didn't work.
But now, I realized that somehow md5 of those two bios.bin files were different. So maybe
I haven't in fact installed your package but I was only going to...

Now md5 sums are correct, and it should work.

Last edited by argon (2013-07-13 20:23:22)

Offline

#233 2013-07-13 20:31:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

argon wrote:
nbhs wrote:
argon wrote:

Ok, I'll try to be quick here. What helped me was in fact pointing out to qemu where seabios has its
roms. After I've compiled seabios given on the first page, I added

-L /path/to/seabios/out/ -L /usr/share/qemu/bios.bin

to the command line and my GPU responded and now I am able to use it without any problems.
However I'm not sure whether mobo BIOS update I did, was helpful - probably.

I also have to say, I am also using patched linux-mainline and qemu.
My hardware:
MB: ASUS M4A89TD PRO USB3.0
Host GPU: ASUS NVIDIA GTX460
Guest GPU: GIGABYTE NVIDIA 7600GT

Sorry for the late reply i was kinda busy working, so you're saying it didnt work with the -bios command? my seabios package should install it to /usr/share/qemu/bios.bin by default

No problem, I really enjoyed digging in the issue by myself smile. And no, it didn't work.
But now, I realized that somehow md5 of those two bios.bin files were different. So maybe
I haven't in fact installed your package but I was only going to...

Now md5 sums are correct, and it should work.

Im actually surprised you got that card working gratz.

Offline

#234 2013-07-16 06:29:59

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone and thanks for this tutorial, exiting stuff.

Do you guys know if there's any chance on getting this to work on a Mac Pro (2009). I guess
after a firmware update VT-d support should be present. I have OS X and Arch dual boot setup
but for some reason the /sys/bus/pci/drivers/vfio-pci folder doesn't exist. No iommu_group links
in the graphics card folders either.

I have a GeForce 9500 GT as a primary and I was hoping to passthrough a Radeon 5400 series
device.

uname -r gives me 3.9.9-1-ARCH which I guess should have the necessary options on.

I can post whatever additional info you need, thanks.

Offline

#235 2013-07-16 09:33:54

resr
Member
Registered: 2013-07-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri

I had a similar experience which I'm presently working through. I can't speak for mac, but I had the same symptoms with my new arch system.

I added

intel_iommu=on

to my kernel parameters, which gives me "[    0.000000] Intel-IOMMU: enabled" from a

dmesg | grep IOMMU

.

But I wasn't getting any iommu_group links under /sys/bus/pci/devices/<device>, the same as yourself.

My /sys/kernel/iommu_groups/ folder is empty too.

I'm about to rebuild the kernel, making sure that the

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

options are there.

And as per rabcor's experience halfway through the thread I'm going to try CONFIG_INTEL_IOMMU_DEFAULT_ON=y as well.

I'm a little bit different from the thread instructions as I'm only trying to pass a firewire card through, so I've skipped the patched seabios step.

I have things in my /sys/bus/pci/drivers/vfio-pci folder, but I've recompiled once.

Offline

#236 2013-07-16 10:22:05

resr
Member
Registered: 2013-07-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Don't mind me. I've found that VT-d isn't a feature of my 3770K cpu.

Bit silly paying the extra for the K as I haven't overclocked it yet.

Offline

#237 2013-07-16 11:01:47

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok I've made some progress. I set the intel_iommu=on and vfio_iommu_type1.allow_unsafe_interrupts=1 in grub and now I have the iommu_group folders. I also modprobed vfio-pci and the drivers are present also.

I can also bind a device without any (visible) error messages using the script. Testing with qemu starts a black screen (as it should) but no picture from the other graphics card.

I'd appreciate if someone can tell me how to troubleshoot the problem. I had the proprietary nvidia drivers for my host installed first, but I removed them and installed nouveau instead. No change I guess.

Qemu version: QEMU emulator version 1.5.1

Offline

#238 2013-07-16 15:43:57

argon
Member
Registered: 2013-07-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri:
Hm, I don't know, whether removing nvidia drivers was helpful. However, you need
kernel and qemu with pci-resetting support. You can find linux-mainline (3.10), qemu with
mentioned patches in this thread and you'll have to compile them and install. Also you'll need
to use patched seabios - again, link available in this thread, few pages back.

This is exactly what I did and what seem to work for me.

Last edited by argon (2013-07-16 15:45:30)

Offline

#239 2013-07-16 19:26:27

perfectsine
Member
Registered: 2013-06-28
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs:

Do you know if it would be possible to run a headless setup with one video card as "secondary" passthrough?   I have a server that I want to test this on but, it doesn't have a built-in graphics adapter.

Offline

#240 2013-07-16 19:32:46

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

perfectsine wrote:

nbhs:

Do you know if it would be possible to run a headless setup with one video card as "secondary" passthrough?   I have a server that I want to test this on but, it doesn't have a built-in graphics adapter.

Using this guide probably not, though if you have a radeon card you could pass it through as a secondary device to qemu that means both the emulated card and the radeon card will be present on the vm, note that im not sure if this is working anymore since ive been using vfio instead of "pci-assing" for a long time ( last time i tried it worked )

Offline

#241 2013-07-16 23:51:15

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri wrote:

Ok I've made some progress. I set the intel_iommu=on and vfio_iommu_type1.allow_unsafe_interrupts=1 in grub and now I have the iommu_group folders. I also modprobed vfio-pci and the drivers are present also.

I can also bind a device without any (visible) error messages using the script. Testing with qemu starts a black screen (as it should) but no picture from the other graphics card.

I'd appreciate if someone can tell me how to troubleshoot the problem. I had the proprietary nvidia drivers for my host installed first, but I removed them and installed nouveau instead. No change I guess.

Qemu version: QEMU emulator version 1.5.1

Could you try giving a shot using the patched builds?

Offline

#242 2013-07-17 10:40:59

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok I have now the patched mainline kernel, patched QEMU, patched Seabios and it .... works!

Thanks so much. I can give out my system specs later if someone else is trying to do this on a Mac Pro or with a similar setup.

Offline

#243 2013-07-17 13:53:16

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Congrats smile I´d like to see the specs, too. ´Trying to build a small list of components that work.

Offline

#244 2013-07-17 18:16:26

argon
Member
Registered: 2013-07-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Low-end NVIDIA 210 also works.

Offline

#245 2013-07-17 18:58:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri wrote:

Ok I have now the patched mainline kernel, patched QEMU, patched Seabios and it .... works!

Thanks so much. I can give out my system specs later if someone else is trying to do this on a Mac Pro or with a similar setup.

Awesome please post you specs! i assume you got it working on intel right (well duh its a mac!)? I also updated the qemu patch yesterday, it seems i forgot to include some late patches that contain some quirks for the vga's, so i recommend people who couldnt get it to work to try the lastest builds

Also you can try doing this with mac osx, here's a guide on how to get mac running on qemu: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/, unfortunately i couldnt get it to run on my amd system but it should work on intel, it would be sweet if you can get vga passthrough working with a mac install!

Last edited by nbhs (2013-07-17 19:19:43)

Offline

#246 2013-07-17 19:31:45

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Also you can try doing this with mac osx, here's a guide on how to get mac running on qemu: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/, unfortunately i couldnt get it to run on my amd system but it should work on intel, it would be sweet if you can get vga passthrough working with a mac install!

I do have a snow leopard VM with some AMD patched kernel which works fine in VMware also in VirtualBox (no guest tools, though). But I failed to get it running in Qemu - with VGA passthrough that would be so cool, indeed.

Offline

#247 2013-07-18 10:25:55

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

I would like to install Windows 8 without any VGA-Passthrough and faced with terrible performance issue I suggest with storage: in takes more than half a hour to pass from partition selection dialog to coping files. A long time coping percentage stay at 0%. Is it OK? Is it any way to improve performance? Live Ubuntu works well.

I use the following lines for storage:
-device ahci,bus=pcie.0,id=ahci
-drive file=/home/nbhs/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

Offline

#248 2013-07-18 13:40:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear All,

I would like to install Windows 8 without any VGA-Passthrough and faced with terrible performance issue I suggest with storage: in takes more than half a hour to pass from partition selection dialog to coping files. A long time coping percentage stay at 0%. Is it OK? Is it any way to improve performance? Live Ubuntu works well.

I use the following lines for storage:
-device ahci,bus=pcie.0,id=ahci
-drive file=/home/nbhs/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

Could you post the entire qemu cmdline? You could also try using the faster virtio blk

Last edited by nbhs (2013-07-18 13:41:13)

Offline

#249 2013-07-18 20:07:33

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -M q35 -nodefconfig -readconfig ~/new/qemu/src/qemu-1.5.1/docs/q35-chipset.cfg -m 2048  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/run/media/user/CopiedData/Soft/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/home/user/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

Offline

#250 2013-07-18 20:45:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

qemu-system-x86_64 -M q35 -nodefconfig -readconfig ~/new/qemu/src/qemu-1.5.1/docs/q35-chipset.cfg -m 2048  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/run/media/user/CopiedData/Soft/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/home/user/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

Have you tried to copy the iso image to your hard drive and install it from there?

Last edited by nbhs (2013-07-18 20:45:51)

Offline

#251 2013-07-18 21:22:16

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/run/media/user/CopiedData - is a hard drive with NTFS filesystem (/ is placed on SSD, soft archive, media, etc on usual hard drive)
I have tried vice versa: place installation image on SSD and windows to hard drive - 10 minutes passed and partitioning window is shown.
I also tried place windows to SSD and installation image to RAM disk - result is the same - freeze.

Offline

#252 2013-07-18 21:29:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

/run/media/user/CopiedData - is a hard drive with NTFS filesystem (/ is placed on SSD, soft archive, media, etc on usual hard drive)
I have tried vice versa: place installation image on SSD and windows to hard drive - 10 minutes passed and partitioning window is shown.
I also tried place windows to SSD and installation image to RAM disk - result is the same - freeze.

Does this also happen using the the ide or virtio controller instead of ahci?

Offline

#253 2013-07-18 21:34:16

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried to execute with virtio:

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/home/abrilevskiy/windows.img \
-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

windows installer does not recognize any hard drive

Offline

#254 2013-07-18 21:38:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

I tried to execute with virtio:

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/home/abrilevskiy/windows.img \
-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

windows installer does not recognize any hard drive

Try the ide controller, its on the guide, im really not sure whats causing your problem, i think another user had the same problem a few pages back, if that doesnt work try using the old chipset (remove the -M q35 part)

Last edited by nbhs (2013-07-18 21:39:47)

Offline

#255 2013-07-18 21:53:35

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried:
qemu-system-x86_64 -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot d \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd \
-drive file=/home/abrilevskiy/windows.img,if=ide,format=raw,index=2

but VM could not boot: boot drive is not found

Offline

#256 2013-07-18 22:11:33

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It looks like I mistaken with parameters.

qemu-system-x86_64 -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot d \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd  -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/abrilevskiy/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

but I receive: qemu-system-x86_64: -device ide-cd,bus=ide.1,drive=isocd: Property 'ide-cd.drive' can't take value 'isocd', it's in use

Offline

#257 2013-07-18 22:13:33

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

I tried:
qemu-system-x86_64 -m 4096  -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot d \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd \
-drive file=/home/abrilevskiy/windows.img,if=ide,format=raw,index=2

but VM could not boot: boot drive is not found

qemu-system-x86_64 -m 1024 -M q35 -enable-kvm -boot menu=on -cpu host -device piix4-ide,bus=pcie.0 -drive file=/home/nbhs/windows.img,id=windisk,format=raw -device ide-hd,bus=ide.0,drive=windisk -drive file=/home/nbhs/windows8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

works for me

EDIT:

qemu-system-x86_64 -m 1024 -enable-kvm -boot menu=on -cpu host -hda /home/nbhs/windows.img -cdrom windows8.iso

also works (-hda and -cdrom wont work using the q35 chipset), to use virtio i think all you need to do is add if=virtio at the end and it will automatically add the virtio controller too., tho you'll probably need the drivers, ex:

qemu-system-x86_64 -m 1024 -enable-kvm -boot menu=on -cpu host -drive file=/home/nbhs/windows.img,id=windisk,format=raw,if=virtio -drive file=/home/nbhs/windows8.iso,id=isocd,if=virtio

Last edited by nbhs (2013-07-18 22:26:56)

Offline

#258 2013-07-18 22:19:37

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -m 4096  -M q35 -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot menu=on \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd  -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/abrilevskiy/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

works for me, thank you!

Offline

#259 2013-07-18 22:25:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

qemu-system-x86_64 -m 4096  -M q35 -enable-kvm -cpu host  \
-smp 2,sockets=1,cores=2,threads=1 -vga std \
-L ~/tmp/src/seabios-1.7.3/out/ \
-device piix4-ide -boot menu=on \
-drive file=/home/abrilevskiy/ramdisk/9200.16384.120725-1247_x64frev_Enterprise_VL_HRM_CENA_X64FREV_EN-US_DVD.iso,id=isocd  -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/abrilevskiy/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

works for me, thank you!

No problem once you install it you might want to use virtio drivers, or the ahci controller, see my edited post above.

Offline

#260 2013-07-24 14:18:43

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Soon I will have the necessary equipment to do this, but aside from the VGA I'm interestend in passing through my PCIe audio device.
I've got an Asus Xonar Essence STX and the motherboard's integrated sound.
I'd like to use the latter for the host and a Windows VM, reserving the former for a specific Linux VM.
I'd connect my headphones to the Xonar.
Is such a configuration even possible, or I should just give up and connect my headphones to the internal audio (and that won't do)?

Offline

#261 2013-07-24 14:35:58

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Evonat wrote:

Soon I will have the necessary equipment to do this, but aside from the VGA I'm interestend in passing through my PCIe audio device.
I've got an Asus Xonar Essence STX and the motherboard's integrated sound.
I'd like to use the latter for the host and a Windows VM, reserving the former for a specific Linux VM.
I'd connect my headphones to the Xonar.
Is such a configuration even possible, or I should just give up and connect my headphones to the internal audio (and that won't do)?

Well it should be possible, i am able to passthrough my integrated sound card to a vm and it works fine

Offline

#262 2013-07-25 00:54:17

reacocard
Member
Registered: 2008-05-25
Posts: 20
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Evonat wrote:

Soon I will have the necessary equipment to do this, but aside from the VGA I'm interestend in passing through my PCIe audio device.
I've got an Asus Xonar Essence STX and the motherboard's integrated sound.
I'd like to use the latter for the host and a Windows VM, reserving the former for a specific Linux VM.
I'd connect my headphones to the Xonar.
Is such a configuration even possible, or I should just give up and connect my headphones to the internal audio (and that won't do)?

You can probably pass-through the second card just fine like any other pci device. As far as the headphones go, if you want audio from both cards I'd suggest either just combining the outputs by cables with something like this http://amzn.com/B0016LDZ36 or connecting the line-out of the integrated audio to the line-in on the Xonar. The latter method has the downside that audio from the integrated card becomes dependent on the proper operation of the VM with the Xonar however.

Offline

#263 2013-07-26 00:54:15

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Testing if its working out

Lets test if its working, as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

[...]

You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,
"-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" creates a pcie root port and we attach our gpu to it.
We must also use x-vga=on option on our gpu (and -vga none) otherwise it wont work


@nbhs:

This is a GREAT tutorial.  Thanks so much for creating it!

Because I'm not quite ready to invest the time into applying the suggested patches, I thought I'd test my luck by trying this tutorial on a daily build of Ubuntu 13.10 that included the following, but without any patches:

Mainline Kernel 3.11 RC2 qemu 1.5.0 seabios 1.7.3

And I added the following to /etc/modules: 

vfio vfio_iommu_type1 vfio_pci pci_stub kvm kvm_amd

THE GOOD NEWS:  Even without the patches or custom kernel, the above suggested test was successful.  That is, I saw the black qemu window on my main display, and seabios ouput on my monitor from my passthru'd card saying it cant find anything to boot.  So far, so good.

Now to try booting a VM:

nbhs wrote:

DISK:

Creating and passing a disk image to qemu:

By default -M q35 creates an emulated sata controller, but it seems its not working on windows yet, so we attach the old ahci controller:

-device ahci,bus=pcie.0,id=ahci

[...]

And attach a disk to it:

-drive file=/home/nbhs/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

I had previously created a Windows 7 HVM using Virt-Manager under KVM (but without vfio), and so I plugged into the above the appropriate path to the Windows 7 image.   Much to my delight, when I attempted to boot the HVM, the monitor from my passthru'd card displayed boot information after which the Windows 7 welcome screen appeared for a few seconds.   But just when I was about to celebrate.... BSOD.

So I'm wondering:

(1)  Am I mistaken to think that, barring other problems, the Windows 7 HVM that I previously created using Virt-Manager under KVM should boot?

(2) Provided that the Windows 7 HVM that I created as described above should boot, what is the likelihood that the recommended patches will cure the BSOD that I'm experiencing?  That is, were either of the patches created to address issues that arise late into the boot phase, like I'm experiencing?  Or were the patches created to address different issues?


UPDATE:  I managed to get this working.   Have a look HERE for details.

Last edited by GizmoChicken (2013-07-27 06:48:54)

Offline

#264 2013-07-26 01:23:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:

Testing if its working out

Lets test if its working, as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

[...]

You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,
"-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" creates a pcie root port and we attach our gpu to it.
We must also use x-vga=on option on our gpu (and -vga none) otherwise it wont work


@nbhs:

This is a GREAT tutorial.  Thanks so much for creating it!

Because I'm not quite ready to invest the time into applying the suggested patches, I thought I'd test my luck by trying this tutorial on a daily build of Ubuntu 13.10 that included the following, but without any patches:

Mainline Kernel 3.11 RC2 qemu 1.5.0 seabios 1.7.3

And I added the following to /etc/modules: 

vfio vfio_iommu_type1 vfio_pci pci_stub kvm kvm_amd

THE GOOD NEWS:  Even without the patches or custom kernel, the test was successful.  That is, I saw the black qemu window on my main display, and seabios ouput on my monitor from my passthru'd card saying it cant find anything to boot.  So far, so good.

Now to try booting a VM:

nbhs wrote:

DISK:

Creating and passing a disk image to qemu:

By default -M q35 creates an emulated sata controller, but it seems its not working on windows yet, so we attach the old ahci controller:

-device ahci,bus=pcie.0,id=ahci

[...]

And attach a disk to it:

-drive file=/home/nbhs/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

I had previously created a Windows 7 HVM using Virt-Manager under KVM (but without vfio), and so I plugged into the above the appropriate path to the Windows 7 image.   Much to my delight, when I attempted to boot the HVM, the monitor from my passthru'd card displayed boot information after which the Windows 7 welcome screen appeared for a few seconds.   But just when I was about to celebrate.... BSOD.

So I'm wondering:

(1)  Am I mistaken to think that, barring other problems, the Windows 7 HVM that I previously created using Virt-Manager under KVM should boot?

(2) Provided that the Windows 7 HVM that I created as described above should boot, what is the likelihood that the recommended patches will cure the BSOD that I'm experiencing?  That is, were either of the patches created to address issues that arise late into the boot phase, like I'm experiencing?  Or were the patches created to address different issues?

Thank you, would you mind mentioning your build?

1) It should
2) what bsod are you experiencing?, in my experience using these patches will solve most problems, since without them once you shutdown/reset the vm the gpu will stay in a bad state, and most people are having 0 problems with these patches

Offline

#265 2013-07-26 02:26:29

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Thank you, would you mind mentioning your build?

1) It should
2) what bsod are you experiencing?, in my experience using these patches will solve most problems, since without them once you shutdown/reset the vm the gpu will stay in a bad state, and most people are having 0 problems with these patches

@nbhs:

Thanks much for the quick reply!

As for the BSOD, it appears and disappears very quickly, and so unfortunately, I haven't had a chance to read the details.  But I can tell you that it always occurs just about when I would expect to see the Windows 7 login prompt.

As for my current test system, it includes:

MB:  ASUS M5A99FX PRO R2.0 ("options vfio_iommu_type1 allow_unsafe_interrupts=1" applied to overcome IOMMU issue)
CPU:  AMD Phenom II X4 955  (I have access to a newer CPU, but this one is usually reliable and works well with Xen.)
RAM: 20 GB ddr3 1600
GPUs: I've tried various combinations of a Radeon HD6670, an Nvidia GT610, and an Nvidia GTX550Ti.

This GPU combination passed your initial test (no drive attached) and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination also passed your initial test  and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Radeon HD6670 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination passed your initial test, but the screen attached to the secondary GPU remained BLANK (without any BSOD) when attempting to boot from an image:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Radeon HD6670 <-- Windows display

I'm surprised that the monitor attached to the Radeon HD6670 remained blank when attempting to boot from an image.  My Radeon HD6670 consistently works well with GPU passthrough under Xen, whereas GPU passthough has never worked with either of my Nvidia cards under Xen.

When I get a chance, I'll try with the patches applied to qemu and seabios.   

REQUEST:  I'm a Linux neophyte.  Would you be so kind as to suggest a tutorial on how to apply patches?

Last edited by GizmoChicken (2013-07-26 02:30:37)

Offline

#266 2013-07-26 03:01:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:

Thank you, would you mind mentioning your build?

1) It should
2) what bsod are you experiencing?, in my experience using these patches will solve most problems, since without them once you shutdown/reset the vm the gpu will stay in a bad state, and most people are having 0 problems with these patches

@nbhs:

Thanks much for the quick reply!

As for the BSOD, it appears and disappears very quickly, and so unfortunately, I haven't had a chance to read the details.  But I can tell you that it always occurs just about when I would expect to see the Windows 7 login prompt.

As for my current test system, it includes:

MB:  ASUS M5A99FX PRO R2.0 ("options vfio_iommu_type1 allow_unsafe_interrupts=1" applied to overcome IOMMU issue)
CPU:  AMD Phenom II X4 955  (I have access to a newer CPU, but this one is usually reliable and works well with Xen.)
RAM: 20 GB ddr3 1600
GPUs: I've tried various combinations of a Radeon HD6670, an Nvidia GT610, and an Nvidia GTX550Ti.

This GPU combination passed your initial test (no drive attached) and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination also passed your initial test  and allows for seeing the Windows welcome screen before a BSOD:

Primary GPU: Radeon HD6670 <-- Ubuntu display, using opensource module
Secondary GPU: Nvidia GTX550Ti <-- Windows display

This GPU combination passed your initial test, but the screen attached to the secondary GPU remained BLANK (without any BSOD) when attempting to boot from an image:

Primary GPU: Nvidia GT610 <-- Ubuntu display, using opensource module
Secondary GPU: Radeon HD6670 <-- Windows display

I'm surprised that the monitor attached to the Radeon HD6670 remained blank when attempting to boot from an image.  My Radeon HD6670 consistently works well with GPU passthrough under Xen, whereas GPU passthough has never worked with either of my Nvidia cards under Xen.

When I get a chance, I'll try with the patches applied to qemu and seabios.   

REQUEST:  I'm a Linux neophyte.  Would you be so kind as to suggest a tutorial on how to apply patches?

Perhaps the radeon card requires some quirks that are included in the patches, it is indeed weird that it did not work, you can also try to do a clean install of windows without virt-manager (using the qemu command line as described in the guide), and if you are passing through the radeon card use the eject method also described in the guide, my initial test passing through the geforce 470 gtx was without any patches using a clean kernel and qemu 1.5, also take into consideration that some driver combinations on the host might produce some problems (its on the "issues" section), as for a patching guide well i guess you can google "how to patch on linux", i would also say doing a crash course on arch might be good, you dont actually have to patch anything since i provide the required packages (patched and all), and its not hard at all once you get it running, plus the community here is great and im sure any problem you have can be resolved.If you indeed want to try it i recommend installing it on a virtual machine first to get the hang of it.

EDIT: forgot to add i havent really tried seabios 1.7.3, i did try master from git before it was released and i got a BSOD saying it wasnt an acpi compilant system blah blah blah, thats why i recommend using 1.7.2 + a patch

Last edited by nbhs (2013-07-26 03:03:39)

Offline

#267 2013-07-26 03:42:33

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Perhaps the radeon card requires some quirks that are included in the patches, it is indeed weird that it did not work, you can also try to do a clean install of windows without virt-manager (using the qemu command line as described in the guide), and if you are passing through the radeon card use the eject method also described in the guide, my initial test passing through the geforce 470 gtx was without any patches using a clean kernel and qemu 1.5, also take into consideration that some driver combinations on the host might produce some problems (its on the "issues" section), as for a patching guide well i guess you can google "how to patch on linux", i would also say doing a crash course on arch might be good, you dont actually have to patch anything since i provide the required packages (patched and all), and its not hard at all once you get it running, plus the community here is great and im sure any problem you have can be resolved.If you indeed want to try it i recommend installing it on a virtual machine first to get the hang of it.

EDIT: forgot to add i havent really tried seabios 1.7.3, i did try master from git before it was released and i got a BSOD saying it wasnt an acpi compilant system blah blah blah, thats why i recommend using 1.7.2 + a patch


Thanks much.  When I get a chance, I'll create a clean Windows VM.  And I might even bite the bullet and try the patches.  But since I want to stick with Ubuntu and like to keep my system up-to-date, I'm hoping that the patches will make their way upstream soon.

By the way, I saw that you reported the IOMMU problem with your motherboard to Asus.   A few of us got together and wrote to the same Asus customer rep who has volunteered to be a single point of contact for this issue.  If you'd like to write to that customer rep, please send me a pm and I'll provide his contact information.

Oh, would you happen to know of any AMD 990FX-based motherboards that don't suffer from IOMMU issues?

Offline

#268 2013-07-26 09:02:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:

Perhaps the radeon card requires some quirks that are included in the patches, it is indeed weird that it did not work, you can also try to do a clean install of windows without virt-manager (using the qemu command line as described in the guide), and if you are passing through the radeon card use the eject method also described in the guide, my initial test passing through the geforce 470 gtx was without any patches using a clean kernel and qemu 1.5, also take into consideration that some driver combinations on the host might produce some problems (its on the "issues" section), as for a patching guide well i guess you can google "how to patch on linux", i would also say doing a crash course on arch might be good, you dont actually have to patch anything since i provide the required packages (patched and all), and its not hard at all once you get it running, plus the community here is great and im sure any problem you have can be resolved.If you indeed want to try it i recommend installing it on a virtual machine first to get the hang of it.

EDIT: forgot to add i havent really tried seabios 1.7.3, i did try master from git before it was released and i got a BSOD saying it wasnt an acpi compilant system blah blah blah, thats why i recommend using 1.7.2 + a patch


Thanks much.  When I get a chance, I'll create a clean Windows VM.  And I might even bite the bullet and try the patches.  But since I want to stick with Ubuntu and like to keep my system up-to-date, I'm hoping that the patches will make their way upstream soon.

By the way, I saw that you reported the IOMMU problem with your motherboard to Asus.   A few of us got together and wrote to the same Asus customer rep who has volunteered to be a single point of contact for this issue.  If you'd like to write to that customer rep, please send me a pm and I'll provide his contact information.

Oh, would you happen to know of any AMD 990FX-based motherboards that don't suffer from IOMMU issues?

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

Offline

#269 2013-07-26 09:57:30

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
GizmoChicken wrote:

I saw that you reported the IOMMU problem with your motherboard to Asus....

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

Yep, I saw your earlier post:

nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

My motherboard's IVRS  errors seem to be slightly different from your motherboard's IVRS errors.  My board's errors include:  1)  "ivrs_ioapic[5] not in IVRS table";  2)  "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found"  I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.

How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?

Last edited by GizmoChicken (2013-07-26 09:58:41)

Offline

#270 2013-07-26 18:43:50

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:

Oh, would you happen to know of any AMD 990FX-based motherboards that don't suffer from IOMMU issues?

The Gigabyte 990FXA-UD5 rev. 3.0 I use has no issues with interrrupt remapping. Same for the 990FXA-UD3 as far as I know.
The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

Offline

#271 2013-07-26 18:56:45

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.


i'm sorry for my poor english wirting skills…

Offline

#272 2013-07-26 19:06:37

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:
GizmoChicken wrote:

I saw that you reported the IOMMU problem with your motherboard to Asus....

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

Yep, I saw your earlier post:

nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

My motherboard's IVRS  errors seem to be slightly different from your motherboard's IVRS errors.  My board's errors include:  1)  "ivrs_ioapic[5] not in IVRS table";  2)  "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found"  I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.

How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

Last edited by nbhs (2013-07-26 19:16:12)

Offline

#273 2013-07-26 19:09:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
teekay wrote:

The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

Have you tried appending iommu=pt to your grub cfg?

Last edited by nbhs (2013-07-26 19:16:56)

Offline

#274 2013-07-26 19:47:32

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
andy123 wrote:

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

Have you tried appending iommu=pt to your grub cfg?

syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these

AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]

The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.


i'm sorry for my poor english wirting skills…

Offline

#275 2013-07-26 19:50:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
andy123 wrote:

In case you tried this, did you manage to get the Marvell controller working inside a VM? For me, I got IO_PAGE_FAULTs in dmesg and Windows gave me a BSOD and the only fix I could find applies to the intel-iommu driver (http://lkml.indiana.edu/hypermail/linux … 00079.html).
Also, technically it's not a bug in the amd-iommu driver, but in the Marvell Controller hardware. My board is a AsRock 990FX Extreme4, so same chipset and SATA controller as the Gigabyte.

Have you tried appending iommu=pt to your grub cfg?

syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these

AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]

The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.

Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

if you need the rom you can do:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

or using pci-assign instead of vfio

Last edited by nbhs (2013-07-26 19:54:51)

Offline

#276 2013-07-26 20:51:40

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

if you need the rom you can do:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

or using pci-assign instead of vfio

No idea how I didn't think of that yet, but it sadly doesn't seem to solve the problem. But why excactly should it work passed through, when the situation is like this: IOMMU in UEFI on → pata-marvell driver (on host) barely works, as in less than 4mb/s with hdparm or dd after a few minutes to recognize a device. IOMMU off, normal speeds, several 100mb/s from SSD.
Anyway, vfio with or without romfile and behind the pcie root port or not produce BSODs in windows 7+8 guests while loading the driver and pci-assign kind of crashes the host

Jul 26 22:36:58 desk kernel: ---[ end trace efd4627dfc7dbb30 ]--- Jul 26 22:36:58 desk kernel: RSP <ffff8804475e1ce0> Jul 26 22:36:58 desk kernel: RIP [<ffffffff81141c64>] free_huge_page+0x124/0x140 Jul 26 22:36:58 desk kernel: Code: 10 00 00 00 ad de 48 89 43 20 48 b8 00 02 20 00 00 00 ad de 48 89 43 28 e8 ca e9 ff ff 49 ff 4c 24 38 41 ff 4c 24 70 eb 8f 0f 0b <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 66 66 2e 0f Jul 26 22:36:58 desk kernel: [<ffffffff81621986>] system_call_fastpath+0x1a/0x1f Jul 26 22:36:58 desk kernel: [<ffffffff8161d82e>] ? do_page_fault+0xe/0x20 Jul 26 22:36:58 desk kernel: [<ffffffff81174cc1>] SyS_ioctl+0x81/0xa0 Jul 26 22:36:58 desk kernel: [<ffffffff8112f4d8>] ? do_munmap+0x2b8/0x3e0 Jul 26 22:36:58 desk kernel: [<ffffffff81174a5d>] do_vfs_ioctl+0x2dd/0x4c0 Jul 26 22:36:58 desk kernel: [<ffffffff814b3f38>] vfio_fops_unl_ioctl+0x78/0x360 Jul 26 22:36:58 desk kernel: [<ffffffff8161d544>] ? __do_page_fault+0x2e4/0x5c0 Jul 26 22:36:58 desk kernel: [<ffffffff814b5c31>] vfio_iommu_type1_ioctl+0x4b1/0x7a0 Jul 26 22:36:58 desk kernel: [<ffffffff814b52ee>] vfio_dma_unmap+0xe/0x20 Jul 26 22:36:58 desk kernel: [<ffffffff814b51bf>] __vfio_dma_do_unmap.isra.3+0x7f/0xa0 Jul 26 22:36:58 desk kernel: [<ffffffff814b511a>] put_pfn+0x3a/0x60 Jul 26 22:36:58 desk kernel: [<ffffffff8110e10c>] put_page+0x2c/0x40 Jul 26 22:36:58 desk kernel: [<ffffffff8110df15>] put_compound_page+0x55/0x220 Jul 26 22:36:58 desk kernel: [<ffffffff8110de5f>] __put_compound_page+0x1f/0x40 Jul 26 22:36:58 desk kernel: Call Trace: Jul 26 22:36:58 desk kernel: ffff880445b88428 0000000000000000 000000000f400000 0000000000000002 Jul 26 22:36:58 desk kernel: ffffffff8110de5f ffffea000f400000 ffff8804475e1d58 ffffffff8110df15 Jul 26 22:36:58 desk kernel: ffffea000f000000 ffffea000f000000 ffffea000f00001c ffff8804475e1d10 Jul 26 22:36:58 desk kernel: Stack: Jul 26 22:36:58 desk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jul 26 22:36:58 desk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jul 26 22:36:58 desk kernel: CR2: 00007f509648a180 CR3: 000000033e0b7000 CR4: 00000000000407f0 Jul 26 22:36:58 desk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jul 26 22:36:58 desk kernel: FS: 00007f509617d900(0000) GS:ffff88045ec00000(0000) knlGS:0000000000000000 Jul 26 22:36:58 desk kernel: R13: 0000000000000000 R14: 0000000000000246 R15: ffff8804475e1fd8 Jul 26 22:36:58 desk kernel: R10: 0000000000000000 R11: 000ffffffffff000 R12: ffffffff81ce3e20 Jul 26 22:36:58 desk kernel: RBP: ffff8804475e1cf8 R08: 0000000000000001 R09: ffff88044892b2a0 Jul 26 22:36:58 desk kernel: RDX: 800000000000001c RSI: 0000000000000003 RDI: 0000000040000000 Jul 26 22:36:58 desk kernel: RAX: 0000000000000000 RBX: ffffea000f000000 RCX: 0000000000000012 Jul 26 22:36:58 desk kernel: RSP: 0018:ffff8804475e1ce0 EFLAGS: 00010213 Jul 26 22:36:58 desk kernel: RIP: 0010:[<ffffffff81141c64>] [<ffffffff81141c64>] free_huge_page+0x124/0x140 Jul 26 22:36:58 desk kernel: task: ffff880443d85040 ti: ffff8804475e0000 task.ti: ffff8804475e0000 Jul 26 22:36:58 desk kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./990FX Extreme4, BIOS P2.00 10/15/2012 Jul 26 22:36:58 desk kernel: CPU: 0 PID: 947 Comm: qemu-system-x86 Tainted: P O 3.10.3-1-ck #1 Jul 26 22:36:58 desk kernel: usbhid hid ata_generic ahci pata_acpi libahci ohci_hcd ehci_pci xhci_hcd ehci_hcd libata usbcore usb_common scsi_mod Jul 26 22:36:58 desk kernel: Modules linked in: tun nfsd auth_rpcgss oid_registry nfs_acl bridge stp llc ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_na Jul 26 22:36:58 desk kernel: invalid opcode: 0000 [#1] PREEMPT SMP Jul 26 22:36:58 desk kernel: kernel BUG at mm/hugetlb.c:627! Jul 26 22:36:58 desk kernel: ------------[ cut here ]------------

and qemu says something about wrong PCI something size, not a multiple of 4k? If this might be relevant, I'll reproduce it and copy&paste the output.
It's been a while, so is this the right syntax for pci-assign?

-device pci-assign,host=02:00.0,romfile=/root/1b4b-9120_v1001033.bin,multifunction=on \ -device pci-assign,host=02:00.1,rombar=0 

I still think it's a hardware error on Marvell's side that might be worked arround in the amd_iommu driver, but it would be nice if there was another way to solve this.


i'm sorry for my poor english wirting skills…

Offline

#277 2013-07-26 21:05:14

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
andy123 wrote:
nbhs wrote:

Have you tried appending iommu=pt to your grub cfg?

syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these

AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]

The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.

Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

if you need the rom you can do:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

or using pci-assign instead of vfio

The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Last edited by teekay (2013-07-26 21:08:44)

Offline

#278 2013-07-26 21:07:06

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
nbhs wrote:
andy123 wrote:

syslinux.cfg and yes. (I just checked /proc/cmdline)
At first there are the option rom errors, which I might have fixed but then there are these

AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.1 domain=0x0000 address=0x00000000af418000 flags=0x0050]

The thing is, I forward 02:00.0 not 02:00.1, but the Marvell Controller does some strange things I don't understand, because I have no idea how PCIe works.

Have you tried passing through both? the marvell device is prolly a multifunction device, so you should pass it through as such, maybe even behind a pcie root port like the gpu, something like this:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

if you need the rom you can do:

-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=02:00.0,bus=root.2,addr=00.0,multifunction=on,romfile=/path/to/your/rom.bin \ -device vfio-pci,host=02:00.1,bus=root.2,addr=00.1

or using pci-assign instead of vfio

The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

Offline

#279 2013-07-26 21:16:01

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
teekay wrote:

The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?

p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".


i'm sorry for my poor english wirting skills…

Offline

#280 2013-07-26 21:24:39

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
teekay wrote:

The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?

p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

In the qemu command I use

-device vfio-pci,host=03:00.0,bus=pcie.0

and it works fine, no error messages on the host side. But as said, I needed to bind it to pci-stub, otherwhise the host's ahci driver would go nuts (but that's fully unrelated to qemu, passthrough and all that).

Offline

#281 2013-07-26 21:31:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
teekay wrote:

The IO_PAGE_FAULT stuff is what I meant with "it doesn't work in Linux if IOMMU is enabled".
I added it to pci_stub.ids boot parameter so the host ignores it, and passed it through as a simple pcie device. Works well in Win7 with built-in drivers here.

Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?

p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

Offline

#282 2013-07-26 21:41:35

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

Good point, nbhs is correct: it's the ahci driver. The pata-marvell is unrelated, so blacklisting it won't help. That's why only pci-stub can stop the host from acting upon the controller.

└» zcat /proc/config.gz |grep PATA_MARVELL # CONFIG_PATA_MARVELL is not set

Offline

#283 2013-07-26 21:42:23

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
andy123 wrote:
nbhs wrote:

Yeah that another thing you can try, binding it to pci-stub before the driver grabs it

Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?

p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

That's what I though at first, but "lspci -nnvvv" tells me this:

02:00.0 IDE interface [0101]: Marvell Technology Group Ltd. 88SE91A0 SATA 6Gb/s Controller [1b4b:91a0] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO]) Subsystem: ASRock Incorporation Device [1849:91a0] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 44 Region 0: I/O ports at d090 [size=8] Region 1: I/O ports at d080 [size=4] Region 2: I/O ports at d070 [size=8] Region 3: I/O ports at d060 [size=4] Region 4: I/O ports at d050 [size=16] Region 5: Memory at fe615000 (32-bit, non-prefetchable) [size=2K] Expansion ROM at fe600000 [disabled] [size=64K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: pata_marvell, pata_acpi, ata_generic 02:00.1 IDE interface [0101]: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO]) Subsystem: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 45 Region 0: I/O ports at d040 [size=8] Region 1: I/O ports at d030 [size=4] Region 2: I/O ports at d020 [size=8] Region 3: I/O ports at d010 [size=4] Region 4: I/O ports at d000 [size=16] Region 5: Memory at fe614000 (32-bit, non-prefetchable) [size=16] Expansion ROM at fe610000 [disabled] [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: pata_marvell, pata_acpi, ata_generic

And I found this patch on the kernel mailinglist, that seems to be responsible for this, but as said it works well without IOMMU.
And yes, this is indepented of AHCI or IDE setting in the UEFI. But iirc, it's always detected as an IDE controller in windows (bare-metal), too.
There are some quite strange controllers on this board, the USB3 is from Etron, it works better than the Marvell SATA controller, but sometimes freezes and caused BSODs in a guest, that's why I pass though one of the usb2 controllers now.

edit:

teekay wrote:

Good point, nbhs is correct: it's the ahci driver. The pata-marvell is unrelated, so blacklisting it won't help. That's why only pci-stub can stop the host from acting upon the controller.

Interpret the lspci output above for yourself, but I read from it, that I blacklisted the right driver. Please tell me if I didn't.

Last edited by andy123 (2013-07-26 21:48:49)


i'm sorry for my poor english wirting skills…

Offline

#284 2013-07-26 21:45:59

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
andy123 wrote:

Well, I blacklisted the pata-marvell driver, because when it loaded it slowed the boot by several minutes, so that should already be the case.
Teekay, what parameters do you use excactly to pass it though as a simple pcie device? Do you get Option rom errors?

p.s.: I contacted AsRock a few days ago, but all I got back until now was something along the lines of "we notified some department in Taiwan about it and will write back in a week".

Excuse my ignorance but isnt the pata driver needed for ide only? shouldnt it be using the ahci module?

That's what I though at first, but "lspci -nnvvv" tells me this:

02:00.0 IDE interface [0101]: Marvell Technology Group Ltd. 88SE91A0 SATA 6Gb/s Controller [1b4b:91a0] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO]) Subsystem: ASRock Incorporation Device [1849:91a0] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 44 Region 0: I/O ports at d090 [size=8] Region 1: I/O ports at d080 [size=4] Region 2: I/O ports at d070 [size=8] Region 3: I/O ports at d060 [size=4] Region 4: I/O ports at d050 [size=16] Region 5: Memory at fe615000 (32-bit, non-prefetchable) [size=2K] Expansion ROM at fe600000 [disabled] [size=64K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: pata_marvell, pata_acpi, ata_generic 02:00.1 IDE interface [0101]: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4] (rev 12) (prog-if 8f [Master SecP SecO PriP PriO]) Subsystem: Marvell Technology Group Ltd. 88SE912x IDE Controller [1b4b:91a4] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 45 Region 0: I/O ports at d040 [size=8] Region 1: I/O ports at d030 [size=4] Region 2: I/O ports at d020 [size=8] Region 3: I/O ports at d010 [size=4] Region 4: I/O ports at d000 [size=16] Region 5: Memory at fe614000 (32-bit, non-prefetchable) [size=16] Expansion ROM at fe610000 [disabled] [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: pata_marvell, pata_acpi, ata_generic

And I found this patch on the kernel mailinglist, that seems to be responsible for this, but as said it works well without IOMMU.
And yes, this is indepented of AHCI oder IDE setting in the UEFI. But iirc, it's always detected as an IDE controller in windows (bare-metal), too.
There are some quite strange controllers on this board, the USB3 is from Etron, it works better than the Marvell SATA controller, but sometimes freezes and caused BSODs in a guest, that's why I pass though one of the usb2 controllers now.

Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci

Offline

#285 2013-07-26 21:48:21

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci

To quote myself:

andy123 wrote:

And yes, this is indepented of AHCI or IDE setting in the UEFI.

Last edited by andy123 (2013-07-26 21:48:34)


i'm sorry for my poor english wirting skills…

Offline

#286 2013-07-26 21:55:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci

To quote myself:

andy123 wrote:

And yes, this is indepented of AHCI or IDE setting in the UEFI.

Im sorry i overlooked that, does this behavior happens when you disable iommu?

Offline

#287 2013-07-26 21:59:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
andy123 wrote:
nbhs wrote:

Well you problem seems to be its running on ide mode, check your mobo configuration and try to switch it to ahci

To quote myself:

andy123 wrote:

And yes, this is indepented of AHCI or IDE setting in the UEFI.

Im sorry i overlooked that, does this behavior happens when you disable iommu?

EDIT: also make sure you disable "combined mode", this should only affect the amd controller though
EDIT2: from the guide http://www.manualslib.com/manual/434060 … ml?page=59

Last edited by nbhs (2013-07-26 22:07:21)

Offline

#288 2013-07-26 22:14:57

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
nbhs wrote:
andy123 wrote:

To quote myself:

Im sorry i overlooked that, does this behavior happens when you disable iommu?

EDIT: also make sure you disable "combined mode", this should only affect the amd controller though
EDIT2: from the guide http://www.manualslib.com/manual/434060 … ml?page=59

No Problem. I have this guide as paper, but it's kind of worthless, since it depicts the old, 1.x version of the UEFI, so wrong defaults and stuff.
As stated somewhere above, with IOMMU off, the driver works good (normal SSD speeds and devices recognized normally).


i'm sorry for my poor english wirting skills…

Offline

#289 2013-07-26 22:18:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
nbhs wrote:

Im sorry i overlooked that, does this behavior happens when you disable iommu?

EDIT: also make sure you disable "combined mode", this should only affect the amd controller though
EDIT2: from the guide http://www.manualslib.com/manual/434060 … ml?page=59

No Problem. I have this guide as paper, but it's kind of worthless, since it depicts the old, 1.x version of the UEFI, so wrong defaults and stuff.
As stated somewhere above, with IOMMU off, the driver works good (normal SSD speeds and devices recognized normally).

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

EDIT: something to read if you're interested http://www.projectosx.com/forum/index.p … topic=1150
http://wiki.debian.org/InstallingDebian … acBook/3-1

Last edited by nbhs (2013-07-26 22:27:08)

Offline

#290 2013-07-26 22:27:04

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.


i'm sorry for my poor english wirting skills…

Offline

#291 2013-07-26 22:35:51

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

modinfo ahci
parm: marvell_enable:Marvell SATA via AHCI (1 = enabled) (int)

So you could try to add  ahci.marvell_enable=1 as a kernel parameter or to /etc/modprobe.d/marvel.conf, i also found http://forum.mandriva.com/en/viewtopic.php?t=120279, so yeah if that doesnt work i dunno

EDIT: from wikipedia:

88SE91xx chipsets Linux support

Linux SATA driver
Appears to be supported by the standard AHCI driver, however out of the box is not recognised by the AHCI driver - AHCI driver needs to be taught to be loaded for the relevant PCI vendor and product ID.

Check lspci -nnk for the PCI vendor and product ID. i.e. in the following instance the relevant part is "1b4b:9192"

04:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. Device [1b4b:9192] (re11)

To associate these IDs with the AHCI driver run

/bin/echo 1b4b 9192 > /sys/bus/pci/drivers/ahci/new_id

Last edited by nbhs (2013-07-26 23:09:11)

Offline

#292 2013-07-26 23:19:10

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

The Gigabyte 990FXA-UD5 rev. 3.0 I use has no issues with interrrupt remapping. Same for the 990FXA-UD3 as far as I know.
The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

nbhs wrote:

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

@teekay and nbhs:

Thanks much!  I think I'll give Gigabyte a try next time.

nbhs wrote:

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

@nbhs:

Thanks for this too.  As you suggested, I passed "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" as kernel parameters, and now I no longer need the allow_unsafe_interrupts module.  Great!

nbhs wrote:

Have you tried appending iommu=pt to your grub cfg?

I've seen this "iommu=pt" parameter mentioned many times, but despite my search attempts, I have yet to figure out what it does.   Is there simple explanation of what this parameter does?

Offline

#293 2013-07-26 23:25:55

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
teekay wrote:

The Gigabyte 990FXA-UD5 rev. 3.0 I use has no issues with interrrupt remapping. Same for the 990FXA-UD3 as far as I know.
The only issue this board has is, if IOMMU is enabled, the Marvell SATA controller doesn't see any drives in Linux (a bug in the amd_iommu driver) but then again the marvell controller can be passed through to the VM.

nbhs wrote:

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

@teekay and nbhs:

Thanks much!  I think I'll give Gigabyte a try next time.

nbhs wrote:

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

@nbhs:

Thanks for this too.  As you suggested, I passed "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" as kernel parameters, and now I no longer need the allow_unsafe_interrupts module.  Great!

nbhs wrote:

Have you tried appending iommu=pt to your grub cfg?

I've seen this "iommu=pt" parameter mentioned many times, but despite my search attempts, I have yet to figure out what it does.   Is there simple explanation of what this parameter does?

I believe this disables DMA remapping

Offline

#294 2013-07-26 23:33:10

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
GizmoChicken wrote:

I've seen this "iommu=pt" parameter mentioned many times, but despite my search attempts, I have yet to figure out what it does.   Is there simple explanation of what this parameter does?

I believe this disables DMA remapping

Thanks!

Offline

#295 2013-07-27 06:44:34

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

IT WORKS! 

For me, the trick for ending the BSODs that I previously reported was to replace the ahci controller ("-device ahci,bus=pcie.0,id=ahci") with an ide controller ("-device piix4-ide,bus=pcie.0").  It took me a few tries to figure out that the change in controller necessitated  replacing "-device ide-hd,bus=ahci.0,drive=disk" with "-device ide-hd,drive=disk" (no "bus=...,") compared to the original example.

Here's a minimalist version of the qemu commands that I'm using:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \ -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root,addr=00.1 \ -device piix4-ide,bus=pcie.0 \ -drive file=/images/Win7HVM.img,id=disk,format=raw -device ide-hd,drive=disk

Even with an outdated Nvidia driver (downloaded with Windows updates), I have HDMI audio and a Windows experience of 6.9 for desktop graphics and gaming graphics.  But something tells me that a newer driver would improve matters.

Here's a summary of my setup:

MB: ASUS M5A99FX PRO R2.0 w/ BIOS v2005 CPU: AMD Phenom II X4 955 RAM: 20 GB DDR3 1600 Primary GPU: Radeon HD6670 w/ a Catalyst 13.6-beta driver on the host Seconday GPU: Nvidia GTX550Ti w/ an Nvidia driver (from Windows updates) on the guest Ubuntu 13.10 (current daily build) Kernel 3.10.0-5 (from Ubuntu, no patches) seabios 1.7.3 (from Ununtu, no patches) qemu 1.5.0 (from Ununtu, no patches)

Since I'm using a stock Ubuntu kernel, I added the following to /etc/modules:

vfio vfio_iommu_type1 vfio_pci kvm kvm_amd amd_iommu_v2 pci_stub

Also, here's the GRUB cmdline that I'm using:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci-stub.ids=10de:1244,10de:0bee"

Using BIOS v2005, IOMMU works flawlessly on my ASUS M5A99FX PRO R2.0 motherboard.  Using older BIOS versions with that motherboard, IOMMU errors occurred, but those errors could be cured by adding "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" to the GRUB cmdline (as suggested by nbhs).  (Note that the pci addresses and values "[5]" and "[6]" may need to be changed, depending on setup.)

Using BIOS v2005, "dmesg | grep AMD-Vi" gives me the following output:

$ dmesg | grep AMD-Vi AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 AMD-Vi: Interrupt remapping enabled AMD-Vi: Lazy IO/TLB flushing enabled

I haven't tested much, but so far, the only major hiccup has been a frozen GPU when attempting to passthough an Ubuntu guest to the GPU.  (I never was able to get my Ubuntu guest to boot with vfio passthough.)

Now, if only I could use VirtManager to start/stop/monitor this VM.

EDIT 1:  I made a few minor tweaks since I first posted the above.  For details, have a look here:  https://bbs.archlinux.org/viewtopic.php … 7#p1313007
EDIT 2:  Updated to reflect that BIOS v2005 cures previously reported IOMMU errors and other small tweaks.

Last edited by GizmoChicken (2013-11-08 04:03:16)

Offline

#296 2013-07-27 08:09:02

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:

I haven't tested much, but so far, the only major hiccup has been a frozen GPU when attempting to passthough an Ubuntu guest to the GPU.  (I never was able to get my Ubuntu guest to boot with vfio passthough.)

Small update:  I just noticed that, when booting from an iso and running in "try it" mode, I am able to passthough an Ubuntu 13.10 guest to the GPU.  So my guess is that vfio doesn't like something about the Ubuntu VM that I had previously created using VirtManager.  At some point, I'll try creating a fresh Ubuntu VM and test again.

Offline

#297 2013-07-27 08:58:23

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:

IT WORKS! 

For me, the trick for ending the BSODs that I previously reported was to replace the ahci controller ("-device ahci,bus=pcie.0,id=ahci") with an ide controller ("-device piix4-ide,bus=pcie.0").  It took me a few tries to figure out that the change in controller necessitated  replacing "-device ide-hd,bus=ahci.0,drive=disk" with "-device ide-hd,drive=disk" (no "bus=...,") compared to the original example.

Here's a minimalist version of the qemu commands that I'm using:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \ -device vfio-pci,host=02:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root,addr=00.1 \ -device piix4-ide,bus=pcie.0 \ -drive file=/images/Win7HVM.img,id=disk,format=raw -device ide-hd,drive=disk

Even with an outdated Nvidia driver (downloaded with Windows updates), I have HDMI audio and a Windows experience of 6.9 for desktop graphics and gaming graphics.  But something tells me that a newer driver would improve matters.

Here's a summary of my setup:

MB: ASUS M5A99FX PRO R2.0 CPU: AMD Phenom II X4 955 RAM: 20 GB DDR3 1600 Primary GPU: Radeon HD6670 w/ a Catalyst 13.6-beta driver on the host Seconday GPU: Nvidia GTX550Ti w/ an Nvidia driver (from Windows updates) on the guest Ubuntu 13.10 (current daily build) Kernel 3.10.0-5 (from Ubuntu, no patches) qemu 1.5.0 (from Ununtu, no patches) seabios 1.7.3 (from Ununtu, no patches)

Since I'm using a stock Ubuntu kernel, I added the following to /etc/modules:

vfio vfio_iommu_type1 vfio_pci kvm kvm_amd amd_iommu_v2 pci_stub

Also, here's the GRUB cmdline that I'm using:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=on ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 pci-stub.ids=10de:1244,10de:0bee"

As I mentioned elsewhere, the addition of "ivrs_ioapic[5]=00:14.0" and "ivrs_ioapic[6]=00:00.1" (as suggested by nbhs) seems to cure the IOMMU errors with ASUS M5A99FX PRO R2.0 motherboard.  (Note that the pci addresses and values "[5]" and "[6]" may need to be changed, depending on setup.)

$ dmesg | grep AMD-Vi AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 AMD-Vi: Interrupt remapping enabled AMD-Vi: Lazy IO/TLB flushing enabled

I haven't tested much, but so far, the only major hiccup has been a frozen GPU when attempting to passthough an Ubuntu guest to the GPU.  (I never was able to get my Ubuntu guest to boot with vfio passthough.)

Now, if only I could use VirtManager to start/stop/monitor this VM.

Congrats, your problem is you installed windows with virt-manager using the ide controller instead of the "sata" controller (ahci), and windows doesnt like that (it doesnt like that even on bare hardware), see this post on the same problem https://bbs.archlinux.org/viewtopic.php … 0#p1298880, also "amd_iommu=on" is not a valid kernel parameter as the kernel automatically enables it on amd boards

Last edited by nbhs (2013-07-27 08:59:17)

Offline

#298 2013-07-27 10:35:12

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Congrats, your problem is you installed windows with virt-manager using the ide controller instead of the "sata" controller (ahci), and windows doesnt like that (it doesnt like that even on bare hardware), see this post on the same problem https://bbs.archlinux.org/viewtopic.php … 0#p1298880, also "amd_iommu=on" is not a valid kernel parameter as the kernel automatically enables it on amd boards

Yep, that probably explains it.  I'll create fresh VMs (Windows and Ubuntu) using sata/ahci next week and will test whether they too run with the unpatched versions of qemu and seabios along with the stock Ubuntu 3.10 kernel.

Oh, I tested my Nvidia GT610 as yet another secondary GPU, and it seems to be working great too.  Not quite as fast as my Nvidia GTX550Ti, but plenty  fast for Netflix and Youtube.

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

Last edited by GizmoChicken (2013-07-27 10:42:05)

Offline

#299 2013-07-27 11:35:39

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

I am still unable get Windows running with ATI HD7750 card:
I could pass first part of windows installation (partitionig and files
copying), but then VM reboots and I get black screen.
Alex Williamson proposed to use "vga-current" branch of :
git://github.com/awilliam/qemu-vfio.git
git://github.com/awilliam/linux-vfio.git

Could you please check if I created right PKGBUILD files?

Linux vfio: http://www.fileswap.com/dl/K7s93euILN/
Qemu vfio: http://www.fileswap.com/dl/Oie83lou8B/

Last edited by myweb (2013-07-27 11:37:51)

Offline

#300 2013-07-27 13:46:47

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Offline

#301 2013-07-27 16:41:29

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I seem to be having the same issue that apoapo and myweb are running through.

- With the Radeon as the primary video device, I get apoapo's dirty stream of errors :

system-x86_64: vfio_bar_write(,0x13ff50, 0x36703067,4) failed: Device or resource busy

and this in dmesg:

[ 763.408269] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref] [ 763.408280] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref] [ 763.408291] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]

With Intel integrated as the primary device, no errors come up, but the Radeon never initializes.

The latter seems like an easier start point. Is there any way to force that card to turn on during the SEABios boot? Any way to log why it's not doing so?

Offline

#302 2013-07-27 17:46:59

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
andy123 wrote:
nbhs wrote:

Yes but is it using the ahci or pata driver?, from what im getting enabling iommu switches the marvell controller into ide mode is this correct?, i believe there is a way to switch the controller into ahci using grub using the setpci command, i know some mac users did that when dual booting windows on some intel controller, im not sure how or if this is possible on yours

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

modinfo ahci
parm: marvell_enable:Marvell SATA via AHCI (1 = enabled) (int)

So you could try to add  ahci.marvell_enable=1 as a kernel parameter or to /etc/modprobe.d/marvel.conf, i also found http://forum.mandriva.com/en/viewtopic.php?t=120279, so yeah if that doesnt work i dunno

EDIT: from wikipedia:

88SE91xx chipsets Linux support

Linux SATA driver
Appears to be supported by the standard AHCI driver, however out of the box is not recognised by the AHCI driver - AHCI driver needs to be taught to be loaded for the relevant PCI vendor and product ID.

Check lspci -nnk for the PCI vendor and product ID. i.e. in the following instance the relevant part is "1b4b:9192"

04:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. Device [1b4b:9192] (re11)

To associate these IDs with the AHCI driver run

/bin/echo 1b4b 9192 > /sys/bus/pci/drivers/ahci/new_id

I read that wikipedia article a while back, but it doesn't work. With ahci.marvell_enable=1 the driver even gets loaded automatically, but it doesn't recognize any devices (at least with iommu on, but the pata-marvell driver works decent with iommu off, so why try to force the ahci?). Relevant dmesg (dmesg|egrep -i (ata|ahci|scsi))

[ 0.955526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.990232] SCSI subsystem initialized [ 0.993158] ACPI: bus type ATA registered [ 0.993369] libata version 3.00 loaded. [ 0.994387] pata_acpi 0000:02:00.1: enabling device (0000 -> 0003) [ 1.238497] ahci 0000:00:11.0: version 3.0 [ 1.238729] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 1.238734] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 1.240134] scsi0 : ahci [ 1.240202] scsi1 : ahci [ 1.240277] scsi2 : ahci [ 1.240333] scsi3 : ahci [ 1.240418] scsi4 : ahci [ 1.240470] scsi5 : ahci [ 1.240513] ata1: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b100 irq 19 [ 1.240515] ata2: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b180 irq 19 [ 1.240517] ata3: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b200 irq 19 [ 1.240519] ata4: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b280 irq 19 [ 1.240521] ata5: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b300 irq 19 [ 1.240523] ata6: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b380 irq 19 [ 1.240619] ahci 0000:02:00.0: irq 75 for MSI/MSI-X [ 1.240632] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1 [ 1.702088] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.702107] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.702123] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.703241] ata6.00: ATA-8: ST31000520AS, CC32, max UDMA/133 [ 1.703243] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 1.704090] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.704561] ata6.00: configured for UDMA/133 [ 1.705090] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.705109] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.705619] ata1.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133 [ 1.705620] ata1.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.706205] ata1.00: configured for UDMA/133 [ 1.706285] scsi 0:0:0:0: Direct-Access ATA WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5 [ 1.708208] ata3.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133 [ 1.708210] ata3.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.710489] ata2.00: ATA-7: SAMSUNG HD103SI, 1AG01118, max UDMA7 [ 1.710491] ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.711470] ata4.00: ATA-8: WDC WD20EARX-00PASB0, 51.0AB51, max UDMA/133 [ 1.711472] ata4.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.713723] ata5.00: ATA-8: ADATA SP900, 5.0.2a, max UDMA/133 [ 1.713725] ata5.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.714324] ata3.00: configured for UDMA/133 [ 1.716961] ata2.00: configured for UDMA/133 [ 1.717032] scsi 1:0:0:0: Direct-Access ATA SAMSUNG HD103SI 1AG0 PQ: 0 ANSI: 5 [ 1.717202] scsi 2:0:0:0: Direct-Access ATA SAMSUNG HD204UI 1AQ1 PQ: 0 ANSI: 5 [ 1.718470] ata4.00: configured for UDMA/133 [ 1.718535] scsi 3:0:0:0: Direct-Access ATA WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5 [ 1.723710] ata5.00: configured for UDMA/133 [ 1.723771] scsi 4:0:0:0: Direct-Access ATA ADATA SP900 5.0. PQ: 0 ANSI: 5 [ 1.723902] scsi 5:0:0:0: Direct-Access ATA ST31000520AS CC32 PQ: 0 ANSI: 5 [ 2.248026] ahci 0000:02:00.0: controller reset failed (0x80000001) [ 2.248100] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode [ 2.248102] ahci 0000:02:00.0: flags: [ 2.248239] scsi6 : ahci [ 2.248291] ata7: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75 [ 2.557575] ata7: SATA link down (SStatus 0 SControl 300) [ 2.565869] sd 4:0:0:0: [sde] Attached SCSI disk [ 2.579963] sd 5:0:0:0: [sdf] Attached SCSI disk [ 2.585895] sd 1:0:0:0: [sdb] Attached SCSI disk [ 2.589825] sd 2:0:0:0: [sdc] Attached SCSI disk [ 3.039930] sd 3:0:0:0: [sdd] Attached SCSI disk [ 3.083346] sd 0:0:0:0: [sda] Attached SCSI disk [ 3.863782] scsi7 : pata_marvell [ 3.864187] scsi8 : pata_marvell [ 3.864574] ata8: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45 [ 3.864577] ata9: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45 [ 165.303195] ata9: soft resetting link [ 165.465217] ata9: EH complete [ 179.817277] ata8: soft resetting link [ 179.969106] ata8: EH complete [ 189.161840] ata7: hard resetting link [ 189.467441] ata7: SATA link down (SStatus 0 SControl 300) [ 189.467461] ata7: EH complete [ 315.833140] ata8: soft resetting link [ 315.984546] ata8: EH complete [ 318.974146] ata8: soft resetting link [ 319.125082] ata8: EH complete [ 328.019533] scsi9 : pata_marvell [ 328.019951] scsi10 : pata_marvell [ 328.020417] ata10: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45 [ 328.020426] ata11: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45 [ 373.887292] ata11: soft resetting link [ 374.048856] ata11: EH complete [ 381.099605] ata10: soft resetting link [ 381.250912] ata10: EH complete [ 409.101457] ahci 0000:02:00.0: irq 75 for MSI/MSI-X [ 409.101485] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1 [ 410.106469] ahci 0000:02:00.0: controller reset failed (0x80000003) [ 410.106481] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode [ 410.106483] ahci 0000:02:00.0: flags: [ 410.106655] scsi11 : ahci [ 410.106706] ata12: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75 [ 410.412017] ata12: SATA link down (SStatus 0 SControl 300) [ 427.452877] ata12: hard resetting link [ 427.757703] ata12: SATA link down (SStatus 0 SControl 300) [ 427.757721] ata12: EH complete [ 443.326795] scsi12 : pata_marvell [ 443.327090] scsi13 : pata_marvell [ 443.327348] ata13: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45 [ 443.327354] ata14: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45 [ 455.198604] ata13: soft resetting link [ 455.349916] ata13: EH complete [ 460.648310] ata14: soft resetting link [ 460.809941] ata14: EH complete

As you can see, I played around with the different drivers (pata-marvell loads normally now), but no devices are recognized. Windows VM still BSODs.


i'm sorry for my poor english wirting skills…

Offline

#303 2013-07-27 19:06:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:

Congrats, your problem is you installed windows with virt-manager using the ide controller instead of the "sata" controller (ahci), and windows doesnt like that (it doesnt like that even on bare hardware), see this post on the same problem https://bbs.archlinux.org/viewtopic.php … 0#p1298880, also "amd_iommu=on" is not a valid kernel parameter as the kernel automatically enables it on amd boards

Yep, that probably explains it.  I'll create fresh VMs (Windows and Ubuntu) using sata/ahci next week and will test whether they too run with the unpatched versions of qemu and seabios along with the stock Ubuntu 3.10 kernel.

Oh, I tested my Nvidia GT610 as yet another secondary GPU, and it seems to be working great too.  Not quite as fast as my Nvidia GTX550Ti, but plenty  fast for Netflix and Youtube.

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?


If they're the same vendor yeah you'll have problems with with pci stub, what you can do is flash one card and modify the vendor id, or buy another one from a different vendor, also dont forget to check out my post on performance report/improvements https://bbs.archlinux.org/viewtopic.php … 1#p1270311

Offline

#304 2013-07-27 19:07:45

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear All,

I am still unable get Windows running with ATI HD7750 card:
I could pass first part of windows installation (partitionig and files
copying), but then VM reboots and I get black screen.
Alex Williamson proposed to use "vga-current" branch of :
git://github.com/awilliam/qemu-vfio.git
git://github.com/awilliam/linux-vfio.git

Could you please check if I created right PKGBUILD files?

Linux vfio: http://www.fileswap.com/dl/K7s93euILN/
Qemu vfio: http://www.fileswap.com/dl/Oie83lou8B/

So it does work at some point but after doing a reboot it doesnt work anymore?

Offline

#305 2013-07-27 19:34:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

I seem to be having the same issue that apoapo and myweb are running through.

- With the Radeon as the primary video device, I get apoapo's dirty stream of errors :

system-x86_64: vfio_bar_write(,0x13ff50, 0x36703067,4) failed: Device or resource busy

and this in dmesg:

[ 763.408269] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref] [ 763.408280] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref] [ 763.408291] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]

With Intel integrated as the primary device, no errors come up, but the Radeon never initializes.

The latter seems like an easier start point. Is there any way to force that card to turn on during the SEABios boot? Any way to log why it's not doing so?

Well assuming you have 2 cards (the IGP and the radeon) the IGP should always be the primary one since you cant pass it through and passing through your primary card isnt supported, are you using my builds i posted on the main guide(kernel, qemu and seabios)?

Last edited by nbhs (2013-07-27 19:35:26)

Offline

#306 2013-07-27 19:47:31

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
andy123 wrote:

I highly doubt that. It's ~00:30 here and I tested this a few hours ago, so I might be wrong but the driver is matched via tha pci id and that shouldn't change. Since I blacklisted the driver, I think I remember loading it manually and testing the drive, but I should probably go to bed, soon.

modinfo ahci
parm: marvell_enable:Marvell SATA via AHCI (1 = enabled) (int)

So you could try to add  ahci.marvell_enable=1 as a kernel parameter or to /etc/modprobe.d/marvel.conf, i also found http://forum.mandriva.com/en/viewtopic.php?t=120279, so yeah if that doesnt work i dunno

EDIT: from wikipedia:

88SE91xx chipsets Linux support

Linux SATA driver
Appears to be supported by the standard AHCI driver, however out of the box is not recognised by the AHCI driver - AHCI driver needs to be taught to be loaded for the relevant PCI vendor and product ID.

Check lspci -nnk for the PCI vendor and product ID. i.e. in the following instance the relevant part is "1b4b:9192"

04:00.0 RAID bus controller [0104]: Marvell Technology Group Ltd. Device [1b4b:9192] (re11)

To associate these IDs with the AHCI driver run

/bin/echo 1b4b 9192 > /sys/bus/pci/drivers/ahci/new_id

I read that wikipedia article a while back, but it doesn't work. With ahci.marvell_enable=1 the driver even gets loaded automatically, but it doesn't recognize any devices (at least with iommu on, but the pata-marvell driver works decent with iommu off, so why try to force the ahci?). Relevant dmesg (dmesg|egrep -i (ata|ahci|scsi))

[ 0.955526] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.990232] SCSI subsystem initialized [ 0.993158] ACPI: bus type ATA registered [ 0.993369] libata version 3.00 loaded. [ 0.994387] pata_acpi 0000:02:00.1: enabling device (0000 -> 0003) [ 1.238497] ahci 0000:00:11.0: version 3.0 [ 1.238729] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 1.238734] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 1.240134] scsi0 : ahci [ 1.240202] scsi1 : ahci [ 1.240277] scsi2 : ahci [ 1.240333] scsi3 : ahci [ 1.240418] scsi4 : ahci [ 1.240470] scsi5 : ahci [ 1.240513] ata1: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b100 irq 19 [ 1.240515] ata2: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b180 irq 19 [ 1.240517] ata3: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b200 irq 19 [ 1.240519] ata4: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b280 irq 19 [ 1.240521] ata5: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b300 irq 19 [ 1.240523] ata6: SATA max UDMA/133 abar m1024@0xfe70b000 port 0xfe70b380 irq 19 [ 1.240619] ahci 0000:02:00.0: irq 75 for MSI/MSI-X [ 1.240632] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1 [ 1.702088] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.702107] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.702123] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.703241] ata6.00: ATA-8: ST31000520AS, CC32, max UDMA/133 [ 1.703243] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 1.704090] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.704561] ata6.00: configured for UDMA/133 [ 1.705090] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 1.705109] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 1.705619] ata1.00: ATA-8: WDC WD10EADS-00L5B1, 01.01A01, max UDMA/133 [ 1.705620] ata1.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.706205] ata1.00: configured for UDMA/133 [ 1.706285] scsi 0:0:0:0: Direct-Access ATA WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 5 [ 1.708208] ata3.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133 [ 1.708210] ata3.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.710489] ata2.00: ATA-7: SAMSUNG HD103SI, 1AG01118, max UDMA7 [ 1.710491] ata2.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.711470] ata4.00: ATA-8: WDC WD20EARX-00PASB0, 51.0AB51, max UDMA/133 [ 1.711472] ata4.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA [ 1.713723] ata5.00: ATA-8: ADATA SP900, 5.0.2a, max UDMA/133 [ 1.713725] ata5.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 1.714324] ata3.00: configured for UDMA/133 [ 1.716961] ata2.00: configured for UDMA/133 [ 1.717032] scsi 1:0:0:0: Direct-Access ATA SAMSUNG HD103SI 1AG0 PQ: 0 ANSI: 5 [ 1.717202] scsi 2:0:0:0: Direct-Access ATA SAMSUNG HD204UI 1AQ1 PQ: 0 ANSI: 5 [ 1.718470] ata4.00: configured for UDMA/133 [ 1.718535] scsi 3:0:0:0: Direct-Access ATA WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5 [ 1.723710] ata5.00: configured for UDMA/133 [ 1.723771] scsi 4:0:0:0: Direct-Access ATA ADATA SP900 5.0. PQ: 0 ANSI: 5 [ 1.723902] scsi 5:0:0:0: Direct-Access ATA ST31000520AS CC32 PQ: 0 ANSI: 5 [ 2.248026] ahci 0000:02:00.0: controller reset failed (0x80000001) [ 2.248100] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode [ 2.248102] ahci 0000:02:00.0: flags: [ 2.248239] scsi6 : ahci [ 2.248291] ata7: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75 [ 2.557575] ata7: SATA link down (SStatus 0 SControl 300) [ 2.565869] sd 4:0:0:0: [sde] Attached SCSI disk [ 2.579963] sd 5:0:0:0: [sdf] Attached SCSI disk [ 2.585895] sd 1:0:0:0: [sdb] Attached SCSI disk [ 2.589825] sd 2:0:0:0: [sdc] Attached SCSI disk [ 3.039930] sd 3:0:0:0: [sdd] Attached SCSI disk [ 3.083346] sd 0:0:0:0: [sda] Attached SCSI disk [ 3.863782] scsi7 : pata_marvell [ 3.864187] scsi8 : pata_marvell [ 3.864574] ata8: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45 [ 3.864577] ata9: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45 [ 165.303195] ata9: soft resetting link [ 165.465217] ata9: EH complete [ 179.817277] ata8: soft resetting link [ 179.969106] ata8: EH complete [ 189.161840] ata7: hard resetting link [ 189.467441] ata7: SATA link down (SStatus 0 SControl 300) [ 189.467461] ata7: EH complete [ 315.833140] ata8: soft resetting link [ 315.984546] ata8: EH complete [ 318.974146] ata8: soft resetting link [ 319.125082] ata8: EH complete [ 328.019533] scsi9 : pata_marvell [ 328.019951] scsi10 : pata_marvell [ 328.020417] ata10: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45 [ 328.020426] ata11: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45 [ 373.887292] ata11: soft resetting link [ 374.048856] ata11: EH complete [ 381.099605] ata10: soft resetting link [ 381.250912] ata10: EH complete [ 409.101457] ahci 0000:02:00.0: irq 75 for MSI/MSI-X [ 409.101485] ahci 0000:02:00.0: forcing PORTS_IMPL to 0x1 [ 410.106469] ahci 0000:02:00.0: controller reset failed (0x80000003) [ 410.106481] ahci 0000:02:00.0: AHCI 0000.0000 1 slots 1 ports ? Gbps 0x1 impl SATA mode [ 410.106483] ahci 0000:02:00.0: flags: [ 410.106655] scsi11 : ahci [ 410.106706] ata12: SATA max UDMA/133 abar m2048@0xfe615000 port 0xfe615100 irq 75 [ 410.412017] ata12: SATA link down (SStatus 0 SControl 300) [ 427.452877] ata12: hard resetting link [ 427.757703] ata12: SATA link down (SStatus 0 SControl 300) [ 427.757721] ata12: EH complete [ 443.326795] scsi12 : pata_marvell [ 443.327090] scsi13 : pata_marvell [ 443.327348] ata13: PATA max UDMA/100 cmd 0xd040 ctl 0xd030 bmdma 0xd000 irq 45 [ 443.327354] ata14: PATA max UDMA/133 cmd 0xd020 ctl 0xd010 bmdma 0xd008 irq 45 [ 455.198604] ata13: soft resetting link [ 455.349916] ata13: EH complete [ 460.648310] ata14: soft resetting link [ 460.809941] ata14: EH complete

As you can see, I played around with the different drivers (pata-marvell loads normally now), but no devices are recognized. Windows VM still BSODs.

I'm unsure what your goal is meanwhile. If your goal is to get it passed through to the VM, I'll summarize what I did to get it working in Win7 without issues. If your goal is to get it working in Linux while IOMMU is enabled in the BIOS, I can't help.

My goal was to get it passed through to the VM, in AHCI mode of course, as the controller is completely useless in the host anyway due to, let's call it "the missing workaround" in the amd-iommu driver.
I didn't test the controller in IDE mode, because IDE emulation is worthless (I'd rather pass through a disk attached to my internal 5-port JMB393 controller than one in IDE emulation mode on the Marvell SATA3 controller) - I do not even have the pata-marvell driver enabled in my kernel config for that reason. So I

* set the controller to AHCI mode in UEFI/BIOS (actually no UEFI here, the Gigabyte has a legacy BIOS mode)
* added its vendor:id to pci-stub.ids boot parameter (as said, it's completely useless for the host anyways, so make the AHCI driver ignore it completely)
* passed it through to the qemu VM using "-device vfio-pci,host=${SATA_DEVICE},bus=pcie.0" (actually it shows up with two entries in lscpi, one of wich seems to be for the E-SATA ports (they also do not work in the Linux host, same issues))
* Win7 automatically installed some ATA drivers for it (I didn't download anything anywhere)
* it works, without issues

Why is it that you refuse to bind it to pci-stub and instead mess around with blacklisting drivers in different modes? I don't get it, please excuse me if I just missed your point or got something wrong from your posts.

Last edited by teekay (2013-07-27 20:00:40)

Offline

#307 2013-07-27 20:18:09

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

I'm unsure what your goal is meanwhile. If your goal is to get it passed through to the VM, I'll summarize what I did to get it working in Win7 without issues. If your goal is to get it working in Linux while IOMMU is enabled in the BIOS, I can't help.

My goal was to get it passed through to the VM, in AHCI mode of course, as the controller is completely useless in the host anyway due to, let's call it "the missing workaround" in the amd-iommu driver.
I didn't test the controller in IDE mode, because IDE emulation is worthless (I'd rather pass through a disk attached to my internal 5-port JMB393 controller than one in IDE emulation mode). I do not even have the pata-marvell driver enabled in my kernel config for that reason. To sum it up, the marvell controller is completely useless for the host OS here. So I

* set the controller to AHCI mode in UEFI/BIOS (actually no UEFI here, the Gigabyte has a legacy BIOS mode)
* added its vendor:id to pci-stub.ids boot parameter (as said, it's completely useless for the host anyways, so make the AHCI driver ignore it completely)
* passed it through to the qemu VM using "-device vfio-pci,host=${SATA_DEVICE},bus=pcie.0" (actually it shows up with two entries in lscpi, one of wich seems to be for the E-SATA ports (they also do not work in the Linux host, same issues))
* Win7 automatically installed some ATA drivers for it (I didn't download anything anywhere)
* it works, without issues

Why is it that you refuse to bind it to pci-stub and instead mess around with blacklisting drivers in different modes? I don't get it, please excuse my ignorance.

My goal is to get it passed through to the VM, but IOMMU has to be enabled in order to to that. I play around with the controller on the host, because I can't imagine, that the controller will work in a guest, bot not work on the host, while it works perfectly fine on the host with IOMMU disabled, but refuses to work with IOMMU enabled. So why should it work in the guest?
Also, where's the difference between loading no driver at all and binding to pci-stub? While pata-marvell was blacklisted and I hadn't added ahci.marvell_enable=1 to my kernel parameters no driver got loaded (see lspci -vvvnn earlier…).
I'll try pci-stub later, but we don't have the exact same controller (both are buggy, but maybe in different ways) and boards from a different manufacturer, so different things might work or not.
Also, the Windows installer doesn't even find drives on mine without loading a special driver from the mainboard dvd, was that the same for yours? My intention is to boot with this controller from a SSD, because trim and stuff might be strange through virtio.

Note about the E-SATA ports: I think they are handled by the same controller, arent they?


i'm sorry for my poor english wirting skills…

Offline

#308 2013-07-27 20:22:20

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
GizmoChicken wrote:

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

If they're the same vendor yeah you'll have problems with with pci stub, what you can do is flash one card and modify the vendor id, or buy another one from a different vendor.

I was afraid of that.  Don't know if my borderline obsessive-compulsive disorder would permit me to use different venders. :-)  That, and as much as I am loathe to compliment Asus, their version probably has among best cooling fins for the price.  I'll look into the possibility of flashing one card with a modified vendor ID.

It's a shame that we can't use pciback, in place of pci-stub, to grab the GPU before a driver binds it.  I found an outdated Xen wiki entry discussing how pciback could be loaded (presumably in a Xen environment) as module if not recompiled in the kernel.

I don't suppose you've heard of anyone getting the pciback module (or a derivative thereof) to load outside the Xen environment?

Offline

#309 2013-07-27 20:38:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:
GizmoChicken wrote:

QUESTION:  Although it's a bit slow, I really like that my GT610 is fanless, and therefore silent.  And so I'm thinking that I might like a matching GT610 for use as a primary card on my host.  But I'm wondering, if my primary and secondary cards are identical (both GT610s), would I be able to use pci-stub to bind selectively my secondary card?   That is, in the case where the primary and secondary cards (and presumably their identifies) are identical, would there be some means by which pci-stub could distinguish the two cards, such as by pci-slot location?

If they're the same vendor yeah you'll have problems with with pci stub, what you can do is flash one card and modify the vendor id, or buy another one from a different vendor.

I was afraid of that.  Don't know if my borderline obsessive-compulsive disorder would permit me to use different venders. :-)  That, and as much as I am loathe to compliment Asus, their version probably has among best cooling fins for the price.  I'll look into the possibility of flashing one card with a modified vendor ID.

It's a shame that we can't use pciback, in place of pci-stub, to grab the GPU before a driver binds it.  I found an outdated Xen wiki entry discussing how pciback could be loaded (presumably in a Xen environment) as module if not recompiled in the kernel.

I don't suppose you've heard of anyone getting the pciback module (or a derivative thereof) to load outside the Xen environment?

Well now that i think about it it might work if you bind it manually to the pci-stub module but to avoid problems just flash the bios

Offline

#310 2013-07-27 21:25:51

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

My goal is to get it passed through to the VM, but IOMMU has to be enabled in order to to that. I play around with the controller on the host, because I can't imagine, that the controller will work in a guest, bot not work on the host, while it works perfectly fine on the host with IOMMU disabled, but refuses to work with IOMMU enabled. So why should it work in the guest?

Well, my only explanation is: it's the driver that fails. My board has some Etron USB3 controller. It works perfectly fine in Linux, but the driver for Windows 7 caused BSOD gallore, no matter if it's in a VM or a "real" install. So that's a counter example for that Marvell controller which, if IOMMU is enabled, totally fails in Linux, but works fine in Windows. You did find the bug report and the post about the intel-iommu driver having a workaround, but not the amd-iommu one. (I found the same posts when I was looking for the problem)

Also, where's the difference between loading no driver at all and binding to pci-stub? While pata-marvell was blacklisted and I hadn't added ahci.marvell_enable=1 to my kernel parameters no driver got loaded (see lspci -vvvnn earlier…).

I have no idea, but my thinking was: if it (currently) doesn't work anyway in the host and just causes page fault and what not, and there is a way to make the host ignore it before any driver even tries to act on it (disabling AHCI driver is no option), I'll use it, Yay for pci-stub.

I'll try pci-stub later, but we don't have the exact same controller (both are buggy, but maybe in different ways) and boards from a different manufacturer, so different things might work or not.

I realized that from your lscpi output, yes. But as they seem to act almost the same in the Linux, it's worth a try for you.

Also, the Windows installer doesn't even find drives on mine without loading a special driver from the mainboard dvd, was that the same for yours? My intention is to boot with this controller from a SSD, because trim and stuff might be strange through virtio.

I haven't tried that. My Windows VM install is actually a converted VirtualBox image, with vbox tools uninstalled, virtio drivers added, and the install itself now resides on a qcow image. The Marvell SATA only provides a data disk inside the VM. After I added it in Qemu, Windows automatically installed drivers for it, and it "just worked"™

Note about the E-SATA ports: I think they are handled by the same controller, arent they?

The same driver, yes (ahci if running in AHCI mode), but on my board the marvell exhibits itself as 2 separate controllers in lspci, and I _believe_ one of them is for the e-sata ports, I haven't verified that though (I replaced my external RAID bay with a larger case and an internal port multiplier, that external IcyDock thing was way too noisy). I pci-stub'ed both (as the have same vendor and id), but maybe I just passed through the right one by pure luck.

Last edited by teekay (2013-07-27 21:34:40)

Offline

#311 2013-07-27 21:39:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Offline

#312 2013-07-28 11:03:38

pcworld
Member
From: Germany
Registered: 2013-07-28
Posts: 6
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

I'm now in windows, but I still can't boot with "-vga none", so I have two graphics cards and Windows picks the wrong one.

Does that mean VGA passthrough worked for the GTX 650 (as the guest)?

nbhs wrote:
Kamek wrote:

Is there a way to share a single GPU between the host and the guest ? I only have one GTX 670 in my rig, but I'm sick and tired of rebooting every time I want to play a game.

Not using this method, if you had a radeon card you could use kvm pci-assign and pass it as secondary but on NV cards this is the only reliable method (unless you're willing to try xen, and its counteless unreliable hacks)

Is there any other method that works with only one Nvidia card and KVM? It seems Xen works with only one graphics card, when you boot the VM your graphics card is passed through to the guest system. (If Xen worked properly with Nvidia cards, that is.)
The host system could still be controlled via VNC/X Forwarding or similarly, and when I shut down the guest system I'd like the graphics card to be reassigned to my host system.

I'm considering buying a GTX 650 Ti Boost, since many (fanboys?) say Nvidia's proprietary Linux drivers were much better than fglrx (I don't know if it's true).
On the CPU side, the AMD FX-8350 and Intel i7-2600 are on my list, though I'm rather leaning towards the AMD one since it's cheaper and officially supports overclocking. Advantage of the i7 would be its integrated GPU, but apparently it's not too reliable if it's your primary GPU and you want to forward/VGA-passthrough the other one.

Edit: Of course, great thanks for your tutorial! I've come across many VGA-passthrough threads (mostly concering Xen), and this one seems to be one of very few that are not dead (yet).

Last edited by pcworld (2013-07-28 11:09:37)

Offline

#313 2013-07-28 13:41:46

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Well assuming you have 2 cards (the IGP and the radeon) the IGP should always be the primary one since you cant pass it through and passing through your primary card isnt supported, are you using my builds i posted on the main guide(kernel, qemu and seabios)?

- I have the SeaBIOS 1.7.2 w/ patches (extension dirty? is that the right one?)
- Is there a patch for Kernel 3.9? I can't update to 3.10 as it breaks DKMS, and ZFS-DKMS runs my primary storage
- Do I need a patch Qemu or is the latest git version already updated for VFIO-reset? If not, which version do I clone to patch?

I apologize for putting my noobishness on front row here, but if anyone has a command set for pulling down and patching the kernel/qemu, I'd much appreciate it.

Last edited by mukiex (2013-07-28 14:22:42)

Offline

#314 2013-07-28 14:34:48

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

Last edited by myweb (2013-07-28 14:38:48)

Offline

#315 2013-07-28 19:06:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

Hm,

I have builded new kernel, but can't execute qemu:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

But do you see any output on your passthrough'd card? what seabios version are you using? can you get more info about the BSOD?

Last edited by nbhs (2013-07-28 19:16:06)

Offline

#316 2013-07-28 19:08:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:
nbhs wrote:

Well assuming you have 2 cards (the IGP and the radeon) the IGP should always be the primary one since you cant pass it through and passing through your primary card isnt supported, are you using my builds i posted on the main guide(kernel, qemu and seabios)?

- I have the SeaBIOS 1.7.2 w/ patches (extension dirty? is that the right one?)
- Is there a patch for Kernel 3.9? I can't update to 3.10 as it breaks DKMS, and ZFS-DKMS runs my primary storage
- Do I need a patch Qemu or is the latest git version already updated for VFIO-reset? If not, which version do I clone to patch?

I apologize for putting my noobishness on front row here, but if anyone has a command set for pulling down and patching the kernel/qemu, I'd much appreciate it.

Right now you'll need a patched seabios, patched kernel, and patched qemu until they get merged upstream, using archlinux you can use my builds, you can fetch the kernel and qemu from alex williamson's trees https://github.com/awilliam/qemu-vfio and https://github.com/awilliam/linux-vfio branch vga-current

Last edited by nbhs (2013-07-28 19:13:20)

Offline

#317 2013-07-28 19:09:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pcworld wrote:
andy123 wrote:

I'm now in windows, but I still can't boot with "-vga none", so I have two graphics cards and Windows picks the wrong one.

Does that mean VGA passthrough worked for the GTX 650 (as the guest)?

nbhs wrote:
Kamek wrote:

Is there a way to share a single GPU between the host and the guest ? I only have one GTX 670 in my rig, but I'm sick and tired of rebooting every time I want to play a game.

Not using this method, if you had a radeon card you could use kvm pci-assign and pass it as secondary but on NV cards this is the only reliable method (unless you're willing to try xen, and its counteless unreliable hacks)

Is there any other method that works with only one Nvidia card and KVM? It seems Xen works with only one graphics card, when you boot the VM your graphics card is passed through to the guest system. (If Xen worked properly with Nvidia cards, that is.)
The host system could still be controlled via VNC/X Forwarding or similarly, and when I shut down the guest system I'd like the graphics card to be reassigned to my host system.

I'm considering buying a GTX 650 Ti Boost, since many (fanboys?) say Nvidia's proprietary Linux drivers were much better than fglrx (I don't know if it's true).
On the CPU side, the AMD FX-8350 and Intel i7-2600 are on my list, though I'm rather leaning towards the AMD one since it's cheaper and officially supports overclocking. Advantage of the i7 would be its integrated GPU, but apparently it's not too reliable if it's your primary GPU and you want to forward/VGA-passthrough the other one.

Edit: Of course, great thanks for your tutorial! I've come across many VGA-passthrough threads (mostly concering Xen), and this one seems to be one of very few that are not dead (yet).

You'll need 2 cards one for the host, and one for the vm, right now amd cpus seems more successful at getting it working

Last edited by nbhs (2013-07-28 19:16:49)

Offline

#318 2013-07-28 21:01:51

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:
nbhs wrote:

Try these, it will fetch and build Alex's qemu and linux trees, branch 'vga-current'

linux-vga-current.tar.gz
qemu-vga-current.tar.gz

they build just fine, i havent tried anything else though, you'll also need dtc-git: https://aur.archlinux.org/packages/dtc-git/

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

But do you see any output on your passthrough'd card? what seabios version are you using? can you get more info about the BSOD?

I see windows installation loading logo on passthrough'd card and then BSOD on passthrough'd card. BSOD: ACPI_BIOS_ERROR
SeaBIOS compiled from git: 1.7.3-18-g7093aa5

Also I see the following message in console when starting QEMU:
Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.

Last edited by myweb (2013-07-28 21:03:44)

Offline

#319 2013-07-28 21:26:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

Thank you, I have builded and installed the files, but still get:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

I have tried "modprobe kvm", but it does not help

modprobe kvm-intel solved the issue, but loading windows installation is failed with BSOD (your PC run into problem and need to reboot)

But do you see any output on your passthrough'd card? what seabios version are you using? can you get more info about the BSOD?

I see windows installation loading logo on passthrough'd card and then BSOD on passthrough'd card. BSOD: ACPI_BIOS_ERROR
SeaBIOS compiled from git: 1.7.3-18-g7093aa5

Also I see the following message in console when starting QEMU:
Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.

The acpi bios thing its because of seabios, use my package (1.7.2 patched), the other message is likely because you're using the wrong kernel without pcie bus reset support

Last edited by nbhs (2013-07-28 21:27:07)

Offline

#320 2013-07-28 21:33:14

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying to get this working for days now.. works on Xen but it's not usable due to all the crashing and poor performance.

I've had enough of my intel board. Looking to buy a AM3+ gigabyte board... anyone have any suggestions that works with this?

EDIT: Also, I would like the integrated AMD graphics to work on the host.

Last edited by BulliteShot (2013-07-28 21:34:23)

Offline

#321 2013-07-28 21:37:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

I've been trying to get this working for days now.. works on Xen but it's not usable due to all the crashing and poor performance.

I've had enough of my intel board. Looking to buy a AM3+ gigabyte board... anyone have any suggestions that works with this?

EDIT: Also, I would like the integrated AMD graphics to work on the host.

Have you tried this guide on your intel build?

Offline

#322 2013-07-28 21:45:02

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I'm getting the same problem as a guy a few pages back... "This device can not find enough free resources that it can use. (Code 12)"... apparently something to do with using the intel integrated graphics. I read the post that was linked but it seems like a dead end?

- I can't get the graphics card to show the BIOS.
- If I boot the ArchLinux ISO in the VM then it shows it's loading screen on the graphics card.
- Windows shows the device in task manager and AMD installs the catalyst drivers.

[root@localhost ~]# dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.021443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.021447] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.021516] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.297980] IOMMU 0 0xfed90000: using Queued invalidation [ 0.297981] IOMMU 1 0xfed91000: using Queued invalidation [ 0.436381] IOMMU: software identity mapping for device 0000:00:00.0 [ 0.436382] IOMMU: software identity mapping for device 0000:00:01.0 [ 0.436383] IOMMU: software identity mapping for device 0000:00:14.0 [ 0.436384] IOMMU: software identity mapping for device 0000:00:16.0 [ 0.436385] IOMMU: software identity mapping for device 0000:00:1a.0 [ 0.436386] IOMMU: software identity mapping for device 0000:00:1b.0 [ 0.436387] IOMMU: software identity mapping for device 0000:00:1c.0 [ 0.436388] IOMMU: software identity mapping for device 0000:00:1c.2 [ 0.436389] IOMMU: software identity mapping for device 0000:00:1d.0 [ 0.436390] IOMMU: software identity mapping for device 0000:00:1f.0 [ 0.436391] IOMMU: software identity mapping for device 0000:00:1f.2 [ 0.436392] IOMMU: software identity mapping for device 0000:00:1f.3 [ 0.436396] IOMMU: software identity mapping for device 0000:01:00.0 [ 0.436397] IOMMU: software identity mapping for device 0000:01:00.1 [ 0.436401] IOMMU: software identity mapping for device 0000:02:00.0 [ 0.436402] IOMMU: Setting RMRR: [ 0.436410] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff] [ 0.436696] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9d82000 - 0xc9d8efff] [ 0.436701] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9d82000 - 0xc9d8efff] [ 0.436702] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9d82000 - 0xc9d8efff] [ 0.436704] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.436705] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [root@localhost ~]# 
[root@localhost ~]# dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.705428] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705438] pci-stub 0000:01:00.0: claimed by stub [ 0.705442] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705448] pci-stub 0000:01:00.1: claimed by stub [root@localhost ~]# 
[ 60.412232] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 60.438127] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 60.438134] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 2 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive if=none,format=raw,file=/root/win_os.raw,id=disk \ -device scsi-hd,drive=disk \ -net nic,model=virtio \ -net tap,ifname=vm-win,script=no,downscript=no

Offline

#323 2013-07-28 21:50:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

Yes, I'm getting the same problem as a guy a few pages back... "This device can not find enough free resources that it can use. (Code 12)"... apparently something to do with using the intel integrated graphics. I read the post that was linked but it seems like a dead end?

- I can't get the graphics card to show the BIOS.
- If I boot the ArchLinux ISO in the VM then it shows it's loading screen on the graphics card.
- Windows shows the device in task manager and AMD installs the catalyst drivers.

[root@localhost ~]# dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.021443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.021447] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.021516] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.297980] IOMMU 0 0xfed90000: using Queued invalidation [ 0.297981] IOMMU 1 0xfed91000: using Queued invalidation [ 0.436381] IOMMU: software identity mapping for device 0000:00:00.0 [ 0.436382] IOMMU: software identity mapping for device 0000:00:01.0 [ 0.436383] IOMMU: software identity mapping for device 0000:00:14.0 [ 0.436384] IOMMU: software identity mapping for device 0000:00:16.0 [ 0.436385] IOMMU: software identity mapping for device 0000:00:1a.0 [ 0.436386] IOMMU: software identity mapping for device 0000:00:1b.0 [ 0.436387] IOMMU: software identity mapping for device 0000:00:1c.0 [ 0.436388] IOMMU: software identity mapping for device 0000:00:1c.2 [ 0.436389] IOMMU: software identity mapping for device 0000:00:1d.0 [ 0.436390] IOMMU: software identity mapping for device 0000:00:1f.0 [ 0.436391] IOMMU: software identity mapping for device 0000:00:1f.2 [ 0.436392] IOMMU: software identity mapping for device 0000:00:1f.3 [ 0.436396] IOMMU: software identity mapping for device 0000:01:00.0 [ 0.436397] IOMMU: software identity mapping for device 0000:01:00.1 [ 0.436401] IOMMU: software identity mapping for device 0000:02:00.0 [ 0.436402] IOMMU: Setting RMRR: [ 0.436410] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff] [ 0.436696] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9d82000 - 0xc9d8efff] [ 0.436701] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9d82000 - 0xc9d8efff] [ 0.436702] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9d82000 - 0xc9d8efff] [ 0.436704] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.436705] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [root@localhost ~]# 
[root@localhost ~]# dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.705428] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705438] pci-stub 0000:01:00.0: claimed by stub [ 0.705442] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705448] pci-stub 0000:01:00.1: claimed by stub [root@localhost ~]# 
[ 60.412232] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 60.438127] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 60.438134] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 2 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive if=none,format=raw,file=/root/win_os.raw,id=disk \ -device scsi-hd,drive=disk \ -net nic,model=virtio \ -net tap,ifname=vm-win,script=no,downscript=no

Are you using the lastest builds i posted?  iommu=pt is an amd thing only (i think for intel is intel_iommu=pt) and i have no idea what igfx_on does, it seems myweb has partially got it to work on his intel board (except the bsod which im sure its a seabios thing), perhaps he knows more since i dont have an intel board to test

Offline

#324 2013-07-28 21:55:33

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

The acpi bios thing its because of seabios, use my package (1.7.2 patched), the other message is likely because you're using the wrong kernel without pcie bus reset support

Yes, You are right - I have no BSOD with 1.7.2 but I could not apply kernel-vfio-vga-reset.patch to Linux kernel from vga-current branch: I get

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

Last edited by myweb (2013-07-28 21:58:34)

Offline

#325 2013-07-28 22:09:50

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:

Dear All,

I am still unable get Windows running with ATI HD7750 card:
I could pass first part of windows installation (partitionig and files
copying), but then VM reboots and I get black screen.
Alex Williamson proposed to use "vga-current" branch of :
git://github.com/awilliam/qemu-vfio.git
git://github.com/awilliam/linux-vfio.git

Could you please check if I created right PKGBUILD files?

Linux vfio: http://www.fileswap.com/dl/K7s93euILN/
Qemu vfio: http://www.fileswap.com/dl/Oie83lou8B/

So it does work at some point but after doing a reboot it doesnt work anymore?

Yes, You are right but it does not work anymore until VM is not restarted (qemu).
But when I restart Qemu and boot to windows, windows says that installation was unexpectedly interrupted and asks to reinstall.

Offline

#326 2013-07-28 22:12:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:

The acpi bios thing its because of seabios, use my package (1.7.2 patched), the other message is likely because you're using the wrong kernel without pcie bus reset support

Yes, You are right - I have no BSOD with 1.7.2 but I could not apply kernel-vfio-vga-reset.patch to Linux kernel from vga-current branch: I get

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

You dont need to patch that kernel, i extracted the commits from that branch and use it against vanilla kernel, and it probably doesnt work anymore after you reboot the vm because you're using the wrong kernel (it cant reset the card), i posted a qemu and kernel builds for you on page 13 you can use that or build it yourself

Last edited by nbhs (2013-07-28 22:20:23)

Offline

#327 2013-07-28 22:15:38

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
BulliteShot wrote:

Yes, I'm getting the same problem as a guy a few pages back... "This device can not find enough free resources that it can use. (Code 12)"... apparently something to do with using the intel integrated graphics. I read the post that was linked but it seems like a dead end?

- I can't get the graphics card to show the BIOS.
- If I boot the ArchLinux ISO in the VM then it shows it's loading screen on the graphics card.
- Windows shows the device in task manager and AMD installs the catalyst drivers.

[root@localhost ~]# dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.021443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.021447] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.021516] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.297980] IOMMU 0 0xfed90000: using Queued invalidation [ 0.297981] IOMMU 1 0xfed91000: using Queued invalidation [ 0.436381] IOMMU: software identity mapping for device 0000:00:00.0 [ 0.436382] IOMMU: software identity mapping for device 0000:00:01.0 [ 0.436383] IOMMU: software identity mapping for device 0000:00:14.0 [ 0.436384] IOMMU: software identity mapping for device 0000:00:16.0 [ 0.436385] IOMMU: software identity mapping for device 0000:00:1a.0 [ 0.436386] IOMMU: software identity mapping for device 0000:00:1b.0 [ 0.436387] IOMMU: software identity mapping for device 0000:00:1c.0 [ 0.436388] IOMMU: software identity mapping for device 0000:00:1c.2 [ 0.436389] IOMMU: software identity mapping for device 0000:00:1d.0 [ 0.436390] IOMMU: software identity mapping for device 0000:00:1f.0 [ 0.436391] IOMMU: software identity mapping for device 0000:00:1f.2 [ 0.436392] IOMMU: software identity mapping for device 0000:00:1f.3 [ 0.436396] IOMMU: software identity mapping for device 0000:01:00.0 [ 0.436397] IOMMU: software identity mapping for device 0000:01:00.1 [ 0.436401] IOMMU: software identity mapping for device 0000:02:00.0 [ 0.436402] IOMMU: Setting RMRR: [ 0.436410] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff] [ 0.436696] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9d82000 - 0xc9d8efff] [ 0.436701] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9d82000 - 0xc9d8efff] [ 0.436702] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9d82000 - 0xc9d8efff] [ 0.436704] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.436705] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [root@localhost ~]# 
[root@localhost ~]# dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.705428] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705438] pci-stub 0000:01:00.0: claimed by stub [ 0.705442] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705448] pci-stub 0000:01:00.1: claimed by stub [root@localhost ~]# 
[ 60.412232] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 60.438127] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 60.438134] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 2 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive if=none,format=raw,file=/root/win_os.raw,id=disk \ -device scsi-hd,drive=disk \ -net nic,model=virtio \ -net tap,ifname=vm-win,script=no,downscript=no

Are you using the lastest builds i posted?  iommu=pt is an amd thing only (i think for intel is intel_iommu=pt) and i have no idea what igfx_on does, it seems myweb has partially got it to work on his intel board (except the bsod which im sure its a seabios thing), perhaps he knows more since i dont have an intel board to test

I just managed to get it working (using secondary graphics)

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host \ -smp 2 \ -bios /usr/share/qemu/bios.bin \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1

For anyone interested in my hardware;
Mobo: B75-D3V, BIOS F8 04/17/2013 (The integrated LAN on this board is utter crap... maybe I have a dodgy board but i'm using an Intel e1000e PCI LAN card to avoid a mountain of problems)
CPU: i5-3570
GPU: Radeon HD 7950 Gigabyte MSI Twin Fozr III 3GB

It's a shame I can't get the BIOS level working though. If switching to an AMD board would work I would consider it well worthwhile. I just want to make sure I buy one which is confirmed to be working as "upgrading" once will cost enough xD

I compiled the sources using your scripts today so I think i'm using the latest versions.

[root@localhost ~]# qemu-system-x86_64 --version QEMU emulator version 1.5.1, Copyright (c) 2003-2008 Fabrice Bellard [root@localhost ~]# uname -a Linux localhost 3.10.1-1-mainline #1 SMP PREEMPT Sun Jul 28 17:04:33 IST 2013 x86_64 GNU/Linux

Don't know how to check the seabios version, but I did use your script to build it.

I've tried using a lot of kernel param combinations but I see no difference. Now i'm using "pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on,pass-through,pt" but it doesn't look right to me! XD

Last edited by BulliteShot (2013-07-28 22:16:21)

Offline

#328 2013-07-28 22:18:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:
nbhs wrote:
BulliteShot wrote:

Yes, I'm getting the same problem as a guy a few pages back... "This device can not find enough free resources that it can use. (Code 12)"... apparently something to do with using the intel integrated graphics. I read the post that was linked but it seems like a dead end?

- I can't get the graphics card to show the BIOS.
- If I boot the ArchLinux ISO in the VM then it shows it's loading screen on the graphics card.
- Windows shows the device in task manager and AMD installs the catalyst drivers.

[root@localhost ~]# dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.021443] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.021447] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.021516] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.297980] IOMMU 0 0xfed90000: using Queued invalidation [ 0.297981] IOMMU 1 0xfed91000: using Queued invalidation [ 0.436381] IOMMU: software identity mapping for device 0000:00:00.0 [ 0.436382] IOMMU: software identity mapping for device 0000:00:01.0 [ 0.436383] IOMMU: software identity mapping for device 0000:00:14.0 [ 0.436384] IOMMU: software identity mapping for device 0000:00:16.0 [ 0.436385] IOMMU: software identity mapping for device 0000:00:1a.0 [ 0.436386] IOMMU: software identity mapping for device 0000:00:1b.0 [ 0.436387] IOMMU: software identity mapping for device 0000:00:1c.0 [ 0.436388] IOMMU: software identity mapping for device 0000:00:1c.2 [ 0.436389] IOMMU: software identity mapping for device 0000:00:1d.0 [ 0.436390] IOMMU: software identity mapping for device 0000:00:1f.0 [ 0.436391] IOMMU: software identity mapping for device 0000:00:1f.2 [ 0.436392] IOMMU: software identity mapping for device 0000:00:1f.3 [ 0.436396] IOMMU: software identity mapping for device 0000:01:00.0 [ 0.436397] IOMMU: software identity mapping for device 0000:01:00.1 [ 0.436401] IOMMU: software identity mapping for device 0000:02:00.0 [ 0.436402] IOMMU: Setting RMRR: [ 0.436410] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff] [ 0.436696] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9d82000 - 0xc9d8efff] [ 0.436701] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9d82000 - 0xc9d8efff] [ 0.436702] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9d82000 - 0xc9d8efff] [ 0.436704] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.436705] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [root@localhost ~]# 
[root@localhost ~]# dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on iommu=pt [ 0.705428] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705438] pci-stub 0000:01:00.0: claimed by stub [ 0.705442] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.705448] pci-stub 0000:01:00.1: claimed by stub [root@localhost ~]# 
[ 60.412232] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 60.438127] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 60.438134] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 2 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive if=none,format=raw,file=/root/win_os.raw,id=disk \ -device scsi-hd,drive=disk \ -net nic,model=virtio \ -net tap,ifname=vm-win,script=no,downscript=no

Are you using the lastest builds i posted?  iommu=pt is an amd thing only (i think for intel is intel_iommu=pt) and i have no idea what igfx_on does, it seems myweb has partially got it to work on his intel board (except the bsod which im sure its a seabios thing), perhaps he knows more since i dont have an intel board to test

I just managed to get it working (using secondary graphics)

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host \ -smp 2 \ -bios /usr/share/qemu/bios.bin \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1

For anyone interested in my hardware;
Mobo: B75-D3V, BIOS F8 04/17/2013 (The integrated LAN on this board is utter crap... maybe I have a dodgy board but i'm using an Intel e1000e PCI LAN card to avoid a mountain of problems)
CPU: i5-3570
GPU: Radeon HD 7950 Gigabyte MSI Twin Fozr III 3GB

It's a shame I can't get the BIOS level working though. If switching to an AMD board would work I would consider it well worthwhile. I just want to make sure I buy one which is confirmed to be working as "upgrading" once will cost enough xD

I compiled the sources using your scripts today so I think i'm using the latest versions.

[root@localhost ~]# qemu-system-x86_64 --version QEMU emulator version 1.5.1, Copyright (c) 2003-2008 Fabrice Bellard [root@localhost ~]# uname -a Linux localhost 3.10.1-1-mainline #1 SMP PREEMPT Sun Jul 28 17:04:33 IST 2013 x86_64 GNU/Linux

Don't know how to check the seabios version, but I did use your script to build it.

I've tried using a lot of kernel param combinations but I see no difference. Now i'm using "pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on,igfx_on,pass-through,pt" but it doesn't look right to me! XD

Yes, passing through a radeon card as secondary should work (wont work with nvidia though) but'll probably experience performance degradation/host crashes, if you do, you can solve it using the "eject" method i posted in the guide

Last edited by nbhs (2013-07-28 22:22:27)

Offline

#329 2013-07-28 22:23:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@BulliteShot
I also just realized you forgot to add "-vga none" using vfio, it wont work without it, also just use intel_iommu=on

Last edited by nbhs (2013-07-28 22:26:10)

Offline

#330 2013-07-28 22:37:55

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:
nbhs wrote:

The acpi bios thing its because of seabios, use my package (1.7.2 patched), the other message is likely because you're using the wrong kernel without pcie bus reset support

Yes, You are right - I have no BSOD with 1.7.2 but I could not apply kernel-vfio-vga-reset.patch to Linux kernel from vga-current branch: I get

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

You dont need to patch that kernel, i extracted the commits from that branch and use it against vanilla kernel, and it probably doesnt work anymore after you reboot the vm because you're using the wrong kernel (it cant reset the card), i posted a qemu and kernel builds for you on page 13 you can use that or build it yourself

I use kernel and qemu compiled from your builds posted on page 13. Also I get the same behaviour (black screen after windows reboot) with kernel and qemu compiled from builds posted on the first page: I did not get any warnings about reset, but saw black screen when windows reboot.

Last edited by myweb (2013-07-28 22:40:20)

Offline

#331 2013-07-28 22:38:29

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nbhs
Yes, the host does freeze when the VM is rebooted. I tried the eject script you provided and it works. Thank you.

I just tried rebooted to try only "intel_iommu=on" and adding "-vga none". Unfortunately I just see the Qemu console and no input from the graphics card.

Going to try using the GPU ROM now. I tried it earlier but I can't remember if I used -vga none"

Last edited by BulliteShot (2013-07-28 22:38:54)

Offline

#332 2013-07-28 22:39:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

Yes, You are right - I have no BSOD with 1.7.2 but I could not apply kernel-vfio-vga-reset.patch to Linux kernel from vga-current branch: I get

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

You dont need to patch that kernel, i extracted the commits from that branch and use it against vanilla kernel, and it probably doesnt work anymore after you reboot the vm because you're using the wrong kernel (it cant reset the card), i posted a qemu and kernel builds for you on page 13 you can use that or build it yourself

I use kernel and qemu compiled from your builds posted on page 13. Also I get the same behaviour (black screen after windows reboot) with kernel and qemu builder from builds posted on the first page: I did not get any warnings about reset, but saw black screen when windows reboot.

have you tried:

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

Offline

#333 2013-07-28 22:43:58

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

@nbhs
Yes, the host does freeze when the VM is rebooted. I tried the eject script you provided and it works. Thank you.

I just tried rebooted to try only "intel_iommu=on" and adding "-vga none". Unfortunately I just see the Qemu console and no input from the graphics card.

Going to try using the GPU ROM now. I tried it earlier but I can't remember if I used -vga none"

well there's not much benefits from passing it through as primary or secondary other than seeing the seabios boot messages and having to use the eject script thingy

Offline

#334 2013-07-28 22:45:46

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Myweb
You could also try loading the bios rom from a dump too

Last edited by nbhs (2013-07-28 22:47:32)

Offline

#335 2013-07-28 22:53:25

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:
nbhs wrote:

You dont need to patch that kernel, i extracted the commits from that branch and use it against vanilla kernel, and it probably doesnt work anymore after you reboot the vm because you're using the wrong kernel (it cant reset the card), i posted a qemu and kernel builds for you on page 13 you can use that or build it yourself

I use kernel and qemu compiled from your builds posted on page 13. Also I get the same behaviour (black screen after windows reboot) with kernel and qemu builder from builds posted on the first page: I did not get any warnings about reset, but saw black screen when windows reboot.

have you tried:

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

I have tried, but it does not help.

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0
nbhs wrote:

@Myweb
You could also try loading the bios rom from a dump too

Do you mean VGA bios (romfile)?

Is it correct, that at first I need to fix: Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.?

Last edited by myweb (2013-07-28 22:56:14)

Offline

#336 2013-07-28 22:56:34

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nbhs
Yes, here is my script running on root.

vfio-bind 0000:01:00.0 0000:01:00.1 modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

The reason I am after primary pass-through is because I'm waiting for Windows to BSOD in the background, reset and then crash the entire system >_< I suppose I can disable automatic rebooting when a BSOD happens but I don't know how reliable that is.

If anyone else has a 7950 running on primary pass-through please let me know what CPU & Motherboard you're using.

By the way - anyone who is using a KVM or the cirrus VGA to control the mouse and keyboard take a look at Synergy. It works using the network and you can define a hotkey to switch between hosts and copy the clipboard.

Offline

#337 2013-07-28 23:06:32

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It looks like I was wrong with description of issue:
Windows installation starts, copy files, reboot VM, Windows installation continue, I see message Getting device ready 0...100% and then screen switched to black.
Using VGA bios (romfile) does not help.

Offline

#338 2013-07-28 23:06:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

I use kernel and qemu compiled from your builds posted on page 13. Also I get the same behaviour (black screen after windows reboot) with kernel and qemu builder from builds posted on the first page: I did not get any warnings about reset, but saw black screen when windows reboot.

have you tried:

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

I have tried, but it does not help.

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0
nbhs wrote:

@Myweb
You could also try loading the bios rom from a dump too

Do you mean VGA bios (romfile)?

Is it correct, that at first I need to fix: Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.?

Yes your vga romfile, could you post your uname -a? using the packages posted on the first page does not produce this warning message correct?

Offline

#339 2013-07-28 23:07:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

It looks like I was wrong with description of issue:
Windows installation starts, copy files, reboot VM, Windows installation continue, I see message Getting device ready 0...100% and then screen switched to black.
Using VGA bios (romfile) does not help.

Oh, have you tried to install it first without vga passthrough?

Offline

#340 2013-07-28 23:08:22

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:
nbhs wrote:

have you tried:

Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp.h.rej patching file drivers/pci/hotplug/pciehp_core.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/hotplug/pciehp_core.c.rej patching file drivers/pci/hotplug/pciehp_hpc.c Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/pci/hotplug/pciehp_hpc.c.rej patching file drivers/pci/pci.c Reversed (or previously applied) patch detected! Skipping patch. 5 out of 5 hunks ignored -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/vfio/pci/vfio_pci.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/pci/vfio_pci.c.rej patching file drivers/vfio/vfio.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/vfio/vfio.c.rej patching file include/linux/pci.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/pci.h.rej patching file include/linux/pci_hotplug.h Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file include/linux/pci_hotplug.h.rej patching file include/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vfio.h.rej patching file include/uapi/linux/vfio.h Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file include/uapi/linux/vfio.h.rej ==> ERROR: A failure occurred in prepare(). Aborting...

I have tried, but it does not help.

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0
nbhs wrote:

@Myweb
You could also try loading the bios rom from a dump too

Do you mean VGA bios (romfile)?

Is it correct, that at first I need to fix: Attempt to reset PCI bus for VGA support failed (Device or resource busy).  VGA may not work.?

Yes your vga romfile, could you post your uname -a? using the packages posted on the first page does not produce this warning message correct?

uname: 3.11.0-1-mainline #1 SMP PREEMPT Sun Jul 28 16:25:59 UTC 2013 x86_64 GNU/Linux
using the packages posted on the first page does not produce this warning message - this is correct

Offline

#341 2013-07-28 23:10:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

@nbhs
Yes, here is my script running on root.

vfio-bind 0000:01:00.0 0000:01:00.1 modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

The reason I am after primary pass-through is because I'm waiting for Windows to BSOD in the background, reset and then crash the entire system >_< I suppose I can disable automatic rebooting when a BSOD happens but I don't know how reliable that is.

If anyone else has a 7950 running on primary pass-through please let me know what CPU & Motherboard you're using.

By the way - anyone who is using a KVM or the cirrus VGA to control the mouse and keyboard take a look at Synergy. It works using the network and you can define a hotkey to switch between hosts and copy the clipboard.

Wait i dont understand you're trying to get windows to BSOD?

Offline

#342 2013-07-28 23:11:07

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:

It looks like I was wrong with description of issue:
Windows installation starts, copy files, reboot VM, Windows installation continue, I see message Getting device ready 0...100% and then screen switched to black.
Using VGA bios (romfile) does not help.

Oh, have you tried to install it first without vga passthrough?

Yes, but when I run VM with vga passthrough specified Windows can't boot and asks to repair. Repair is failed due to it cant find a partition (I gues due to virtio-blk using)

Offline

#343 2013-07-28 23:14:55

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

It looks like I was wrong with description of issue:
Windows installation starts, copy files, reboot VM, Windows installation continue, I see message Getting device ready 0...100% and then screen switched to black.
Using VGA bios (romfile) does not help.

Oh, have you tried to install it first without vga passthrough?

Yes, but when I run VM with vga passthrough specified Windows can't boot and asks to repair. Repair is failed due to it cant find a partition (I gues due to virtio-blk using)

Have you tried the ide or ahci controller?

Offline

#344 2013-07-29 08:49:39

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could we have a separate thread to focus on building vfio-reset? This opening post is AMAZINGLY comprehensive for getting things up and running, and it almost feels like the building process would be better served a separate discussion.

nbhs : Found your 3.9 building package: http://www.filesend.net/download.php?f= … 468af53600   Thanks!

Last edited by mukiex (2013-07-29 10:08:45)

Offline

#345 2013-07-29 11:56:29

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
BulliteShot wrote:

@nbhs
Yes, here is my script running on root.

vfio-bind 0000:01:00.0 0000:01:00.1 modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

The reason I am after primary pass-through is because I'm waiting for Windows to BSOD in the background, reset and then crash the entire system >_< I suppose I can disable automatic rebooting when a BSOD happens but I don't know how reliable that is.

If anyone else has a 7950 running on primary pass-through please let me know what CPU & Motherboard you're using.

By the way - anyone who is using a KVM or the cirrus VGA to control the mouse and keyboard take a look at Synergy. It works using the network and you can define a hotkey to switch between hosts and copy the clipboard.

Wait i dont understand you're trying to get windows to BSOD?

No, what I mean is that the eject script is only going to run when the VM is cleanly shutdown.

If windows crashes or resets due to a BSOD then it's going to freeze the host when it boots up again.

Offline

#346 2013-07-29 12:06:00

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

linux-mainline.tar.gz (3.10.1)
qemu.tar.gz (1.5.1)
Or
Qemu 1.5.1 patch: qemu-vfio-vga-reset.patch.tar.gz
Kernel 3.10.1 patch: kernel-vfio-vga-reset.patch.tar.gz

I'm using the top 2 files. What's the difference between the top 2 and the bottom 2 files?

Offline

#347 2013-07-29 12:43:40

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

linux-mainline.tar.gz (3.10.1)
qemu.tar.gz (1.5.1)
Or
Qemu 1.5.1 patch: qemu-vfio-vga-reset.patch.tar.gz
Kernel 3.10.1 patch: kernel-vfio-vga-reset.patch.tar.gz

I'm using the top 2 files. What's the difference between the top 2 and the bottom 2 files?

Ah ha! I had the same thing!

One of them ONLY contains the patch. Advanced users only, I guess.

The other one is patch/PKGbuild setup. That's probably the one you want. You essentially hop in, run "makepkg".

If any dependencies are missing, you pacman -S them. Eventually you'll a .zx file with the package name. You'll want to pacman -U that file.
I'm really excited here. I really wanna write an uber-noob guide to get this going smile

Offline

#348 2013-07-29 12:58:22

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok so I think I installed the right package because I did do the makepkg route. Are these Alex Williamson's branches?

Last edited by BulliteShot (2013-07-29 13:02:17)

Offline

#349 2013-07-29 16:24:54

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't win. I got qemu, Seabios, and the kernel built with reset support. Still the same issue. Radeon never initializes unless it's the primary. Is this a motherboard issue? Would there be a way to fake it by having Linux send an image to it beforehand?

Offline

#350 2013-07-29 17:07:46

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sounds like you're having the opposite problem i'm having. I can only get my radon card to work as a secondary adapter using pci-attach (but first you have to enable cirrus VGA and then install the catalyst drivers... after rebooting, the cirrus VGA will stop responding at the windows boot logo and then you'll the graphics card will take over)

Last edited by BulliteShot (2013-07-29 17:08:03)

Offline

#351 2013-07-29 19:03:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

Ok so I think I installed the right package because I did do the makepkg route. Are these Alex Williamson's branches?

yes

mukiex wrote:

I can't win. I got qemu, Seabios, and the kernel built with reset support. Still the same issue. Radeon never initializes unless it's the primary. Is this a motherboard issue? Would there be a way to fake it by having Linux send an image to it beforehand?

Most amd boards + cpu work fine, it seems people with intel are having some problems, though some have been successful with or without (with another discrete card) the IGP as primary

mukiex wrote:

Could we have a separate thread to focus on building vfio-reset? This opening post is AMAZINGLY comprehensive for getting things up and running, and it almost feels like the building process would be better served a separate discussion.

nbhs : Found your 3.9 building package: http://www.filesend.net/download.php?f= … 468af53600   Thanks!

You should try the lastest packages i posted i might have forgotten a comit from alex tree

the building process is on the arch wiki, like you mentioned, unpack the file then run

makepkg -s

then:

pacman -U something_packagever_arch.pkg.tar.xz

Last edited by nbhs (2013-07-29 19:12:40)

Offline

#352 2013-07-29 19:11:56

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm not sure if I should ask my question here, but since the "problem" is somewhat related and others may have had the same doubts, I hope this is the right place.

I've got 3 video cards:
- Club3D Radeon HD 4770 750MHz (Memory: 512MB GDDR5 3200MHz)
- XFX Radeon HD 6570 650MHz (Memory: 1GB DDR3 1300MHz)
- Sapphire Radeon HD 5450 650MHz (Memory: 1GB DDR3 1300MHz)

I want to mount them on the same motherboard (ASRock Z87 Extreme6) and passthrough them to different VMs: the 4770 to my main Windows guest, the 6570 to my main Linux guest and I'll be leaving the 5450 to the host.

The thing is, the only configuration the 3 PCIe (3.0, though all the cards are 2.0) slots in my motherboard can assume when all of them are used is x8 x4 x4. Should I even care and put the most powerful card (that would be the 4770) in the x8 slot or even that can't possibly hope to saturate a x8 slot, and so I should just place them in the more aestetically pleasing/cooling efficient order?

Also I read that dual/tri monitor configurations use much more PCIe bandwidth, and I would connect at least two displays to the 5450, even though the cards that would be rendering things for those same displays would be the other two, the 5450 being only the one to which they are directly connected (since I understand the monitors must be connected to the host's graphic card, or I'm wrong?). So should I put the 5450 in the x8 slot after all?

Yeah I'm pretty confused here. Hope you can clarify all this mess for me.

Offline

#353 2013-07-29 19:15:09

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Evonat wrote:

I'm not sure if I should ask my question here, but since the "problem" is somewhat related and others may have had the same doubts, I hope this is the right place.

I've got 3 video cards:
- Club3D Radeon HD 4770 750MHz (Memory: 512MB GDDR5 3200MHz)
- XFX Radeon HD 6570 650MHz (Memory: 1GB DDR3 1300MHz)
- Sapphire Radeon HD 5450 650MHz (Memory: 1GB DDR3 1300MHz)

I want to mount them on the same motherboard (ASRock Z87 Extreme6) and passthrough them to different VMs: the 4770 to my main Windows guest, the 6570 to my main Linux guest and I'll be leaving the 5450 to the host.

The thing is, the only configuration the 3 PCIe (3.0, though all the cards are 2.0) slots in my motherboard can assume when all of them are used is x8 x4 x4. Should I even care and put the most powerful card (that would be the 4770) in the x8 slot or even that can't possibly hope to saturate a x8 slot, and so I should just place them in the more aestetically pleasing/cooling efficient order?

Also I read that dual/tri monitor configurations use much more PCIe bandwidth, and I would connect at least two displays to the 5450, even though the cards that would be rendering things for those same displays would be the other two, the 5450 being only the one to which they are directly connected (since I understand the monitors must be connected to the host's graphic card, or I'm wrong?). So should I put the 5450 in the x8 slot after all?

Yeah I'm pretty confused here. Hope you can clarify all this mess for me.

You should probably start with 2 cards and see if it works smile

Last edited by nbhs (2013-07-29 19:23:16)

Offline

#354 2013-07-29 19:29:12

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well yeah, I'm creating problems when I don't even know if it'll work in the first place. But what can I do, I still haven't got hold of the necessary equipment, aside from the video cards that I already own. Sorry. roll

But disregarding for a moment whether it'll work or not, just to satisfy my curiosity, would it be more ideal to put the most powerful card in the fastest slot, or reserve it for the host's card to which the displays are connected to?

Offline

#355 2013-07-29 19:34:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Evonat wrote:

Well yeah, I'm creating problems when I don't even know if it'll work in the first place. But what can I do, I still haven't got hold of the necessary equipment, aside from the video cards that I already own. Sorry. roll

But disregarding for a moment whether it'll work or not, just to satisfy my curiosity, would it be more ideal to put the most powerful card in the fastest slot, or reserve it for the host's card to which the displays are connected to?

Well you'll need the first slot for the host gpu so unless you can get it to work with the igp, you should put the 5450 on the first one, also why would you want another linux guest?

Last edited by nbhs (2013-07-29 19:35:12)

Offline

#356 2013-07-29 21:11:19

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems if I set my primary card in the host's BIOS to PCI instead of Integrated then ArchLinux boots on the host using my radeon card! :s Then when I boot the VM, the display goes blank. Since I've blacklisted and used pci-stub driver then why is ArchLinux still using the radeon card?

[root@localhost ~]# cat /etc/modprobe.d/blacklist.conf blacklist radeon

lspci -v...

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950] (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. Device 2761 Flags: fast devsel, IRQ 16 Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f7e00000 (64-bit, non-prefetchable) [size=256K] I/O ports at e000 [size=256] Expansion ROM at f7e40000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] #13 Capabilities: [2d0] #1b Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] Subsystem: Micro-Star International Co., Ltd. Device aaa0 Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at f7e60000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

It seems to still be using the Kernel module... is it possible that this is causing the problem?

[root@localhost ~]# dmesg | grep Kernel [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/md1 ro quiet pci-stub.ids=1002:679a,1002:aaa0 intel_iommu=on

Last edited by BulliteShot (2013-07-29 21:13:10)

Offline

#357 2013-07-29 21:13:10

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Evonat wrote:

Well yeah, I'm creating problems when I don't even know if it'll work in the first place. But what can I do, I still haven't got hold of the necessary equipment, aside from the video cards that I already own. Sorry. roll

But disregarding for a moment whether it'll work or not, just to satisfy my curiosity, would it be more ideal to put the most powerful card in the fastest slot, or reserve it for the host's card to which the displays are connected to?

Well you'll need the first slot for the host gpu so unless you can get it to work with the igp, you should put the 5450 on the first one.

If I remember correctly, some Gigabyte motherboards allow for selecting which of PCIe slot boots first, or at least they did at one time.  So with a board that allows for such a selection, the host gpu could be placed in a slower slot.

Also, I once inadvertently added the card in my first PCIe slot to the pci-stub list in grub cmdline.  To my surprise, the card in the second PCIe slot displayed my host on boot.  Although such an approach certainly is not optimal, it might be worth a try.

Offline

#358 2013-07-30 14:04:32

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

Sounds like you're having the opposite problem i'm having. I can only get my radon card to work as a secondary adapter using pci-attach (but first you have to enable cirrus VGA and then install the catalyst drivers... after rebooting, the cirrus VGA will stop responding at the windows boot logo and then you'll the graphics card will take over)

Can you do pci-attach with the q35 motherboard? I'll gladly take a workaround.

Offline

#359 2013-07-30 14:06:08

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:
BulliteShot wrote:

Sounds like you're having the opposite problem i'm having. I can only get my radon card to work as a secondary adapter using pci-attach (but first you have to enable cirrus VGA and then install the catalyst drivers... after rebooting, the cirrus VGA will stop responding at the windows boot logo and then you'll the graphics card will take over)

Can you do pci-attach with the q35 motherboard? I'll gladly take a workaround.

No sorry, just the default motherboard. If I set using q35 then the Windows device manager complains that the device doesn't have enough resources to start.

Offline

#360 2013-07-30 15:00:14

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

No sorry, just the default motherboard. If I set using q35 then the Windows device manager complains that the device doesn't have enough resources to start.

Yeah, that's where I'm at.

The most aggravating part is the Qemu provides ZERO logs to figure out why it's not starting up.

Another bit I noticed is that when Intel is primary and the Radeon card is secondary, not only dose the image not show up on-screen, NOTHING LOADS. There is zero hard drive activity. If I could find out why that'd be great, but I get NOTHING for error messages. Heck, I can't find ANY Qemu logs, ANYWHERE. dmesg gives me this:

[ 2956.122793] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 2956.122800] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

Last edited by mukiex (2013-07-30 15:04:08)

Offline

#361 2013-07-30 16:36:38

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well the damn thing just randomly blue screened while I was running Steam chat on the VM and watching youtube on the host (not demanding in the slightest)... so i'm gonna try plugging in a 2nd card and disable the Intel integrated graphics to see what happens.

I couldn't see the error because I couldn't open the VNC fast enough.. but it was an ati----.sys driver that caused the BSOD

Offline

#362 2013-07-30 17:47:35

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And it WORKS! Here's what I did...

1) Went into the BIOS and Disabled Integrated Graphics
2) Still in the BIOS, changed the primary graphics from Integrated to PCI
3) Saves changes and powered off my system.
4) Moved my 7950 GPU to my 2nd slot on the mobo.
5) Inserted a 7790 GPU from my girlfriend's PC into the 1st slot on my mobo.
6) Booted, changed my kernel pci-stub params to match the PCI ID of my 7950 then rebooted.
7) Host archlinux booted on the 7790 in my first slot. Booted the VM using the q35 method and the BIOS loaded on the screen attached to the 7950.

So I can confirm using the Intel Integrated Graphics breaks the VFIO passthrough but if you use a 2nd card and disable the integrated graphics then it should work!

I'm getting a BSOD in windows now but I bet that's because it doesn't like the change to the q35 motherboard. I'm currently reinstalling it.

Last edited by BulliteShot (2013-07-30 17:48:36)

Offline

#363 2013-07-30 18:48:10

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I GOT IT!!!

Okay, so this is the most bum-backwards solution you will ever see, but:

- Boot with Intel GFX as primary
- Bind Intel graphics to PCI-STUB but NOT VFIO (the latter will lock your system)
- SUCCESS (graphics card turns on without issue)

And now to fix my USB binding... can I bind USB 3.0?

Offline

#364 2013-07-30 19:12:09

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Awsome guys nice to see it works, yes you can bind your usb3 controller thats why i do with my vm

Offline

#365 2013-07-30 19:14:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

I GOT IT!!!

Okay, so this is the most bum-backwards solution you will ever see, but:

- Boot with Intel GFX as primary
- Bind Intel graphics to PCI-STUB but NOT VFIO (the latter will lock your system)
- SUCCESS (graphics card turns on without issue)

And now to fix my USB binding... can I bind USB 3.0?

Wait so you're using binding the primary card ( the igp ) to pci-stub that that allowed it to work?, It might be that the intel driver is the one causing problems then, take into consideration that the host wont use the it and you'll lose 3d acceleration

Last edited by nbhs (2013-07-30 19:20:15)

Offline

#366 2013-07-30 19:16:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

IF you got it working please post how and your hardware specs so we can all know (i might even get an intel setup)

Offline

#367 2013-07-30 20:12:57

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

I GOT IT!!!

Okay, so this is the most bum-backwards solution you will ever see, but:

- Boot with Intel GFX as primary
- Bind Intel graphics to PCI-STUB but NOT VFIO (the latter will lock your system)
- SUCCESS (graphics card turns on without issue)

And now to fix my USB binding... can I bind USB 3.0?

mukiex, you sweet sweet genius! big_smile That fixed it for me too!

@nbhs: My board is B75-D3V

Offline

#368 2013-07-30 20:16:19

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:
mukiex wrote:

I GOT IT!!!

Okay, so this is the most bum-backwards solution you will ever see, but:

- Boot with Intel GFX as primary
- Bind Intel graphics to PCI-STUB but NOT VFIO (the latter will lock your system)
- SUCCESS (graphics card turns on without issue)

And now to fix my USB binding... can I bind USB 3.0?

mukiex, you sweet sweet genius! big_smile That fixed it for me too!

@nbhs: My board is B75-D3V

So it looks like the intel driver its the one causing problems

Offline

#369 2013-07-30 20:29:18

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's not the intel driver causing issues: I found this fix originally on two separate discrete cards.

The Intel card isn't being loaded on the guest, but if it's not bound to pci-stub, the guests actual card won't ever turn on.

Offline

#370 2013-07-30 20:31:51

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

It's not the intel driver causing issues: I found this fix originally on two separate discrete cards.

The Intel card isn't being loaded on the guest, but if it's not bound to pci-stub, the guests actual card won't ever turn on.

Have you tried blacklisting the intel driver instead of binding it to pci-stub?

Last edited by nbhs (2013-07-30 20:32:13)

Offline

#371 2013-07-30 22:08:16

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I install the ATI drivers I get a BSOD at boot with atikmpag.sys sad

Offline

#372 2013-07-30 22:16:51

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

When I install the ATI drivers I get a BSOD at boot with atikmpag.sys sad

Are you using the lastest version? im using catalyst 12.10 with win8, no bsods or anything, try installing the drivers manually from the device manager

Offline

#373 2013-07-30 22:26:55

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Driver installed is 12.104.0.0 along with Catalyst 13.4 and I'm using Windows 7 in the VM. I'll mess around with the drivers and see if it likes the BETA or 12.10 versions.

Windows boots fine with the standard driver, but thats not much better than the standard VGA in Qemu to be honest tongue

Offline

#374 2013-07-30 22:28:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

Driver installed is 12.104.0.0 along with Catalyst 13.4 and I'm using Windows 7 in the VM. I'll mess around with the drivers and see if it likes the BETA or 12.10 versions.

Windows boots fine with the standard driver, but thats not much better than the standard VGA in Qemu to be honest tongue

Try catalyst 12.10, works for me

Offline

#375 2013-07-30 22:52:02

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

* set the controller to AHCI mode in UEFI/BIOS (actually no UEFI here, the Gigabyte has a legacy BIOS mode)
* added its vendor:id to pci-stub.ids boot parameter (as said, it's completely useless for the host anyways, so make the AHCI driver ignore it completely)
* passed it through to the qemu VM using "-device vfio-pci,host=${SATA_DEVICE},bus=pcie.0" (actually it shows up with two entries in lscpi, one of wich seems to be for the E-SATA ports (they also do not work in the Linux host, same issues))
* Win7 automatically installed some ATA drivers for it (I didn't download anything anywhere)
* it works, without issues

Sorry for the late reply and thanks for your attemp to help, I tried to replicate your setup already a few days ago but kind of gave up (it works anyways…). The Windows 7 install disk doesn't recognize the device and (an already installed version of) Windows 8 BSODs after bootup, I can write doen the exact error message in case anyone knows what these mean.
What I didn't try is a fresh, "untouched" in a qcow file and attach it there. And for some reason I think I didn't try linux guests.
I'll probably try those options if I find the motivation, or I might replace 2 of the 1TB disks in my PC with one 2 or 3TB disk (I should do that anyways), attach the SSD to the amd controller and use virtio.
Maybe someday, someone will work around this in the amd-iommu driver or something else will happen. If I look at my graphics drivers problems on my notebook with an e350 apu (amd graphics), the problems I had with it in the last ~1.5 years are mostly solved by switching to the open source driver, which wasn't ready back then. What I want to say is that waiting or doing nothing is sometimes the best solution to non-critical issues.


i'm sorry for my poor english wirting skills…

Offline

#376 2013-07-31 09:21:07

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
BulliteShot wrote:

Driver installed is 12.104.0.0 along with Catalyst 13.4 and I'm using Windows 7 in the VM. I'll mess around with the drivers and see if it likes the BETA or 12.10 versions.

Windows boots fine with the standard driver, but thats not much better than the standard VGA in Qemu to be honest tongue

Try catalyst 12.10, works for me

I've tried 12.10, 13.4 and 13.6 BETA... no luck. Tried passing the VGA BIOS through too, but no luck there either hmm

Edit: We really need a list of hardware combinations which is confirmed to be working.

Last edited by BulliteShot (2013-07-31 11:22:17)

Offline

#377 2013-07-31 11:53:21

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I updated my system, which resulted in a nonworking DomU hmm
Guess that´s the point where i throw away my old approach with pci-stub and the unpatched system. I am going to try vfio-vga and report my story :-)

@Bullite: So you´re setup was already working? I´d like to take your mobo into my list of vt-d hardware (*edit: sorry i missed the part with ati bsod*). But i am interested in collecting working combinations also.

Last edited by Diggo (2013-07-31 11:53:59)

Offline

#378 2013-07-31 11:59:44

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It "works" using the old pci-attach without vfio. I seem to experience random BSOD of the VM due to the ATI driver... and then because it crashed without being able to eject the graphics card, the reset crashes the entire system.

That's why I would like to upgrade to hardware that works using this new vfio method.

My hardware is listed a few packages back.

Offline

#379 2013-07-31 12:02:37

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i noted it here (http://www.techdudes.de/752/iommu-und-vt-d-taugliche-hardware/ - it´s german *g* so maybe it is not that useful for you). I haven´t read a lot in the last moths on vfio, what´s the major advantage over pci-attach?

Last edited by Diggo (2013-07-31 12:05:14)

Offline

#380 2013-07-31 12:53:01

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:

Ok i noted it here (http://www.techdudes.de/752/iommu-und-vt-d-taugliche-hardware/ - it´s german *g* so maybe it is not that useful for you). I haven´t read a lot in the last moths on vfio, what´s the major advantage over pci-attach?

The vfio-reset patch and BIOS-level access from the VGA are the only advantages I am aware of.

If there was a similar fix for pci-attach then i'd be happy

Offline

#381 2013-07-31 17:18:30

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Try catalyst 12.10, works for me

Thanks for the heads up on blacklisting the Intel driver, it worked!

I'm getting a blue screen on windows loads up like many other people. Tried catalyst 12.10. Gonna try 12-6 as it's still after my card's release date, but I don't have high points. 'bout to knock out, I'll update you guys when it's done. Thankfully, being on ZFS makes rolling back to a pre-driver Windows install pretty trivial.

Edit note: Yep, bluescreen on atikmpag.sys , same on Catalyst 12.10 or 12.6

Last edited by mukiex (2013-07-31 17:26:45)

Offline

#382 2013-07-31 18:36:45

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:
nbhs wrote:

Try catalyst 12.10, works for me

Thanks for the heads up on blacklisting the Intel driver, it worked!

I'm getting a blue screen on windows loads up like many other people. Tried catalyst 12.10. Gonna try 12-6 as it's still after my card's release date, but I don't have high points. 'bout to knock out, I'll update you guys when it's done. Thankfully, being on ZFS makes rolling back to a pre-driver Windows install pretty trivial.

Edit note: Yep, bluescreen on atikmpag.sys , same on Catalyst 12.10 or 12.6

I (was) pretty sure this was because we were using Intel Graphics on the host... but i've disabled Intel Graphics in the BIOS and tried with a 2nd radeon card and I'm seeing the same BSOD. Now I have no idea what the problem could be.

Offline

#383 2013-07-31 19:10:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

have you guys tried installing only the driver without the hdmi driver and amd control panel? you could also try playing with the intel_iommu= options

Last edited by nbhs (2013-07-31 19:12:06)

Offline

#384 2013-08-01 10:59:30

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey nbhs. Just to make sure, are there any drivers we need to install that would improve things before doing Catalyst? Is there a specific method?

Right now I'm installing catalyst by letting the installer extract (and closing it), and then right-clicking the VGA device, and pointing "update driver" to that installation folder. Is there a more in-depth method?

Quick note for searchability: It's the issue with atikmpag.sys ending with timestamp 51ae5d89

I'll let you guys know if I find a solution.

Last edited by mukiex (2013-08-01 13:24:53)

Offline

#385 2013-08-01 13:33:45

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

Hey nbhs. Just to make sure, are there any drivers we need to install that would improve things before doing Catalyst? Is there a specific method?

Right now I'm installing catalyst by letting the installer extract (and closing it), and then right-clicking the VGA device, and pointing "update driver" to that installation folder. Is there a more in-depth method?

Quick note for searchability: It's the issue with atikmpag.sys ending with timestamp 51ae5d89

I'll let you guys know if I find a solution.

Nope, no driver needed, all i did was run the catalyst installer then rebooted

EDIT: have you tried playing with the intel_iommu= options like i mentioned? for example intel_iommu=pt,igfx_off

Last edited by nbhs (2013-08-01 13:38:07)

Offline

#386 2013-08-01 13:55:12

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Haven't tried intel_iommu=pt, but I've tried both igfx_ options. I'll let you know what my results are.

Result: With intel_iommu=pt , iommu_group disappears

cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

Last edited by mukiex (2013-08-01 14:04:46)

Offline

#387 2013-08-01 14:05:06

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Offline

#388 2013-08-01 14:22:43

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

Offline

#389 2013-08-01 15:16:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

Offline

#390 2013-08-01 15:39:08

pcworld
Member
From: Germany
Registered: 2013-07-28
Posts: 6
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, …

How to do that? I'm considering buying an "ASUS M5A97 Evo R2.0", however I would still like to be able to use IOMMU if that board suffers from the same issue.

Offline

#391 2013-08-01 16:00:49

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

nbhs wrote:
BulliteShot wrote:

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

*sigh* I for one am sick of Intel pushing these limitations.

Goodbye Intel, hello AMD!

Offline

#392 2013-08-01 16:06:11

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pcworld wrote:
jgott wrote:

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, …

How to do that? I'm considering buying an "ASUS M5A97 Evo R2.0", however I would still like to be able to use IOMMU if that board suffers from the same issue.

there's a "ivrs_override=" kernel parameter, IOMMU should work even without it, the workaround is to enable interrupt remapping support on these boards

Last edited by nbhs (2013-08-01 16:10:34)

Offline

#393 2013-08-01 16:11:22

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thank you very much for your guide.
I'm running into some problems when I try to passthrough a gpu to a guest.

Hardware:

  • Xeon E3-1230v2

  • ASRock Fatal1ty Z77 professional

  • NVIDIA GTX-470 (Host)

  • ATI Radeon 5450 (Guest)

# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on cryptdevice=/dev/sda2:main quiet
# lspci -tv -[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller +-01.0-[01]--+-00.0 NVIDIA Corporation GF100 [GeForce GTX 470] | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller +-01.1-[02]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller +-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 +-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 +-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller +-1c.0-[03]-- +-1c.7-[04-0d]----00.0-[05-0d]--+-01.0-[06]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-04.0-[07-08]----00.0-[08]-- | +-05.0-[09]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-06.0-[0a]----00.0 Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe | +-07.0-[0b]----00.0 Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller | +-08.0-[0c]----00.0 Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe | \-09.0-[0d]--+-00.0 VIA Technologies, Inc. VT6315 Series Firewire Controller | \-00.1 VIA Technologies, Inc. VT6415 PATA IDE Host Controller +-1d.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 +-1f.0 Intel Corporation Z77 Express Chipset LPC Controller +-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Using the second pcie slot, both gpus are in the same iommu_group (thats why i can't bind all devices in the group to vfio).

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group ../../../../kernel/iommu_groups/1 # ls -l /sys/kernel/iommu_groups/1/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <- pci-bridge lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <- pci-bridge lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

Using the third pcie slot, the second gpu is in the same iommu_group with the NIC's and stuff.

# lspci | grep ATI 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] # readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group ../../../../kernel/iommu_groups/6 # ls -l /sys/kernel/iommu_groups/6/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

Question:
Is there a workaround/fix for this or should i get a new mobo? big_smile

Thank you!

Offline

#394 2013-08-01 16:24:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:

Hi,

thank you very much for your guide.
I'm running into some problems when I try to passthrough a gpu to a guest.

Hardware:

  • Xeon E3-1230v2

  • ASRock Fatal1ty Z77 professional

  • NVIDIA GTX-470 (Host)

  • ATI Radeon 5450 (Guest)

# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on cryptdevice=/dev/sda2:main quiet
# lspci -tv -[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller +-01.0-[01]--+-00.0 NVIDIA Corporation GF100 [GeForce GTX 470] | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller +-01.1-[02]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller +-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 +-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 +-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller +-1c.0-[03]-- +-1c.7-[04-0d]----00.0-[05-0d]--+-01.0-[06]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-04.0-[07-08]----00.0-[08]-- | +-05.0-[09]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-06.0-[0a]----00.0 Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe | +-07.0-[0b]----00.0 Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller | +-08.0-[0c]----00.0 Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe | \-09.0-[0d]--+-00.0 VIA Technologies, Inc. VT6315 Series Firewire Controller | \-00.1 VIA Technologies, Inc. VT6415 PATA IDE Host Controller +-1d.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 +-1f.0 Intel Corporation Z77 Express Chipset LPC Controller +-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Using the second pcie slot, both gpus are in the same iommu_group (thats why i can't bind all devices in the group to vfio).

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group ../../../../kernel/iommu_groups/1 # ls -l /sys/kernel/iommu_groups/1/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <- pci-bridge lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <- pci-bridge lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 1 17:40 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

Using the third pcie slot, the second gpu is in the same iommu_group with the NIC's and stuff.

# lspci | grep ATI 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] # readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group ../../../../kernel/iommu_groups/6 # ls -l /sys/kernel/iommu_groups/6/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

Question:
Is there a workaround/fix for this or should i get a new mobo? big_smile

Thank you!

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Last edited by nbhs (2013-08-01 16:26:11)

Offline

#395 2013-08-01 16:47:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:
mukiex wrote:

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

nbhs wrote:
BulliteShot wrote:

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

*sigh* I for one am sick of Intel pushing these limitations.

Goodbye Intel, hello AMD!

I think instead of spending money perhaps you could try posting you issue on the kvm and/or qemu mailing list, remember this is still an experimental feature being worked on smile

Last edited by nbhs (2013-08-01 16:49:15)

Offline

#396 2013-08-02 00:31:47

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
cmdr wrote:

-snip-

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Last edited by cmdr (2013-08-02 01:53:54)

Offline

#397 2013-08-02 02:34:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

-snip-

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Last edited by nbhs (2013-08-02 02:46:52)

Offline

#398 2013-08-02 10:32:46

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok I was meant to post my specs earlier, but summer vacation got in the way smile.

So. I'm using a 2009 Mac Pro with OS X and Arch Linux dual boot as a basis. 2 Xeon processors (8 cores) and 20 gigs of ram smile. Modified kernel, QEMU and Seabios - downloaded from this thread.

Host graphics card - nVidia GeForce 9500 GT (Mac Edition) (Mac EFI boot loader needs this in the first PCIe slot to boot unfortunately.) Tested graphics cards to work in the second 16x PCIe slot: Radeon HD 5450 and Radeon HD 7750.

So on their own they work fine. But I've also tried a 3-graphics-cards-setup to have two VMs. At this point it's not so stable anymore. I can have one Windows 8 VM with the HD 7750 running together with Ubuntu 12.04 with HD5450, but as soon as I install FGLRX or catalyst driver for it- it crashes and won't boot anymore. Also various error messages from the kernel can be seen during start and shutdown. Windows installation fails even earlier. So at this setup the HD 5450 is on a 4x PCIe slot.

Now the problem might be similar to what was posted earlier about the iommu_groups, the HD 7750 (on the 2nd 16x slot) shows only:

0000:06:00.0 0000:06:00.1

But the HD 5450 (on one of the free 4x slots) shows:

0000:02:04.0 0000:04:00.0 0000:04:00.1

So it includes one of the pcie switches:

02:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch (rev 01)

Could this be the source of the problem? And could I change the iommu_groups somehow.

Thanks again,
Teemu Kämäräinen

Offline

#399 2013-08-02 12:07:29

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
cmdr wrote:
nbhs wrote:

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Offline

#400 2013-08-02 12:11:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).


I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Interesting, thanks a lot for your report

Offline

#401 2013-08-02 12:18:16

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).


I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Thanks, this might help me too in my quest for three VMs with each having their own graphics cards. So what device are you giving the acs override command here

pcie_acs_override=id:8086:1e1e

. Is it a bridge or something else?

-Teemu

Offline

#402 2013-08-02 12:26:05

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri wrote:
cmdr wrote:
nbhs wrote:

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Thanks, this might help me too in my quest for three VMs with each having their own graphics cards. So what device are you giving the acs override command here

pcie_acs_override=id:8086:1e1e

. Is it a bridge or something else?

-Teemu

http://pci-ids.ucw.cz/read/PC/8086/1e1e

Offline

#403 2013-08-02 12:33:07

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

reintseri wrote:
cmdr wrote:
nbhs wrote:

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Thanks, this might help me too in my quest for three VMs with each having their own graphics cards. So what device are you giving the acs override command here

pcie_acs_override=id:8086:1e1e

. Is it a bridge or something else?

-Teemu

Yes, it's a pci root port/bridge. Before i used 'pcie_acs_override" every device behind that root port/bridge was in the same iommu_group.

# ls -l /sys/kernel/iommu_groups/6/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0 lrwxrwxrwx 1 root root 0 Aug 1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

With the changed cmdline the devices are now in their own groups:

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 2 14:32 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0 lrwxrwxrwx 1 root root 0 Aug 2 14:32 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 2 14:32 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1 ]# ls -l /sys/bus/pci/devices/0000\:05\:00.0/iommu_group/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 2 14:32 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0 # ls -l /sys/bus/pci/devices/0000\:0a\:00.0/iommu_group/devices/ total 0 lrwxrwxrwx 1 root root 0 Aug 2 14:32 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0

Offline

#404 2013-08-02 14:14:22

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I keep forgetting to post this:

If your Intel system won't boot with iommu=on enabled, it's a bug. Straight-up. I've gotten Xen working with passthrough PCI (though not at DomU boot) as far back as Ubuntu 12.04. However, under KVM at the moment, setting that option will result in your system not booting if you have a 0000:05:00.0 pci device. I tried other options, but that's basically what it came down to. Remove the extra PCI device. It will work. It's messed up, I know.

Offline

#405 2013-08-02 14:32:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

I keep forgetting to post this:

If your Intel system won't boot with iommu=on enabled, it's a bug. Straight-up. I've gotten Xen working with passthrough PCI (though not at DomU boot) as far back as Ubuntu 12.04. However, under KVM at the moment, setting that option will result in your system not booting if you have a 0000:05:00.0 pci device. I tried other options, but that's basically what it came down to. Remove the extra PCI device. It will work. It's messed up, I know.

It's not a bug, to enable the iommu on intel the correct kernel parameter is intel_iommu=on, iommu=on might be a xen parameter but its completely unrelated to kvm, since xen is a type1 hypervisor with its own kernel.

Offline

#406 2013-08-02 15:13:59

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).


I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

I installed the ACS-patch also and now everything works! Also the 4x PCI slots now have the devices in their own iommu groups. Thanks for this info. So now I have 2 VM's both with their own graphics cards on top of the host smile. Arch + Windows 8 + Ubuntu 12.04.

Now I need another card to test 4 ... (and a PCIe extender cable, since I've run out of room a bit in the card department..)

Last edited by reintseri (2013-08-02 15:14:44)

Offline

#407 2013-08-02 22:00:46

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

Note that I've tried pcie_acs_override = downstream, multifunciton, and singling out the vendor/device numbers for one of the root port. This output did not change.
iommu grouping on my motherboard is insane. If the videocard is on the first PCI express port, I only get the video card and the root PCI-E port. However, moving the card to the other port gives me this.

Via  /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ :

lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:00:1c.4 -> ../../../../devices/pci0000:00/0000:00:1c.4 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:00:1c.6 -> ../../../../devices/pci0000:00/0000:00:1c.6 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:03:00.0 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.6/0000:04:00.0

lspci for those devices:

00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 03:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0) 04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)

Offline

#408 2013-08-02 22:29:31

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

Hey guys,

Note that I've tried pcie_acs_override = downstream, multifunciton, and singling out the vendor/device numbers for one of the root port. This output did not change.
iommu grouping on my motherboard is insane. If the videocard is on the first PCI express port, I only get the video card and the root PCI-E port. However, moving the card to the other port gives me this.

Via  /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ :

lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:00:1c.4 -> ../../../../devices/pci0000:00/0000:00:1c.4 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:00:1c.6 -> ../../../../devices/pci0000:00/0000:00:1c.6 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.4/0000:03:00.0 lrwxrwxrwx 1 root root 0 Aug 2 16:52 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.6/0000:04:00.0

lspci for those devices:

00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 03:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0) 04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)

Sorry to ask but did you apply the patch first?

Offline

#409 2013-08-02 23:08:09

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs, is that patch included in the vfio-reset build? Sorry, I had assumed it was because of the patch date.

Offline

#410 2013-08-02 23:10:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

nbhs, is that patch included in the vfio-reset build? Sorry, I had assumed it was because of the patch date.

No its not, its here: http://www.spinics.net/lists/kvm/msg92163.html or here: http://marc.info/?l=linux-kernel&m=136993923507507&w=2 i did not include it since its not in alex tree and technically its unrelated

Last edited by nbhs (2013-08-02 23:12:11)

Offline

#411 2013-08-03 19:47:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
mukiex wrote:

nbhs, is that patch included in the vfio-reset build? Sorry, I had assumed it was because of the patch date.

No its not, its here: http://www.spinics.net/lists/kvm/msg92163.html or here: http://marc.info/?l=linux-kernel&m=136993923507507&w=2 i did not include it since its not in alex tree and technically its unrelated

I re uploaded the linux-mainline package to include this patch its on the first page

Offline

#412 2013-08-04 11:29:05

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I re uploaded the linux-mainline package to include this patch its on the first page

Thank you so much! Adding "pcie_acs_override=downstream" took care of the iommu_groups silliness. Now only the graphics card pieces show up.

The blue screen seems to happen across multiple driver versions (tried about 4 different drivers between the month my card launched and the latest beta). Same file: atikmpag.sys

Now, I don't know if this helps, but the situation is also bad with an Nvidia card: I swapped it out, reverted to a fresh, driver-less install (yay, ZFS), and installed it. No blue-screen, but also no working driver; driver fails to start and the system reverts to a pre-driver setup.


nbhs: I want to make 110% sure I'm not screwing it up on my end. Each piece of the puzzle I'm running the vfio-reset stuff (built from your packages; thanks again), but I just want to make sure I've installed each bit correctly; is there anything to look out for to verify this?

Last edited by mukiex (2013-08-04 11:32:04)

Offline

#413 2013-08-04 17:23:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:
nbhs wrote:

I re uploaded the linux-mainline package to include this patch its on the first page

Thank you so much! Adding "pcie_acs_override=downstream" took care of the iommu_groups silliness. Now only the graphics card pieces show up.

The blue screen seems to happen across multiple driver versions (tried about 4 different drivers between the month my card launched and the latest beta). Same file: atikmpag.sys

Now, I don't know if this helps, but the situation is also bad with an Nvidia card: I swapped it out, reverted to a fresh, driver-less install (yay, ZFS), and installed it. No blue-screen, but also no working driver; driver fails to start and the system reverts to a pre-driver setup.


nbhs: I want to make 110% sure I'm not screwing it up on my end. Each piece of the puzzle I'm running the vfio-reset stuff (built from your packages; thanks again), but I just want to make sure I've installed each bit correctly; is there anything to look out for to verify this?

Have you tried with windows 8?

Offline

#414 2013-08-04 23:14:31

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried Windows 7 64Bit and the vm runs fine. But when i install the latest ATI driver (ONLY the display driver) i get a bsod on reboot. Downloading Windows 8 now; 6 hours left...

Offline

#415 2013-08-06 13:33:46

licious
Member
Registered: 2013-08-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Specs :
cpu:Intel i7 3770
motherboard : Asus P8H77-I
host gpu : Intel HD 4000
guest gpu : Nvidia GT 620
Host : Ubuntu 13.04 with patched kernel 3.10.1
grub: GRUB_CMDLINE_LINUX="intel_iommu=on pci-stub.ids=8086:0151,10de:0f01,10de:0bea vfio_iommu_type1.allow_unsafe_interrupts=1"
guest : Ubuntu 13.04, kernel 3.8.0-27-generic (stock)


I've built the patched kernel, seabios and qemu - but I can't get an output on the screen connected to the GT620 when I use vfio. Qemu output is always blank / black as well. If I use "-device pci-assign" instead of "-device vfio-pci" then my guest os initializes the GT620 when starting X on the guest (no output at all before X starts). I can only get an output if I'm using the Nouveau-drivers on the guest, Nvidia binary drivers won't work. I've tried patching the drivers as well, but I've had no luck in getting the patched drivers to work. The output from dmesg on the guest :
[   17.260770] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
[   17.260778] NVRM: rm_init_adapter(0) failed

the weird thing is that it seems like hdmi sound is initialized :
[   13.078985] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input5
[   13.079086] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input6
[   13.079149] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input7
[   13.079225] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input8

Also, when using -device vfio-pci I get an error message saying "qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/xbmc/GT620.rom: Warning, device 0000:01:00.0 does not support reset". This message does not appear when I use pci-assign, most likely because no kernel modules have been loaded for the GPU yet. I thought the patched kernel and/ or qemu would take care of resetting the device as well ?

Offline

#416 2013-08-06 17:58:36

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, today i took a chance to migrate to vfio-pci. But i got stuck at the point, where to start the DomU the first time. If i read the posts here right, the patched versions are for VGA-reset only (so for testing, i actually don´t need them?).

The last thing i did was to add the ivrs entry to the commandline. The AMD-Vi error has gone, but still the outcome is the same "Could not initialize SDL(No available video device) - exiting". There must be something I am missing (?) smile

Bash command:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 Could not initialize SDL(No available video device) - exiting


Some other Outputs:


cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=328873c3-cc85-4c75-8b77-ac9a7763d532 ro ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 iommu=pt iommu=1 pci-stub.ids=1002:6739,1002:aa88

lspci:

... 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] ...

lspci-n:

01:00.0 0300: 1002:6739 01:00.1 0403: 1002:aa88

Dmesg:

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.5-1-ARCH (tobias@T-POWA-LX) (gcc version 4.8.1 20130725 (prerelease) (GCC) ) #1 SMP PREEMPT Mon Aug 5 08:04:22 CEST 2013 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=328873c3-cc85-4c75-8b77-ac9a7763d532 ro ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 iommu=pt iommu=1 pci-stub.ids=1002:6739,1002:aa88 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bda5efff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bda5f000-0x00000000bdaacfff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdaad000-0x00000000bdab6fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdab7000-0x00000000bdab7fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdab8000-0x00000000bdadcfff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdadd000-0x00000000bdaedfff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdaee000-0x00000000bdb02fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdb03000-0x00000000bdb04fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdb05000-0x00000000bdb0dfff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdb0e000-0x00000000bdb13fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdb14000-0x00000000bdb75fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdb76000-0x00000000bdd78fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdd79000-0x00000000bdefffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec20000-0x00000000fec20fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed61000-0x00000000fed70fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fef00000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100001000-0x000000013fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: To be filled by O.E.M. To be filled by O.E.M./M5A99X EVO, BIOS 1604 10/16/2012 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] No AGP bridge found [ 0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF write-through [ 0.000000] C0000-C7FFF write-protect [ 0.000000] C8000-EBFFF uncachable [ 0.000000] EC000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask FFFF80000000 write-back [ 0.000000] 1 base 000080000000 mask FFFFC0000000 write-back [ 0.000000] 2 base 0000BDF00000 mask FFFFFFF00000 uncachable [ 0.000000] 3 base 0000BE000000 mask FFFFFE000000 uncachable [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] TOM2: 0000000140000000 aka 5120M [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: update [mem 0xbdf00000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0xbdf00 max_arch_pfn = 0x400000000 [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01b22000, 0x01b22fff] PGTABLE [ 0.000000] BRK [0x01b23000, 0x01b23fff] PGTABLE [ 0.000000] BRK [0x01b24000, 0x01b24fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x13fe00000-0x13fffffff] [ 0.000000] [mem 0x13fe00000-0x13fffffff] page 2M [ 0.000000] BRK [0x01b25000, 0x01b25fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x13c000000-0x13fdfffff] [ 0.000000] [mem 0x13c000000-0x13fdfffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x100001000-0x13bffffff] [ 0.000000] [mem 0x100001000-0x1001fffff] page 4k [ 0.000000] [mem 0x100200000-0x13bffffff] page 2M [ 0.000000] BRK [0x01b26000, 0x01b26fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x00100000-0xbda5efff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0xbd9fffff] page 2M [ 0.000000] [mem 0xbda00000-0xbda5efff] page 4k [ 0.000000] init_memory_mapping: [mem 0xbdd79000-0xbdefffff] [ 0.000000] [mem 0xbdd79000-0xbdefffff] page 4k [ 0.000000] RAMDISK: [mem 0x37a58000-0x37d23fff] [ 0.000000] ACPI: RSDP 00000000000f0450 00024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 00000000bdaad070 00054 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 00000000bdab5d18 000F4 (v04 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130328/tbfadt-603) [ 0.000000] ACPI: DSDT 00000000bdaad158 08BBC (v02 ALASKA A M I 00000000 INTL 20051117) [ 0.000000] ACPI: FACS 00000000bdb0ef80 00040 [ 0.000000] ACPI: APIC 00000000bdab5e10 0009E (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: MCFG 00000000bdab5eb0 0003C (v01 ALASKA A M I 01072009 MSFT 00010013) [ 0.000000] ACPI: HPET 00000000bdab5ef0 00038 (v01 ALASKA A M I 01072009 AMI 00000004) [ 0.000000] ACPI: SSDT 00000000bdab6048 008BC (v01 AMD POWERNOW 00000001 AMD 00000001) [ 0.000000] ACPI: IVRS 00000000bdab5f80 000C8 (v01 AMD RD890S 00202031 AMD 00000000) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] Scanning NUMA topology in Northbridge 24 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x13fffffff] [ 0.000000] NODE_DATA [mem 0x13fff7000-0x13fffbfff] [ 0.000000] [ffffea0000000000-ffffea0004ffffff] PMD -> [ffff88013b600000-ffff88013f5fffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0xbda5efff] [ 0.000000] node 0: [mem 0xbdd79000-0xbdefffff] [ 0.000000] node 0: [mem 0x100001000-0x13fffffff] [ 0.000000] On node 0 totalpages: 1039233 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12080 pages used for memmap [ 0.000000] DMA32 zone: 773094 pages, LIFO batch:31 [ 0.000000] Normal zone: 4096 pages used for memmap [ 0.000000] Normal zone: 262143 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x808 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x05] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 5, version 33, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x06] address[0xfec20000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 6, version 33, address 0xfec20000, GSI 24-55 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ2 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x43538210 base: 0xfed00000 [ 0.000000] smpboot: Allowing 8 CPUs, 4 hotplug CPUs [ 0.000000] nr_irqs_gsi: 72 [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 000000000009e000 [ 0.000000] PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000 [ 0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000 [ 0.000000] PM: Registered nosave memory: 00000000bda5f000 - 00000000bdaad000 [ 0.000000] PM: Registered nosave memory: 00000000bdaad000 - 00000000bdab7000 [ 0.000000] PM: Registered nosave memory: 00000000bdab7000 - 00000000bdab8000 [ 0.000000] PM: Registered nosave memory: 00000000bdab8000 - 00000000bdadd000 [ 0.000000] PM: Registered nosave memory: 00000000bdadd000 - 00000000bdaee000 [ 0.000000] PM: Registered nosave memory: 00000000bdaee000 - 00000000bdb03000 [ 0.000000] PM: Registered nosave memory: 00000000bdb03000 - 00000000bdb05000 [ 0.000000] PM: Registered nosave memory: 00000000bdb05000 - 00000000bdb0e000 [ 0.000000] PM: Registered nosave memory: 00000000bdb0e000 - 00000000bdb14000 [ 0.000000] PM: Registered nosave memory: 00000000bdb14000 - 00000000bdb76000 [ 0.000000] PM: Registered nosave memory: 00000000bdb76000 - 00000000bdd79000 [ 0.000000] PM: Registered nosave memory: 00000000bdf00000 - 00000000f8000000 [ 0.000000] PM: Registered nosave memory: 00000000f8000000 - 00000000fc000000 [ 0.000000] PM: Registered nosave memory: 00000000fc000000 - 00000000fec00000 [ 0.000000] PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000 [ 0.000000] PM: Registered nosave memory: 00000000fec01000 - 00000000fec10000 [ 0.000000] PM: Registered nosave memory: 00000000fec10000 - 00000000fec11000 [ 0.000000] PM: Registered nosave memory: 00000000fec11000 - 00000000fec20000 [ 0.000000] PM: Registered nosave memory: 00000000fec20000 - 00000000fec21000 [ 0.000000] PM: Registered nosave memory: 00000000fec21000 - 00000000fed00000 [ 0.000000] PM: Registered nosave memory: 00000000fed00000 - 00000000fed01000 [ 0.000000] PM: Registered nosave memory: 00000000fed01000 - 00000000fed61000 [ 0.000000] PM: Registered nosave memory: 00000000fed61000 - 00000000fed71000 [ 0.000000] PM: Registered nosave memory: 00000000fed71000 - 00000000fed80000 [ 0.000000] PM: Registered nosave memory: 00000000fed80000 - 00000000fed90000 [ 0.000000] PM: Registered nosave memory: 00000000fed90000 - 00000000fef00000 [ 0.000000] PM: Registered nosave memory: 00000000fef00000 - 0000000100000000 [ 0.000000] PM: Registered nosave memory: 0000000100000000 - 0000000100001000 [ 0.000000] e820: [mem 0xbdf00000-0xf7ffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:8 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 29 pages/cpu @ffff88013fc00000 s86144 r8192 d24448 u262144 [ 0.000000] pcpu-alloc: s86144 r8192 d24448 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1022972 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=328873c3-cc85-4c75-8b77-ac9a7763d532 ro ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 iommu=pt iommu=1 pci-stub.ids=1002:6739,1002:aa88 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Node 0: aperture @ f8000000 size 64 MB [ 0.000000] Memory: 4009760k/5242880k available (4921k kernel code, 1085948k absent, 147172k reserved, 4045k data, 1124k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] Dump stacks of tasks blocking RCU-preempt GP. [ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8. [ 0.000000] NR_IRQS:4352 nr_irqs:1288 16 [ 0.000000] spurious 8259A interrupt: IRQ7. [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] allocated 16777216 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.003333] tsc: Detected 3009.740 MHz processor [ 0.000003] Calibrating delay loop (skipped), value calculated using timer frequency.. 6021.29 BogoMIPS (lpj=10032466) [ 0.000006] pid_max: default: 32768 minimum: 301 [ 0.000033] Security Framework initialized [ 0.000040] AppArmor: AppArmor disabled by boot time parameter [ 0.000265] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.001579] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.002179] Mount-cache hash table entries: 256 [ 0.002377] Initializing cgroup subsys memory [ 0.002389] Initializing cgroup subsys devices [ 0.002391] Initializing cgroup subsys freezer [ 0.002392] Initializing cgroup subsys net_cls [ 0.002393] Initializing cgroup subsys blkio [ 0.002416] tseg: 00bdf00000 [ 0.002418] CPU: Physical Processor ID: 0 [ 0.002419] CPU: Processor Core ID: 0 [ 0.002421] mce: CPU supports 6 MCE banks [ 0.002425] LVT offset 0 assigned for vector 0xf9 [ 0.002430] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8 Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64 tlb_flushall_shift: 4 [ 0.002506] Freeing SMP alternatives: 20k freed [ 0.003085] ACPI: Core revision 20130328 [ 0.005536] ACPI: All ACPI Tables successfully acquired [ 0.089671] ftrace: allocating 19566 entries in 77 pages [ 0.179081] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.212064] smpboot: CPU0: AMD Athlon(tm) II X4 640 Processor (fam: 10, model: 05, stepping: 03) [ 0.316828] Performance Events: AMD PMU driver. [ 0.316832] ... version: 0 [ 0.316833] ... bit width: 48 [ 0.316833] ... generic registers: 4 [ 0.316834] ... value mask: 0000ffffffffffff [ 0.316835] ... max period: 00007fffffffffff [ 0.316836] ... fixed-purpose events: 0 [ 0.316837] ... event mask: 000000000000000f [ 0.336903] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.343565] smpboot: Booting Node 0, Processors #1 #2 #3 [ 0.386697] Brought up 4 CPUs [ 0.386701] smpboot: Total of 4 processors activated (24087.16 BogoMIPS) [ 0.387576] devtmpfs: initialized [ 0.388975] PM: Registering ACPI NVS region [mem 0xbda5f000-0xbdaacfff] (319488 bytes) [ 0.388984] PM: Registering ACPI NVS region [mem 0xbdab7000-0xbdab7fff] (4096 bytes) [ 0.388985] PM: Registering ACPI NVS region [mem 0xbdadd000-0xbdaedfff] (69632 bytes) [ 0.388987] PM: Registering ACPI NVS region [mem 0xbdb03000-0xbdb04fff] (8192 bytes) [ 0.388988] PM: Registering ACPI NVS region [mem 0xbdb0e000-0xbdb13fff] (24576 bytes) [ 0.388990] PM: Registering ACPI NVS region [mem 0xbdb76000-0xbdd78fff] (2109440 bytes) [ 0.389867] RTC time: 20:47:39, date: 08/06/13 [ 0.389916] NET: Registered protocol family 16 [ 0.390044] node 0 link 0: io port [b000, ffff] [ 0.390046] TOM: 00000000c0000000 aka 3072M [ 0.390048] Fam 10h mmconf [mem 0xe0000000-0xefffffff] [ 0.390050] node 0 link 0: mmio [c0000000, fef0ffff] ==> [c0000000, dfffffff] and [f0000000, fef0ffff] [ 0.390053] TOM2: 0000000140000000 aka 5120M [ 0.390055] bus: [bus 00-1f] on node 0 link 0 [ 0.390056] bus: 00 [io 0x0000-0xffff] [ 0.390057] bus: 00 [mem 0xc0000000-0xdfffffff] [ 0.390058] bus: 00 [mem 0xf0000000-0xffffffff] [ 0.390059] bus: 00 [mem 0x140000000-0xfcffffffff] [ 0.390097] ACPI: bus type PCI registered [ 0.390165] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.390167] PCI: not using MMCONFIG [ 0.390168] PCI: Using configuration type 1 for base access [ 0.390169] PCI: Using configuration type 1 for extended access [ 0.390333] mtrr: your CPUs had inconsistent variable MTRR settings [ 0.390334] mtrr: probably your BIOS does not setup all CPUs. [ 0.390335] mtrr: corrected configuration. [ 0.390842] bio: create slab <bio-0> at 0 [ 0.391015] ACPI: Added _OSI(Module Device) [ 0.391016] ACPI: Added _OSI(Processor Device) [ 0.391018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.391019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.391718] ACPI: EC: Look up EC in DSDT [ 0.392488] ACPI: Executed 2 blocks of module-level executable AML code [ 0.407260] ACPI: Interpreter enabled [ 0.407272] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130328/hwxface-568) [ 0.407283] ACPI: (supports S0 S1 S3 S4 S5) [ 0.407285] ACPI: Using IOAPIC for interrupt routing [ 0.407393] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.407424] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.419889] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.419975] ACPI: No dock devices found. [ 0.432894] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.433116] PCI host bridge to bus 0000:00 [ 0.433120] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.433122] pci_bus 0000:00: root bus resource [io 0x0000-0x03af] [ 0.433123] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7] [ 0.433125] pci_bus 0000:00: root bus resource [io 0x03b0-0x03df] [ 0.433126] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.433128] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.433130] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff] [ 0.433131] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xffffffff] [ 0.433142] pci 0000:00:00.0: [1002:5a14] type 00 class 0x060000 [ 0.433227] pci 0000:00:00.2: [1002:5a23] type 00 class 0x080600 [ 0.433331] pci 0000:00:02.0: [1002:5a16] type 01 class 0x060400 [ 0.433360] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 0.433392] pci 0000:00:02.0: System wakeup disabled by ACPI [ 0.433421] pci 0000:00:04.0: [1002:5a18] type 01 class 0x060400 [ 0.433449] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold [ 0.433480] pci 0000:00:04.0: System wakeup disabled by ACPI [ 0.433507] pci 0000:00:06.0: [1002:5a1a] type 01 class 0x060400 [ 0.433534] pci 0000:00:06.0: PME# supported from D0 D3hot D3cold [ 0.433567] pci 0000:00:06.0: System wakeup disabled by ACPI [ 0.433595] pci 0000:00:0a.0: [1002:5a1d] type 01 class 0x060400 [ 0.433622] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold [ 0.433659] pci 0000:00:0a.0: System wakeup disabled by ACPI [ 0.433693] pci 0000:00:11.0: [1002:4391] type 00 class 0x010601 [ 0.433708] pci 0000:00:11.0: reg 10: [io 0xf040-0xf047] [ 0.433715] pci 0000:00:11.0: reg 14: [io 0xf030-0xf033] [ 0.433722] pci 0000:00:11.0: reg 18: [io 0xf020-0xf027] [ 0.433728] pci 0000:00:11.0: reg 1c: [io 0xf010-0xf013] [ 0.433735] pci 0000:00:11.0: reg 20: [io 0xf000-0xf00f] [ 0.433742] pci 0000:00:11.0: reg 24: [mem 0xfeb0b000-0xfeb0b3ff] [ 0.433816] pci 0000:00:12.0: [1002:4397] type 00 class 0x0c0310 [ 0.433826] pci 0000:00:12.0: reg 10: [mem 0xfeb0a000-0xfeb0afff] [ 0.433878] pci 0000:00:12.0: System wakeup disabled by ACPI [ 0.433910] pci 0000:00:12.2: [1002:4396] type 00 class 0x0c0320 [ 0.433923] pci 0000:00:12.2: reg 10: [mem 0xfeb09000-0xfeb090ff] [ 0.433982] pci 0000:00:12.2: supports D1 D2 [ 0.433983] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot [ 0.434003] pci 0000:00:12.2: System wakeup disabled by ACPI [ 0.434033] pci 0000:00:13.0: [1002:4397] type 00 class 0x0c0310 [ 0.434043] pci 0000:00:13.0: reg 10: [mem 0xfeb08000-0xfeb08fff] [ 0.434096] pci 0000:00:13.0: System wakeup disabled by ACPI [ 0.434142] pci 0000:00:13.2: [1002:4396] type 00 class 0x0c0320 [ 0.434155] pci 0000:00:13.2: reg 10: [mem 0xfeb07000-0xfeb070ff] [ 0.434214] pci 0000:00:13.2: supports D1 D2 [ 0.434215] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot [ 0.434236] pci 0000:00:13.2: System wakeup disabled by ACPI [ 0.434267] pci 0000:00:14.0: [1002:4385] type 00 class 0x0c0500 [ 0.434342] pci 0000:00:14.2: [1002:4383] type 00 class 0x040300 [ 0.434358] pci 0000:00:14.2: reg 10: [mem 0xfeb00000-0xfeb03fff 64bit] [ 0.434405] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold [ 0.434419] pci 0000:00:14.2: System wakeup disabled by ACPI [ 0.434445] pci 0000:00:14.3: [1002:439d] type 00 class 0x060100 [ 0.434523] pci 0000:00:14.4: [1002:4384] type 01 class 0x060401 [ 0.434558] pci 0000:00:14.4: System wakeup disabled by ACPI [ 0.434583] pci 0000:00:14.5: [1002:4399] type 00 class 0x0c0310 [ 0.434593] pci 0000:00:14.5: reg 10: [mem 0xfeb06000-0xfeb06fff] [ 0.434652] pci 0000:00:14.5: System wakeup disabled by ACPI [ 0.434680] pci 0000:00:16.0: [1002:4397] type 00 class 0x0c0310 [ 0.434690] pci 0000:00:16.0: reg 10: [mem 0xfeb05000-0xfeb05fff] [ 0.434746] pci 0000:00:16.0: System wakeup disabled by ACPI [ 0.434777] pci 0000:00:16.2: [1002:4396] type 00 class 0x0c0320 [ 0.434791] pci 0000:00:16.2: reg 10: [mem 0xfeb04000-0xfeb040ff] [ 0.434850] pci 0000:00:16.2: supports D1 D2 [ 0.434851] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot [ 0.434874] pci 0000:00:16.2: System wakeup disabled by ACPI [ 0.434903] pci 0000:00:18.0: [1022:1200] type 00 class 0x060000 [ 0.434964] pci 0000:00:18.1: [1022:1201] type 00 class 0x060000 [ 0.435018] pci 0000:00:18.2: [1022:1202] type 00 class 0x060000 [ 0.435071] pci 0000:00:18.3: [1022:1203] type 00 class 0x060000 [ 0.435127] pci 0000:00:18.4: [1022:1204] type 00 class 0x060000 [ 0.435223] pci 0000:01:00.0: [1002:6739] type 00 class 0x030000 [ 0.435237] pci 0000:01:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.435249] pci 0000:01:00.0: reg 18: [mem 0xfea20000-0xfea3ffff 64bit] [ 0.435256] pci 0000:01:00.0: reg 20: [io 0xe000-0xe0ff] [ 0.435270] pci 0000:01:00.0: reg 30: [mem 0xfea00000-0xfea1ffff pref] [ 0.435305] pci 0000:01:00.0: supports D1 D2 [ 0.435370] pci 0000:01:00.1: [1002:aa88] type 00 class 0x040300 [ 0.435385] pci 0000:01:00.1: reg 10: [mem 0xfea40000-0xfea43fff 64bit] [ 0.435450] pci 0000:01:00.1: supports D1 D2 [ 0.443351] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.443358] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.443361] pci 0000:00:02.0: bridge window [mem 0xfea00000-0xfeafffff] [ 0.443364] pci 0000:00:02.0: bridge window [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.443432] pci 0000:02:00.0: [10ec:8168] type 00 class 0x020000 [ 0.443445] pci 0000:02:00.0: reg 10: [io 0xd000-0xd0ff] [ 0.443464] pci 0000:02:00.0: reg 18: [mem 0xd0804000-0xd0804fff 64bit pref] [ 0.443476] pci 0000:02:00.0: reg 20: [mem 0xd0800000-0xd0803fff 64bit pref] [ 0.443528] pci 0000:02:00.0: supports D1 D2 [ 0.443529] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.453345] pci 0000:00:04.0: PCI bridge to [bus 02] [ 0.453352] pci 0000:00:04.0: bridge window [io 0xd000-0xdfff] [ 0.453356] pci 0000:00:04.0: bridge window [mem 0xd0800000-0xd08fffff 64bit pref] [ 0.453436] pci 0000:03:00.0: [197b:2362] type 00 class 0x010185 [ 0.453459] pci 0000:03:00.0: reg 10: [io 0xc040-0xc047] [ 0.453470] pci 0000:03:00.0: reg 14: [io 0xc030-0xc033] [ 0.453480] pci 0000:03:00.0: reg 18: [io 0xc020-0xc027] [ 0.453490] pci 0000:03:00.0: reg 1c: [io 0xc010-0xc013] [ 0.453500] pci 0000:03:00.0: reg 20: [io 0xc000-0xc00f] [ 0.453510] pci 0000:03:00.0: reg 24: [mem 0xfe910000-0xfe9101ff] [ 0.453520] pci 0000:03:00.0: reg 30: [mem 0x00000000-0x0000ffff pref] [ 0.453560] pci 0000:03:00.0: PME# supported from D3hot [ 0.463340] pci 0000:00:06.0: PCI bridge to [bus 03] [ 0.463346] pci 0000:00:06.0: bridge window [io 0xc000-0xcfff] [ 0.463349] pci 0000:00:06.0: bridge window [mem 0xfe900000-0xfe9fffff] [ 0.463435] pci 0000:04:00.0: [197b:2362] type 00 class 0x010185 [ 0.463458] pci 0000:04:00.0: reg 10: [io 0xb040-0xb047] [ 0.463468] pci 0000:04:00.0: reg 14: [io 0xb030-0xb033] [ 0.463478] pci 0000:04:00.0: reg 18: [io 0xb020-0xb027] [ 0.463488] pci 0000:04:00.0: reg 1c: [io 0xb010-0xb013] [ 0.463498] pci 0000:04:00.0: reg 20: [io 0xb000-0xb00f] [ 0.463508] pci 0000:04:00.0: reg 24: [mem 0xfe810000-0xfe8101ff] [ 0.463518] pci 0000:04:00.0: reg 30: [mem 0x00000000-0x0000ffff pref] [ 0.463558] pci 0000:04:00.0: PME# supported from D3hot [ 0.473334] pci 0000:00:0a.0: PCI bridge to [bus 04] [ 0.473340] pci 0000:00:0a.0: bridge window [io 0xb000-0xbfff] [ 0.473343] pci 0000:00:0a.0: bridge window [mem 0xfe800000-0xfe8fffff] [ 0.473425] pci 0000:05:05.0: [102b:0519] type 00 class 0x030000 [ 0.473439] pci 0000:05:05.0: reg 10: [mem 0xfe710000-0xfe713fff] [ 0.473449] pci 0000:05:05.0: reg 14: [mem 0xd0000000-0xd07fffff pref] [ 0.473490] pci 0000:05:05.0: reg 30: [mem 0xfe700000-0xfe70ffff pref] [ 0.473554] pci 0000:00:14.4: PCI bridge to [bus 05] (subtractive decode) [ 0.473559] pci 0000:00:14.4: bridge window [mem 0xfe700000-0xfe7fffff] [ 0.473562] pci 0000:00:14.4: bridge window [mem 0xd0000000-0xd07fffff pref] [ 0.473564] pci 0000:00:14.4: bridge window [io 0x0000-0x03af] (subtractive decode) [ 0.473565] pci 0000:00:14.4: bridge window [io 0x03e0-0x0cf7] (subtractive decode) [ 0.473567] pci 0000:00:14.4: bridge window [io 0x03b0-0x03df] (subtractive decode) [ 0.473568] pci 0000:00:14.4: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.473570] pci 0000:00:14.4: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.473572] pci 0000:00:14.4: bridge window [mem 0x000c0000-0x000dffff] (subtractive decode) [ 0.473573] pci 0000:00:14.4: bridge window [mem 0xc0000000-0xffffffff] (subtractive decode) [ 0.473589] acpi PNP0A03:00: ACPI _OSC support notification failed, disabling PCIe ASPM [ 0.473591] acpi PNP0A03:00: Unable to request _OSC control (_OSC support mask: 0x08) [ 0.473916] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 7 10 11 14 15) *0 [ 0.473968] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 7 10 11 14 15) *0 [ 0.474019] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 7 10 11 14 15) *0 [ 0.474068] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 10 11 14 15) *0 [ 0.474108] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 7 10 11 14 15) *0 [ 0.474139] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 7 10 11 14 15) *0 [ 0.474171] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 7 10 11 14 15) *0 [ 0.474202] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 7 10 11 14 15) *0 [ 0.474297] acpi root: \_SB_.PCI0 notify handler is installed [ 0.474332] Found 1 acpi root devices [ 0.474350] ACPI: EC: GPE = 0xa, I/O: command/status = 0x66, data = 0x62 [ 0.474538] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.474543] vgaarb: device added: PCI:0000:05:05.0,decodes=io+mem,owns=io+mem,locks=none [ 0.474544] vgaarb: loaded [ 0.474545] vgaarb: bridge control possible 0000:05:05.0 [ 0.474545] vgaarb: bridge control possible 0000:01:00.0 [ 0.474583] PCI: Using ACPI for IRQ routing [ 0.480744] PCI: pci_cache_line_size set to 64 bytes [ 0.480829] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff] [ 0.480831] e820: reserve RAM buffer [mem 0xbda5f000-0xbfffffff] [ 0.480832] e820: reserve RAM buffer [mem 0xbdf00000-0xbfffffff] [ 0.480908] NetLabel: Initializing [ 0.480909] NetLabel: domain hash size = 128 [ 0.480909] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.480919] NetLabel: unlabeled traffic allowed by default [ 0.480939] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.480942] hpet0: 3 comparators, 32-bit 14.318180 MHz counter [ 0.482962] Switching to clocksource hpet [ 0.486666] pnp: PnP ACPI init [ 0.486681] ACPI: bus type PNP registered [ 0.486790] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved [ 0.486794] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.487224] system 00:01: [io 0x040b] has been reserved [ 0.487226] system 00:01: [io 0x04d6] has been reserved [ 0.487228] system 00:01: [io 0x0c00-0x0c01] has been reserved [ 0.487229] system 00:01: [io 0x0c14] has been reserved [ 0.487231] system 00:01: [io 0x0c50-0x0c51] has been reserved [ 0.487233] system 00:01: [io 0x0c52] has been reserved [ 0.487234] system 00:01: [io 0x0c6c] has been reserved [ 0.487236] system 00:01: [io 0x0c6f] has been reserved [ 0.487237] system 00:01: [io 0x0cd0-0x0cd1] has been reserved [ 0.487239] system 00:01: [io 0x0cd2-0x0cd3] has been reserved [ 0.487240] system 00:01: [io 0x0cd4-0x0cd5] has been reserved [ 0.487242] system 00:01: [io 0x0cd6-0x0cd7] has been reserved [ 0.487243] system 00:01: [io 0x0cd8-0x0cdf] has been reserved [ 0.487245] system 00:01: [io 0x0800-0x089f] has been reserved [ 0.487247] system 00:01: [io 0x0b20-0x0b3f] has been reserved [ 0.487250] system 00:01: [io 0x0900-0x090f] has been reserved [ 0.487252] system 00:01: [io 0x0910-0x091f] has been reserved [ 0.487253] system 00:01: [io 0xfe00-0xfefe] has been reserved [ 0.487256] system 00:01: [mem 0xfec00000-0xfec00fff] could not be reserved [ 0.487258] system 00:01: [mem 0xfee00000-0xfee00fff] has been reserved [ 0.487259] system 00:01: [mem 0xfed80000-0xfed8ffff] has been reserved [ 0.487261] system 00:01: [mem 0xfed61000-0xfed70fff] has been reserved [ 0.487263] system 00:01: [mem 0xfec10000-0xfec10fff] has been reserved [ 0.487265] system 00:01: [mem 0xfed00000-0xfed00fff] has been reserved [ 0.487266] system 00:01: [mem 0xffc00000-0xffffffff] has been reserved [ 0.487269] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.487372] system 00:02: [io 0x0290-0x02af] has been reserved [ 0.487374] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.487385] pnp 00:03: [dma 4] [ 0.487401] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active) [ 0.487432] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.487450] pnp 00:05: Plug and Play ACPI device, IDs PNP0800 (active) [ 0.487498] system 00:06: [io 0x04d0-0x04d1] has been reserved [ 0.487500] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.487524] pnp 00:07: Plug and Play ACPI device, IDs PNP0c04 (active) [ 0.487553] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.487671] system 00:09: [mem 0xfeb20000-0xfeb23fff] could not be reserved [ 0.487673] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.487750] system 00:0a: [mem 0xfec20000-0xfec200ff] could not be reserved [ 0.487752] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.487889] pnp 00:0b: Plug and Play ACPI device, IDs PNP0103 (active) [ 0.487894] pnp: PnP ACPI: found 12 devices [ 0.487895] ACPI: bus type PNP unregistered [ 0.494204] pci 0000:00:06.0: BAR 15: assigned [mem 0xd0900000-0xd09fffff pref] [ 0.494209] pci 0000:00:0a.0: BAR 15: assigned [mem 0xd0a00000-0xd0afffff pref] [ 0.494211] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.494214] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.494217] pci 0000:00:02.0: bridge window [mem 0xfea00000-0xfeafffff] [ 0.494219] pci 0000:00:02.0: bridge window [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.494222] pci 0000:00:04.0: PCI bridge to [bus 02] [ 0.494223] pci 0000:00:04.0: bridge window [io 0xd000-0xdfff] [ 0.494227] pci 0000:00:04.0: bridge window [mem 0xd0800000-0xd08fffff 64bit pref] [ 0.494230] pci 0000:03:00.0: BAR 6: assigned [mem 0xd0900000-0xd090ffff pref] [ 0.494231] pci 0000:00:06.0: PCI bridge to [bus 03] [ 0.494233] pci 0000:00:06.0: bridge window [io 0xc000-0xcfff] [ 0.494236] pci 0000:00:06.0: bridge window [mem 0xfe900000-0xfe9fffff] [ 0.494238] pci 0000:00:06.0: bridge window [mem 0xd0900000-0xd09fffff pref] [ 0.494241] pci 0000:04:00.0: BAR 6: assigned [mem 0xd0a00000-0xd0a0ffff pref] [ 0.494242] pci 0000:00:0a.0: PCI bridge to [bus 04] [ 0.494244] pci 0000:00:0a.0: bridge window [io 0xb000-0xbfff] [ 0.494246] pci 0000:00:0a.0: bridge window [mem 0xfe800000-0xfe8fffff] [ 0.494281] pci 0000:00:0a.0: bridge window [mem 0xd0a00000-0xd0afffff pref] [ 0.494284] pci 0000:00:14.4: PCI bridge to [bus 05] [ 0.494288] pci 0000:00:14.4: bridge window [mem 0xfe700000-0xfe7fffff] [ 0.494291] pci 0000:00:14.4: bridge window [mem 0xd0000000-0xd07fffff pref] [ 0.494499] pci_bus 0000:00: resource 4 [io 0x0000-0x03af] [ 0.494500] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7] [ 0.494502] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df] [ 0.494503] pci_bus 0000:00: resource 7 [io 0x0d00-0xffff] [ 0.494505] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff] [ 0.494507] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff] [ 0.494508] pci_bus 0000:00: resource 10 [mem 0xc0000000-0xffffffff] [ 0.494510] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.494511] pci_bus 0000:01: resource 1 [mem 0xfea00000-0xfeafffff] [ 0.494513] pci_bus 0000:01: resource 2 [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.494515] pci_bus 0000:02: resource 0 [io 0xd000-0xdfff] [ 0.494516] pci_bus 0000:02: resource 2 [mem 0xd0800000-0xd08fffff 64bit pref] [ 0.494518] pci_bus 0000:03: resource 0 [io 0xc000-0xcfff] [ 0.494520] pci_bus 0000:03: resource 1 [mem 0xfe900000-0xfe9fffff] [ 0.494521] pci_bus 0000:03: resource 2 [mem 0xd0900000-0xd09fffff pref] [ 0.494523] pci_bus 0000:04: resource 0 [io 0xb000-0xbfff] [ 0.494524] pci_bus 0000:04: resource 1 [mem 0xfe800000-0xfe8fffff] [ 0.494526] pci_bus 0000:04: resource 2 [mem 0xd0a00000-0xd0afffff pref] [ 0.494527] pci_bus 0000:05: resource 1 [mem 0xfe700000-0xfe7fffff] [ 0.494529] pci_bus 0000:05: resource 2 [mem 0xd0000000-0xd07fffff pref] [ 0.494531] pci_bus 0000:05: resource 4 [io 0x0000-0x03af] [ 0.494532] pci_bus 0000:05: resource 5 [io 0x03e0-0x0cf7] [ 0.494533] pci_bus 0000:05: resource 6 [io 0x03b0-0x03df] [ 0.494535] pci_bus 0000:05: resource 7 [io 0x0d00-0xffff] [ 0.494536] pci_bus 0000:05: resource 8 [mem 0x000a0000-0x000bffff] [ 0.494538] pci_bus 0000:05: resource 9 [mem 0x000c0000-0x000dffff] [ 0.494539] pci_bus 0000:05: resource 10 [mem 0xc0000000-0xffffffff] [ 0.494576] NET: Registered protocol family 2 [ 0.494761] TCP established hash table entries: 32768 (order: 7, 524288 bytes) [ 0.494924] TCP bind hash table entries: 32768 (order: 7, 524288 bytes) [ 0.495078] TCP: Hash tables configured (established 32768 bind 32768) [ 0.495131] TCP: reno registered [ 0.495141] UDP hash table entries: 2048 (order: 4, 65536 bytes) [ 0.495165] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes) [ 0.495264] NET: Registered protocol family 1 [ 1.555947] pci 0000:05:05.0: Boot video device [ 1.555953] PCI: CLS 64 bytes, default 64 [ 1.555998] Unpacking initramfs... [ 1.600551] Freeing initrd memory: 2864k freed [ 1.601842] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.601843] AMD-Vi: Interrupt remapping enabled [ 1.601858] pci 0000:00:00.2: irq 72 for MSI/MSI-X [ 1.602091] AMD-Vi: Initialized for Passthrough Mode [ 1.684133] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.684135] software IO TLB [mem 0xb9a5f000-0xbda5f000] (64MB) mapped at [ffff8800b9a5f000-ffff8800bda5efff] [ 1.684267] LVT offset 1 assigned for vector 0x400 [ 1.684271] IBS: LVT offset 1 assigned [ 1.684303] perf: AMD IBS detected (0x0000001f) [ 1.684326] Scanning for low memory corruption every 60 seconds [ 1.684502] audit: initializing netlink socket (disabled) [ 1.684512] type=2000 audit(1375822060.419:1): initialized [ 1.693786] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.695305] VFS: Disk quotas dquot_6.5.2 [ 1.695355] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.695547] msgmni has been set to 7837 [ 1.695958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 1.696084] io scheduler noop registered [ 1.696087] io scheduler deadline registered [ 1.696119] io scheduler cfq registered (default) [ 1.696535] vesafb: mode is 800x600x32, linelength=3200, pages=0 [ 1.696538] vesafb: scrolling: redraw [ 1.696539] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0 [ 1.696642] vesafb: framebuffer at 0xd0000000, mapped to 0xffffc90010800000, using 1920k, total 1920k [ 1.836131] Console: switching to colour frame buffer device 100x37 [ 1.974614] fb0: VESA VGA frame buffer device [ 1.974645] GHES: HEST is not enabled! [ 1.974720] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 1.975117] Linux agpgart interface v0.103 [ 1.975160] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 1.975567] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.975592] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.975820] mousedev: PS/2 mouse device common for all mice [ 1.975864] rtc_cmos 00:04: RTC can wake from S4 [ 1.975992] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0 [ 1.976016] rtc_cmos 00:04: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 1.976024] cpuidle: using governor ladder [ 1.976025] cpuidle: using governor menu [ 1.976077] drop_monitor: Initializing network drop monitor service [ 1.976134] TCP: cubic registered [ 1.976209] NET: Registered protocol family 10 [ 1.976352] NET: Registered protocol family 17 [ 1.976361] Key type dns_resolver registered [ 1.976651] PM: Hibernation image not present or could not be loaded. [ 1.976659] registered taskstats version 1 [ 1.977428] Magic number: 9:819:800 [ 1.977528] rtc_cmos 00:04: setting system clock to 2013-08-06 20:47:41 UTC (1375822061) [ 1.978344] Freeing unused kernel memory: 1124k freed [ 1.978626] Write protecting the kernel read-only data: 8192k [ 1.981160] Freeing unused kernel memory: 1212k freed [ 1.982425] Freeing unused kernel memory: 408k freed [ 1.997013] systemd-udevd[58]: starting version 204 [ 1.998130] pci-stub: add 1002:6739 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.998486] pci-stub 0000:01:00.0: claimed by stub [ 1.998548] pci-stub: add 1002:AA88 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.998590] pci-stub 0000:01:00.1: claimed by stub [ 2.024612] SCSI subsystem initialized [ 2.024825] ACPI: bus type USB registered [ 2.024864] usbcore: registered new interface driver usbfs [ 2.024877] usbcore: registered new interface driver hub [ 2.028146] usbcore: registered new device driver usb [ 2.028278] ACPI: bus type ATA registered [ 2.028894] libata version 3.00 loaded. [ 2.028976] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.029187] ehci-pci: EHCI PCI platform driver [ 2.029391] ehci-pci 0000:00:12.2: EHCI Host Controller [ 2.029402] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.029407] ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 1 [ 2.029425] QUIRK: Enable AMD PLL fix [ 2.029428] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 2.029440] ehci-pci 0000:00:12.2: debug port 1 [ 2.029492] ehci-pci 0000:00:12.2: irq 17, io mem 0xfeb09000 [ 2.038798] ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00 [ 2.038974] hub 1-0:1.0: USB hub found [ 2.038978] hub 1-0:1.0: 5 ports detected [ 2.039226] ehci-pci 0000:00:13.2: EHCI Host Controller [ 2.039231] ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 2 [ 2.039234] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 2.039243] ehci-pci 0000:00:13.2: debug port 1 [ 2.039276] ehci-pci 0000:00:13.2: irq 21, io mem 0xfeb07000 [ 2.048873] ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00 [ 2.049091] hub 2-0:1.0: USB hub found [ 2.049097] hub 2-0:1.0: 5 ports detected [ 2.049357] ehci-pci 0000:00:16.2: EHCI Host Controller [ 2.049363] ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 3 [ 2.049366] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 2.049375] ehci-pci 0000:00:16.2: debug port 1 [ 2.049409] ehci-pci 0000:00:16.2: irq 23, io mem 0xfeb04000 [ 2.058786] ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00 [ 2.058998] hub 3-0:1.0: USB hub found [ 2.059003] hub 3-0:1.0: 4 ports detected [ 2.059231] ohci_hcd 0000:00:12.0: OHCI Host Controller [ 2.059237] ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 4 [ 2.059270] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfeb0a000 [ 2.116269] hub 4-0:1.0: USB hub found [ 2.116276] hub 4-0:1.0: 5 ports detected [ 2.116497] ohci_hcd 0000:00:13.0: OHCI Host Controller [ 2.116502] ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 5 [ 2.116528] ohci_hcd 0000:00:13.0: irq 20, io mem 0xfeb08000 [ 2.172906] hub 5-0:1.0: USB hub found [ 2.172913] hub 5-0:1.0: 5 ports detected [ 2.173134] ohci_hcd 0000:00:14.5: OHCI Host Controller [ 2.173139] ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 6 [ 2.173154] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfeb06000 [ 2.229540] hub 6-0:1.0: USB hub found [ 2.229547] hub 6-0:1.0: 2 ports detected [ 2.229744] ohci_hcd 0000:00:16.0: OHCI Host Controller [ 2.229749] ohci_hcd 0000:00:16.0: new USB bus registered, assigned bus number 7 [ 2.229804] ohci_hcd 0000:00:16.0: irq 22, io mem 0xfeb05000 [ 2.286169] hub 7-0:1.0: USB hub found [ 2.286177] hub 7-0:1.0: 4 ports detected [ 2.286369] ahci 0000:00:11.0: version 3.0 [ 2.286604] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 2.286607] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part [ 2.288415] scsi0 : ahci [ 2.288607] scsi1 : ahci [ 2.288821] scsi2 : ahci [ 2.289018] scsi3 : ahci [ 2.289198] scsi4 : ahci [ 2.289383] scsi5 : ahci [ 2.289485] ata1: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b100 irq 19 [ 2.289487] ata2: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b180 irq 19 [ 2.289490] ata3: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b200 irq 19 [ 2.289492] ata4: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b280 irq 19 [ 2.289494] ata5: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b300 irq 19 [ 2.289496] ata6: SATA max UDMA/133 abar m1024@0xfeb0b000 port 0xfeb0b380 irq 19 [ 2.289724] ahci 0000:03:00.0: AHCI 0001.0100 32 slots 2 ports 3 Gbps 0x3 impl SATA mode [ 2.289727] ahci 0000:03:00.0: flags: 64bit ncq pm led clo pmp pio slum part [ 2.290127] scsi6 : ahci [ 2.290308] scsi7 : ahci [ 2.290403] ata7: SATA max UDMA/133 abar m512@0xfe910000 port 0xfe910100 irq 51 [ 2.290406] ata8: SATA max UDMA/133 abar m512@0xfe910000 port 0xfe910180 irq 51 [ 2.290610] ahci 0000:04:00.0: AHCI 0001.0100 32 slots 2 ports 3 Gbps 0x3 impl SATA mode [ 2.290613] ahci 0000:04:00.0: flags: 64bit ncq pm led clo pmp pio slum part [ 2.291008] scsi8 : ahci [ 2.291214] scsi9 : ahci [ 2.291310] ata9: SATA max UDMA/133 abar m512@0xfe810000 port 0xfe810100 irq 47 [ 2.291313] ata10: SATA max UDMA/133 abar m512@0xfe810000 port 0xfe810180 irq 47 [ 2.608518] ata9: SATA link down (SStatus 0 SControl 300) [ 2.608552] ata7: SATA link down (SStatus 0 SControl 300) [ 2.608560] ata2: SATA link down (SStatus 0 SControl 300) [ 2.608604] ata5: SATA link down (SStatus 0 SControl 300) [ 2.608606] ata10: SATA link down (SStatus 0 SControl 300) [ 2.608653] ata8: SATA link down (SStatus 0 SControl 300) [ 2.608654] ata4: SATA link down (SStatus 0 SControl 300) [ 2.608685] ata6: SATA link down (SStatus 0 SControl 300) [ 2.608713] ata3: SATA link down (SStatus 0 SControl 300) [ 2.685092] tsc: Refined TSC clocksource calibration: 3010.011 MHz [ 2.685104] Switching to clocksource tsc [ 2.765042] usb 7-4: new full-speed USB device number 2 using ohci_hcd [ 2.775047] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 2.775798] ata1.00: ATA-8: ST9160827AS, 3.AAA, max UDMA/133 [ 2.775801] ata1.00: 312581808 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 2.776269] ata1.00: configured for UDMA/133 [ 2.776476] scsi 0:0:0:0: Direct-Access ATA ST9160827AS 3.AA PQ: 0 ANSI: 5 [ 2.777492] sd 0:0:0:0: [sda] 312581808 512-byte logical blocks: (160 GB/149 GiB) [ 2.777643] sd 0:0:0:0: [sda] Write Protect is off [ 2.777645] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 2.777684] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.836861] sda: sda1 sda2 sda3 sda4 [ 2.837752] sd 0:0:0:0: [sda] Attached SCSI disk [ 2.934093] hidraw: raw HID events driver (C) Jiri Kosina [ 2.954737] usbcore: registered new interface driver usbhid [ 2.954741] usbhid: USB HID core driver [ 2.955529] input: Microsoft Microsoft® Nano Transceiver v2.0 as /devices/pci0000:00/0000:00:16.0/usb7/7-4/7-4:1.0/input/input0 [ 2.955580] hid-generic 0003:045E:0745.0001: input,hidraw0: USB HID v1.11 Keyboard [Microsoft Microsoft® Nano Transceiver v2.0] on usb-0000:00:16.0-4/input0 [ 2.956007] input: Microsoft Microsoft® Nano Transceiver v2.0 as /devices/pci0000:00/0000:00:16.0/usb7/7-4/7-4:1.1/input/input1 [ 2.956144] hid-generic 0003:045E:0745.0002: input,hidraw1: USB HID v1.11 Mouse [Microsoft Microsoft® Nano Transceiver v2.0] on usb-0000:00:16.0-4/input1 [ 2.970799] input: Microsoft Microsoft® Nano Transceiver v2.0 as /devices/pci0000:00/0000:00:16.0/usb7/7-4/7-4:1.2/input/input2 [ 2.970969] hid-generic 0003:045E:0745.0003: input,hiddev0,hidraw2: USB HID v1.11 Device [Microsoft Microsoft® Nano Transceiver v2.0] on usb-0000:00:16.0-4/input2 [ 4.117161] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 5.056668] systemd[1]: systemd 204 running in system mode. (+PAM -LIBWRAP -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ) [ 5.100219] systemd[1]: Set hostname to <Virtking>. [ 5.778615] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory. See system logs and 'systemctl status display-manager.service' for details. [ 5.778815] systemd[1]: Starting Forward Password Requests to Wall Directory Watch. [ 5.778859] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ 5.778870] systemd[1]: Expecting device sys-subsystem-net-devices-eth0.device... [ 5.781751] systemd[1]: Starting Remote File Systems. [ 5.784017] systemd[1]: Reached target Remote File Systems. [ 5.784026] systemd[1]: Starting LVM2 metadata daemon socket. [ 5.786533] systemd[1]: Listening on LVM2 metadata daemon socket. [ 5.786541] systemd[1]: Starting /dev/initctl Compatibility Named Pipe. [ 5.789397] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe. [ 5.789404] systemd[1]: Starting Device-mapper event daemon FIFOs. [ 5.792079] systemd[1]: Listening on Device-mapper event daemon FIFOs. [ 5.792087] systemd[1]: Starting Delayed Shutdown Socket. [ 5.794432] systemd[1]: Listening on Delayed Shutdown Socket. [ 5.794462] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. [ 5.798466] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ 5.798531] systemd[1]: Starting udev Kernel Socket. [ 5.800678] systemd[1]: Listening on udev Kernel Socket. [ 5.800714] systemd[1]: Starting udev Control Socket. [ 5.802918] systemd[1]: Listening on udev Control Socket. [ 5.802926] systemd[1]: Starting Encrypted Volumes. [ 5.805116] systemd[1]: Reached target Encrypted Volumes. [ 5.805129] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch. [ 5.805161] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 5.805168] systemd[1]: Starting Paths. [ 5.806901] systemd[1]: Reached target Paths. [ 5.806914] systemd[1]: Starting Journal Socket. [ 5.808928] systemd[1]: Listening on Journal Socket. [ 5.838913] systemd[1]: Started Set Up Additional Binary Formats. [ 5.838933] systemd[1]: Starting Setup Virtual Console... [ 5.841446] systemd[1]: Starting udev Coldplug all Devices... [ 5.844029] systemd[1]: Starting Create static device nodes in /dev... [ 5.853689] systemd[1]: Starting Apply Kernel Variables... [ 5.856159] systemd[1]: Mounting POSIX Message Queue File System... [ 5.859606] systemd[1]: Starting Load Kernel Modules... [ 5.921078] systemd[1]: Mounting Debug File System... [ 5.983231] systemd[1]: Starting Journal Service... [ 6.105438] systemd[1]: Started Journal Service. [ 6.105490] systemd[1]: Mounting Huge Pages File System... [ 6.167710] systemd[1]: Expecting device dev-sda3.device... [ 6.227811] systemd[1]: Starting File System Check on Root Device... [ 6.287283] systemd[1]: Mounting Temporary Directory... [ 6.534911] systemd[1]: Expecting device dev-sda1.device... [ 6.648418] systemd[1]: Started Create static device nodes in /dev. [ 6.704478] systemd[1]: Started Apply Kernel Variables. [ 6.760296] systemd[1]: Mounted POSIX Message Queue File System. [ 6.815358] systemd[1]: Mounted Debug File System. [ 6.869511] systemd[1]: Mounted Huge Pages File System. [ 6.921472] systemd[1]: Mounted Temporary Directory. [ 6.972157] systemd[1]: Started udev Coldplug all Devices. [ 6.972232] systemd[1]: Starting udev Kernel Device Manager... [ 7.158863] systemd-udevd[157]: starting version 204 [ 7.453239] kvm: Nested Virtualization enabled [ 7.453244] kvm: Nested Paging enabled [ 7.956172] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input3 [ 7.956179] ACPI: Power Button [PWRB] [ 7.962263] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input4 [ 7.962270] ACPI: Power Button [PWRF] [ 7.964106] wmi: Mapper loaded [ 8.038713] EXT4-fs (sda2): re-mounted. Opts: data=ordered [ 8.214618] microcode: CPU0: patch_level=0x010000c8 [ 8.493444] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 8.493726] r8169 0000:02:00.0: irq 73 for MSI/MSI-X [ 8.493898] r8169 0000:02:00.0 eth0: RTL8168evl/8111evl at 0xffffc90011292000, c8:60:00:85:03:7f, XID 0c900800 IRQ 73 [ 8.493901] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko] [ 8.549878] MCE: In-kernel MCE decoding enabled. [ 8.558059] EDAC MC: Ver: 3.0.0 [ 8.559106] AMD64 EDAC driver v3.4.0 [ 8.559163] EDAC amd64: DRAM ECC disabled. [ 8.559181] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load. Either enable ECC checking or force module loading by setting 'ecc_enable_override'. (Note that use of the override may cause unknown side effects.) [ 8.808637] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 8.885809] input: HDA ATI SB Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card0/input5 [ 8.885839] input: HDA ATI SB Line Out Side as /devices/pci0000:00/0000:00:14.2/sound/card0/input6 [ 8.885860] input: HDA ATI SB Line Out CLFE as /devices/pci0000:00/0000:00:14.2/sound/card0/input7 [ 8.885883] input: HDA ATI SB Line Out Surround as /devices/pci0000:00/0000:00:14.2/sound/card0/input8 [ 8.885906] input: HDA ATI SB Line Out Front as /devices/pci0000:00/0000:00:14.2/sound/card0/input9 [ 8.885930] input: HDA ATI SB Line as /devices/pci0000:00/0000:00:14.2/sound/card0/input10 [ 8.885959] input: HDA ATI SB Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input11 [ 8.885981] input: HDA ATI SB Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card0/input12 [ 8.906550] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0 [ 8.963091] input: PC Speaker as /devices/platform/pcspkr/input/input13 [ 9.093876] microcode: CPU1: patch_level=0x010000c8 [ 9.093902] microcode: CPU2: patch_level=0x010000c8 [ 9.093920] microcode: CPU3: patch_level=0x010000c8 [ 9.094006] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 9.189679] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver v0.05 [ 9.189726] sp5100_tco: PCI Revision ID: 0x42 [ 9.189759] sp5100_tco: Using 0xfed80b00 for watchdog MMIO address [ 9.189774] sp5100_tco: Last reboot was not triggered by watchdog. [ 9.189875] sp5100_tco: initialized (0xffffc90012458b00). heartbeat=60 sec (nowayout=0) [ 9.250142] acpi-cpufreq: overriding BIOS provided _PSD data [ 9.309658] asus_wmi: ASUS WMI generic driver loaded [ 9.347460] asus_wmi: Initialization: 0x0 [ 9.347523] asus_wmi: BIOS WMI version: 0.9 [ 9.347637] asus_wmi: SFUN value: 0x0 [ 9.348546] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input14 [ 9.350939] asus_wmi: Disabling ACPI video driver [ 10.188668] Adding 8388604k swap on /dev/sda3. Priority:-1 extents:1 across:8388604k FS [ 10.502171] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem [ 10.525220] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered [ 11.634768] it87: Found IT8721F chip at 0x290, revision 1 [ 12.172445] r8169 0000:02:00.0 eth0: link down [ 12.172480] r8169 0000:02:00.0 eth0: link down [ 12.172533] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 14.096652] r8169 0000:02:00.0 eth0: link up [ 14.096672] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 15.830233] ip_tables: (C) 2000-2006 Netfilter Core Team [ 15.875583] ip6_tables: (C) 2000-2006 Netfilter Core Team [ 26.940720] Bridge firewalling registered [ 27.204314] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 27.267605] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready [ 146.061275] VFIO - User Level meta-driver version: 0.3 [ 158.100190] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

Last edited by Diggo (2013-08-06 18:01:19)

Offline

#417 2013-08-06 21:54:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:

So, today i took a chance to migrate to vfio-pci. But i got stuck at the point, where to start the DomU the first time. If i read the posts here right, the patched versions are for VGA-reset only (so for testing, i actually don´t need them?).

You need them

It also looks like your trying to passthrough your primary vga device, that wont work, the SDL error is because it doesnt have permission to access the X server, as a regular user try:

xhost +

Last edited by nbhs (2013-08-06 22:01:51)

Offline

#418 2013-08-06 22:07:45

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

licious wrote:

Specs :
cpu:Intel i7 3770
motherboard : Asus P8H77-I
host gpu : Intel HD 4000
guest gpu : Nvidia GT 620
Host : Ubuntu 13.04 with patched kernel 3.10.1
grub: GRUB_CMDLINE_LINUX="intel_iommu=on pci-stub.ids=8086:0151,10de:0f01,10de:0bea vfio_iommu_type1.allow_unsafe_interrupts=1"
guest : Ubuntu 13.04, kernel 3.8.0-27-generic (stock)


I've built the patched kernel, seabios and qemu - but I can't get an output on the screen connected to the GT620 when I use vfio. Qemu output is always blank / black as well. If I use "-device pci-assign" instead of "-device vfio-pci" then my guest os initializes the GT620 when starting X on the guest (no output at all before X starts). I can only get an output if I'm using the Nouveau-drivers on the guest, Nvidia binary drivers won't work. I've tried patching the drivers as well, but I've had no luck in getting the patched drivers to work. The output from dmesg on the guest :
[   17.260770] NVRM: RmInitAdapter failed! (0x26:0x38:1170)
[   17.260778] NVRM: rm_init_adapter(0) failed

the weird thing is that it seems like hdmi sound is initialized :
[   13.078985] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input5
[   13.079086] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input6
[   13.079149] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input7
[   13.079225] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1c.0/0000:01:00.1/sound/card0/input8

Also, when using -device vfio-pci I get an error message saying "qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/xbmc/GT620.rom: Warning, device 0000:01:00.0 does not support reset". This message does not appear when I use pci-assign, most likely because no kernel modules have been loaded for the GPU yet. I thought the patched kernel and/ or qemu would take care of resetting the device as well ?

From previous posts it looks like the intel driver also causes problems. Try blacklisting it and check if you get output with vfio

Offline

#419 2013-08-07 09:42:55

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

It also looks like your trying to passthrough your primary vga device, that wont work, the SDL error is because it doesnt have permission to access the X server, as a regular user try:

xhost +

Sorry, i had the primary card cut out. It is a oldschool Matrox Millenium smile

05:05.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA 2064W [Millennium] (rev 01)

I had no X server In the last setup -> I installed it and the xorg-xhost too, but still noch change. I will now try your patched kernels.

*edit*

Found the following in this thread smile

nbhs wrote:

Qemu by default creates a sdl window, if you want to lauch qemu without X launch qemu with this option

-nographic

Seems like it is doing something now smile I can´t say what, because i am not at home and i just try this over SSH wink ... but hope rises again big_smile
Thanks for all your support so far!

Last edited by Diggo (2013-08-07 10:31:45)

Offline

#420 2013-08-07 20:55:18

Diggo
Member
Registered: 2013-05-29
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm... beeing home a while and playing around with this, i installed the x server, some lxde for googling and so son ... qemu now starts without any SDL Error (inside a x session), but its just giving me a new blank window.  At least qemu seems to be running now.
Sadly the output on the tv-screen (where the passed through card is connected via hdmi) is still just black. I don´t see the seabios messages. Removing "-vga none" shows the seabios then in the mentioned qemu window.
So i guess qemu is running into seabios, but i can´t see it on the passed-through card.

Is there any more debug-information i can grab?

*update* Got it! Looked like something in the patched qemu got corrupted. Redownloaded nbhs´ packages and everything went on smoothly wink

Last edited by Diggo (2013-08-07 22:52:10)

Offline

#421 2013-08-11 03:07:21

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'll start saying that I'm trying this with gentoo so I'm restricted to certain package versions; please don't scorn me.

I'm using a 3.10.5 kernel patched with:
- kernel-vfio-vga-reset.patch.tar.gz
- pci: Enable overrides for missing ACS capabilities

QEMU 1.5.2 patched with:
- qemu-vfio-vga-reset.patch.tar.gz

seabios 1.7.2.2 patched with:
- pciinit: Enable default VGA device


The cards:

# lspci [...] 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks PRO [Radeon HD 6570] 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Turks/Whistler HDMI Audio [Radeon HD 6000 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV740 PRO [Radeon HD 4770] 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV710/730 HDMI Audio [Radeon HD 4000 series] 03:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cedar [Radeon HD 5000/6000/7350 Series] 03:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cedar HDMI Audio [Radeon HD 5400/6300 Series]
# lspci -n [...] 01:00.0 0300: 1002:6759 01:00.1 0403: 1002:aa90 02:00.0 0300: 1002:94b3 02:00.1 0403: 1002:aa38 03:00.0 0300: 1002:68f9 03:00.1 0403: 1002:aa68

Kernel options:

# cat .config | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y
# cat .config | grep IOMMU CONFIG_GART_IOMMU=y CONFIG_CALGARY_IOMMU=y CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y CONFIG_IOMMU_HELPER=y CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_IOMMU_API=y CONFIG_IOMMU_SUPPORT=y # CONFIG_AMD_IOMMU is not set CONFIG_INTEL_IOMMU=y CONFIG_INTEL_IOMMU_DEFAULT_ON=y CONFIG_INTEL_IOMMU_FLOPPY_WA=y # CONFIG_IOMMU_DEBUG is not set # CONFIG_IOMMU_STRESS is not set
# cat .config | grep CONFIG_CMDLINE CONFIG_CMDLINE_BOOL=y CONFIG_CMDLINE="root=/dev/sda2 pci-stub.ids=1002:94b3,1002:aa38,1002:68f9,1002:aa68 pcie_acs_override=downstream kvm_intel.emulate_invalid_guest_state=0 vfio_iommu_type1.allow_unsafe_interrupts=1" # CONFIG_CMDLINE_OVERRIDE is not set

My vfio-bind (I use this because all the cards used to be on the same IOMMU group, and even though now it's not the case anymore the result is the same):

#!/bin/bash for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

The cards get properly binded:

# ls -l /sys/bus/pci/drivers/vfio-pci/ total 0 lrwxrwxrwx 1 root root 0 Aug 11 05:02 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 lrwxrwxrwx 1 root root 0 Aug 11 05:02 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 lrwxrwxrwx 1 root root 0 Aug 11 05:02 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:03:00.0 lrwxrwxrwx 1 root root 0 Aug 11 05:02 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:03:00.1 --w------- 1 root root 4096 Aug 11 05:02 bind lrwxrwxrwx 1 root root 0 Aug 11 05:02 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 Aug 11 04:51 new_id --w------- 1 root root 4096 Aug 11 05:02 remove_id --w------- 1 root root 4096 Aug 11 04:51 uevent --w------- 1 root root 4096 Aug 11 05:02 unbind

dmesg when I start the test with qemu:

[ 33.741710] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003) [ 33.926077] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002) [...] [ 195.150275] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) [ 195.337386] vfio-pci 0000:02:00.1: enabling device (0000 -> 0002)

Result after all this: the screen stays black.

I tried swapping the cards and the displays, and nothing changes.
I want to think that I'm overlooking something simple. I just hope that is the case.

Well, one thing that bothers me is that some time after boot vgaarb does this:

[ 22.719722] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 22.719724] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:03:00.0

But I don't know how relevant it can be. 02:00.0 doesn't work either after all.


EDIT:

I think I'm getting closer: I tried booting archlinux-2013.08.01-dual.iso instead of just testing the bios screen. The display was still black, but I tried grabbing the input and hitting enter and after a few seconds I saw the prompt and it worked.

Now, there are still a few problems:
- If I halt the guest system, qemu doesn't close itself, and even after I forcefully kill it the last image still stays in the monitor and doesn't go away until I re-run qemu. I'm guessing this means the card isn't getting reset?
- I also tried a Windows installation disc, but with that the display stayed black and when I tried to re-run the archlinux CD I got a kernel panic after a while (drm_kms_helper: panic occurred, switching back to text console). From then on the only way to get it to work again is to reboot the host.

Meanwhile dmesg... (2 "good" runs, 1 try with the Windows CD and the last one that ended with a kernel panic)

[ 32.052373] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003) [ 32.237194] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002) [ 37.622600] kvm [2066]: vcpu0 unhandled rdmsr: 0x345 [ 39.575702] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X [ 41.154701] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X [ 134.870611] kvm [2082]: vcpu0 unhandled rdmsr: 0x345 [ 136.815406] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X [ 138.310509] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X [ 183.069869] kvm [2093]: vcpu0 unhandled rdmsr: 0x345 [ 185.015290] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X [ 186.511395] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X [ 360.835017] kvm [2163]: vcpu0 unhandled rdmsr: 0x345 [ 362.786883] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X [ 364.322938] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X

EDIT2:

Just tried to install an archlinux guest. Managed to complete the installation but when I rebooted the VM I couldn't get any kind of output from the display connected to the card. Switching to the cirrus card I could see that everything was fine and the card was there when I did lspci.
I also tried dumping the bioses of all my cards (got to mount them one at at time on the first PCIe slot since I couldn't manage to get the roms otherwise, damn Input/Output errors) and to run it with them. Nothing changed.


EDIT3:

Oh, I think I may have got it.
Changing the framebuffer device with "fbcon=map:1" lets the kernel use the right one.
So this means there is some other framebuffer device that gets priority during the boot process, namely:

fb0: EFI VGA frame buffer device

Is this part of the Q35? Time to investigate.

Last edited by Evonat (2013-08-12 05:05:34)

Offline

#422 2013-08-11 22:30:00

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hopefully we'll have this last issue snubbed out eventually. I'm good on non-vfio-pci, but vfio still blue-screens on that Radeon driver.

If this helps, I noticed also that the usb PCI device I pass through is also incredibly slow. As in, my mouse movement is very... choppy? Jerky? And the USB hotspot I hooked up worked but downloaded things VERY slowly (devices hooked up to the same spot via Wi-Fi did not have this issue). My hopes are that it's a software bug somewhere down the line, and possibly is related to having 000:05+ pci devices breaking iommu.

Offline

#423 2013-08-12 05:05:46

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I decided to try installing Windows 8, but since I couldn't manage to use the right card from the boot, I decided to try installing it with the cirrus and switching to the ATI afterwards, leaving the PCIe devices there (1), or just removing them and adding them after the installation is complete (2), or just hot plugging them once the system is fully installed (3).
Sure enough, all three failed:
BSOD with:

VIDEO_TDR_FAILURE (atikmpag.sys)

Now I'll try with pci-assign, which I wanted to avoid, but I'm losing hope.
It's quite disappointing since I built this system specifically with this in mind, and even though it's still quite immature it seemed the "cleanest" solution and I wanted to avoid the bloaded xen and vmware.


EDIT:

Almost forgot to say, the screen turned on before the BSOD, and stayed on even afterwards. On, and black.


EDIT2:

Well that worked. Well, almost: the jerky mouse makes the system unusable. The keyboard works fine and the animations seem fluid enough. Just the damn mouse. I had the same problem with the cirrus card, plus a constant "white refresh" in the the VM window (I don't know how to explain it... have you even seen what happens when you point an analog video camera to a CRT display? That.)
I even though that the problem was that I still had the VM window output enabled, so I tried "-vga none", but the screen stayed off, probably Windows refuses to boot because it can't find the primary graphic adapter or something like that; I also thought of "-nographic", but I don't know how can I grab mouse and keyboard after I start it like that.

Last edited by Evonat (2013-08-12 05:30:31)

Offline

#424 2013-08-12 14:46:47

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm trying to make it work with an archlinux guest. It seems to work fine. That is, until I decide that I want X:

X.Org X Server 1.14.2 Release Date: 2013-06-25 [ 99.537] X Protocol Version 11, Revision 0 [ 99.539] Build Operating System: Linux 3.9.7-1-ARCH x86_64 [ 99.541] Current Operating System: Linux Arch 3.10.5-1-ARCH #1 SMP PREEMPT Mon Aug 5 08:04:22 CEST 2013 x86_64 [ 99.541] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=75e17ae6-53fe-42ec-9ef6-a144c6fe404a rw fbcon=map:1 quiet [ 99.544] Build Date: 01 July 2013 10:48:42AM [ 99.546] [ 99.548] Current version of pixman: 0.30.2 [ 99.552] Before reporting problems, check http://wiki.x.org to make sure that you have the latest version. [ 99.552] Markers: (--) probed, (**) from config file, (==) default setting, (++) from command line, (!!) notice, (II) informational, (WW) warning, (EE) error, (NI) not implemented, (??) unknown. [ 99.560] (==) Log file: "/var/log/Xorg.0.log", Time: Mon Aug 12 16:28:20 2013 [ 99.563] (==) Using config directory: "/etc/X11/xorg.conf.d" [ 99.565] (==) Using system config directory "/usr/share/X11/xorg.conf.d" [ 99.565] (==) No Layout section. Using the first Screen section. [ 99.565] (==) No screen section available. Using defaults. [ 99.565] (**) |-->Screen "Default Screen Section" (0) [ 99.565] (**) | |-->Monitor "<default monitor>" [ 99.565] (==) No monitor specified for screen "Default Screen Section". Using a default monitor configuration. [ 99.565] (==) Automatically adding devices [ 99.565] (==) Automatically enabling devices [ 99.565] (==) Automatically adding GPU devices [ 99.565] (WW) The directory "/usr/share/fonts/OTF/" does not exist. [ 99.565] Entry deleted from font path. [ 99.565] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/100dpi/". [ 99.565] Entry deleted from font path. [ 99.565] (Run 'mkfontdir' on "/usr/share/fonts/100dpi/"). [ 99.565] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/75dpi/". [ 99.565] Entry deleted from font path. [ 99.565] (Run 'mkfontdir' on "/usr/share/fonts/75dpi/"). [ 99.565] (==) FontPath set to: /usr/share/fonts/misc/, /usr/share/fonts/TTF/, /usr/share/fonts/Type1/ [ 99.565] (==) ModulePath set to "/usr/lib/xorg/modules" [ 99.565] (II) The server relies on udev to provide the list of input devices. If no devices become available, reconfigure udev or disable AutoAddDevices. [ 99.565] (II) Loader magic: 0x7fdc20 [ 99.565] (II) Module ABI versions: [ 99.565] X.Org ANSI C Emulation: 0.4 [ 99.565] X.Org Video Driver: 14.1 [ 99.565] X.Org XInput driver : 19.1 [ 99.565] X.Org Server Extension : 7.0 [ 99.565] (II) xfree86: Adding drm device (/dev/dri/card0) [ 99.567] (--) PCI:*(0:1:0:0) 1002:68f9:174b:e233 rev 0, Mem @ 0xe0000000/268435456, 0xfea00000/131072, I/O @ 0x0000c000/256, BIOS @ 0x????????/131072 [ 99.569] Initializing built-in extension Generic Event Extension [ 99.571] Initializing built-in extension SHAPE [ 99.573] Initializing built-in extension MIT-SHM [ 99.575] Initializing built-in extension XInputExtension [ 99.577] Initializing built-in extension XTEST [ 99.579] Initializing built-in extension BIG-REQUESTS [ 99.581] Initializing built-in extension SYNC [ 99.582] Initializing built-in extension XKEYBOARD [ 99.584] Initializing built-in extension XC-MISC [ 99.586] Initializing built-in extension SECURITY [ 99.588] Initializing built-in extension XINERAMA [ 99.589] Initializing built-in extension XFIXES [ 99.591] Initializing built-in extension RENDER [ 99.592] Initializing built-in extension RANDR [ 99.594] Initializing built-in extension COMPOSITE [ 99.596] Initializing built-in extension DAMAGE [ 99.597] Initializing built-in extension MIT-SCREEN-SAVER [ 99.599] Initializing built-in extension DOUBLE-BUFFER [ 99.601] Initializing built-in extension RECORD [ 99.602] Initializing built-in extension DPMS [ 99.604] Initializing built-in extension X-Resource [ 99.606] Initializing built-in extension XVideo [ 99.607] Initializing built-in extension XVideo-MotionCompensation [ 99.609] Initializing built-in extension XFree86-VidModeExtension [ 99.611] Initializing built-in extension XFree86-DGA [ 99.612] Initializing built-in extension XFree86-DRI [ 99.614] Initializing built-in extension DRI2 [ 99.614] (II) "glx" will be loaded by default. [ 99.614] (II) LoadModule: "dri2" [ 99.614] (II) Module "dri2" already built-in [ 99.614] (II) LoadModule: "glamoregl" [ 99.614] (II) Loading /usr/lib/xorg/modules/libglamoregl.so [ 99.615] (II) Module glamoregl: vendor="X.Org Foundation" [ 99.615] compiled for 1.14.0, module version = 0.5.0 [ 99.615] ABI class: X.Org ANSI C Emulation, version 0.4 [ 99.615] (II) LoadModule: "glx" [ 99.615] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so [ 99.615] (II) Module glx: vendor="X.Org Foundation" [ 99.615] compiled for 1.14.2, module version = 1.0.0 [ 99.615] ABI class: X.Org Server Extension, version 7.0 [ 99.615] (==) AIGLX enabled [ 99.617] Loading extension GLX [ 99.617] (==) Matched ati as autoconfigured driver 0 [ 99.617] (==) Matched ati as autoconfigured driver 1 [ 99.617] (==) Matched vesa as autoconfigured driver 2 [ 99.617] (==) Matched modesetting as autoconfigured driver 3 [ 99.617] (==) Matched fbdev as autoconfigured driver 4 [ 99.617] (==) Assigned the driver to the xf86ConfigLayout [ 99.617] (II) LoadModule: "ati" [ 99.617] (II) Loading /usr/lib/xorg/modules/drivers/ati_drv.so [ 99.617] (II) Module ati: vendor="X.Org Foundation" [ 99.617] compiled for 1.14.2, module version = 7.2.0 [ 99.617] Module class: X.Org Video Driver [ 99.617] ABI class: X.Org Video Driver, version 14.1 [ 99.617] (II) LoadModule: "radeon" [ 99.617] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so [ 99.617] (II) Module radeon: vendor="X.Org Foundation" [ 99.617] compiled for 1.14.2, module version = 7.2.0 [ 99.617] Module class: X.Org Video Driver [ 99.617] ABI class: X.Org Video Driver, version 14.1 [ 99.617] (II) LoadModule: "vesa" [ 99.617] (WW) Warning, couldn't open module vesa [ 99.617] (II) UnloadModule: "vesa" [ 99.617] (II) Unloading vesa [ 99.617] (EE) Failed to load module "vesa" (module does not exist, 0) [ 99.617] (II) LoadModule: "modesetting" [ 99.617] (WW) Warning, couldn't open module modesetting [ 99.617] (II) UnloadModule: "modesetting" [ 99.617] (II) Unloading modesetting [ 99.617] (EE) Failed to load module "modesetting" (module does not exist, 0) [ 99.617] (II) LoadModule: "fbdev" [ 99.617] (WW) Warning, couldn't open module fbdev [ 99.617] (II) UnloadModule: "fbdev" [ 99.617] (II) Unloading fbdev [ 99.617] (EE) Failed to load module "fbdev" (module does not exist, 0) [ 99.617] (II) RADEON: Driver for ATI Radeon chipsets: ATI Radeon Mobility X600 (M24) 3150 (PCIE), ATI FireMV 2400 (PCI), ATI Radeon Mobility X300 (M24) 3152 (PCIE), ATI FireGL M24 GL 3154 (PCIE), ATI FireMV 2400 3155 (PCI), ATI Radeon X600 (RV380) 3E50 (PCIE), ATI FireGL V3200 (RV380) 3E54 (PCIE), ATI Radeon IGP320 (A3) 4136, ATI Radeon IGP330/340/350 (A4) 4137, ATI Radeon 9500 AD (AGP), ATI Radeon 9500 AE (AGP), ATI Radeon 9600TX AF (AGP), ATI FireGL Z1 AG (AGP), ATI Radeon 9800SE AH (AGP), ATI Radeon 9800 AI (AGP), ATI Radeon 9800 AJ (AGP), ATI FireGL X2 AK (AGP), ATI Radeon 9600 AP (AGP), ATI Radeon 9600SE AQ (AGP), ATI Radeon 9600XT AR (AGP), ATI Radeon 9600 AS (AGP), ATI FireGL T2 AT (AGP), ATI Radeon 9650, ATI FireGL RV360 AV (AGP), ATI Radeon 7000 IGP (A4+) 4237, ATI Radeon 8500 AIW BB (AGP), ATI Radeon IGP320M (U1) 4336, ATI Radeon IGP330M/340M/350M (U2) 4337, ATI Radeon Mobility 7000 IGP 4437, ATI Radeon 9000/PRO If (AGP/PCI), ATI Radeon 9000 Ig (AGP/PCI), ATI Radeon X800 (R420) JH (AGP), ATI Radeon X800PRO (R420) JI (AGP), ATI Radeon X800SE (R420) JJ (AGP), ATI Radeon X800 (R420) JK (AGP), ATI Radeon X800 (R420) JL (AGP), ATI FireGL X3 (R420) JM (AGP), ATI Radeon Mobility 9800 (M18) JN (AGP), ATI Radeon X800 SE (R420) (AGP), ATI Radeon X800XT (R420) JP (AGP), ATI Radeon X800 VE (R420) JT (AGP), ATI Radeon X850 (R480) (AGP), ATI Radeon X850 XT (R480) (AGP), ATI Radeon X850 SE (R480) (AGP), ATI Radeon X850 PRO (R480) (AGP), ATI Radeon X850 XT PE (R480) (AGP), ATI Radeon Mobility M7 LW (AGP), ATI Mobility FireGL 7800 M7 LX (AGP), ATI Radeon Mobility M6 LY (AGP), ATI Radeon Mobility M6 LZ (AGP), ATI FireGL Mobility 9000 (M9) Ld (AGP), ATI Radeon Mobility 9000 (M9) Lf (AGP), ATI Radeon Mobility 9000 (M9) Lg (AGP), ATI FireMV 2400 PCI, ATI Radeon 9700 Pro ND (AGP), ATI Radeon 9700/9500Pro NE (AGP), ATI Radeon 9600TX NF (AGP), ATI FireGL X1 NG (AGP), ATI Radeon 9800PRO NH (AGP), ATI Radeon 9800 NI (AGP), ATI FireGL X2 NK (AGP), ATI Radeon 9800XT NJ (AGP), ATI Radeon Mobility 9600/9700 (M10/M11) NP (AGP), ATI Radeon Mobility 9600 (M10) NQ (AGP), ATI Radeon Mobility 9600 (M11) NR (AGP), ATI Radeon Mobility 9600 (M10) NS (AGP), ATI FireGL Mobility T2 (M10) NT (AGP), ATI FireGL Mobility T2e (M11) NV (AGP), ATI Radeon QD (AGP), ATI Radeon QE (AGP), ATI Radeon QF (AGP), ATI Radeon QG (AGP), ATI FireGL 8700/8800 QH (AGP), ATI Radeon 8500 QL (AGP), ATI Radeon 9100 QM (AGP), ATI Radeon 7500 QW (AGP/PCI), ATI Radeon 7500 QX (AGP/PCI), ATI Radeon VE/7000 QY (AGP/PCI), ATI Radeon VE/7000 QZ (AGP/PCI), ATI ES1000 515E (PCI), ATI Radeon Mobility X300 (M22) 5460 (PCIE), ATI Radeon Mobility X600 SE (M24C) 5462 (PCIE), ATI FireGL M22 GL 5464 (PCIE), ATI Radeon X800 (R423) UH (PCIE), ATI Radeon X800PRO (R423) UI (PCIE), ATI Radeon X800LE (R423) UJ (PCIE), ATI Radeon X800SE (R423) UK (PCIE), ATI Radeon X800 XTP (R430) (PCIE), ATI Radeon X800 XL (R430) (PCIE), ATI Radeon X800 SE (R430) (PCIE), ATI Radeon X800 (R430) (PCIE), ATI FireGL V7100 (R423) (PCIE), ATI FireGL V5100 (R423) UQ (PCIE), ATI FireGL unknown (R423) UR (PCIE), ATI FireGL unknown (R423) UT (PCIE), ATI Mobility FireGL V5000 (M26) (PCIE), ATI Mobility FireGL V5000 (M26) (PCIE), ATI Mobility Radeon X700 XL (M26) (PCIE), ATI Mobility Radeon X700 (M26) (PCIE), ATI Mobility Radeon X700 (M26) (PCIE), ATI Radeon X550XTX 5657 (PCIE), ATI Radeon 9100 IGP (A5) 5834, ATI Radeon Mobility 9100 IGP (U3) 5835, ATI Radeon XPRESS 200 5954 (PCIE), ATI Radeon XPRESS 200M 5955 (PCIE), ATI Radeon 9250 5960 (AGP), ATI Radeon 9200 5961 (AGP), ATI Radeon 9200 5962 (AGP), ATI Radeon 9200SE 5964 (AGP), ATI FireMV 2200 (PCI), ATI ES1000 5969 (PCI), ATI Radeon XPRESS 200 5974 (PCIE), ATI Radeon XPRESS 200M 5975 (PCIE), ATI Radeon XPRESS 200 5A41 (PCIE), ATI Radeon XPRESS 200M 5A42 (PCIE), ATI Radeon XPRESS 200 5A61 (PCIE), ATI Radeon XPRESS 200M 5A62 (PCIE), ATI Radeon X300 (RV370) 5B60 (PCIE), ATI Radeon X600 (RV370) 5B62 (PCIE), ATI Radeon X550 (RV370) 5B63 (PCIE), ATI FireGL V3100 (RV370) 5B64 (PCIE), ATI FireMV 2200 PCIE (RV370) 5B65 (PCIE), ATI Radeon Mobility 9200 (M9+) 5C61 (AGP), ATI Radeon Mobility 9200 (M9+) 5C63 (AGP), ATI Mobility Radeon X800 XT (M28) (PCIE), ATI Mobility FireGL V5100 (M28) (PCIE), ATI Mobility Radeon X800 (M28) (PCIE), ATI Radeon X850 5D4C (PCIE), ATI Radeon X850 XT PE (R480) (PCIE), ATI Radeon X850 SE (R480) (PCIE), ATI Radeon X850 PRO (R480) (PCIE), ATI unknown Radeon / FireGL (R480) 5D50 (PCIE), ATI Radeon X850 XT (R480) (PCIE), ATI Radeon X800XT (R423) 5D57 (PCIE), ATI FireGL V5000 (RV410) (PCIE), ATI Radeon X700 XT (RV410) (PCIE), ATI Radeon X700 PRO (RV410) (PCIE), ATI Radeon X700 SE (RV410) (PCIE), ATI Radeon X700 (RV410) (PCIE), ATI Radeon X700 SE (RV410) (PCIE), ATI Radeon X1800, ATI Mobility Radeon X1800 XT, ATI Mobility Radeon X1800, ATI Mobility FireGL V7200, ATI FireGL V7200, ATI FireGL V5300, ATI Mobility FireGL V7100, ATI Radeon X1800, ATI Radeon X1800, ATI Radeon X1800, ATI Radeon X1800, ATI Radeon X1800, ATI FireGL V7300, ATI FireGL V7350, ATI Radeon X1600, ATI RV505, ATI Radeon X1300/X1550, ATI Radeon X1550, ATI M54-GL, ATI Mobility Radeon X1400, ATI Radeon X1300/X1550, ATI Radeon X1550 64-bit, ATI Mobility Radeon X1300, ATI Mobility Radeon X1300, ATI Mobility Radeon X1300, ATI Mobility Radeon X1300, ATI Radeon X1300, ATI Radeon X1300, ATI RV505, ATI RV505, ATI FireGL V3300, ATI FireGL V3350, ATI Radeon X1300, ATI Radeon X1550 64-bit, ATI Radeon X1300/X1550, ATI Radeon X1600, ATI Radeon X1300/X1550, ATI Mobility Radeon X1450, ATI Radeon X1300/X1550, ATI Mobility Radeon X2300, ATI Mobility Radeon X2300, ATI Mobility Radeon X1350, ATI Mobility Radeon X1350, ATI Mobility Radeon X1450, ATI Radeon X1300, ATI Radeon X1550, ATI Mobility Radeon X1350, ATI FireMV 2250, ATI Radeon X1550 64-bit, ATI Radeon X1600, ATI Radeon X1650, ATI Radeon X1600, ATI Radeon X1600, ATI Mobility FireGL V5200, ATI Mobility Radeon X1600, ATI Radeon X1650, ATI Radeon X1650, ATI Radeon X1600, ATI Radeon X1300 XT/X1600 Pro, ATI FireGL V3400, ATI Mobility FireGL V5250, ATI Mobility Radeon X1700, ATI Mobility Radeon X1700 XT, ATI FireGL V5200, ATI Mobility Radeon X1700, ATI Radeon X2300HD, ATI Mobility Radeon HD 2300, ATI Mobility Radeon HD 2300, ATI Radeon X1950, ATI Radeon X1900, ATI Radeon X1950, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI Radeon X1900, ATI AMD Stream Processor, ATI Radeon X1900, ATI Radeon X1950, ATI RV560, ATI RV560, ATI Mobility Radeon X1900, ATI RV560, ATI Radeon X1950 GT, ATI RV570, ATI RV570, ATI FireGL V7400, ATI RV560, ATI Radeon X1650, ATI Radeon X1650, ATI RV560, ATI Radeon 9100 PRO IGP 7834, ATI Radeon Mobility 9200 IGP 7835, ATI Radeon X1200, ATI Radeon X1200, ATI Radeon X1200, ATI Radeon X1200, ATI Radeon X1200, ATI RS740, ATI RS740M, ATI RS740, ATI RS740M, ATI Radeon HD 2900 XT, ATI Radeon HD 2900 XT, ATI Radeon HD 2900 XT, ATI Radeon HD 2900 Pro, ATI Radeon HD 2900 GT, ATI FireGL V8650, ATI FireGL V8600, ATI FireGL V7600, ATI Radeon 4800 Series, ATI Radeon HD 4870 x2, ATI Radeon 4800 Series, ATI Radeon HD 4850 x2, ATI FirePro V8750 (FireGL), ATI FirePro V7760 (FireGL), ATI Mobility RADEON HD 4850, ATI Mobility RADEON HD 4850 X2, ATI Radeon 4800 Series, ATI FirePro RV770, AMD FireStream 9270, AMD FireStream 9250, ATI FirePro V8700 (FireGL), ATI Mobility RADEON HD 4870, ATI Mobility RADEON M98, ATI Mobility RADEON HD 4870, ATI Radeon 4800 Series, ATI Radeon 4800 Series, ATI FirePro M7750, ATI M98, ATI M98, ATI M98, ATI Mobility Radeon HD 4650, ATI Radeon RV730 (AGP), ATI Mobility Radeon HD 4670, ATI FirePro M5750, ATI Mobility Radeon HD 4670, ATI Radeon RV730 (AGP), ATI RV730XT [Radeon HD 4670], ATI RADEON E4600, ATI Radeon HD 4600 Series, ATI RV730 PRO [Radeon HD 4650], ATI FirePro V7750 (FireGL), ATI FirePro V5700 (FireGL), ATI FirePro V3750 (FireGL), ATI Mobility Radeon HD 4830, ATI Mobility Radeon HD 4850, ATI FirePro M7740, ATI RV740, ATI Radeon HD 4770, ATI Radeon HD 4700 Series, ATI Radeon HD 4770, ATI FirePro M5750, ATI RV610, ATI Radeon HD 2400 XT, ATI Radeon HD 2400 Pro, ATI Radeon HD 2400 PRO AGP, ATI FireGL V4000, ATI RV610, ATI Radeon HD 2350, ATI Mobility Radeon HD 2400 XT, ATI Mobility Radeon HD 2400, ATI RADEON E2400, ATI RV610, ATI FireMV 2260, ATI RV670, ATI Radeon HD3870, ATI Mobility Radeon HD 3850, ATI Radeon HD3850, ATI Mobility Radeon HD 3850 X2, ATI RV670, ATI Mobility Radeon HD 3870, ATI Mobility Radeon HD 3870 X2, ATI Radeon HD3870 X2, ATI FireGL V7700, ATI Radeon HD3850, ATI Radeon HD3690, AMD Firestream 9170, ATI Radeon HD 4550, ATI Radeon RV710, ATI Radeon RV710, ATI Radeon RV710, ATI Radeon HD 4350, ATI Mobility Radeon 4300 Series, ATI Mobility Radeon 4500 Series, ATI Mobility Radeon 4500 Series, ATI FirePro RG220, ATI Mobility Radeon 4330, ATI RV630, ATI Mobility Radeon HD 2600, ATI Mobility Radeon HD 2600 XT, ATI Radeon HD 2600 XT AGP, ATI Radeon HD 2600 Pro AGP, ATI Radeon HD 2600 XT, ATI Radeon HD 2600 Pro, ATI Gemini RV630, ATI Gemini Mobility Radeon HD 2600 XT, ATI FireGL V5600, ATI FireGL V3600, ATI Radeon HD 2600 LE, ATI Mobility FireGL Graphics Processor, ATI Radeon HD 3470, ATI Mobility Radeon HD 3430, ATI Mobility Radeon HD 3400 Series, ATI Radeon HD 3450, ATI Radeon HD 3450, ATI Radeon HD 3430, ATI Radeon HD 3450, ATI FirePro V3700, ATI FireMV 2450, ATI FireMV 2260, ATI FireMV 2260, ATI Radeon HD 3600 Series, ATI Radeon HD 3650 AGP, ATI Radeon HD 3600 PRO, ATI Radeon HD 3600 XT, ATI Radeon HD 3600 PRO, ATI Mobility Radeon HD 3650, ATI Mobility Radeon HD 3670, ATI Mobility FireGL V5700, ATI Mobility FireGL V5725, ATI Radeon HD 3200 Graphics, ATI Radeon 3100 Graphics, ATI Radeon HD 3200 Graphics, ATI Radeon 3100 Graphics, ATI Radeon HD 3300 Graphics, ATI Radeon HD 3200 Graphics, ATI Radeon 3000 Graphics, SUMO, SUMO, SUMO2, SUMO2, SUMO2, SUMO2, SUMO, SUMO, SUMO, SUMO, SUMO, SUMO, SUMO, SUMO, ATI Radeon HD 4200, ATI Radeon 4100, ATI Mobility Radeon HD 4200, ATI Mobility Radeon 4100, ATI Radeon HD 4290, ATI Radeon HD 4250, AMD Radeon HD 6310 Graphics, AMD Radeon HD 6310 Graphics, AMD Radeon HD 6250 Graphics, AMD Radeon HD 6250 Graphics, AMD Radeon HD 6300 Series Graphics, AMD Radeon HD 6200 Series Graphics, PALM, PALM, PALM, CYPRESS, ATI FirePro (FireGL) Graphics Adapter, ATI FirePro (FireGL) Graphics Adapter, ATI FirePro (FireGL) Graphics Adapter, AMD Firestream 9370, AMD Firestream 9350, ATI Radeon HD 5800 Series, ATI Radeon HD 5800 Series, ATI Radeon HD 5800 Series, ATI Radeon HD 5800 Series, ATI Radeon HD 5900 Series, ATI Radeon HD 5900 Series, ATI Mobility Radeon HD 5800 Series, ATI Mobility Radeon HD 5800 Series, ATI FirePro (FireGL) Graphics Adapter, ATI FirePro (FireGL) Graphics Adapter, ATI Mobility Radeon HD 5800 Series, ATI Radeon HD 5700 Series, ATI Radeon HD 5700 Series, ATI Radeon HD 6700 Series, ATI Radeon HD 5700 Series, ATI Radeon HD 6700 Series, ATI Mobility Radeon HD 5000 Series, ATI Mobility Radeon HD 5000 Series, ATI Mobility Radeon HD 5570, ATI FirePro (FireGL) Graphics Adapter, ATI FirePro (FireGL) Graphics Adapter, ATI Radeon HD 5670, ATI Radeon HD 5570, ATI Radeon HD 5500 Series, REDWOOD, ATI Mobility Radeon HD 5000 Series, ATI Mobility Radeon HD 5000 Series, ATI Mobility Radeon Graphics, ATI Mobility Radeon Graphics, CEDAR, ATI FirePro (FireGL) Graphics Adapter, ATI FirePro (FireGL) Graphics Adapter, ATI FirePro 2270, CEDAR, ATI Radeon HD 5450, CEDAR, CEDAR, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, CAYMAN, AMD Radeon HD 6900 Series, AMD Radeon HD 6900 Series, CAYMAN, CAYMAN, CAYMAN, AMD Radeon HD 6900M Series, Mobility Radeon HD 6000 Series, BARTS, BARTS, Mobility Radeon HD 6000 Series, Mobility Radeon HD 6000 Series, BARTS, BARTS, BARTS, BARTS, AMD Radeon HD 6800 Series, AMD Radeon HD 6800 Series, AMD Radeon HD 6700 Series, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, TURKS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, CAICOS, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, ARUBA, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, TAHITI, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, PITCAIRN, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, VERDE, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, OLAND, HAINAN, HAINAN, HAINAN, HAINAN, HAINAN, HAINAN, BONAIRE, BONAIRE, BONAIRE, BONAIRE, BONAIRE, BONAIRE, BONAIRE, BONAIRE, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI, KABINI [ 99.620] (--) using VT number 2 [ 99.631] (II) [KMS] Kernel modesetting enabled. [ 99.631] (EE) No devices detected. [ 99.631] (EE) Fatal server error: [ 99.631] (EE) no screens found(EE) [ 99.631] (EE) Please consult the The X.Org Foundation support at http://wiki.x.org for help. [ 99.631] (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information. [ 99.631] (EE) 

It looks like it can't identify the card, despite detecting it and loading the right drivers:

[ 99.567] (--) PCI:*(0:1:0:0) 1002:68f9:174b:e233 rev 0, Mem @ 0xe0000000/268435456, 0xfea00000/131072, I/O @ 0x0000c000/256, BIOS @ 0x????????/131072
[ 99.617] (II) RADEON: Driver for ATI Radeon chipsets: [...]

Really, I don't get it.


EDIT:

Oh, that wasn't hard, I just needed to create a small xorg config file with this to use the right bus:

Section "Device" Identifier "Card0" Driver "radeon" BusID "PCI:1:0:0" EndSection

The mouse is still unusable though. I think I'll connect some old mouse for the host and I'll passthrough the current one hotplugging it on occurrence on various guests.

Last edited by Evonat (2013-08-12 15:10:36)

Offline

#425 2013-08-14 23:25:50

irtimmer
Member
Registered: 2013-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have try this on my new computer (Nvdia GTX 660 and Intel Hasswell 4600). I have patched seabios, qemu and kernel mentioned in the opening post. It works while using the ArchLinux cd but I can't get it work under Seabios and Windows. It looks like there is still something missing in Seabios to properly initialize the GPU. Using ArchLinux to visualize Linux isn't very useful for me so I hope someone knows how to fix this. I could of course provide some more debug information if someone can tell me what is useful.

While running qemu with -vga none it is by default not possible to grab the keyboard and mouse in the Qemu window. After searching for an option to enable this while not using a emulated video card driver I have found somekind of a hack. If you add '-device qxl' to the Qemu command you would get a second video card which is showed in the Qemu window. In my testing it looks it won't cause conflicts (probably because it doesn't have a video bios).


I can read and write in English, but my english grammar is just horrible.

Offline

#426 2013-08-15 05:37:27

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

irtimmer wrote:

I have try this on my new computer (Nvdia GTX 660 and Intel Hasswell 4600). I have patched seabios, qemu and kernel mentioned in the opening post. It works while using the ArchLinux cd but I can't get it work under Seabios and Windows. It looks like there is still something missing in Seabios to properly initialize the GPU. Using ArchLinux to visualize Linux isn't very useful for me so I hope someone knows how to fix this. I could of course provide some more debug information if someone can tell me what is useful.

While running qemu with -vga none it is by default not possible to grab the keyboard and mouse in the Qemu window. After searching for an option to enable this while not using a emulated video card driver I have found somekind of a hack. If you add '-device qxl' to the Qemu command you would get a second video card which is showed in the Qemu window. In my testing it looks it won't cause conflicts (probably because it doesn't have a video bios).

I have a very similar problem: I manage to install windows using a cirrus/qxl/whatever card, and using pci-assign instead of vfio-pci my card works and I can even disable the "secondary display" from Windows so that only my display is on and the QEMU windows stays black.
But if I disable whatever graphic card I selected with -vga, Windows doesn't boot. The display stays black and nothing happens. If I try to gracefully power down QEMU with system_powerdown after a few seconds it turns off, so it's not like it's freezed or something.
I'd really like to know what is happening, so I even tried to add a secondary qxl adapter with:

-device qxl,bus=root.1,addr=01.0 \ -spice port=54321,disable-ticketing

But I can't see anything from remote-viewer and I noticed that Windows doesn't even have the drivers for it (it says it's an "unknown adapter" from the device manager), so at this point I'm at a loss.
I could be happy and start using it with a -vga adapter enabled but disabled from Windows, but it still bothers me.

Offline

#427 2013-08-15 11:00:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you guys tried blacklisting the intel driver on the host some people reported problems using it

Offline

#428 2013-08-15 12:02:12

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I haven't even compiled them in the kernel, let alone modprobed them!

Offline

#429 2013-08-15 16:43:24

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh, also, here's a funny thing: I can boot the guest arch with -vga none only when I'm using the unpatched seabios. I patch it, and the display doesn't turn on.
I guess there must be a problem with the identification of the primary VGA card in the seabios or qemu. I hope I can figure out how the code works and see if I can write me a patch.

Offline

#430 2013-08-16 15:17:59

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:

IT WORKS!

I’ve made a few minor tweaks to my system since the time of my previous posts.   For those interested, here’s my current setup, which seems to be running Windows 7 Ultimate (64-bit) guests nearly flawlessly:

MB: ASUS M5A99FX PRO R2.0 w/ BIOS v2005 CPU: AMD Phenom II X4 955 RAM: 20 GB DDR3 1600 Host GPU: Radeon HD6670 w/ Catalyst 13.4 driver Guest GPU: Nvidia GTX550Ti w/ Nvidia 331.82 driver Ubuntu 13.10 (fully updated) Kernel 3.12.0 (from Ubuntu kernel-ppa/mainline, no patches) or 3.11.0-14 (from Ubuntu repository, no patches) Qemu 1.7.0 (from ppa:jacob/virtualisation, no patches) or 1.5.0 (from Ubuntu repository, no patches) Seabios 1.7.3 (from Ubuntu repository, no patches)

Using the Nvidia 331.82 driver, I have HDMI audio and a Windows experience score of 7.3 for both desktop graphics and gaming graphics.  My Windows experience scores for processor and memory are 7.3 and 7.5, respectively.  My Windows experience score for disk access is currently an unimpressive 6.0, but I'm sure that score could be improved through many means.  Note that, although kernel 3.11.0-14 (from the Ubuntu repository, no patches) paired with Qemu 1.5.0 (from the Ubuntu repository, no patches) works well with my hardware, mainline kernel 3.12.0 (from Ubuntu kernel-ppa/mainline, no patches) paired with Qemu 1.7.0 (from ppa:jacob/virtualisation, no patches), likely works with a broader range of hardware since the latter combination supports bus reset.  Between these two options, I recommend using mainline kernel 3.12 and Qemu 1.7.0.

Here are the start up commands that I'm using:

sudo vfio-bind 0000:03:00.0 0000:03:00.1 0000:00:16.0 0000:00:16.2 && \ sudo qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 4096 \ -cpu host \ -daemonize \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \ -device vfio-pci,host=03:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/storage/images/Win7HVM.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:16.0,bus=pcie.0 \ -device vfio-pci,host=00:16.2,bus=pcie.0 \

Using BIOS v2005, IOMMU seems to work nearly flawlessly on my ASUS M5A99FX PRO R2.0 motherboard and "dmesg | grep AMD-Vi" gives me the following output:

$ dmesg | grep AMD-Vi [ 1.149303] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.149305] AMD-Vi: Interrupt remapping enabled [ 1.159117] AMD-Vi: Lazy IO/TLB flushing enabled

Since I'm using a stock Ubuntu kernel, which is missing a few components needed for vfio, I added the following to /etc/modules:

pci_stub vfio vfio_iommu_type1 vfio_pci # vfio_pci_vga (For now, this component is in the stock Ubuntu kernel.) kvm kvm_amd

Note that if using an Intel CPU, you would want to replace "kvm_amd" with "kvm_intel" in the /etc/modules file described above.

After adding the above to /etc/modules, although probably not necessary, I updated initramfs using the following:

sudo update-initramfs -u

So that pci-stub will claim my Nvidia GTX550Ti GPU at boot, I edited my Grub cmdline found in /etc/default/grub to read as follows:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci-stub.ids=10de:1244,10de:0bee"

Note that I used lspci to determine the value "10de:1244,10de:0bee" that corresponds to my Nvidia GTX550Ti GPU.

After editing the Grub cmdline, I updated Grub using the following:

sudo update-grub

After booting with the Grub cmdline edited as above, "dmesg | grep pci-stub" gives me the following output:

$ dmesg | grep pci-stub [ 6.430514] pci-stub: add 10DE:1244 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 6.430538] pci-stub 0000:03:00.0: claimed by stub [ 6.430554] pci-stub: add 10DE:0BEE sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 6.430566] pci-stub 0000:03:00.1: claimed by stub

For the sake of describing with completeness the steps that I took, I'll add that, following nbhs's tutorial, I created /usr/bin/vfio-bind, and added to it the following:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

I changed the file's permissions to 755:

chmod 755 /usr/bin/vfio-bind

Also following nbhs's tutorial, I created /etc/vfio-pci.cfg and added to it the following:

DEVICES="0000:03:00.0 0000:03:00.1 0000:00:16.0 0000:00:16.2"

I used lspci to determine which pci addresses to add to the above file.  The "0000:03:00.0" and "0000:03:00.1" addresses correspond to my Nvidia GTX550Ti  GPU.  The "0000:00:16.0" and "0000:00:16.2" addresses correspond to an otherwise unused USB 2.0 controller, which allows me to pass through a separate keyboard and mouse.   Of course, other addresses can be added in addition to, or in place of, the above mentioned addresses that I'm using.  But note that, in my case, adding and binding any pci addresses corresponding to any USB 3.0 controller prevents the guest from booting, and at least in a few instances, corrupted my guests, rendering them unbootable.  So use caution if attempting to pass through a USB 3.0 controller.

Finally, and also following nbhs's tutorial, I created /etc/systemd/system/binds-vfio-pci.service and added to it the following:

[Unit] Description=Binds devices to vfio-pci After=syslog.target [Service] EnvironmentFile=-/etc/vfio-pci.cfg Type=oneshot RemainAfterExit=yes ExecStart=-/usr/bin/vfio-bind $DEVICES [Install] WantedBy=multi-user.target

As I said, the above setup seems to be running Windows 7 Ultimate (64-bit) guests nearly flawlessly.

I’ve also managed to passthrough my Nvidia GPU to an Ubuntu 13.10 guest.  The Ubuntu VM starts up and runs reasonably well with GPU passthrough.  However, unlike with the Windows VM, which consistently shuts down without incident for me, the Ubuntu VM often hangs on exit, sometimes requiring reboot of the host to clear the GPU.  Or at least that was the case with Qemu 1.5.  I’ll be testing Ubuntu VMs with Qemu 1.7 soon.

EDITS:  Updated to reflect that BIOS v2005 cures previously reported IOMMU errors and other small tweaks, including the use of mainline kernel 3.12 and Qemu 1.7.0 from ppa:jacob/virtualisation.

Last edited by GizmoChicken (2013-12-08 05:22:25)

Offline

#431 2013-08-17 14:29:20

mikeserv
Member
Registered: 2013-08-17
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My config:

Gigabyte 970A-D3 v3.0 (latest FD firmware)
AMD Phenom II X6 1045T 2.7ghz
2x8GB DDR3 1600; 2X4GB DDR3 1600
2GB Nvidia GT640 (flashed with GOP compatible rom emailed me from eVGA upon request last year)
120GB Kingston SSD + 2X3TB WD Green; bcache + btrfs RAID0 data, RAID1 metadata array

I'd really like to run the linux hypervisor headless and passthrough the 1 video card to a Windows installation. I've seen the 2 gpu requirement a few times in this thread and am curious about it. Why is it necessary if you don't really care to view the host? Or, if x is a requirement, can I configure the host to use framebuffer only like with xvfb or xvnc?

-Mike

Offline

#432 2013-08-17 19:31:01

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mikeserv wrote:

I'd really like to run the linux hypervisor headless and passthrough the 1 video card to a Windows installation. I've seen the 2 gpu requirement a few times in this thread and am curious about it. Why is it necessary if you don't really care to view the host? Or, if x is a requirement, can I configure the host to use framebuffer only like with xvfb or xvnc?

As to whether the VFIO component of KVM will work in a headless mode, I honestly don't know.  So I'll leave that question for others to answer.

Although I can't directly answer your question, I can offer one possible alternative configuration:  Have you considered installing a cheap video card in your (probably unused) pci slot for use as a "primary" card on your host?

In my case, I have a cheap "ATI Rage XL Pro" card (about $10 on ebay) installed in an otherwise unused pci slot of a XenServer setup.   I only use the card to access xsconsole, which is a non-graphical interface, and the card works well for that purpose.   To make this work, I set my bios to boot pci (rather than pcie) first.

If your bios allows for selecting pci to boot first, which I think it does, then although not truly headless, you could probably use an "ATI Rage XL Pro"pci graphics card (or possibly something a bit more potent, depending on your needs) as the primary card for your KVM w/VFIO host.  Such a card would not only be cheap, but would also draw very little power.  (Given your goals, you also might want to consider open source XenServer.  But that's another topic.)

Last edited by GizmoChicken (2013-08-17 19:44:47)

Offline

#433 2013-08-17 20:02:31

mikeserv
Member
Registered: 2013-08-17
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have had very little luck with Xen. Of course I last tried all of this Feb. At the time I couldn't find a single verifiable report of anyone successfully passing through a Fermi+ Nvidia to a Windows Vista+ guest. If I remember correctly it was due to Seabios memory space limitations and large Nvidia Roms. I requested the GOP ROM specifically in the hopes that qemu's UEFI firmware at the time could get past that but I got fed up and quit before I tried it.

Anyway, I kicked myself then (again!) for the GPU impulse buy (dammit Fry's) and figured I could just wait. I did some cool stuff with multiple lxc guests, vts and XDMCP, which I mean to do again as well... And I checked Google again today just to look for any possible improvements and found this!

The way I figure it I should be able to both. I was thinking that while the Windows guest ran I wouldn't have any need of the card because the tv is the machine's only local display and could power it down, rmmod nvidia, and modprobe pci-stub it, then when wanting to switch back I could reverse that. Too far-fetched?

Still, I probably should grab a cheap second GPU, I guess. Or maybe a PSU (gotta cheap diablotek 450 that came with the barebones kit now) and a nicer AMD... For Linux the 640 is more than adequate but it's a dog gaming board... I guess I'd probably need to verify I could keep it cool, too...

So thanks, Gizmo, I'll definitely consider your suggestion but would still love to hear from anyone else that may know of a workable single GPU config...

Offline

#434 2013-08-18 16:45:10

g.art
Member
Registered: 2013-08-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys!

Currently my setup is Host GPU - GTS250 (nouveau), Guest GPU which running Windows 7 Ultimate 64-bit - Radeon 6930. Everything run smooth with VFIO and your guide (thx to nbhs btw). But right now i got Radeon 5850 and i wanna use it as my host GPU, and i'm wondering is it possible to not blacklist the radeon(OSS) module in host? Cause i wanna use it simultanesly with my guest running Windows, like in my current config.

Offline

#435 2013-08-18 20:59:15

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

g.art wrote:

Hi guys!

Currently my setup is Host GPU - GTS250 (nouveau), Guest GPU which running Windows 7 Ultimate 64-bit - Radeon 6930. Everything run smooth with VFIO and your guide (thx to nbhs btw). But right now i got Radeon 5850 and i wanna use it as my host GPU, and i'm wondering is it possible to not blacklist the radeon(OSS) module in host? Cause i wanna use it simultanesly with my guest running Windows, like in my current config.

You could use pci-stub to "hide" the 6930 from the host's radeon driver. It's also explained in the guide how to do that.

Offline

#436 2013-08-21 20:09:52

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).

Offline

#437 2013-08-21 22:27:30

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).

I can't say if you're the only one, but those packages work fine for me:

seabios 1.7.2.2-1 qemu-vga-current 1.6.0.g7b4b0e9-1

i'm sorry for my poor english wirting skills…

Offline

#438 2013-08-28 03:25:37

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.

What could be wrong?

Offline

#439 2013-08-29 11:27:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

Hi,

I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.

What could be wrong?

It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.

Last edited by nbhs (2013-08-29 11:27:39)

Offline

#440 2013-08-29 11:54:47

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Chetyre wrote:

Hi,

I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.

What could be wrong?

It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.


Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.

I'm not really sure where to begin with this.

Offline

#441 2013-08-29 12:15:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:
nbhs wrote:
Chetyre wrote:

Hi,

I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.

What could be wrong?

It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.


Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.

I'm not really sure where to begin with this.

pci-back i a xen module, you shouldnt bind anything to it

Offline

#442 2013-08-29 12:17:46

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Chetyre wrote:
nbhs wrote:

It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.


Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.

I'm not really sure where to begin with this.

pci-back i a xen module, you shouldnt bind anything to it


sorry, I'm still using xen so I got confused. I meant pci-stub.

Offline

#443 2013-08-29 13:15:51

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:
nbhs wrote:
Chetyre wrote:

Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.

I'm not really sure where to begin with this.

pci-back i a xen module, you shouldnt bind anything to it


sorry, I'm still using xen so I got confused. I meant pci-stub.

Have you tried booting without binding the controller to pci-stub?

Offline

#444 2013-08-29 14:49:02

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got different error messages, but if it is similar to my case you could try detaching all drives from one controller and booting and if you still get errors try the other controller.
My "solution" to the problem is: not using the second sata controller. If it turns out you really have the exact same problem, then you are lucky, because there is a patch for it for intel hardware (not for amd, which I use).


i'm sorry for my poor english wirting skills…

Offline

#445 2013-08-29 23:16:25

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, I tried both your suggestions, but still the same thing.

I managed to boot with intel_iommu=on ONCE, but only after I removed everything from pci-stub.ids. but as I said it only worked once, and then I just got a black screen with nothing in it. but when I do that I can change to tty1, but it just stays there doing nothing

Offline

#446 2013-08-31 20:52:05

Lekensteyn
Member
From: Netherlands
Registered: 2012-06-19
Posts: 179
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am new to this IOMMU thing, just installed an i7 3770 on a GA-Z68X-UD3H-B3 (rev 1.3), BIOS U1l. Using -device pci-assign,host=xx:yy.z successfully passes a USB controller and NIC (when ran as root).

After looking around, it seems that "VFIO" must be used for passing a VGA card, but I cannot simply unbind all devices under an IOMMU group, that will render me without networking and disk! Can you (desktop owners, not laptop) post the output of:

(for i in /sys/kernel/iommu_groups/*;do echo "IOMMU: $i";for dev in $i/devices/*;do lspci -s "${dev##*/}"; done; echo;done)

Output for my machine:

IOMMU: /sys/kernel/iommu_groups/0 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) IOMMU: /sys/kernel/iommu_groups/1 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) IOMMU: /sys/kernel/iommu_groups/2 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) IOMMU: /sys/kernel/iommu_groups/3 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05) IOMMU: /sys/kernel/iommu_groups/4 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5) 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5) 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5) 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5) 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5) 02:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) IOMMU: /sys/kernel/iommu_groups/5 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) IOMMU: /sys/kernel/iommu_groups/6 00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05) 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)

There is no discrete video card attached, I first need to get these groups more sane.

Edit:
I just noticed that the devices are either located under bus 00, device 1c or that they are the 00:1c.x device. From http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM: VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.

Am I out of luck then with this motherboard?

Last edited by Lekensteyn (2013-08-31 21:30:09)

Offline

#447 2013-09-01 03:58:30

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
GizmoChicken wrote:
nbhs wrote:

Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue

Yep, I saw your earlier post:

nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

My motherboard's IVRS  errors seem to be slightly different from your motherboard's IVRS errors.  My board's errors include:  1)  "ivrs_ioapic[5] not in IVRS table";  2)  "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found"  I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.

How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

In capitulation to my repeated nagging and pestering, ASUS finally sent me a patched BIOS (M5A99FX-PRO-R20-ASUS-9903.CAP) that attempts to address the IVRS  errors that I was having with my ASUS M5A99FX PRO R2.0 motherboard. 

Much to my surprise and satisfaction, the patched BIOS resolves ALL the IVRS  errors that I had with my motherboard, eliminating the need for the kernel IVRS override.  Yippee!

The support rep couldn't tell me when a version of the BIOS that includes the patch will be available for download from ASUS.  Hopefully soon.

Last edited by GizmoChicken (2013-09-01 08:44:36)

Offline

#448 2013-09-01 10:59:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lekensteyn wrote:

I am new to this IOMMU thing, just installed an i7 3770 on a GA-Z68X-UD3H-B3 (rev 1.3), BIOS U1l. Using -device pci-assign,host=xx:yy.z successfully passes a USB controller and NIC (when ran as root).

After looking around, it seems that "VFIO" must be used for passing a VGA card, but I cannot simply unbind all devices under an IOMMU group, that will render me without networking and disk! Can you (desktop owners, not laptop) post the output of:

(for i in /sys/kernel/iommu_groups/*;do echo "IOMMU: $i";for dev in $i/devices/*;do lspci -s "${dev##*/}"; done; echo;done)

Output for my machine:

IOMMU: /sys/kernel/iommu_groups/0 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) IOMMU: /sys/kernel/iommu_groups/1 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) IOMMU: /sys/kernel/iommu_groups/2 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) IOMMU: /sys/kernel/iommu_groups/3 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05) IOMMU: /sys/kernel/iommu_groups/4 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5) 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5) 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5) 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5) 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5) 02:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) IOMMU: /sys/kernel/iommu_groups/5 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) IOMMU: /sys/kernel/iommu_groups/6 00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05) 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)

There is no discrete video card attached, I first need to get these groups more sane.

Edit:
I just noticed that the devices are either located under bus 00, device 1c or that they are the 00:1c.x device. From http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM: VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.

Am I out of luck then with this motherboard?

See: https://bbs.archlinux.org/viewtopic.php … 1#p1307091

Offline

#449 2013-09-01 11:00:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:
GizmoChicken wrote:

Yep, I saw your earlier post:



My motherboard's IVRS  errors seem to be slightly different from your motherboard's IVRS errors.  My board's errors include:  1)  "ivrs_ioapic[5] not in IVRS table";  2)  "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found"  I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.

How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

In capitulation to my repeated nagging and pestering, ASUS finally sent me a patched BIOS (M5A99FX-PRO-R20-ASUS-9903.CAP) that attempts to address the IVRS  errors that I was having with my ASUS M5A99FX PRO R2.0 motherboard. 

Much to my surprise and satisfaction, the patched BIOS resolves ALL the IVRS  errors that I had with my motherboard, eliminating the need for the kernel IVRS override.  Yippee!

The support rep couldn't tell me when a version of the BIOS that includes the patch will be available for download from ASUS.  Hopefully soon.

That is good news! i wonder if they'll release an updated bios for the rest of their boards with this issue

Last edited by nbhs (2013-09-01 11:00:58)

Offline

#450 2013-09-01 14:09:08

Lekensteyn
Member
From: Netherlands
Registered: 2012-06-19
Posts: 179
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Lekensteyn wrote:

[..]
I just noticed that the devices are either located under bus 00, device 1c or that they are the 00:1c.x device. From http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM: VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.

Am I out of luck then with this motherboard?

See: https://bbs.archlinux.org/viewtopic.php … 1#p1307091

Thank you, I encountered these patches earlier today. Perhaps someone could edit the first post to make a note of this patch?

Note: this may break the safety provided by IOMMU, devices can bypass the IOMMU and reach other devices in the same IOMMU domain. Though when compared with pci-assign, it is not that bad. (pci-assign requires you running as root to access the full PCI config space (which requires CAP_SYS_ADMIN)). See also http://www.linux-kvm.org/wiki/images/b/ … m-VFIO.pdf, page 29.

Last edited by Lekensteyn (2013-09-01 14:36:42)

Offline

#451 2013-09-02 19:39:51

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I figured out what is wrong, but not how to solve it.

Turns out I really shouldn`t have enabled the systemd service to vfio-bind... That was really stupid, and yet took me an eternity to notice.

anyway what happens is in my interpretation (I read very little on the subject): there is too much stuff in my iommu group for some reason. When the script runs it binds too many devices and makes the computer unusable. Is this really a thing that happens? I tried enabling only my ati card, but 4 devices were grabbed by vfio.

On the test script this produced a qemu window, but no output on the card.

When I try binding the rest of what I need, my system becomes pretty much unusable. Out of the 24 devices in my system it grabs 15 of them.

Last edited by Chetyre (2013-09-02 19:54:14)

Offline

#452 2013-09-03 12:00:28

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

I think I figured out what is wrong, but not how to solve it.

Turns out I really shouldn`t have enabled the systemd service to vfio-bind... That was really stupid, and yet took me an eternity to notice.

anyway what happens is in my interpretation (I read very little on the subject): there is too much stuff in my iommu group for some reason. When the script runs it binds too many devices and makes the computer unusable. Is this really a thing that happens? I tried enabling only my ati card, but 4 devices were grabbed by vfio.

On the test script this produced a qemu window, but no output on the card.

When I try binding the rest of what I need, my system becomes pretty much unusable. Out of the 24 devices in my system it grabs 15 of them.

see: https://bbs.archlinux.org/viewtopic.php … 7#p1319587

Offline

#453 2013-09-03 12:43:04

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Chetyre wrote:

I think I figured out what is wrong, but not how to solve it.

Turns out I really shouldn`t have enabled the systemd service to vfio-bind... That was really stupid, and yet took me an eternity to notice.

anyway what happens is in my interpretation (I read very little on the subject): there is too much stuff in my iommu group for some reason. When the script runs it binds too many devices and makes the computer unusable. Is this really a thing that happens? I tried enabling only my ati card, but 4 devices were grabbed by vfio.

On the test script this produced a qemu window, but no output on the card.

When I try binding the rest of what I need, my system becomes pretty much unusable. Out of the 24 devices in my system it grabs 15 of them.

see: https://bbs.archlinux.org/viewtopic.php … 7#p1319587


Thanks. I'll try this tonight. Maybe the first post could be edited to draw attention to the acs patch? I just realized that it was included in your patched kernel while trying to patch it myself.

Offline

#454 2013-09-04 09:04:26

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

all steps before are succeed, but when i try to run vfio-bind it says

# vfio-bind 0000:01:00.0 ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

In all /sys/bus/pci/devices/XXXX:XX:XX.X/ directory "iommu_group" missed.

My distro is Ubuntu 13.10

Where I'm wrong?

Thanks in advance.

Offline

#455 2013-09-04 11:25:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gosheto wrote:

Hello,

all steps before are succeed, but when i try to run vfio-bind it says

# vfio-bind 0000:01:00.0 ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

In all /sys/bus/pci/devices/XXXX:XX:XX.X/ directory "iommu_group" missed.

My distro is Ubuntu 13.10

Where I'm wrong?

Thanks in advance.

Check if you cpu supports VT-D/AMD-VI and its enabled

Offline

#456 2013-09-04 11:47:39

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello nbhs,

Thank you for quick reply!
My cpu is Intel Q9550 and support VT-d.
Mainboard is Asus Rampage Extreme with X48 chipset, who also support VT-d according Intel Chipsets
In my current BIOS VT-d enable/disable setting missing.
After activating Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line,
dmesg | grep -e DMAR -e IOMMU
return
[ 0.000000] Intel-IOMMU: enabled

Any ideas why "iommu_group" directories missed?

Thanks!

Offline

#457 2013-09-04 23:23:42

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

acs_override worked, but I encountered another problem: blank display that does nothing. If I use vga cirrus or std, windows boots fine and installs drivers for everything correctly, including the video card. But device manager says it can't free enough resources.

However, if I boot with vga none, it simply gets stuck with a blank qemu window in my monitor forever. No hdd activity or anything.

On xen, if I have a standard vga adaptor enabled and passthrough a video card, once windows boots it disables the qemu vga device and lets the gpu take over. Is there anyway to force that behavior? Disabling it on device manager does nothing, it resets during the reboot.

Offline

#458 2013-09-04 23:41:14

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

acs_override worked, but I encountered another problem: blank display that does nothing. If I use vga cirrus or std, windows boots fine and installs drivers for everything correctly, including the video card. But device manager says it can't free enough resources.

However, if I boot with vga none, it simply gets stuck with a blank qemu window in my monitor forever. No hdd activity or anything.

On xen, if I have a standard vga adaptor enabled and passthrough a video card, once windows boots it disables the qemu vga device and lets the gpu take over. Is there anyway to force that behavior? Disabling it on device manager does nothing, it resets during the reboot.

You aren't using a nvidia card with the proprietary driver on the host, are you?


i'm sorry for my poor english wirting skills…

Offline

#459 2013-09-04 23:44:12

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
Chetyre wrote:

acs_override worked, but I encountered another problem: blank display that does nothing. If I use vga cirrus or std, windows boots fine and installs drivers for everything correctly, including the video card. But device manager says it can't free enough resources.

However, if I boot with vga none, it simply gets stuck with a blank qemu window in my monitor forever. No hdd activity or anything.

On xen, if I have a standard vga adaptor enabled and passthrough a video card, once windows boots it disables the qemu vga device and lets the gpu take over. Is there anyway to force that behavior? Disabling it on device manager does nothing, it resets during the reboot.

You aren't using a nvidia card with the proprietary driver on the host, are you?


No, I'm using the intel igp and trying to passthrough a radeon card

Offline

#460 2013-09-05 21:07:05

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

vfio_iommu_type1 won`t  load
modprobe vfio_iommu_type1
return
ERROR: could not insert 'vfio_iommu_type1': No such device
Maybe MB (Rampage Extreme X48) not supported VT-d? sad
X48 is capable but Asus BIOS not have option to enable him. sad

Offline

#461 2013-09-09 17:27:46

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, so I just tried to get this working with linux 3.11 but recieved these error messages upon starting qemu

qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to set iommu for container: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to setup container for group 18 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to get group 18 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device 'vfio-pci' could not be initialized

I simply applied the same patch to the kernel (I was kind of suprised that even worked).
Since 3.10 will be a lts kernel, should we simply stick with it?


i'm sorry for my poor english wirting skills…

Offline

#462 2013-09-10 03:26:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Hi, so I just tried to get this working with linux 3.11 but recieved these error messages upon starting qemu

qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to set iommu for container: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to setup container for group 18 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to get group 18 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device 'vfio-pci' could not be initialized

I simply applied the same patch to the kernel (I was kind of suprised that even worked).
Since 3.10 will be a lts kernel, should we simply stick with it?

Are you running it as root? im using kernel 3.11 + qemu 1.5.2 here without problems

here's my package:

linux-mainline.tar.gz

Hopefully these patches wont be needed anymore for 3.12 and the next qemu release see: http://marc.info/?l=kvm&m=137865971705085&w=2 and http://qemu.11.n7.nabble.com/PATCH-vfio … 23928.html

i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.

Last edited by nbhs (2013-09-10 03:55:19)

Offline

#463 2013-09-11 15:18:57

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Are you running it as root? im using kernel 3.11 + qemu 1.5.2 here without problems

here's my package:

linux-mainline.tar.gz

Hopefully these patches wont be needed anymore for 3.12 and the next qemu release see: http://marc.info/?l=kvm&m=137865971705085&w=2 and http://qemu.11.n7.nabble.com/PATCH-vfio … 23928.html

i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.

I think I was running it as root, so my guess is that I was using the wrong patch or configured the kernel wrong somehow. I based my new config off of yours, used your patches and it works now. Thanks for the package.
It's good to hear that the patches might not be needed in 3.12.

About 1.6, my qemu package ist called "qemu-vga-current 1.6.0.g7b4b0e9-1", I sadly lost the pkgbuild recently, so I have no idea how that was created, but it works for me together with "seabios 1.7.2.2-1".
Thanks again for your help.

Last edited by andy123 (2013-09-11 15:19:20)


i'm sorry for my poor english wirting skills…

Offline

#464 2013-09-13 06:08:59

finesse
Member
Registered: 2013-01-25
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Greetings all.

Thanks for this how-to nbhs!

I've read every post here but can't get over something very simple. Im left believing that my board has buggy IOMMU support, pity since I bought it a few years ago wanting to do this very thing and options were limited.

System

i5-2500 (non-K)
ASRock Z68 Pro 3 Mobo
16gig ram
Intel IGPU on host Arch (with your custom built mainline kernel)
Some no name brand 7970 for the guest which currently might as well not exist

Im using syslinux, here is my kernel line:

root=UUID=069ddd96-02d7-4b5a-9b8c-d71403fb4579 ro quiet init=/usr/lib/systemd/systemd intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream pci-stub.ids=1002:6798,1002:aaa0 initrd=../initramfs-linux-mainline.img BOOT_IMAGE=../vmlinuz-linux-mainline 

Ive pretty much tried everything suggested in here so there's stuff in there that probably doesn't need to be, but that is how it was on my last attempt.

Other outputs:

@Shodan ~ $ dmesg |grep IOMMU [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] Intel-IOMMU: enabled
@Shodan ~ $ dmesg |grep pci-stub [ 0.509170] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.509176] pci-stub 0000:01:00.0: claimed by stub [ 0.509180] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.509183] pci-stub 0000:01:00.1: claimed by stub
@Shodan ~ $ lspci |grep VGA 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]

Vt-d is enabled in the bios, however disabling it and rebooting still produced the dmesg output of it getting enabled so I don't know if it's lying or just referring to the module and not hardware (if thats even possible)

I compiled and installed from your first post with no issue.

The problem herein lies with the iommu_groups not being created, there were 3 other posters here with that same error but nothing they did has fixed it for me yet. I have blacklisted IGP, radeon as well as pci-stub'ing them.

For reference:

@Shodan ~ $ sudo vfio-bind 0000:01:00.0 0000:01:00.1 ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory ls: cannot access /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices: No such file or directory

Your script doesn't lie...

@Shodan ~ $ sudo find /sys/bus |grep iommu_group @Shodan ~ :( $ 

Any thoughts? Im just doing random stuff now, not sure where to go from here cept maybe buy a new motherboard.

Thanks, Joe

Offline

#465 2013-09-15 00:23:51

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1

return

qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Any ideas?

Offline

#466 2013-09-15 13:01:32

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

my primary and secondary VGA is in one IOMMU group, and i cant bind them to vfio.
I found in old posts here (https://bbs.archlinux.org/viewtopic.php … 1#p1307091) that must apply ACS-patch from here - http://www.spinics.net/lists/kvm/msg92163.html
Because i`m new in Linux world, can provide me step by step, how to apply this patch?
I'm using Ubuntu 13.10 - 3.11.0-7-generic

Thank you in advance!

Offline

#467 2013-09-16 04:12:20

GizmoChicken
Member
Registered: 2013-07-25
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
GizmoChicken wrote:
nbhs wrote:

yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:

vfio_iommu_type1.allow_unsafe_interrupts=1

Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

In capitulation to my repeated nagging and pestering, ASUS finally sent me a patched BIOS (M5A99FX-PRO-R20-ASUS-9903.CAP) that attempts to address the IVRS  errors that I was having with my ASUS M5A99FX PRO R2.0 motherboard. 

Much to my surprise and satisfaction, the patched BIOS resolves ALL the IVRS  errors that I had with my motherboard, eliminating the need for the kernel IVRS override.  Yippee!

The support rep couldn't tell me when a version of the BIOS that includes the patch will be available for download from ASUS.  Hopefully soon.

That is good news! i wonder if they'll release an updated bios for the rest of their boards with this issue

I recently learned that, in addition to the fix for my M5A99FX PRO R2.0 motherboard, ASUS issued a similar "test BIOS" for a Sabertooth 990FX R2.0.  I don't know for certain what ASUS has planned for other motherboards, but I'm under the impression ASUS will at least make an effort to add the fix to future BIOS releases... perhaps not the next stable BIOS release, but eventually.

Anyone wanting a "test BIOS" for an ASUS motherboard that uses the AMD 990FX or 970FX chipsets should consider contacting ASUS and providing them with enough information to get to the correct person.  ASUS might issue a custom "test BIOS" for other motherboards upon request, but no promises.

Here's some information that may help to get to the right person at ASUS:
Name of my primary contact at ASUS: lijay_quan
Name of "test BIOS" issued for the ASUS M5A99FX PRO R2:  M5A99FX-PRO-R20-ASUS-9903.CAP
Name of "test BIOS" issued for the ASUS Sabertooth 990FX R2:  SABERTOOTH-990FX-R20-ASUS-9901.CAP

Last edited by GizmoChicken (2013-09-16 04:38:44)

Offline

#468 2013-09-17 11:27:03

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.

So I wasn't the only one :-)

teekay some pages ago wrote:

Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).

Offline

#469 2013-09-17 14:01:30

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fully confirm. It works!

running on
Gigabyte AM3+ 990FXA-UD7/UD3
AMD FX-8150/8350
ATI 5450/Nvidia 210 - as primary.
ATI 7970/7850 - as passthrough.
Windows 7/8 fully works.

Qemu-git
Seabios from qemu


Freezing when installing, was solved. The problem was in image file permissions.
Now I want to try to pass another card in the guest machine, to make a crossfire.

Offline

#470 2013-09-17 15:20:44

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

first of: thanks for this great topic!

I think about setting up a System with VGA-Passthrough, and i have still some questions:

Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?

Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?

Offline

#471 2013-09-17 15:44:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

Hi,

first of: thanks for this great topic!

I think about setting up a System with VGA-Passthrough, and i have still some questions:

Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?

Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?

Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).

Offline

#472 2013-09-17 16:09:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
nbhs wrote:

i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.

So I wasn't the only one :-)

teekay some pages ago wrote:

Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).

Im gonna skip 1.6 and wait for kernel 3.12 and the next qemu release which like i mentioned before shouldnt need any extra patches.

Offline

#473 2013-09-17 16:18:43

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:

Hi,

first of: thanks for this great topic!

I think about setting up a System with VGA-Passthrough, and i have still some questions:

Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?

Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?

Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).

So theres no way of using the mouse for both systems like for example a shared soundcard?

Offline

#474 2013-09-17 16:45:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
nbhs wrote:
SpacePirate wrote:

Hi,

first of: thanks for this great topic!

I think about setting up a System with VGA-Passthrough, and i have still some questions:

Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?

Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?

Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).

So theres no way of using the mouse for both systems like for example a shared soundcard?

http://synergy-foss.org/

Offline

#475 2013-09-17 17:16:22

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:
nbhs wrote:

Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).

So theres no way of using the mouse for both systems like for example a shared soundcard?

http://synergy-foss.org/

That looks really nice! Thanks a lot for the help!

Offline

#476 2013-09-17 17:23:43

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need help.

I use this script to start VM, what I need to change to add another one graphics card to VM?

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" #SND=true #SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="06:06.0" GPU="01:00.0" GPU_SND="01:00.1" VBIOS="/home/cd/rom.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

Last edited by chUpa (2013-09-17 17:28:53)

Offline

#477 2013-09-17 23:01:05

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

While reading the Thread I`ve got another question:

Is it possible to pass the GPU in the first PCI Slot to the Guest System? If not, is this a performance issue since many Mainboards only have PCIe x4 in the second slot...

Offline

#478 2013-09-18 05:30:11

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

While reading the Thread I`ve got another question:

Is it possible to pass the GPU in the first PCI Slot to the Guest System? If not, is this a performance issue since many Mainboards only have PCIe x4 in the second slot...

I do not see any problem in this. In my motherboard, I have 6 pcie ports. My host using card that which is in the lower slot. Its address 0000:05:00.0. Passthrough cards have addresses 0000:01:00.0/0000:02:00.0.

Offline

#479 2013-09-18 12:18:29

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:
SpacePirate wrote:

While reading the Thread I`ve got another question:

Is it possible to pass the GPU in the first PCI Slot to the Guest System? If not, is this a performance issue since many Mainboards only have PCIe x4 in the second slot...

I do not see any problem in this. In my motherboard, I have 6 pcie ports. My host using card that which is in the lower slot. Its address 0000:05:00.0. Passthrough cards have addresses 0000:01:00.0/0000:02:00.0.

Okay, thats nice, i think i just missunderstood some earlier post then.

Offline

#480 2013-09-18 14:24:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:

I need help.

I use this script to start VM, what I need to change to add another one graphics card to VM?

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" #SND=true #SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="06:06.0" GPU="01:00.0" GPU_SND="01:00.1" VBIOS="/home/cd/rom.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

You could add a "GPU2", "GPU_SND2", and "VBIOS2" variable, then add this

if [ -n "$GPU2" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS2" ]; then DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,rombar=0,romfile=$VBIOS2,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND2" ]; then DEV+=" -device vfio-pci,host=$GPU_SND2,addr=0.1,bus=root.2" fi fi

Offline

#481 2013-09-18 14:25:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
chUpa wrote:
SpacePirate wrote:

While reading the Thread I`ve got another question:

Is it possible to pass the GPU in the first PCI Slot to the Guest System? If not, is this a performance issue since many Mainboards only have PCIe x4 in the second slot...

I do not see any problem in this. In my motherboard, I have 6 pcie ports. My host using card that which is in the lower slot. Its address 0000:05:00.0. Passthrough cards have addresses 0000:01:00.0/0000:02:00.0.

Okay, thats nice, i think i just missunderstood some earlier post then.

As long as its not your primary card you could use any slot you want

Offline

#482 2013-09-18 15:21:44

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
chUpa wrote:

I need help.

I use this script to start VM, what I need to change to add another one graphics card to VM?

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" #SND=true #SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="06:06.0" GPU="01:00.0" GPU_SND="01:00.1" VBIOS="/home/cd/rom.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

You could add a "GPU2", "GPU_SND2", and "VBIOS2" variable, then add this

if [ -n "$GPU2" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS2" ]; then DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,rombar=0,romfile=$VBIOS2,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND2" ]; then DEV+=" -device vfio-pci,host=$GPU_SND2,addr=0.1,bus=root.2" fi fi

I tried to do this, here is my edited script:

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" SND=true SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="" GPU0="01:00.0" GPU_SND0="01:00.1" GPU1="02:00.0" GPU_SND1="02:00.1" VBIOS0="/home/cd/rom0.rom" VBIOS1="/home/cd/rom1.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU0" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS0" ]; then DEV+=" -device vfio-pci,host=$GPU0,x-vga=on,rombar=0,romfile=$VBIOS0,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU0,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND0" ]; then DEV+=" -device vfio-pci,host=$GPU_SND0,addr=0.1,bus=root.1" fi fi if [ -n "$GPU1" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS1" ]; then DEV+=" -device vfio-pci,host=$GPU1,x-vga=on,rombar=0,romfile=$VBIOS1,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU1,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND1" ]; then DEV+=" -device vfio-pci,host=$GPU_SND1,addr=0.1,bus=root.2" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

It starts without errors, but there is no output from the graphics card (from both I get - no signal detected).
When I comment the variables:

GPU1="02:00.0" GPU_SND1="02:00.1" VBIOS1="/home/cd/rom1.rom"

works as before

Spasibo for the help, I hope we will find a solution (:

Russia and Uruguay 27 December 2011 signed a visa-free status. I travel much. We can have a beer or coffee. ((:

Last edited by chUpa (2013-09-18 15:33:09)

Offline

#483 2013-09-18 15:42:28

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:
nbhs wrote:
chUpa wrote:

I need help.

I use this script to start VM, what I need to change to add another one graphics card to VM?

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" #SND=true #SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="06:06.0" GPU="01:00.0" GPU_SND="01:00.1" VBIOS="/home/cd/rom.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

You could add a "GPU2", "GPU_SND2", and "VBIOS2" variable, then add this

if [ -n "$GPU2" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS2" ]; then DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,rombar=0,romfile=$VBIOS2,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND2" ]; then DEV+=" -device vfio-pci,host=$GPU_SND2,addr=0.1,bus=root.2" fi fi

I tried to do this, here is my edited script:

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" SND=true SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="" GPU0="01:00.0" GPU_SND0="01:00.1" GPU1="02:00.0" GPU_SND1="02:00.1" VBIOS0="/home/cd/rom0.rom" VBIOS1="/home/cd/rom1.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU0" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS0" ]; then DEV+=" -device vfio-pci,host=$GPU0,x-vga=on,rombar=0,romfile=$VBIOS0,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU0,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND0" ]; then DEV+=" -device vfio-pci,host=$GPU_SND0,addr=0.1,bus=root.1" fi fi if [ -n "$GPU1" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS1" ]; then DEV+=" -device vfio-pci,host=$GPU1,x-vga=on,rombar=0,romfile=$VBIOS1,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU1,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND1" ]; then DEV+=" -device vfio-pci,host=$GPU_SND1,addr=0.1,bus=root.2" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

It starts without errors, but there is no output from the graphics card (from both I get - no signal detected).
When I comment the variables:

GPU1="02:00.0" GPU_SND1="02:00.1" VBIOS1="/home/cd/rom1.rom"

works as before

Spasibo for the help, I hope we will find a solution (:

Are you using radeon cards?

chUpa wrote:

Russia and Uruguay 27 December 2011 signed a visa-free status. I travel much. We can have a beer or coffee. ((:

Sure man anytime smile

Offline

#484 2013-09-18 15:55:09

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
chUpa wrote:
nbhs wrote:

You could add a "GPU2", "GPU_SND2", and "VBIOS2" variable, then add this

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" #SND=true #SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="06:06.0" GPU="01:00.0" GPU_SND="01:00.1" VBIOS="/home/cd/rom.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

I tried to do this, here is my edited script:

if [ -n "$GPU2" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS2" ]; then DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,rombar=0,romfile=$VBIOS2,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND2" ]; then DEV+=" -device vfio-pci,host=$GPU_SND2,addr=0.1,bus=root.2" fi fi

It starts without errors, but there is no output from the graphics card (from both I get - no signal detected).
When I comment the variables:

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" SND=true SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="" GPU0="01:00.0" GPU_SND0="01:00.1" GPU1="02:00.0" GPU_SND1="02:00.1" VBIOS0="/home/cd/rom0.rom" VBIOS1="/home/cd/rom1.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU0" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS0" ]; then DEV+=" -device vfio-pci,host=$GPU0,x-vga=on,rombar=0,romfile=$VBIOS0,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU0,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND0" ]; then DEV+=" -device vfio-pci,host=$GPU_SND0,addr=0.1,bus=root.1" fi fi if [ -n "$GPU1" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS1" ]; then DEV+=" -device vfio-pci,host=$GPU1,x-vga=on,rombar=0,romfile=$VBIOS1,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU1,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND1" ]; then DEV+=" -device vfio-pci,host=$GPU_SND1,addr=0.1,bus=root.2" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

works as before

Spasibo for the help, I hope we will find a solution (:

Are you using radeon cards?

chUpa wrote:

Russia and Uruguay 27 December 2011 signed a visa-free status. I travel much. We can have a beer or coffee. ((:

ATI Radeon (gigabyte HD 7970 gv-r7970c-3gd)

Sure man anytime smile

You have the summer there soon ... (:

Last edited by chUpa (2013-09-18 15:55:52)

Offline

#485 2013-09-18 16:00:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:
nbhs wrote:
chUpa wrote:

I tried to do this, here is my edited script:

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm -nographic" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" SND="" #SND=true #SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="tablet host:058f:6254 host:046d:c526 host:045e:0745" PCI_DEVICES="06:06.0" GPU="01:00.0" GPU_SND="01:00.1" VBIOS="/home/cd/rom.rom" HDD="/home/cd/vhdd/win8x64.img" CDROM="" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 set -m $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu echo "All Done!"

It starts without errors, but there is no output from the graphics card (from both I get - no signal detected).
When I comment the variables:

if [ -n "$GPU2" ]; then DEV+=" -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2" if [ -n "$VBIOS2" ]; then DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,rombar=0,romfile=$VBIOS2,addr=0.0,multifunction=on,bus=root.2" else DEV+=" -device vfio-pci,host=$GPU2,x-vga=on,addr=0.0,multifunction=on,bus=root.2" fi if [ -n "$GPU_SND2" ]; then DEV+=" -device vfio-pci,host=$GPU_SND2,addr=0.1,bus=root.2" fi fi

works as before

Spasibo for the help, I hope we will find a solution (:

Are you using radeon cards?

chUpa wrote:

Russia and Uruguay 27 December 2011 signed a visa-free status. I travel much. We can have a beer or coffee. ((:

ATI Radeon (gigabyte HD 7970 gv-r7970c-3gd)

Sure man anytime smile

You have the summer there soon ... (:

Try without "x-vga=on" on GPU1

Last edited by nbhs (2013-09-18 16:00:58)

Offline

#486 2013-09-18 16:05:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just uploaded qemu 1.6, kernel 3.11 and seabios 1.7.3 packages to the main guide, qemu 1.6 seems to require seabios 1.7.3

Offline

#487 2013-09-18 16:10:22

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Try without "x-vga=on" on GPU1

still no signal

Offline

#488 2013-09-18 16:19:00

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:
nbhs wrote:

Try without "x-vga=on" on GPU1

still no signal

Im assuming its working ok with one card, but if you passthrough another card it doesnt work, right? Could you post your hardware specs? you can try to passthrough the secondary card as a regular pci device

-device vfio-pci,host=02:00.0

Offline

#489 2013-09-18 16:33:02

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, correctly. With one card Windows work fine, and i have to install Catalyst.

I have 2 radeon 01:00.0/1 and 02:00.0/1 and Nvidia 210 05:00.0/1 (open source driver), radeon and fglrx modules blacklisted.

I try add my second card in PCI_DEVICES="02:00.0" its not working, shows no signal too.

just in case, i was add my ati cards in pci-stub, and try. Nothing has changed.

lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port C) 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI Express bridge for GPP2 port 1 00:11.0 RAID bus controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [RAID5 mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 05:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce 210] (rev a2) 05:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1) 06:06.0 Multimedia audio controller: Creative Labs SB X-Fi 06:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06) 08:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
lspci -n 00:00.0 0600: 1002:5a14 (rev 02) 00:00.2 0806: 1002:5a23 00:02.0 0604: 1002:5a16 00:03.0 0604: 1002:5a17 00:09.0 0604: 1002:5a1c 00:0a.0 0604: 1002:5a1d 00:0c.0 0604: 1002:5a20 00:11.0 0104: 1002:4393 (rev 40) 00:12.0 0c03: 1002:4397 00:12.2 0c03: 1002:4396 00:13.0 0c03: 1002:4397 00:13.2 0c03: 1002:4396 00:14.0 0c05: 1002:4385 (rev 42) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 00:15.0 0604: 1002:43a0 00:15.1 0604: 1002:43a1 00:15.2 0604: 1002:43a2 00:16.0 0c03: 1002:4397 00:16.2 0c03: 1002:4396 00:18.0 0600: 1022:1600 00:18.1 0600: 1022:1601 00:18.2 0600: 1022:1602 00:18.3 0600: 1022:1603 00:18.4 0600: 1022:1604 00:18.5 0600: 1022:1605 01:00.0 0300: 1002:6798 01:00.1 0403: 1002:aaa0 02:00.0 0300: 1002:6798 02:00.1 0403: 1002:aaa0 03:00.0 0c03: 1b6f:7023 (rev 01) 04:00.0 0106: 1b4b:9172 (rev 11) 05:00.0 0300: 10de:0a65 (rev a2) 05:00.1 0403: 10de:0be3 (rev a1) 06:06.0 0401: 1102:0005 06:0e.0 0c00: 1106:3044 (rev c0) 07:00.0 0200: 10ec:8168 (rev 06) 08:00.0 0c03: 1b6f:7023 (rev 01) 09:00.0 0106: 1b4b:9172 (rev 11)

Last edited by chUpa (2013-09-18 16:38:23)

Offline

#490 2013-09-18 16:36:04

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:
chUpa wrote:

I do not see any problem in this. In my motherboard, I have 6 pcie ports. My host using card that which is in the lower slot. Its address 0000:05:00.0. Passthrough cards have addresses 0000:01:00.0/0000:02:00.0.

Okay, thats nice, i think i just missunderstood some earlier post then.

As long as its not your primary card you could use any slot you want

When does a graphic card become my primary one?

Offline

#491 2013-09-18 16:39:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
nbhs wrote:
SpacePirate wrote:

Okay, thats nice, i think i just missunderstood some earlier post then.

As long as its not your primary card you could use any slot you want

When does a graphic card become my primary one?

Its the one that you see the POST messages

Offline

#492 2013-09-18 16:41:29

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
nbhs wrote:
SpacePirate wrote:

Okay, thats nice, i think i just missunderstood some earlier post then.

As long as its not your primary card you could use any slot you want

When does a graphic card become my primary one?

I think you should choose it in you BIOS. I boot from PEG3 card.

Offline

#493 2013-09-18 16:44:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:

Yes, correctly. With one card Windows work fine, and i have to install Catalyst.

I have 2 radeon 01:00.0/1 and 02:00.0/1 and Nvidia 210 05:00.0/1 (open source driver), radeon and fglrx modules blacklisted.

I try add my second card in PCI_DEVICES="02:00.0" its not working, shows no signal too.

just in case, i was add my ati cards in pci-stub, and try. Nothing has changed.

lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port C) 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI Express bridge for GPP2 port 1 00:11.0 RAID bus controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [RAID5 mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 05:00.0 VGA compatible controller: NVIDIA Corporation GT218 [GeForce 210] (rev a2) 05:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1) 06:06.0 Multimedia audio controller: Creative Labs SB X-Fi 06:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06) 08:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
lspci -n 00:00.0 0600: 1002:5a14 (rev 02) 00:00.2 0806: 1002:5a23 00:02.0 0604: 1002:5a16 00:03.0 0604: 1002:5a17 00:09.0 0604: 1002:5a1c 00:0a.0 0604: 1002:5a1d 00:0c.0 0604: 1002:5a20 00:11.0 0104: 1002:4393 (rev 40) 00:12.0 0c03: 1002:4397 00:12.2 0c03: 1002:4396 00:13.0 0c03: 1002:4397 00:13.2 0c03: 1002:4396 00:14.0 0c05: 1002:4385 (rev 42) 00:14.2 0403: 1002:4383 (rev 40) 00:14.3 0601: 1002:439d (rev 40) 00:14.4 0604: 1002:4384 (rev 40) 00:14.5 0c03: 1002:4399 00:15.0 0604: 1002:43a0 00:15.1 0604: 1002:43a1 00:15.2 0604: 1002:43a2 00:16.0 0c03: 1002:4397 00:16.2 0c03: 1002:4396 00:18.0 0600: 1022:1600 00:18.1 0600: 1022:1601 00:18.2 0600: 1022:1602 00:18.3 0600: 1022:1603 00:18.4 0600: 1022:1604 00:18.5 0600: 1022:1605 01:00.0 0300: 1002:6798 01:00.1 0403: 1002:aaa0 02:00.0 0300: 1002:6798 02:00.1 0403: 1002:aaa0 03:00.0 0c03: 1b6f:7023 (rev 01) 04:00.0 0106: 1b4b:9172 (rev 11) 05:00.0 0300: 10de:0a65 (rev a2) 05:00.1 0403: 10de:0be3 (rev a1) 06:06.0 0401: 1102:0005 06:0e.0 0c00: 1106:3044 (rev c0) 07:00.0 0200: 10ec:8168 (rev 06) 08:00.0 0c03: 1b6f:7023 (rev 01) 09:00.0 0106: 1b4b:9172 (rev 11)

Are you sure the nvidia card is your primary card? have you tried to comment out the VBIOS variable?

Offline

#494 2013-09-18 16:56:38

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Are you sure the nvidia card is your primary card? have you tried to comment out the VBIOS variable?

I draw my conclusions from the fact that the bios and arch boot and work from my nvidia card, even if I turn off second HDMI cable from my PC.

when I did like this:

#GPU1="02:00.0" #GPU_SND1="02:00.1" #VBIOS0="/home/cd/rom0.rom" #VBIOS1="/home/cd/rom1.rom"

all working fine. The only difference is fan in my ati card is worked louder until not windows boot.

Last edited by chUpa (2013-09-18 17:06:05)

Offline

#495 2013-09-18 17:48:13

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

and when i try this:

PCI_DEVICES="02:00.0 02:00.1" GPU0="01:00.0" GPU_SND0="01:00.1" #GPU1="02:00.0" #GPU_SND1="02:00.1" VBIOS0="/home/cd/rom0.rom" (or #VBIOS0="/home/cd/rom0.rom") #VBIOS1="/home/cd/rom1.rom"

I have no effect. No signal as global result. (:
But my credo never give up, hold on to the result! (;

Last edited by chUpa (2013-09-18 17:51:08)

Offline

#496 2013-09-18 18:08:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:

and when i try this:

PCI_DEVICES="02:00.0 02:00.1" GPU0="01:00.0" GPU_SND0="01:00.1" #GPU1="02:00.0" #GPU_SND1="02:00.1" VBIOS0="/home/cd/rom0.rom" (or #VBIOS0="/home/cd/rom0.rom") #VBIOS1="/home/cd/rom1.rom"

I have no effect. No signal as global result. (:
But my credo never give up, hold on to the result! (;

comment out the VBIOS line

Offline

#497 2013-09-18 18:18:45

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

comment out the VBIOS line


Tried both variants

Offline

#498 2013-09-19 08:17:05

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Today has updated the system. Now i have

Linux barby 3.11.1-1-ARCH #1 SMP PREEMPT Sat Sep 14 19:30:21 CEST 2013 x86_64 GNU/Linux

and

qemu-git 1.7.r28759.g6c2679f-1
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux root=UUID=df9ed763-fa79-41fd-8314-ca4a108ee274 rw pci-stub.ids=1002:6798,1002:aaa0,1102:0005
ls /etc/modprobe.d/ blacklist.conf kvm.conf vfio_iommu_type1.conf
cat /etc/modprobe.d/blacklist.conf blacklist radeon blacklist fglrx
cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1
cat /etc/modprobe.d/vfio_iommu_type1.conf options vfio_iommu_type1 allow_unsafe_interrupts=1
ls /etc/modules-load.d/ kvm_amd.conf kvm.conf vfio.conf vfio_iommu_type1.conf vfio_pci.conf
cat /etc/modules-load.d/kvm_amd.conf kvm_amd
cat /etc/modules-load.d/kvm.conf kvm
 cat /etc/modules-load.d/vfio.conf vfio
cat /etc/modules-load.d/vfio_iommu_type1.conf vfio_iommu_type1
cat /etc/modules-load.d/vfio_pci.conf vfio_pci

Without any patches. Nothing has changed. Only one card working. When I try a second, I have no signal.

Last edited by chUpa (2013-09-19 08:18:45)

Offline

#499 2013-09-19 19:44:32

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

i`m now here:
FluxBB bbcode test

Second monitor, connected to passthrough VGA - no output (black screen).

Boot guest using this script:

/usr/bin/qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 4096 \ -cpu host \ -bios /home/gosheto/Downloads/bios.bin \ -vga cirrus \ -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/home/gosheto/Downloads/Palit.GTX580.1536.101102.rom \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/gosheto/VM/windowsq35.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/home/gosheto/VM/windows7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \ -boot menu=on

If using
-vga none and x-vga=on get black QEMU windows and system freeze after 10 seconds (1 of cpu get 100% load).

In dmesg get this messages:

vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:05:00.1 hiding ecap 0x0@0x100 kvm: zapping shadow pages for mmio generation wraparound assign device 0:5:0.0 assign device 0:5:0.1 etc

Any ideas?

PS: 0000:05:00.1 (NVIDIA HD Audio working properly)

Thank you in advance!

Last edited by gosheto (2013-09-19 19:46:42)

Offline

#500 2013-09-19 19:52:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gosheto wrote:

Hello,

i`m now here:
http://s15.postimg.org/ad6dnw2zv/Selection_001.png

Second monitor, connected to passthrough VGA - no output (black screen).

Boot guest using this script:

/usr/bin/qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 4096 \ -cpu host \ -bios /home/gosheto/Downloads/bios.bin \ -vga cirrus \ -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/home/gosheto/Downloads/Palit.GTX580.1536.101102.rom \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/gosheto/VM/windowsq35.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/home/gosheto/VM/windows7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \ -boot menu=on

If using
-vga none and x-vga=on get black QEMU windows and system freeze after 10 seconds (1 of cpu get 100% load).

In dmesg get this messages:

vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:05:00.1 hiding ecap 0x0@0x100 kvm: zapping shadow pages for mmio generation wraparound assign device 0:5:0.0 assign device 0:5:0.1 etc

Any ideas?

PS: 0000:05:00.1 (NVIDIA HD Audio working properly)

Thank you in advance!

Never seen that message before what kernel/qemu version are you using?

Offline

#501 2013-09-20 08:05:55

gosheto
Member
Registered: 2013-09-04
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I`m using latest stable kernel 3.11.1 with your linux-mainline.tar.gz (3.11) - all patches and configured with your config.x86_64
Qemu version 1.6.0 patched with your qemu.tar.gz (1.6)
SeaBIOS v1.7.3 QEMU/KVM image - bios.bin-1.7.3.gz

Offline

#502 2013-09-21 03:52:55

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gosheto wrote:

Hi,

I`m using latest stable kernel 3.11.1 with your linux-mainline.tar.gz (3.11) - all patches and configured with your config.x86_64
Qemu version 1.6.0 patched with your qemu.tar.gz (1.6)
SeaBIOS v1.7.3 QEMU/KVM image - bios.bin-1.7.3.gz

Have you tried the previous qemu version? im having some problems myself using qemu 1.6 ( system crawls and eventually hangs at windows login prompt)

Offline

#503 2013-09-23 02:19:59

Smasher816
Member
From: Cyberspace of Missouri
Registered: 2012-03-15
Posts: 32
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Awesome write  nbhs. Unfortunately I am having some strange issues.

I have an AMD FX processor, a Gigabyte motherboard (with IOMMU support), an Nvidia GTX graphics card (for Linux, primary card in bios, proprietary driver), and an ATI Radeon HD 5770 (for Windows). I successfully got vga passthrough working with Xen in the past, but it made the dom0 graphics acceleration fail. I am now attempting this KVM method which looks promising.

If I do not have the nvidia module built and boot  into the command line I can successfully see the ati card working. However, once I install the nvidia module and boot into X, all I get is a black screen on the second card.

I am attempting to test the passthrough with these commands:

$ sudo vfio-bind 0000:01:00.0 0000:01:00.1 $ sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Installed packages (patched versions from first post):

linux-mainline 3.11-1 linux-mainline-headers 3.11-1 qemu 1.6.0-9 seabios 1.7.3-1 nvidia-304xx 304.108-5 (rebuilt for linux-mainline kernel)

lspci output:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770] Subsystem: PC Partner Limited Device e132 Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] Subsystem: PC Partner Limited Device aa58 Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

Edit: Fixed. See this patch http://lists.gnu.org/archive/html/qemu- … 9_I13p.txt

Last edited by Smasher816 (2013-09-23 02:51:42)


(Arch) Linux is user friendly, its just very selective of its friends

Offline

#504 2013-09-23 02:47:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Smasher816 wrote:

Awesome write  nbhs. Unfortunately I am having some strange issues.

I have an AMD FX processor, a Gigabyte motherboard (with IOMMU support), an Nvidia GTX graphics card (for Linux, primary card in bios, proprietary driver), and an ATI Radeon HD 5770 (for Windows). I successfully got vga passthrough working with Xen in the past, but it made the dom0 graphics acceleration fail. I am now attempting this KVM method which looks promising.

If I do not have the nvidia module built and boot  into the command line I can successfully see the ati card working. However, once I install the nvidia module and boot into X, all I get is a black screen on the second card.

I am attempting to test the passthrough with these commands:

$ sudo vfio-bind 0000:01:00.0 0000:01:00.1 $ sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Installed packages (patched versions from first post):

linux-mainline 3.11-1 linux-mainline-headers 3.11-1 qemu 1.6.0-9 seabios 1.7.3-1 nvidia-304xx 304.108-5 (rebuilt for linux-mainline kernel)

lspci output:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770] Subsystem: PC Partner Limited Device e132 Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] Subsystem: PC Partner Limited Device aa58 Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

Its on the guide

nbhs wrote:

Using NVIDIA proprietary drivers on the requires a patch: see http://www.mail-archive.com/qemu-devel@ … 74066.html  patch (posted by andy123):  http://bpaste.net/show/109185/

Good luck!

Last edited by nbhs (2013-09-23 02:48:03)

Offline

#505 2013-09-23 02:56:14

Smasher816
Member
From: Cyberspace of Missouri
Registered: 2012-03-15
Posts: 32
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Smasher816 wrote:

<snip>

Its on the guide

nbhs wrote:

Using NVIDIA proprietary drivers on the requires a patch: see http://www.mail-archive.com/qemu-devel@ … 74066.html  patch (posted by andy123):  http://bpaste.net/show/109185/

Good luck!

Thanks for the quick reply. It works now big_smile

I ended up finding a link to the same patch on page 16. I read through the first post once before actually attempting anything. I guess when I skimmed over it a second time I managed to miss that link.


(Arch) Linux is user friendly, its just very selective of its friends

Offline

#506 2013-09-23 03:04:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Smasher816 wrote:
nbhs wrote:
Smasher816 wrote:

<snip>

Its on the guide

nbhs wrote:

Using NVIDIA proprietary drivers on the requires a patch: see http://www.mail-archive.com/qemu-devel@ … 74066.html  patch (posted by andy123):  http://bpaste.net/show/109185/

Good luck!

Thanks for the quick reply. It works now big_smile

I ended up finding a link to the same patch on page 16. I read through the first post once before actually attempting anything. I guess when I skimmed over it a second time I managed to miss that link.

Nice, i just realized there was a typo i forgot the word "host", anyway check out my second post on "performance improvements", specifically disabling npt.

Offline

#507 2013-09-24 01:23:21

Smasher816
Member
From: Cyberspace of Missouri
Registered: 2012-03-15
Posts: 32
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright. My windows.img (Windows 7, raw file) used with Xen works under KVM too.

I am now attempting to virtualize the physical Windows partition I dual-boot into. My Arch Linux install is on an ssd (sda1), and Windows is on a hard drive (sdb3). lspci only shows one SATA controller, therefore I am passing through my whole second hard drive (which has its own bootloader).

Passing /dev/sdb to Qemu shows the Syslinux bootloader, and then begins booting Windows. Unfortunately it quickly blue screens of death with the error "*** STOP: 0x0000007B (0xFFFFF880009A97E8,0xFFFFFFFFC0000034,0x0000000000000000,0x0000000000000000)". I believe this means that Windows does not like the Qemu disk controller (piix4-ide). When booting in in ahci mode it will error before Windows starts (bios is in IDE mode if I remember correctly). Is it possible to have qemu use a different device Windows 7 will recognize? Perhaps there is a way to boot into Windows and install some drivers before virtualizing it with qemu?

I also have an issue with audio. When using the two lines from your original post any sounds made in the Windows vm will stutter badly. With Xen I passed through my onboard audio controller, and used a dedicated sound card for Linux. I then had the speaker port of one plugged into line in of the other. When trying this with KVM I face issues. When I use vfio-bind on 0000:00:14.2 my sound stops working. Note: alsa is configured to use the CMI8768 card and the onboard should be disabled with "pci-stubs.ids=....".

The two cards are:

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 05:06.0 Multimedia audio controller: C-Media Electronics Inc CMI8738/CMI8768 PCI Audio (rev 10)

Thanks in advance for any help.

Last edited by Smasher816 (2013-09-24 01:36:48)


(Arch) Linux is user friendly, its just very selective of its friends

Offline

#508 2013-09-24 03:16:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Smasher816 wrote:

Alright. My windows.img (Windows 7, raw file) used with Xen works under KVM too.

I am now attempting to virtualize the physical Windows partition I dual-boot into. My Arch Linux install is on an ssd (sda1), and Windows is on a hard drive (sdb3). lspci only shows one SATA controller, therefore I am passing through my whole second hard drive (which has its own bootloader).

Passing /dev/sdb to Qemu shows the Syslinux bootloader, and then begins booting Windows. Unfortunately it quickly blue screens of death with the error "*** STOP: 0x0000007B (0xFFFFF880009A97E8,0xFFFFFFFFC0000034,0x0000000000000000,0x0000000000000000)". I believe this means that Windows does not like the Qemu disk controller (piix4-ide). When booting in in ahci mode it will error before Windows starts (bios is in IDE mode if I remember correctly). Is it possible to have qemu use a different device Windows 7 will recognize? Perhaps there is a way to boot into Windows and install some drivers before virtualizing it with qemu?

I also have an issue with audio. When using the two lines from your original post any sounds made in the Windows vm will stutter badly. With Xen I passed through my onboard audio controller, and used a dedicated sound card for Linux. I then had the speaker port of one plugged into line in of the other. When trying this with KVM I face issues. When I use vfio-bind on 0000:00:14.2 my sound stops working. Note: alsa is configured to use the CMI8768 card and the onboard should be disabled with "pci-stubs.ids=....".

The two cards are:

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 05:06.0 Multimedia audio controller: C-Media Electronics Inc CMI8738/CMI8768 PCI Audio (rev 10)

Thanks in advance for any help.

you'll need to modfiy the windows registry i believe some people had this problem some pages back, about the sound you could try another driver instead of pulse, like also or oss

Offline

#509 2013-09-25 13:34:10

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow. Many pages in here big_smile.

Have to do some reading and will probably give the vfio a try again the next days.
Unfortunately I didn't found time/motivation to do more testing/using with the pci-assign in the past.

Was there some progress with vfio and Intel IGPs (eg in i5) as primary GPU?

Greetings
apo

Offline

#510 2013-09-25 18:14:27

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just wanted to start trying this on  my new machine. When building the qemu-git package from the AUR it conflicts with some files by seasbios...
Do i still need seabios or was its functionality merged into qemu itself?

EDIT: I just recognized something in the first post. The output of "dmesg | grep pci-stub" contains one line more than one would expect with the posted kernel parameters. I suppose the next pci device is a USB-Controller you added in your configuration (?).

EDIT2: So i tried to run this on my machine, but it does not seem to work. I installed the linux-mainline (3.11), qemu (1.6) and seabios (1.7.3) from the packagebuilds you posted. IOMMU is enabled, the pci-bind kernel boot parameter is set and i execute the vfio-bind script using your systemctl service.

# dmesg | grep -iE 'iommu|vfio|pci-bind' [ 0.848089] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 0.970713] VFIO - User Level meta-driver version: 0.3 [ 229.648012] vfio-pci 0000:06:00.0: enabling device (0000 -> 0003) [ 476.262919] vfio-pci 0000:06:00.1: enabling device (0000 -> 0002) # lspci 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] # lspci -n 06:00.0 0300: 1002:6779 06:00.1 0403: 1002:aa98 # cat /etc/vfio-pci.cfg DEVICES="0000:06:00.1 0000:06:00.1"

This is the script i use to run the VM: (it`s easier to edit the commands in there)

#!/bin/sh qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1

When i run the vm, my main screen goes black after switching to tty2 and back again its normal again (i suppose its a Gnome or X bug). I have an empty qemu window. After plugging my monitor to the second GPU i get no signal from it.
Some thoughts i had on what could cause the problem:
- maybe the 6450 doesnt support switching? I want to use my 7870 but i cant choose my primary boot device in bios so i need to change the slots (havent done that yet)
- do i need to have the monitor plugged in, when the vm starts?

Any help is greatly appreciated!

Last edited by SpacePirate (2013-09-26 10:18:46)

Offline

#511 2013-09-26 15:54:31

Endrik
Member
Registered: 2013-09-26
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

Wow. Many pages in here big_smile.

Have to do some reading and will probably give the vfio a try again the next days.
Unfortunately I didn't found time/motivation to do more testing/using with the pci-assign in the past.

Was there some progress with vfio and Intel IGPs (eg in i5) as primary GPU?

Greetings
apo

I don't know if there has been any progress with the IGP - when i try to passtrough my intel Card qemu complains:

qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get device 0000:00:02.0
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized


passing the radeon card works fine - except the "bug" that after a guest reboot only the seabios message is passed, after "grub init" the display stays blank and the guest doesnt boot up hmm

i am currently using qemu/seabios from git and kernel 3.11.0 (debian) with vga-vfio enabled

I have no clue why i can't pass-trough the intel-gpu, the i915 driver ist blacklisted and the radeon is primary GPU. Starting qemu without x-vga=on boots up the guest - but the monitor stays off and the guest kernel complains:
[    3.883569] [drm] Initialized drm 1.1.0 20060810
[    3.893808] [drm] Memory usable by graphics device = 2048M
[    3.895049] i915 0000:01:00.0: setting latency timer to 64
[    3.897831] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to 61208
[    3.897834] [drm:hsw_unclaimed_reg_check] *ERROR* Unclaimed write to 61208
[    3.897836] [drm:hsw_unclaimed_reg_check] *ERROR* Unclaimed write to 6120c
[    3.898233] i915 0000:01:00.0: irq 46 for MSI/MSI-X
[    3.981291] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    3.981301] [drm] Driver supports precise vblank timestamp query.
[    3.981317] i915 0000:01:00.0: BAR 6: can't assign [??? 0x00000000 flags 0x0] (bogus alignment)
[    3.981321] [drm] failed to find VBIOS tables
....

Maybe someone found a workaround?
Greetings

Edit1: When booting the host up with the intel as primary device and running the qemu command to pass-through the intel card with x-vga on: qemu does not complain, but screen turns off (login prompt from the host vanishes and the monitor turns off)

Last edited by Endrik (2013-09-26 16:33:15)

Offline

#512 2013-09-26 20:14:33

Smasher816
Member
From: Cyberspace of Missouri
Registered: 2012-03-15
Posts: 32
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Smasher816 wrote:

Alright. My windows.img (Windows 7, raw file) used with Xen works under KVM too.

<snip>

you'll need to modfiy the windows registry i believe some people had this problem some pages back, about the sound you could try another driver instead of pulse, like also or oss

Just a quick update. I got a fresh virtual installation to work well, but could not convert my previous install. Sound works great using alsa and "-soundhw ac97" along with the ac97 drivers in Windows 7. Both virtio disk and network (bridged tap) are working well.

The Windows Experience Index gives me 7.4 for the processor, 7.9 for memory, 7.2 for graphics, and 4.8 for disk (its an old hard drive). Overall the virtual machine runs great big_smile

I am still unable to pass through my physical sata controller and sound card because they are part of a ten device iommu_group. I have already tried using pcie_acs_override=... with downstream, mulitplex, and the vendor ids of various controllers to no avail. As it stands it would be nice to get more manageable iommu groups, but it is not a game breaker. Thanks again for the awesome guide nbhs.

Edit: is it possible to use usb 2.0 with pci passthrough? Windows keeps yelling at me saying "this device can perform better" if it was plugged into a usb 2 port.

Last edited by Smasher816 (2013-09-27 03:29:15)


(Arch) Linux is user friendly, its just very selective of its friends

Offline

#513 2013-09-26 22:56:00

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey again.

I upgraded qemu, seabios and the kernel with the builds on the first page since things weren't working before anyway, but now when I try to launch the test machine I get this error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Error: Failed to setup INTx fd: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Also, the kvm module isn't loading on boot. I've added it to modules-load.d but it doesn't want to work

dmesg shows this when I try creating the vm:

genirq: Flags mismatch irq 17. 00000080 (vfio-intx(0000:02:00.0)) vs. 00002000 (kvm:0000:07:00.0)

Last edited by Chetyre (2013-09-26 23:12:47)

Offline

#514 2013-09-27 08:58:57

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So i swapped the Cards now, to check if maybe passing the 7870 works and now the qemu process crashes...:
http://ideone.com/5fxU8z
Any ideas on that?

Offline

#515 2013-09-29 12:09:02

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey,

My problem is that I keep getting "ls: cannot access /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices: No such file or directory"
I have an nvidia 9800GT as the system's GPU and I want to bind my GTX 660 Ti.
I had nvidia drivers, but I saw in page 3 that they have a bug and that's why I don't have the "iommu_group" directory.
Then I changed to nouveau. The problem remains.
Then I built your linux-mainline (I was on 3.11.1-1). The problem still remains.
I can't bind my 660... :\

`dmesg| grep -i pci-`

[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=e78cde76-376a-4d00-b994-18453659643a quiet add_efi_memmap rw pci-stub.ids=10de:0e0a,10de:1183 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=e78cde76-376a-4d00-b994-18453659643a quiet add_efi_memmap rw pci-stub.ids=10de:0e0a,10de:1183 [ 0.792129] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 0.881226] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.881234] pci-stub 0000:02:00.1: claimed by stub [ 0.881240] pci-stub: add 10DE:1183 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.881244] pci-stub 0000:02:00.0: claimed by stub

My mobo and CPU (Asus P9X79 Deluxe and i7-3820) support interrupt remapping (VT-d) and I've enabled it, so there is no need for the "vfio_iommu_type1.allow_unsafe_interrupts=1" at my kernel line. Right?

Thank you in advance

Offline

#516 2013-09-29 14:27:50

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just a wild guess, did you load the module / compile it into the kernel?

Last edited by SpacePirate (2013-09-29 14:30:34)

Offline

#517 2013-09-29 14:29:19

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes. I use the patched linux-mainline kernel and I double-checked it.

Offline

#518 2013-10-02 13:35:27

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any help please?
I would be very glad if I could enjoy bf3 without dual-booting...

Offline

#519 2013-10-02 14:09:35

El_Presidente
Member
Registered: 2013-10-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Any help please?
I would be very glad if I could enjoy bf3 without dual-booting...

i think you have not enabled vt-d within the kernel. i dont see "intel_iommu=on" in your kernel command line.

Offline

#520 2013-10-02 14:36:19

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ooo, thank you very much... it now binds smile
I'll edit if I find any other problem...

EDIT: I get blank screen when I try to run qemu (the patched for mainline one) and in dmesg I get: "vfio-pci 0000:02:00.0: Invalid ROM contents"
Googling proved fruitless and I saw the same problem on the first page and there was no solution...
Also qemu reports: "qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:02:00.0 does not support reset"

EDIT 2: Long boot cause found, completely irrelevant

Last edited by ttouch (2013-10-04 19:47:18)

Offline

#521 2013-10-03 19:17:58

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

I just wanted to start trying this on  my new machine. When building the qemu-git package from the AUR it conflicts with some files by seasbios...
Do i still need seabios or was its functionality merged into qemu itself?

EDIT: I just recognized something in the first post. The output of "dmesg | grep pci-stub" contains one line more than one would expect with the posted kernel parameters. I suppose the next pci device is a USB-Controller you added in your configuration (?).

EDIT2: So i tried to run this on my machine, but it does not seem to work. I installed the linux-mainline (3.11), qemu (1.6) and seabios (1.7.3) from the packagebuilds you posted. IOMMU is enabled, the pci-bind kernel boot parameter is set and i execute the vfio-bind script using your systemctl service.

# dmesg | grep -iE 'iommu|vfio|pci-bind' [ 0.848089] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 0.970713] VFIO - User Level meta-driver version: 0.3 [ 229.648012] vfio-pci 0000:06:00.0: enabling device (0000 -> 0003) [ 476.262919] vfio-pci 0000:06:00.1: enabling device (0000 -> 0002) # lspci 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] # lspci -n 06:00.0 0300: 1002:6779 06:00.1 0403: 1002:aa98 # cat /etc/vfio-pci.cfg DEVICES="0000:06:00.1 0000:06:00.1"

This is the script i use to run the VM: (it`s easier to edit the commands in there)

#!/bin/sh qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1

When i run the vm, my main screen goes black after switching to tty2 and back again its normal again (i suppose its a Gnome or X bug). I have an empty qemu window. After plugging my monitor to the second GPU i get no signal from it.
Some thoughts i had on what could cause the problem:
- maybe the 6450 doesnt support switching? I want to use my 7870 but i cant choose my primary boot device in bios so i need to change the slots (havent done that yet)
- do i need to have the monitor plugged in, when the vm starts?

Any help is greatly appreciated!

Could you post your specs?

Offline

#522 2013-10-03 21:51:01

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:

...

Could you post your specs?

CPU:   AMD FX-8350
MB:     Gigabyte 970A-UD3
GPU1: Gigabyte HD7870 GHz
GPU2: Saphire HD6450

+ 16GiB RAM

Offline

#523 2013-10-03 21:54:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
nbhs wrote:
SpacePirate wrote:

...

Could you post your specs?

CPU:   AMD FX-8350
MB:     Gigabyte 970A-UD3
GPU1: Gigabyte HD7870 GHz
GPU2: Saphire HD6450

+ 16GiB RAM

Are you using the catalyst drivers?, are you using pci-stub to "hide" the card?

Last edited by nbhs (2013-10-03 21:58:13)

Offline

#524 2013-10-04 00:02:51

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:
nbhs wrote:

Could you post your specs?

CPU:   AMD FX-8350
MB:     Gigabyte 970A-UD3
GPU1: Gigabyte HD7870 GHz
GPU2: Saphire HD6450

+ 16GiB RAM

Are you using the catalyst drivers?, are you using pci-stub to "hide" the card?

Yes i use pci-stub, but the open source drivers.

Offline

#525 2013-10-04 20:58:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#526 2013-10-05 02:46:34

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

Awsome news! thanks a lot Alex!

Offline

#527 2013-10-05 15:02:53

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

So with this new progress i will need a linux 3.12-rc1 or later and a pull from alex` repository (https://github.com/awilliam/qemu-vfio/r … 20131003.0)?
It looks like seabios was merged into qemu so i wont need to install it?

Offline

#528 2013-10-05 15:12:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
aw wrote:

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

So with this new progress i will need a linux 3.12-rc1 or later and a pull from alex` repository (https://github.com/awilliam/qemu-vfio/r … 20131003.0)?
It looks like seabios was merged into qemu so i wont need to install it?

Right, you can use seabios as-is in qemu.  If you use my tag note that it does not include the patch in the 2nd link.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#529 2013-10-05 15:19:53

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SpacePirate wrote:
aw wrote:

...

So with this new progress i will need a linux 3.12-rc1 or later and a pull from alex` repository (https://github.com/awilliam/qemu-vfio/r … 20131003.0)?
It looks like seabios was merged into qemu so i wont need to install it?

Right, you can use seabios as-is in qemu.  If you use my tag note that it does not include the patch in the 2nd link.

Alright, I will try to get it running with this. Since i use AMD i dont think i need that patch.

Offline

#530 2013-10-05 16:44:04

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so I tried 3.12-rc3 and qemu from https://github.com/awilliam/qemu-vfio.
It resulted in a lot of error messages in dmesg, but it worked.
Afterwards, if I start qemu, it says

file_ram_alloc: can't mmap RAM pages: Cannot allocate memory

and does't seem to use my 4 1gb hugepages…
I'll reboot later and try again. My current kernel is compiled with "# CONFIG_DEBUG_KERNEL is not set" for some reason, is a recompile with "CONFIG_DEBUG_KERNEL=y" likely to give useful information?

While I'm at it, has there been any news on the broken hardware front (http://lkml.indiana.edu/hypermail/linux … 00079.html)?


i'm sorry for my poor english wirting skills…

Offline

#531 2013-10-05 17:20:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Ok, so I tried 3.12-rc3 and qemu from https://github.com/awilliam/qemu-vfio.
It resulted in a lot of error messages in dmesg, but it worked.
Afterwards, if I start qemu, it says

file_ram_alloc: can't mmap RAM pages: Cannot allocate memory

and does't seem to use my 4 1gb hugepages…
I'll reboot later and try again. My current kernel is compiled with "# CONFIG_DEBUG_KERNEL is not set" for some reason, is a recompile with "CONFIG_DEBUG_KERNEL=y" likely to give useful information?

While I'm at it, has there been any news on the broken hardware front (http://lkml.indiana.edu/hypermail/linux … 00079.html)?

dmesg errors are all from kvm, not vfio.  Do these happen regardless of an assigned device?  What if you don't use hugepages?  What if you boot the guest with a single CPU?  WRT broken hardware, no time lately.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#532 2013-10-05 19:33:35

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
andy123 wrote:

dmesg errors are all from kvm, not vfio.  Do these happen regardless of an assigned device?  What if you don't use hugepages?  What if you boot the guest with a single CPU?  WRT broken hardware, no time lately.

Yeah, right. To clarify, the vm starts despite the mmap error and without "-mem-path" the error goes away. Not assinging devices or using only one cpu doesn't change anything.
I did some testing and the first vm start after a reboot results in no error from qemu, but the dmesg errors and everything afterwards returns "file_ram_alloc: can't mmap RAM pages: Cannot allocate memory" (but dmesg stays the same).
I tried something simple as a first start to try to reduce error sources:

/usr/bin/qemu-system-x86_64 -enable-kvm -m 4096 -mem-path /dev/hugepages -drive file=/files/vm/arch.img,id=drive0,if=virtio,media=disk -monitor stdio

In case I have the same problem with upstream stable qemu, any idea where to report this?


i'm sorry for my poor english wirting skills…

Offline

#533 2013-10-05 20:00:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
aw wrote:

dmesg errors are all from kvm, not vfio.  Do these happen regardless of an assigned device?  What if you don't use hugepages?  What if you boot the guest with a single CPU?  WRT broken hardware, no time lately.

Yeah, right. To clarify, the vm starts despite the mmap error and without "-mem-path" the error goes away. Not assinging devices or using only one cpu doesn't change anything.

What happens with 2M hugepages?

andy123 wrote:

I did some testing and the first vm start after a reboot results in no error from qemu, but the dmesg errors and everything afterwards returns "file_ram_alloc: can't mmap RAM pages: Cannot allocate memory" (but dmesg stays the same).
I tried something simple as a first start to try to reduce error sources:

/usr/bin/qemu-system-x86_64 -enable-kvm -m 4096 -mem-path /dev/hugepages -drive file=/files/vm/arch.img,id=drive0,if=virtio,media=disk -monitor stdio

In case I have the same problem with upstream stable qemu, any idea where to report this?

My qemu tree only modifies vfio, so if it happens without an assigned device, it should happen on qemu.git.  If you can narrow it down (or even if you can't), please report it to:

kvm@vger.kernel.org, cc qemu-devel@nongnu.org


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#534 2013-10-06 00:32:51

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What happens with 2M hugepages?

My qemu tree only modifies vfio, so if it happens without an assigned device, it should happen on qemu.git.  If you can narrow it down (or even if you can't), please report it to:

kvm@vger.kernel.org, cc qemu-devel@nongnu.org

There are no problems with 2M hugepages.
Thanks for those adresses, I did some futher testing and I noticed, that the hugepages seem to "fill up". With 4x1G hugepages, I can start and stop 2 virtual machines with 2gb ram with dmesg errors, but without qemu telling me, that it can't mmap, 4 virtualmachines if they use 1gb each.


i'm sorry for my poor english wirting skills…

Offline

#535 2013-10-06 03:19:16

tarcisioe
Member
Registered: 2013-10-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am trying this with the following setup:

CPU: i7 2600 (supports VT-d)
Motherboard: Gigabyte GA-P67A-UD3-B3 (Current BIOS F4a, VT-d is enabled on BIOS setup)
Video cards:
- GeForce GT 620 (for Linux host)
- GeForce GTX 650 Ti (for Windows)

I can't seem to turn Intel_IOMMU on, though. When I try, Arch boots to weird messages about "can't find DMAR" for various devices and finally hangs, unable to find the hard drive.

The kernel line I am (trying to, actually) using is:
linux    /vmlinuz-linux root=UUID=89c6d568-9a2f-4de8-ad98-5f210c6a7163 rw  quiet iommu=1 intel_iommu=on pci-stub.ids=10de:11c6,10de:0b0e

I can't seem to get past this. Without intel_iommu=on, I get
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.0/iommu_group/devices: No such file or directory
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.1/iommu_group/devices: No such file or directory

when trying to run vfio-bind.

Any help will be much appreciated, so thanks in advance.

Offline

#536 2013-10-07 15:54:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tarcisioe wrote:

Hi,

I am trying this with the following setup:

CPU: i7 2600 (supports VT-d)
Motherboard: Gigabyte GA-P67A-UD3-B3 (Current BIOS F4a, VT-d is enabled on BIOS setup)
Video cards:
- GeForce GT 620 (for Linux host)
- GeForce GTX 650 Ti (for Windows)

I can't seem to turn Intel_IOMMU on, though. When I try, Arch boots to weird messages about "can't find DMAR" for various devices and finally hangs, unable to find the hard drive.

The kernel line I am (trying to, actually) using is:
linux    /vmlinuz-linux root=UUID=89c6d568-9a2f-4de8-ad98-5f210c6a7163 rw  quiet iommu=1 intel_iommu=on pci-stub.ids=10de:11c6,10de:0b0e

I can't seem to get past this. Without intel_iommu=on, I get
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.0/iommu_group/devices: No such file or directory
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.1/iommu_group/devices: No such file or directory

when trying to run vfio-bind.

Any help will be much appreciated, so thanks in advance.

IOMMU support is required for VFIO.  Make sure VT-d support is enabled in your BIOS, update to a newer BIOS, and possibly contact your motherboard vendor about fixing their VT-d support.  Perhaps others with this board can share their experience.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#537 2013-10-08 22:04:13

tarcisioe
Member
Registered: 2013-10-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

IOMMU support is required for VFIO.  Make sure VT-d support is enabled in your BIOS, update to a newer BIOS, and possibly contact your motherboard vendor about fixing their VT-d support.  Perhaps others with this board can share their experience.

Indeed, updating to the newest BIOS (F9) made me able to boot with intel_iommu=on. For anyone who may care, the option "M.I.T. Intelligent Tweaker > Miscellaneous Settings > Virtualization technology" apparently enables VT-x as well as VT-d on the newer Gigabyte BIOS for GA-P67A-UD3-B3. There is no option called "VT-d" as it was called on F4a.

Now dmesg tells me that Intel IOMMU is on, but I can't seem to get the iommu_group directories under /sys/bus/pci/devices/0000:01:00.1 yet. I can't load vfio_iommu_type1 directly, but I'll try to compile it built-in just in case.

I'm not with my PC right now but I can post any other needed information as soon as I get home.

Thanks smile

Offline

#538 2013-10-08 22:14:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tarcisioe wrote:
aw wrote:

IOMMU support is required for VFIO.  Make sure VT-d support is enabled in your BIOS, update to a newer BIOS, and possibly contact your motherboard vendor about fixing their VT-d support.  Perhaps others with this board can share their experience.

Indeed, updating to the newest BIOS (F9) made me able to boot with intel_iommu=on. For anyone who may care, the option "M.I.T. Intelligent Tweaker > Miscellaneous Settings > Virtualization technology" apparently enables VT-x as well as VT-d on the newer Gigabyte BIOS for GA-P67A-UD3-B3. There is no option called "VT-d" as it was called on F4a.

Now dmesg tells me that Intel IOMMU is on, but I can't seem to get the iommu_group directories under /sys/bus/pci/devices/0000:01:00.1 yet. I can't load vfio_iommu_type1 directly, but I'll try to compile it built-in just in case.

I'm not with my PC right now but I can post any other needed information as soon as I get home.

Thanks smile

If there's no iommu_group link or /sys/kernel/iommu_groups/ is not populated, that suggests that intel-iommu isn't really enabled.  Those are provided by the iommu driver, not vfio.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#539 2013-10-09 12:08:28

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry to bother again,

I get output only when I omit -vga none.
Is the card used? Or I use qemu's graphics emulation?

Offline

#540 2013-10-09 19:13:32

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Offline

#541 2013-10-09 19:15:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Sorry to bother again,

I get output only when I omit -vga none.
Is the card used? Or I use qemu's graphics emulation?

Have you tried the lastest kernel rc + alex qemu tree?

aw wrote:

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

Last edited by nbhs (2013-10-09 19:19:08)

Offline

#542 2013-10-09 19:17:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Its on the main guide, you must add a sata/ide/virtio controller then attach a disk to it

Offline

#543 2013-10-09 19:37:51

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Its on the main guide, you must add a sata/ide/virtio controller then attach a disk to it

Ah, now i see it, thanks!

EDIT: I`m afraid I can`t get Windows to install. It stops at "Windows data is being expanded (0%)..." i waited for a while but I cant see any progress... Any ideas?

Last edited by SpacePirate (2013-10-09 20:31:16)

Offline

#544 2013-10-10 03:19:22

tarcisioe
Member
Registered: 2013-10-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

If there's no iommu_group link or /sys/kernel/iommu_groups/ is not populated, that suggests that intel-iommu isn't really enabled.  Those are provided by the iommu driver, not vfio.

Oh, I didn't check /sys/kernel/iommu_groups. It is indeed not populated. How can intel-iommu not be really enabled, though? I even got a bunch of errors when trying to use it with my old BIOS, and they were fixed by updating it to the latest one.

Don't want this second graphics card to be a waste of money, I am positive that my CPU supports IOMMU and was informed by Gigabyte's technical support that my motherboard indeed supports VT-d.

Any suggestions?

Offline

#545 2013-10-10 03:23:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
nbhs wrote:
SpacePirate wrote:

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Its on the main guide, you must add a sata/ide/virtio controller then attach a disk to it

Ah, now i see it, thanks!

EDIT: I`m afraid I can`t get Windows to install. It stops at "Windows data is being expanded (0%)..." i waited for a while but I cant see any progress... Any ideas?

AHCI is currently extremely slow in QEMU, much slower than IDE on 440FX.  You can use pidstat to see if there's actually I/O to the QEMU process.  Suggest waiting it out and switch to virtio once installed (or try to attach IDE or switch to 440FX model).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#546 2013-10-10 03:24:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tarcisioe wrote:
aw wrote:

If there's no iommu_group link or /sys/kernel/iommu_groups/ is not populated, that suggests that intel-iommu isn't really enabled.  Those are provided by the iommu driver, not vfio.

Oh, I didn't check /sys/kernel/iommu_groups. It is indeed not populated. How can intel-iommu not be really enabled, though? I even got a bunch of errors when trying to use it with my old BIOS, and they were fixed by updating it to the latest one.

Don't want this second graphics card to be a waste of money, I am positive that my CPU supports IOMMU and was informed by Gigabyte's technical support that my motherboard indeed supports VT-d.

Any suggestions?

dmesg or it didn't happen.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#547 2013-10-10 04:26:54

tarcisioe
Member
Registered: 2013-10-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

dmesg or it didn't happen.

LOL ok.

[root@jack tarcisioe]# dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled

Anything else I should grep for?

Offline

#548 2013-10-10 06:27:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've uploaded new builds for kernel 3.12-rc4, qemu-git + alex patches, and the lastest seabios, all working fine on my pc, ive also cleaned up the guide a bit to reflect the lastest changes

Offline

#549 2013-10-10 10:49:48

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe you could include a note that the AHCI controller will be slow in the first post. using the IDE controller it works fine for me now.

Also i think the best practice to create qemu images is using qemu-img instead of dd, isnt it? I dont know if there are any differences but so far i have only seen the use of qemu-img in other tutorials/wikis/the documentation.

I now got an Windows installation running but cant install the driver for the 6450 on the guest. When I use the installer I get a BSOD and if i try to use Windows` driver update functionality it detects the GPU but says "Failed to read from device"...

Last edited by SpacePirate (2013-10-10 11:12:57)

Offline

#550 2013-10-10 13:32:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tarcisioe wrote:
aw wrote:

dmesg or it didn't happen.

LOL ok.

[root@jack tarcisioe]# dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled

Anything else I should grep for?

You should have a lot more output than that, for example:

$ dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.020973] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.020977] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.021052] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.414246] IOMMU 0 0xfed90000: using Queued invalidation
[    0.414248] IOMMU 1 0xfed91000: using Queued invalidation
[    0.414251] IOMMU: Setting RMRR:
[    0.414276] IOMMU: Setting identity map for device 0000:00:02.0 [0xdb800000 - 0xdf9fffff]
[    0.415007] IOMMU: Setting identity map for device 0000:00:1d.0 [0xda3ba000 - 0xda3d0fff]
[    0.415044] IOMMU: Setting identity map for device 0000:00:1a.0 [0xda3ba000 - 0xda3d0fff]
[    0.415077] IOMMU: Setting identity map for device 0000:00:14.0 [0xda3ba000 - 0xda3d0fff]
[    0.415095] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.415108] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#551 2013-10-10 13:37:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

I now got an Windows installation running but cant install the driver for the 6450 on the guest. When I use the installer I get a BSOD and if i try to use Windows` driver update functionality it detects the GPU but says "Failed to read from device"...

I moved my 7850 to a different system the other day and hit a BSOD about failing to reset the adapter.  I had to revert to Catalyst 13.4 to get it working.  Catalyst 13.9 was the broken version.  Maybe someone could find 13.5,.6,.7,.8 and figure out which one introduced the break.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#552 2013-10-10 13:46:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I've uploaded new builds for kernel 3.12-rc4, qemu-git + alex patches, and the lastest seabios, all working fine on my pc, ive also cleaned up the guide a bit to reflect the lastest changes

It hasn't landed yet, but we're about to experience a regression in VGA arbiter support for i915.  Apparently making i915 work correctly with the arbiter results in too much of a performance hit for X.  Once 81b5c7bc8de3e6f63419139c2fc91bf81dea8a7d is reverted, IGD will claim all VGA accesses when i915 is loaded on the host.  So, sorry folks, if you have IGD you'll soon need to patch your kernel again.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#553 2013-10-10 14:16:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SpacePirate wrote:

I now got an Windows installation running but cant install the driver for the 6450 on the guest. When I use the installer I get a BSOD and if i try to use Windows` driver update functionality it detects the GPU but says "Failed to read from device"...

I moved my 7850 to a different system the other day and hit a BSOD about failing to reset the adapter.  I had to revert to Catalyst 13.4 to get it working.  Catalyst 13.9 was the broken version.  Maybe someone could find 13.5,.6,.7,.8 and figure out which one introduced the break.

Maybe those were internal releases, 13.4 and 13.9 seems to be the best granularity we have from AMD.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#554 2013-10-10 17:49:33

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SpacePirate wrote:

I now got an Windows installation running but cant install the driver for the 6450 on the guest. When I use the installer I get a BSOD and if i try to use Windows` driver update functionality it detects the GPU but says "Failed to read from device"...

I moved my 7850 to a different system the other day and hit a BSOD about failing to reset the adapter.  I had to revert to Catalyst 13.4 to get it working.  Catalyst 13.9 was the broken version.  Maybe someone could find 13.5,.6,.7,.8 and figure out which one introduced the break.

That didnt work for me. I`ve tried to use Catalyst 13.4, still a BSOD when installing the driver and the "Failed to read from device" error.

Offline

#555 2013-10-11 03:56:08

tarcisioe
Member
Registered: 2013-10-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You should have a lot more output than that, for example:

(...)

Now that seems bad. Wonder if I am missing some software configuration, something in my kernel, or am just stuck with a motherboard that just claims to be VT-d compatible.

Offline

#556 2013-10-12 19:47:19

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

First post here.
I've been following this great topic for a while now and finally had the time to set up a system to play with GPU passthrough again (tried it with Xen before, with mixed results).
I do run into some issues though. When using the vfio-bind script a Call Trace is displayed on screen and in dmesg. This doesn't look good to me, but don't know if I can ignore it. Also, when ignoring the call trace and starting qemu to start installing Windows 7 Home Premium SP1 64 bit, a BSOD appeared.

Compiled kernel 3.12-rc4 with necessary config items, checked dmesg and iommu_groups, blacklisted device drivers, checked BDFs and device id's, set up kernel parameter with pci-stub, compiled Alex' qemu-vfio with the NoSnoop patch, compiled latest seabios. Kernel, qemu-vfio and seabios were compiled from source. The .config for the kernel is based on the one from Debian (3.2.0-4-amd64).
As far as I can tell I did all the necessary steps described here, but might have missed or misunderstood something. If more information is required, please ask. Have logged nearly everything I did. Hope someone can help me out.


System specs:
Motherboard: ASRock Z77 WS , BIOS v1.20 , VT-d enabled in BIOS
CPU: Intel i7 3770
Memory: 16 GB
GPU1: Sapphire AMD/ATI HD6870 1GB
GPU2: Asus GTX460 1GB

Intel GPU is set as primary graphics adapter.

The OS is Debian 7, minimal net-install without a GUI, only SSH and basic system tools options selected. Some extra packages installed for compiling etc.
I manage this system with SSH from another system (Win7/putty).

Kernel line in grub.cfg:

linux   /boot/vmlinuz-3.12.0-rc4-kvm-vfio-gpu01 root=/dev/mapper/vg00-root ro pci-stub.ids=1002:6738,1002:aa88,10de:0e22,10de:0beb,8086:1e2d

The id's are for the HD6870 + audio, GTX460 + audio and an USB 2.0 controller; in that order.

When starting the vfio-bind script (from this topic, without "modprobe vfio-pci" since it is compiled in the kernel), I get a Call Trace on my console and in dmesg when running:
# vfio-bind 0000:04:00.0 0000:04:00.1 0000:00:1a.0

[13211.267950] pcie_pme 0000:00:01.0:pcie01: unloading service driver pcie_pme
[13211.268146] ------------[ cut here ]------------
[13211.268165] WARNING: CPU: 0 PID: 19356 at drivers/pci/pci.c:1430 pci_disable_device+0x4d/0x74()
[13211.268183] Device pcieport
[13211.268183] disabling already-disabled device
[13211.268195] Modules linked in: autofs4 hid_generic usbhid hid nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop x86_pkg_temp_thermal coretemp iTCO_wdt iTCO_vendor_support crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 ablk_helper cryptd lrw gf128mul glue_helper mxm_wmi evdev snd_pcm snd_page_alloc snd_timer snd psmouse soundcore microcode serio_raw pcspkr i2c_i801 i2c_core lpc_ich mfd_core ehci_pci ehci_hcd wmi video acpi_cpufreq button processor thermal_sys ext4 crc16 jbd2 mbcache dm_mod sg sd_mod sr_mod crc_t10dif cdrom crct10dif_common ahci tg3 libahci libata ptp pps_core libphy scsi_mod xhci_hcd usbcore usb_common
[13211.268387] CPU: 0 PID: 19356 Comm: vfio-bind Not tainted 3.12.0-rc4-kvm-vfio-gpu01 #1
[13211.268404] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 WS, BIOS P1.20 08/13/2012
[13211.268422]  0000000000000000 0000000000000009 ffffffff813b19a1 ffff8800ace7dda8
[13211.268455]  ffffffff810693c7 ffff88042bc4c6c0 ffffffff81217da3 ffffffff81664bf8
[13211.268498]  ffff88042eb96000 ffffffff81664b90 ffff88042eb96000 ffffffff81664bf8
[13211.268541] Call Trace:
[13211.268564]  [<ffffffff813b19a1>] ? dump_stack+0x41/0x51
[13211.268590]  [<ffffffff810693c7>] ? warn_slowpath_common+0x78/0x90
[13211.268616]  [<ffffffff81217da3>] ? pci_disable_device+0x4d/0x74
[13211.268642]  [<ffffffff81069477>] ? warn_slowpath_fmt+0x45/0x4a
[13211.268668]  [<ffffffff81217d50>] ? do_pci_disable_device+0x41/0x47
[13211.268694]  [<ffffffff81217da3>] ? pci_disable_device+0x4d/0x74
[13211.268722]  [<ffffffff8121a44e>] ? pci_device_remove+0x38/0x83
[13211.268750]  [<ffffffff812a879f>] ? __device_release_driver+0x82/0xdb
[13211.268778]  [<ffffffff812a8a3f>] ? device_release_driver+0x1a/0x25
[13211.268805]  [<ffffffff812a7eb0>] ? unbind_store+0x59/0x89
[13211.268832]  [<ffffffff8118e435>] ? sysfs_write_file+0xd3/0x109
[13211.268859]  [<ffffffff811395f6>] ? vfs_write+0x9e/0x104
[13211.268883]  [<ffffffff811398ed>] ? SyS_write+0x51/0x85
[13211.268910]  [<ffffffff813ba8e2>] ? system_call_fastpath+0x16/0x1b
[13211.268935] ---[ end trace 1209721fe0f238ac ]---

0000:04:00.0 and 0000:04:00.1 is a GTX460 as second GPU + audio
0000:00:1a.0 is the USB 2.0 controller
When using the script with the HD6870 after a reboot, it results in the same Call Trace.
When executing the script a second time after the Call Trace, it doesn't give the Call Trace again.


Ignoring the Call Trace and trying to execute qemu results in Windows booting up but gives a BSOD when the animated Windows logo appears:
1. "Windows is loading files" appears
2. "Windows is starting", where the animated Windows logo should appear and then,
3. BSOD with PAGE_FAULT_IN_NONPAGED_AREA and STOP error 0x00000050

Qemu command line:

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -nographic \
-smp 2,sockets=1,cores=2,threads=1 -bios /usr/local/bin/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/vg00/win7gtx460.1,cache=none,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/data/win7hp-sp1-x64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-boot dc

Also disabled Intel GPU in BIOS, but that didn't help.

Thanks in advance.


----

Other things I noticed:
1. With the NoSnoop patch I don't get the following message anymore on my Nvidia GPU:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:04:00.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1: Warning, device 0000:04:00.1 does not support reset

2. When compiling Alex' qemu-vfio I got an error:
ERROR: DTC not present
After installing libfdt-dev that error went away and I was able to compile.

3. When starting qemu without disks (like the command line above, but without the last three lines), it stays in the seabios screen because it has nothing to boot from. This looks logical to me.
But when a logical volume is mounted (as I did), it displays a message on the seabios screen:
[ 1033.718172] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
I was under the assumption the VM 'owns' the display and that no OS messages should be displayed there. Maybe it's because "quiet" was removed from the kernel line in grub.cfg .
This could also been seen in the BSOD screen, but in that case, the screen went up one line and nothing was printed. Didn't look any further into it.

Last edited by empie (2013-10-12 22:11:44)

Offline

#557 2013-10-12 20:03:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm too lazy to try to quote the above, so...

The vfio-bind script in this thread is broken, it should not be unbinding devices bound to pcieport.  I can imagine a number of people could be having problems just due to this.  The report above shows a backtrace from pci_disable_device() from this.  We absolutely do not want root ports to be disabled.  Leave them bound to pcieport.  vfio-pci will let you do this.

BSOD - I'm noticing this too and it seems to be something broken in latest qemu/kvm when using "-cpu host".  Try dropping this parameter and just use the QEMU CPU type.

If you get host console messages out to the assigned graphics device, it's because the console driver you're using on the host doesn't make use of the VGA arbiter.  vgacon for example is this way.  It writes to VGA no matter where it's routed.  The device specific drivers generally do make use of the VGA arbiter, but it can be hit or miss there too.  With an i7 I assume your host VGA is IGD, so you should be using the i915 drivers, which as of v3.12-rc4 still uses the VGA arbiter correctly, but see a bit back in the thread about how that's soon to change.

The "does not support reset" messages are gone.  This is part of the new bus reset interface in v3.12.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#558 2013-10-12 20:25:53

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Alex,
Will look into the vfio-bind script and qemu cpu type tomorrow. VGA arbiter is new to me, will take a look at that also.
I did blacklist the i915 driver. Will 'whitelist' i915 again.
Will post my progress here.
Thanks!

Offline

#559 2013-10-13 19:57:16

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Never could get xen working. Thanks to everyone that got this working and the excellent instructions. I spent about two weeks trying this new method for pass through with no luck but with the latest patches recently posted vfio vga-passthrough works perfectly. After reading all the horror stories about ASUS motherboards I thought that I may have a motherboard with a broken bios but apparently ASUS Z87 Pro has a working vt-d option with lastest bios.

ASUS Z87 Pro MotherBoard Bios Version 1405
Intel iGPU primary display
ASUS Radeon HD 6450 --> vga passthrough on VM. Also a nice bonus is
the HDMI audio on the HD 6450 is working great also.

HD 6450 works perfectly with AMD Catalyst 13.11 beta 10/08/2013 release.

Only problem is USB and this is off topic, I don't understand the new USB syntax for qemu. Could someone kindly update the examples for using USB in the first post because those examples don't work with qemu-git version? I have tried passing the usb devices to the VM but this does not work. I had to passthrough the whole USB device but still can't get USB hotplugging to work. Anyone have some updated examples using USB and USB hotplugging?

Lastly what about the iGPU patch for intel iGPU, when should be apply it and what will happen if it is not applied after the patch discussed previously is reverted?

Thanks ;-)

Offline

#560 2013-10-13 20:15:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alphahere wrote:

Never could get xen working. Thanks to everyone that got this working and the excellent instructions. I spent about two weeks trying this new method for pass through with no luck but with the latest patches recently posted vfio vga-passthrough works perfectly. After reading all the horror stories about ASUS motherboards I thought that I may have a motherboard with a broken bios but apparently ASUS Z87 Pro has a working vt-d option with lastest bios.

ASUS Z87 Pro MotherBoard Bios Version 1405
Intel iGPU primary display
ASUS Radeon HD 6450 --> vga passthrough on VM. Also a nice bonus is
the HDMI audio on the HD 6450 is working great also.

HD 6450 works perfectly with AMD Catalyst 13.11 beta 10/08/2013 release.

Awesome, great to have success reports!

alphahere wrote:

Only problem is USB and this is off topic, I don't understand the new USB syntax for qemu. Could someone kindly update the examples for using USB in the first post because those examples don't work with qemu-git version? I have tried passing the usb devices to the VM but this does not work. I had to passthrough the whole USB device but still can't get USB hotplugging to work. Anyone have some updated examples using USB and USB hotplugging?

I usually just pass through specific devices using something like:

-device usb-host,hostbus=3,hostaddr=3

This does not support hotplug.  If you use vfio-pci to attach the USB devices, hotplug should work.  I'd recommend assigning the whole devices, ie. one full set of OHCI/UHCI functions and EHCI functions, mirroring the function layout of the host system.

alphahere wrote:

Lastly what about the iGPU patch for intel iGPU, when should be apply it and what will happen if it is not applied after the patch discussed previously is reverted?

Thanks ;-)

The revert will be in v3.12-rc5, it went in on Friday.  To restore behavior reapply 81b5c7bc8de3e6f63419139c2fc91bf81dea8a7d and 6e1b4fdad5157bb9e88777d525704aba24389bee.  Something like 'git show 81b5c7bc | patch -p1' should do it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#561 2013-10-13 20:44:15

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Got i915 out of the blacklist and got no more OS messages in seabios and BSOD screen.
Took a look into vfio-bind script. Looks like it wants to unbind/bind all the other ID's in /sys/bus/pci/devices/0000\:04\:00.0/iommu_group/devices/. Now using only echo statements for the specific ID's. No more Call Trace.
Replacing -cpu host with -cpu SandyBridge got the VM started.

Offline

#562 2013-10-13 21:12:25

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone experienced this behaviour?

I tried to add the devices now manually to the vfio-pci driver, as shown in the vfio documentation.
I dont get the vfio-pci warning anymore, but this in dmesg:

[ 1344.776333] pcieport 0000:00:15.0: driver skip pci_set_master, fix it! [ 1344.776340] vfio-pci 0000:06:00.0: enabling device (0000 -> 0003) [ 1344.846595] vfio-pci 0000:06:00.1: enabling device (0000 -> 0002)

When i turn on the VM my screen goes black, when switching to tty2 and back again everything works fine again. except that i dont get any output on the screen attached to the passed-through GPU.
I cant seem to get any progress on that, any ideas?

Offline

#563 2013-10-13 21:29:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alphahere wrote:

Only problem is USB and this is off topic, I don't understand the new USB syntax for qemu. Could someone kindly update the examples for using USB in the first post because those examples don't work with qemu-git version? I have tried passing the usb devices to the VM but this does not work. I had to passthrough the whole USB device but still can't get USB hotplugging to work. Anyone have some updated examples using USB and USB hotplugging?

Works for me using qemu-git.

This is how i do it:

lsusb

Bus 007 Device 002: ID 1532:0039 Razer USA, Ltd 

then i add this line to qemu

-usb -usbdevice host:1532:0039

Offline

#564 2013-10-13 21:40:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The vfio-bind script in this thread is broken, it should not be unbinding devices bound to pcieport.

empie wrote:

Got i915 out of the blacklist and got no more OS messages in seabios and BSOD screen.
Took a look into vfio-bind script. Looks like it wants to unbind/bind all the other ID's in /sys/bus/pci/devices/0000\:04\:00.0/iommu_group/devices/. Now using only echo statements for the specific ID's. No more Call Trace.

I fixed this on the first post, thanks

Last edited by nbhs (2013-10-13 21:50:27)

Offline

#565 2013-10-14 00:18:45

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oops!!

USB problem was my fault. Sorry about that post. I went back and started checking everything again.

Forgot to compile USB support in qemu.  I setup a test  system to try this out and forgot to install libusb. Stupid mistake. Whatever.

ERROR: User requested feature libusb
       configure was not able to find it

Thanks!

Offline

#566 2013-10-14 02:56:21

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After the compiling USB support in qemu. I started getting BSOD in windows. I decided to go back and double check everything. I tested the following changes three times to verify the causes of the BSOD.

After compiling USB support in qemu immediately had BSOD with "-cpu host" and changing this option to "-cpu qemu64" stopped the BSOD.

Another BSOD was caused by the following line to enable hdmi audio on the radeon video card.
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Changed to the following and sound is working great again without BSOD.
-device vfio-pci,host=01:00.1,bus=pcie.0

Hope this helps someone.

Last edited by alphahere (2013-10-14 04:42:49)

Offline

#567 2013-10-15 06:50:36

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, I'm wanting to get this set up on my rig, but I have an unlocked ivy bridge i5 which doesn't support vt-d, and I have an asus motherboard which isn't supported by linux at all, so I'm going to have to get a new mobo and a new cpu. So first off my question is, should I be looking to go for haswell, or should I stick with ivy bridge or even go for a sandy bridge? which is best supported? Also I need the most powerful cpu that I can get because I also play ps2 games and gc/wii games in an emulator (which is why i got the unlocked cpu to begin with).

So I'm thinking I'll go for an i7 haswell or an i7 ivy bridge basic.


Anyways I have a few other questions aswell. Do you need to have a seperate keyboard for the vm or can you just seemlessly run through the 2 vm's using the same keyboard/mouse?


And I only have 1 screen as of right now. Is it possible to have the vm output video through the dom0's graphics card but use the domu card for the processing? Or is there a practical way to vnc or something of the like so that you can have a single monitor setup?

For my graphics cards I'm going to be using an nvidia gtx580 and a fairly new radeon graphics card. the nvidia card is the better card so I would prefer to have that running in the domu and the radeon running in the dom0, but would it be better to have it running the other way around? sorry for the scrub questions, but I just want to get some answers before i start sinking cash into this.

also what z77 mobo's do you guys recommend? and how much power should my psu be able to put out to support this system?

edit: also can you control the audio of the vm via pulse audio volume control? Currently I'm controling the volume of all my software using that.

Last edited by risho (2013-10-15 08:48:19)

Offline

#568 2013-10-15 09:15:30

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

So, I'm wanting to get this set up on my rig, but I have an unlocked ivy bridge i5 which doesn't support vt-d, and I have an asus motherboard which isn't supported by linux at all, so I'm going to have to get a new mobo and a new cpu. So first off my question is, should I be looking to go for haswell, or should I stick with ivy bridge or even go for a sandy bridge? which is best supported? Also I need the most powerful cpu that I can get because I also play ps2 games and gc/wii games in an emulator (which is why i got the unlocked cpu to begin with).

So I'm thinking I'll go for an i7 haswell or an i7 ivy bridge basic.


Anyways I have a few other questions aswell. Do you need to have a seperate keyboard for the vm or can you just seemlessly run through the 2 vm's using the same keyboard/mouse?


And I only have 1 screen as of right now. Is it possible to have the vm output video through the dom0's graphics card but use the domu card for the processing? Or is there a practical way to vnc or something of the like so that you can have a single monitor setup?

For my graphics cards I'm going to be using an nvidia gtx580 and a fairly new radeon graphics card. the nvidia card is the better card so I would prefer to have that running in the domu and the radeon running in the dom0, but would it be better to have it running the other way around? sorry for the scrub questions, but I just want to get some answers before i start sinking cash into this.

also what z77 mobo's do you guys recommend? and how much power should my psu be able to put out to support this system?

edit: also can you control the audio of the vm via pulse audio volume control? Currently I'm controling the volume of all my software using that.


Afaik you cannot pass the Output of one GPU through the other. I guess it would be possible to achieve that by streaming the Image over IP to the Host, but i guess that isnt a good idea for gaming.

For the input question you can pass the Controller which controlls the Mouse/Keyboard to the VM or use Synergy, which nbhs mentioned a few pages earlier in this topic.

I dont think there is any difference in hardware support for VT-d for the different Core-i generations.

Offline

#569 2013-10-15 09:32:53

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

is there any preference between intel and amd?

I'm actually considering getting an AMD FX-8350, since it looks like it can be oc'ed and still supports hardware passthrough.

another question I have is if I'm going to be using 2 dedicated graphics cards do i need to make sure that i get a motherboard that supports running 2 pci express slots at x16 concurantly?

can you guys recommend a good amd mobo and a good intel mobo that will support 2 powerful graphics cards running simultaneously?

edit:

I've done some reading and it really seems like amd pales in comparison for what I'm trying to do. Are there any unlocked intel cpu's that support vt-d?

edit2:

It looks like the ivy bridge e editions support vt-d including the unlocked versions.

can anyone confirm or deny this?


http://www.newegg.com/Product/Product.a … 6819116939 does this support vt-d?

Last edited by risho (2013-10-15 10:24:04)

Offline

#570 2013-10-15 11:17:25

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The CPU you linked supports VT-d, but it costs three times the price of a FX-8350 smile

Last edited by SpacePirate (2013-10-15 11:17:39)

Offline

#571 2013-10-15 11:31:18

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just tried to pass through something else than my GPU (USB3.0 Controller) - that works perfectly fine. So I the vfio works, with script, iommu etc. Yet I cannot produce any output from the passed through GPU.

EDIT: Some progress! When i start the VM using the test-command from the first post, the screen on the passed-through GPU actually turns on and the fans of the graphicscard! Unfortunatly it doesnt go any further than that. Also this output on dmesg on the host (it should start to get interesting from line 1050 on). When i dont shut down the qemu process, this crashes my host, it goes on with these statements:

AMD-Vi: Completion-Wait loop timed out

and

AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=06:00.0 address=0x0000000438544b90]

and the input devices start to lag behind, and at some point the screen starts only showing dmesg output....

Last edited by SpacePirate (2013-10-15 22:02:09)

Offline

#572 2013-10-16 07:16:51

1Samildanach
Member
Registered: 2012-06-18
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

And I only have 1 screen as of right now. Is it possible to have the vm output video through the dom0's graphics card but use the domu card for the processing? Or is there a practical way to vnc or something of the like so that you can have a single monitor setup?

One option is to use a KVM switch. The short version is that you plug it into the appropriate places, then press a button to switch your keyboard, monitor, and mouse between the two. I have been using, and satisfied with, an Aten CS682, but there are other models and brands out there which may better suite your individual needs.

risho wrote:

edit: also can you control the audio of the vm via pulse audio volume control? Currently I'm controling the volume of all my software using that.

Personally, I've opted for getting a dedicated sound card and passing the onboard through to the VM, and having audio hooked into the KVM switch along with all the rest.

Offline

#573 2013-10-16 08:02:58

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

1Samildanach wrote:

One option is to use a KVM switch. The short version is that you plug it into the appropriate places, then press a button to switch your keyboard, monitor, and mouse between the two. I have been using, and satisfied with, an Aten CS682, but there are other models and brands out there which may better suite your individual needs.

i always wondered about something like that. nice to know that's what it's called. i'll definately consider that thanks!

Personally, I've opted for getting a dedicated sound card and passing the onboard through to the VM, and having audio hooked into the KVM switch along with all the rest.


well i use a headset so and i want to be able to seemlessly switch between vm and host while using the same audio. like having a game using sound in the vm and talking on skype on the host simultaneously.

Offline

#574 2013-10-16 08:52:21

1Samildanach
Member
Registered: 2012-06-18
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

i always wondered about something like that. nice to know that's what it's called. i'll definately consider that thanks!

No worries smile. It does make for a mess of cables, but that's a minor quibble, and the advantages (no mess of software, for one thing) far outweighs it.

risho wrote:

well i use a headset so and i want to be able to seemlessly switch between vm and host while using the same audio. like having a game using sound in the vm and talking on skype on the host simultaneously.

Ach, that does complicate matters. My switch allows one to switch just KVM or audio focus (so have audio coming from one machine while doing stuff to the other), but that still doesn't do you any good if want sound from both at once. However, it can certainly be done with software; all you need is something that can stream your audio output from Windows over the (virtual) network to Arch. I just decided it was more complication than I could be bothered with when I already had a switch tongue.

The obvious 'simple' solution for Skype is to use the Windows client, but I imagine you have other examples in mind as well.

Offline

#575 2013-10-16 09:30:54

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so, i've decided I would really like to get an intel 4930k, and I even found a motherboard that I was looking at getting. it's EVGA X79 Dark. I wasn't sure if it would support vt-d and linux so I contacted their customer service and asked them directly. they emailed me back:

Hi ****,

The motherboard will support Intel Visualization Technology. Unfortunately we do not support or have any drivers for Linux and only have drivers and support for Windows.

Regards,
EVGA

I asked them SPECIFICALLY if it supported vt-d, and that seems to say that it does, but it doesn't have drivers to support linux. does that mean it won't work or does that just mean it doesn't support it actively and that the kernel would have to support it itself?

anyone have any idea, and if it won't work does anyone know a 2011 slot mobo that will work and supports ivy bridge e(not just sandy bridge)?






No worries smile. It does make for a mess of cables, but that's a minor quibble, and the advantages (no mess of software, for one thing) far outweighs it.

i'm sure all of us are intimately firmillial with a mess of cables hahaha. my cable set it would probably scare some people away.

Last edited by risho (2013-10-16 09:32:12)

Offline

#576 2013-10-16 10:26:55

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

so, i've decided I would really like to get an intel 4930k, and I even found a motherboard that I was looking at getting. it's EVGA X79 Dark. I wasn't sure if it would support vt-d and linux so I contacted their customer service and asked them directly. they emailed me back:

Hi ****,

The motherboard will support Intel Visualization Technology. Unfortunately we do not support or have any drivers for Linux and only have drivers and support for Windows.

Regards,
EVGA

I asked them SPECIFICALLY if it supported vt-d, and that seems to say that it does, but it doesn't have drivers to support linux. does that mean it won't work or does that just mean it doesn't support it actively and that the kernel would have to support it itself?

anyone have any idea, and if it won't work does anyone know a 2011 slot mobo that will work and supports ivy bridge e(not just sandy bridge)?

I dont think you will find a vendor who supports Linux officially. The drivers are available in the Linux kernel so you shouldnt have any trouble.

Offline

#577 2013-10-16 11:03:40

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

interestingly the person called it "visualization" technology... makes it seem a little off.. can I really trust this guy? i know people can easily make typoes especially when you're doing something as monotonous as answering customer service emails... but still.

the whole point of me buying the mobo is solely based aroundit supporting vt-d, can anyone confirm that this motherboard supports vt-d? it's also possible they are assuming i meant vt-x or read it as vt-x when i tried to make it absolutely clear that i was asking specifically about vt-d.

Offline

#578 2013-10-16 11:37:43

1Samildanach
Member
Registered: 2012-06-18
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

interestingly the person called it "visualization" technology... makes it seem a little off.. can I really trust this guy? i know people can easily make typoes especially when you're doing something as monotonous as answering customer service emails... but still.

the whole point of me buying the mobo is solely based aroundit supporting vt-d, can anyone confirm that this motherboard supports vt-d? it's also possible they are assuming i meant vt-x or read it as vt-x when i tried to make it absolutely clear that i was asking specifically about vt-d.

Could throw another email at them saying that you assume "visualization" was a typo, but want to double check before you cough up $400 because IOMMU is a crucial requirement for you (and use the term "IOMMU"; if they don't know what it is they'll hopefully spend the 10 secs to check what it is, thereby creating an opportunity for error correction if they've misread/understood you).

It'll be annoying if you have to send it back, but if you have them on record specifically saying it'll do the job then at least getting a refund should be a no brainer.

EDIT: Oh, and I recall MSI being the most helpful/responsive when I was checking for MB compatibility ages ago (it was unfortunate that their products didn't quite tick other boxes, I felt kinda sad that I didn't give them any business), followed closely by Gigabyte (the brand I went with in the end). I couldn't find any way to even contact ASrock, and I ended up contacting ASUS by telephone yikes... and didn't end up with anything useful from them 'cause the bloke couldn't find anything about vt-d. I don't think I tried any other  manufacturers.

So you're not alone in having trouble getting reliable information even from the source. But if the EVGA X79 doesn't work out, I recommend just emailing a few manufacturers and asking for a list of models that support vt-d and the socket type you want, then cross anything unsatisfactory off the list and seeing what you have left.

[/ramble]

Last edited by 1Samildanach (2013-10-16 11:58:40)

Offline

#579 2013-10-16 16:15:06

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!

I'm a long-time secondary-passthrough user who is fed up with fragility of this setup. Any time the gpu hangs it takes all the host down with itself. And the gpu will hang at least once in a month, you can bet on it. Besides, even though device removal in the guest helps a bit, every VM reboot is still risky with a good chance of something happening that will require the host reboot.

Long story short, today I've tried vfio-vga. Thanks everybody who contributed to this thread, it's really helpful! But... has anybody had the 'gpu reset timeout' BSOD in Win7?

I'm using:
kernel-3.12.0-rc5 + the vfio patch from rc6 + reapplied vgaarb patches.
qemu-git-1680d4857 (the latest at this moment) + Alex' nosnoop patch. The other patches are already in upstream.
seabios-1.7.3.2 (since there are no official builds yet, it's my own build with gcc-4.8.1).
Xeon E3-1245v2
SuperMicro X9SAE-V
Intel P4000 (host)
Radeon HD7750 (guest)

Binding stuff with:

echo "1002 683f" > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind

^ the same for hdmi audio, and usb.

Command line:

qemu-kvm \ -machine q35,accel=kvm \ -cpu SandyBridge,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -smp 4,cores=4,threads=1 \ -m 4G -mem-prealloc -mem-path /hugepages \ -boot order=c \ -device piix4-ide,bus=pcie.0 \ -drive file=/dev/vg0/gpu-snap,id=disk,format=raw,discard=unmap -device ide-hd,bus=ide.0,drive=disk \ -drive file=/vms/drivers/virtio-win-0.1-65.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -display none -vga none \ -device virtio-net,netdev=t10,mac=52:54:00:05:cd:79 \ -netdev tap,id=t10,vhost=on,ifname=tap10,script=no,downscript=no \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1a.0 \ -name "win7-gpu"

Completely new Windows 7 x64 install.

VGA functions work. I've installed Win7 using the output of the gpu, but once I install the gpu drivers Windows BSODs right after the "Starting Windows" screen. It says "device reset timeout." This happens with Catalyst 13.4 and 13.9. With secondary passthrough I use Catalyst 12.10; on vfio-vga setup, 12.10 just freezes in the "Starting Windows" screen (no BSOD, just the VM freeze). Has anybody encountred such issues? Is there any way to fix them?

P.S. Tried everything I could with VGA output to force host freeze, but it didn't. Impressive!

P.P.S. Looks like many people have stuttering audio in VMs, I've never had such an issue. I use the audio output the gpu provides over HDMI. The VM has no emulated audio whatsoever. So if your setup allows HDMI audio, just use it. smile

Offline

#580 2013-10-16 16:30:16

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I had a similar problem. Change this line below
        -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
TO THIS:
        -device vfio-pci,host=01:00.1,bus=pcie.0 \

Changing the above line stopped BSOD's or windows crashes on boot. See page 23 for my post for more details and another cause of BSOD's.

Hope this helps.

Last edited by alphahere (2013-10-16 17:39:27)

Offline

#581 2013-10-16 17:03:31

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How could I miss it?! So embarrassing. :\ Anyway, thank you very much! Everything works flawlessly now!

Even more, Windows Experice Score is slightly higher now. In previous setup: 7.4 (cpu), 7.6(mem), 7.2(gpu). Now: 7.5,7.9,7.5 on the same host.  I guess it's thanks to the new kernel, qemu, and addition of hv_apicv and hv_spinlocks.

Offline

#582 2013-10-16 23:14:43

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I try to passthrough one of my USB Controllers, qemu fails with

qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: vfio: error, group 7 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: vfio: failed to get group 7 qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: Device 'vfio-pci' could not be initialized

Pretty sure this has something to do with my huge iommu group. Can I safely apply the acs_override patch to the 3.12 kernel?

Offline

#583 2013-10-20 18:41:25

cjn
Member
Registered: 2013-10-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I'm trying to pass-through a Radeon HD 3870 but don't have much luck doing so. I'm using

Linux 3.12.0-rc6
QEMU head as of today (includes SeaBIOS) + nosnoop patch
ASRock 990FX Extreme3
AMD FX-8350
GeForce 210 (host)
Radeon HD 3870 (guest)

I'm currently running on console, no frame buffer device drivers loaded. I start QEMU like this

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/local/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/dev/shm/windows.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/root/en_windows_7_home_premium_with_sp1_x64_dvd_u_676549.iso,id=isocd \ -device ide-cd,bus=ahci.1,drive=isocd \ -nographic -usb -usbdevice host:04fe:0006 -usbdevice host:046d:c05b

The passed USB devices are my keyboard and my mouse.

What happens is that the Windows installer BSODs with a "STOP: 0x0000007E" error (with a reference to "atikmdag.sys") after "Completing installation", at which point I can mostly just SSH to the host to shut it down in an orderly fashion. Killing the QEMU process does not restore keyboard/mouse functionality to the host.

Of note is that most things seem to work if I replace the Radeon HD 3870 with a Radeon HD 5450. The Windows installation completes, as does the subsequent Catalyst installation (of version 13.9). Issues remain with guest shutdown/reboot but these can probably be overcome.

Anyway, it is the Radeon HD 3870 I want in my Windows box. What can I do to debug this problem?

Offline

#584 2013-10-21 06:08:28

1Samildanach
Member
Registered: 2012-06-18
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cjn wrote:

Anyway, it is the Radeon HD 3870 I want in my Windows box. What can I do to debug this problem?

As I recall, the card must support Function Level Reset. Given that you've got an older card there, it seems a good place to start. Post any output from this (replace the domain:bus:slot as appropriate):

su -c 'lspci -vvs 02:00.0 | grep FLR'

If that comes up with nothing, the card is probably incapable of being passed through (:(). If you want to double check, drop the grep pipe, post the output, and we can take a look smile.

Last edited by 1Samildanach (2013-10-21 06:09:35)

Offline

#585 2013-10-21 08:30:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, I have yet to see a VGA card that does support flr.  We're probably missing quirks for older radeon cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#586 2013-10-21 08:54:20

1Samildanach
Member
Registered: 2012-06-18
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hmm, I have yet to see a VGA card that does support flr.  We're probably missing quirks for older radeon cards.

It was just something I recall from when I was reading up on stuff ages ago that would be easy to check, though it was probably from Xen documentation (this was back when passthrough only really existed in Xen).

Offline

#587 2013-10-21 16:49:46

cjn
Member
Registered: 2013-10-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

1Samildanach wrote:

As I recall, the card must support Function Level Reset.

No FLR it seems:

# lspci -vvs 04:00.0 | grep FLR ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
aw wrote:

We're probably missing quirks for older radeon cards.

Can any logging be enabled in QEMU (or a kernel module?) that will help identify the quirk required for my card?

Otherwise, I'm quite prepared to test patches. :-)

Offline

#588 2013-10-21 20:03:34

pablocrossa
Member
Registered: 2013-10-21
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there

I've been following this thread for a while and now I finally started putting things together. My current setup:
Laptop
i7-4800MQ (VT-d support and working, passthrough of most devices works)
Intel HD IGP (Whichever comes with that CPU)
Nvidia 770m GTX in Optimus setup

I believe the rest of the specifications are irrelevant; however, it is important to understand what Optimus does behind the scenes:
Intel IGP is connected to Display + all outputs
Nvidia GPU is connected to nowhere
Nvidia have a software solution to copy the 'framebuffer' from the Nvidia card onto the Intel

I am also on Debian, not Arch; however I am using the kernel + qemu + seabios from here.

Now, I have correctly enabled IOMMU and am using PCI-stub correctly; no nouveau or i915 drivers are loaded. I have created a Windows 8 VM that boots with the following command line:
qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu Haswell \
-smp 4,sockets=1,cores=2,threads=2 \
-bios /usr/share/qemu/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/root/hd.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/mnt/root/win8.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-boot dc \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0 \ #Nvidia
-vnc 0.0.0.0:1,1234 \
-usb -usbdevice tablet \
-vga vmware \ #the same happens with Cirrus
-device vfio-pci,host=00:1b.0,bus=pcie.0 \ #SoundCard
-device vfio-pci,host=00:03.0,bus=pcie.0 \ #Intel HD audio controller
-device vfio-pci,host=00:02.0,bus=pcie.0 #Intel IGP

If I comment out the last 2 lines the VM works fine (the Nvidia with newest drivers is registered under the device manager as working)(EDIT: I do this through VNC) and the soundcard works (Windows 'bings' when you adjust the sound level). If I don't comment the last two lines like above Windows boots and then everything freezes except the cursor (no way to click on the Metro interface). Then I believe memory corruption happens as sometimes I can Ctrl+C the Qemu process, other times not; after Ctrl+C sometimes if I try launching software an error comes up about an incorrect reference to a library (that doesn't come up before launching the VM). If I type poweroff (I do all of this through SSH as the screen powers off when I pass the IGP to the VM) sometimes it outputs that the system is halting now but doesn't halt after waiting for a while or it just returns without outputting anything (and doesn't halt). I've been out of internet for the last week so I have some Qs I still haven't been able to read from the source (will get onto it ASAP):

What exactly does the x-vga parameter do? Does it signal SeaBios which GPU to use as primary? I have tried every combination and it crashes.
Why do we need the "-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \" line? Can we not pass the GPU/IGP to pcie.0 directly?
I believe the memory corruption might be hapenning because the Linux kernel doesn't give up the VGA memory or does BIOS calls to the VGA BIOS while the VM initializes it, making memory go everywhere from both hosts, causing the lockups and messups (this is only an idea); could it be so? Can KVM do a 'headless' passthrough? What other causes could the memory corruption have?
Why aren't Intel IGPs working here? Why do they work with Xen in headless mode? I just want to understand it to see where to look and why this happens.

Thanks for all and I hope you can help, I'm really close to having a great setup smile

Last edited by pablocrossa (2013-10-21 20:04:54)

Offline

#589 2013-10-23 17:59:18

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

When I try to passthrough one of my USB Controllers, qemu fails with

qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: vfio: error, group 7 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: vfio: failed to get group 7 qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=pcie.0: Device 'vfio-pci' could not be initialized

Pretty sure this has something to do with my huge iommu group. Can I safely apply the acs_override patch to the 3.12 kernel?

I have the same issue and I get no success  with applying acs_override patch to the 3.12 kernel.

Offline

#590 2013-10-23 18:02:54

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could anyone please provide qemu-git PKGBUILD with alex qemu patches or alex qemu tree + NoSnoop?
Thank you in advance!

Last edited by myweb (2013-10-23 18:06:06)

Offline

#591 2013-10-23 19:25:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Could anyone please provide qemu-git PKGBUILD with alex qemu patches or alex qemu tree + NoSnoop?
Thank you in advance!

qemu-git.tar.gz

Offline

#592 2013-10-23 19:33:26

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:

Could anyone please provide qemu-git PKGBUILD with alex qemu patches or alex qemu tree + NoSnoop?
Thank you in advance!

qemu-git.tar.gz

Thank you!
Is it ok that git://git.qemu.org/qemu.git is used instead of git://github.com/awilliam/qemu-vfio.git tags/vfio-pci-for-qemu-20131003.0 ?

Offline

#593 2013-10-23 19:44:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

Could anyone please provide qemu-git PKGBUILD with alex qemu patches or alex qemu tree + NoSnoop?
Thank you in advance!

qemu-git.tar.gz

Thank you!
Is it ok that git://git.qemu.org/qemu.git is used instead of git://github.com/awilliam/qemu-vfio.git tags/vfio-pci-for-qemu-20131003.0 ?

Alex patches have been merged upstream except the nosnoop patch, yes its ok smile

Last edited by nbhs (2013-10-23 20:00:11)

Offline

#594 2013-10-23 20:09:02

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you!
And one more question: could you please help to apply acs_override patch to the 3.12 kernel?

Offline

#595 2013-10-23 20:24:37

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Thank you!
And one more question: could you please help to apply acs_override patch to the 3.12 kernel?

linux-mainline.tar.gz

Offline

#596 2013-10-23 20:38:19

chUpa
Member
From: Chelyabinsk, Russia
Registered: 2013-09-17
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

II found a solution, crossfire work in KVM. On x2 AMD Radeon HD 7970.

Offline

#597 2013-10-23 20:39:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chUpa wrote:

II found a solution, crossfire work in KVM. On x2 AMD Radeon HD 7970.

Interesting, could you share the solution?

Offline

#598 2013-10-23 21:16:45

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

warning: cannot resolve "libcacard-git", a dependency of "qemu-git"
:: The following package cannot be upgraded due to unresolvable dependencies:
      qemu-git

Aur does not provide libcacard-git

Last edited by myweb (2013-10-23 21:17:08)

Offline

#599 2013-10-23 21:18:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

warning: cannot resolve "libcacard-git", a dependency of "qemu-git"
:: The following package cannot be upgraded due to unresolvable dependencies:
      qemu-git

Aur does not provide libcacard-git

The qemu-git PKGBUILD builds 2 packages, qemu-git and libcacard-git, it should be built on the same folder as qemu-git, so you should install it first then qemu

Last edited by nbhs (2013-10-23 21:19:25)

Offline

#600 2013-10-23 21:35:25

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you and I am sorry for my inattention.
I have started VM via command bellow, but get black monitor - no signal... could you please help to investigate the issue?

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  -vga none \
-smp 1,sockets=1,cores=2,threads=1 \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-drive file=/run/media/abrilevskiy/CopiedData/Soft/saucy-desktop-amd64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/home/abrilevskiy/windows.img \
-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

I have tried:
modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0
but it did not help


dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.080035] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.080040] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.080110] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.366619] IOMMU 0 0xfed90000: using Queued invalidation
[    0.366620] IOMMU 1 0xfed91000: using Queued invalidation
[    0.366621] IOMMU: Setting RMRR:
[    0.366629] IOMMU: Setting identity map for device 0000:00:02.0 [0x9f800000 - 0xaf9fffff]
[    0.367753] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9d5e6000 - 0x9d617fff]
[    0.367773] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9d5e6000 - 0x9d617fff]
[    0.367789] IOMMU: Setting identity map for device 0000:00:14.0 [0x9d5e6000 - 0x9d617fff]
[    0.367800] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.367807] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]


Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on,igfx_on,pass-through pcie_acs_override=downstream quiet

Last edited by myweb (2013-10-23 21:42:36)

Offline

#601 2013-10-23 21:42:45

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Thank you and I am sorry for my inattention.
I have started VM via command bellow, but get black monitor - no signal... could you please help to investigate the issue?

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  -vga none \
-smp 1,sockets=1,cores=2,threads=1 \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-drive file=/run/media/abrilevskiy/CopiedData/Soft/saucy-desktop-amd64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/home/abrilevskiy/windows.img \
-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on

I have tried:
modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0
but it did not help

post your hardware specs, and any message you get.

Offline

#602 2013-10-23 21:46:39

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i5-3470
AsRock Z77 pro4
Nvidia GT640 - Grid K1 (01:00.0)
ATI HD 7750 (02:00.0)

[ 0.368249] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.368333] ------------[ cut here ]------------ [ 0.368338] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90() [ 0.368339] Modules linked in: [ 0.368341] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.12.0-1-mainline #1 [ 0.368342] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Pro4, BIOS P1.70 01/17/2013 [ 0.368344] 0000000000000009 ffff880243811d20 ffffffff814eb993 0000000000000000 [ 0.368346] ffff880243811d58 ffffffff81061b5d ffff880243560098 ffff880243560000 [ 0.368347] ffff880243560098 0000000000000000 ffff880246c097e0 ffff880243811d68 [ 0.368349] Call Trace: [ 0.368354] [<ffffffff814eb993>] dump_stack+0x54/0x8d [ 0.368357] [<ffffffff81061b5d>] warn_slowpath_common+0x7d/0xa0 [ 0.368359] [<ffffffff81061c3a>] warn_slowpath_null+0x1a/0x20 [ 0.368361] [<ffffffff812c3e47>] pci_find_upstream_pcie_bridge+0x77/0x90 [ 0.368364] [<ffffffff813d9d8d>] intel_iommu_add_device+0x4d/0x220 [ 0.368366] [<ffffffff813d0fb0>] ? bus_set_iommu+0x50/0x50 [ 0.368368] [<ffffffff813d0fda>] add_iommu_group+0x2a/0x50 [ 0.368371] [<ffffffff81385c93>] bus_for_each_dev+0x63/0xa0 [ 0.368372] [<ffffffff813d0fa8>] bus_set_iommu+0x48/0x50 [ 0.368375] [<ffffffff819279bc>] intel_iommu_init+0xaad/0xba2 [ 0.368377] [<ffffffff810523e5>] ? free_init_pages+0x75/0xa0 [ 0.368380] [<ffffffff818e2156>] ? maybe_link.part.1+0x10f/0x10f [ 0.368382] [<ffffffff818ea9a3>] ? memblock_find_dma_reserve+0x124/0x124 [ 0.368384] [<ffffffff818ea9b5>] pci_iommu_init+0x12/0x3c [ 0.368386] [<ffffffff810020fa>] do_one_initcall+0xfa/0x1b0 [ 0.368389] [<ffffffff81081f05>] ? parse_args+0x225/0x3f0 [ 0.368391] [<ffffffff818e1067>] kernel_init_freeable+0x15a/0x1e0 [ 0.368393] [<ffffffff818e089a>] ? do_early_param+0x88/0x88 [ 0.368396] [<ffffffff814de1a0>] ? rest_init+0x90/0x90 [ 0.368398] [<ffffffff814de1ae>] kernel_init+0xe/0x190 [ 0.368400] [<ffffffff814fa4bc>] ret_from_fork+0x7c/0xb0 [ 0.368402] [<ffffffff814de1a0>] ? rest_init+0x90/0x90 [ 0.368405] ---[ end trace 8e9a9c119d6070f7 ]---
sudo journalctl --no-pager | grep vfio Oct 23 00:41:47 abrilevskiy-pc.lan sudo[1078]: root : TTY=pts/1 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:01:00.0 0000:01:00.1 Oct 23 00:41:49 abrilevskiy-pc.lan sudo[1094]: root : TTY=pts/1 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:02:00.0 0000:02:00.1 Oct 23 00:43:38 abrilevskiy-pc.lan sudo[1168]: root : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:00:1a.0 Oct 23 00:43:42 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 00:44:02 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 00:44:02 abrilevskiy-pc.lan kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Oct 23 00:44:02 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 00:54:58 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 00:54:58 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 00:55:29 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 00:55:29 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 00:56:23 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 00:56:23 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 01:04:07 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:05:13 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:05:57 abrilevskiy-pc.lan kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:09:36 abrilevskiy-pc.lan sudo[1671]: abrilevskiy : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:02:00.0 0000:02:00.1 Oct 23 01:09:36 abrilevskiy-pc.lan kernel: usb_storage vfio_pci vfio_iommu_type1 vfio fuse isl6423 stv6110x stv090x x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support microcode joydev btusb bluetooth rfkill psmouse pcspkr serio_raw budget budget_core ttpci_eeprom saa7146 snd_hda_codec_hdmi dvb_core r8169 mii snd_hda_codec_realtek i915 snd_hda_intel snd_hda_codec intel_agp intel_gtt snd_hwdep snd_pcm drm_kms_helper video snd_page_alloc i2c_i801 drm snd_timer acpi_cpufreq snd button evdev mei_me i2c_algo_bit processor mei soundcore i2c_core shpchp lpc_ich hid_generic ext4 crc16 mbcache jbd2 usbhid hid sd_mod ahci libahci libata scsi_mod ehci_pci ehci_hcd xhci_hcd usbcore usb_common Oct 23 01:09:36 abrilevskiy-pc.lan kernel: Oct 23 01:09:36 abrilevskiy-pc.lan kernel: CPU: 2 PID: 1672 Comm: vfio-bind Tainted: G W 3.12.0-1-mainline #1 Oct 23 01:09:43 localhost.localdomain sudo[1767]: root : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:02:00.0 0000:02:00.1 Oct 23 01:09:51 localhost.localdomain kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:09:51 localhost.localdomain kernel: vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) Oct 23 01:09:51 localhost.localdomain kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 Oct 23 01:10:19 localhost.localdomain kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:10:19 localhost.localdomain kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 Oct 23 01:10:53 localhost.localdomain kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:10:53 localhost.localdomain kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 Oct 23 01:11:06 localhost.localdomain kernel: vfio-pci 0000:02:00.0: irq 45 for MSI/MSI-X Oct 23 01:11:12 localhost.localdomain kernel: vfio-pci 0000:02:00.1: irq 48 for MSI/MSI-X Oct 23 01:14:12 localhost.localdomain kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:14:12 localhost.localdomain kernel: vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 Oct 23 01:14:21 localhost.localdomain kernel: vfio_cap_init: 0000:00:1a.0 hiding cap 0xa Oct 23 01:14:21 localhost.localdomain kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:20:17 abrilevskiy-pc.lan sudo[1325]: root : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:01:00.0 0000:01:00.1 Oct 23 20:20:21 abrilevskiy-pc.lan sudo[1344]: root : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:02:00.0 0000:02:00.1 Oct 23 20:23:04 abrilevskiy-pc.lan kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Oct 23 20:23:04 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:23:53 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:24:32 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:26:51 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:30:30 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:38:28 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:40:02 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:40:48 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 23 20:41:43 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 24 00:24:33 abrilevskiy-pc.lan sudo[3886]: root : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:01:00.0 0000:01:00.1 Oct 24 00:24:40 abrilevskiy-pc.lan sudo[4036]: root : TTY=pts/2 ; PWD=/home/abrilevskiy ; USER=root ; COMMAND=./vfio-bind 0000:02:00.0 0000:02:00.1 Oct 24 00:25:12 abrilevskiy-pc.lan kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Oct 24 00:25:12 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 24 00:26:17 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 24 00:28:20 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Oct 24 00:32:28 abrilevskiy-pc.lan kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Last edited by myweb (2013-10-23 21:50:16)

Offline

#603 2013-10-23 21:50:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

i5-3470
AsRock Z77 pro4
Nvidia GT640 - Grid K1 (01:00.0)
ATI HD 7750 (02:00.0)

[    0.368249] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.368333] ------------[ cut here ]------------
[    0.368338] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90()
[    0.368339] Modules linked in:
[    0.368341] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.12.0-1-mainline #1
[    0.368342] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Pro4, BIOS P1.70 01/17/2013
[    0.368344]  0000000000000009 ffff880243811d20 ffffffff814eb993 0000000000000000
[    0.368346]  ffff880243811d58 ffffffff81061b5d ffff880243560098 ffff880243560000
[    0.368347]  ffff880243560098 0000000000000000 ffff880246c097e0 ffff880243811d68
[    0.368349] Call Trace:
[    0.368354]  [<ffffffff814eb993>] dump_stack+0x54/0x8d
[    0.368357]  [<ffffffff81061b5d>] warn_slowpath_common+0x7d/0xa0
[    0.368359]  [<ffffffff81061c3a>] warn_slowpath_null+0x1a/0x20
[    0.368361]  [<ffffffff812c3e47>] pci_find_upstream_pcie_bridge+0x77/0x90
[    0.368364]  [<ffffffff813d9d8d>] intel_iommu_add_device+0x4d/0x220
[    0.368366]  [<ffffffff813d0fb0>] ? bus_set_iommu+0x50/0x50
[    0.368368]  [<ffffffff813d0fda>] add_iommu_group+0x2a/0x50
[    0.368371]  [<ffffffff81385c93>] bus_for_each_dev+0x63/0xa0
[    0.368372]  [<ffffffff813d0fa8>] bus_set_iommu+0x48/0x50
[    0.368375]  [<ffffffff819279bc>] intel_iommu_init+0xaad/0xba2
[    0.368377]  [<ffffffff810523e5>] ? free_init_pages+0x75/0xa0
[    0.368380]  [<ffffffff818e2156>] ? maybe_link.part.1+0x10f/0x10f
[    0.368382]  [<ffffffff818ea9a3>] ? memblock_find_dma_reserve+0x124/0x124
[    0.368384]  [<ffffffff818ea9b5>] pci_iommu_init+0x12/0x3c
[    0.368386]  [<ffffffff810020fa>] do_one_initcall+0xfa/0x1b0
[    0.368389]  [<ffffffff81081f05>] ? parse_args+0x225/0x3f0
[    0.368391]  [<ffffffff818e1067>] kernel_init_freeable+0x15a/0x1e0
[    0.368393]  [<ffffffff818e089a>] ? do_early_param+0x88/0x88
[    0.368396]  [<ffffffff814de1a0>] ? rest_init+0x90/0x90
[    0.368398]  [<ffffffff814de1ae>] kernel_init+0xe/0x190
[    0.368400]  [<ffffffff814fa4bc>] ret_from_fork+0x7c/0xb0
[    0.368402]  [<ffffffff814de1a0>] ? rest_init+0x90/0x90
[    0.368405] ---[ end trace 8e9a9c119d6070f7 ]---

myweb wrote:

Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on,igfx_on,pass-through pcie_acs_override=downstream quiet

Boot with intel_iommu=on and remove the other stuff, also nvidia drivers on the host requires a patch

myweb wrote:

...
-device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
...

Are you trying to passthrough your primary card?

Last edited by nbhs (2013-10-23 21:57:30)

Offline

#604 2013-10-23 22:02:38

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Are you trying to passthrough your primary card?

No, my primary card is Integrated video.
I compiled kernel with pcie_acs_override support, booted with:
Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on pcie_acs_override=downstream quiet and got working Live Ubuntu on ATI HD 7750. Also, Windows 8 is installed and working!!! Thank you very much!!! Graphics performance is 7.3 for 3d/2d.

But I can not do this for Nvidia...

Last edited by myweb (2013-10-23 22:49:53)

Offline

#605 2013-10-23 23:15:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:

Are you trying to passthrough your primary card?

No, my primary card is Integrated video.
I compiled kernel with pcie_acs_override support, booted with:
Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on pcie_acs_override=downstream quiet and got working Live Ubuntu on ATI HD 7750. Also, Windows 8 is installed and working!!! Thank you very much!!! Graphics performance is 7.3 for 3d/2d.

But I can not do this for Nvidia...

nice!

Offline

#606 2013-10-23 23:51:40

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally got my setup working! What caused the trouble was apparently a buggy IOMMU implementation on my Mainboard (Gigabyte 970A-UD3). I switched to an Asrock 990FX Extreme3 since that was the only affordable Mainboard which was confirmed to be working in this topic i could find.

Thanks for the help!

Offline

#607 2013-10-24 10:52:40

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:
nbhs wrote:

Are you trying to passthrough your primary card?

No, my primary card is Integrated video.
I compiled kernel with pcie_acs_override support, booted with:
Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on pcie_acs_override=downstream quiet and got working Live Ubuntu on ATI HD 7750. Also, Windows 8 is installed and working!!! Thank you very much!!! Graphics performance is 7.3 for 3d/2d.

But I can not do this for Nvidia...

nice!

Could you please specify what steps I should perform in order to investigate why my Nvidia card does not work?

Offline

#608 2013-10-24 17:01:44

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

No, my primary card is Integrated video.
I compiled kernel with pcie_acs_override support, booted with:
Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on pcie_acs_override=downstream quiet and got working Live Ubuntu on ATI HD 7750. Also, Windows 8 is installed and working!!! Thank you very much!!! Graphics performance is 7.3 for 3d/2d.

But I can not do this for Nvidia...

nice!

Could you please specify what steps I should perform in order to investigate why my Nvidia card does not work?

If you are trying to passthrough your nvidia card (and use your ATI for the host), either try to get rid of the nvidia drivers on the host, or simply pci-stub it.
Or are you trying to pass through both? (that's a bit unclear..)

Offline

#609 2013-10-24 21:57:56

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would like to setup and run simultaneously 2 virtual PC:
#1 Windows with ATI 7750 passthrough - works
#2 Linux with Nvidia GT 640 passthrough - not works
Host Linux should use Intel video card embeded in CPU

with -vga none I always get "no signal" on Nvidia GT 640 passthrough.
If boot to Kubuntu 12.10 live without specified "-vga none" monitor detecs signal and show black screen or some console messages from boot.
Newer version of Kubuntu does not boot to live: KDE crashed somehow (before dialog asking boot type: Live or Install), shows GUI that asking to enter password


nouveau and radeon modules are blacklisted

kubuntu_13.11_pass.png
kubuntu_13.10_syslog_nou.png

Seabios log:

Start bios (version ?-20130921_165137-mnt-storage-buildroots-extra-x86_64-eric) No Xen hypervisor found. Running on KVM Ram Size=0xb0000000 (0x0000000050000000 high) Relocating init from 0x000e22d1 to 0xaffe0950 (size 62952) Found QEMU fw_cfg CPU Mhz=3200 === PCI bus & bridge init === PCI: pci_bios_init_bus_rec bus = 0x0 PCI: pci_bios_init_bus_rec bdf = 0xe0 PCI: primary bus = 0x0 PCI: secondary bus = 0xff -> 0x1 PCI: pci_bios_init_bus_rec bus = 0x1 PCI: subordinate bus = 0x0 -> 0x1 === PCI device probing === Found 10 PCI devices (max PCI bus is 01) === PCI new allocation pass #1 === PCI: check devices PCI: secondary bus 1 size 00001000 type io PCI: secondary bus 1 size 02000000 type mem PCI: secondary bus 1 size 20000000 type prefmem === PCI new allocation pass #2 === PCI: map device bdf=00:01.0 bar 1, addr 0000d000, size 00000040 [io] PCI: map device bdf=00:1f.3 bar 4, addr 0000d040, size 00000040 [io] PCI: map device bdf=00:03.0 bar 4, addr 0000d080, size 00000020 [io] PCI: map device bdf=00:1f.2 bar 4, addr 0000d0a0, size 00000020 [io] PCI: map device bdf=00:01.0 bar 6, addr fe000000, size 00040000 [mem] PCI: map device bdf=00:01.0 bar 0, addr fe040000, size 00020000 [mem] PCI: map device bdf=00:04.0 bar 0, addr fe060000, size 00004000 [mem] PCI: map device bdf=00:02.0 bar 0, addr fe064000, size 00001000 [mem] PCI: map device bdf=00:03.0 bar 5, addr fe065000, size 00001000 [mem] PCI: map device bdf=00:1f.2 bar 5, addr fe066000, size 00001000 [mem] PCI: map device bdf=01:00.0 bar 5, addr 0000c000, size 00000080 [io] PCI: map device bdf=01:00.0 bar 0, addr fc000000, size 01000000 [mem] PCI: map device bdf=01:00.0 bar 6, addr fd000000, size 00080000 [mem] PCI: map device bdf=01:00.0 bar 1, addr d0000000, size 10000000 [prefmem] PCI: map device bdf=01:00.0 bar 3, addr e0000000, size 02000000 [prefmem] PCI: init bdf=00:00.0 id=8086:29c0 PCI: init bdf=00:01.0 id=8086:100e PCI: init bdf=00:02.0 id=8086:1e2d PCI: init bdf=00:03.0 id=8086:2922 PCI: init bdf=00:04.0 id=10de:0e1b PCI: init bdf=00:1c.0 id=8086:3420 PCI: init bdf=00:1f.0 id=8086:2918 Q35 LPC init: elcr=00 0c Using pmtimer, ioport 0xb008, freq 3579 kHz PCI: init bdf=00:1f.2 id=8086:2922 PCI: init bdf=00:1f.3 id=8086:2930 PCI: init bdf=01:00.0 id=10de:0fc1 PCI: Enabling 01:00.0 for primary VGA PCI: Setting VGA enable on bridge 00:1c.0 Found 1 cpu(s) max supported 1 cpu(s) MP table addr=0x000f17c0 MPC table addr=0x000f17d0 size=232 SMBIOS ptr=0x000f17a0 table=0x000f1670 size=301 ACPI DSDT=0xafffd8e0 ACPI tables: RSDP=0x000f1640 RSDT=0xafffd8a0 Scan for VGA option rom Running option rom at c000:0003

Last edited by myweb (2013-10-24 22:19:12)

Offline

#610 2013-10-24 22:10:33

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I`ve got some trouble getting the sound working for my setup (everything else works fine). When I use your script from the first post i hear some scratches when the VM starts but windows doesnt have any soundoutput.
The device manager has an "AMD High Definition Audio device" and an "Unknown device". Do you know what I could be doing wrong?

Offline

#611 2013-10-24 23:05:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

I`ve got some trouble getting the sound working for my setup (everything else works fine). When I use your script from the first post i hear some scratches when the VM starts but windows doesnt have any soundoutput.
The device manager has an "AMD High Definition Audio device" and an "Unknown device". Do you know what I could be doing wrong?

What OS are you using? 32bit or 64bit? there should be a "high definition audio device" somewhere on the device manager thats the emulated sound card

EDIT: i updated my script to use cpuset (https://code.google.com/p/cpuset/)

Last edited by nbhs (2013-10-24 23:11:20)

Offline

#612 2013-10-25 06:05:01

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have enabled vfio debug, please find Debug output via link: https://dl.dropboxusercontent.com/u/526 … ug.log.zip
and stubbed Nvidia card:

dmesg | grep stub [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:0fc1,10de:0e1b quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=51388cdb-7737-469b-80db-268b2f41e16c ro intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:0fc1,10de:0e1b quiet [ 0.418370] pci-stub: add 10DE:0FC1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.418383] pci-stub 0000:01:00.0: claimed by stub [ 0.418392] pci-stub: add 10DE:0E1B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.418399] pci-stub 0000:01:00.1: claimed by stub

Last edited by myweb (2013-10-25 06:18:15)

Offline

#613 2013-10-25 12:00:51

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:

I`ve got some trouble getting the sound working for my setup (everything else works fine). When I use your script from the first post i hear some scratches when the VM starts but windows doesnt have any soundoutput.
The device manager has an "AMD High Definition Audio device" and an "Unknown device". Do you know what I could be doing wrong?

What OS are you using? 32bit or 64bit? there should be a "high definition audio device" somewhere on the device manager thats the emulated sound card

EDIT: i updated my script to use cpuset (https://code.google.com/p/cpuset/)

I use Windows and Arch 64Bit.

Offline

#614 2013-10-25 21:23:15

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
nbhs wrote:
SpacePirate wrote:

I`ve got some trouble getting the sound working for my setup (everything else works fine). When I use your script from the first post i hear some scratches when the VM starts but windows doesnt have any soundoutput.
The device manager has an "AMD High Definition Audio device" and an "Unknown device". Do you know what I could be doing wrong?

What OS are you using? 32bit or 64bit? there should be a "high definition audio device" somewhere on the device manager thats the emulated sound card

EDIT: i updated my script to use cpuset (https://code.google.com/p/cpuset/)

I use Windows and Arch 64Bit.

Well unless you're using an old version of windows XP the audio drivers should be automatically installed.

UMzAKXW.jpg

Offline

#615 2013-10-27 13:56:31

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Recently upgrade packages to the latest version.

I had to apply the two patches list below for vga-passthrough for i915 (built in graphics card). The i915 was is the host and not the guest. This was recommend in a previous post and works well.

http://o.cs.uvic.ca:20810/perl/cid.pl?c … ba24389bee
http://o.cs.uvic.ca:20810/perl/cid.pl?c … f81dea8a7d

Anyway will there be a more permanent solution for the Intel GPU in the future?

This may not be the best place to post this question but I thought I give it a shot here anyway.

Thanks.

Offline

#616 2013-10-27 16:37:57

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
SpacePirate wrote:
nbhs wrote:

What OS are you using? 32bit or 64bit? there should be a "high definition audio device" somewhere on the device manager thats the emulated sound card

EDIT: i updated my script to use cpuset (https://code.google.com/p/cpuset/)

I use Windows and Arch 64Bit.

Well unless you're using an old version of windows XP the audio drivers should be automatically installed.

http://i.imgur.com/UMzAKXW.jpg

I use Windows 7. For me it looks almost the same like your screenshot, but instead of the "High Definition Audio Device" I`ve got an "Unkown Device". When I uninstalled it and searched for plug`n`play devices in the Devicemanager, it took a while, and then I`ve got an BSOD.

Is there another Device that I could use?
Could the Error be produces by the fact, that the iso i used for installation is fairly old and that I dont have every update for Win7 yet, since it only updates some things per reboot (-.-) ?

EDIT: i switched the device to "intel-hda" (ich6) and now it works perfectly

Last edited by SpacePirate (2013-10-27 21:31:11)

Offline

#617 2013-10-28 18:13:53

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

Could you please help to disable i915 driver right after boot?
I do not need GUI on IGD, I just need that all PCIE videocards are available for passthrough. As far as I understand (Alex, thank you for explanation) i915 driver does not allow to execute PCIE passthrough videocard in guest VM.
Thank you.

Offline

#618 2013-10-28 18:43:12

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AAA!!! I blacklisted i915 module, booted to multi-user.target and Nvidia GT640 is successfully displayed Seabios messages and booted to Kubutu Live!!!

Offline

#619 2013-10-28 19:42:51

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

AAA!!! I blacklisted i915 module, booted to multi-user.target and Nvidia GT640 is successfully displayed Seabios messages and booted to Kubutu Live!!!

Gratz, have you tried applying the patches from the previous alphahere post?

alphahere wrote:

Recently upgrade packages to the latest version.

I had to apply the two patches list below for vga-passthrough for i915 (built in graphics card). The i915 was is the host and not the guest. This was recommend in a previous post and works well.

http://o.cs.uvic.ca:20810/perl/cid.pl?c … ba24389bee
http://o.cs.uvic.ca:20810/perl/cid.pl?c … f81dea8a7d

Anyway will there be a more permanent solution for the Intel GPU in the future?

This may not be the best place to post this question but I thought I give it a shot here anyway.

Thanks.

Offline

#620 2013-10-28 20:20:23

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
myweb wrote:

AAA!!! I blacklisted i915 module, booted to multi-user.target and Nvidia GT640 is successfully displayed Seabios messages and booted to Kubutu Live!!!

Gratz, have you tried applying the patches from the previous alphahere post?

alphahere wrote:

Recently upgrade packages to the latest version.

I had to apply the two patches list below for vga-passthrough for i915 (built in graphics card). The i915 was is the host and not the guest. This was recommend in a previous post and works well.

http://o.cs.uvic.ca:20810/perl/cid.pl?c … ba24389bee
http://o.cs.uvic.ca:20810/perl/cid.pl?c … f81dea8a7d

Anyway will there be a more permanent solution for the Intel GPU in the future?

This may not be the best place to post this question but I thought I give it a shot here anyway.

Thanks.

I did not apply the patches - I can not open the links...

Offline

#621 2013-10-28 21:29:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

AAA!!! I blacklisted i915 module, booted to multi-user.target and Nvidia GT640 is successfully displayed Seabios messages and booted to Kubutu Live!!!

Gratz, have you tried applying the patches from the previous alphahere post?

alphahere wrote:

Recently upgrade packages to the latest version.

I had to apply the two patches list below for vga-passthrough for i915 (built in graphics card). The i915 was is the host and not the guest. This was recommend in a previous post and works well.

http://o.cs.uvic.ca:20810/perl/cid.pl?c … ba24389bee
http://o.cs.uvic.ca:20810/perl/cid.pl?c … f81dea8a7d

Anyway will there be a more permanent solution for the Intel GPU in the future?

This may not be the best place to post this question but I thought I give it a shot here anyway.

Thanks.

I did not apply the patches - I can not open the links...

First link

Second link

Offline

#622 2013-10-28 21:44:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
nbhs wrote:
myweb wrote:

AAA!!! I blacklisted i915 module, booted to multi-user.target and Nvidia GT640 is successfully displayed Seabios messages and booted to Kubutu Live!!!

Gratz, have you tried applying the patches from the previous alphahere post?

alphahere wrote:

Recently upgrade packages to the latest version.

I had to apply the two patches list below for vga-passthrough for i915 (built in graphics card). The i915 was is the host and not the guest. This was recommend in a previous post and works well.

http://o.cs.uvic.ca:20810/perl/cid.pl?c … ba24389bee
http://o.cs.uvic.ca:20810/perl/cid.pl?c … f81dea8a7d

Anyway will there be a more permanent solution for the Intel GPU in the future?

This may not be the best place to post this question but I thought I give it a shot here anyway.

Thanks.

I did not apply the patches - I can not open the links...

Glad you got it working, I suppose blacklisting i915 is another way to work around it's broken-ness with VGA arbiter, but vgacon doesn't play nicely with arbitration either, it just doesn't opt-out.  Here are (hopefully) working links to the i915 patches:

http://git.kernel.org/cgit/linux/kernel … f81dea8a7d
http://git.kernel.org/cgit/linux/kernel … ba24389bee


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#623 2013-10-28 21:53:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
myweb wrote:
nbhs wrote:

Gratz, have you tried applying the patches from the previous alphahere post?

I did not apply the patches - I can not open the links...

Glad you got it working, I suppose blacklisting i915 is another way to work around it's broken-ness with VGA arbiter, but vgacon doesn't play nicely with arbitration either, it just doesn't opt-out.  Here are (hopefully) working links to the i915 patches:

http://git.kernel.org/cgit/linux/kernel … f81dea8a7d
http://git.kernel.org/cgit/linux/kernel … ba24389bee

And how did the 7750 work?  Did you happen to be using a pre-rc5 kernel that already included these?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#624 2013-10-29 04:58:55

coolper22@gmail.com
Member
From: Shanghai,China
Registered: 2013-10-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

first, thanks for the greatest post.

I got i915 (built in graphics card) and GeForce GTX 650 working on windows 7 64bit, the hardware list:
Mainboard: GIGABYTE Z87M-D3H
CPU: i5-4430
Graphic Card: i915 (built in graphics card)
Graphic Card: GeForce GTX 650
I use kernel 3.12 rc6 with two patches about i915 that previous post, and qemu from git master with NoSnoop patch.

But on windows 7 32bit, in device manager,the "Intel(R) 5520/5500/X58 I/O 集线器 PCI Express 根端口 0 - 3420" driver show error code 12,This device cannot find enough free resources that it can use.
I try boot windows 7 32bit using q35(or only -readconfig ../docs/q35-chipset.cfg) without graphics card passthrough,it has the same problem.
I also try seabios rel-1.7.3.2,rel-1.7.3.1,rel-1.7.3,rel-1.7.2.2 and qemu 1.6.1,but no success.

Can somebody give me some suggestions?
I build qemu,seabios on Ubuntu 12.04 64bit.

Anybody passthrough graphics card to windows 7 32 bit successfully?

Anybody passthrough graphics card to windows xp 32 bit successfully?

Last edited by coolper22@gmail.com (2013-10-29 08:15:34)

Offline

#625 2013-10-29 05:48:39

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Will try blacklisting the i915 again. Up till now only Code 43 with Nvidia with i915 not on blacklist.
Got my 6870 working with 13.9 driver. Had to do the WorkaroundInstall=true trick to get it working even when it's missing a file. Played Skyrim and Tombraider for a few minutes and looked good. Used Intel audio for sound.
Using Kernel 3.12.0-rc4 on my ASrock Z77 WS and i7 3770.

@myweb: How do you do the vfio binding? Per device, or using the script from nbhs?

Is the pcie_acs_override useful? Haven't used it up till now.

Offline

#626 2013-10-29 13:07:14

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

And how did the 7750 work?

Windows 8 performance test shows: Graphics performance is 7.3 for 3d/2d.
But I noticed mouse hanging (not smooth moving) when Windows asks Administrator privileges (all screen covered with gray and just one dialog, that asks about administrator privileges, is active)

aw wrote:

Did you happen to be using a pre-rc5 kernel that already included these?

No

I need help one more time ... I get no success to boot already installed Windows 7 to real drive.
Partition table looks like:
sda1 EFI
sda2 Linux Swap
sda3 GPT system
sda4 Arch Linux (KVM qemu host)
sda5 Windows 7

I specified:
-drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

But I got black screen, not windows booting.
Could you please help to boot from real hard drive?

Offline

#627 2013-10-29 14:17:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:

And how did the 7750 work?

Windows 8 performance test shows: Graphics performance is 7.3 for 3d/2d.
But I noticed mouse hanging (not smooth moving) when Windows asks Administrator privileges (all screen covered with gray and just one dialog, that asks about administrator privileges, is active)

aw wrote:

Did you happen to be using a pre-rc5 kernel that already included these?

No

I need help one more time ... I get no success to boot already installed Windows 7 to real drive.
Partition table looks like:
sda1 EFI
sda2 Linux Swap
sda3 GPT system
sda4 Arch Linux (KVM qemu host)
sda5 Windows 7

I specified:
-drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

But I got black screen, not windows booting.
Could you please help to boot from real hard drive?

This is effectively the same as moving a disk with a windows install from one physical system to another.  Windows does not do well with this.  There are those that can hack it to work, but for best results, it's recommended to use a separate install.

Last edited by aw (2013-10-29 14:17:41)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#628 2013-10-29 17:16:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I gave a talk about VFIO-VGA support at KVM Forum last week.  Unfortunately due to technology malfunction there's no video, but the slides are linked here:

https://plus.google.com/107691536544094 … y8EHhJ6PWz


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#629 2013-10-29 17:21:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

empie wrote:

Is the pcie_acs_override useful? Haven't used it up till now.

It's useful if you have an IOMMU group that includes too many devices.  Intel didn't include ACS support for the root ports in many of their chipsets and processors, without it we can't guarantee isolation between devices.  We're working with Intel to get a whitelist of root ports to resolve this.  In the meantime, yes the pcie_acs_override patch can be used as a workaround.  Note that it only changes IOMMU group topology.  It doesn't specifically fix or enabling anything new wrt VGA.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#630 2013-10-29 18:38:50

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It's useful if you have an IOMMU group that includes too many devices.  Intel didn't include ACS support for the root ports in many of their chipsets and processors, without it we can't guarantee isolation between devices.  We're working with Intel to get a whitelist of root ports to resolve this.  In the meantime, yes the pcie_acs_override patch can be used as a workaround.  Note that it only changes IOMMU group topology.  It doesn't specifically fix or enabling anything new wrt VGA.

Ok, thanks. Will give it a go in the next couple of days.

Offline

#631 2013-10-29 20:33:49

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use the following script to bind devices:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

without "pcie_acs_override=downstream" "ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/" shows:

0000:00:1c.0 0000:00:1c.4 0000:00:1c.5 0000:00:1c.7 0000:02:00.0 0000:02:00.1 0000:03:00.0 0000:04:01.0 0000:05:00.0 0000:06:00.0

with:

0000:02:00.0 0000:02:00.1

Offline

#632 2013-10-29 20:38:46

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nbhs, @aw
I think it would be great to create some kind of Wiki page with summary/list of successful configuration (CPU, Motherboard, VideoCard) and short note of specific settings for particular configuration fo example.
Do we have that page?

Is it correct that in order to use real partition along with virtual drive I need to specify:

-drive file=/home/nbhs/windows.img,id=bootDisk,format=raw -device ide-hd,bus=ahci.0,drive=bootDisk -drive file=/dev/sdb,id=dataDisk,format=raw -device ide-hd,bus=ahci.1,drive=dataDisk

Could you please clarify how to specify boot order for seabios: what drive to boot?

Last edited by myweb (2013-10-30 11:09:15)

Offline

#633 2013-10-30 15:00:47

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hey

just wanted to chime in to report success on VGA passthrough with inspiration found in this Thread.

my Hardware
Gigabyte GA970A-UD3 (BIOS F8 beta with fixed ACPI-IVRS Table)
AMD Phenom II x6 1090T
EVGA Geforce GTX 285
Sapphire Radeon HD 5670

My Software
Debian Jessie
Kernel 3.12rc6 (Jessie Stock 3.10-3 got no vfio-pci-vga enabled anyway)
qemu-kvm 1.6.0+dfsg-2 (debian unstable)
Windows7 x64 in VM with latest Nvidia Driver

Only downside is that i have the reboot issue. e.g. i can't reboot the Windows VM without rebooting the Host aswell.

Cheers for the Inspiration and Hints

Offline

#634 2013-10-30 15:04:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blitzschlag wrote:

hey

just wanted to chime in to report success on VGA passthrough with inspiration found in this Thread.

my Hardware
Gigabyte GA970A-UD3 (BIOS F8 beta with fixed ACPI-IVRS Table)
AMD Phenom II x6 1090T
EVGA Geforce GTX 285
Sapphire Radeon HD 5670

My Software
Debian Jessie
Kernel 3.12rc6 (Jessie Stock 3.10-3 got no vfio-pci-vga enabled anyway)
qemu-kvm 1.6.0+dfsg-2 (debian unstable)
Windows7 x64 in VM with latest Nvidia Driver

Only downside is that i have the reboot issue. e.g. i can't reboot the Windows VM without rebooting the Host aswell.

Cheers for the Inspiration and Hints

Reboot should be fixed with current qemu.git or wait for 1.7 when it comes out.  Just to be clear, you're assigning the GTX285 and the HD5670 is your host display, right?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#635 2013-10-30 15:10:33

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Reboot should be fixed with current qemu.git or wait for 1.7 when it comes out.  Just to be clear, you're assigning the GTX285 and the HD5670 is your host display, right?

ah, cool

Yes, the GTX 285 is passed through to the Windows VM

Offline

#636 2013-10-30 17:16:09

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thanks for this thread! Using this setup for a couple of years under XEN - but with KVM and VFIO its easier and - the most important part - more stable (with xen the Guest-Reboot was not always successful - with KVM it is!).

Anyways i converted most of the manual qemu options into libvirt-xml - main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually.
Another positive effect is Host Startup/Shutdown handling due to the libvirt init script - they stop/start your Guest on Host startup/shutdown.

<domain type='kvm' id='26' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>windows7</name> <uuid>xxxxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxx</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='6'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-1,4-5'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-1.5'>hvm</type> <boot dev='hd'/> <smbios mode='host'/> </os> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/ssd/win7system'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/mapper/win7games_cached'/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </disk> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0' model='none'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='00:xx:xx:xx:xx:xx'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <memballoon model='none'> <alias name='balloon0'/> </memballoon> </devices> <seclabel type='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.2,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.7,bus=pcie.0'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/> <qemu:env name='DISPLAY' value=':0'/> <qemu:env name='QEMU_PA_SAMPLES' value='128'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> </qemu:commandline> </domain>

Adapt UUID and MAC adress - and of course your PCI-IDs...

Greetings
K.

Last edited by kaeptnb (2013-10-30 20:54:40)

Offline

#637 2013-10-30 17:39:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great tip!  I've parsed it down to what I think are the important parts if anyone wants to try to adapt their own guest xml.  libvirt does have vfio support now, but it doesn't (and shouldn't) know about the x-vga=on option.  If you just want to add ",x-vga=on" to a libvirt managed guest you can follow the instructions here to make a wrapper around the emulator. (a different bios shouldn't be necessary on qemu.git)

kaeptnb wrote:

                                                                                                       
  <os>                                                                                                                 
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>                                                                 
    ...
  </os>                                                                                                                 
  ...
  <devices>                                                                                                             
    <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator>                                                         
    ...
  </devices>
  ...
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/>
    ...
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/>
    ...
  </qemu:commandline>


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#638 2013-10-30 17:49:40

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iirc if you pass through qemu args, you need the xmlns so that libvirt can validate the xml:

xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'

(1st line)

And regarding the CPU Pinning/Isolation: the "isolcpu" Kernel Parameter gets very handy - in my case : "isolcpus=2,3,6,7"

BTW it REALLY makes a difference (i went through great lenghts to benchmark it) to keep in mind cache trashing - so try to pass through the corresponding CPUs when you have a HT CPU like me (i7-920).

IRQ Balance is another hint - i have bad experience with the irqbalance daemon, so i made myself a ugly startup-script which puts the most intense interrupts to non-guest CPUs:

for i in $(grep -i ahci     /proc/interrupts | awk -F: '{print $1}') ; do echo 2 >/proc/irq/$i/smp_affinity ; done
...

Greetings!
K.

Offline

#639 2013-10-30 18:15:16

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Is it correct that in order to use real partition along with virtual drive I need to specify:

-drive file=/home/nbhs/windows.img,id=bootDisk,format=raw -device ide-hd,bus=ahci.0,drive=bootDisk -drive file=/dev/sdb,id=dataDisk,format=raw -device ide-hd,bus=ahci.1,drive=dataDisk

Could you please clarify how to specify boot order for seabios: what drive to boot?

I use this to first boot from a slipstreamed iso, then the snapshot of the vm-disk on a logical volume:

-drive file=/dev/vg01/win7gtx560ti-slip4.1-snap01,cache=none,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/data/win7hp-sp1-x64-ss20131008-rt7-4.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-boot dc

And thanks for the info regarding the binding of devices.

Last edited by empie (2013-10-30 18:17:18)

Offline

#640 2013-10-30 18:23:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

...
main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually
...

Have you tried using cset and taskset? https://code.google.com/p/cpuset/, thats what i use on my script (on the front page) to pin my vm vcpus (6) to my cpu cores (2-7)

Last edited by nbhs (2013-10-30 18:27:09)

Offline

#641 2013-10-30 20:26:24

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, that works (taskset or cgroups) - but doesnt ensure 1:1 pinning;
i.e. i can say qemu with its 4 vcpus should run on cpus 2,3,6,7 - but nothing ensures the qemu threads aren't hopping between these 4 real cpus.
libvirt ensures that - it stops qemu after initialization, pins each thread, and continues qemu...

sad qemu doesn't support this directly - but OTOH its not bad to have libvirt and with all its benefits running, so...

Greetings

Offline

#642 2013-10-30 20:30:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

Yes, that works (taskset or cgroups) - but doesnt ensure 1:1 pinning;
i.e. i can say qemu with its 4 vcpus should run on cpus 2,3,6,7 - but nothing ensures the qemu threads aren't hopping between these 4 real cpus.
libvirt ensures that - it stops qemu after initialization, pins each thread, and continues qemu...

sad qemu doesn't support this directly - but OTOH its not bad to have libvirt and with all its benefits running, so...

Greetings

I believe qemu creates a thread per vcpu, so you can pin each thread to a core using taskset

Offline

#643 2013-10-30 20:33:10

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, each vcpu is a thread.
but which thread is which vcpu (important regarding cache-trashing)?
and (thats maybe just my incompetence) - how can you work with threads and taskset (afaik it only nows -p - the PID) ?

Offline

#644 2013-10-30 20:35:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

'info cpus' from the monitor gives the TID for each vcpu


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#645 2013-10-30 20:40:33

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ah nice. and taskset can handle TID also with -p i guess?

but the shell magic to automise this is probably more hassle than to adapt to libvirt wink

anyways - its worth the hassle either way - fewer microlags - i never tried it (i just use it for starting game.exe) but i bet you could even use it as a DAW station wink

Greetings
K.

Edit: "it" - being the windows7 kvm guest

Last edited by kaeptnb (2013-10-30 20:41:51)

Offline

#646 2013-10-30 20:46:26

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

regarding latency:
i noticed a huge latency penalty with "-usbdevice tablet " and "-vnc 0.0.0.0:10" - oddly?!

meassurable with DPC Latency Checker ( http://www.thesycon.de/deu/latency_check.shtml )

Offline

#647 2013-10-30 23:32:52

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

Hi,

thanks for this thread! Using this setup for a couple of years under XEN - but with KVM and VFIO its easier and - the most important part - more stable (with xen the Guest-Reboot was not always successful - with KVM it is!).

Anyways i converted most of the manual qemu options into libvirt-xml - main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually.
Another positive effect is Host Startup/Shutdown handling due to the libvirt init script - they stop/start your Guest on Host startup/shutdown.

<domain type='kvm' id='26' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>windows7</name> <uuid>xxxxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxx</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='6'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-1,4-5'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-1.5'>hvm</type> <boot dev='hd'/> <smbios mode='host'/> </os> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/ssd/win7system'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/mapper/win7games_cached'/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </disk> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0' model='none'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='00:xx:xx:xx:xx:xx'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <memballoon model='none'> <alias name='balloon0'/> </memballoon> </devices> <seclabel type='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.2,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.7,bus=pcie.0'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/> <qemu:env name='DISPLAY' value=':0'/> <qemu:env name='QEMU_PA_SAMPLES' value='128'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> </qemu:commandline> </domain>

Adapt UUID and MAC adress - and of course your PCI-IDs...

Greetings
K.


I tried creating my own xml based on yours, but virsh fails with:

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What could be wrong?

Offline

#648 2013-10-30 23:37:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

I tried creating my own xml based on yours, but virsh fails with:

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What could be wrong?

I'd guess you need to change a few things in /etc/libvirt/qemu.conf

user & group should probably be root
clear_emulator_capabilities should probably be 0

You'll still need to setup the devices with vfio prior to starting the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#649 2013-10-30 23:51:01

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Chetyre wrote:

I tried creating my own xml based on yours, but virsh fails with:

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What could be wrong?

I'd guess you need to change a few things in /etc/libvirt/qemu.conf

user & group should probably be root
clear_emulator_capabilities should probably be 0

You'll still need to setup the devices with vfio prior to starting the guest.

Now it gives me a different error

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

Offline

#650 2013-10-30 23:58:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

Now it gives me a different error

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#651 2013-10-31 00:44:14

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Chetyre wrote:

Now it gives me a different error

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

Last edited by Chetyre (2013-10-31 00:46:00)

Offline

#652 2013-10-31 00:56:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:
aw wrote:
Chetyre wrote:

Now it gives me a different error

error: Failed to start domain windows7 error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

have you tried adding "/dev/vfio/15" to cgroup_device_acl =[..]  on /etc/libvirt/qemu.conf?

Offline

#653 2013-10-31 02:50:10

sabzn
Member
Registered: 2013-10-31
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys.
Sorry to bother you, but I just can't get this to work as it should.
For quite some time I've been trying to get Windows 7 to work with VGA-Passthrough using my 7970.
The VM starts fine and VGA-Passthrough seems to be working to some extend (the VM's output is shown through the passthrough'd GPU). Dmesg shows no errors either.
As I said, the video output through the VM works fine as long as I don't install the Catalyst drivers.
Whenever I try to install them, the VM BSOD's during the next restart with the "device reset timeout" error mentioned on page 23/24 in this thread, but the suggested fixes didn't work for me.
When I manually load the VGA BIOS (with rombar=0,romfile=/home/user/7970.rom), the VM does not bluescreen, but the Catalyst driver throws an error message saying that it failed to load the driver.
The GPU itself seems to be recognized by the Device Manager and programs like GPU-z, but there is still no acceleration available (see link below)
http://picpaste.com/pics/qlq1Pz79.1383184744.png

So far I have tried Catalyst 12.4, 12.10, 13.4 and 13.11 beta, but all of them show the same results.

Specs:
Intel i5-4670, integrated GPU used for the host
AsRock z87 Pro4
AMD Radeon HD 7970, used for passthrough
Kernel 3.12-rc6 with the i915 patches applied.

I have yet to apply the pcie_acs_override patch, but I don't think the problem is related to this one.


If you need any other information about the system such as exact configurations or logs, please let me know.

best regards

Last edited by sabzn (2013-10-31 02:52:00)

Offline

#654 2013-10-31 04:14:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sabzn wrote:

Hello guys.
Sorry to bother you, but I just can't get this to work as it should.
For quite some time I've been trying to get Windows 7 to work with VGA-Passthrough using my 7970.
The VM starts fine and VGA-Passthrough seems to be working to some extend (the VM's output is shown through the passthrough'd GPU). Dmesg shows no errors either.
As I said, the video output through the VM works fine as long as I don't install the Catalyst drivers.
Whenever I try to install them, the VM BSOD's during the next restart with the "device reset timeout" error mentioned on page 23/24 in this thread, but the suggested fixes didn't work for me.
When I manually load the VGA BIOS (with rombar=0,romfile=/home/user/7970.rom), the VM does not bluescreen, but the Catalyst driver throws an error message saying that it failed to load the driver.
The GPU itself seems to be recognized by the Device Manager and programs like GPU-z, but there is still no acceleration available (see link below)
http://picpaste.com/pics/qlq1Pz79.1383184744.png

So far I have tried Catalyst 12.4, 12.10, 13.4 and 13.11 beta, but all of them show the same results.

Specs:
Intel i5-4670, integrated GPU used for the host
AsRock z87 Pro4
AMD Radeon HD 7970, used for passthrough
Kernel 3.12-rc6 with the i915 patches applied.

I have yet to apply the pcie_acs_override patch, but I don't think the problem is related to this one.


If you need any other information about the system such as exact configurations or logs, please let me know.

best regards

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#655 2013-10-31 07:14:16

sabzn
Member
Registered: 2013-10-31
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

I'm fairly certain that it's using q35. -M q35 is set in the start script and even programs in windows hint that the q35 chipset is being used.

I will check it again when I'm at home though.

Offline

#656 2013-10-31 12:17:14

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,
Could you please help to find the best solution to share drive (array) between two virtual machines in the same time:
1. Real machine: qemu host, combine real hard drive to ZFS partition
2. Virtual Linux machine should has access to ZFS partition
3. Virtual Windows machine should has access to ZFS partition
I am sorry for offtop question, but it would be great if someone point me to right direction.

Offline

#657 2013-10-31 12:29:31

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Chetyre wrote:
aw wrote:

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

have you tried adding "/dev/vfio/15" to cgroup_device_acl =[..]  on /etc/libvirt/qemu.conf?

I had the same "Operation not permitted" issue, and that fixed it. Thanks!

Having it in libvirt is pretty neat! (Until now I had a systemd --user service)

Last edited by teekay (2013-10-31 12:32:53)

Offline

#658 2013-10-31 12:38:54

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear All,
Could you please help to find the best solution to share drive (array) between two virtual machines in the same time:
1. Real machine: qemu host, combine real hard drive to ZFS partition
2. Virtual Linux machine should has access to ZFS partition
3. Virtual Windows machine should has access to ZFS partition
I am sorry for offtop question, but it would be great if someone point me to right direction.

zfs set sharesmb=on pool/fs maybe?

http://zfsonlinux.org/faq.html#HowDoISetupShares

Last edited by teekay (2013-10-31 12:40:04)

Offline

#659 2013-10-31 13:42:04

sabzn
Member
Registered: 2013-10-31
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sabzn wrote:
aw wrote:

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

I'm fairly certain that it's using q35. -M q35 is set in the start script and even programs in windows hint that the q35 chipset is being used.

I will check it again when I'm at home though.


I just checked again and it is indeed using q35. The CPU-z Mainboard tab shows the same output as in OP's screenshots.

Do you think that this error might be related to the lack of the pcie_acs_override patch?

Offline

#660 2013-10-31 13:44:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sabzn wrote:
sabzn wrote:
aw wrote:

Based on the GPU being listed as installed on a PCI bus and it's bus address of 00:02.0, I'm guessing that you're using the default 440fx machine model.  I think most folks here are having better success using the q35 model.

I'm fairly certain that it's using q35. -M q35 is set in the start script and even programs in windows hint that the q35 chipset is being used.

I will check it again when I'm at home though.


I just checked again and it is indeed using q35. The CPU-z Mainboard tab shows the same output as in OP's screenshots.

Do you think that this error might be related to the lack of the pcie_acs_override patch?

No.  Even if you're using q35 you're still placing the GPU on the root complex rather than behind a root port.  It would be on something other than bus 0 if it were behind a root port.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#661 2013-10-31 19:22:01

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
nbhs wrote:
Chetyre wrote:

Even without acs_override that group has always been its own. Only 02:00.0 and 02:00.1 (audio) are there. Both are bound to vfio.

Besides, it works when I call qemu directly. I get a lot of BSODs eventually but it works.

have you tried adding "/dev/vfio/15" to cgroup_device_acl =[..]  on /etc/libvirt/qemu.conf?

I had the same "Operation not permitted" issue, and that fixed it. Thanks!

Having it in libvirt is pretty neat! (Until now I had a systemd --user service)

Guess I spoke too soon.

I can either get VFIO working, OR the ICH9 audio emulation with pulseaudio. But not both.

If I configure qemu to run as my main user, the VM starts and I get output on the passed through card, but the resolution is low, and also the passed through USB3 controller isn't working (so, no mouse -.-). The /dev/vfio/16 /dev/vfio/17 /dev/vfio/7 /dev/vfio/vfio devices are chown'ed.
In Qemu log I see

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f0fe95d3ce0, 0x0, 0xb0000000, 0x7f0e44000000) = -12 (C annot allocate memory) qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f0fe95d3ce0, 0x100000000, 0xd0000000, 0x7f0ef4000000) = -12 (Cannot allocate memory) qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0x0, 0xc0000, 0x7f0e44000000) = -12 (Cannot allocate memory) qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0xc0000, 0x20000, 0x7f0fcf600000) = -12 (Cannot allocate memory) qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0xe0000, 0x20000, 0x7f0fdc200000) = -12 (Cannot allocate memory) qemu-system-x86_64: vfio_dma_map(0x7f0fe95d3ce0, 0x100000, 0xaff00000, 0x7f0e44100000) = -12 (Cannot allocate memory) ...

And dmesg many of those

[ 9507.459310] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

and some

[ 8845.605035] AMD-Vi: Event logged [IO_PAGE_FAULT device=05:00.0 domain=0x001f address=0x00000001c8cec800 flags=0x0000] [ 8847.018681] AMD-Vi: Event logged [IO_PAGE_FAULT device=04:00.0 domain=0x001f address=0x00000001bf330000 flags=0x0000]

If I run as root, VFIO works fine, but it cannot connect to my user's pulseaudio session:

pulseaudio: pa_context_connect() failed pulseaudio: Reason: Connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver

My domain .xml has

 <qemu:env name='QEMU_PA_SAMPLES' value='128'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>

I must be missing something, either way.

Last edited by teekay (2013-10-31 19:25:07)

Offline

#662 2013-10-31 19:41:16

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Teekay,

Pulseaudio is running as a your user, qemu running as root may have difficulties to connect to its session i guess.

I stopped bothering with pulseaudio and qemu's sound emulation, as i couln't get the a/v latency below an acceptable minimum.
When playing e.g. this A/V Sync test http://www.youtube.com/watch?v=cGgf_dbDMsw - i always had a noteable lag (with above settings QEMU_PA_SAMPLES=128 it was minimum lag - but still noteable).
So i gave up with a fully virtualized soundcard and use a USB connected one.

Regarding mouse support i can recommend synergy: Its fast enough even for gaming, and when you press the Scroll-Lock button, the mouse gets locked which helps many fullscreen-applications to function perfectly with synergy.

Offline

#663 2013-10-31 20:17:23

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

Hi Teekay,

Pulseaudio is running as a your user, qemu running as root may have difficulties to connect to its session i guess.

I stopped bothering with pulseaudio and qemu's sound emulation, as i couln't get the a/v latency below an acceptable minimum.
When playing e.g. this A/V Sync test http://www.youtube.com/watch?v=cGgf_dbDMsw - i always had a noteable lag (with above settings QEMU_PA_SAMPLES=128 it was minimum lag - but still noteable).
So i gave up with a fully virtualized soundcard and use a USB connected one.

Regarding mouse support i can recommend synergy: Its fast enough even for gaming, and when you press the Scroll-Lock button, the mouse gets locked which helps many fullscreen-applications to function perfectly with synergy.

Hi kaeptnb,

thanks for your input.

When using qemu command line (as user), both vfio and ICH9 emulation to pulseaudio works fine. For mouse I pass through a dedicated USB3 controller and use a KVM-switch, so no issues there. Audio is working fine for what I need it, too.

I just wonder what's missing to get VFIO fully working when running as user in libvirt. I did
* chown tom:tom /dev/vfio/{17,18,19}
* chmod 666 /dev/vfio/vfio
* set memlock to 6GB + 2GB overhead in limits.conf (the VM is confiured to run with 6GB)
* set user, group and "clear_emulator_capabilities = 0" in /etc/libvirt/qemu.conf

Still same RLIMIT_MEMLOCK and "vfio_dma_map .. Cannot allocate memory" errors.

I'd really like to run as user, not root - Audio is not the only issue here (I could just enable the onboard HDA, pass it through and plug it into my host MAUDIO Delta's input) - but,  running Windows as root sounds scary! big_smile

Last edited by teekay (2013-10-31 21:48:12)

Offline

#664 2013-10-31 22:46:41

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes you're right, the need to run it as root is suboptimal, but to be honest i think IF any windows process is capabale to get out of kvm its already on the kernel-level - so getting root may be easy anyhow.

But yes - i also would like to run libvirt/qemu as non-root...

Offline

#665 2013-10-31 22:51:18

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oh i overread the main part: you can sucessfully run qemu as a user manually!
that never worked for me, when i startet qemu as a user i got the same or similar messages like you.

Last edited by kaeptnb (2013-10-31 22:52:38)

Offline

#666 2013-10-31 23:22:38

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

oh i overread the main part: you can sucessfully run qemu as a user manually!
that never worked for me, when i startet qemu as a user i got the same or similar messages like you.

Yup, that works fine. It just required to chown the /dev/vfio devices and the hugetlbfs area.

Having it in libvirt would be cosmetic. I have some LXC containers managed with libvirt, so this would put everything in a unique interface. But libvirt causes me a headache, no idea where it takes the RLIMIT_MEMLOCK value from. It seems to ignore limits.conf

Offline

#667 2013-11-01 08:32:50

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

but the shell magic to automise this is probably more hassle than to adapt to libvirt wink

Here is a script I use to pin vcpus. It works, though it's ugly:

qemu [your usual options] -qmp unix:./qmp-sock,server,nowait --monitor stdio i=0 echo '{"execute":"qmp_capabilities"} {"execute":"query-cpus"}' | socat UNIX-CONNECT:./qmp-sock STDIO | grep -Po '"thread_id": (\d)*' | awk '{print $2}' | while read tid do taskset -pc $((i++)) $tid done

Now a question regarding libvirt.
Is it possible to launch just one vm with root privileges in libvirt? All my other vms work fine with regular user privileges, I don't want to switch all of them to root because of one special vm.

Offline

#668 2013-11-01 16:09:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

libvirt does have vfio support, so many of you may have any easier time getting the latest libvirt and enabling vfio that way.  Then it's a matter of a small wrapper script to add the ,x-vga=on option for the VGA device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#669 2013-11-01 18:23:14

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got a bit further regarding the libvirt-as-user issue. On the libvirt mailing list there were some comments about recently added patches the make libvirt set a proper RLIMIT_MEMLOCK value, if either memory is locked in the domain, or if VFIO is configured somewhere.

This is were your comment comes in, @aw

aw wrote:

libvirt does have vfio support, so many of you may have any easier time getting the latest libvirt and enabling vfio that way.  Then it's a matter of a small wrapper script to add the ,x-vga=on option for the VGA device.

I'm using latest libvirt-git, and I'm struggling to convert this

-device vfio-pci,host=03:00.0,bus=pcie.0

into libvirt xml.
Using "virsh edit" I tried like this:

 <hostdev type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev>

which generated this address
<address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
and this command line:

-device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.2,addr=0x6

It made the RLIMIT_MEMLOCK issues go away, but it also made the VM run into a boot->crash->boot loop.

Not sure if I'm doing something wrong.. is there a pcie address type maybe?

Offline

#670 2013-11-02 16:16:00

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I Have a problem passing through my PCIe Combo Card to the virtual machine, because it has the same ASM1062 Serial ATA Controller as I have onboard.

# lspci -nn 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05) 00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05) 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) 00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5) 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) 00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 [8086:8c1a] (rev d5) 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) 00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] [1002:6819] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] 03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) 05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 06:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) 07:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) 07:02.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)

I think 05:00.0 to 09:00.0 would be the combo card, but all 3 SATA Controller have [1b21:0612] vendor/device id. Any ideas what would be the best way to pass this whole card to the virtual machine? Do I need to pass those PCI Bridges, too?

Thanks in advance
Frank

Offline

#671 2013-11-02 20:37:58

cjn
Member
Registered: 2013-10-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I got hold of another Radeon HD 5450. I'm trying to pass it through to a Windows 7 guest while using an identical graphics card on the host, which is now running the radeon frame buffer driver (but no X). I still don't have much luck. In fact, the host crashes hard as soon as I start QEMU, right when the guest is loading the Windows 7 installer and the guest graphics card says "Windows is loading files...".

This is the relevant snippet from syslog when the crash happens

Nov 2 15:32:20 host kernel: [ 746.027441] VFIO - User Level meta-driver version: 0.3 Nov 2 15:35:04 host kernel: [ 911.086167] pcieport 0000:00:0d.0: driver skip pci_set_master, fix it! Nov 2 15:35:04 host kernel: [ 911.086173] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) Nov 2 15:35:07 host acpid: input device has been disconnected, fd 7 Nov 2 15:35:07 host kernel: [ 914.162057] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none Nov 2 15:35:08 host kernel: [ 914.777896] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci Nov 2 15:35:08 host kernel: [ 915.138530] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci Nov 2 15:35:09 host kernel: [ 915.571288] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci Nov 2 15:35:09 host kernel: [ 915.935931] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci

and this is the VFIO debug output (warning: 4.5MB of logs). Not all of the relevant kernel messages are syslogged, unfortunately. Some only reach the console. I've seen crashes in both SATA and USB code.

My configuration is currently

Linux 3.12.0-rc7
QEMU head as of today (includes SeaBIOS) + nosnoop patch + debug-vfio patch
ASRock 990FX Extreme3
AMD FX-8350
Radeon HD 5450 (host, PCI address 01:00)
Radeon HD 5450 (guest, PCI address 05:00)

I start QEMU like this

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/local/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/dev/shm/windows.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/root/en_windows_7_home_premium_with_sp1_x64_dvd_u_676549.iso,id=isocd \ -device ide-cd,bus=ahci.1,drive=isocd \ -nographic -usb -usbdevice host:04fe:0006 -usbdevice host:046d:c05b

The passed USB devices are my keyboard and my mouse.

Due to the fact that my two graphics cards have identical PCI IDs

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68] 05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1] 05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

and pci-stub accepts only PCI IDs as parameters (and not PCI addresses), I have

# Bind both graphics cards to the pci-stub driver and then unbind the one for # the host, making sure that the pci-stub driver is loaded before the drm # driver (and, in turn, before the radeon driver). options pci-stub ids=1002:68e1,1002:aa68 install pci-stub /sbin/modprobe --ignore-install pci-stub $CMDLINE_OPTS; /usr/local/sbin/pci-stub-unbind 0000:01:00.0 0000:01:00.1 softdep drm pre: pci-stub

in /etc/modprobe.d/pci-stub.conf and

#!/bin/sh for dev in "$@" do echo $dev > /sys/bus/pci/drivers/pci-stub/unbind done

in /usr/local/sbin/pci-stub-unbind, to bind the guest graphics card to the pci-stub driver and the host graphics card to the radeon driver.

Anyway, how do I get rid of the crashes? They seem to be caused by the radeon driver because I get no crashes without it.

Offline

#672 2013-11-03 10:58:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cjn wrote:

Hi all,

I got hold of another Radeon HD 5450. I'm trying to pass it through to a Windows 7 guest while using an identical graphics card on the host, which is now running the radeon frame buffer driver (but no X). I still don't have much luck. In fact, the host crashes hard as soon as I start QEMU, right when the guest is loading the Windows 7 installer and the guest graphics card says "Windows is loading files...".

This is the relevant snippet from syslog when the crash happens

Nov 2 15:32:20 host kernel: [ 746.027441] VFIO - User Level meta-driver version: 0.3 Nov 2 15:35:04 host kernel: [ 911.086167] pcieport 0000:00:0d.0: driver skip pci_set_master, fix it! Nov 2 15:35:04 host kernel: [ 911.086173] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) Nov 2 15:35:07 host acpid: input device has been disconnected, fd 7 Nov 2 15:35:07 host kernel: [ 914.162057] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none Nov 2 15:35:08 host kernel: [ 914.777896] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci Nov 2 15:35:08 host kernel: [ 915.138530] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci Nov 2 15:35:09 host kernel: [ 915.571288] usb 6-2.4.1: reset full-speed USB device number 5 using ohci-pci Nov 2 15:35:09 host kernel: [ 915.935931] usb 6-2.4.2: reset low-speed USB device number 6 using ohci-pci

and this is the VFIO debug output (warning: 4.5MB of logs). Not all of the relevant kernel messages are syslogged, unfortunately. Some only reach the console. I've seen crashes in both SATA and USB code.

My configuration is currently

Linux 3.12.0-rc7
QEMU head as of today (includes SeaBIOS) + nosnoop patch + debug-vfio patch
ASRock 990FX Extreme3
AMD FX-8350
Radeon HD 5450 (host, PCI address 01:00)
Radeon HD 5450 (guest, PCI address 05:00)

I start QEMU like this

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/local/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/dev/shm/windows.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/root/en_windows_7_home_premium_with_sp1_x64_dvd_u_676549.iso,id=isocd \ -device ide-cd,bus=ahci.1,drive=isocd \ -nographic -usb -usbdevice host:04fe:0006 -usbdevice host:046d:c05b

The passed USB devices are my keyboard and my mouse.

Due to the fact that my two graphics cards have identical PCI IDs

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68] 05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1] 05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

and pci-stub accepts only PCI IDs as parameters (and not PCI addresses), I have

# Bind both graphics cards to the pci-stub driver and then unbind the one for # the host, making sure that the pci-stub driver is loaded before the drm # driver (and, in turn, before the radeon driver). options pci-stub ids=1002:68e1,1002:aa68 install pci-stub /sbin/modprobe --ignore-install pci-stub $CMDLINE_OPTS; /usr/local/sbin/pci-stub-unbind 0000:01:00.0 0000:01:00.1 softdep drm pre: pci-stub

in /etc/modprobe.d/pci-stub.conf and

#!/bin/sh for dev in "$@" do echo $dev > /sys/bus/pci/drivers/pci-stub/unbind done

in /usr/local/sbin/pci-stub-unbind, to bind the guest graphics card to the pci-stub driver and the host graphics card to the radeon driver.

Anyway, how do I get rid of the crashes? They seem to be caused by the radeon driver because I get no crashes without it.

You're supposed to bind the card to vfio-pci not pci-stub

Offline

#673 2013-11-03 12:51:37

cjn
Member
Registered: 2013-10-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You're supposed to bind the card to vfio-pci not pci-stub

Thanks, but I already do that, using the vfio-bind script. I run

vfio-bind 0000:05:00.0 0000:05:00.1

right before starting QEMU. This should unbind the graphics card from the pci-stub driver and bind it to the vfio-pci driver.

Also, my IOMMU group looks fine, as far as I can see

root@host:~# ll /sys/bus/pci/devices/0000\:05\:00.0/iommu_group/devices/ total 0 lrwxrwxrwx 1 root root 0 Nov 3 13:37 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:0d.0/0000:05:00.0 lrwxrwxrwx 1 root root 0 Nov 3 13:37 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:0d.0/0000:05:00.1

Have you had any luck passing through your Radeon HD 5450 (with the radeon driver loaded)? Or have you only used it on the host?

Offline

#674 2013-11-04 14:14:17

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Hello,
I Have a problem passing through my PCIe Combo Card to the virtual machine, because it has the same ASM1062 Serial ATA Controller as I have onboard.

# lspci -nn 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05) 00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05) 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) 00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5) 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) 00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 [8086:8c1a] (rev d5) 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) 00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] [1002:6819] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] 03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) 05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 06:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) 07:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) 07:02.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)

I think 05:00.0 to 09:00.0 would be the combo card, but all 3 SATA Controller have [1b21:0612] vendor/device id. Any ideas what would be the best way to pass this whole card to the virtual machine? Do I need to pass those PCI Bridges, too?

Thanks in advance
Frank

I think I wrote this in a wrong way. I could pass them to the VM by using their enumeration, but how do I tell my kernel to ignore them on boot with pci-stub?

Offline

#675 2013-11-04 17:47:27

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

like this:

kernel commandline parameter: pci-stub.ids=1002:6819,1002:aab0

the id's are not the PCI Bus IDs, but the Device Class/Subclass - you get them with "lspci -vn"

Offline

#676 2013-11-04 18:41:41

Unnamed_Hero
Member
Registered: 2012-06-12
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi to all. I have a problem starting test qemu machine.
I'm using kernel 3.11.6, qemu 1.6.1-2

I bind Radeon card (2 ids) with vfio-bind script. (BTW, should I bind Radeon's PCI-E with this script?)

My error message:

qemu-system-x86_64: -device vfio-pci,host=00:01.1,bus=root.1,addr=00.1: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver


My card list:

01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]

Nvidia is for Linux host, Radeon is for Windows guest.

My IOMMU list is

lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <--PCI-E 0 < Nvidia lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <--PCI-E 1 < Radeon lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 <-- Nvidia lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 <-- Nvidia lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 <-- Radeon lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 <-- Radeon

So, I can't unbind my system video.

What should I do?

Last edited by Unnamed_Hero (2013-11-04 18:42:24)

Offline

#677 2013-11-04 19:06:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Unnamed_Hero wrote:

Hi to all. I have a problem starting test qemu machine.
I'm using kernel 3.11.6, qemu 1.6.1-2

I bind Radeon card (2 ids) with vfio-bind script. (BTW, should I bind Radeon's PCI-E with this script?)

My error message:

qemu-system-x86_64: -device vfio-pci,host=00:01.1,bus=root.1,addr=00.1: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver


My card list:

01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]

Nvidia is for Linux host, Radeon is for Windows guest.

My IOMMU list is

lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <--PCI-E 0 < Nvidia lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <--PCI-E 1 < Radeon lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 <-- Nvidia lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 <-- Nvidia lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 <-- Radeon lrwxrwxrwx 1 root root 0 ноя 4 21:03 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 <-- Radeon

So, I can't unbind my system video.

What should I do?

First, things are not going to work with the kernel and qemu versions you have.  See the first post for where to get the right versions.  You will need to build from source.

Next, if your IOMMU group doesn't let you separate devices it's because the root ports do not support PCI ACS and we therefore must assume that peer-to-peer traffic between subordinates of those roots is possible.  The long term fix for this is Intel properly supporting ACS on root ports and providing a white list for existing devices.  We're working with them to do that.  In the interim, several people here are using this patch: https://lkml.org/lkml/2013/5/30/513  This was rejected upstream, so you'll need to patch it in.

Finally, I assume by "PCI-E" you mean the root port the device is attached to (00:01.0 & 00:01.1 in your listing), and no, that should not be bound to vfio-pci.

Also note that if you're using the proprietary nvidia driver in the host it will grab the VGA arbiter lock and never release it.  The result will be that the first VGA access made by the guest for the Radeon device will hang.  There's a patch floating around to modify the nvidia driver wrapper to avoid this, perhaps someone can share a link.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#678 2013-11-04 19:13:33

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

There's a patch floating around to modify the nvidia driver wrapper to avoid this, perhaps someone can share a link.

That's actually in the first post, too under "ISSUES".


i'm sorry for my poor english wirting skills…

Offline

#679 2013-11-04 19:27:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have uploaded kernel 3.12.0 (includes acs override patch + i915 patches) and qemu-git (with NoSnoop patch) on the first post

Last edited by nbhs (2013-11-04 19:27:35)

Offline

#680 2013-11-04 19:55:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like some dudes got OSX mavericks working on qemu-kvm on intel hardware http://www.linux.org.ru/gallery/screenshots/9732976, also see: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/

Offline

#681 2013-11-04 20:03:07

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear all,

Could you please help to solve Nvidia driver issue on the guest (nouveau driver works OK, but I need Nvidia due to hardware video acceleration):

[ 42.409] Loading extension NV-GLX [ 44.409] (EE) NVIDIA(GPU-0): WAIT: (E, 0, 0x917d)

I receive the messages when I run startx and when I run xbmc via  "xinit /usr/bin/xbmc-standalone" I receive the following:

[ 5709.592] (EE) NVIDIA(0): The NVIDIA kernel module does not appear to be receiving [ 5709.592] (EE) NVIDIA(0): interrupts generated by the NVIDIA GPU at PCI:1:0:0. [ 5709.592] (EE) NVIDIA(0): Please see Chapter 8: Common Problems in the README for [ 5709.592] (EE) NVIDIA(0): additional information. [ 5709.592] (EE) NVIDIA(0): Failed to initialize the NVIDIA graphics device! [ 5709.592] (EE) NVIDIA(0): Failing initialization of X screen 0

lspci -v shows the following (IRQ 16 is shared between GPU and SMBus Controller)

00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) Subsystem: Red Hat, Inc Device 1100 Physical Slot: 31 Flags: fast devsel, IRQ 16 I/O ports at b100 [size=64] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GT 640] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 840e Physical Slot: 0 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at fc000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at e0000000 (64-bit, prefetchable) [size=32M] I/O ports at c000 [size=128] [virtual] Expansion ROM at fd000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: nvidia Kernel modules: nouveau, nvidia

Could you please help to "disable" sharing GPU IRQ?

Last edited by myweb (2013-11-04 21:40:31)

Offline

#682 2013-11-04 21:38:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear all,

Could you please help to solve Nvidia driver issue on the guest (nouveau driver works OK, but I need Nvidia due to hardware video acceleration):

[ 42.409] Loading extension NV-GLX [ 44.409] (EE) NVIDIA(GPU-0): WAIT: (E, 0, 0x917d)

I receive the messages when I run startx and when I run xbmc via  "xinit /usr/bin/xbmc-standalone" I receive the following:

[ 5709.592] (EE) NVIDIA(0): The NVIDIA kernel module does not appear to be receiving [ 5709.592] (EE) NVIDIA(0): interrupts generated by the NVIDIA GPU at PCI:1:0:0. [ 5709.592] (EE) NVIDIA(0): Please see Chapter 8: Common Problems in the README for [ 5709.592] (EE) NVIDIA(0): additional information. [ 5709.592] (EE) NVIDIA(0): Failed to initialize the NVIDIA graphics device! [ 5709.592] (EE) NVIDIA(0): Failing initialization of X screen 0

Have you tried NVreg_EnableMSI=1 as a nvidia module options?  Check /proc/interrupts on the host to see if we're seeing an interrupt there (grep vfio /proc/interrupts).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#683 2013-11-04 21:48:32

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sudo modprobe nvidia NVreg_EnableMSI=1

does not help

grep vfio /proc/interrupts

16: 5513 3621 960 109 IR-IO-APIC-fasteoi vfio-intx(0000:00:1a.0), vfio-intx(0000:01:00.0) 17: 393 209 261 120 IR-IO-APIC-fasteoi saa7146 (0), vfio-intx(0000:01:00.1)

0000:00:1a.0 - is USB controller
I run qemu with the following options

qemu-system-x86_64 -M q35 -m 4096 -enable-kvm -cpu host -vga none -nographic -boot menu=on \ -smp 4,sockets=4,cores=4,threads=4 \ -device ahci,bus=pcie.0,id=ahci \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -device vfio-pci,host=00:1a.0,bus=pcie.0 \ -drive file=/dev/sda6,id=rootDisk,format=raw -device ide-hd,bus=ahci.0,drive=rootDisk \ -drive file=/dev/sda7,id=swapDisk,format=raw -device ide-hd,bus=ahci.1,drive=swapDisk \ -netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0

I removed USB controller from qemu options and see the following in Xorg.log

[ 176.854] (II) NVIDIA(0): Validated MetaModes: [ 176.854] (II) NVIDIA(0): "DFP-0:nvidia-auto-select{}" [ 176.854] (II) NVIDIA(0): Virtual screen size determined to be 1920 x 1200 [ 176.889] (--) NVIDIA(0): DPI set to (90, 87); computed from "UseEdidDpi" X config [ 176.889] (--) NVIDIA(0): option [ 176.889] (--) Depth 24 pixmap format is 32 bpp [ 176.890] (II) NVIDIA: Using 3072.00 MB of virtual memory for indirect memory [ 176.890] (II) NVIDIA: access. [ 176.894] (II) NVIDIA(0): ACPI: failed to connect to the ACPI event daemon; the daemon [ 176.894] (II) NVIDIA(0): may not be running or the "AcpidSocketPath" X [ 176.894] (II) NVIDIA(0): configuration option may not be set correctly. When the [ 176.894] (II) NVIDIA(0): ACPI event daemon is available, the NVIDIA X driver will [ 176.894] (II) NVIDIA(0): try to use it to receive ACPI event notifications. For [ 176.894] (II) NVIDIA(0): details, please see the "ConnectToAcpid" and [ 176.894] (II) NVIDIA(0): "AcpidSocketPath" X configuration options in Appendix B: X [ 176.894] (II) NVIDIA(0): Config Options in the README. [ 176.896] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select{}" [ 179.920] Loading extension NV-GLX [ 181.920] (EE) NVIDIA(GPU-0): WAIT: (E, 0, 0x917d)

Last edited by myweb (2013-11-04 22:00:19)

Offline

#684 2013-11-04 22:01:45

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

like this:

kernel commandline parameter: pci-stub.ids=1002:6819,1002:aab0

the id's are not the PCI Bus IDs, but the Device Class/Subclass - you get them with "lspci -vn"

# lspci -vnn 03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:0612] Flags: bus master, fast devsel, latency 0, IRQ 47 I/O ports at d050 [size=8] I/O ports at d040 [size=4] I/O ports at d030 [size=8] I/O ports at d020 [size=4] I/O ports at d000 [size=32] Memory at f0d00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: ahci [...] 05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:0612] Flags: bus master, fast devsel, latency 0, IRQ 48 I/O ports at b050 [size=8] I/O ports at b040 [size=4] I/O ports at b030 [size=8] I/O ports at b020 [size=4] I/O ports at b000 [size=32] Memory at f0b00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: ahci [...] 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASMedia Technology Inc. Device [1b21:1060] Flags: bus master, fast devsel, latency 0, IRQ 49 I/O ports at a050 [size=8] I/O ports at a040 [size=4] I/O ports at a030 [size=8] I/O ports at a020 [size=4] I/O ports at a000 [size=32] Memory at f0910000 (32-bit, non-prefetchable) [size=512] Expansion ROM at f0900000 [disabled] [size=64K] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: ahci [...]

So you mean I should not use vendor:device [1b21:0612] but from the ?Subsystem? ID: [1b21:1060] as pci-stub kernelparameter?
Does it matter at this point, that it is a pci-express x1 card, not just pci?

Offline

#685 2013-11-04 22:20:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
sudo modprobe nvidia NVreg_EnableMSI=1

does not help

grep vfio /proc/interrupts

16: 5513 3621 960 109 IR-IO-APIC-fasteoi vfio-intx(0000:00:1a.0), vfio-intx(0000:01:00.0) 17: 393 209 261 120 IR-IO-APIC-fasteoi saa7146 (0), vfio-intx(0000:01:00.1)

You're still using INTx here.  Is that modprobe run from a clean guest boot without the nvidia module previously loaded?  Running modprobe against an already loaded module of course does nothing.  Does a Windows guest with Nvidia drivers work?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#686 2013-11-04 22:23:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:
kaeptnb wrote:

like this:

kernel commandline parameter: pci-stub.ids=1002:6819,1002:aab0

the id's are not the PCI Bus IDs, but the Device Class/Subclass - you get them with "lspci -vn"

So you mean I should not use vendor:device [1b21:0612] but from the ?Subsystem? ID: [1b21:1060] as pci-stub kernelparameter?
Does it matter at this point, that it is a pci-express x1 card, not just pci?

You should use vendor:device, that's what the -n option provides.  The first post is mistaken on what those IDs actually are.  PCIe vs conventional PCI doesn't matter


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#687 2013-11-04 22:34:47

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nvidia under Windows works fine. Modprobe run from a clean guest boot without the nvidia module previously loaded.

grep vfio /proc/interrupts 17: 572 239 284 142 IR-IO-APIC-fasteoi saa7146 (0), vfio-intx(0000:01:00.1) 49: 1 0 0 0 IR-PCI-MSI-edge vfio-msi[0](0000:01:00.0)

Offline

#688 2013-11-05 02:07:52

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello. I think I'm running into a problem regarding interrupt handling when trying to pass through my NVIDIA GTX 650 card into OS X Mavericks.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GTX 650] [10de:0fc6] (rev a1)

Booting OS X with stock QEMU and GTX 650 passed though results in extremly sluggish graphics. Every now and then Finder process gets killed with the following logged into dmesg:

NVDA(OpenGL): Channel exception! Exception type = 0x8 DMA Engine Error (FIFO Error 8)

However, passing through the same GTX 650 into Windows 7 install works fine. I've compared "cat /proc/interrupts | grep vfio" output after booting Windows 7 and OS X:

Windows 7:

 16: 405613 0 0 0 1 0 0 0 IR-IO-APIC-fasteoi vfio-intx(0000:01:00.0), firewire_ohci

OS X:

 67: 1 0 0 0 0 0 0 0 IR-PCI-MSI-edge vfio-msi[0](0000:01:00.0)

Also, when running OS X the following backtrace is occasionally seen on the host:

[35849.691253] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [35910.481384] vfio-pci 0000:01:00.0: irq 67 for MSI/MSI-X [36467.745050] irq 16: nobody cared (try booting with the "irqpoll" option) [36467.745054] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.12.0+ #8 [36467.745055] Hardware name: [36467.745056] 0000000000000000 ffffffff81390dff ffff880806f71a00 ffffffff8107d0bb [36467.745059] ffff880806f71a00 0000000000000000 ffff880806f71a00 ffffffff8107d463 [36467.745060] 0000000000000000 0000000000000010 ffff880806f71a00 0000000000000000 [36467.745062] Call Trace: [36467.745063] <IRQ> [<ffffffff81390dff>] ? dump_stack+0x41/0x51 [36467.745071] [<ffffffff8107d0bb>] ? __report_bad_irq+0x2c/0xb7 [36467.745072] [<ffffffff8107d463>] ? note_interrupt+0x14b/0x1ce [36467.745074] [<ffffffff8107b7a1>] ? handle_irq_event_percpu+0x120/0x187 [36467.745076] [<ffffffff8107b83a>] ? handle_irq_event+0x32/0x4d [36467.745078] [<ffffffff8107dce6>] ? handle_fasteoi_irq+0x80/0xb4 [36467.745080] [<ffffffff8100ebb2>] ? handle_irq+0x18/0x20 [36467.745081] [<ffffffff8100e82b>] ? do_IRQ+0x40/0x95 [36467.745083] [<ffffffff8139522d>] ? common_interrupt+0x6d/0x6d [36467.745084] <EOI> [<ffffffff812b5b3e>] ? arch_local_irq_enable+0x7/0x8 [36467.745087] [<ffffffff812b5e4e>] ? cpuidle_enter_state+0x50/0xa9 [36467.745089] [<ffffffff812b5f84>] ? cpuidle_idle_call+0xdd/0x132 [36467.745091] [<ffffffff810140ab>] ? arch_cpu_idle+0x5/0x17 [36467.745093] [<ffffffff8107af6b>] ? cpu_startup_entry+0x10c/0x165 [36467.745095] [<ffffffff816b8d33>] ? start_kernel+0x3e8/0x3f3 [36467.745097] [<ffffffff816b876e>] ? repair_env_string+0x54/0x54 [36467.745099] [<ffffffff816b8120>] ? early_idt_handlers+0x120/0x120 [36467.745100] [<ffffffff816b8593>] ? x86_64_start_kernel+0xef/0xf8 [36467.745101] handlers: [36467.745114] [<ffffffffa0144bdb>] irq_handler [firewire_ohci] [36467.745115] Disabling IRQ #16

My theory is that although QEMU/VFIO configures the card to use MSI, it continues to use conventional IRQ and OS X drivers never receive any interrupts. I couldn't find any option to disable MSI emulation when using VFIO, so I've tried the most obvious way of commenting out the following lines in hw/misc/vfio.c:

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index fdbb492..459c4ee 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -2800,14 +2804,14 @@ static int vfio_add_std_cap(VFIODevice *vdev, uint8_t pos) switch (cap_id) { case PCI_CAP_ID_MSI: - ret = vfio_setup_msi(vdev, pos); + ret = 0; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup vfio_setup_msi(vdev, pos); */ break; case PCI_CAP_ID_EXP: vfio_check_pcie_flr(vdev, pos); ret = vfio_setup_pcie_cap(vdev, pos, size); break; case PCI_CAP_ID_MSIX: - ret = vfio_setup_msix(vdev, pos); + ret = 0; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup vfio_setup_msix(vdev, pos); */ break; case PCI_CAP_ID_PM: vfio_check_pm_reset(vdev, pos);

After rebuilding QEMU with this patch applied interrupt handler switches to vfio-intx instead of vfio-msi and OS X graphics become smooth. Thus, in my case there is something wrong going on when MSI is being utilized.

Offline

#689 2013-11-05 03:14:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cryptonymous wrote:

My theory is that although QEMU/VFIO configures the card to use MSI, it continues to use conventional IRQ and OS X drivers never receive any interrupts. I couldn't find any option to disable MSI emulation when using VFIO, so I've tried the most obvious way of commenting out the following lines in hw/misc/vfio.c:

After rebuilding QEMU with this patch applied interrupt handler switches to vfio-intx instead of vfio-msi and OS X graphics become smooth. Thus, in my case there is something wrong going on when MSI is being utilized.

Interesting observation.  By removing those lines you're indeed not exposing MSI or MSI-X capabilities to the guest.  The guest has no choice but to use INTx.  I know I've used MSI successfully with a Quadro card and Linux guest, but I don't think I've tried a Linux guest with MSI on a GeForce (I had a Radeon running with MSI just the other day).  What does 'sudo lspci -vvv' from the host show for the GTX650 with OSX running without your qemu patch?  Random guess; if lspci shows DisINTx- try using setpci to enable it (ex. dev=01:00.0; val=$(setpci -s $dev COMMAND); newval=$(printf %04x $(( 0x$val | 0x400 ))); setpci -s $dev COMMAND=$newval; echo "restore with: setpci -s $dev COMMAND=$val").  If it's DisINTx- and making it DisINT+ makes any difference, perhaps there's a bug in the emulation of that bit.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#690 2013-11-05 04:48:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
cryptonymous wrote:

My theory is that although QEMU/VFIO configures the card to use MSI, it continues to use conventional IRQ and OS X drivers never receive any interrupts. I couldn't find any option to disable MSI emulation when using VFIO, so I've tried the most obvious way of commenting out the following lines in hw/misc/vfio.c:

After rebuilding QEMU with this patch applied interrupt handler switches to vfio-intx instead of vfio-msi and OS X graphics become smooth. Thus, in my case there is something wrong going on when MSI is being utilized.

Interesting observation.  By removing those lines you're indeed not exposing MSI or MSI-X capabilities to the guest.  The guest has no choice but to use INTx.  I know I've used MSI successfully with a Quadro card and Linux guest, but I don't think I've tried a Linux guest with MSI on a GeForce (I had a Radeon running with MSI just the other day).  What does 'sudo lspci -vvv' from the host show for the GTX650 with OSX running without your qemu patch?  Random guess; if lspci shows DisINTx- try using setpci to enable it (ex. dev=01:00.0; val=$(setpci -s $dev COMMAND); newval=$(printf %04x $(( 0x$val | 0x400 ))); setpci -s $dev COMMAND=$newval; echo "restore with: setpci -s $dev COMMAND=$val").  If it's DisINTx- and making it DisINT+ makes any difference, perhaps there's a bug in the emulation of that bit.

Testing a Linux guest w/ a GTX660, I can confirm that things go bad when using the NV_RegEnableMSI=1 option.  This worked fine for me with a Quadro card.  I see a single interrupt and the monitor goes out of sync and never comes back after OS bootup.  I'll try to find some time to dig into it.  I also note that the audio driver specifically checks for nvidia devices and blacklists MSI on snd-hda-intel, so at some point nvidia hasn't played nicely with Linux MSI support.  Perhaps there's a reason why the Linux nvidia driver defaults to INTx as well.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#691 2013-11-05 07:37:04

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

with the following options I get working Nvidia drivers in the guest:

options snd-hda-intel enable_msi=0 options nvidia NVreg_EnableMSI=0

Btw, I found interesting release notice from Nvidia for 325.08 BETA (http://www.nvidia.com/object/linux-disp … river.html)

"Changed the default PCIe interrupt delivery method from virtual-wire to MSI. Note that if the NVIDIA Linux driver fails to initialize with an error indicating that it is not receiving interrupts, MSI can be disabled by setting the module parameter "NVreg_EnableMSI=0" when loading the NVIDIA kernel module. - See more at: http://www.nvidia.com/object/linux-display-ia32-325.08-driver.html#sthash.bXXFjZEK.dpuf"

Last edited by myweb (2013-11-05 09:34:40)

Offline

#692 2013-11-05 11:12:35

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What does 'sudo lspci -vvv' from the host show for the GTX650 with OSX running without your qemu patch?  Random guess; if lspci shows DisINTx- try using setpci to enable it (ex. dev=01:00.0; val=$(setpci -s $dev COMMAND); newval=$(printf %04x $(( 0x$val | 0x400 ))); setpci -s $dev COMMAND=$newval; echo "restore with: setpci -s $dev COMMAND=$val").  If it's DisINTx- and making it DisINT+ makes any difference, perhaps there's a bug in the emulation of that bit.

When idling at OS X login screen with unpatched QEMU, 'sudo lspci -vvv -s 01:00.0' shows the following:

01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GTX 650] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. Device 2804 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 67 Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] Expansion ROM at f7000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00478 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+ Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900 v1] #19 Kernel driver in use: vfio-pci

So it has DisINTx+ flag. Changing it to DisINTx- via setpci does not make any visible difference. After rebooting the guest without exiting QEMU the flag is back to DisINTx+.

Offline

#693 2013-11-05 20:50:29

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After my system ran fine for a week or so now I dont get any sound from the vm at all again.
It starts up with this messages:

Nov 05 21:41:07 host windows-vm[1825]: ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave Nov 05 21:41:07 host windows-vm[1825]: alsa: Could not initialize DAC Nov 05 21:41:07 host windows-vm[1825]: alsa: Failed to open `default': Nov 05 21:41:07 host windows-vm[1825]: alsa: Reason: Device or resource busy Nov 05 21:41:07 host windows-vm[1825]: ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave Nov 05 21:41:07 host windows-vm[1825]: alsa: Could not initialize DAC Nov 05 21:41:07 host windows-vm[1825]: alsa: Failed to open `default': Nov 05 21:41:07 host windows-vm[1825]: alsa: Reason: Device or resource busy Nov 05 21:41:07 host windows-vm[1825]: audio: Failed to create voice `dac'

I tried googling the issue but I didnt find anything... Have any of you encountered that problem before?

Offline

#694 2013-11-08 00:30:35

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You may remember me from earlier in the thread, i tried this but gave up when i couldn't get Qemu to run. I'm back now with the AUR version of qemu-git and linux-mainline as well as the seabios provided here.

I have an Nvidia GTX 670 (01:00.0) and Nvidia GTX 550-Ti (02:00.0)

I have applied this to my cmdline: (The GTX 670 and its Audio)

pci-stub.ids=10de:1189,10de:0e0a

However i'm not actually sure if this is working or not, seeing as the computer is using this GPU when in terminal. When i try to start X if this option is on, i get an error that says the Nvidia Drivers are failing to load the GTX card (PCI 01:00.0), i even set the xorg configuration file to use "PCI:02:00:0" as BusID but that wont work either, and if i try to run the vfio-bind script for the GTX, i get an error just like the one i got the last time i tried to do this.

ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

I think last time setting

CONFIG_INTEL_IOMMU_DEFAULT_ON=y

fixed that specific error(i'm gonna try that again). But how am i to tell the Nvidia drivers to ignore the GTX 670?

Last edited by rabcor (2013-11-08 00:33:38)

Offline

#695 2013-11-08 00:47:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

You may remember me from earlier in the thread, i tried this but gave up when i couldn't get Qemu to run. I'm back now with the AUR version of qemu-git and linux-mainline as well as the seabios provided here.

I have an Nvidia GTX 670 (01:00.0) and Nvidia GTX 550-Ti (02:00.0)

I have applied this to my cmdline: (The GTX 670 and its Audio)

pci-stub.ids=10de:1189,10de:0e0a

However i'm not actually sure if this is working or not, seeing as the computer is using this GPU when in terminal. When i try to start X if this option is on, i get an error that says the Nvidia Drivers are failing to load the GTX card (PCI 01:00.0), i even set the xorg configuration file to use "PCI:02:00:0" as BusID but that wont work either, and if i try to run the vfio-bind script for the GTX, i get an error just like the one i got the last time i tried to do this.

ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

I think last time setting

CONFIG_INTEL_IOMMU_DEFAULT_ON=y

fixed that specific error(i'm gonna try that again). But how am i to tell the Nvidia drivers to ignore the GTX 670?

Since you're using intel make sure to use the packages i provided instead of the AUR ones, on intel systems you need to boot with intel_iommu=on, add pci-stub to /etc/mkinitcpio.conf in the MODULES section, and rebuild it

mkinitcpio -p linux-mainline

Make sure the Geforce 670 isnt your primary card, and if you're gonna use the nvidia drivers on the host you'll need to patch them (patch on the "issues" section)

Last edited by nbhs (2013-11-08 00:48:25)

Offline

#696 2013-11-08 00:58:33

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Since you're using intel make sure to use the packages i provided instead of the AUR ones, on intel systems you need to boot with intel_iommu=on, add pci-stub to /etc/mkinitcpio.conf in the MODULES section, and rebuild it

mkinitcpio -p linux-mainline

Make sure the Geforce 670 isnt your primary card, and if you're gonna use the nvidia drivers on the host you'll need to patch them (patch on the "issues" section)

by intel_iommu=on do you mean in the cmdline? something like this?

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=10de:1189,10de:0e0a quiet"

as for mkinitcpio.conf and the command, i already did that. I'll switch to your linux-mainline and qemu packages then.

Also, how do i make sure GeForce 670 isn't my primary card? does it have to do with which PCI port on the motherboard it is placed? or is it something i can configure somewhere? (bios only lets me select between iGPU/PCIe, it doesn't let me specify which PCIe slot is primary i think)

Last edited by rabcor (2013-11-08 01:01:43)

Offline

#697 2013-11-08 01:22:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
nbhs wrote:

Since you're using intel make sure to use the packages i provided instead of the AUR ones, on intel systems you need to boot with intel_iommu=on, add pci-stub to /etc/mkinitcpio.conf in the MODULES section, and rebuild it

mkinitcpio -p linux-mainline

Make sure the Geforce 670 isnt your primary card, and if you're gonna use the nvidia drivers on the host you'll need to patch them (patch on the "issues" section)

by intel_iommu=on do you mean in the cmdline? something like this?

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=10de:1189,10de:0e0a quiet"

as for mkinitcpio.conf and the command, i already did that. I'll switch to your linux-mainline and qemu packages then.

Also, how do i make sure GeForce 670 isn't my primary card? does it have to do with which PCI port on the motherboard it is placed? or is it something i can configure somewhere? (bios only lets me select between iGPU/PCIe, it doesn't let me specify which PCIe slot is primary i think)

Yes you have to add intel_iommu=on to your grub config file or whatever bootloader you use, your primary card is the one you see boot messages on

Offline

#698 2013-11-08 01:31:45

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've switched the slots of the GPUs, 550-Ti is now in the primary slot and 670 in the secondary slot (i see boot messages and stuff on the 550 now, it also seems to play better with my BIOS too, 670 had a resolution problem until official drivers were loaded) if i boot into windows(dualboot) like this strangely enough the drivers get confused (probably because the older/weaker card is in the primary slot)

I still get the same error in X11 though

(PCI 1:0:0 is now the 550-Ti which i want to be using in linux, and 670 is at 2:0:0 which i would like to use in the VM, i didn't tell my system to do anything with the 550)

Failed to initialize the NVIDIA GPU at PCI:1:0:0. Please check yuor system's kernel log for additional error messages... Screen(s) found, but none have a usable configuration.

it is quite the pain. i tried to install the linux-mainline you uploaded, it compiled but it didn't install for some reason, i'll have to try again hmm

Last edited by rabcor (2013-11-08 01:53:30)

Offline

#699 2013-11-08 11:04:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I've switched the slots of the GPUs, 550-Ti is now in the primary slot and 670 in the secondary slot (i see boot messages and stuff on the 550 now, it also seems to play better with my BIOS too, 670 had a resolution problem until official drivers were loaded) if i boot into windows(dualboot) like this strangely enough the drivers get confused (probably because the older/weaker card is in the primary slot)

I still get the same error in X11 though

(PCI 1:0:0 is now the 550-Ti which i want to be using in linux, and 670 is at 2:0:0 which i would like to use in the VM, i didn't tell my system to do anything with the 550)

Failed to initialize the NVIDIA GPU at PCI:1:0:0. Please check yuor system's kernel log for additional error messages... Screen(s) found, but none have a usable configuration.

it is quite the pain. i tried to install the linux-mainline you uploaded, it compiled but it didn't install for some reason, i'll have to try again hmm

First you have to build it:

makepkg -sf

Then install it:

sudo pacman -U linux-mainline-*

Offline

#700 2013-11-09 16:55:14

cmorrow
Member
Registered: 2013-11-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-git (with and without dsnoop patches) is giving me a Windows Stop Error (0x0000007E) when I boot the DVD (immediately after "Starting Windows" with no O/S installed yet) whenever I use q35 as the machine type. 

Anyone else seeing this?  I downgraded to qemu 1.6.1 and the issue went away.

(This is prior to any vfio passthrough, I just kept running into bluescreens and was trying to whittle down the configuration to barebones)
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga std \
-boot menu=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
-device piix4-ide,bus=pcie.0 \
-drive file=/home/windows.img,id=disk,format=raw -device ide-hd,drive=disk \
-drive file=/rpool/Public/ISO/Windows7HomePremium64bit.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Offline

#701 2013-11-09 18:05:21

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try using "-cpu qemu64" in stead of "-cpu host". That solved my issues.

Offline

#702 2013-11-09 18:18:29

cmorrow
Member
Registered: 2013-11-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Changing the -cpu option worked.  Thanks!

Offline

#703 2013-11-11 03:03:06

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

Hi,

thanks for this thread! Using this setup for a couple of years under XEN - but with KVM and VFIO its easier and - the most important part - more stable (with xen the Guest-Reboot was not always successful - with KVM it is!).

Anyways i converted most of the manual qemu options into libvirt-xml - main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually.
Another positive effect is Host Startup/Shutdown handling due to the libvirt init script - they stop/start your Guest on Host startup/shutdown.

<domain type='kvm' id='26' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>windows7</name> <uuid>xxxxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxx</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='6'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-1,4-5'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-1.5'>hvm</type> <boot dev='hd'/> <smbios mode='host'/> </os> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/ssd/win7system'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <source dev='/dev/mapper/win7games_cached'/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </disk> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0' model='none'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='00:xx:xx:xx:xx:xx'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <memballoon model='none'> <alias name='balloon0'/> </memballoon> </devices> <seclabel type='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.2,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1d.7,bus=pcie.0'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/> <qemu:env name='DISPLAY' value=':0'/> <qemu:env name='QEMU_PA_SAMPLES' value='128'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> </qemu:commandline> </domain>

Adapt UUID and MAC adress - and of course your PCI-IDs...

Greetings
K.

Thank 
I forgot  (alias name=) -.-'
But thanks to all of you to be able to do what I wanted to do 15 years ago
I would like to give my contribution to the cpu AMD FX, are able to work well with the double of the processes with respect to the number of cores
I have a fx 4100 will dedicate 3 cores for the guest but they will setup with hyper-treading
How distro I'm using gentoo

Offline

#704 2013-11-11 22:09:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Testing a Linux guest w/ a GTX660, I can confirm that things go bad when using the NV_RegEnableMSI=1 option.  This worked fine for me with a Quadro card.  I see a single interrupt and the monitor goes out of sync and never comes back after OS bootup.  I'll try to find some time to dig into it.  I also note that the audio driver specifically checks for nvidia devices and blacklists MSI on snd-hda-intel, so at some point nvidia hasn't played nicely with Linux MSI support.  Perhaps there's a reason why the Linux nvidia driver defaults to INTx as well.

Good news/bad news...

Good news: I figured out the problem and posted a patch that solves it on my GTX660: http://lists.nongnu.org/archive/html/qe … 01390.html

Bad news: This means that for every MSI interrupt, the Nvidia driver does a write to MMIO space at an offset that QEMU needs to trap in order to "virtualize" the device.  That may negate any performance benefit of MSI vs INTx.  As noted in the patch we might be able to use an ioeventfd to make handling more asynchronous, but the overhead is probably still more than INTx, which we can handle without exiting to QEMU userspace.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#705 2013-11-12 01:33:29

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:
kaeptnb wrote:

oh i overread the main part: you can sucessfully run qemu as a user manually!
that never worked for me, when i startet qemu as a user i got the same or similar messages like you.

Yup, that works fine. It just required to chown the /dev/vfio devices and the hugetlbfs area.

Having it in libvirt would be cosmetic. I have some LXC containers managed with libvirt, so this would put everything in a unique interface. But libvirt causes me a headache, no idea where it takes the RLIMIT_MEMLOCK value from. It seems to ignore limits.conf

I have the same problem , and change the user and group in the root on /etc/libvirt/qemu.conf and libvirtd.conf work fine no output the error -.-'
the same problem occurs even if you launch qemu with "sudo-u user qemu ......"

Offline

#706 2013-11-12 02:18:52

a123
Member
Registered: 2013-11-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all, I can passthrough my VGA to VM successfully using both pci-assign and vfio-pci. What's the difference between these two methods?

Offline

#707 2013-11-12 02:30:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

a123 wrote:

Hi all, I can passthrough my VGA to VM successfully using both pci-assign and vfio-pci. What's the difference between these two methods?

pci-assign is effectively deprecated upstream.  It has a poor device ownership model, it's a patchwork of security issues, it relies on userspace to protect the kernel against peer-to-peer isolation, and kvm would really like to get away from being a device driver and managing the iommu.  vfio-pci is meant to replace it.  Additionally, pci-assign doesn't actually have VGA support.  You can only use it to assign a graphics card as a secondary device to the VM.  If you can do this, great.  It avoids a number of issues with VGA.  vfio-pci should work equally well with that model, but also supports use as a primary VGA device with the x-vga=on option, which seems to enable quite a few more cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#708 2013-11-12 03:00:00

a123
Member
Registered: 2013-11-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
a123 wrote:

Hi all, I can passthrough my VGA to VM successfully using both pci-assign and vfio-pci. What's the difference between these two methods?

pci-assign is effectively deprecated upstream.  It has a poor device ownership model, it's a patchwork of security issues, it relies on userspace to protect the kernel against peer-to-peer isolation, and kvm would really like to get away from being a device driver and managing the iommu.  vfio-pci is meant to replace it.  Additionally, pci-assign doesn't actually have VGA support.  You can only use it to assign a graphics card as a secondary device to the VM.  If you can do this, great.  It avoids a number of issues with VGA.  vfio-pci should work equally well with that model, but also supports use as a primary VGA device with the x-vga=on option, which seems to enable quite a few more cards.

Does VGA passthrough and GPU passthrough mean the same thing?  Is there a way to show the display of video output by SDL or VNC without connectting a physical monitor to the passthroughed vga card?

Offline

#709 2013-11-12 03:08:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

a123 wrote:
aw wrote:
a123 wrote:

Hi all, I can passthrough my VGA to VM successfully using both pci-assign and vfio-pci. What's the difference between these two methods?

pci-assign is effectively deprecated upstream.  It has a poor device ownership model, it's a patchwork of security issues, it relies on userspace to protect the kernel against peer-to-peer isolation, and kvm would really like to get away from being a device driver and managing the iommu.  vfio-pci is meant to replace it.  Additionally, pci-assign doesn't actually have VGA support.  You can only use it to assign a graphics card as a secondary device to the VM.  If you can do this, great.  It avoids a number of issues with VGA.  vfio-pci should work equally well with that model, but also supports use as a primary VGA device with the x-vga=on option, which seems to enable quite a few more cards.

Does VGA passthrough and GPU passthrough mean the same thing?  Is there a way to show the display of video output by SDL or VNC without connectting a physical monitor to the passthroughed vga card?

To me, VGA passthrough means you're actually making use of VGA, which means it's the primary VM display including VM BIOS output.  GPU passthrough is without VGA, which mostly means a secondary VM display.  GPU passthrough might be a primary display in a legacy-free environment (UEFI).  VNC is only available using an in-band server (one running in the VM).  Out-of-band SDL/VNC is only for emulated graphics.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#710 2013-11-12 10:41:33

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone!

I`m yet another guy extremely interested in passing through my GPU to get rid of dual-boot once & for all. Thanks a lot for working on this, the effort is very much appreciated.

Can someone please confirm that using the instructions provided in the first post should be enough to pass through a NVidia GTX 680 card? Previously for Xen this was possible only after hardmodding the card into "Multi-OS capable" GRID K2 or Quadro K5000 card.

Offline

#711 2013-11-12 14:38:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Hi everyone!

I`m yet another guy extremely interested in passing through my GPU to get rid of dual-boot once & for all. Thanks a lot for working on this, the effort is very much appreciated.

Can someone please confirm that using the instructions provided in the first post should be enough to pass through a NVidia GTX 680 card? Previously for Xen this was possible only after hardmodding the card into "Multi-OS capable" GRID K2 or Quadro K5000 card.

It works with a GTX660 for me.  Depending on whether your primary host graphics is Intel, you may need to patch the host kernel for i915 VGA arbiter fixes.  Also if on an Intel system you'll need to patch QEMU to disable NoSnoop on the device.  I have a plan for solving the latter problem more permanently, but it won't come until v3.13 and QEMU 1.8.  The VGA arbiter problem is going to take longer.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#712 2013-11-12 14:45:38

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It works with a GTX660 for me.  Depending on whether your primary host graphics is Intel, you may need to patch the host kernel for i915 VGA arbiter fixes.  Also if on an Intel system you'll need to patch QEMU to disable NoSnoop on the device.  I have a plan for solving the latter problem more permanently, but it won't come until v3.13 and QEMU 1.8.  The VGA arbiter problem is going to take longer.

Thanks for reply, this gives me hope! I may get away without putting soldering iron to my GTX 680. Also KVM has a stronger appeal to me, I`ve used it in the past. I`m running Debian, but I don't see that as a show stopper; I`m also fine with patching the kernel and QEMU ( running with Intel HD 2500 as primary GPU ). I`ll definitely give it a shot.

Last edited by ilya80 (2013-11-12 14:56:37)

Offline

#713 2013-11-12 19:14:05

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some months and after getting a new GPU I wanted to try this again. This time I am using a AMD 7750 for the host and a NVIDIA GTX 470 for the guest.
When I tried to install windows 7 in my guest I got bsod showing "page fault in nonpaged area" but I solved that by changing "-cpu host" to "-cpu qemu64" (I think I only get bsod when I set memory higher than 1024 *confused*).
After lots of reboots I finally got what I wanted... big_smile

Thank you very much!

vmofdeath50sam.png

For the record:
I only installed the packages from the first post (qemu-git, linux-mainline, seabios) then added pci-id's to grub and blacklisted nouveau.

  • Xeon E3-1230v2

  • ASRock Fatal1ty Z77 professional

  • ATI Radeon 7750 (Host)

  • NVIDIA GTX-470 (Guest)

# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/main-root rw cryptdevice=/dev/sda2:main lang=de locale=de_DE.UTF-8 quiet intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:68f9,1002:aa68
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu qemu64 -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/zh32/MyMachines/windows2.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/home/zh32/Images/bie764129g.iso,id=isocd \ -device ide-cd,bus=ahci.1,drive=isocd \ -boot menu=on \ -usb -usbdevice host:04f3:0210 -usbdevice host:05ac:1006 -usbdevice host:05ac:0221

Offline

#714 2013-11-12 19:24:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:

After some months and after getting a new GPU I wanted to try this again. This time I am using a AMD 7750 for the host and a NVIDIA GTX 470 for the guest.
When I tried to install windows 7 in my guest I got bsod showing "page fault in nonpaged area" but I solved that by changing "-cpu host" to "-cpu qemu64" (I think I only get bsod when I set memory higher than 1024 *confused*).
After lots of reboots I finally got what I wanted... big_smile

Thank you very much!

Congrats, have you tried using -cpu SandyBridge ?

Offline

#715 2013-11-12 21:39:20

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!

I’m currently stuck at the first qemu-run where I’m supposed to see the SeaBIOS-Output on my third screen (plugged into the dedicated graphics card). Actually it flickers every time I execute

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

which means, I guess, that the device is successfully reset. The third display is in standby when booting the host but actually stays on after I first executed qemu. Also the main screen on the IGP (but not the secondary) shows artifacts when starting qemu but I can get rid of them by changing to tty2 and back to X. There are no error messages whatsoever, qemu runs until I kill it. I also tried to use a ROM file for the VGA bios but that didn’t change anything.

My setup:

ASRock Z87 Extreme6
Core i4771
HD4850

The HD4850 is for testing only since I’m waiting for R9 290X with a reasonable custom cooling system.

dmesg (relevant info only):

255:[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/-root rw cryptdevice=UUID=:_crypt intel_iommu=on pci-stub.ids=1002:9442,1002:aa30 256:[ 0.000000] Intel-IOMMU: enabled 315:[ 0.058278] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a 317:[ 0.058282] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da 320:[ 0.058347] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 614:[ 0.667449] IOMMU 0 0xfed90000: using Queued invalidation 615:[ 0.667450] IOMMU 1 0xfed91000: using Queued invalidation 616:[ 0.667451] IOMMU: Setting RMRR: 617:[ 0.667459] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] 621:[ 0.668724] IOMMU: Prepare 0-16MiB unity mapping for LPC 685:[ 0.696931] pci-stub: add 1002:9442 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 686:[ 0.696936] pci-stub 0000:02:00.0: claimed by stub 687:[ 0.696940] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 688:[ 0.696943] pci-stub 0000:02:00.1: claimed by stub 1125: [ 3436.413121] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

/sys/

$> ls /sys/kernel/iommu_groups/1/devices lrwxrwxrwx 1 root root 0 12. Nov 21:01 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 12. Nov 21:01 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 lrwxrwxrwx 1 root root 0 12. Nov 21:01 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 lrwxrwxrwx 1 root root 0 12. Nov 21:01 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 $> ls /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0 /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0: lrwxrwxrwx 1 root root 0 12. Nov 20:23 driver -> ../../../../bus/pci/drivers/vfio-pci lrwxrwxrwx 1 root root 0 12. Nov 21:01 iommu_group -> ../../../../kernel/iommu_groups/1 […] /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.1: lrwxrwxrwx 1 root root 0 12. Nov 20:26 driver -> ../../../../bus/pci/drivers/vfio-pci lrwxrwxrwx 1 root root 0 12. Nov 21:01 iommu_group -> ../../../../kernel/iommu_groups/1 […] $> ls /sys/bus/pci/drivers/vfio-pci/ lrwxrwxrwx 1 root root 0 12. Nov 23:04 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 lrwxrwxrwx 1 root root 0 12. Nov 23:04 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 --w------- 1 root root 4,0K 12. Nov 23:04 bind lrwxrwxrwx 1 root root 0 12. Nov 23:04 module -> ../../../../module/vfio_pci --w------- 1 root root 4,0K 12. Nov 20:04 new_id --w------- 1 root root 4,0K 12. Nov 23:04 remove_id --w------- 1 root root 4,0K 12. Nov 20:04 uevent --w------- 1 root root 4,0K 12. Nov 23:04 unbind

lspci -v

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4850] (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] MSI Radeon HD 4850 512MB GDDR3 Flags: fast devsel, IRQ 17 Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f0020000 (64-bit, non-prefetchable) [size=64K] I/O ports at e000 [size=256] Expansion ROM at f0000000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: vfio-pci Kernel modules: radeon 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] Flags: bus master, fast devsel, latency 0, IRQ 18 Memory at f0030000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

Unfortunately I cannot use another card since at least the first two PCIe-Ports are in the same iommu group.

I used linux-mainline (AUR), qemu-git (AUR) and seabios (from the first post).

Now I’m a bit stuck because I don’t see any error messages I could relate to this.

The following seems to be more of an (unrelated) IGP problem.

kernel: [drm:ring_stuck] *ERROR* Kicking stuck wait on blitter ring kernel: [drm] no progress on blitter ring kernel: [drm:i915_set_reset_status] *ERROR* blitter ring hung flushing bo (0x8aa000 ctx 0) at 0xfac kernel: [drm] stuck on render ring kernel: [drm:i915_set_reset_status] *ERROR* render ring hung inside bo (0xa309000 ctx 1) at 0xa3092d4 kernel: [drm:ring_stuck] *ERROR* Kicking stuck wait on blitter ring

Can you probably give me a hint where to look?

Last edited by blacky (2013-11-12 22:09:47)

Offline

#716 2013-11-12 22:16:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

Can you probably give me a hint where to look?

Use the packages i provided instead of the AUR ones

Offline

#717 2013-11-12 22:17:55

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

Hi!

I’m currently stuck at the first qemu-run where I’m supposed to see the SeaBIOS-Output on my third screen (plugged into the dedicated graphics card). Actually it flickers every time I execute

......

Can you probably give me a hint where to look?

Hi!

I guess your problems should be solved with the advice AW gave me:

"Depending on whether your primary host graphics is Intel, you may need to patch the host kernel for i915 VGA arbiter fixes. Also if on an Intel system you'll need to patch QEMU to disable NoSnoop on the device"

Links to the mentioned patches are in the updated first post of the topic. Do you have those patches already adopted?

Offline

#718 2013-11-12 22:20:20

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Use the packages i provided instead of the AUR ones

I thought that those were o.k. too since you linked them

nbhs wrote:

qemu-git.tar.gz (includes NoSnoop patch) OR qemu-git from AUR
seabios.tar.gz (1.7.3.2)
linux-mainline.tar.gz (3.12.0 includes acs override patch and i915 fixes) OR  linux-mainline from AUR

But of course I’ll follow your advise, thanks!

@ilya80: Oh, ok, didn’t catch that.

Offline

#719 2013-11-12 22:28:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

[snip] Also the main screen on the IGP (but not the secondary) shows artifacts when starting qemu but I can get rid of them by changing to tty2 and back to X.

sounds like lack of i915 VGA arbiter patch

blacky wrote:

[snip]
The following seems to be more of an (unrelated) IGP problem.

kernel: [drm:ring_stuck] *ERROR* Kicking stuck wait on blitter ring kernel: [drm] no progress on blitter ring kernel: [drm:i915_set_reset_status] *ERROR* blitter ring hung flushing bo (0x8aa000 ctx 0) at 0xfac kernel: [drm] stuck on render ring kernel: [drm:i915_set_reset_status] *ERROR* render ring hung inside bo (0xa309000 ctx 1) at 0xa3092d4 kernel: [drm:ring_stuck] *ERROR* Kicking stuck wait on blitter ring

Can you probably give me a hint where to look?

very related.  As others noted, mainline doesn't work for this yet.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#720 2013-11-13 00:33:44

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

-snip-

Congrats, have you tried using -cpu SandyBridge ?

Yes, I just reinstalled the machine using '-cpu SandyBridge' and with a own sata-controller. Works fine.

Offline

#721 2013-11-13 19:10:41

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

Bind the GPU:

vfio-bind 0000:07:00.0 0000:07:00.1

cat /etc/vfio-pci.cfg

DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1"

GL!


quoto this because I wish I could do the reverse, for when I shutdown the vm
because if I restart the vm I have very bad performance
I have to reboot the host because everything works well

Offline

#722 2013-11-13 19:24:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mafferri wrote:

quoto this because I wish I could do the reverse, for when I shutdown the vm
because if I restart the vm I have very bad performance
I have to reboot the host because everything works well

You can do the reverse.  Here are the scripts I use:

vfio-group <group #>:

#!/bin/sh if [ ! -e /sys/kernel/iommu_groups/$1 ]; then echo "IOMMU group $1 not found" exit 1 fi if [ ! -e /sys/bus/pci/drivers/vfio-pci ]; then sudo modprobe vfio-pci fi for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do if [ -e /sys/kernel/iommu_groups/$1/devices/$i/driver ]; then if [ "$(basename $(readlink -f \ /sys/kernel/iommu_groups/$1/devices/$i/driver))" != \ "pcieport" ]; then echo $i | sudo tee \ /sys/kernel/iommu_groups/$1/devices/$i/driver/unbind fi fi done for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do if [ ! -e /sys/kernel/iommu_groups/$1/devices/$i/driver ]; then VEN=$(cat /sys/kernel/iommu_groups/$1/devices/$i/vendor) DEV=$(cat /sys/kernel/iommu_groups/$1/devices/$i/device) echo $VEN $DEV | sudo tee \ /sys/bus/pci/drivers/vfio-pci/new_id fi done

vfio-ungroup <group #>

#!/bin/sh if [ ! -e /sys/kernel/iommu_groups/$1 ]; then echo "IOMMU group $1 not found" exit 1 fi for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do VEN=$(cat /sys/kernel/iommu_groups/$1/devices/$i/vendor) DEV=$(cat /sys/kernel/iommu_groups/$1/devices/$i/device) echo $VEN $DEV | sudo tee \ /sys/bus/pci/drivers/vfio-pci/remove_id echo $i | sudo tee \ /sys/kernel/iommu_groups/$1/devices/$i/driver/unbind done for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do echo $i | sudo tee /sys/bus/pci/drivers_probe done

lsgroup

#!/bin/sh BASE="/sys/kernel/iommu_groups" for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do GROUP=$(basename $i) echo "### Group $GROUP ###" for j in $(find $i/devices -type l); do DEV=$(basename $j) echo -n " " lspci -s $DEV done done

I'm not sure how this is going to help your performance though, it's pretty random whether re-loading the host driver is going to resolve anything that the guest driver can't.  For graphics devices, if you're using kernel+qemu that support bus reset, that should provide something nearly as good as a host reboot.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#723 2013-11-13 20:59:37

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi

First i Thank you for the tutorial.

I'm able to use two or three VM with reel GPU.

But i have a problem i want to use Virt Manager to start, stop, wake or add new USB passthroug but i'm not able to do that because of the difrence between qemu args and virt manager xml config.

Does any body have more luck with that ?

I'm on Ubuntu 13.10 with 3.12 kernel (with patch) qemu master (with patch).

Thanks

PS: I'm french

Offline

#724 2013-11-13 21:02:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Hi

First i Thank you for the tutorial.

I'm able to use two or three VM with reel GPU.

But i have a problem i want to use Virt Manager to start, stop, wake or add new USB passthroug but i'm not able to do that because of the difrence between qemu args and virt manager xml config.

Does any body have more luck with that ?

I'm on Ubuntu 13.10 with 3.12 kernel (with patch) qemu master (with patch).

Thanks

PS: I'm french

See this post by kaeptnb: https://bbs.archlinux.org/viewtopic.php … 0#p1343400

Would you mind posting your hardware specs?

Last edited by nbhs (2013-11-13 21:08:44)

Offline

#725 2013-11-13 21:30:10

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So i made some test with this methode but no luck. Virt Manager say to me a lot of difrente error.
Sometime AppArmor error, sometime connection reset by peer.

My Hardware is:

Corei5 4670S so HD4600 for the host
24Go ram (3*8Go)
Asrock Z87 Extrem 6
AMD HD7770 for the first VM
AMD HD5570 for the second VM
AMD HD5570 fot the third Vm

All of the graphic card work.

For the moment i've only Windows 7 on the first VM, the seconde and third come later.

Offline

#726 2013-11-13 21:45:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

So i made some test with this methode but no luck. Virt Manager say to me a lot of difrente error.
Sometime AppArmor error, sometime connection reset by peer.

My Hardware is:

Corei5 4670S so HD4600 for the host
24Go ram (3*8Go)
Asrock Z87 Extrem 6
AMD HD7770 for the first VM
AMD HD5570 for the second VM
AMD HD5570 fot the third Vm

All of the graphic card work.

For the moment i've only Windows 7 on the first VM, the seconde and third come later.

Well try disabling apparmor, you could also use the qemu monitor for your vm's, launch qemu with -monitor stdio

Offline

#727 2013-11-13 22:21:17

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:

Testing a Linux guest w/ a GTX660, I can confirm that things go bad when using the NV_RegEnableMSI=1 option.  This worked fine for me with a Quadro card.  I see a single interrupt and the monitor goes out of sync and never comes back after OS bootup.  I'll try to find some time to dig into it.  I also note that the audio driver specifically checks for nvidia devices and blacklists MSI on snd-hda-intel, so at some point nvidia hasn't played nicely with Linux MSI support.  Perhaps there's a reason why the Linux nvidia driver defaults to INTx as well.

Good news/bad news...

Good news: I figured out the problem and posted a patch that solves it on my GTX660: http://lists.nongnu.org/archive/html/qe … 01390.html

Bad news: This means that for every MSI interrupt, the Nvidia driver does a write to MMIO space at an offset that QEMU needs to trap in order to "virtualize" the device.  That may negate any performance benefit of MSI vs INTx.  As noted in the patch we might be able to use an ioeventfd to make handling more asynchronous, but the overhead is probably still more than INTx, which we can handle without exiting to QEMU userspace.

Thanks, "3950205 vfio-pci: Fix Nvidia MSI ACK through 0x88000 quirk" has fixed the interrupt issue on my GTX 650 in OS X guest.

Offline

#728 2013-11-13 22:37:37

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Well try disabling apparmor, you could also use the qemu monitor for your vm's, launch qemu with -monitor stdio

So i've disabled apparmor but it does not slove my problem.

I've tried a lot of differente .xml but some time it does not show in virt manager, sometime it's say Connection reset by peer (when i put pc-q35-1.7 instead of 440).

I'm completly lost ^^.

ps: my .xml

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit windows7test
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows7test</name>
  <uuid>79afd66a-f479-8c9c-c66f-2ac53f1c110c</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-1.7'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='host'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='monitor'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writethrough' io='native'/>
      <source file='/home/val/windows2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:4c:d9:7a'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=03:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/>
  </qemu:commandline>
</domain>

Last edited by Val532 (2013-11-13 23:03:54)

Offline

#729 2013-11-13 23:25:13

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using everything provided in post#1, including startup script, but as soon as I try to pass pci/pcie devices I get this error:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error opening /dev/vfio/9: No such file or directory qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9 qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized cset: --> deactivating/reseting shielding cset: **> shielding not active on system All Done!!

What does it mean, why does /dev/vfio/9 not exist?
What did I do wrong?

My versions of your scripts:
Assign PCI devices to pci-stub

modprobe pci_stub # 2. onboard Ethernetcontroller/Lankarte echo "8086 1539" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:04:00.0 > /sys/bus/pci/devices/0000:04:00.0/driver/unbind echo 0000:04:00.0 > /sys/bus/pci/drivers/pci-stub/bind # PCI bridge der PCIe Karte: echo "10b5 8603" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:06:00.0 > /sys/bus/pci/devices/0000:06:00.0/driver/unbind echo 0000:06:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:07:01.0 > /sys/bus/pci/devices/0000:07:01.0/driver/unbind echo 0000:07:01.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:07:02.0 > /sys/bus/pci/devices/0000:07:02.0/driver/unbind echo 0000:07:02.0 > /sys/bus/pci/drivers/pci-stub/bind # S-ATA Controller PCIe: echo "1b21 0612" > /sys/bus/pci/drivers/pci-stub/new_id echo "1b21 1060" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:08:00.0 > /sys/bus/pci/devices/0000:08:00.0/driver/unbind echo 0000:08:00.0 > /sys/bus/pci/drivers/pci-stub/bind # USB3 Controller der PCIe Karte: echo "1912 0015" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:09:00.0 > /sys/bus/pci/devices/0000:09:00.0/driver/unbind echo 0000:09:00.0 > /sys/bus/pci/drivers/pci-stub/bind

Startup Script

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) QEMU=/home/neocron/qemu-1.6.50/bin/qemu-system-x86_64 CSET=/home/neocron/qemu-git/cpuset-1.5.6/usr/bin/cset VCPUS="8" #must be > 0 < cpu core number CPU="Haswell" MEM="8192" MEM_PATH="" BIOS="/home/neocron/qemu-1.6.50/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" USB_DEVICES="" #PCI_DEVICES="00:11.0 04:00.0 05:00.0 06:00.0" PCI_DEVICES="04:00.0 06:00.0 07:01.0 07:02.0 08:00.0 09:00.0" #PCI_DEVICES="" GPU="01:00.0" GPU_SND="01:00.1" #SND=true SND=false SND_DRIVER_OPTS="QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170" #VBIOS="/home/neocron/qemu-1.6.50/share/qemu/vgabios-6950.bin" VBIOS="" CDROM="/home/neocron/Dokumente/KVM/Win7.SP1.AiO.PreActivated.Multilanguage/Win_7_AiO_Ar-En-Fr-Ge_Pre-Activated.iso" #HDD="/dev/mapper/vgRaidHDDs-win7ult" HDD="/media/BigStorage/Temp/windowskvm.img" #HDD="" QEMU_ARGS=" -M q35 -enable-kvm -monitor stdio" EXTRA_ARGS="-boot menu=on" if [ -n "$GPU" ]; then EXTRA_ARGS+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1,rombar=0,romfile=$VBIOS" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$MEM" ]; then MEMORY+="-m $MEM" if [ -n "$MEM_PATH" ]; then MEMORY+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi $CSET shield -c $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) set -m $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then $CSET shield --shield --pid $t taskset -pc $i $t let i++ fi fi done break fi done ) & fg %1 wait $CSET shield --reset echo "All Done!!"

lspci:

$ sudo lspci -vnn 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) Subsystem: ASRock Incorporation Device [1849:0c08] Flags: bus master, fast devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: e0000000-f00fffff Capabilities: [88] Subsystem: ASRock Incorporation Device [1849:0c01] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [a0] Express Root Port (Slot+), MSI 00 Capabilities: [100] Virtual Channel Capabilities: [140] Root Complex Link Capabilities: [d94] #19 Kernel driver in use: pcieport 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) (prog-if 00 [VGA controller]) Subsystem: ASRock Incorporation Device [1849:041a] Flags: bus master, fast devsel, latency 0, IRQ 59 Memory at f0400000 (64-bit, non-prefetchable) [size=4M] Memory at d0000000 (64-bit, prefetchable) [size=256M] I/O ports at f000 [size=64] Expansion ROM at <unassigned> [disabled] Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [d0] Power Management version 2 Capabilities: [a4] PCI Advanced Features Kernel driver in use: i915 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) Subsystem: ASRock Incorporation Device [1849:0c0c] Flags: bus master, fast devsel, latency 0, IRQ 67 Memory at f0e34000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00 Kernel driver in use: snd_hda_intel 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) (prog-if 30 [XHCI]) Subsystem: ASRock Incorporation Device [1849:8c31] Flags: bus master, medium devsel, latency 0, IRQ 50 Memory at f0e20000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Kernel driver in use: xhci_hcd 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) Subsystem: ASRock Incorporation Device [1849:8c3a] Flags: bus master, fast devsel, latency 0, IRQ 60 Memory at f0e3f000 (64-bit, non-prefetchable) [size=16] Capabilities: [50] Power Management version 3 Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+ Kernel driver in use: mei_me 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05) Subsystem: ASRock Incorporation Device [1849:153b] Flags: bus master, fast devsel, latency 0, IRQ 66 Memory at f0e00000 (32-bit, non-prefetchable) [size=128K] Memory at f0e3d000 (32-bit, non-prefetchable) [size=4K] I/O ports at f080 [disabled] [size=32] Capabilities: [c8] Power Management version 2 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [e0] PCI Advanced Features Kernel driver in use: e1000e 00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05) (prog-if 20 [EHCI]) Subsystem: ASRock Incorporation Device [1849:8c2d] Flags: bus master, medium devsel, latency 0, IRQ 16 Memory at f0e3c000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci-pci 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) Subsystem: ASRock Incorporation Device [1849:1151] Flags: bus master, fast devsel, latency 0, IRQ 68 Memory at f0e30000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: snd_hda_intel 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 Capabilities: [40] Express Root Port (Slot-), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: ASRock Incorporation Device [1849:8c10] Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f0d00000-f0dfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: ASRock Incorporation Device [1849:8c14] Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: f0c00000-f0cfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: ASRock Incorporation Device [1849:8c16] Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 0000b000-0000bfff Memory behind bridge: f0b00000-f0bfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: ASRock Incorporation Device [1849:8c18] Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 [8086:8c1a] (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=06, subordinate=09, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: f0800000-f0afffff Prefetchable memory behind bridge: 00000000cfa00000-00000000cfbfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: ASRock Incorporation Device [1849:8c1a] Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) (prog-if 20 [EHCI]) Subsystem: ASRock Incorporation Device [1849:8c26] Flags: bus master, medium devsel, latency 0, IRQ 23 Memory at f0e3b000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci-pci 00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05) Subsystem: ASRock Incorporation Device [1849:8c44] Flags: bus master, medium devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Device [1849:8c02] Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 46 I/O ports at f0d0 [size=8] I/O ports at f0c0 [size=4] I/O ports at f0b0 [size=8] I/O ports at f0a0 [size=4] I/O ports at f060 [size=32] Memory at f0e3a000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [70] Power Management version 3 Capabilities: [a8] SATA HBA v1.0 Kernel driver in use: ahci 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) Subsystem: ASRock Incorporation Device [1849:8c22] Flags: medium devsel, IRQ 18 Memory at f0e39000 (64-bit, non-prefetchable) [size=256] I/O ports at f040 [size=32] 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] [1002:6819] (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device [1458:2553] Flags: fast devsel, IRQ 16 Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f0000000 (64-bit, non-prefetchable) [size=256K] I/O ports at e000 [size=256] Expansion ROM at f0040000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] #13 Capabilities: [2d0] #1b Kernel driver in use: vfio-pci 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] Subsystem: Gigabyte Technology Co., Ltd Device [1458:aab0] Flags: fast devsel, IRQ 17 Memory at f0060000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci 03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:0612] Flags: bus master, fast devsel, latency 0, IRQ 47 I/O ports at d050 [size=8] I/O ports at d040 [size=4] I/O ports at d030 [size=8] I/O ports at d020 [size=4] I/O ports at d000 [size=32] Memory at f0d00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: ahci 04:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) Subsystem: ASRock Incorporation Device [1849:1539] Flags: fast devsel, IRQ 19 Memory at f0c00000 (32-bit, non-prefetchable) [size=128K] I/O ports at c000 [disabled] [size=32] Memory at f0c20000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable- Count=5 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number bc-5f-f4-ff-ff-cd-17-77 Capabilities: [1a0] Transaction Processing Hints Kernel driver in use: pci-stub 05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:0612] Flags: bus master, fast devsel, latency 0, IRQ 48 I/O ports at b050 [size=8] I/O ports at b040 [size=4] I/O ports at b030 [size=8] I/O ports at b020 [size=4] I/O ports at b000 [size=32] Memory at f0b00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: ahci 06:00.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) (prog-if 00 [Normal decode]) Flags: fast devsel Memory at f0a00000 (32-bit, non-prefetchable) [size=16K] Bus: primary=06, secondary=07, subordinate=09, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: f0800000-f09fffff Prefetchable memory behind bridge: 00000000cfa00000-00000000cfbfffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. Device [10b5:8603] Capabilities: [100] Device Serial Number aa-86-02-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting <?> Capabilities: [148] Virtual Channel Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?> Kernel driver in use: pci-stub 07:01.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) (prog-if 00 [Normal decode]) Flags: fast devsel Bus: primary=07, secondary=08, subordinate=08, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: f0900000-f09fffff Prefetchable memory behind bridge: 00000000cfa00000-00000000cfbfffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. Device [10b5:8603] Capabilities: [100] Device Serial Number aa-86-02-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?> Kernel driver in use: pci-stub 07:02.0 PCI bridge [0604]: PLX Technology, Inc. Device [10b5:8603] (rev aa) (prog-if 00 [Normal decode]) Flags: fast devsel Bus: primary=07, secondary=09, subordinate=09, sec-latency=0 Memory behind bridge: f0800000-f08fffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. Device [10b5:8603] Capabilities: [100] Device Serial Number aa-86-02-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=028 <?> Kernel driver in use: pci-stub 08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASMedia Technology Inc. Device [1b21:1060] Flags: fast devsel, IRQ 18 I/O ports at a050 [size=8] I/O ports at a040 [size=4] I/O ports at a030 [size=8] I/O ports at a020 [size=4] I/O ports at a000 [size=32] Memory at f0910000 (32-bit, non-prefetchable) [size=512] Expansion ROM at f0900000 [disabled] [size=64K] Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [78] Power Management version 3 Capabilities: [80] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Kernel driver in use: pci-stub 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) (prog-if 30 [XHCI]) Subsystem: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] Flags: fast devsel, IRQ 19 Memory at f0800000 (64-bit, non-prefetchable) [size=8K] Capabilities: [50] Power Management version 3 Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [90] MSI-X: Enable- Count=8 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Latency Tolerance Reporting Kernel driver in use: pci-stub

Offline

#730 2013-11-13 23:29:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

...
Assign PCI devices to pci-stub
...

You have to assign your devices to vfio-pci not pci-stub

Offline

#731 2013-11-13 23:33:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Norcoen wrote:

...
Assign PCI devices to pci-stub
...

You have to assign your devices to vfio-pci not pci-stub

And do not bind bridges to anything, leave them attached to pcieport.  Maybe start with something simple, like one device instead of practically everything in the box.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#732 2013-11-13 23:36:17

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mafferri wrote:

quoto this because I wish I could do the reverse, for when I shutdown the vm
because if I restart the vm I have very bad performance
I have to reboot the host because everything works well

You can do the reverse.  Here are the scripts I use:

vfio-group <group #>:

#!/bin/sh if [ ! -e /sys/kernel/iommu_groups/$1 ]; then echo "IOMMU group $1 not found" exit 1 fi if [ ! -e /sys/bus/pci/drivers/vfio-pci ]; then sudo modprobe vfio-pci fi for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do if [ -e /sys/kernel/iommu_groups/$1/devices/$i/driver ]; then if [ "$(basename $(readlink -f \ /sys/kernel/iommu_groups/$1/devices/$i/driver))" != \ "pcieport" ]; then echo $i | sudo tee \ /sys/kernel/iommu_groups/$1/devices/$i/driver/unbind fi fi done for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do if [ ! -e /sys/kernel/iommu_groups/$1/devices/$i/driver ]; then VEN=$(cat /sys/kernel/iommu_groups/$1/devices/$i/vendor) DEV=$(cat /sys/kernel/iommu_groups/$1/devices/$i/device) echo $VEN $DEV | sudo tee \ /sys/bus/pci/drivers/vfio-pci/new_id fi done

vfio-ungroup <group #>

#!/bin/sh if [ ! -e /sys/kernel/iommu_groups/$1 ]; then echo "IOMMU group $1 not found" exit 1 fi for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do VEN=$(cat /sys/kernel/iommu_groups/$1/devices/$i/vendor) DEV=$(cat /sys/kernel/iommu_groups/$1/devices/$i/device) echo $VEN $DEV | sudo tee \ /sys/bus/pci/drivers/vfio-pci/remove_id echo $i | sudo tee \ /sys/kernel/iommu_groups/$1/devices/$i/driver/unbind done for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do echo $i | sudo tee /sys/bus/pci/drivers_probe done

lsgroup

#!/bin/sh BASE="/sys/kernel/iommu_groups" for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do GROUP=$(basename $i) echo "### Group $GROUP ###" for j in $(find $i/devices -type l); do DEV=$(basename $j) echo -n " " lspci -s $DEV done done

I'm not sure how this is going to help your performance though, it's pretty random whether re-loading the host driver is going to resolve anything that the guest driver can't.  For graphics devices, if you're using kernel+qemu that support bus reset, that should provide something nearly as good as a host reboot.

much more convenient to handle thanks;)

Offline

#733 2013-11-13 23:50:53

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your insanely fast reply smile
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9 qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

Using the lsgroup script you just posted I got this:

$ ./lsgroup.sh ### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 06:00.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) 07:01.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) 07:02.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) 08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

Why is there so much in group 9? From what I know, I can only passthrough complete groups.
But why would my PCIe Card be in the same group as a whole lot of onboard stuff?
Or is this the same problem as I had some pages back, because the PCIe Card has the same vendor:device ids as the onboard asmedia chip? (I think vfio-bind.sh automaticly gets them)

Offline

#734 2013-11-14 00:05:01

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Thank you for your insanely fast reply smile
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9 qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

Using the lsgroup script you just posted I got this:

$ ./lsgroup.sh ### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 06:00.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) 07:01.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) 07:02.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) 08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

Why is there so much in group 9? From what I know, I can only passthrough complete groups.
But why would my PCIe Card be in the same group as a whole lot of onboard stuff?
Or is this the same problem as I had some pages back, because the PCIe Card has the same vendor:device ids as the onboard asmedia chip? (I think vfio-bind.sh automaticly gets them)

If you built the kernel i provided try this workarround: https://bbs.archlinux.org/viewtopic.php … 5#p1342995

Offline

#735 2013-11-14 00:40:36

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, I just built 3.12 from kernel.org with the config changes

 CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

I'll try building your kernel. Guess that will take some time, since I have to patch it manually. (And built my first kernel a week ago...^^)

Offline

#736 2013-11-14 00:48:55

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

I'll try building your kernel. Guess that will take some time, since I have to patch it manually. (And built my first kernel a week ago...^^)

If you're using arch,  just unpack it, open a console then build it like this:

makepkg -s

then install it:

sudo pacman -U linux-mainline-*

Same for the qemu and seabios packages i linked.

Last edited by nbhs (2013-11-14 00:49:27)

Offline

#737 2013-11-14 02:16:44

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Thank you for your insanely fast reply smile
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9 qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

resolve this , add the group /dev/vfio/9 in the /etc/libvirt/qemu.conf  edit: I thought you were the one who posted the libvirt domain
and "chmod 666 /dev/vfio/16 /dev/vfio/17 /dev/vfio/7 /dev/vfio/vfio " only this test

Last edited by mafferri (2013-11-14 02:18:24)

Offline

#738 2013-11-14 02:22:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mafferri wrote:
Norcoen wrote:

Thank you for your insanely fast reply smile
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9 qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

resolve this , add the group /dev/vfio/9 in the /etc/libvirt/qemu.conf  edit: I thought you were the one who posted the libvirt domain
and "chmod 666 /dev/vfio/16 /dev/vfio/17 /dev/vfio/7 /dev/vfio/vfio " only this test

That's not the problem, the error message indicates how to fix it.  Norcoen needs the ACS override patch and to boot with an option to prune the group down to something manageable.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#739 2013-11-14 05:32:33

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I work this night to make able to use virt-manager with GPU passtrough and it work now !!!

You need libvirt 1.1.4 and Virt-Manage 0.10.

Offline

#740 2013-11-14 14:16:06

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys!

Need a bit of help here. My config:

Asrock Z68 Extreme 3 Gen 3, latest BIOS, Vt-d enabled in bios
Intel i5 3470, with HD 2500 as primary adapter, used in host system
NVidia GTX 680 as secondary adapter, nouveau blacklisted in host system - intended for passthrough
Running fresh Debian sid

I started on the whole thing, so far I`ve done the following:

1) Downloaded 3.12 source, applied "i915_fixes.patch" and "override_for_missing_acs_capabilities.patch", built with VFIO compiled into the kernel and rebooted. Tried both with and without "vfio_iommu_type1.allow_unsafe_interrupts=1" in kernel command line.
2) Cloned qemu git, applied "vfio_disallow_device_from_using_nosnoop_transactions.patch", built and installed via "make install"
3) Cloned seabios and built, have no use for it yet though.

My problem is that when I try to bind the GPU using "sudo vfio-bind 0000:01:00.0 0000:01:00.0", it fails with the following in /var/log/kern.log:

Nov 14 17:54:18 cave-lin kernel: [ 145.427257] vfio-pci: probe of 0000:01:00.0 failed with error -22 Nov 14 17:54:18 cave-lin kernel: [ 145.472245] vfio-pci: probe of 0000:01:00.0 failed with error -22 Nov 14 17:54:18 cave-lin kernel: [ 145.472252] vfio-pci: probe of 0000:01:00.1 failed with error -22

I`m sure device is not yet claimed by any driver, "sudo lspci -k" gives:

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) Subsystem: NVIDIA Corporation Device 0969 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) Subsystem: NVIDIA Corporation Device 0969

Any pointers on what I have missed / where I should look ?

Thanks!

Offline

#741 2013-11-14 14:20:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Hi guys!

Need a bit of help here. My config:

Asrock Z68 Extreme 3 Gen 3, latest BIOS, Vt-d enabled in bios
Intel i5 3470, with HD 2500 as primary adapter, used in host system
NVidia GTX 680 as secondary adapter, nouveau blacklisted in host system - intended for passthrough
Running fresh Debian sid

I started on the whole thing, so far I`ve done the following:

1) Downloaded 3.12 source, applied "i915_fixes.patch" and "override_for_missing_acs_capabilities.patch", built with VFIO compiled into the kernel and rebooted. Tried both with and without "vfio_iommu_type1.allow_unsafe_interrupts=1" in kernel command line.
2) Cloned qemu git, applied "vfio_disallow_device_from_using_nosnoop_transactions.patch", built and installed via "make install"
3) Cloned seabios and built, have no use for it yet though.

My problem is that when I try to bind the GPU using "sudo vfio-bind 0000:01:00.0 0000:01:00.0", it fails with the following in /var/log/kern.log:

Nov 14 17:54:18 cave-lin kernel: [ 145.427257] vfio-pci: probe of 0000:01:00.0 failed with error -22 Nov 14 17:54:18 cave-lin kernel: [ 145.472245] vfio-pci: probe of 0000:01:00.0 failed with error -22 Nov 14 17:54:18 cave-lin kernel: [ 145.472252] vfio-pci: probe of 0000:01:00.1 failed with error -22

I`m sure device is not yet claimed by any driver, "sudo lspci -k" gives:

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) Subsystem: NVIDIA Corporation Device 0969 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) Subsystem: NVIDIA Corporation Device 0969

Any pointers on what I have missed / where I should look ?

Thanks!

Boot with intel_iommu=on


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#742 2013-11-14 14:36:42

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Boot with intel_iommu=on

That did it, thanks! Both devices now successfully managed by vfio-pci. I`ll follow the guide in the first post. Maybe you could provide few clarifications: should I copy seabios binaries produced after building it somewhere into qemu? Or just try firing up qemu?

Offline

#743 2013-11-14 14:46:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
aw wrote:

Boot with intel_iommu=on

That did it, thanks! Both devices now successfully managed by vfio-pci. I`ll follow the guide in the first post. Maybe you could provide few clarifications: should I copy seabios binaries produced after building it somewhere into qemu? Or just try firing up qemu?

Build it somewhere then fire qemu with qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin ...

Offline

#744 2013-11-14 15:06:42

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Build it somewhere then fire qemu with qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin ...

Right, now I read it. Okay this requires me looking at the monitors, so I`ll have to come home first.

Thanks!

Offline

#745 2013-11-14 15:07:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The bios provided by qemu should be fine.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#746 2013-11-14 16:49:11

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

If you're using arch,  just unpack it, open a console then build it like this:

makepkg -s

then install it:

sudo pacman -U linux-mainline-*

Same for the qemu and seabios packages i linked.

Unfortunately I'm stuck with Fedora and/or CentOS if I don't want to use Windows here.
I already built the qemu and seabios packages you provided (~2 weeks ago, have there been any changes I need in the meantime?)

Now I just read the code in the PKGBUILD file and did the patching manually. No errors so far. But I'm kinda stuck on the .config file.
I copied the oldconfig from /boot/config/3.11.7 and compared it to your config.x86_64.

I think (hope) it's save, to leave configs on, that arent specified in your config, for example:
Your config: # CONFIG_KALLSYMS_ALL is not set
My oldconfig: CONFIG_KALLSYMS_ALL=y
Result: CONFIG_KALLSYMS_ALL=y

I think this also applies the other way around, when you set it, but it's not specified in my oldconfig yet:
Your config: CONFIG_TICK_CPU_ACCOUNTING=y
My oldconfig: # CONFIG_TICK_CPU_ACCOUNTING is not set
Result: CONFIG_TICK_CPU_ACCOUNTING=y

I hope I'm right to this point. Now what do I do with differences like this:

Your Config: My oldconfig: # # # RCU Subsystem # RCU Subsystem # # CONFIG_TREE_PREEMPT_RCU=y | CONFIG_TREE_RCU=y CONFIG_PREEMPT_RCU=y | # CONFIG_PREEMPT_RCU is not set CONFIG_RCU_STALL_COMMON=y CONFIG_RCU_STALL_COMMON=y

Last edited by Norcoen (2013-11-14 16:54:44)

Offline

#747 2013-11-14 16:52:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The fedora kernel config is fine to use as a starting point.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#748 2013-11-14 16:58:25

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But why isn't CONFIG_TREE_PREEMPT_RCU present in the fedora kernel config, while CONFIG_TREE_RCU isn't present in nbhs' config?
Aren't those related somehow? Or is it complety irrelevant?

Again, thank you for those insanely fast replies smile

Last edited by Norcoen (2013-11-14 17:04:30)

Offline

#749 2013-11-14 16:59:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

But why isn't CONFIG_TREE_PREEMPT_RCU present in the fedora kernel config, while CONFIG_TREE_RCU is present isn't present in nbhs' config?
AreN't those related somehow? Or is it complety irrelevant?

Again, thank you for those insanely fast replies smile

Completely irrelevant to vga assignment


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#750 2013-11-14 18:41:29

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's working!
Booting kernel 3.12 with the provided patches and "pcie_acs_override=downstream" as a parameter, lsgroup now gives me this output:

$ ./lsgroup.sh ### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 10 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) ### Group 11 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 12 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) ### Group 13 ### 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) ### Group 14 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 15 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) ### Group 16 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 17 ### 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 18 ### 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) ### Group 19 ### 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 20 ### 06:00.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) ### Group 21 ### 07:01.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) ### Group 22 ### 07:02.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa) ### Group 23 ### 08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 24 ### 09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

I can boot the Windows7 image with this. It's working fine as far as I can tell. Thank you so much smile

I got an other little problem now, because Win7 does not recognize mouse and keyboard on the pcie-controller until I install the drivers for it, I need to passthrough my hosts usb devices.

$ lsusb Bus 002 Device 002: ID 8087:8000 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:8008 Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 013: ID 046d:c227 Logitech, Inc. G15 Refresh Keyboard Bus 003 Device 011: ID 046d:c226 Logitech, Inc. G15 Refresh Keyboard Bus 003 Device 010: ID 1532:0016 Razer USA, Ltd DeathAdder Mouse Bus 003 Device 009: ID 10d5:1234 Uni Class Technology Co., Ltd Bus 003 Device 008: ID 0424:2524 Standard Microsystems Corp. USB MultiSwitch Hub Bus 003 Device 002: ID 174c:2074 ASMedia Technology Inc. Bus 003 Device 007: ID 046d:c227 Logitech, Inc. G15 Refresh Keyboard Bus 003 Device 006: ID 046d:c226 Logitech, Inc. G15 Refresh Keyboard Bus 003 Device 005: ID 1532:0016 Razer USA, Ltd DeathAdder Mouse Bus 003 Device 004: ID 10d5:1234 Uni Class Technology Co., Ltd Bus 003 Device 003: ID 0424:2524 Standard Microsystems Corp. USB MultiSwitch Hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Mouse and keyboard are 2x in this list, because both of the kvm-switch cables are connected to the host.
I want to passthrough the secondary ones, so the line in the startscript looks like this:

USB_DEVICES="host:046d:c226 host:046d:c227 host:1532:0016"

giving me this output+error:

$ ./startWin7.sh cset: **> CPUSPEC "-4-3" has bad group "-4-3" $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS QEMU 1.6.50 monitor - type 'help' for more information (qemu) cset: **> shielding not active on system taskset: Ungültige Option -- 4 Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]] Options: -a, --all-tasks operate on all the tasks (threads) for a given pid -p, --pid operate on existing given pid -c, --cpu-list display and specify cpus in list format -h, --help display this help -V, --version output version information The default behavior is to run a new command: taskset 03 sshd -b 1024 You can retrieve the mask of an existing task: taskset -p 700 Or set it: taskset -p 03 700 List format uses a comma-separated list instead of a mask: taskset -pc 0,3,7-11 700 Ranges in list format can take a stride argument: e.g. 0-31:2 is equivalent to mask 0x55555555 For more information see taskset(1). cset: **> shielding not active on system taskset: Ungültige Option -- 3 Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]] Options: -a, --all-tasks operate on all the tasks (threads) for a given pid -p, --pid operate on existing given pid -c, --cpu-list display and specify cpus in list format -h, --help display this help -V, --version output version information The default behavior is to run a new command: taskset 03 sshd -b 1024 You can retrieve the mask of an existing task: taskset -p 700 Or set it: taskset -p 03 700 List format uses a comma-separated list instead of a mask: taskset -pc 0,3,7-11 700 Ranges in list format can take a stride argument: e.g. 0-31:2 is equivalent to mask 0x55555555 For more information see taskset(1). cset: **> shielding not active on system taskset: Ungültige Option -- 2 Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]] Options: -a, --all-tasks operate on all the tasks (threads) for a given pid -p, --pid operate on existing given pid -c, --cpu-list display and specify cpus in list format -h, --help display this help -V, --version output version information The default behavior is to run a new command: taskset 03 sshd -b 1024 You can retrieve the mask of an existing task: taskset -p 700 Or set it: taskset -p 03 700 List format uses a comma-separated list instead of a mask: taskset -pc 0,3,7-11 700 Ranges in list format can take a stride argument: e.g. 0-31:2 is equivalent to mask 0x55555555 For more information see taskset(1). cset: **> shielding not active on system taskset: Ungültige Option -- 1 Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]] Options: -a, --all-tasks operate on all the tasks (threads) for a given pid -p, --pid operate on existing given pid -c, --cpu-list display and specify cpus in list format -h, --help display this help -V, --version output version information The default behavior is to run a new command: taskset 03 sshd -b 1024 You can retrieve the mask of an existing task: taskset -p 700 Or set it: taskset -p 03 700 List format uses a comma-separated list instead of a mask: taskset -pc 0,3,7-11 700 Ranges in list format can take a stride argument: e.g. 0-31:2 is equivalent to mask 0x55555555 For more information see taskset(1). cset: **> shielding not active on system pid 6715's current affinity list: 0-7 pid 6715's new affinity list: 0 qemu: could not add USB device 'host:046d:c226' cset: **> shielding not active on system taskset: failed to get pid 6755's affinity: Kein passender Prozess gefunden cset: **> shielding not active on system taskset: failed to get pid 6804's affinity: Kein passender Prozess gefunden cset: **> shielding not active on system taskset: failed to get pid 6853's affinity: Kein passender Prozess gefunden cset: --> deactivating/reseting shielding cset: **> shielding not active on system All Done!!

Do I Need to unbind them somehow? Like the PCI devices?
Before I just selected them in virt-manager and they've been passed through without manually unbinding them.
Or is my syntax incorrect?

//edit:
ah, now I see that vendor:device IDs are the same again, like the problem I had with the sata-controller.
Is it possible to be more specific, on which device I want to pass?
virt-manager defines it in this xml structure:

<hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> <address bus='3' device='15'/> </source> </hostdev>

//edit2:
I found this: http://www.linux-kvm.org/page/USB_Host_ … d_to_Guest
I tried changing the startscript from

USB_DEVICES="host:046d:c226 host:046d:c227 host:1532:0016"

to:

USB_DEVICES="usb-host,hostbus=3,hostaddr=10 usb-host,hostbus=3,hostaddr=11 usb-host,hostbus=3,hostaddr=13"

but that did not work either.

Last edited by Norcoen (2013-11-14 19:02:23)

Offline

#751 2013-11-14 18:49:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to lower your vcpu count or my script will break, thats why you're seeing all those error

Ex: change VCPUS=8 to VCPUS=6

Last edited by nbhs (2013-11-14 19:02:54)

Offline

#752 2013-11-14 19:05:43

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

//edit:
ah, now I see that vendor:device IDs are the same again, like the problem I had with the sata-controller.
Is it possible to be more specific, on which device I want to pass?
virt-manager defines it in this xml structure:

<hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> <address bus='3' device='15'/> </source> </hostdev>

//edit2:
I found this: http://www.linux-kvm.org/page/USB_Host_ … d_to_Guest
I tried changing the startscript from

USB_DEVICES="host:046d:c226 host:046d:c227 host:1532:0016"

to:

USB_DEVICES="usb-host,hostbus=3,hostaddr=10 usb-host,hostbus=3,hostaddr=11 usb-host,hostbus=3,hostaddr=13"

but that did not work either.

I changed the above things and I lowered the count to 4.
[do I have to set VCPUS="7" to use all 8 cores (4+4 HT)?]

Now this is my output:

$ sudo ./startWin7.sh cset: --> activating shielding: cset: moving 368 tasks from root into system cpuset... [==================================================]% cset: "system" cpuset of CPUSPEC(4-7) with 368 tasks running cset: "user" cpuset of CPUSPEC(0-3) with 0 tasks running $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS QEMU 1.6.50 monitor - type 'help' for more information (qemu) cset: --> shielding following pidspec: 7983 cset: done pid 7983's current affinity list: 0-3 pid 7983's new affinity list: 0 cset: --> shielding following pidspec: 7984 cset: done pid 7984's current affinity list: 0-3 pid 7984's new affinity list: 1 cset: --> shielding following pidspec: 8029 cset: done pid 8029's current affinity list: 0-3 pid 8029's new affinity list: 2 qemu: could not add USB device 'usb-host,hostbus=3,hostaddr=10' cset: --> shielding following pidspec: 8070 cset: **> skipped 1 task, not in origination set "system" cset: --> hint: perhaps use --force if sure of command cset: **> tasks do not match all criteria, none moved taskset: failed to get pid 8070's affinity: Kein passender Prozess gefunden cset: --> deactivating/reseting shielding cset: moving 0 tasks from "/user" user set to root set... cset: moving 368 tasks from "/system" system set to root set... [==================================================]% cset: deleting "/user" and "/system" sets cset: done All Done!!

Offline

#753 2013-11-14 19:19:48

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys!

I`m able to pass through GTX 680 no problems, and able to shutdown / restart VM as many times as I want. I get qemu bios on the card ouptut.
I`m still far away from full featured windows inside linux, but this is awesome stuff.

Thanks a lot, especially nbhs and aw!

PS. Did a quick installation of Windows 7, install went fine but I got a blue screen after reboot. I guess there will be one or two hiccups still smile

PPS: Must be a ahci / ide controller mismatch. I`m getting it w/o passed through VGA ( dont have the time to see the code ) I`ll try more tomorrow.

Last edited by ilya80 (2013-11-14 20:05:09)

Offline

#754 2013-11-14 19:23:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:
Norcoen wrote:

//edit:
ah, now I see that vendor:device IDs are the same again, like the problem I had with the sata-controller.
Is it possible to be more specific, on which device I want to pass?
virt-manager defines it in this xml structure:

<hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> <address bus='3' device='15'/> </source> </hostdev>

//edit2:
I found this: http://www.linux-kvm.org/page/USB_Host_ … d_to_Guest
I tried changing the startscript from

USB_DEVICES="host:046d:c226 host:046d:c227 host:1532:0016"

to:

USB_DEVICES="usb-host,hostbus=3,hostaddr=10 usb-host,hostbus=3,hostaddr=11 usb-host,hostbus=3,hostaddr=13"

but that did not work either.

I changed the above things and I lowered the count to 4.
[do I have to set VCPUS="7" to use all 8 cores (4+4 HT)?]

Now this is my output:

$ sudo ./startWin7.sh cset: --> activating shielding: cset: moving 368 tasks from root into system cpuset... [==================================================]% cset: "system" cpuset of CPUSPEC(4-7) with 368 tasks running cset: "user" cpuset of CPUSPEC(0-3) with 0 tasks running $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS QEMU 1.6.50 monitor - type 'help' for more information (qemu) cset: --> shielding following pidspec: 7983 cset: done pid 7983's current affinity list: 0-3 pid 7983's new affinity list: 0 cset: --> shielding following pidspec: 7984 cset: done pid 7984's current affinity list: 0-3 pid 7984's new affinity list: 1 cset: --> shielding following pidspec: 8029 cset: done pid 8029's current affinity list: 0-3 pid 8029's new affinity list: 2 qemu: could not add USB device 'usb-host,hostbus=3,hostaddr=10' cset: --> shielding following pidspec: 8070 cset: **> skipped 1 task, not in origination set "system" cset: --> hint: perhaps use --force if sure of command cset: **> tasks do not match all criteria, none moved taskset: failed to get pid 8070's affinity: Kein passender Prozess gefunden cset: --> deactivating/reseting shielding cset: moving 0 tasks from "/user" user set to root set... cset: moving 368 tasks from "/system" system set to root set... [==================================================]% cset: deleting "/user" and "/system" sets cset: done All Done!!

Try using the commandline instead of my script,  perhaps you built qemu without usb support?

Last edited by nbhs (2013-11-14 19:23:36)

Offline

#755 2013-11-14 20:30:40

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to rebuilt qemu as provided in the first post now.
I'm a bit stuck with configure now.

I'm getting this error:

ERROR: pa check failed Make sure to have the pa libs and headers installed.

using

./configure --prefix=/home/user/qemu-1.7.r29486.g5c5432e --sysconfdir=/etc --audio-drv-list='pa alsa sdl oss' \ --python=/usr/bin/python2 --smbd=/usr/bin/smbd \ --enable-docs --libexecdir=/usr/lib/qemu \ --disable-gtk --enable-linux-aio --enable-seccomp \ --enable-spice --localstatedir=/var

Do I need "pa" - what does it do?
[//edit: I guess pa means pulseaudio in context with alsa and oss... could have figured that out earlier...] - I have pulseaudi installed, guess I need to check the libs.
Also, what do I set my --libexecdir to, with my given prefix? Or is it just fine?
I don't have /usr/bin/smbd, is that the samba daemon?

Last edited by Norcoen (2013-11-14 20:43:16)

Offline

#756 2013-11-14 20:42:42

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi i need help ^^.

When i start my VM with :

sudo qemu-system-x86_64 -M q35 -m 8G -enable-kvm -cpu Haswell -vga none -nographic -parallel none -boot menu=on \
-bios /usr/share/qemu/bios.bin \
-smp 4,sockets=1,cores=4,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-drive file=/home/val/windows2,id=disk \
-device ide-hd,bus=ahci.0,drive=disk \
-usb -usbdevice host:046d:c019 -usbdevice host:0603:00f2 \
-net nic,model=e1000 -net tap,ifname=tap0

All work ok.

But if i start my VM with Virt-Manager with this xml :

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows7-test2
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows7-test2</name>
  <uuid>652e4382-4dc8-7349-cdbf-359d333aba08</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-1.7'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='host'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='rtm'/>
    <feature policy='require' name='invpcid'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='bmi2'/>
    <feature policy='require' name='smep'/>
    <feature policy='require' name='avx2'/>
    <feature policy='require' name='hle'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='fsgsbase'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='require' name='rdtscp'/>
    <feature policy='require' name='avx'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='aes'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='popcnt'/>
    <feature policy='require' name='movbe'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='sse4.2'/>
    <feature policy='require' name='sse4.1'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='fma'/>
    <feature policy='require' name='pclmuldq'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writethrough' io='native'/>
      <source file='/home/val/windows2'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:07:1c:aa'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0603'/>
        <product id='0x00f2'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc019'/>
      </source>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='bios.bin'/>
  </qemu:commandline>
</domain>

I get a BSOD 0x00000116  atikmpag.sys if i remove the ati driver, the VM work good but no graphique acceleration.

Any idea ?

Edit: I found the problem to slove it juste put clear_emulator_capabilities=0 in qemu.conf

Last edited by Val532 (2013-11-14 21:00:27)

Offline

#757 2013-11-14 20:44:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

I'm trying to rebuilt qemu as provided in the first post now.
I'm a bit stuck with configure now.

I'm getting this error:

ERROR: pa check failed Make sure to have the pa libs and headers installed.

using

./configure --prefix=/home/user/qemu-1.7.r29486.g5c5432e --sysconfdir=/etc --audio-drv-list='pa alsa sdl oss' \ --python=/usr/bin/python2 --smbd=/usr/bin/smbd \ --enable-docs --libexecdir=/usr/lib/qemu \ --disable-gtk --enable-linux-aio --enable-seccomp \ --enable-spice --localstatedir=/var

Do I need "pa" - what does it do?
Also, what do I set my --libexecdir to, with my given prefix? Or is it just fine?
I don't have /usr/bin/smbd, is that the samba daemon?

pa is pulseaudio, you have to install the pulseaudio-libs package i think

Offline

#758 2013-11-14 20:53:08

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I needed the lib-pulseaudio-devel package, also alsa and oss devel libs after that...^^

//edit2:
After some more missing libs, configure succeeded.
Now when I'm trying to compile it with male I'm getting this SDL error:

$ make GEN alpha-softmmu/config-devices.mak GEN arm-softmmu/config-devices.mak GEN cris-softmmu/config-devices.mak GEN i386-softmmu/config-devices.mak GEN lm32-softmmu/config-devices.mak GEN m68k-softmmu/config-devices.mak GEN microblazeel-softmmu/config-devices.mak GEN microblaze-softmmu/config-devices.mak GEN mips64el-softmmu/config-devices.mak GEN mips64-softmmu/config-devices.mak GEN mipsel-softmmu/config-devices.mak GEN mips-softmmu/config-devices.mak GEN moxie-softmmu/config-devices.mak GEN or32-softmmu/config-devices.mak GEN ppc64-softmmu/config-devices.mak GEN ppcemb-softmmu/config-devices.mak GEN ppc-softmmu/config-devices.mak GEN s390x-softmmu/config-devices.mak GEN sh4eb-softmmu/config-devices.mak GEN sh4-softmmu/config-devices.mak GEN sparc64-softmmu/config-devices.mak GEN sparc-softmmu/config-devices.mak GEN unicore32-softmmu/config-devices.mak GEN x86_64-softmmu/config-devices.mak GEN xtensaeb-softmmu/config-devices.mak GEN xtensa-softmmu/config-devices.mak GEN alpha-linux-user/config-devices.mak GEN armeb-linux-user/config-devices.mak GEN arm-linux-user/config-devices.mak GEN cris-linux-user/config-devices.mak GEN i386-linux-user/config-devices.mak GEN m68k-linux-user/config-devices.mak GEN microblazeel-linux-user/config-devices.mak GEN microblaze-linux-user/config-devices.mak GEN mips64el-linux-user/config-devices.mak GEN mips64-linux-user/config-devices.mak GEN mipsel-linux-user/config-devices.mak GEN mips-linux-user/config-devices.mak GEN mipsn32el-linux-user/config-devices.mak GEN mipsn32-linux-user/config-devices.mak GEN or32-linux-user/config-devices.mak GEN ppc64abi32-linux-user/config-devices.mak GEN ppc64-linux-user/config-devices.mak GEN ppc-linux-user/config-devices.mak GEN s390x-linux-user/config-devices.mak GEN sh4eb-linux-user/config-devices.mak GEN sh4-linux-user/config-devices.mak GEN sparc32plus-linux-user/config-devices.mak GEN sparc64-linux-user/config-devices.mak GEN sparc-linux-user/config-devices.mak GEN unicore32-linux-user/config-devices.mak GEN x86_64-linux-user/config-devices.mak GEN config-all-devices.mak GEN config-host.h DEP tests/dumptrees.c DEP tests/trees.S DEP tests/testutils.c DEP tests/value-labels.c DEP tests/asm_tree_dump.c DEP tests/truncated_property.c DEP tests/path_offset_aliases.c DEP tests/add_subnode_with_nops.c DEP tests/dtbs_equal_unordered.c DEP tests/dtb_reverse.c DEP tests/dtbs_equal_ordered.c DEP tests/extra-terminating-null.c DEP tests/incbin.c DEP tests/boot-cpuid.c DEP tests/phandle_format.c DEP tests/path-references.c DEP tests/references.c DEP tests/string_escapes.c DEP tests/del_node.c DEP tests/del_property.c DEP tests/setprop.c DEP tests/set_name.c DEP tests/rw_tree1.c DEP tests/open_pack.c DEP tests/nopulate.c DEP tests/mangle-layout.c DEP tests/move_and_save.c DEP tests/sw_tree1.c DEP tests/nop_node.c DEP tests/nop_property.c DEP tests/setprop_inplace.c DEP tests/notfound.c DEP tests/get_alias.c DEP tests/node_offset_by_compatible.c DEP tests/node_check_compatible.c DEP tests/node_offset_by_phandle.c DEP tests/node_offset_by_prop_value.c DEP tests/parent_offset.c DEP tests/supernode_atdepth_offset.c DEP tests/get_path.c DEP tests/get_phandle.c DEP tests/getprop.c DEP tests/get_name.c DEP tests/path_offset.c DEP tests/subnode_offset.c DEP tests/find_property.c DEP tests/root_node.c DEP tests/get_mem_rsv.c DEP libfdt/fdt_strerror.c DEP libfdt/fdt_rw.c DEP libfdt/fdt_sw.c DEP libfdt/fdt_wip.c DEP libfdt/fdt_ro.c DEP libfdt/fdt.c DEP ftdump.c LEX convert-dtsv0-lexer.lex.c DEP convert-dtsv0-lexer.lex.c DEP util.c DEP srcpos.c BISON dtc-parser.tab.c DEP dtc-parser.tab.c LEX dtc-lexer.lex.c DEP dtc-lexer.lex.c DEP treesource.c DEP livetree.c DEP fstree.c DEP flattree.c DEP dtc.c DEP data.c DEP checks.c CHK version_gen.h UPD version_gen.h DEP dtc.c CHK version_gen.h CC libfdt/fdt.o CC libfdt/fdt_ro.o CC libfdt/fdt_wip.o CC libfdt/fdt_sw.o CC libfdt/fdt_rw.o CC libfdt/fdt_strerror.o AR libfdt/libfdt.a ar: creating libfdt/libfdt.a a - libfdt/fdt.o a - libfdt/fdt_ro.o a - libfdt/fdt_wip.o a - libfdt/fdt_sw.o a - libfdt/fdt_rw.o a - libfdt/fdt_strerror.o GEN qemu-options.def GEN qmp-commands.h GEN qapi-types.h GEN qapi-visit.h GEN trace/generated-events.h GEN trace/generated-tracers.h GEN tests/test-qapi-types.h GEN tests/test-qapi-visit.h GEN tests/test-qmp-commands.h CHK version_gen.h CC tests/qemu-iotests/socket_scm_helper.o LINK tests/qemu-iotests/socket_scm_helper lt CC stubs/arch-query-cpu-def.lo lt CC stubs/clock-warp.lo lt CC stubs/cpu-get-clock.lo lt CC stubs/cpu-get-icount.lo lt CC stubs/dump.lo lt CC stubs/fdset-add-fd.lo lt CC stubs/fdset-find-fd.lo lt CC stubs/fdset-get-fd.lo lt CC stubs/fdset-remove-fd.lo lt CC stubs/gdbstub.lo lt CC stubs/get-fd.lo lt CC stubs/get-vm-name.lo lt CC stubs/iothread-lock.lo lt CC stubs/migr-blocker.lo lt CC stubs/mon-is-qmp.lo lt CC stubs/mon-printf.lo lt CC stubs/mon-print-filename.lo lt CC stubs/mon-protocol-event.lo lt CC stubs/mon-set-error.lo lt CC stubs/pci-drive-hot-add.lo lt CC stubs/reset.lo lt CC stubs/set-fd-handler.lo lt CC stubs/slirp.lo lt CC stubs/sysbus.lo lt CC stubs/uuid.lo lt CC stubs/vm-stop.lo lt CC stubs/vmstate.lo lt CC stubs/cpus.lo lt CC libcacard/cac.lo lt CC libcacard/event.lo lt CC libcacard/vcard.lo lt CC libcacard/vreader.lo lt CC libcacard/vcard_emul_nss.lo lt CC libcacard/vcard_emul_type.lo lt CC libcacard/card_7816.lo lt CC libcacard/vcardt.lo lt CC util/osdep.lo lt CC util/cutils.lo lt CC util/qemu-timer-common.lo lt CC util/error.lo lt CC util/qemu-error.lo lt CC util/oslib-posix.lo lt CC util/qemu-thread-posix.lo GEN trace/generated-events.c lt CC trace/generated-events.lo lt CC trace/default.lo lt CC trace/control.lo GEN trace/generated-tracers.c lt CC trace/generated-tracers.lo lt LINK libcacard.la GEN libcacard.pc GEN qemu-options.texi GEN qemu-monitor.texi GEN qemu-img-cmds.texi GEN qemu-doc.html GEN qemu-tech.html GEN qemu.1 GEN qemu-img.1 GEN qemu-nbd.8 GEN qmp-commands.txt GEN fsdev/virtfs-proxy-helper.1 GEN qapi-types.c GEN qga/qapi-generated/qga-qapi-types.h GEN qga/qapi-generated/qga-qapi-visit.h GEN qga/qapi-generated/qga-qmp-commands.h CC qapi-types.o GEN qapi-visit.c CC qapi-visit.o CC qga/commands.o CC qga/guest-agent-command-state.o CC qga/main.o CC qga/commands-posix.o CC qga/channel-posix.o GEN qga/qapi-generated/qga-qapi-types.c CC qga/qapi-generated/qga-qapi-types.o GEN qga/qapi-generated/qga-qapi-visit.c CC qga/qapi-generated/qga-qapi-visit.o GEN qga/qapi-generated/qga-qmp-marshal.c CC qga/qapi-generated/qga-qmp-marshal.o CC qapi/qapi-visit-core.o CC qapi/qapi-dealloc-visitor.o CC qapi/qmp-input-visitor.o CC qapi/qmp-output-visitor.o CC qapi/qmp-registry.o CC qapi/qmp-dispatch.o CC qapi/string-input-visitor.o CC qapi/string-output-visitor.o CC qapi/opts-visitor.o CC qobject/qint.o CC qobject/qstring.o CC qobject/qdict.o CC qobject/qlist.o CC qobject/qfloat.o CC qobject/qbool.o CC qobject/qjson.o CC qobject/json-lexer.o CC qobject/json-streamer.o CC qobject/json-parser.o CC qobject/qerror.o CC trace/generated-events.o CC util/unicode.o CC util/event_notifier-posix.o CC util/qemu-openpty.o CC util/envlist.o CC util/path.o CC util/host-utils.o CC util/cache-utils.o CC util/module.o CC util/bitmap.o CC util/bitops.o CC util/hbitmap.o CC util/fifo8.o CC util/acl.o CC util/compatfd.o CC util/iov.o CC util/aes.o CC util/qemu-config.o CC util/qemu-sockets.o CC util/uri.o CC util/notify.o CC util/qemu-option.o CC util/qemu-progress.o CC util/hexdump.o CC util/crc32c.o CC util/throttle.o AR libqemuutil.a AR libqemustub.a LINK qemu-ga CC qemu-nbd.o CC async.o CC thread-pool.o CC nbd.o CC block.o CC blockjob.o CC main-loop.o CC iohandler.o CC qemu-timer.o CC aio-posix.o CC qemu-io-cmds.o CC qemu-coroutine.o CC qemu-coroutine-lock.o CC qemu-coroutine-io.o CC qemu-coroutine-sleep.o CC coroutine-ucontext.o CC block/raw_bsd.o CC block/cow.o CC block/qcow.o CC block/vdi.o CC block/vmdk.o CC block/cloop.o CC block/dmg.o CC block/bochs.o CC block/vpc.o CC block/vvfat.o CC block/qcow2.o CC block/qcow2-refcount.o CC block/qcow2-cluster.o CC block/qcow2-snapshot.o CC block/qcow2-cache.o CC block/qed.o CC block/qed-gencb.o CC block/qed-l2-cache.o CC block/qed-table.o CC block/qed-cluster.o CC block/qed-check.o CC block/parallels.o CC block/blkdebug.o CC block/blkverify.o CC block/snapshot.o CC block/qapi.o CC block/raw-posix.o CC block/linux-aio.o CC block/nbd.o CC block/sheepdog.o CC block/curl.o LINK qemu-nbd GEN qemu-img-cmds.h CC qemu-img.o LINK qemu-img CC qemu-io.o LINK qemu-io CC fsdev/virtfs-proxy-helper.o CC fsdev/virtio-9p-marshal.o LINK fsdev/virtfs-proxy-helper CC libcacard/vscclient.o lt LINK vscclient CC qemu-bridge-helper.o LINK qemu-bridge-helper CC blockdev.o CC blockdev-nbd.o CC readline.o CC qdev-monitor.o CC device-hotplug.o CC os-posix.o CC migration.o CC migration-tcp.o CC qemu-char.o CC block-migration.o CC page_cache.o CC xbzrle.o CC migration-exec.o CC migration-unix.o CC migration-fd.o CC spice-qemu-char.o CC bt-host.o CC bt-vhci.o CC dma-helpers.o CC vl.o CC tpm.o CC qemu-seccomp.o GEN qmp-marshal.c CC qmp-marshal.o CC qmp.o CC hmp.o CC qemu-log.o CC tcg-runtime.o CC audio/audio.o CC audio/noaudio.o CC audio/wavaudio.o CC audio/mixeng.o CC audio/sdlaudio.o audio/sdlaudio.c:24:17: schwerwiegender Fehler: SDL.h: Datei oder Verzeichnis nicht gefunden #include <SDL.h> ^ Kompilierung beendet. make: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_3.html topic_4.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html [audio/sdlaudio.o] Fehler 1

I have installed all the sdl-devel libs. What am I missing?

Last edited by Norcoen (2013-11-14 21:17:06)

Offline

#759 2013-11-14 21:26:46

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Couldnt hold myself from playing around a bit more big_smile

Installed windows, initial BSOD was because of -cpu host , needed -cpu SandyBridge instead.

Installed NVidia drivers but they give me Error 43. I think aw had more patches around that might solve this. Also GTX 680 is now my secondary GPU, this might be the problem.

Offline

#760 2013-11-14 21:30:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

I have installed all the sdl-devel libs. What am I missing?

Most distros have some sort of build dependency tool, ex:

yum-builddep qemu
apt-get build-dep qemu

It's generally a better starting point that recursively trying to build and installing missing dependencies.  I've never used arch, but I'd guess it has a similar tool


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#761 2013-11-14 21:33:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Couldnt hold myself from playing around a bit more big_smile

Installed windows, initial BSOD was because of -cpu host , needed -cpu SandyBridge instead.

Installed NVidia drivers but they give me Error 43. I think aw had more patches around that might solve this. Also GTX 680 is now my secondary GPU, this might be the problem.

Code 43 error - http://lists.nongnu.org/archive/html/qe … 01417.html

This didn't end up being the final solution, but it works for Geforce cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#762 2013-11-14 21:42:57

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Code 43 error - http://lists.nongnu.org/archive/html/qe … 01417.html
This didn't end up being the final solution, but it works for Geforce cards.

Isnt this vfio_disallow_device_from_using_nosnoop_transactions.patch ? I have pulled qemu from git and applied this patch successfully before testing.

Is there any way to debug this?

Last edited by ilya80 (2013-11-14 22:24:25)

Offline

#763 2013-11-15 03:44:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
aw wrote:

Code 43 error - http://lists.nongnu.org/archive/html/qe … 01417.html
This didn't end up being the final solution, but it works for Geforce cards.

Isnt this vfio_disallow_device_from_using_nosnoop_transactions.patch ? I have pulled qemu from git and applied this patch successfully before testing.

Is there any way to debug this?

Yes, those are the same patches.  Code 43 is pretty difficult to debug, google for it and you'll find videos of people baking their GPU to try to re-flow the solder.  If you have the nosnoop patch, looks at the GPU in the host with 'lspci -vvv' while Windows is running.  You should see NoSnoop with a + or - next to it.  If it's - then the nosnoop patch is doing it's job.  If it's + I can point you at some other patches.  Otherwise I'd suggest comparing your VM setup versus others that have reported success with similar cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#764 2013-11-15 07:48:10

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Right I got to check it while Windows is running.

Last edited by ilya80 (2013-11-15 08:23:16)

Offline

#765 2013-11-15 09:44:25

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, when VM is running, Nvidia VGA Adapter has NoSnoop-, but Nvidia HDMI Audio has NoSnoop+

Can that affect the drivers in guest?

Also, I wonder would saving the card BIOS and then supplying it to qemu make any difference?

Thanks,
Ilya.

Last edited by ilya80 (2013-11-15 11:25:36)

Offline

#766 2013-11-15 11:38:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Also GTX 680 is now my secondary GPU, this might be the problem.

Have you disabled the emulated vga?

Offline

#767 2013-11-15 11:44:37

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
ilya80 wrote:

Also GTX 680 is now my secondary GPU, this might be the problem.

Have you disabled the emulated vga?

Yes, I`ve got -vga none in qemu command line. Without it passthrough doesnt work I think. Here's how I start qemu:

kalujny@cave-lin:~$ sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -vga none -bios /opt/kalujny/kernel/seabios/out/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/opt/kalujny/thrash/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -usbdevice tablet -usb -usbdevice host:046d:c311 -net nic -net user VNC server running on `::1:5900'

Last edited by ilya80 (2013-11-15 11:46:28)

Offline

#768 2013-11-15 14:20:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

OK, when VM is running, Nvidia VGA Adapter has NoSnoop-, but Nvidia HDMI Audio has NoSnoop+

Can that affect the drivers in guest?

Also, I wonder would saving the card BIOS and then supplying it to qemu make any difference?

I don't know if these will make a difference, but should eliminate the question of whether the problem is related to NoSnoop.  This is the current upstream solution.  Kernel:
https://git.kernel.org/cgit/linux/kerne … a429732ef4
https://git.kernel.org/cgit/linux/kerne … 860dae4d56
https://git.kernel.org/cgit/linux/kerne … 7612b7f235

QEMU:
http://lists.nongnu.org/archive/html/qe … 00090.html
http://lists.nongnu.org/archive/html/qe … 00091.html

Remove the previous QEMU nosnoop patch with these fixes.  I doubt saving the card BIOS would make any difference.  If you get seabios output to the monitor and the windows boot animation then it's working correctly.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#769 2013-11-15 14:22:40

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot! I`ll try these and see what comes out of it.

There's always an option of hardmodding the card into GRID K2

Offline

#770 2013-11-15 17:14:22

gentoo-nut
Member
Registered: 2013-11-14
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello all

I am very close to a fully working VFIO passthrough.  I had been using the legacy KVM pci-assign in the past but it's just not reliable enough - having to reboot the host is unnacceptable to me.  nbhs, I believe we know each other from a certain other forum, under a certain other aliases -- thank you for your reply the other day.  I am making good progress.

I am using all of the software versions and patches linked in the first post.  The windows guest works perfectly, I can stop and restart it many times with the x-vga card and the host thankfully remains quite healthy.  I do not have any error messages in dmesg or the qemu logs, I believe everything is in good shape.

I'm just having an issue getting the guest to behave with the official AMD GPU drivers installed.  If I don't install these, again everything is fine but there's no 3d acceleration and that's the whole point here.  smile   

Originally I was not using the ioh3420 device.  When I installed the guest drivers, windows would always say "code 10" on the GPU -- which I've found means that it thinks it is not "properly connected to a root port", so enter the ioh3420.

I've added the ioh3420 to my command line and send the vga/hdmi device through that controller instead. However now I see the "starting windows" animated logo for about 3 seconds, and it just blacks out and reboots at "launch startup repair".  It's possible there is a blue screen here but I am just not seeing it.  If I use send-key KEY_DOWN KEY_ENTER I again get the logo for 3 seconds, rinse repeat.  I would guess this is right at the point where the AMD driver loads and tries to get a hold of the GPU.  Unfortunately I do not see any text or info to tell me why it's restarting....

Any ideas what I'm running into here?   Unfortunately I am at work now and cannot share any logs/commands used, but I can later.  I guess I could try re-installing the guest with the ioh3420 present from the get-go, but I'm wondering if anybody has any thoughts on what could be going on and/or how to debug it. 

ASUS Z9PE-D8WS
2x E5-2687w
EVGA Nvidia GTX570  used for host
ASUS AMD 7870HD passed to guest
IOMMU group 32:   0000:82:00.0  pci bridge of some sort, { 0000:83:00.0, 0000:83:00.1, 0000:83:00.2 } nec usb controller passed to guest (works perfect, cannot pass the 82)
IOMMU group 33:   0000:84:00.0  pci bridge of some sort, { 0000:85:00.0, 0000:85:00.1, 0000:85:00.2 } second nec usb controller passed to guest (works perfect, cannot pass the 84)
IOMMU group 34:  { 0000:86:00.0, 0000:86:00.1 } ASUS 7870, passed to guest - almost works right.

I am using aw's scripts on the previous page to bind those three groups to vfio on startup, works great.

Interestingly, nbhs pointed me towards patching the nvidia-drivers for a VGA_ARB fix.  I'm not sure if I still need this or not.  When I apply the patch, the windows installer never gets past the starting windows animated logo - when it tries to flip the display mode for the "blue install now screen", it never gets there.  When I don't apply the patch, everything works - but I have seen ARTIFACTS on my screen for the host occasionally, but not recently since getting everything set right.  Perhaps nvidia is the problem with the guest. 

Another question -- if you were to do this from scratch, which cards would you use for the host, and which for the guest?  I have too much time and money invested in this for it to not work, I don't mind exchanging cards.  I do have an AST onboard video, I could go with a 3 way setup of onboard for host, AMD for guest1, NVIDIA for guest2.

Thanks for any input!!

Last edited by gentoo-nut (2013-11-15 17:22:38)

Offline

#771 2013-11-15 17:21:07

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gentoo-nut wrote:

I guess I could try reinstalling the guest with the ioh3420 present from the get-go

I guess this is a good idea.

Also in the first post in troubleshooting area there were portions about running host with dedicated cards. Have you gone through that?

Also AST onboard video may be the trouble. I have this weird idea of assigning it to pci-stub in kernel boot options.

Last edited by ilya80 (2013-11-15 17:24:43)

Offline

#772 2013-11-15 17:41:39

gentoo-nut
Member
Registered: 2013-11-14
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

^ The AST card is currently disabled in the bios and the VGA backplate is unplugged from the board - so that is all well out of the picture for the time being.  But yes, if I did elect to add it back in, I would put it in the pci-stub-ids listing in grub.conf.  I may eventually go to that 3 way model and use the host as a "pure host" - not a desktop at all.  If I could find out the ideal cards to pass to a linux and windows guests I may go that way in a month or so.  For now I just want to get my golf simulator software rolling with existing hardware...  smile

I will review the first post again (has to be the 5th time now... smile ) and I will certainly wipe the guest install and try again with the ioh3420 for starters when I get home.

Thank you for the quick reply!!

Offline

#773 2013-11-15 18:00:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gentoo-nut wrote:

Hello all

I am very close to a fully working VFIO passthrough.  I had been using the legacy KVM pci-assign in the past but it's just not reliable enough - having to reboot the host is unnacceptable to me.  nbhs, I believe we know each other from a certain other forum, under a certain other aliases -- thank you for your reply the other day.  I am making good progress.

I am using all of the software versions and patches linked in the first post.  The windows guest works perfectly, I can stop and restart it many times with the x-vga card and the host thankfully remains quite healthy.  I do not have any error messages in dmesg or the qemu logs, I believe everything is in good shape.

I'm just having an issue getting the guest to behave with the official AMD GPU drivers installed.  If I don't install these, again everything is fine but there's no 3d acceleration and that's the whole point here.  smile   

Originally I was not using the ioh3420 device.  When I installed the guest drivers, windows would always say "code 10" on the GPU -- which I've found means that it thinks it is not "properly connected to a root port", so enter the ioh3420.

I've added the ioh3420 to my command line and send the vga/hdmi device through that controller instead. However now I see the "starting windows" animated logo for about 3 seconds, and it just blacks out and reboots at "launch startup repair".  It's possible there is a blue screen here but I am just not seeing it.  If I use send-key KEY_DOWN KEY_ENTER I again get the logo for 3 seconds, rinse repeat.  I would guess this is right at the point where the AMD driver loads and tries to get a hold of the GPU.  Unfortunately I do not see any text or info to tell me why it's restarting....

Any ideas what I'm running into here?   Unfortunately I am at work now and cannot share any logs/commands used, but I can later.  I guess I could try re-installing the guest with the ioh3420 present from the get-go, but I'm wondering if anybody has any thoughts on what could be going on and/or how to debug it.

Windows doesn't much like hardware changes, so it's worth a shot to reinstall.  Others have noted success with Radeon cards when they attach the audio device to bus=pcie.0 rather than as function 1 of the GPU behind the root port.

gentoo-nut wrote:

ASUS Z9PE-D8WS
2x E5-2687w
EVGA Nvidia GTX570  used for host
ASUS AMD 7870HD passed to guest
IOMMU group 32:   0000:82:00.0  pci bridge of some sort, { 0000:83:00.0, 0000:83:00.1, 0000:83:00.2 } nec usb controller passed to guest (works perfect, cannot pass the 82)
IOMMU group 33:   0000:84:00.0  pci bridge of some sort, { 0000:85:00.0, 0000:85:00.1, 0000:85:00.2 } second nec usb controller passed to guest (works perfect, cannot pass the 84)
IOMMU group 34:  { 0000:86:00.0, 0000:86:00.1 } ASUS 7870, passed to guest - almost works right.

I am using aw's scripts on the previous page to bind those three groups to vfio on startup, works great.

Interestingly, nbhs pointed me towards patching the nvidia-drivers for a VGA_ARB fix.  I'm not sure if I still need this or not.  When I apply the patch, the windows installer never gets past the starting windows animated logo - when it tries to flip the display mode for the "blue install now screen", it never gets there.  When I don't apply the patch, everything works - but I have seen ARTIFACTS on my screen for the host occasionally, but not recently since getting everything set right.  Perhaps nvidia is the problem with the guest.

The nvidia VGA arbiter fix is required if you intend to use the nvidia driver in the host.  Without this the host driver will lock the VGA resources and never release them.  If you were experiencing this problem the guest would be blocked as soon as it tries to do a VGA access, so it sounds like you must not be running nvidia in the host.

gentoo-nut wrote:

Another question -- if you were to do this from scratch, which cards would you use for the host, and which for the guest?  I have too much time and money invested in this for it to not work, I don't mind exchanging cards.  I do have an AST onboard video, I could go with a 3 way setup of onboard for host, AMD for guest1, NVIDIA for guest2.

Thanks for any input!!

Surprisingly I'd say we have a bit better success assigning Nvidia cards than we do Radeon cards.  Possibly due to the better reverse engineered hardware documentation through the nouveau project.  So, you might have better luck using the Radeon for the host and assigning the Nvidia.  Your 3-headed approach may work, but it depends on what the graphics is onboard and how well it participates in VGA arbitration.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#774 2013-11-15 18:45:58

gentoo-nut
Member
Registered: 2013-11-14
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Windows doesn't much like hardware changes, so it's worth a shot to reinstall.  Others have noted success with Radeon cards when they attach the audio device to bus=pcie.0 rather than as function 1 of the GPU behind the root port.

The nvidia VGA arbiter fix is required if you intend to use the nvidia driver in the host.  Without this the host driver will lock the VGA resources and never release them.  If you were experiencing this problem the guest would be blocked as soon as it tries to do a VGA access, so it sounds like you must not be running nvidia in the host.

Surprisingly I'd say we have a bit better success assigning Nvidia cards than we do Radeon cards.  Possibly due to the better reverse engineered hardware documentation through the nouveau project.  So, you might have better luck using the Radeon for the host and assigning the Nvidia.  Your 3-headed approach may work, but it depends on what the graphics is onboard and how well it participates in VGA arbitration.

Thanks, if the reinstall doesn't work I will also try moving the hdmi audio device over.

I AM using the nvidia-drivers package with the nvidia module loaded and working.  I am using version 331.20 from gentoo portage ~amd64.  Without the patch it seems to be fine.  With the patch I can't get through the installer.   Perhaps they just fixed the problem in this version?  I cannot get a version prior to 331.20 to compile against kernel 3.12.  (Without at least hacking the build to say - <=kernel-3.11  + <=kernel-3.12  -- or probably updating the old code to be compatible with a changed kernel API, which I'm not going to attempt).  Which version of the driver are you using that requires the patch? The patch does still apply cleanly to 331.20.

If I still run into issues, I will try switching the Nvidia / AMD cards in the slots and give that a shot as you suggested.  Either way is fine with me.  I also have a second 7870 sitting here from when I was experimenting with Xen last year, so I could try that with nvidia totally out of the picture.  However when I tried that last year, the fglrx driver would ALWAYS forcefully claim both cards despite pci-stub already claiming one, so I was never able to try passing one and keeping one.  I don't remember what happened when trying the radeon module, I think that was about the point where I got fed up with Xen in general and did something else.  smile  I do remember the nvidia card was a big no go for dom0 and domU.

Let me ask this very simply:  if anyone has this working perfectly - what exact model card do you have in the host, what driver, and which for the guest?  I am totally fine with ebay-ing all of my graphics cards and starting fresh with a known good setup.

Thanks again for the replies!!

Offline

#775 2013-11-15 19:41:22

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I could only wish someone at NVidia with access to drivers development environment took sometime to repro the code 43 stuff to check whats actually going wrong. Dreams, dreams...

Wonder if installing linux guest with nouveau drivers & debug enabled kernel would do us any good. I think I could pull that off.

Offline

#776 2013-11-15 23:21:44

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I rebuilt qemu now, qemu-1.7.r29486.g5c5432e
Still the same error with usb passthrough.
Can I increase verbosity or is there some sort of error log to search for more information, why it could not add the usb device?
Without the machine boots up fine.

Offline

#777 2013-11-16 00:53:20

gentoo-nut
Member
Registered: 2013-11-14
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Windows doesn't much like hardware changes, so it's worth a shot to reinstall.  Others have noted success with Radeon cards when they attach the audio device to bus=pcie.0 rather than as function 1 of the GPU behind the root port.

Well ... I did the re-install this time with the ioh3420 root there .. and no dice, same thing

THEN .. I moved the HDMI audio function back over to pci.2 as you suggested, kept the VGA over on the ioh, and BAM.  Great success.  Full 3D accel and I can reboot the thing freely, meanwhile the host is rock solid stable.  AW, thank you x100000000000 for that suggestion.  And even more for all of your hard work, great work mind you, and great idea for the VFIO mechanism.  This is truly great stuff, once it becomes easier we can imagine how popular this type of setup could be!!  (seriously, this is awesome)

And nbhs, thank you again for putting the software recipe together on the first post, as well as for all of the help you have given me over the past couple months.   It's been a long often very stressful journey, but I think I am finally finished.   I'll be doing some stability testing over the next few hours and will report back one final time with some screenshots and a summary of what all I've ran into.  BUT IT WORKS, I am so happy right now.... smile smile

[ actually the hdmi audio shows code 10, can't start, as expected I guess.  But who cares, I really don't even want that in the first place.  smile ]

Last edited by gentoo-nut (2013-11-16 01:17:49)

Offline

#778 2013-11-16 06:04:41

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

I rebuilt qemu now, qemu-1.7.r29486.g5c5432e
Still the same error with usb passthrough.
Can I increase verbosity or is there some sort of error log to search for more information, why it could not add the usb device?
Without the machine boots up fine.

Hi!

It really looks like you have built QEMU without usb support. If your qemu starts, can you connect to it via VNC and issue "info usb" and "info hostusb" commands?

To build with usb support you need to run ./configure --help and check the options - there are options to enable it. Then you may have to install some libs.

Offline

#779 2013-11-16 06:06:32

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed debian as guest and nouveau doesnt start. Since its all opensource I thought there can be a way to get more diagnostics out of it... any hints?

Thanks,
Ilya.

Offline

#780 2013-11-16 10:49:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gentoo-nut wrote:

...

Nice to see you finally got it working!

Offline

#781 2013-11-16 12:44:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

I rebuilt qemu now, qemu-1.7.r29486.g5c5432e
Still the same error with usb passthrough.
Can I increase verbosity or is there some sort of error log to search for more information, why it could not add the usb device?
Without the machine boots up fine.

Maybe perhaps you're missing some usb dependencies see this post: https://bbs.archlinux.org/viewtopic.php … 5#p1337135

Offline

#782 2013-11-16 13:34:49

kaladis
Member
Registered: 2013-11-16
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a P15SM Notebook with i7-4800MQ with integrated graphics and Radeon 8970. There are 3 ports for graphics: Display Port, Mini Display Port and HDMI.

I would like to run X on one port (+ Notebook screen) using the IGP and a Windows VM with the discrete GPU on another port.

Is this possible? If so, how do configure which GPU is used for which port?

So far I got the below commands running fine - just without any graphic output:

echo "8086 0c01" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
sudo qemu-system-x86_64 -M q35 -m 8G -enable-kvm -cpu Haswell -vga none -parallel none -bios /usr/share/qemu/bios.bin -smp 4,sockets=1,cores=4,threads=1 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device ahci,bus=pcie.0,id=ahci -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -vnc :1

Thanks a lot

Offline

#783 2013-11-16 16:22:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaladis wrote:

I have a P15SM Notebook with i7-4800MQ with integrated graphics and Radeon 8970. There are 3 ports for graphics: Display Port, Mini Display Port and HDMI.

I would like to run X on one port (+ Notebook screen) using the IGP and a Windows VM with the discrete GPU on another port.

Is this possible? If so, how do configure which GPU is used for which port?

So far I got the below commands running fine - just without any graphic output:

echo "8086 0c01" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
sudo qemu-system-x86_64 -M q35 -m 8G -enable-kvm -cpu Haswell -vga none -parallel none -bios /usr/share/qemu/bios.bin -smp 4,sockets=1,cores=4,threads=1 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device ahci,bus=pcie.0,id=ahci -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -vnc :1

Thanks a lot

Unless you have the schematic for the laptop we can't really tell you whether it's even possible.  Is there a hardware mux that can switch individual output ports to each graphics device independently?  Who knows.  I don't know about IGD/Radeon laptops, but modern IGD/Nvidia laptops don't attach the Nvidia device to any output, the IGD and Nvidia are meant to work together.  The Nvidia renders into buffers which are displayed by the IGD.  I have an older, pre-optimus Asus UL30VT with discrete Nvidia + IGD.  In this case I've figured out that laptop panel is switchable between graphics using a mux in ACPI.  Even then I can only get standard VGA drivers to work, the accelerated drivers just turn the screen black.

So, for your device, it may be possible to get this to work, it may not.  Either way it's going to take some digging and guesswork on your part to figure out if outputs can be switched and how to do it.  It may be telling if the device works with standard Intel and AMD graphics drivers or whether it requires a custom driver from the laptop vendor.  If it requires a custom driver, I doubt you're going to have much success.  Good luck.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#784 2013-11-16 18:24:00

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:
nbhs wrote:

Use the packages i provided instead of the AUR ones

I thought that those were o.k. too since you linked them

nbhs wrote:

qemu-git.tar.gz (includes NoSnoop patch) OR qemu-git from AUR
seabios.tar.gz (1.7.3.2)
linux-mainline.tar.gz (3.12.0 includes acs override patch and i915 fixes) OR  linux-mainline from AUR

But of course I’ll follow your advise, thanks!

… and it was good advise. Although I’m not completely finished just yet I can now tell you that I successfully booted Windows.

Maybe interesting facts:

* migrated existing Windows 7 installation from its own physical machine into a virtual environment
* Using IGP on board graphics from ASRock Z87 Extreme6 for Arch Linux Host works
* Successfully passed through Radeon HD 4850 to Windows

Findings:

* Booting Windows from ssd using a virtualized AHCI controller did not work. Although the Windows repair disk did find the installation nothing I tried to fix it (bootsect, bcdedit, bootrec, etc.) worked. Error messages included but were not limited to “Windows failed to start. A recent hardware or software change might be the cause.”, “An error occurred while attempting to read the boot configuration data” and “An error occurred while attempting to read the boot configuration data”. Then I decided to boot up Windows on the physical host an install all drivers for both the physical machine and virtio. Since I then tried booting (in qemu) using virtio straight away and it worked I don’t know if this or any of the other drivers did the trick.
* It might not have been necessary to decrypt the Windows main drive if I had installed the virtio-drivers before. Also: Don’t decrypt using the rescue CD from TrueCrypt in Qemu. They tell you it will be painfully slow and they mean that. Also it’s single threaded.

Next step:

* Try to pass Creative X-Fi through to Windows. I’m not very optimistic on this one; it looks like it may be sharing it’s IOMMU group with a ton of other stuff
* Set up cpuset
* Fine tuning

Thank you so very, very much, nbhs!

I wanted to try this with Xen back in 2008 but unfortunately my BIOS (yes, only that) did not support VT-d. Now I’m finally there an this is thanks to you and all the other fine people around here. Thank you all!

Offline

#785 2013-11-16 18:41:23

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

...
* Try to pass Creative X-Fi through to Windows. I’m not very optimistic on this one; it looks like it may be sharing it’s IOMMU group with a ton of other stuff
...

Try using the acs override workarround

Offline

#786 2013-11-16 23:03:42

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Hi!

It really looks like you have built QEMU without usb support. If your qemu starts, can you connect to it via VNC and issue "info usb" and "info hostusb" commands?

To build with usb support you need to run ./configure --help and check the options - there are options to enable it. Then you may have to install some libs.

nbhs wrote:

Maybe perhaps you're missing some usb dependencies see this post: https://bbs.archlinux.org/viewtopic.php … 5#p1337135

I think I fixed it now. Even tough configure didn't complain anymore, and I did a "yum-builddep qemu" as aw suggested, libusb-devel and some other stuff was still missing.

$ ./configure --prefix=/home/user/qemu-1.7.r29486.g5c5432e --sysconfdir=/etc --audio-drv-list='pa alsa sdl oss' \ > --python=/usr/bin/python2 --smbd=/sbin/smbd \ > --enable-docs --libexecdir=/home/user/qemu-1.7.r29486.g5c5432e/libexec \ > --disable-gtk --enable-linux-aio --enable-seccomp \ > --enable-spice --localstatedir=/var Install prefix /home/user/qemu-1.7.r29486.g5c5432e BIOS directory /home/user/qemu-1.7.r29486.g5c5432e/share/qemu binary directory /home/user/qemu-1.7.r29486.g5c5432e/bin library directory /home/user/qemu-1.7.r29486.g5c5432e/lib libexec directory /home/user/qemu-1.7.r29486.g5c5432e/libexec include directory /home/user/qemu-1.7.r29486.g5c5432e/include config directory /etc local state directory /var Manual directory /home/user/qemu-1.7.r29486.g5c5432e/share/man ELF interp prefix /usr/gnemul/qemu-%M Source path /home/user/qemu-git/test/qemu-1.7.r29486.g5c5432e C compiler cc Host C compiler cc C++ compiler c++ Objective-C compiler cc CFLAGS -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g -D_FILE_OFFSET_BITS=64 -D__USE_FILE_OFFSET64 -I/usr/include/glusterfs QEMU_CFLAGS -Werror -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -I/usr/include/p11-kit-1 -I/usr/include/libpng15 -I/usr/include/spice-server -I/usr/include/cacard -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/nss3 -I/usr/include/nspr4 -I/usr/include/spice-1 -I/usr/include/nss3 -I/usr/include/nspr4 -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libusb-1.0 -I/usr/include/pixman-1 LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g make make install install python /usr/bin/python2 smbd /sbin/smbd host CPU x86_64 host big endian no target list alpha-softmmu arm-softmmu cris-softmmu i386-softmmu lm32-softmmu m68k-softmmu microblazeel-softmmu microblaze-softmmu mips64el-softmmu mips64-softmmu mipsel-softmmu mips-softmmu moxie-softmmu or32-softmmu ppc64-softmmu ppcemb-softmmu ppc-softmmu s390x-softmmu sh4eb-softmmu sh4-softmmu sparc64-softmmu sparc-softmmu unicore32-softmmu x86_64-softmmu xtensaeb-softmmu xtensa-softmmu alpha-linux-user armeb-linux-user arm-linux-user cris-linux-user i386-linux-user m68k-linux-user microblazeel-linux-user microblaze-linux-user mips64el-linux-user mips64-linux-user mipsel-linux-user mips-linux-user mipsn32el-linux-user mipsn32-linux-user or32-linux-user ppc64abi32-linux-user ppc64-linux-user ppc-linux-user s390x-linux-user sh4eb-linux-user sh4-linux-user sparc32plus-linux-user sparc64-linux-user sparc-linux-user unicore32-linux-user x86_64-linux-user tcg debug enabled no gprof enabled no sparse enabled no strip binaries yes profiler no static build no -Werror enabled yes pixman system SDL support yes GTK support no curses support yes curl support yes mingw32 support no Audio drivers pa alsa sdl oss Block whitelist (rw) Block whitelist (ro) VirtFS support yes VNC support yes VNC TLS support yes VNC SASL support yes VNC JPEG support yes VNC PNG support yes VNC WS support yes xen support no brlapi support yes bluez support yes Documentation yes GUEST_BASE yes PIE yes vde support no Linux AIO support yes ATTR/XATTR support yes Install blobs yes KVM support yes RDMA support no TCG interpreter no fdt support yes preadv support yes fdatasync yes madvise yes posix_madvise yes sigev_thread_id yes uuid support yes libcap-ng support no vhost-net support yes vhost-scsi support yes Trace backend nop Trace output file trace-<pid> spice support yes (0.12.6/0.12.4) rbd support yes xfsctl support yes nss used yes libusb yes usb net redir yes GLX support yes libiscsi support yes build guest agent yes QGA VSS support no seccomp support yes coroutine backend ucontext coroutine pool yes GlusterFS support yes virtio-blk-data-plane yes gcov gcov gcov enabled no TPM support no libssh2 support yes TPM passthrough no QOM debugging yes vhdx yes

So libusb is now saying "yes", are some of the others saying "no" important?

I can now passthrough the usb devices I need and managed to install Win7Ultimate in a 30GB .img-file as shown in the first post. (Just for now, until it works)
My install process looked like this:
1. boot from win7.iso
2. select "disk" and start installation
3. reboot
4. installation process finishes
5. reboot
6. initial setup of win7: chose computer name, select network and password, etc.

BUT

When it's finished, it never boots the actual system, after seabios it just stops with a blinking underline "_" while one of my cpu cores runs at 100% load all the time.
I tried this many times now, with installation on an .img-file, a real /dev/sdj disk and even by passing through the  pcie-controller with a disk attached (unfortunately windows states it cant be installed on this disk, when doing so)

Is this a known problem someone ran into before? My old installation still works fine, my wildest guess is that using emulated q35 chipset with an image of win7 from 2010 is just not working.
Any ideas?

Offline

#787 2013-11-17 00:46:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

...

Dont use my script, use the command line or use this one:

#!/bin/sh QEMU=/usr/bin/qemu-system-x86_64 VCPUS="6" CPU="Opteron_G4" MEM="8192" MEM_PATH="/dev/hugepages" BIOS="/usr/share/qemu/bios.bin" NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" USB_DEVICES="" PCI_DEVICES="00:11.0 04:00.0 05:00.0 06:00.0" GPU="07:00.0" GPU_SND="07:00.1" SND=true SND_DRIVER_OPTS="QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa " VBIOS="/usr/local/share/qemu/vgabios-6950.bin" CDROM="" HDD="" QEMU_ARGS=" -M q35 -enable-kvm -monitor stdio" EXTRA_ARGS="-boot menu=on" if [ -n "$GPU" ]; then EXTRA_ARGS+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1,romfile=$VBIOS" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$MEM" ]; then MEMORY+="-m $MEM" if [ -n "$MEM_PATH" ]; then MEMORY+=" -mem-prealloc -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS echo "All Done!!"

Last edited by nbhs (2013-11-17 00:50:28)

Offline

#788 2013-11-17 01:57:46

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

same problem hmm
This time I didn't even get to configure Win7, it stopped after the installation process rebootet the 2nd time.
I see the line "SeaBIOS (version rel-1.7.3.2-0-[...]" above the blinking cursor.

After this I stopped qemu and started it again so it startet where it left off, finishing the installation.
After configuring win7 I even got to see the desktop for a few seconds until it decided to do a reboot... and now: back to the blinking cursor. Does not boot anymore. Doesn't matter how often I reset qemu.
Only thing I'm wondering about, is the qemu-console showing "QEMU 1.6.90 monitor - type 'help' for more information".
Shouldn't it be 1.7.x?

Last edited by Norcoen (2013-11-17 01:58:32)

Offline

#789 2013-11-17 08:26:25

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

same problem hmm

If you fail booting / installing win - try various settings for -cpu option for qemu. host, qemu64, your actual CPU type, etc. Thats what helped me.

Offline

#790 2013-11-17 13:12:33

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks. You were right with the -cpu options. smile
I just changed CPU="Haswell" in the script to CPU="host" and it is booting fine now, didn't even need to re-install the system.

I'm currently installing drivers to check if everything is working. But I wonder, why the machine has access to the internet... I set NET="" in the script

#!/bin/sh QEMU=/home/user/qemu-1.7.r29486.g5c5432e/bin/qemu-system-x86_64 VCPUS="8" CPU="host" MEM="8192" #MEM_PATH="/dev/hugepages" MEM_PATH="" BIOS="/home/user/qemu-1.7.r29486.g5c5432e/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" NET="" #USB_DEVICES="" USB_DEVICES="host:1267:0210" PCI_DEVICES="04:00.0 08:00.0 09:00.0" #PCI_DEVICES="" GPU="01:00.0" GPU_SND="01:00.1" SND=false SND_DRIVER_OPTS="QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa " VBIOS="" CDROM="/home/user/Dokumente/KVM/Win7.iso" HDD="/media/BigStorage/Temp/windowskvm.img" QEMU_ARGS=" -M q35 -enable-kvm -monitor stdio" EXTRA_ARGS="-boot menu=on" if [ -n "$GPU" ]; then EXTRA_ARGS+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1,romfile=$VBIOS" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$MEM" ]; then MEMORY+="-m $MEM" if [ -n "$MEM_PATH" ]; then MEMORY+=" -mem-prealloc -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS echo "All Done!!"

Still Windows7 has "Intel(R) PRO/1000 MT-Networkconnection".
IP-Adress: 10.0.2.15
Subnet: 255.255.255.0
Gateway: 10.0.2.2
DHCP: 10.0.2.2
DNS: 10.0.2.3

Where is this coming from? The adress looks like the one I got from bridged network in virt-manager.

//Edit:
Unfortunately, I'm getting a bluescreen with atikmpag.sys now everytime I try to boot the system hmm

Last edited by Norcoen (2013-11-17 13:26:25)

Offline

#791 2013-11-17 16:40:15

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Unfortunately, I'm getting a bluescreen with atikmpag.sys now everytime I try to boot the system hmm

"I get a BSOD 0x00000116  atikmpag.sys if i remove the ati driver, the VM work good but no graphique acceleration.
Edit: I found the problem to slove it juste put clear_emulator_capabilities=0 in qemu.conf"

Is blue screen code the same? You could try this.

Offline

#792 2013-11-17 17:35:59

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
after playing around with my guest i still have 2 unsolved problems.
1. When i reboot the guest the screen stays black. I have to reboot the host to get the guest working again.
2. qemu does not show the pa or alsa driver hence sound is not working in the guest. (qemu-git PKGBUILD from first post, libpulse & pulseaudio packages installed)

# qemu-system-x86_64 -audio-help | grep Name Name: oss Name: none Name: wav

Maybe someone can help me fixing this? big_smile

Thanks in advance!

Offline

#793 2013-11-17 19:03:14

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

"I get a BSOD 0x00000116  atikmpag.sys if i remove the ati driver, the VM work good but no graphique acceleration.
Edit: I found the problem to slove it juste put clear_emulator_capabilities=0 in qemu.conf"

Is blue screen code the same? You could try this.

Yes, I'm getting this one:
2013-11-17195947jpgeOvpEsU.jpg
Where is qemu.conf lcoated usually? Or do I have to create it manually?

Last edited by Norcoen (2013-11-17 19:03:34)

Offline

#794 2013-11-17 19:15:19

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:
ilya80 wrote:

"I get a BSOD 0x00000116  atikmpag.sys if i remove the ati driver, the VM work good but no graphique acceleration.
Edit: I found the problem to slove it juste put clear_emulator_capabilities=0 in qemu.conf"

Is blue screen code the same? You could try this.

Yes, I'm getting this one:
http://pichoster.adbt2.ath.cx/viewpic/y/2013-11-17195947jpgeOvpEsU.jpg
Where is qemu.conf lcoated usually? Or do I have to create it manually?

Dude use google. Its in /etc/libvirt/qemu.conf usually.

Offline

#795 2013-11-17 20:50:09

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry, I was looking in /etc/qemu/, and it wasn't there, I forgot the connection to libvirtd.
I already have "clear_emulator_capabilities=0" in said qemu.conf, because before finding this board, I followed another tutorial and applied the last paragraph of this site:
http://fedoraproject.org/wiki/How_to_de … assignment

So, I have the same bluescreen error, but your solution somehow does not apply for me hmm

//edit:
I found out, the qemu-git I built isn't looking in /etc/libvirt/ but in /etc/qemu, so I copied my qemu.conf to that location, now getting this error:

qemu-system-x86_64:/etc/qemu/qemu.conf:198: no group defined

Line 198 is

user = "root"

So I'm stuck again...

//edit2:
I played around a bit with the config, and the seems to be around the group for the QEMU processes:

# The group for QEMU processes run by the system instance. It can be # specified in a similar way to user. group = "root"

If I delete that line, I get a parser error, but if I set it to any group, root, my user, qemu, kvm, I just get the error above, that no group is defined.

Last edited by Norcoen (2013-11-17 21:26:54)

Offline

#796 2013-11-17 21:09:13

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I'm not sure how this is going to help your performance though, it's pretty random whether re-loading the host driver is going to resolve anything that the guest driver can't.  For graphics devices, if you're using kernel+qemu that support bus reset, that should provide something nearly as good as a host reboot.

have the kernel 3.12.0-rc7
I do not know exactly what it was anyway because I did set the values of kvm and kvm-amd.
Solved in this way



/proc/cmdline

root=PARTUUID=F942477B-23BB-4880-A611-17B3CF4040A7 ro ivrs_ioapic[5]=00:14.0 acpi_sleep=s3_bios,s3_mode vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.allow_unsafe_assigned_interrupts=1 kvm.ignore_msrs=1 kvm_amd.npt=1 pci-stub.ids=1002:6738,1002:aa88,1033:0194

dmesg

[ 6289.907200] kvm_get_msr_common: 70 callbacks suppressed [ 6289.907203] kvm [2383]: vcpu0 ignored rdmsr: 0xe8 [ 6289.907208] kvm [2383]: vcpu0 ignored rdmsr: 0xe7 [ 6289.911413] kvm [2383]: vcpu0 ignored rdmsr: 0xe8 [ 6289.911417] kvm [2383]: vcpu0 ignored rdmsr: 0xe7 [ 6289.969551] kvm [2383]: vcpu1 ignored rdmsr: 0xe8 [ 6289.969557] kvm [2383]: vcpu1 ignored rdmsr: 0xe7 [ 6289.973760] kvm [2383]: vcpu1 ignored rdmsr: 0xe8 [ 6289.973765] kvm [2383]: vcpu1 ignored rdmsr: 0xe7

What could this message be?


here's what starts libvirt

2013-11-17 17:31:32.777+0000: starting up
LC_ALL=C PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.7.3 HOME=/root USER=root QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name dcdcdcd,process=qemu:dcdcdcd -S -machine pc-q35-1.6,accel=kvm,usb=off,mem-merge=off -cpu host -m 3993 -realtime mlock=on -smp 2,sockets=1,cores=3,threads=1 -uuid 34bbd363-5c7a-9147-dff1-c04d90d335eb -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/dcdcdcd.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -no-acpi -boot order=dc,menu=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x2 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device piix3-usb-uhci,id=usb,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/virtio-win-0.1-65.iso,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/dev/mapper/350014ee00384381d,if=none,id=drive-sata0-0-1,format=raw -device ide-hd,bus=ide.1,drive=drive-sata0-0-1,id=sata0-0-1 -netdev tap,fd=18,id=hostnet0,vhost=on,vhostfd=19 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:58:63:38,bus=pci.2,addr=0x2 -device usb-host,hostbus=3,hostaddr=3,id=hostdev0 -device usb-host,hostbus=5,hostaddr=2,id=hostdev1 -device usb-host,hostbus=5,hostaddr=3,id=hostdev2 -bios /usr/share/seabios/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/usr/share/qemu/vbios.rom -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 -device vfio-pci,host=04:00.0,bus=pcie.0 -device vfio-pci,host=05:00.0,bus=pcie.0 .
Domain id=3 is tainted: high-privileges
Domain id=3 is tainted: custom-argv
Domain id=3 is tainted: host-cpu

I can not pass the usb3 controller, the host interfere but I think the drivers is vfio-pci
what I could try?

04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI]) Subsystem: Micro-Star International Co., Ltd. Device 7640 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 51 Region 0: Memory at f7300000 (64-bit, non-prefetchable) [size=8K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [90] MSI-X: Enable+ Count=8 Masked- Vector table: BAR=0 offset=00001000 PBA: BAR=0 offset=00001080 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff Capabilities: [150 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: vfio-pci 05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI]) Subsystem: Micro-Star International Co., Ltd. Device 7640 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 50 Region 0: Memory at f7200000 (64-bit, non-prefetchable) [size=8K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [90] MSI-X: Enable+ Count=8 Masked- Vector table: BAR=0 offset=00001000 PBA: BAR=0 offset=00001080 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff Capabilities: [150 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: vfio-pci

00:00.0 Host bridge should be using a driver?

00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) Subsystem: Micro-Star International Co., Ltd. Device 7640 Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Capabilities: [f0] HyperTransport: MSI Mapping Enable+ Fixed+ Capabilities: [c4] HyperTransport: Slave or Primary Interface Command: BaseUnitID=0 UnitCnt=20 MastHost- DefDir- DUL- Link Control 0: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b+ Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn- Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+ <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b- Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn- Revision ID: 3.00 Link Frequency 0: [c] Link Error 0: <Prot- <Ovfl- <EOC- CTLTm- Link Frequency Capability 0: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend- Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- Link Frequency 1: 200MHz Link Error 1: <Prot- <Ovfl- <EOC- CTLTm- Link Frequency Capability 1: 200MHz- 300MHz- 400MHz- 500MHz- 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend- Error Handling: PFlE- OFlE- PFE- OFE- EOCFE- RFE- CRCFE- SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE- Prefetchable memory behind bridge Upper: 00-00 Bus Number: 00 Capabilities: [40] HyperTransport: Retry Mode Capabilities: [54] HyperTransport: UnitID Clumping Capabilities: [9c] HyperTransport: #1a Capabilities: [70] MSI: Enable- Count=1/4 Maskable- 64bit- Address: 00000000 Data: 0000 00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU) Subsystem: Micro-Star International Co., Ltd. Device 7640 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 72 Capabilities: [40] Secure device <?> Capabilities: [54] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee0f00c Data: 41a1 Capabilities: [64] HyperTransport: MSI Mapping Enable+ Fixed+ 00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port B) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: f4000000-f70fffff Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <8us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #2, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [b0] Subsystem: Micro-Star International Co., Ltd. Device 7640 Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+ Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [190 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport

libvirt dom

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>dcdcdcd</name> <uuid>34bbd363-5c7a-9147-dff1-c04d90d335eb</uuid> <memory unit='KiB'>4088608</memory> <currentMemory unit='KiB'>4088608</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='1'/> <emulatorpin cpuset='0-1,2-3'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-1.6'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <cpu mode='host-passthrough'> <topology sockets='1' cores='3' threads='1'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/virtio-win-0.1-65.iso'/> <target dev='vda' bus='ide'/> <alias name='virtio-disk0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/mapper/350014ee00384381d'/> <target dev='vdb' bus='sata'/> <alias name='virtio-disk1'/> </disk> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1b1c'/> <product id='0x1ab1'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04d9'/> <product id='0x1702'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x192f'/> <product id='0x0416'/> </source> </hostdev> <interface type='network'> <mac address='52:54:00:58:63:38'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <controller type='ide' index='0'/> <memballoon model='none'> <alias name='balloon0'/> </memballoon> </devices> <seclabel type='none'/> <qemu:commandline> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/seabios/bios.bin'/> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/usr/share/qemu/vbios.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/> </qemu:commandline> </domain>

another question I try to pass the sound card but does not feel good

Last edited by mafferri (2013-11-18 00:27:23)

Offline

#797 2013-11-18 20:27:58

juhe
Member
Registered: 2013-08-10
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to let you know I was able to get the thing working on following HW configurations:

AMD Platform:

  • Motherboard: SABERTOOTH 990FX R20 (BIOS rev. 2005)

  • CPU: AMD FX-8350 @4Ghz

  • Primary VGA: Radeon HD 5850

  • Secondary (passed through) VGA: Radeon HD 7870

Intel Platform:

  • Motherboard: P9X79 DELUXE (BIOS rev. 4502)

  • CPU: Intel i7-3930K

  • Primary VGA: Radeon HD 6670

  • Secondary (passed through) VGA: Radeon HD 7850

Both platform have also passed one internal USB HUB for keyboard / mouse and internal Intel HDA audio for sound. Network and disk use virtio interface.

Host OS (on both machines) is Archlinux with recent stock 3.12 kernel, qemu (1.7.r29424.g964668b) and seabios (1.7.3.2) from nbhs' first post.

AMD platform worked basically out of the box, because nbhs has very similar setup (chipset is the same).  Intel, however, was a bit tougher to get working because of Catalyst 13.9 atikmpag.sys' code 116 BSOD which was finally solved thanks to recent posts from aw and gentoo-nut by simply ommiting the HDMI Audio completely (or adding it to pcie.0 bus works as well).

I would like to Thank nbhs, aw and all you guys who share the information about setups, related issues and their solutions on this thread.  I wouldn't be able to get the thing working without you :-).

Offline

#798 2013-11-18 21:20:51

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

juhe wrote:

Just wanted to let you know I was able to get the thing working

Congrats! I envy you smile

juhe wrote:

CPU: Intel i7-3930K

Are you sure that was a K model? Those are not supposed to have VT-d.

Offline

#799 2013-11-18 21:37:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
juhe wrote:

CPU: Intel i7-3930K

Are you sure that was a K model? Those are not supposed to have VT-d.

Never assume you've figured out Intel's naming convention:

http://ark.intel.com/products/63697/Int … GHz?q=3930

VT-d: yes


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#800 2013-11-19 07:24:40

mangobot
Member
Registered: 2013-11-19
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

Thanks for the awesome writeup. My basic hardware is:

cpu: amd FX-6300
gpu: amd 7970
mb: ASUS M5A99FX PRO R2.0 (2005 bios)

I previously used this walkthrough to set up fully working passthrough using two graphics cards (using the 7970 as secondary and a really basic ati card as primary), but I just managed to get it working on my primary card alone, using pci_stub. smile I do see boot graphics and console output from the gfx card (I run all my monitors off of it), but when I run qemu it seems to switch over fine.
I hadn't seen much success with this/had heard a lot about primary passthrough not being possible, but since I'm looking at it, I thought I'd throw my experience into the ring. Now I have more space in my box so I can avoid the Realtek ethernet card!

Offline

#801 2013-11-19 13:08:21

mafferri
Member
Registered: 2013-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

juhe wrote:

Just wanted to let you know I was able to get the thing working on following HW configurations:

AMD Platform:

  • Motherboard: SABERTOOTH 990FX R20 (BIOS rev. 2005)

  • CPU: AMD FX-8350 @4Ghz

  • Primary VGA: Radeon HD 5850

  • Secondary (passed through) VGA: Radeon HD 7870

Intel Platform:

  • Motherboard: P9X79 DELUXE (BIOS rev. 4502)

  • CPU: Intel i7-3930K

  • Primary VGA: Radeon HD 6670

  • Secondary (passed through) VGA: Radeon HD 7850

Both platform have also passed one internal USB HUB for keyboard / mouse and internal Intel HDA audio for sound. Network and disk use virtio interface.

Host OS (on both machines) is Archlinux with recent stock 3.12 kernel, qemu (1.7.r29424.g964668b) and seabios (1.7.3.2) from nbhs' first post.

AMD platform worked basically out of the box, because nbhs has very similar setup (chipset is the same).  Intel, however, was a bit tougher to get working because of Catalyst 13.9 atikmpag.sys' code 116 BSOD which was finally solved thanks to recent posts from aw and gentoo-nut by simply ommiting the HDMI Audio completely (or adding it to pcie.0 bus works as well).

I would like to Thank nbhs, aw and all you guys who share the information about setups, related issues and their solutions on this thread.  I wouldn't be able to get the thing working without you :-).

hello hda-intel will work well on the guest? to me makes strange noises and echoes

How did you pass the usb hub? because I can not even spend an entire controller usb3 -. - '
and I have to pass individually keyboard and mouse

Offline

#802 2013-11-19 14:41:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mangobot wrote:

Hey guys,

Thanks for the awesome writeup. My basic hardware is:

cpu: amd FX-6300
gpu: amd 7970
mb: ASUS M5A99FX PRO R2.0 (2005 bios)

I previously used this walkthrough to set up fully working passthrough using two graphics cards (using the 7970 as secondary and a really basic ati card as primary), but I just managed to get it working on my primary card alone, using pci_stub. smile I do see boot graphics and console output from the gfx card (I run all my monitors off of it), but when I run qemu it seems to switch over fine.
I hadn't seen much success with this/had heard a lot about primary passthrough not being possible, but since I'm looking at it, I thought I'd throw my experience into the ring. Now I have more space in my box so I can avoid the Realtek ethernet card!

The problem with primary graphics is detaching the host drivers or making sure the host driver uses the vga arbiter.  Typically if you free the primary device by blacklisting the framebuffer driver or using pci-stub then you're left with vgacon on the primary VGA.  vgacon is completely unaware of VGA arbitration and writes to whatever device owns VGA routing at the time.  That means you could see host console messages on the guest if the guest is in VGA mode, but even in graphics mode those writes could be corrupting the state of the device in the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#803 2013-11-20 17:12:05

juhe
Member
Registered: 2013-08-10
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

mafferri wrote:

hello hda-intel will work well on the guest? to me makes strange noises and echoes

I'm not sure to which system (AMD/Intel) I maintain you refer to. On Intel, it seems working fine. I didn't do extensive testing/game playing since it is a production machine at work to which I added another VGA for vfio testing (well, I couldn't resit to try it smile). It means I didn't play a game on it, just run the "Kalimba test", i.e. run the Kalimba.mp3 from Music/ folder on windows 7 and did some internet browsing while listening to the music. From the feeling and comparison with the other AMD system I'm maintaining (and on which I run couple of games), the sound rarely stutters, even initial login sound seems clean. The stuttering seems more often happening on the AMD system, where I rarely get clean logon sound. However, after some time (probably when everything important gets cached in memory) it runs fine also on AMD system. BTW, thanks nbhs for the npt=0 parameter to kvm-amd kernel module, it really makes difference.

DPC Latency Checker seems to have lower red bars on the Intel platform, but all this is really just my current feeling (YMMV), more elaborate tests (possibly having both systems side by side, which is not that easily possible) would be necessary to compare the systems properly.

mafferri wrote:

How did you pass the usb hub? because I can not even spend an entire controller usb3 -. - '
and I have to pass individually keyboard and mouse

I pass only USB2 controller (I really meant controller in my previous post, not HUB, I am sorry for the confusion). I use these options in my qemu startup command line for Intel system (which I have currently at hand):

 -device vfio-pci,host=00:1a.0,bus=pcie.0,id=usb \ -device vfio-pci,host=00:1b.0,bus=pcie.0,id=snd \
$ lspci | grep 1[ab]\.0 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06) 00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06)

And both devices get to their own group, thanksfully, even without ACS patch. The lsgroup.sh used below is aw's script stolen from one of his previous post and added ".sh" extension.

$ ./lsgroup.sh | grep -C1 1[ab]\.0 ### Group 9 ### 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06) ### Group 10 ### 00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06) ### Group 11 ###

I use a wireless keyboard and mouse with a single USB transmitter/receiver, which occupies one USB port.

HTH

Offline

#804 2013-11-20 23:40:06

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys. With the latest kernel and patches I finally managed to get it working! ...kinda.

I can't pass my usb 3.0 controller for some reason. It simply won't work. I can try to find another controller that works, but I really wanted it to be this one. For now I passed my usb keyboard and mouse only, but I need either the controller or my HUB (which I also can't pass).

The second problem is that it can't stand a reboot. If I try to reboot it will get stuck on a screen with the seabios version.

Offline

#805 2013-11-21 13:19:09

sbd
Member
Registered: 2013-11-21
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

Long time lurker, first time poster here. Thanks very much for the interesting and informative thread, it's been a great help. I've been trying to get VGA passthrough working for some time, and while I've succeeded in getting a secondary passthrough working with Xen I've yet to succeed with KVM. I'm experiencing the same stop 0x00000116 error seen by several others with Catalyst drivers installed, but haven't had any joy solving it with the suggested methods.

Unfortunately at the moment I'm having to do my testing on a Fedora machine (my other machine is indisposed right now, although I'm hoping to have it available again soon, please excuse the foreign OS). It has the following specs:
Asus P8H61M-LE (B3 revision)
i5-2500
HD5770
I was previously using version 1.6.1 of qemu to test, however I am currently using qemu compiled from Alex's git repo with the tag referenced in post 1:
git://github.com/awilliam/qemu-vfio.git tags/vfio-pci-for-qemu-20131003.0

Kernel is 3.11.8 (no special config) but I have checked and VFIO support is compiled in (as modules).

I have succeeded in getting the 5770 passed through as primary adapter to a Windows 7 guest, however when I install the Catalyst drivers and reboot, I get the stop 0x00000116 error, followed by a hard lockup of the machine (screen goes black, machine goes unresponsive, and I have to hard reboot. Before the catalyst drivers are installed everything is fine, but I am really after accelerated graphics. I have passed through a USB controller as well and this appears to function normally, and the virtio drivers for the network card also work fine.

When the stop error occurs, the machine goes completely unresponsive. No messages show on my other monitor (running off the onboard GPU), and the machine stops responding to keyboard and ping. There's nothing in the logs afterwards either.

I have done/tried the following without success:
- Blacklisting the radeon driver and assigning the graphics card to pci-stub in the grub command line.
- Setting options vfio_iommu_type1 allow_unsafe_interrupts=1 in /etc/modprobe.d/vfio_iommu_type1.conf
- Setting clear_emulator_capabilities = 0 in qemu.conf, and setting user and group to root.
- Adding the various /dev/vfio/x devices to cgroup_device_acl in qemu.conf
- Moving the HDMI audio component (01:00.1) of the 5770 to pcie.0, instead of being attached to the pcie root port.
- Attaching 01:00.1 to a pci port.
- Removing the 01:00.1 from the qemu config completely so it isn't passed through.
- Upgrading my motherboard BIOS

I have the guest running off of a ZFS zpool, with snapshots, so I've been able to rollback and reinstall the catalyst drivers under different configurations (for example, installing the drivers without the HDMI audio component). In each case I've encountered the same stop error.

I've checked my dmesg output and the IOMMU appears to be being detected properly (and I have intel_iommu=on set in /etc/default/grub):

[    0.000000] Intel-IOMMU: enabled
[    0.016716] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.016787] IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.443112] IOMMU 0 0xfed90000: using Queued invalidation
[    0.443114] IOMMU: Setting RMRR:
[    0.443121] IOMMU: Setting identity map for device 0000:00:1d.0 [0xdf0d6000 - 0xdf0e4fff]
[    0.443139] IOMMU: Setting identity map for device 0000:00:1a.0 [0xdf0d6000 - 0xdf0e4fff]
[    0.443148] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.443154] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

However after the entry for device 00:1f.0 (LPC controller) I get this:

[    0.443154] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.443233] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.443290] ------------[ cut here ]------------
[    0.443295] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90()
[    0.443296] Modules linked in:
[    0.443299] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.11.8-200.fc19.x86_64 #1
[    0.443300] Hardware name: System manufacturer System Product Name/P8H61-M LE, BIOS 4501 05/10/2013
[    0.443301]  0000000000000009 ffff8801f3eabce0 ffffffff8164781b 0000000000000000
[    0.443303]  ffff8801f3eabd18 ffffffff8106715d ffff8801f3b04098 ffff8801f3b04000
[    0.443305]  ffff8801f3b04098 ffffffff81cbb0a0 ffff8801f68095a0 ffff8801f3eabd28
[    0.443306] Call Trace:
[    0.443311]  [<ffffffff8164781b>] dump_stack+0x45/0x56
[    0.443313]  [<ffffffff8106715d>] warn_slowpath_common+0x7d/0xa0
[    0.443315]  [<ffffffff8106723a>] warn_slowpath_null+0x1a/0x20
[    0.443317]  [<ffffffff8132b547>] pci_find_upstream_pcie_bridge+0x77/0x90
[    0.443319]  [<ffffffff815262ad>] intel_iommu_add_device+0x4d/0x220
[    0.443322]  [<ffffffff8151d5a0>] ? bus_set_iommu+0x50/0x50
[    0.443323]  [<ffffffff8151d5ca>] add_iommu_group+0x2a/0x50
[    0.443325]  [<ffffffff813effe3>] bus_for_each_dev+0x63/0xa0
[    0.443327]  [<ffffffff8151d598>] bus_set_iommu+0x48/0x50
[    0.443330]  [<ffffffff81d5f4f0>] intel_iommu_init+0xa55/0xb4d
[    0.443332]  [<ffffffff8104f5e5>] ? free_init_pages+0x75/0xa0
[    0.443335]  [<ffffffff81d10694>] ? maybe_link.part.2+0x110/0x110
[    0.443337]  [<ffffffff81d17da9>] ? memblock_find_dma_reserve+0x124/0x124
[    0.443339]  [<ffffffff81d17dbb>] pci_iommu_init+0x12/0x3c
[    0.443341]  [<ffffffff810020fa>] do_one_initcall+0xfa/0x1b0
[    0.443344]  [<ffffffff810866a8>] ? parse_args+0x138/0x400
[    0.443346]  [<ffffffff81d0f078>] kernel_init_freeable+0x177/0x1ff
[    0.443347]  [<ffffffff81d0e898>] ? do_early_param+0x88/0x88
[    0.443350]  [<ffffffff8163db10>] ? rest_init+0x80/0x80
[    0.443352]  [<ffffffff8163db1e>] kernel_init+0xe/0x190
[    0.443354]  [<ffffffff8165696c>] ret_from_fork+0x7c/0xb0
[    0.443356]  [<ffffffff8163db10>] ? rest_init+0x80/0x80
[    0.443361] ---[ end trace 2d03b7eaf7f34bc3 ]---

I don't think it's related though, since I'm not trying to pass through any PCI devices. Here's my lspci for reference:

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 2 (rev b5)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation H61 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)

I know a few other posters have encountered stop 0x00000116 but from what I can tell those issues were resolved for most people by placing 01:00.1 under pcie.0 (or disabling it entirely). Has anyone else had no luck with this method? I'm hoping I've just missed something obvious, but I'm guessing it could also be an issue with my Asus motherboard. My other machine has an Asrock board that I'm hoping to be able to test with soon.

Any advice appreciated. Thanks.

Offline

#806 2013-11-21 20:05:07

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Was the solution with deveject.exe at shutdown deleted from the initial post?
I can't find anything about it anymore.
I used it in an older VM and wanted to add my new PCIe Controller, which seems to have he same problem. But I don't remember where I put the entry to do so. Wasn't the taskplaner or mmc (w/e that is)

Offline

#807 2013-11-22 11:18:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sbd wrote:

...Any advice appreciated. Thanks...

You need the 3.12 kernel, and if you're using the i915 driver you'll also need some fixes

Offline

#808 2013-11-22 11:21:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Was the solution with deveject.exe at shutdown deleted from the initial post?
I can't find anything about it anymore.
I used it in an older VM and wanted to add my new PCIe Controller, which seems to have he same problem. But I don't remember where I put the entry to do so. Wasn't the taskplaner or mmc (w/e that is)

nbhs wrote:

To automate this process three's a tool called "deveject" from http://www.withopf.com/tools/deveject/, extract it somewhere then run it from cmd.
You should see an output like this:

... '@oem4.inf,%amd6719.1%;AMD Radeon HD 6900 Series' 'PCI\VEN_1002&DEV_6719&SUBSYS_31221682&REV_00\4&184E0D10&0&00E0' [REMOVEABLE] ...

Now press the start button and type gpedit.msc
Go to "Computer Configuration"-->"Windows Settings"-->"Scripts"-->"Shutdown"
Click Properties, then "Add.."
On Script Name put the full path to deveject.exe (C:\Windows\System32\deveject.exe in my case)
On Script Parameters -EjectId:"PCI\VEN_1002&DEV_6719&SUBSYS_31221682&REV_00\4&184E0D10&0&00E0" <--change this acording to your case
If its working you should see windows changing the resolution while shutting down.

Offline

#809 2013-11-23 02:09:43

sbd
Member
Registered: 2013-11-21
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the tips guys, I'll give those suggestions a try.

Offline

#810 2013-11-23 13:35:42

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys! Can anyone who got this working check which PCI bus does GPU end up on when booting Windows 7? Its in the device manager.

For me its PCI Bus 28, while on bare metal its PCI bus 0.

Last edited by ilya80 (2013-11-23 13:40:22)

Offline

#811 2013-11-23 13:43:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Hi guys! Can anyone who got this working check which PCI bus does GPU end up on when booting Windows 7? Its in the device manager.

For me its PCI Bus 28, while on bare metal its PCI bus 0.

l41Twxg.jpg

Offline

#812 2013-11-23 16:50:57

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cool this must be something else then.

Offline

#813 2013-11-24 16:22:27

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys!

Today I successfully hardmodded my GTX 680 to report itself as GRID K2.

When I try to passthrough the card, I get no output on either display outputs though. Nothing suspicious in the logs, just:

Nov 24 20:19:42 cave-lin kernel: [ 368.001538] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Nov 24 20:19:42 cave-lin kernel: [ 368.001719] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Nov 24 20:20:20 cave-lin kernel: [ 405.590323] kvm: zapping shadow pages for mmio generation wraparound

I havent changed anything since last time I run my config, except of device ID to assign to pci-stub during boot. Prior to that passthrough worked and I just wasnt able to install Nvidia drivers in guest.

Any advice?

Last edited by ilya80 (2013-11-24 19:39:38)

Offline

#814 2013-11-24 23:58:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Hi guys!

Today I successfully hardmodded my GTX 680 to report itself as GRID K2.

When I try to passthrough the card, I get no output on either display outputs though. Nothing suspicious in the logs, just:

Nov 24 20:19:42 cave-lin kernel: [ 368.001538] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Nov 24 20:19:42 cave-lin kernel: [ 368.001719] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Nov 24 20:20:20 cave-lin kernel: [ 405.590323] kvm: zapping shadow pages for mmio generation wraparound

I havent changed anything since last time I run my config, except of device ID to assign to pci-stub during boot. Prior to that passthrough worked and I just wasnt able to install Nvidia drivers in guest.

Any advice?

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.  However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#815 2013-11-25 07:09:37

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.

With VBIOS you mean the bios on the video card? No I haven't updated that. I`d rather keep the original bios b/c as far as I understand it manages clocks/fans in a way I want it to - as a GTX 680. Is there a way to force seabios to run GPU BIOS even though there is a mismatch?

aw wrote:

However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Thanks for advice! Ill try that.

Update: for some reason WIndows is not able to boot - BSOD with PAGE_FAULT_IN_NONPAGED_AREA code 0x00000050. Sometimes BSOD has different code 0x0000007E ( SYSTEM_THREAD_EXCEPTION_NOT_HANDLED )

I might look into seabios at see if it can be forced to load BIOS even though there is a mismatch.

Last edited by ilya80 (2013-11-25 07:41:30)

Offline

#816 2013-11-25 10:23:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
aw wrote:

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.

With VBIOS you mean the bios on the video card? No I haven't updated that. I`d rather keep the original bios b/c as far as I understand it manages clocks/fans in a way I want it to - as a GTX 680. Is there a way to force seabios to run GPU BIOS even though there is a mismatch?

aw wrote:

However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Thanks for advice! Ill try that.

Update: for some reason WIndows is not able to boot - BSOD with PAGE_FAULT_IN_NONPAGED_AREA code 0x00000050. Sometimes BSOD has different code 0x0000007E ( SYSTEM_THREAD_EXCEPTION_NOT_HANDLED )

I might look into seabios at see if it can be forced to load BIOS even though there is a mismatch.

See my first post on how to load your vbios from a dump, but you`ll still need to modify the devid if you`re dumping it from your card

Last edited by nbhs (2013-11-25 10:26:20)

Offline

#817 2013-11-25 10:29:31

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

See my first post on how to load your vbios from a dump, but you`ll still need to modify the devid if you`re dumping it from your card

Yes, I read it again and see what you mean. I will also have to update checksum after I patch the device ID i think. Thanks for the hint.

Offline

#818 2013-11-25 12:11:56

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to understand cpuset. nbhs' script uses the number of physical cores, but I have 4 physical and 4 logical (I7 3770); so if I want to have 6 vcpus in my guest what do I do? Do I use -smp 3,sockets=1,cores=3,threads=2 or something like that?

Sorry for the dumb question.

Last edited by Chetyre (2013-11-25 13:26:32)

Offline

#819 2013-11-25 13:24:59

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

do I use -smp 3,sockets=1,cores=3,threads=2

I think this would work. I think cpuset is used to pin CPU's to VM, is it what you want to do?

Offline

#820 2013-11-25 13:27:20

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
Chetyre wrote:

do I use -smp 3,sockets=1,cores=3,threads=2

I think this would work. I think cpuset is used to pin CPU's to VM, is it what you want to do?

Yeah, I want to squeeze as much performance from the processor as possible for the VM.

Offline

#821 2013-11-25 13:33:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

I'm trying to understand cpuset. nbhs' script uses the number of physical cores, but I have 4 physical and 4 logical (I7 3770); so if I want to have 6 vcpus in my guest what do I do? Do I use -smp 3,sockets=1,cores=3,threads=2 or something like that?

Sorry for the dumb question.

My script is broken on intel, i didnt take into consideration ht.

Offline

#822 2013-11-25 13:35:22

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Chetyre wrote:

I'm trying to understand cpuset. nbhs' script uses the number of physical cores, but I have 4 physical and 4 logical (I7 3770); so if I want to have 6 vcpus in my guest what do I do? Do I use -smp 3,sockets=1,cores=3,threads=2 or something like that?

Sorry for the dumb question.

My script is broken on intel, i didnt take into consideration ht.

In that case is there a big difference between taskset and cpuset?

Offline

#823 2013-11-25 13:48:00

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:
nbhs wrote:
Chetyre wrote:

I'm trying to understand cpuset. nbhs' script uses the number of physical cores, but I have 4 physical and 4 logical (I7 3770); so if I want to have 6 vcpus in my guest what do I do? Do I use -smp 3,sockets=1,cores=3,threads=2 or something like that?

Sorry for the dumb question.

My script is broken on intel, i didnt take into consideration ht.

In that case is there a big difference between taskset and cpuset?

cpuset is a tool for controlling cgroups, what a did is create cpu cgroup with x cores, then move everything to the system cgroup (in my case 2 cores) and run qemu on the user cgroup

Offline

#824 2013-11-25 14:14:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
aw wrote:

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.

With VBIOS you mean the bios on the video card? No I haven't updated that. I`d rather keep the original bios b/c as far as I understand it manages clocks/fans in a way I want it to - as a GTX 680. Is there a way to force seabios to run GPU BIOS even though there is a mismatch?

You won't have VGA on bare metal without a modified VBIOS either, so our emulation is accurate wink  The only way to force seabios to use the VBIOS would be to modify the code.

ila80 wrote:
aw wrote:

However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Thanks for advice! Ill try that.

Update: for some reason WIndows is not able to boot - BSOD with PAGE_FAULT_IN_NONPAGED_AREA code 0x00000050. Sometimes BSOD has different code 0x0000007E ( SYSTEM_THREAD_EXCEPTION_NOT_HANDLED )

I might look into seabios at see if it can be forced to load BIOS even though there is a mismatch.

IIRC the PAGE_FAULT_IN_NONPAGED_AREA is the manifestation of the -cpu host problem on Nvidia.  Try -cpu qemu64 or one of the CPU specific types.  Note that you can dump the VBIOS, modify it, then load it with romfile= to avoid flashing it back to hardware.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#825 2013-11-25 22:09:29

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I need your help,

In my setup i added a sata3 controler on my VM but when i whant to reboot my VM the sata3 controler break the boot.

My sata3 controler is an Asmedia 1061.

I think the probleme is to reset the controler when the VM reboot.

Any idea ?

Offline

#826 2013-11-26 09:00:39

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
ilya80 wrote:
aw wrote:

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.

With VBIOS you mean the bios on the video card? No I haven't updated that. I`d rather keep the original bios b/c as far as I understand it manages clocks/fans in a way I want it to - as a GTX 680. Is there a way to force seabios to run GPU BIOS even though there is a mismatch?

You won't have VGA on bare metal without a modified VBIOS either, so our emulation is accurate wink  The only way to force seabios to use the VBIOS would be to modify the code.

ila80 wrote:
aw wrote:

However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Thanks for advice! Ill try that.

Update: for some reason WIndows is not able to boot - BSOD with PAGE_FAULT_IN_NONPAGED_AREA code 0x00000050. Sometimes BSOD has different code 0x0000007E ( SYSTEM_THREAD_EXCEPTION_NOT_HANDLED )

I might look into seabios at see if it can be forced to load BIOS even though there is a mismatch.

IIRC the PAGE_FAULT_IN_NONPAGED_AREA is the manifestation of the -cpu host problem on Nvidia.  Try -cpu qemu64 or one of the CPU specific types.  Note that you can dump the VBIOS, modify it, then load it with romfile= to avoid flashing it back to hardware.

AW, thanks for your input, it is really appreciated!

I do boot Win7 on bare metal with my mainboard BIOS and I installed Quadro/Tesla/GRID drivers on it. Plays modern games as it did before smile However my next steps are download bios from the card, patch the device ID and the checksum, and then feed that to seabios. Re: -cpu options I`ve had problems with it before, played with it and ended up booting fine with -cpu host. I`ll give it a some tries for sure though.

Offline

#827 2013-11-29 08:37:11

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmm, i have problems passing through my Etron EJ168 USB 3 Controller.

i bind it to vfio-pci and pass it to the Windows 7 VM. There its recognised and i can install the (latest) driver by Etron. The driver seems to be intalled ok but there is no power on the ports.

I read someone (aw?) got a machine up and running with eJ168?

kernel 3.12 and still only qemu-kvm 1.60 (debian unstable).

Offline

#828 2013-11-29 11:40:21

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blitzschlag wrote:

hmm, i have problems passing through my Etron EJ168 USB 3 Controller.

i bind it to vfio-pci and pass it to the Windows 7 VM. There its recognised and i can install the (latest) driver by Etron. The driver seems to be intalled ok but there is no power on the ports.

I read someone (aw?) got a machine up and running with eJ168?

kernel 3.12 and still only qemu-kvm 1.60 (debian unstable).

$ lspci|grep Etron 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

Hi, the person with those controllers seems to be me big_smile although there might be more than one person with them

I managed to pass them to a Windows (7?) VM quite a while ago and they "worked" but etrons drivers and probably hardware were/are total garbage. The VM constantly threw BSODs at me, my mouse or kvm switch was lagging/randomly resetting when connected to one of the controllers and so on. This doesn't seem to be a problem with vfio though, since people without it are having problems too.

So yeah, this was probably not what you are looking for but my solution was to not use usb3 in the vm but usb2 controllers.


i'm sorry for my poor english wirting skills…

Offline

#829 2013-11-29 12:25:50

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
blitzschlag wrote:

hmm, i have problems passing through my Etron EJ168 USB 3 Controller.

i bind it to vfio-pci and pass it to the Windows 7 VM. There its recognised and i can install the (latest) driver by Etron. The driver seems to be intalled ok but there is no power on the ports.

I read someone (aw?) got a machine up and running with eJ168?

kernel 3.12 and still only qemu-kvm 1.60 (debian unstable).

$ lspci|grep Etron 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

Hi, the person with those controllers seems to be me big_smile although there might be more than one person with them

I managed to pass them to a Windows (7?) VM quite a while ago and they "worked" but etrons drivers and probably hardware were/are total garbage. The VM constantly threw BSODs at me, my mouse or kvm switch was lagging/randomly resetting when connected to one of the controllers and so on. This doesn't seem to be a problem with vfio though, since people without it are having problems too.

So yeah, this was probably not what you are looking for but my solution was to not use usb3 in the vm but usb2 controllers.

ah, ok, thx for the Information.

My main Problem or concern is that just passing through the usb-hdd and emulating usb has really poor performance (7-9MB/s write). At least passing through one of the usb2 Controllers will give me better performance.
I was kinda hoping to be able to use  the usb3 Controller for my usual video-editing (non-professional) stuff. As it is (with emulated usb) i have constant bufferings  which freaks me out ^^

oh, btw, i use Synergy for sharing my Keyboard and Mouse. Works pretty solid even with gaming.

Offline

#830 2013-11-29 12:31:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blitzschlag wrote:
andy123 wrote:
blitzschlag wrote:

hmm, i have problems passing through my Etron EJ168 USB 3 Controller.

i bind it to vfio-pci and pass it to the Windows 7 VM. There its recognised and i can install the (latest) driver by Etron. The driver seems to be intalled ok but there is no power on the ports.

I read someone (aw?) got a machine up and running with eJ168?

kernel 3.12 and still only qemu-kvm 1.60 (debian unstable).

$ lspci|grep Etron 03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

Hi, the person with those controllers seems to be me big_smile although there might be more than one person with them

I managed to pass them to a Windows (7?) VM quite a while ago and they "worked" but etrons drivers and probably hardware were/are total garbage. The VM constantly threw BSODs at me, my mouse or kvm switch was lagging/randomly resetting when connected to one of the controllers and so on. This doesn't seem to be a problem with vfio though, since people without it are having problems too.

So yeah, this was probably not what you are looking for but my solution was to not use usb3 in the vm but usb2 controllers.

ah, ok, thx for the Information.

My main Problem or concern is that just passing through the usb-hdd and emulating usb has really poor performance (7-9MB/s write). At least passing through one of the usb2 Controllers will give me better performance.
I was kinda hoping to be able to use  the usb3 Controller for my usual video-editing (non-professional) stuff. As it is (with emulated usb) i have constant bufferings  which freaks me out ^^

oh, btw, i use Synergy for sharing my Keyboard and Mouse. Works pretty solid even with gaming.

Looks like qemu supports xhci emulation

xhci controller support
-----------------------

There is also xhci host controller support available.  It got a lot
less testing than ehci and there are a bunch of known limitations, so
ehci may work better for you.  On the other hand the xhci hardware
design is much more virtualization-friendly, thus xhci emulation uses
less resources (especially cpu).  If you want to give xhci a try
use this to add the host controller ...

    qemu -device nec-usb-xhci,id=xhci

... then use "bus=xhci.0" when assigning usb devices.

You might want to take a look at that

Offline

#831 2013-11-29 15:52:22

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Looks like qemu supports xhci emulation

xhci controller support
-----------------------

There is also xhci host controller support available.  It got a lot
less testing than ehci and there are a bunch of known limitations, so
ehci may work better for you.  On the other hand the xhci hardware
design is much more virtualization-friendly, thus xhci emulation uses
less resources (especially cpu).  If you want to give xhci a try
use this to add the host controller ...

    qemu -device nec-usb-xhci,id=xhci

... then use "bus=xhci.0" when assigning usb devices.

You might want to take a look at that

hmm,

i tried

-device nec-usb-xhci,id=xhci \ -drive file=/dev/sde1,id=usbhdd,format=raw \ -device usb-storage,bus=xhci.0,drive=usbhdd \

to no avail. there is no new storage device present in my VM.

am i missing something?

Offline

#832 2013-11-29 16:06:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blitzschlag wrote:

hmm,

i tried

-device nec-usb-xhci,id=xhci \ -drive file=/dev/sde1,id=usbhdd,format=raw \ -device usb-storage,bus=xhci.0,drive=usbhdd \

to no avail. there is no new storage device present in my VM.

am i missing something?

Why dont you pass it as a virtio device?


-drive file=/dev/sde1,if=virtio,id=disk

You can get the virtio drivers here: http://alt.fedoraproject.org/pub/alt/vi … st/images/

Last edited by nbhs (2013-11-29 16:07:57)

Offline

#833 2013-11-29 18:09:11

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thy for the Input

if i pass it with

-device nec-usb-xhci,id=xhci \ -drive file=/dev/sde1,if=virtio,id=usbhdd \ -device usb-storage,bus=xhci.0,drive=usbhdd \

i get the error:

Property 'virtio-blk-pci.drive' can't take value 'usbhdd', it's in use

googeling around says that i defined the interface two times.

i'm probably just thick but i can't see where the problem is.

Offline

#834 2013-11-30 00:44:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blitzschlag wrote:

thy for the Input

if i pass it with

-device nec-usb-xhci,id=xhci \ -drive file=/dev/sde1,if=virtio,id=usbhdd \ -device usb-storage,bus=xhci.0,drive=usbhdd \

i get the error:

Property 'virtio-blk-pci.drive' can't take value 'usbhdd', it's in use

googeling around says that i defined the interface two times.

i'm probably just thick but i can't see where the problem is.

Forget about the xhci controller, what i ment was something like this:

qemu-system-x86_64 -drive file=/dev/sde1,if=virtio,id

Offline

#835 2013-11-30 04:21:53

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, great work here so far. I am excited to get my setup to work. I have run into a problem getting my second PCIe vga card (AMD 7870) passed through to the vm. Any attempt in passing it through results in the whole system hanging which mean I need to power down/up the system. There appear to be no logs produced when the attempt is made. The system just hangs. Any insights on what could be happening?

Here are my specs.

Supermicro X7DWA-N
16DDR2 ECC Ram
2x AMD 7870 running open source radeon drivers.
Fakeraid Sil3132 eSata card
Dual Xeon 5462 (VT-x enabled)

I'm at a loss here and have been reading through the posts and other forums. I must be missing something here. Any help is appreciated. Thanks

I have an nvidia 660ti I might attempt to pass through instead just to see if its something there. But I suspect its not the issue.

Last edited by paradexes (2013-11-30 06:47:54)

Offline

#836 2013-11-30 18:21:26

sharkwouter
Member
Registered: 2013-11-30
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This guide is awesome, my Windows installation works great. I do have one problem though, both graphics cards in my system have the same hardware id which makes me unable to run Xorg on my arch installation directly. I could run a second vm, but that seems a bit inefficient to me. Did anyone encounter this issue with 2 identical graphics cards and fix it? I don't know if it helps, but I'm using an AMD FX 8320 processor, and ASUS Sabertooth 990FX mobo and 2 AMD Radeon HD7850 graphics cards.

Offline

#837 2013-12-01 08:32:53

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:

Hey guys, great work here so far. I am excited to get my setup to work. I have run into a problem getting my second PCIe vga card (AMD 7870) passed through to the vm. Any attempt in passing it through results in the whole system hanging which mean I need to power down/up the system. There appear to be no logs produced when the attempt is made. The system just hangs. Any insights on what could be happening?

Here are my specs.

Supermicro X7DWA-N
16DDR2 ECC Ram
2x AMD 7870 running open source radeon drivers.
Fakeraid Sil3132 eSata card
Dual Xeon 5462 (VT-x enabled)

I'm at a loss here and have been reading through the posts and other forums. I must be missing something here. Any help is appreciated. Thanks

I have an nvidia 660ti I might attempt to pass through instead just to see if its something there. But I suspect its not the issue.

Your CPU's have no VT-d support.

Last edited by empie (2013-12-01 08:33:06)

Offline

#838 2013-12-01 10:59:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sharkwouter wrote:

This guide is awesome, my Windows installation works great. I do have one problem though, both graphics cards in my system have the same hardware id which makes me unable to run Xorg on my arch installation directly. I could run a second vm, but that seems a bit inefficient to me. Did anyone encounter this issue with 2 identical graphics cards and fix it? I don't know if it helps, but I'm using an AMD FX 8320 processor, and ASUS Sabertooth 990FX mobo and 2 AMD Radeon HD7850 graphics cards.

On recent kernels im able to unload my secondary card from the radeon module  (before starting X) and then binding it to vfio without crashing the host (i do get some ugly kernel warns though) and without having to use pci-stub, you could try it.

Last edited by nbhs (2013-12-01 11:00:00)

Offline

#839 2013-12-01 13:06:24

sharkwouter
Member
Registered: 2013-11-30
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
sharkwouter wrote:

This guide is awesome, my Windows installation works great. I do have one problem though, both graphics cards in my system have the same hardware id which makes me unable to run Xorg on my arch installation directly. I could run a second vm, but that seems a bit inefficient to me. Did anyone encounter this issue with 2 identical graphics cards and fix it? I don't know if it helps, but I'm using an AMD FX 8320 processor, and ASUS Sabertooth 990FX mobo and 2 AMD Radeon HD7850 graphics cards.

On recent kernels im able to unload my secondary card from the radeon module  (before starting X) and then binding it to vfio without crashing the host (i do get some ugly kernel warns though) and without having to use pci-stub, you could try it.

I'll give it a try. Did anyone manage to pass-through his primary graphics adapter? I'd like to try running 2 vms, but I don't know if that could work.

Offline

#840 2013-12-01 15:04:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

empie wrote:
paradexes wrote:

Hey guys, great work here so far. I am excited to get my setup to work. I have run into a problem getting my second PCIe vga card (AMD 7870) passed through to the vm. Any attempt in passing it through results in the whole system hanging which mean I need to power down/up the system. There appear to be no logs produced when the attempt is made. The system just hangs. Any insights on what could be happening?

Here are my specs.

Supermicro X7DWA-N
16DDR2 ECC Ram
2x AMD 7870 running open source radeon drivers.
Fakeraid Sil3132 eSata card
Dual Xeon 5462 (VT-x enabled)

I'm at a loss here and have been reading through the posts and other forums. I must be missing something here. Any help is appreciated. Thanks

I have an nvidia 660ti I might attempt to pass through instead just to see if its something there. But I suspect its not the issue.

Your CPU's have no VT-d support.

VT-d is only listed for the CPU when the CPU hosts root ports.  This is an older system with a Northbridge-Southbridge architecture.  The datasheet for the 5400 series chipset listed for this motherboard does say it supports VT-d.  Unfortunately I don't have any idea why this is hanging, but I'd definitely try the Nvidia card.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#841 2013-12-02 03:31:08

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

empie wrote:
paradexes wrote:

Hey guys, great work here so far. I am excited to get my setup to work. I have run into a problem getting my second PCIe vga card (AMD 7870) passed through to the vm. Any attempt in passing it through results in the whole system hanging which mean I need to power down/up the system. There appear to be no logs produced when the attempt is made. The system just hangs. Any insights on what could be happening?

Here are my specs.

Supermicro X7DWA-N
16DDR2 ECC Ram
2x AMD 7870 running open source radeon drivers.
Fakeraid Sil3132 eSata card
Dual Xeon 5462 (VT-x enabled)

I'm at a loss here and have been reading through the posts and other forums. I must be missing something here. Any help is appreciated. Thanks

I have an nvidia 660ti I might attempt to pass through instead just to see if its something there. But I suspect its not the issue.

Your CPU's have no VT-d support.

Sorry man intel clearly states the CPU has the support. VT-x And I do have it otherwise I would not even be able to properly set up a 64bit VM. Thats one aspect of this tech that is required. It works, I just am having issues passing the card through to the VM. IOMMU is clearly there. Clearly I am missing something.
http://ark.intel.com/products/virtualizationtechnology Its in the list. E5462

I am giving the nvidia card a try on the host system on the guest as well. Hopefully that yields results. Thanks.

Last edited by paradexes (2013-12-02 03:32:28)

Offline

#842 2013-12-02 03:47:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:
empie wrote:
paradexes wrote:

Hey guys, great work here so far. I am excited to get my setup to work. I have run into a problem getting my second PCIe vga card (AMD 7870) passed through to the vm. Any attempt in passing it through results in the whole system hanging which mean I need to power down/up the system. There appear to be no logs produced when the attempt is made. The system just hangs. Any insights on what could be happening?

Here are my specs.

Supermicro X7DWA-N
16DDR2 ECC Ram
2x AMD 7870 running open source radeon drivers.
Fakeraid Sil3132 eSata card
Dual Xeon 5462 (VT-x enabled)

I'm at a loss here and have been reading through the posts and other forums. I must be missing something here. Any help is appreciated. Thanks

I have an nvidia 660ti I might attempt to pass through instead just to see if its something there. But I suspect its not the issue.

Your CPU's have no VT-d support.

Sorry man intel clearly states the CPU has the support. VT-x And I do have it otherwise I would not even be able to properly set up a 64bit VM. Thats one aspect of this tech that is required. It works, I just am having issues passing the card through to the VM. IOMMU is clearly there. Clearly I am missing something.
http://ark.intel.com/products/virtualizationtechnology Its in the list. E5462

I am giving the nvidia card a try on the host system on the guest as well. Hopefully that yields results. Thanks.

VT-d != VT-x


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#843 2013-12-02 05:29:50

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
paradexes wrote:
empie wrote:

Your CPU's have no VT-d support.

Sorry man intel clearly states the CPU has the support. VT-x And I do have it otherwise I would not even be able to properly set up a 64bit VM. Thats one aspect of this tech that is required. It works, I just am having issues passing the card through to the VM. IOMMU is clearly there. Clearly I am missing something.
http://ark.intel.com/products/virtualizationtechnology Its in the list. E5462

I am giving the nvidia card a try on the host system on the guest as well. Hopefully that yields results. Thanks.

VT-d != VT-x

Agreed and thanks. Just glad its not just me saying it. That said I am still kind of at odds with the host freezing. If anyone has any ideas that would be great. I have added the nvidia card and am going to give it a go with that. Altho I think it may likley happen with that one as well.

Offline

#844 2013-12-02 14:23:49

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,291

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@paradexes

I think you misunderstood aw. VT-x is not the same thing as VT-d. And according to the ark your CPUs do not support VT-d. I don't think VT-d even existed when your CPU hit the market (2007).

The good news is that your mobo does support VT-d, so your setup isn't a total loss. Here's a list of '5000 Sequence' Xeons that support VT-d.

Last edited by alphaniner (2013-12-02 14:35:32)


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#845 2013-12-02 16:20:19

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for clearing it up. Yes you are correct. I did some checking myself. I guess my whole concern is whether I will be able to pass the GPU at this point or not. I am not terribly concerned about any other hardware besides maybe USB hardware.

Offline

#846 2013-12-02 17:24:50

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

IIRC the PAGE_FAULT_IN_NONPAGED_AREA is the manifestation of the -cpu host problem on Nvidia.  Try -cpu qemu64 or one of the CPU specific types.  Note that you can dump the VBIOS, modify it, then load it with romfile= to avoid flashing it back to hardware.

You were right about CPU type - I used SandyBridge and was able to install.

After installing WIn7, I installed GRID K2 drivers, but now windows report Code 12 on NVIDIA GRID K2 device: This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system.

I have tried adding -vga none to the qemu command line and booting like this after I installed the drivers, but i dont get any signal on either video outputs of the card still.

Any ideas anyone?

Thanks,
Ilya.

Offline

#847 2013-12-02 17:32:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
aw wrote:

IIRC the PAGE_FAULT_IN_NONPAGED_AREA is the manifestation of the -cpu host problem on Nvidia.  Try -cpu qemu64 or one of the CPU specific types.  Note that you can dump the VBIOS, modify it, then load it with romfile= to avoid flashing it back to hardware.

You were right about CPU type - I used SandyBridge and was able to install.

After installing WIn7, I installed GRID K2 drivers, but now windows report Code 12 on NVIDIA GRID K2 device: This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system.

I have tried adding -vga none to the qemu command line and booting like this after I installed the drivers, but i dont get any signal on either video outputs of the card still.

Any ideas anyone?

Real GRID cards report the same Code 12 on the q35 chipset model, try 440fx instead.  I can't figure out what resources it's missing, nor have I had time to experiment.  Linux guests will work fine on the q35 model.  Again, you're not going to get pre-boot VGA output unless the VBIOS matches the hardware.  Also, Nvidia does not support GRID in a VGA mode, it's only supported as a secondary device to emulated VGA.  The professional version of the drivers detect that they're running on a VM and behave differently.  The quirks enabled with x-vga=on do not work correctly with those drivers.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#848 2013-12-03 00:46:04

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well it appears installing the nvidia card and blacklisting it from the host sems to resolve the hanging issue. However now I am getting this error. Since the motherboard supports IOMMU would it not support vfio? at least in theory. Any thoughts?

Error starting domain: internal error: Unable to reset PCI device 0000:02:00.0: internal error: Active 0000:02:00.1 devices on bus with 0000:02:00.0, not doing bus reset

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 698, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: Unable to reset PCI device 0000:02:00.0: internal error: Active 0000:02:00.1 devices on bus with 0000:02:00.0, not doing bus reset

Offline

#849 2013-12-03 01:15:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:

Well it appears installing the nvidia card and blacklisting it from the host sems to resolve the hanging issue. However now I am getting this error. Since the motherboard supports IOMMU would it not support vfio? at least in theory. Any thoughts?

Error starting domain: internal error: Unable to reset PCI device 0000:02:00.0: internal error: Active 0000:02:00.1 devices on bus with 0000:02:00.0, not doing bus reset

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 698, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: Unable to reset PCI device 0000:02:00.0: internal error: Active 0000:02:00.1 devices on bus with 0000:02:00.0, not doing bus reset

Suspect you're not using vfio because libvirt only tries to reset devices with legacy KVM device assignment.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#850 2013-12-03 04:12:49

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
paradexes wrote:

Well it appears installing the nvidia card and blacklisting it from the host sems to resolve the hanging issue. However now I am getting this error. Since the motherboard supports IOMMU would it not support vfio? at least in theory. Any thoughts?

Error starting domain: internal error: Unable to reset PCI device 0000:02:00.0: internal error: Active 0000:02:00.1 devices on bus with 0000:02:00.0, not doing bus reset

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 698, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: Unable to reset PCI device 0000:02:00.0: internal error: Active 0000:02:00.1 devices on bus with 0000:02:00.0, not doing bus reset

Suspect you're not using vfio because libvirt only tries to reset devices with legacy KVM device assignment.

Ok. I will keep at it and post my results. Thanks. Being that the motherboard supports IOMMU I assume its possible to make this work?

Offline

#851 2013-12-03 16:26:25

abarahc
Member
Registered: 2010-04-30
Posts: 118

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! sorry if all this questio below are already answered and i didnt get them, but!



Question - 01 - How can i check it?

nbhs wrote:

Setting up vfio and kvm modules

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:


i've read this http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM
and on step 3 we have this command

dmesg | grep -e DMAR -e IOMMU

which i think is the answer for my question! True?

Question - 02 - I got only one line on command's output
the command

dmesg | grep -e DMAR -e IOMMU

must have a positive returning like this

DMAR:DRHD base: 0x000000feb03000 flags: 0x0 IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000

but in my case i got this

[ 0.000000] Intel-IOMMU: enabled

so this is mean i dont have remapping support? or is something else?

Question - 03 - Why vfio?

nbhs wrote:

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

on the tutorial link is mentioned another path to make this procedures. so whats the real difference between pci-stub and vfio-stub?

on my case both solution do not work correctly. every time a try

echo XX:XXX:XX > /sys/bus/pci/drivers/vfio-pci/new_id

or

echo XX:XXX:XX > /sys/bus/pci/drivers/pci-stub/new_id

i got permission issues.

those files are on write mode only! and even after i changing the modes to read/write or read/write/execute nothing change error message about i can write nothing

1. yes i was making this instructions as root
2. yes, i tried "su" and "su -". no changes

Question - 04 So whats the problem with this files?

Question - 05 - Where is this BUS id?

in a few lines of this tuto when you have to pass a devices to qemu you write the correct bus for them. But i really dont know where i can find this info.

sometimes is "bus=root.1" other times are "bus=pcie.0"

so how can i determine?


"...dear cinnamon tear..."

Offline

#852 2013-12-03 16:49:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abarahc wrote:

Hi! sorry if all this questio below are already answered and i didnt get them, but!



Question - 01 - How can i check it?

nbhs wrote:

Setting up vfio and kvm modules

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:


i've read this http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM
and on step 3 we have this command

dmesg | grep -e DMAR -e IOMMU

which i think is the answer for my question! True?

Interrupt remapping depends on IOMMU support, but having IOMMU support doesn't imply interrupt remapping.  You can look for something like "Enabled IRQ remapping in ...... mode", or you can just try it.  vfio won't let you assign a device without it or the option to bypass it, so if it doesn't work, look in dmesg and you'll see a message about how to enable it if that's the problem.

Question - 02 - I got only one line on command's output
the command

dmesg | grep -e DMAR -e IOMMU

must have a positive returning like this

DMAR:DRHD base: 0x000000feb03000 flags: 0x0 IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000

but in my case i got this

[ 0.000000] Intel-IOMMU: enabled

so this is mean i dont have remapping support? or is something else?

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

Question - 03 - Why vfio?

nbhs wrote:

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

on the tutorial link is mentioned another path to make this procedures. so whats the real difference between pci-stub and vfio-stub?

on my case both solution do not work correctly. every time a try

echo XX:XXX:XX > /sys/bus/pci/drivers/vfio-pci/new_id

or

echo XX:XXX:XX > /sys/bus/pci/drivers/pci-stub/new_id

i got permission issues.

those files are on write mode only! and even after i changing the modes to read/write or read/write/execute nothing change error message about i can write nothing

1. yes i was making this instructions as root
2. yes, i tried "su" and "su -". no changes

Question - 04 So whats the problem with this files?

You're using them incorrectly.  Note that the script echos $vendor and $device to the new_id file while you're trying to use the (incomplete) PCI address.  You're getting permission denied because of what you're trying to echo, not because of file permissions.  As to why vfio-pci vs pci-stub, vfio is a new userspace driver interface where qemu is just a userspace driver using it for device assignment.  Legacy kvm device assignment with pci-stub is effectively deprecated.  Also, if you want to do VGA, vfio is the only device assignment interface with this support.

Question - 05 - Where is this BUS id?

in a few lines of this tuto when you have to pass a devices to qemu you write the correct bus for them. But i really dont know where i can find this info.

sometimes is "bus=root.1" other times are "bus=pcie.0"

so how can i determine?

bus= addresses are qemu defined identifiers.  When using the q35 model there is an implicit id=pcie.0 on the root bus.  That means any devices placed there using bus=pcie.0 will appear on the PCIe root complex.  When root ports are added to the topology they create a new bus.  You're able to name that bus with the id= option.  root.1 is just a name some people use, it could just as easily be named bob.  To place a device on that bus, use the bus=bob option.

When specifying a device to be assigned, it's the host= option.  This always takes the form of bus:device.function (ex. 01:00.0) or segment:bus:device.function (ex. 0000:01:00.0).  These match your physical device layout.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#853 2013-12-03 17:08:06

abarahc
Member
Registered: 2010-04-30
Posts: 118

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

Thanks so much to reply so fast!

aw wrote:

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

the "intel_iommu=on" is set as parameter on kernel boot line!

linux /boot/vmlinuz-linux root=UUID=aaed6a49-c921-463c-bc9b-a22b74a6ee4e rw quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0de9,8086:1e20

but i cant find anything about this on my notebook BIOS! (Asus K45VM). This probably means i dont have support to remapping, right? hmm

The pci.stub.ids are based on this output

lspci

00:00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation HM76 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation GF108M [GeForce GT 630M] (rev a1) 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 07) 03:00.0 Network controller: Qualcomm Atheros AR9485 Wireless Network Adapter (rev 01)

lspci -n

00:00.0 0600: 8086:0154 (rev 09) 00:01.0 0604: 8086:0151 (rev 09) 00:02.0 0300: 8086:0166 (rev 09) 00:14.0 0c03: 8086:1e31 (rev 04) 00:16.0 0780: 8086:1e3a (rev 04) 00:1a.0 0c03: 8086:1e2d (rev 04) 00:1b.0 0403: 8086:1e20 (rev 04) 00:1c.0 0604: 8086:1e10 (rev c4) 00:1c.1 0604: 8086:1e12 (rev c4) 00:1d.0 0c03: 8086:1e26 (rev 04) 00:1f.0 0601: 8086:1e59 (rev 04) 00:1f.2 0106: 8086:1e03 (rev 04) 00:1f.3 0c05: 8086:1e22 (rev 04) 01:00.0 0300: 10de:0de9 (rev a1) 02:00.0 0200: 10ec:8168 (rev 07) 03:00.0 0280: 168c:0032 (rev 01)

"...dear cinnamon tear..."

Offline

#854 2013-12-03 17:21:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abarahc wrote:

@aw

Thanks so much to reply so fast!

aw wrote:

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

the "intel_iommu=on" is set as parameter on kernel boot line!

linux /boot/vmlinuz-linux root=UUID=aaed6a49-c921-463c-bc9b-a22b74a6ee4e rw quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0de9,8086:1e20

but i cant find anything about this on my notebook BIOS! (Asus K45VM). This probably means i dont have support to remapping, right? hmm

According to Asus, the CPUs available for this laptop are:

Intel® Core™ i7 3610QM Processor
Intel® Core™ i5 3210M Processor
Intel® Core™ i3 3110M Processor

Cross referencing these to http://ark.intel.com, you do not have VT-d (assuming the rPGA version of 3210M).  Full stop, you can't do device assignment.  But the good news is that it probably would not have worked anyway.  Secondary graphics devices built into laptops are extremely customized to the device, often requiring vendor specific drivers.  I have not heard of any cases of these working when assigned to a VM.  Folks here that are having success are using fully independent discrete graphics cards.  Sorry.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#855 2013-12-03 17:23:36

abarahc
Member
Registered: 2010-04-30
Posts: 118

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
abarahc wrote:

@aw

Thanks so much to reply so fast!

aw wrote:

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

the "intel_iommu=on" is set as parameter on kernel boot line!

linux /boot/vmlinuz-linux root=UUID=aaed6a49-c921-463c-bc9b-a22b74a6ee4e rw quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0de9,8086:1e20

but i cant find anything about this on my notebook BIOS! (Asus K45VM). This probably means i dont have support to remapping, right? hmm

According to Asus, the CPUs available for this laptop are:

Intel® Core™ i7 3610QM Processor
Intel® Core™ i5 3210M Processor
Intel® Core™ i3 3110M Processor

Cross referencing these to http://ark.intel.com, you do not have VT-d (assuming the rPGA version of 3210M).  Full stop, you can't do device assignment.  But the good news is that it probably would not have worked anyway.  Secondary graphics devices built into laptops are extremely customized to the device, often requiring vendor specific drivers.  I have not heard of any cases of these working when assigned to a VM.  Folks here that are having success are using fully independent discrete graphics cards.  Sorry.

Ok buddy :'/
Thank you very much for the explanatiion.


"...dear cinnamon tear..."

Offline

#856 2013-12-03 17:30:31

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

Offline

#857 2013-12-03 17:36:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#858 2013-12-03 17:39:09

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Real GRID cards report the same Code 12 on the q35 chipset model, try 440fx instead.  I can't figure out what resources it's missing, nor have I had time to experiment.  Linux guests will work fine on the q35 model.  Again, you're not going to get pre-boot VGA output unless the VBIOS matches the hardware.  Also, Nvidia does not support GRID in a VGA mode, it's only supported as a secondary device to emulated VGA.  The professional version of the drivers detect that they're running on a VM and behave differently.  The quirks enabled with x-vga=on do not work correctly with those drivers.

Thanks for the suggestions!

I downloaded BIOS from GPU using GPU-Z, patched device ID, updated the CRC, added romfile=/path/to/rom/file to options for VGA portion of GPU, still no luck with output to monitors on VM start. All I get in kern.log is:

Dec 3 21:29:33 cave-lin kernel: [ 286.128960] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Dec 3 21:30:10 cave-lin kernel: [ 324.069324] kvm: zapping shadow pages for mmio generation wraparound Dec 3 21:30:10 cave-lin kernel: [ 324.106954] vfio-pci 0000:01:00.0: irq 57 for MSI/MSI-X

Also I have tried running VM with FX440 chipset, but it doesnt seem to have pcie bus, so no luck starting VM at all:

kalujny@cave-lin:~$ sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/opt/kalujny/thrash/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -usbdevice tablet qemu-system-x86_64: -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1: Bus 'pcie.0' not found 

Offline

#859 2013-12-03 17:46:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:
aw wrote:

Real GRID cards report the same Code 12 on the q35 chipset model, try 440fx instead.  I can't figure out what resources it's missing, nor have I had time to experiment.  Linux guests will work fine on the q35 model.  Again, you're not going to get pre-boot VGA output unless the VBIOS matches the hardware.  Also, Nvidia does not support GRID in a VGA mode, it's only supported as a secondary device to emulated VGA.  The professional version of the drivers detect that they're running on a VM and behave differently.  The quirks enabled with x-vga=on do not work correctly with those drivers.

Thanks for the suggestions!

I downloaded BIOS from GPU using GPU-Z, patched device ID, updated the CRC, added romfile=/path/to/rom/file to options for VGA portion of GPU, still no luck with output to monitors on VM start. All I get in kern.log is:

Dec 3 21:29:33 cave-lin kernel: [ 286.128960] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Dec 3 21:30:10 cave-lin kernel: [ 324.069324] kvm: zapping shadow pages for mmio generation wraparound Dec 3 21:30:10 cave-lin kernel: [ 324.106954] vfio-pci 0000:01:00.0: irq 57 for MSI/MSI-X

Also I have tried running VM with FX440 chipset, but it doesnt seem to have pcie bus, so no luck starting VM at all:

kalujny@cave-lin:~$ sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/opt/kalujny/thrash/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -usbdevice tablet qemu-system-x86_64: -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1: Bus 'pcie.0' not found 

Root ports (ioh3420) are incompatible with 44fx, the device needs to be placed on the host bridge (pci.0) which is the default if not specified.  Try something like this:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin \ -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,addr=07.1 \ -drive file=/opt/kalujny/thrash/windows.img,if=none,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -usbdevice tablet 

(I chose device address 7, move it if you want)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#860 2013-12-03 17:48:01

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
paradexes wrote:

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

Last edited by paradexes (2013-12-03 17:49:15)

Offline

#861 2013-12-03 17:56:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:
aw wrote:
paradexes wrote:

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

I'm sorry to say that I will be incredibly surprised if it has AMD IOMMU (AMD-Vi) support.  AMD-V (no i) is the CPU virtualization, aka SVM.  AFAIK the RD870 (maybe 890) was the first AMD-IOMMU chipset.  Most people using AMD for device assignment are using an 970/990.  Note that documentation and BIOS for AMD system can be very confusing because they'll refer to the onboard GART as an IOMMU.  While originally intended for graphics, it became a general purpose IOMMU for many system.  Unfortunately to support device assignment we need an IOMMU capable of both translation and isolation, the GART only has translation.  Furthermore, I believe it's a window based IOMMU rather than page table based, so even if you wanted to forego isolation, it doesn't even support the model we use for device assignment.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#862 2013-12-04 04:09:13

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
paradexes wrote:
aw wrote:

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

I'm sorry to say that I will be incredibly surprised if it has AMD IOMMU (AMD-Vi) support.  AMD-V (no i) is the CPU virtualization, aka SVM.  AFAIK the RD870 (maybe 890) was the first AMD-IOMMU chipset.  Most people using AMD for device assignment are using an 970/990.  Note that documentation and BIOS for AMD system can be very confusing because they'll refer to the onboard GART as an IOMMU.  While originally intended for graphics, it became a general purpose IOMMU for many system.  Unfortunately to support device assignment we need an IOMMU capable of both translation and isolation, the GART only has translation.  Furthermore, I believe it's a window based IOMMU rather than page table based, so even if you wanted to forego isolation, it doesn't even support the model we use for device assignment.


Well this is interesting. AMD-Vi IS IOMMU. Least according to this wiki, I think I might be good here.  The chipset for the board is reported to be working for VGA passthrough in xen. I would imagine the same would apply here.
http://en.wikipedia.org/wiki/X86_virtualization

Offline

#863 2013-12-04 05:23:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:
aw wrote:
paradexes wrote:

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

I'm sorry to say that I will be incredibly surprised if it has AMD IOMMU (AMD-Vi) support.  AMD-V (no i) is the CPU virtualization, aka SVM.  AFAIK the RD870 (maybe 890) was the first AMD-IOMMU chipset.  Most people using AMD for device assignment are using an 970/990.  Note that documentation and BIOS for AMD system can be very confusing because they'll refer to the onboard GART as an IOMMU.  While originally intended for graphics, it became a general purpose IOMMU for many system.  Unfortunately to support device assignment we need an IOMMU capable of both translation and isolation, the GART only has translation.  Furthermore, I believe it's a window based IOMMU rather than page table based, so even if you wanted to forego isolation, it doesn't even support the model we use for device assignment.


Well this is interesting. AMD-Vi IS IOMMU. Least according to this wiki, I think I might be good here.  The chipset for the board is reported to be working for VGA passthrough in xen. I would imagine the same would apply here.
http://en.wikipedia.org/wiki/X86_virtualization

Found this page - http://www.amd.com/us/products/server/p … tions.aspx

And here http://en.wikipedia.org/wiki/AMD_800_chipset_series indicates that the SR5690 is a rebranding of the RD890, so I agree, you should be good.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#864 2013-12-05 14:33:58

nivs
Member
Registered: 2013-12-05
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'd try to passthrough GTX780 with i7-4771 CPU and Z78 chipset, but got "Invalid ROM contents" in dmesg and 43 or 12 error in windows. Kernel and qemu from first post.

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=b011c3ab-e015-4d13-b5d8-afd54b708603 rw max_loop=64 intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a vfio_iommu_type1.allow_unsafe_interrupts=1
# zcat /proc/config.gz| grep -i vfio 22:05 pts/2 CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

Win 7 x64 and win 8 x64 were installed. Host GPU is Intel, but now it's off, cable connect to GTX 780, i see terminals. All work do in vncserver from notebook or SSH connection with "-nographic -monitor stdio". When i run this:

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/var/lib/libvirt/images/780.rom -device vfio-pci,host=0000:01:00.1,bus=root.1,addr=00.1 -device ahci,id=ahci0,bus=pcie.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pcie.0,addr=0x6 -drive file=/var/lib/libvirt/images/windVirt7.img,if=none,id=drive-sata0-0-0,format=qcow2,cache=none -device ide-hd,bus=ahci0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/ru_7x64.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 -net nic -net bridge,br=xenbr0

in dmesg appers this:

[ 1175.295251] device tap0 entered promiscuous mode [ 1175.295332] xenbr0: port 2(tap0) entered forwarding state [ 1175.295335] xenbr0: port 2(tap0) entered forwarding state [ 1175.919638] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 1175.919685] vfio-pci 0000:01:00.0: Invalid ROM contents [ 1175.954591] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref] [ 1179.035148] kvm: zapping shadow pages for mmio generation wraparound

With -vga none this: http://pastebin.com/MPJnUY3a
And in terminal:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/var/lib/libvirt/images/780.rom: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/var/lib/libvirt/images/780.rom: VFIO 0000:01:00.0 BAR 3 mmap unsupported. Performance may be slow qemu-system-x86_64: -device vfio-pci,host=01:00.1,addr=07.1: Warning, device 0000:01:00.1 does not support reset

With

#!/bin/bash qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin \ -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/var/lib/libvirt/images/780.rom -device vfio-pci,host=01:00.1,addr=07.1 \ -drive file=/var/lib/libvirt/images/virt.img,if=none,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -usbdevice tablet -vga none

i also get this. What should I do to make it finally work? I'm doing it for 4 days, with Xen also didn't work. If i wait about 40 sec with -vga none, in monitor (GTX 780) appeard this http://f-bit.ru/uploads/297542.jpg. 40 sec is time to select "repair" in menu. MB's BIOS up to date, GTX's BIOS too.

5-10 minutes after the start in the terminal there was a lot of messages:

qemu-system-x86_64: vfio_bar_write(,0x12ffc, 0x0, 4) failed: Device or resource busy qemu-system-x86_64: vfio_bar_write(,0x98, 0xbffa41, 4) failed: Device or resource busy qemu-system-x86_64: vfio_bar_write(,0x9c, 0x0, 4) failed: Device or resource busy qemu-system-x86_64: vfio_bar_write(,0x13000, 0xabcdabcd, 4) failed: Device or resource busy qemu-system-x86_64: vfio_bar_write(,0x13004, 0xabcdabcd, 4) failed: Device or resource busy qemu-system-x86_64: vfio_bar_write(,0x13008, 0xabcdabcd, 4) failed: Device or resource busy qemu-system-x86_64: vfio_bar_write(,0x1300c, 0xabcdabcd, 4) failed: Device or resource busy

With qemu debug:

<4>[ 108.175567] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref] <2>[ 110.177869] ERROR: DMA PTE for vPFN 0x0 already set (to 416bc3003 not 403400083)

or

<3>[ 89.103901] irq 16: nobody cared (try booting with the "irqpoll" option) <4>[ 89.106576] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G WC 3.12.0-1-mainline #1 <4>[ 89.106577] Hardware name: Gigabyte Technology Co., Ltd. Z87M-HD3/Z87M-HD3, BIOS F7 08/03/2013 <4>[ 89.106578] ffff88042841d68c ffff88043f203e48 ffffffff814f5a4b ffff88042841d600 <4>[ 89.106580] ffff88043f203e70 ffffffff810b8e22 ffff88042841d600 0000000000000010 <4>[ 89.106581] 0000000000000000 ffff88043f203eb0 ffffffff810b9248 ffffffff813d4402 <4>[ 89.106583] Call Trace: <4>[ 89.106584] <IRQ> [<ffffffff814f5a4b>] dump_stack+0x54/0x8d <4>[ 89.106591] [<ffffffff810b8e22>] __report_bad_irq+0x32/0xd0 <4>[ 89.106593] [<ffffffff810b9248>] note_interrupt+0x138/0x1f0 <4>[ 89.106596] [<ffffffff813d4402>] ? cpuidle_enter_state+0x52/0xc0 <4>[ 89.106598] [<ffffffff810b6c39>] handle_irq_event_percpu+0xf9/0x250 <4>[ 89.106599] [<ffffffff810b6dcd>] handle_irq_event+0x3d/0x60 <4>[ 89.106601] [<ffffffff810b9dba>] handle_fasteoi_irq+0x5a/0x100 <4>[ 89.106605] [<ffffffff81016b0e>] handle_irq+0x1e/0x30 <4>[ 89.106607] [<ffffffff8150690d>] do_IRQ+0x4d/0xc0 <4>[ 89.106609] [<ffffffff814fcfed>] common_interrupt+0x6d/0x6d <4>[ 89.106610] <EOI> [<ffffffff813d4402>] ? cpuidle_enter_state+0x52/0xc0 <4>[ 89.106613] [<ffffffff813d43f8>] ? cpuidle_enter_state+0x48/0xc0 <4>[ 89.106614] [<ffffffff813d4537>] cpuidle_idle_call+0xc7/0x280 <4>[ 89.106616] [<ffffffff8101e30e>] arch_cpu_idle+0xe/0x30 <4>[ 89.106617] [<ffffffff810b6207>] cpu_startup_entry+0x267/0x2e0 <4>[ 89.106620] [<ffffffff814e8254>] rest_init+0x84/0x90 <4>[ 89.106621] [<ffffffff818e2f01>] start_kernel+0x414/0x420 <4>[ 89.106622] [<ffffffff818e28f6>] ? repair_env_string+0x5c/0x5c <4>[ 89.106624] [<ffffffff818e2120>] ? early_idt_handlers+0x120/0x120 <4>[ 89.106625] [<ffffffff818e25de>] x86_64_start_reservations+0x2a/0x2c <4>[ 89.106626] [<ffffffff818e26e8>] x86_64_start_kernel+0x108/0x117 <3>[ 89.106627] handlers: <3>[ 89.107536] [<ffffffffa0056ae0>] usb_hcd_irq [usbcore] <3>[ 89.109648] [<ffffffff813b23f0>] vfio_intx_handler <0>[ 89.111623] Disabling IRQ #16

P.S.: sorry for my English.

Last edited by nivs (2013-12-05 23:40:43)

Offline

#865 2013-12-06 05:43:23

keatsfonam
Member
Registered: 2013-03-28
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT:
Solved the error 43 with AW's current qemu-vfio.  Im having the reboot issue still, working on that.  Thanks for all the great info in this thread.

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

------


Similiar situation as previous poster, can't get past error 43:

all my hardware supports VT-d
I am trying to run host on my intel integrated (ivy bridge) but I see that the i915 driver has caused problems.  I tried running the guest before X even and still couldn't get past the problem.

For the record I am trying to pass through an nvidia gtx 560

dmesg | grep -e IOMMU -e DMAR

[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] ACPI: DMAR 00000000be0f4c60 000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.077767] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.077771] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.077851] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.254226] DMAR: No ATSR found [ 0.254249] IOMMU 0 0xfed90000: using Queued invalidation [ 0.254250] IOMMU 1 0xfed91000: using Queued invalidation [ 0.254251] IOMMU: Setting RMRR: [ 0.254260] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.255421] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdb64000 - 0xbdb8efff] [ 0.255438] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdb64000 - 0xbdb8efff] [ 0.255451] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdb64000 - 0xbdb8efff] [ 0.255459] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.255465] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

/proc/cmdline:

BOOT_IMAGE=../vmlinuz-linux-mainline root=UUID=0ff525e8-6238-4741-a5f3-dec5653ebd14 ro intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1201,10de:0e0c initrd=../initramfs-linux-mainline.img

and tried:

BOOT_IMAGE=../vmlinuz-linux-mainline root=UUID=0ff525e8-6238-4741-a5f3-dec5653ebd14 ro intel_iommu=on initrd=../initramfs-linux-mainline.img

uname -r

3.12.0-1-mainline

tried with the three packages on the first post with the same kconfig
tried both qemu-git and linux-mainline from aur, same problem.

I can boot into windows 7 or windows 8 guest but I get the error 43.  Additionally I cannot reboot or it will lock up the host system.  I tried the patch that aw said fixes the error 43, but it still doesn't seem to work.

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.0 -boot once=d,menu=on -drive file=/home/me/win7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -drive file=/home/me/images/win7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on -vga none -usbdevice host:1532:010b -usbdevice host:046d:c00c

I have tried -cpu IvyBridge

dmesg:

 8.119274] [drm] Initialized drm 1.1.0 20060810 [ 8.127061] [drm] Memory usable by graphics device = 2048M [ 8.127065] i915 0000:00:02.0: setting latency timer to 64 [ 8.146428] i915 0000:00:02.0: irq 62 for MSI/MSI-X [ 8.146440] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010). [ 8.146442] [drm] Driver supports precise vblank timestamp query. [ 8.167917] [drm] Wrong MCH_SSKPD value: 0x20100406 [ 8.167920] [drm] This can cause pipe underruns and display issues. [ 8.167921] [drm] Please upgrade your BIOS to fix this. [ 8.266493] [drm] GMBUS [i915 gmbus dpc] timed out, falling back to bit banging on pin 4 [ 8.277615] fbcon: inteldrmfb (fb0) is primary device [ 8.277752] Console: switching to colour frame buffer device 210x65 [ 8.277762] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device [ 8.277764] i915 0000:00:02.0: registered panic notifier [ 8.277768] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io [ 8.282909] acpi device:43: registered as cooling_device4 [ 8.283082] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 8.283130] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/LNXVIDEO:00/input/input8 [ 8.283204] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0 [ 8.442675] vfio-pci 0000:01:00.0: Invalid ROM contents [ 9.775993] [drm] Enabling RC6 states: RC6 on, RC6p on, RC6pp off [ 19.912534] xenbr0: port 1(enp4s0) entered forwarding state

I think the invalid ROM contents is the key error here, but I might be wrong.
I'm not sure if MCH_SSKPD is relevant.
I dumped my rom and tried to load it and it gave me all sorts of errors about invalid commands and locked up my host system completely forcing a reboot.

dma: command 78 not supported dma: command 78 not supported

And a slew of similiar command not supported messages until I kill the process

dma: command 8c not supported dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: command 5c not supported dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: command 5c not supported dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: command 58 not supported dma: command 58 not supported dma: command 84 not supported dma: command 60 not supported dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: command 4c not supported dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: command 4c not supported dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769 dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769

When I start the VM:

[ 1513.598793] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 1516.405951] usb 2-1.5: reset full-speed USB device number 3 using ehci-pci [ 1516.745059] usb 3-2: reset low-speed USB device number 2 using xhci_hcd [ 1516.758317] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880417f98a00 [ 1516.758331] usb 3-2: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes

The usb and xhci_hcd are spammed 5 or 6 more times, but it seems like the vfio-pci has no error message. 
However I saw some other user's dmesgs that said

assign device 0:X:0.0

Which is missing from mine.  Does that mean the device isn't actually getting assigned?

I'm not sure if the problem stems for qemu or from linux-mainline, so I keep trying different combinations with no luck.
Any help is appreciated, I suspect I may be better off with a radeon card.

Last edited by keatsfonam (2013-12-07 09:29:02)

Offline

#866 2013-12-07 18:41:38

jackun
Member
Registered: 2012-11-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm dual booting and sometimes messing with passthrough and having a dual input monitor, it was getting tiresome to switch monitor's input. So one way to use a gfxcard for passthrough without setting another card to be primary is to force off any FB consoles with kernel argument 'vga=0' and then manually (create a systemd service or something) use fbset and con2fb. Default low res text-only bios console should be still working.

You can get con2fb source from http://www.dafyddcrosby.com/src/con2fb.c
So:

cat /proc/fb

to see that only one fb is setup (probably radeondrmfb or something).

Then with working kms driver and 1920x1080 monitor:

fbset /dev/fb0 -xres 1920 -vxres 1920 -yres 1080 -vyres 1080 #initializes framebuffer and wakes up monitor (??) con2fb /dev/fb0 /dev/tty1 #map ttys to framebuffer. Even though only tty1 is passed, it seems all ttys are move to fb0

Now you can see bios etc. stuff still on your (probably) more powerful card and use it in windows and still use it with passthrough too.
Ofcourse pci-stub etc. the gfxcard you intend to use for passthrough.

Now the question, can you unbind kernel's VGA console without 'vga=0' ? Then 'fbcon=map:1' could be enough probably.

Last edited by jackun (2013-12-07 23:24:41)

Offline

#867 2013-12-07 21:11:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

keatsfonam wrote:

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#868 2013-12-08 02:58:51

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone know of a way to monitor CPU temperature from within the guest and output it to an OSD program like RTSS? Proper cpu statistics would also be really nice.

Offline

#869 2013-12-08 11:33:09

Damon
Member
From: Germany
Registered: 2009-04-09
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to pass a SATA controller of my Supermicro X10SAE to the guest.

Kernel: 3.12.3 with acs-override patch.

/usr/bin/qemu-system-x86_64 --version QEMU emulator version 1.7.50, Copyright (c) 2003-2008 Fabrice Bellard
$ dmesg | grep -e DMAR -e IOMMU [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] ACPI: DMAR 00000000d9508dd0 000B8 (v01 INTEL HSW 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.025640] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.025644] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.025710] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.430025] DMAR: No ATSR found [ 0.430050] IOMMU 0 0xfed90000: using Queued invalidation [ 0.430051] IOMMU 1 0xfed91000: using Queued invalidation [ 0.430053] IOMMU: Setting RMRR: [ 0.430062] IOMMU: Setting identity map for device 0000:00:02.0 [0xdc000000 - 0xde1fffff] [ 0.430254] IOMMU: Setting identity map for device 0000:00:1d.0 [0xd9ea8000 - 0xd9eb6fff] [ 0.430276] IOMMU: Setting identity map for device 0000:00:1a.0 [0xd9ea8000 - 0xd9eb6fff] [ 0.430295] IOMMU: Setting identity map for device 0000:00:14.0 [0xd9ea8000 - 0xd9eb6fff] [ 0.430309] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.430315] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
$ cat /proc/cmdline BOOT_IMAGE=/@/boot/vmlinuz-3.12.3+ root=UUID=4a5ab603-c89c-42fb-93bb-ef541d20f32e ro rootflags=subvol=@ crashkernel=384M-2G:64M,2G-:128M splash quiet intel_iommu=on pcie_acs_override=downstream vt.handoff=7
$ ./lsgroup ### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 2 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller (rev 04) ### Group 5 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 9 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 10 ### 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) ### Group 11 ### 00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d5) ### Group 12 ### 00:1c.7 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #8 (rev d5) ### Group 13 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 14 ### 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) ### Group 15 ### 02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) ### Group 16 ### 03:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) 04:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] ### Group 17 ### 05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) ### Group 18 ### 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

The device I want to pass is in the IOMMU-Group 14

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)

Therefore I bind this group to vfio-pci

./vfio-group 14 0000:00:1f.0 0000:00:1f.2 0x8086 0x8c56 0x8086 0x8c02 0x8086 0x8c22 0x8086 0x8c24

Finally I want to test if it works:

$ /usr/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=00:1f.2,bus=root.1,addr=00.0 qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.2, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.2, no available reset mechanism.

I really don't know what I'm doing wrong or what I have missed to do. Initially I tried to pass the controller using pci-assign but that wasn't possible due to the lack of FLReset.

I would really appreciate any help.

Offline

#870 2013-12-08 14:34:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

linux-mainline.tar.gz (3.13-rc3, acs override patch) Doesnt include i935 vga arbiter fixes
qemu-git.tar.gz Spice disabled due to build errors
seabios-git.tar.gz

Aur:

https://aur.archlinux.org/packages/qemu-git/
https://aur.archlinux.org/packages/linux-mainline/ (still -rc2 as of today)

Last edited by nbhs (2013-12-08 14:41:30)

Offline

#871 2013-12-08 20:34:24

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The the i915 patch does not apply cleanly to linux-3.13-rc3. This is the request when I attempted to patch linux-3.13-rc3.

patch -p1 -i ../vfio-previous/linux-mainline/i915_fixes.patch patching file drivers/gpu/drm/i915/i915_dma.c patching file drivers/gpu/drm/i915/intel_display.c Hunk #1 succeeded at 10585 (offset 544 lines). Hunk #2 succeeded at 10601 with fuzz 1 (offset 544 lines). Hunk #3 succeeded at 11105 (offset 543 lines). patching file include/linux/vgaarb.h patching file drivers/gpu/drm/i915/i915_dma.c Hunk #1 succeeded at 1355 (offset 4 lines). patching file drivers/gpu/drm/i915/intel_display.c Hunk #1 succeeded at 10585 (offset 545 lines). Hunk #2 succeeded at 10592 (offset 545 lines). Hunk #3 succeeded at 10606 with fuzz 1 (offset 545 lines). Hunk #4 succeeded at 10904 with fuzz 1 (offset 552 lines). Hunk #5 succeeded at 11110 (offset 544 lines). patching file drivers/gpu/drm/i915/intel_drv.h Hunk #1 FAILED at 792. 1 out of 1 hunk FAILED -- saving rejects to file drivers/gpu/drm/i915/intel_drv.h.rej

I am  able to pass a usb 3.0 controller to the VM but totally unable to install a driver under windows 7. Complete BSOD when installing the driver. Anyone know of a working usb 3.0 or usb 2.0 controller? I even tried passing the intel motherboard usb devices. Still could not install a driver. If there is additional information please let me know. I have no problem passing a usb device but want to pass a controller.

I think it is the controller. So much for that Black Friday special ;-). At least it was cheap. I think I will try a via usb 3.0 controller. I am looking for pci express card. If anyone has one that works please list the model.

The card I was trying to use is a Vantec Model UGT-PC312

04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])
        Subsystem: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at f7900000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [90] MSI-X: Enable- Count=8 Masked-
                Vector table: BAR=0 offset=00001000
                PBA: BAR=0 offset=00001080
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [150 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Kernel driver in use: vfio-pci

Thanks.

Last edited by alphahere (2013-12-09 03:18:26)

Offline

#872 2013-12-08 23:57:23

binaryplease
Member
Registered: 2012-04-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I cant get pass the step from unbinding step. Everytime i execute the script or do

echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind

my system freezes and i have to switch it of holding the power button.

I have a lenovo y500 with two geforce gt650m cards (SLI).
Im using the nouveau driver. Can anyone help?

Offline

#873 2013-12-09 05:24:16

dalingrin
Member
Registered: 2009-03-18
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried passing through 2 cards in SLI?

Offline

#874 2013-12-09 18:37:39

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alphahere wrote:

I am  able to pass a usb 3.0 controller to the VM but totally unable to install a driver under windows 7. Complete BSOD when installing the driver. Anyone know of a working usb 3.0 or usb 2.0 controller? I even tried passing the intel motherboard usb devices. Still could not install a driver. If there is additional information please let me know. I have no problem passing a usb device but want to pass a controller.

I think it is the controller. So much for that Black Friday special ;-). At least it was cheap. I think I will try a via usb 3.0 controller. I am looking for pci express card. If anyone has one that works please list the model.

The card I was trying to use is a Vantec Model UGT-PC312

04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])

I bought one (because my onboard Etron controllers cause BSOD in Win7 when trying to install the driver).
The one I got works absolutely perfect in both Win 7 and 8.1, it's a

05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])

/me hides, giggeling.

Last edited by teekay (2013-12-09 18:38:04)

Offline

#875 2013-12-09 19:58:16

peaquino
Member
Registered: 2013-12-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I think I read all the posts in this long thread, but unfortunatelly I'm still not able to passthrough my Radeon 7870XT to a Windows 7 VM. My other components are:
- AMD APU A10-5800K (with Radeon 7660D acting as primary VGA)
- Asrock A85 FM2A85X Extreme6
- 16G RAM

lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) I/O Memory Management Unit
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Trinity [Radeon HD 7660D]
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port
00:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03)
00:10.1 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 14)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD] FCH IDE Controller
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 01)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 0)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 3)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 5
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti LE [Radeon HD 7870 XT]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
02:00.0 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.1 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.2 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.3 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.4 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.5 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.6 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.7 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)

I'm using linux-mainline, qemu and seabios from the first page of the thread.

According to dmesg, the board properly activates IOMMU, and the 7870 is properly bound to pci-stub:

[    0.000000] Command line: BOOT_IMAGE=/arch/vmlinuz-linux-mainline root=UUID=c25d5a87-40b9-4a3d-89db-3e263ff8b839 rw quiet pci-stub.ids=1002:679e,1002:aaa0 radeon.dpm=1 radeon.fastfb=1 iommu=pt

dmesg | grep AMD-Vi

[    1.139471] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.139472] AMD-Vi:  Extended features:  PreF PPR GT IA
[    1.139475] AMD-Vi: Interrupt remapping enabled
[    1.139647] AMD-Vi: Initialized for Passthrough Mode

dmesg | grep pci-stub

[    0.000000] Command line: BOOT_IMAGE=/arch/vmlinuz-linux-mainline root=UUID=c25d5a87-40b9-4a3d-89db-3e263ff8b839 rw quiet pci-stub.ids=1002:679e,1002:aaa0 radeon.dpm=1 radeon.fastfb=1 iommu=pt
[    0.000000] Kernel command line: BOOT_IMAGE=/arch/vmlinuz-linux-mainline root=UUID=c25d5a87-40b9-4a3d-89db-3e263ff8b839 rw quiet pci-stub.ids=1002:679e,1002:aaa0 radeon.dpm=1 radeon.fastfb=1 iommu=pt
[    1.175368] pci-stub: add 1002:679E sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.175380] pci-stub 0000:01:00.0: claimed by stub
[    1.175389] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.175395] pci-stub 0000:01:00.1: claimed by stub

I'm trying to run the VM with the following script:

#!/bin/bash QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -cpu qemu64 -M q35 -m 8192 \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -monitor stdio \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/dev/bcache/by-uuid/d79f9dd0-c884-42f7-b5a9-79db832ea9d0,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/myuser/Tahiti.rom \ -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:04fc:05d8 \ -boot menu=on

I can see the BIOS output and Windows starting animation on the monitor connected to the Radeon, but as soon as Catalyst tries to initialize the card, I get a BSOD with atikmpag.sys 116 error.
Additionally, after Seabios initializes the 7870, the color palette in Gnome running on the 7660D is totally messed up (I can to post a photo of the screen, if that could be any help). Qemu monitor is not showing any errors, but I get
the follwing entries in journal:

[ 2204.807168] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 2204.831486] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[ 2204.831500] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[ 2204.873898] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[ 2246.082840] kvm: zapping shadow pages for mmio generation wraparound
[ 2248.441789] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc000 flags=0x0010]
[ 2248.441798] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441801] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0030]
[ 2248.441803] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441805] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]
[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441809] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]

I tried every workaround I could find, nothing seems to work.
About 7 months ago I was able to passthrough on the same hardware using Xen 4.1 (but I abandoned this setup beacause of problems with card resets and host hangs on Windows reboots, I wasn't aware of the dev eject hack then).

I hope I'm just being stupid and overlooked something obvious. If not, what motherboard and processor would you recommend me to replace my APU setup?

Thanks for all pointers!

Offline

#876 2013-12-09 20:03:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also just tried assigning USB3 controllers and they just work for me.  I tried both an onboard Etron EJ168 and a plugin Renesas uPD720200.  Windows 7 guest, drivers installed, seems to work with an old USB thumb drive.  I picked up the Renesas card at Best Buy - http://www.rocketfishproducts.com/produ … 2USB3.html http://www.bestbuy.com/site/2-port-usb- … /1092716.p YMMV


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#877 2013-12-09 20:10:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

peaquino wrote:
#!/bin/bash QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -cpu qemu64 -M q35 -m 8192 \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -monitor stdio \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/dev/bcache/by-uuid/d79f9dd0-c884-42f7-b5a9-79db832ea9d0,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/myuser/Tahiti.rom \ -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:04fc:05d8 \ -boot menu=on

rombar=0, romfile=<something> are conflicting.  The first tells it not to expose a ROM, the second tells it what to expose for the ROM.  I suppose maybe rombar=0 is getting ignored if you're getting output.

I can see the BIOS output and Windows starting animation on the monitor connected to the Radeon, but as soon as Catalyst tries to initialize the card, I get a BSOD with atikmpag.sys 116 error.
Additionally, after Seabios initializes the 7870, the color palette in Gnome running on the 7660D is totally messed up (I can to post a photo of the screen, if that could be any help). Qemu monitor is not showing any errors, but I get

I've seen color palette corruption on AMD as well, even without AMD primary display IIRC.

the follwing entries in journal:

[ 2204.807168] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 2204.831486] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[ 2204.831500] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[ 2204.873898] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[ 2246.082840] kvm: zapping shadow pages for mmio generation wraparound
[ 2248.441789] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc000 flags=0x0010]
[ 2248.441798] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441801] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0030]
[ 2248.441803] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441805] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]
[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441809] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]

The errors are from your audio card.  What about not assigning it at all?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#878 2013-12-09 21:00:57

peaquino
Member
Registered: 2013-12-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The following script doesn't change much: the pagefaults are occuring on 01:00.0 device (main GPU) now (I still get a BSOD on Windows startup).

#!/bin/bash QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -cpu qemu64 -M q35 -m 8192 \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -monitor stdio \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/dev/bcache/by-uuid/d79f9dd0-c884-42f7-b5a9-79db832ea9d0,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:04fc:05d8 \ -boot menu=on
aw wrote:

rombar=0, romfile=<something> are conflicting.  The first tells it not to expose a ROM, the second tells it what to expose for the ROM.  I suppose maybe rombar=0 is getting ignored if you're getting output.

Thanks, I corrected that in the script.

Offline

#879 2013-12-09 21:15:21

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

i'm sorry for my poor english wirting skills…

Offline

#880 2013-12-09 21:23:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

Numerous folks have reported BSODs which were solved by moving the audio device to the root complex rather than exposing it as a sub-function.  It may work, but exposing it on pcie.0 has had more success it seems.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#881 2013-12-10 00:14:29

peaquino
Member
Registered: 2013-12-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
andy123 wrote:

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

Numerous folks have reported BSODs which were solved by moving the audio device to the root complex rather than exposing it as a sub-function.  It may work, but exposing it on pcie.0 has had more success it seems.

That's what I read in th this thread -- I tried both options (exposing the soundcard as subdevice and as separate unit) -- none worked. Not passing the soundcard is not making any visible differences: instad of

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 ...

I get

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 ...

Sometimes the host reboots even before it has a chance to write theses errors to journal.

Moments ago I tried with pci-assign method and Windows Blue Preview: it doesn't BSOD, but the passed 7870 is not turning on and has a yellow exlamantion mark in device manager (Code 10 - if I remember correctly).

Still, thanks a lot! I *really* appreciate your help.

Offline

#882 2013-12-10 03:18:46

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

binaryplease wrote:

I cant get pass the step from unbinding step. Everytime i execute the script or do

echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind

my system freezes and i have to switch it of holding the power button.

I have a lenovo y500 with two geforce gt650m cards (SLI).
Im using the nouveau driver. Can anyone help?

You'll probably have to use pci-stub or blacklist the nouveau driver

Offline

#883 2013-12-10 03:57:15

alphahere
Member
Registered: 2013-10-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I also just tried assigning USB3 controllers and they just work for me.  I tried both an onboard Etron EJ168 and a plugin Renesas uPD720200.  Windows 7 guest, drivers installed, seems to work with an old USB thumb drive.  I picked up the Renesas card at Best Buy - http://www.rocketfishproducts.com/produ … 2USB3.html http://www.bestbuy.com/site/2-port-usb- … /1092716.p YMMV

Thanks also to teekay for reporting that it does work with nec (renasas) usb 3.0 cards. The card does work fine under linux and windows natively.

Anyway I have wasted most of yesterday trying to figure this out and it must be my setup. Will try to compile everything again and see if I can figure it out. I can get a little crazy trying to get stuff to work.

Even the intel controller usb 3.0 on the motherboard does not work with qemu. It must be my setup because the default xhci driver does not even work anymore. This is the default built in usb 3.0 driver in qemu and it does not work!

-device nec-usb-xhci,id=xhci

This comes up as unknown device under windows 7.  I know I had this built in xhci driver working once.

Any hints at what to look for would be greatly apperciated.

Fun times!

Thanks

Offline

#884 2013-12-10 05:42:57

keatsfonam
Member
Registered: 2013-03-28
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
keatsfonam wrote:

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.

Not executing vfio-bind and just starting the vm while the pci-stub is loaded is what I meant.

Offline

#885 2013-12-10 13:48:02

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

keatsfonam wrote:

Not executing vfio-bind and just starting the vm while the pci-stub is loaded is what I meant.

Now that sounds really strange.

Offline

#886 2013-12-11 03:49:08

ddjdx
Member
Registered: 2013-07-16
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi All. Encountering an issue that I cant seem to get a hold of a solution for.

I have installed the 3 packages in the OP, Kernel with default configuration.

I'm able to use pci-stub on my secondary AMD Radeon 5450 adapter and the VIFO-BIND Script appears to work.

When I start up the test Qemu, the screen that is plugged into the passthrough adapter comes out of standby but is blank. And a journalctl -r reveals the following :

Dec 11 14:28:05 tux kernel: ------------[ cut here ]------------ Dec 11 14:28:05 tux kernel: ---[ end trace 59e460fd7afe9fe2 ]--- Dec 11 14:28:05 tux kernel: [<ffffffff815053ad>] system_call_fastpath+0x1a/0x1f Dec 11 14:28:05 tux kernel: [<ffffffff811b60c1>] SyS_ioctl+0x81/0xa0 Dec 11 14:28:05 tux kernel: [<ffffffff811b5e60>] do_vfs_ioctl+0x2e0/0x4c0 Dec 11 14:28:05 tux kernel: [<ffffffffa1042ea7>] vfio_fops_unl_ioctl+0x77/0x340 [vfio] Dec 11 14:28:05 tux kernel: [<ffffffffa1055439>] vfio_iommu_type1_ioctl+0x219/0x290 [vfio_iommu_type1] Dec 11 14:28:05 tux kernel: [<ffffffffa1054bc9>] vfio_dma_do_map+0x209/0x860 [vfio_iommu_type1] Dec 11 14:28:05 tux kernel: [<ffffffff813dab66>] iommu_map+0xe6/0x220 Dec 11 14:28:05 tux kernel: [<ffffffff813e5f95>] intel_iommu_map+0x75/0xa0 Dec 11 14:28:05 tux kernel: [<ffffffff813e5ea7>] __domain_mapping+0x3a7/0x420 Dec 11 14:28:05 tux kernel: [<ffffffff81061c8a>] warn_slowpath_null+0x1a/0x20 Dec 11 14:28:05 tux kernel: [<ffffffff81061bad>] warn_slowpath_common+0x7d/0xa0 Dec 11 14:28:05 tux kernel: [<ffffffff814f6f10>] dump_stack+0x4d/0x6f Dec 11 14:28:05 tux kernel: Call Trace: Dec 11 14:28:05 tux kernel: ffff8800ac626000 0000000000000201 0000000000000200 ffff880bea403c90 Dec 11 14:28:05 tux kernel: ffff880bea403c80 ffffffff81061bad 0000000000000200 0000000be2200083 Dec 11 14:28:05 tux kernel: 0000000000000009 ffff880bea403c48 ffffffff814f6f10 0000000000000000 Dec 11 14:28:05 tux kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./X79-UP4, BIOS F1 07/13/2012 Dec 11 14:28:05 tux kernel: CPU: 5 PID: 1246 Comm: qemu-system-x86 Not tainted 3.13.0-1-mainline #1 Dec 11 14:28:05 tux kernel: snd_usb_audio evdev snd_usbmidi_lib snd_rawmidi dvb_core snd_seq_device rc_core sb_edac edac_core i2c_i801 radeon lpc_ich ttm drm_kms_helper drm i2c_algo_bi Dec 11 14:28:05 tux kernel: Modules linked in: vfio_pci vfio_iommu_type1 vfio tun ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrac Dec 11 14:28:05 tux kernel: WARNING: CPU: 5 PID: 1246 at drivers/iommu/intel-iommu.c:1860 __domain_mapping+0x3a7/0x420() Dec 11 14:28:05 tux kernel: ------------[ cut here ]------------ Dec 11 14:28:05 tux kernel: ERROR: DMA PTE for vPFN 0x0 already set (to bec52f003 not be2200083) Dec 11 14:28:03 tux kernel: vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)

The stack trace is repeated over 100 times and eventually stops with Qemu still running and the secondary screen blank.

I'm not sure what to look at next as the stack trace doesn't seem to tell me anything. Though this line caught my eye:

Dec 11 14:28:05 tux kernel: ERROR: DMA PTE for vPFN 0x0 already set (to bec52f003 not be2200083)

But not sure what it means?

EDIT-
Tinkered around some more. Blacklisting the Radeon driver allows SeaBios to appear on the passthrough card. I'm assuming its because the Radeon driver is connecting to the card first before PCI-stub loads. I'm sure that can be fixed by compiling pci-stub into the kernel.But I'm,still getting 100's of the stack traces listed above. So I'm hesitant to go furthur and install an OS.

I've tried the following:
Disabling off-chipset Marvell SATA controller (Its known to cause problems with IOMMU), No difference.
Disabled Interrupt Mapping in the BIOS and enabled the workaround in the OP. No Difference
Disbaled all other Virtulisation settings in the bios except for VT-D and VT and enabled the Interrupt Mapping work around. No Difference.

Last edited by ddjdx (2013-12-11 08:54:02)

Offline

#887 2013-12-12 06:17:28

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?
This is a headless server and no display drivers are loaded, nouveau is blacklisted

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get device 0000:04:00.0
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

lspci | grep NVIDIA
04:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070-Q] (rev a3)
04:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)

Offline

#888 2013-12-12 06:29:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owen77 wrote:

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?

grep VFIO from your kernel config and it might be pretty obvious.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#889 2013-12-12 07:36:55

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Owen77 wrote:

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?

grep VFIO from your kernel config and it might be pretty obvious.

cat /boot/config-3.12.0-031200-generic | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

lsmod | grep vfio
vfio_pci               37175  0
vfio_iommu_type1       17885  0
vfio                   21113  2 vfio_pci,vfio_iommu_type1

Looks ok to me, anything I'm missing?

A colleague has just suggested x-vga may not be necessary for a headless config, would I be able to still access the NVidia card and install drivers to use CUDA for example with standard pci pass-through?

Offline

#890 2013-12-12 08:12:40

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owen77 wrote:

cat /boot/config-3.12.0-031200-generic | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

Why not compile VFIO support into the kernel instead of building as module?

Offline

#891 2013-12-12 08:55:57

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Why not compile VFIO support into the kernel instead of building as module?

That's how it came.
It's an Ubuntu machine (trying to get this working without replacing the OS if possible) and I took the lazy option of installing a pre-built 3.12 kernel package.

Offline

#892 2013-12-12 09:10:18

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owen77 wrote:
ilya80 wrote:

Why not compile VFIO support into the kernel instead of building as module?

That's how it came.
It's an Ubuntu machine (trying to get this working without replacing the OS if possible) and I took the lazy option of installing a pre-built 3.12 kernel package.

I believe you need both upstream kernel and upstream qemu, prebuilt package wont do it for you.

Offline

#893 2013-12-12 09:24:39

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

I believe you need both upstream kernel and upstream qemu, prebuilt package wont do it for you.

Ok, well I'm using  qemu 1.7 built from source (originally using 1.5 from a package).
I'll build the kernel if necessary but according to this post https://bbs.archlinux.org/viewtopic.php … 9#p1304619 packages should work ok.

Offline

#894 2013-12-12 14:14:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owen77 wrote:
aw wrote:
Owen77 wrote:

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?

grep VFIO from your kernel config and it might be pretty obvious.

cat /boot/config-3.12.0-031200-generic | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

lsmod | grep vfio
vfio_pci               37175  0
vfio_iommu_type1       17885  0
vfio                   21113  2 vfio_pci,vfio_iommu_type1

Looks ok to me, anything I'm missing?

A colleague has just suggested x-vga may not be necessary for a headless config, would I be able to still access the NVidia card and install drivers to use CUDA for example with standard pci pass-through?

CONFIG_VFIO_PCI_VGA is what I was looking for since I think you would have gotten the same error message if it was disabled (there's no need to build VFIO statically into the kernel, that's not the problem)  The problem though is that your Telsa card doesn't support VGA, note that the PCI class code is "3D controller" rather than "VGA compatible controller".  VFIO only enables VGA access to devices that claim to support VGA per the PCI spec.  Your Tesla is handled as if you had asked to enable VGA access to an Ethernet card.

Nvidia does support Tesla, Quadro, and GRID in virtualized environments, but only the K-series of each.  The supported configuration is to assign the GPU as a secondary device to the emulated VGA.  There's no need to use the x-vga option in the case.  For Quadro and GRID cards the Nvidia device becomes the primary display device when the Nvidia driver stack loads.  I don't know if this would work with your card, but VGA doesn't seem to be an option.  Also note that 3.12/1.7 is not sufficient, there are changes in both kernel and qemu that went in after those releases.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#895 2013-12-12 23:55:27

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

CONFIG_VFIO_PCI_VGA is what I was looking for since I think you would have gotten the same error message if it was disabled (there's no need to build VFIO statically into the kernel, that's not the problem)  The problem though is that your Telsa card doesn't support VGA, note that the PCI class code is "3D controller" rather than "VGA compatible controller".  VFIO only enables VGA access to devices that claim to support VGA per the PCI spec.  Your Tesla is handled as if you had asked to enable VGA access to an Ethernet card.

Nvidia does support Tesla, Quadro, and GRID in virtualized environments, but only the K-series of each.  The supported configuration is to assign the GPU as a secondary device to the emulated VGA.  There's no need to use the x-vga option in the case.  For Quadro and GRID cards the Nvidia device becomes the primary display device when the Nvidia driver stack loads.  I don't know if this would work with your card, but VGA doesn't seem to be an option.  Also note that 3.12/1.7 is not sufficient, there are changes in both kernel and qemu that went in after those releases.

Aha, well that explains it. The M2070-Q adds the Quadro features to the M2070 and they advertise it as able to do remote visualisation, so maybe the second technique could work with this card.

I won't clutter up this thread with any more about this as it's clearly a different setup and it looks like we might need newer hardware but if you have any references about the latter option to speed up my research you could share I'd greatly appreciate it, thanks for the help so far.
(For disclosure this is a publicly funded research project and all our work will be written up and posted online for community use as we get things running.)

Last edited by Owen77 (2013-12-12 23:57:55)

Offline

#896 2013-12-13 01:32:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owen77 wrote:
aw wrote:

CONFIG_VFIO_PCI_VGA is what I was looking for since I think you would have gotten the same error message if it was disabled (there's no need to build VFIO statically into the kernel, that's not the problem)  The problem though is that your Telsa card doesn't support VGA, note that the PCI class code is "3D controller" rather than "VGA compatible controller".  VFIO only enables VGA access to devices that claim to support VGA per the PCI spec.  Your Tesla is handled as if you had asked to enable VGA access to an Ethernet card.

Nvidia does support Tesla, Quadro, and GRID in virtualized environments, but only the K-series of each.  The supported configuration is to assign the GPU as a secondary device to the emulated VGA.  There's no need to use the x-vga option in the case.  For Quadro and GRID cards the Nvidia device becomes the primary display device when the Nvidia driver stack loads.  I don't know if this would work with your card, but VGA doesn't seem to be an option.  Also note that 3.12/1.7 is not sufficient, there are changes in both kernel and qemu that went in after those releases.

Aha, well that explains it. The M2070-Q adds the Quadro features to the M2070 and they advertise it as able to do remote visualisation, so maybe the second technique could work with this card.

I won't clutter up this thread with any more about this as it's clearly a different setup and it looks like we might need newer hardware but if you have any references about the latter option to speed up my research you could share I'd greatly appreciate it, thanks for the help so far.
(For disclosure this is a publicly funded research project and all our work will be written up and posted online for community use as we get things running.)

There's certainly a chance that assigning the GPU as a secondary device with emulated VGA could work, try it and see.  Do it just like you would if it's not a graphics device.  If it behaves like Quadro/GRID the display will switch once the Nvidia driver stack loads, so be sure you can get to the desktop over the network via VNC or some other remote access solution first.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#897 2013-12-17 00:17:29

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So what's the status of KVM and VGA passthrough - Are there still specific patches necessary or is this now working out-of-the-box? I just installed Win7 in a VM, passing through my HD7880 and it seemed to work fine. After initial windows install and reboot I installed latest Catalyst, rebooted and it just works smile I did not do any performance tests so far, just wondering if in the meantime all relevant changes are already in the standard packages or anything still needs to be done?

Offline

#898 2013-12-17 05:22:06

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA passthrough is working well for me with a Radeon R9 270X (i.e. a rebranded 7870 GHz Edition), however after installing the latest Catalyst driver version 13.11 beta 9.5 on Windows 7 SP1 I am having trouble booting Windows.

I see 'Starting Windows...' with the windows splash screen animation but then the screen goes black and after a bit the monitor loses signal. In Safe mode, the event log shows error 0x00000116 - MSDN says this: "The VIDEO_TDR_ ERROR bug check has a value of 0x00000116. This indicates that an attempt to reset the display driver and recover from a timeout failed."

I have tried so moving the HDMI audio device (ID 0000:01:00.1) to pcie.0 instead of root.1 as well as eliminating passthrough of the HDMI audio device entirely, neither of which helped. I also tried manually specifying the ROM (rombar=0 romfile=...) from a ROM bin I dumped using GPU-Z. I dumped it from within the VM, which I read may not work but it didn't throw any errors and GPU-Z properly detected the card so I think it dumped the VBIOS correctly. Has anyone encountered something similar, or has any ideas on what I could do?

Full disclosure: I am on Fedora 20 using the virt-preview repository so I have kernel 3.12.5 and qemu-kvm 1.7.0. I'm using the same config as the main howto - AHCI, virtualized audio. Here's the command I use:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name win7 -S -M q35 -m 2048 \ -realtime mlock=off -rtc base=localtime \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/VMs/W7.img,if=none,id=disk,format=qcow2 -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/dev/null,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \ -boot menu=on

VFIO passthrough was normally prior to installing the Catalyst drivers, so my guess is that pass-through is working correctly and the bug is in the machine config or the Windows driver.

Last edited by firewing1 (2013-12-17 05:28:52)

Offline

#899 2013-12-17 06:18:34

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

There's certainly a chance that assigning the GPU as a secondary device with emulated VGA could work, try it and see.  Do it just like you would if it's not a graphics device.  If it behaves like Quadro/GRID the display will switch once the Nvidia driver stack loads, so be sure you can get to the desktop over the network via VNC or some other remote access solution first.

Hmmm, one last question about my slightly tangential issues in case anyone has any suggestions:

  • The Tesla card is being passed through ok, I can login to a guest with the vga drivers and run CUDA or OpenCL programs.

  • If the NVIDIA drivers are installed (tried 319.76 and 331.20) with appropriate xorg.conf created, the VNC screen just freezes after it loads the X server, if I then ssh in to the VM there are no errors in Xorg.0.log but X apps can't find a display, no matter what I set $DISPLAY to.

  • If I start in runlevel 3 and login then run startx, the VNC screen then goes black but I can ssh in and the X server is now running! I can set $DISPLAY to :0.0 and run glxinfo and glxgears at ludicrous speeds from the terminal but have no way of seeing anything.

Offline

#900 2013-12-17 14:05:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owen77 wrote:
aw wrote:

There's certainly a chance that assigning the GPU as a secondary device with emulated VGA could work, try it and see.  Do it just like you would if it's not a graphics device.  If it behaves like Quadro/GRID the display will switch once the Nvidia driver stack loads, so be sure you can get to the desktop over the network via VNC or some other remote access solution first.

Hmmm, one last question about my slightly tangential issues in case anyone has any suggestions:

  • The Tesla card is being passed through ok, I can login to a guest with the vga drivers and run CUDA or OpenCL programs.

  • If the NVIDIA drivers are installed (tried 319.76 and 331.20) with appropriate xorg.conf created, the VNC screen just freezes after it loads the X server, if I then ssh in to the VM there are no errors in Xorg.0.log but X apps can't find a display, no matter what I set $DISPLAY to.

  • If I start in runlevel 3 and login then run startx, the VNC screen then goes black but I can ssh in and the X server is now running! I can set $DISPLAY to :0.0 and run glxinfo and glxgears at ludicrous speeds from the terminal but have no way of seeing anything.

Congratulation, it works.  When the Nvidia driver is used, the emulated VGA is only used for text mode and the GPU is used for graphical mode.  They're mutually exclusive, so it's expected that the emulated VGA "freezes" when X starts.  In your first case, apps can't find any display device, probably because the graphical desktop is sitting at a login prompt.  In the second case, you've started an X session, therefore there is a display to connect to.  This is the way that GRID cards work and Quadro as well if you ignore that it has a physical display connector.  You need to setup remote access to the desktop first.  You can run a vnc server in the guest or use some other remote'ing solution.  If you were expecting X on the emulated VGA with rendering in the GPU, that doesn't happen.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#901 2013-12-17 23:51:26

Owen77
Member
Registered: 2013-12-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Congratulation, it works.  When the Nvidia driver is used, the emulated VGA is only used for text mode and the GPU is used for graphical mode.  They're mutually exclusive, so it's expected that the emulated VGA "freezes" when X starts.  In your first case, apps can't find any display device, probably because the graphical desktop is sitting at a login prompt.  In the second case, you've started an X session, therefore there is a display to connect to.  This is the way that GRID cards work and Quadro as well if you ignore that it has a physical display connector.  You need to setup remote access to the desktop first.  You can run a vnc server in the guest or use some other remote'ing solution.  If you were expecting X on the emulated VGA with rendering in the GPU, that doesn't happen.

Great! Thanks, I don't know why I was expecting the qemu vnc server to switch over, but it makes sense that it's fixed to the emulated vga framebuffer. Looking into vnc installation in the guest now.

Offline

#902 2013-12-18 05:51:28

syllopsium
Member
Registered: 2013-12-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

VGA passthrough is working well for me with a Radeon R9 270X (i.e. a rebranded 7870 GHz Edition), however after installing the latest Catalyst driver version 13.11 beta 9.5 on Windows 7 SP1 I am having trouble booting Windows.

I see 'Starting Windows...' with the windows splash screen animation but then the screen goes black and after a bit the monitor loses signal. In Safe mode, the event log shows error 0x00000116 - MSDN says this: "The VIDEO_TDR_ ERROR bug check has a value of 0x00000116. This indicates that an attempt to reset the display driver and recover from a timeout failed."

I have tried so moving the HDMI audio device (ID 0000:01:00.1) to pcie.0 instead of root.1 as well as eliminating passthrough of the HDMI audio device entirely, neither of which helped. I also tried manually specifying the ROM (rombar=0 romfile=...) from a ROM bin I dumped using GPU-Z. I dumped it from within the VM, which I read may not work but it didn't throw any errors and GPU-Z properly detected the card so I think it dumped the VBIOS correctly. Has anyone encountered something similar, or has any ideas on what I could do?
[..] snip
VFIO passthrough was normally prior to installing the Catalyst drivers, so my guess is that pass-through is working correctly and the bug is in the machine config or the Windows driver.

I have a very similar problem here, but suspect the issue is Windows and Qemu/KVM's virtualised hardware presented, not the passthrough capability.

Alex, is there any way to debug this, or are we at the mercy of AMD/Microsoft/firing up a Windows kernel debugger? I've fiddled around with this a lot and got nowhere. There's nothing in dmesg.

I'm able to successfully pass a 6950 through to a Ubuntu VM using both the radeon and fglrx drivers on an S3210SHLC board (basically a server version of the X38 chipset, using a Core2Quad). That's using the southbridge x4 PCIe 1.x slot (the only one that can run graphics at a sensible rate, due to the enforced chipset limitation of x1 for graphics cards in other slots).

For Windows it always provides a driver timeout under both Windows 7 and 8 x64, whether catalyst's control panel is installed or not (before Catalyst, it works fine unaccelerated). In some cases it seems enabling vmx in the virtual CPU causes problems, but this appears to be less of an issue of late. Using 3.12.0 rc4 and qemu 1.7.5.0

I do have to specify the GPU BIOS when booting the VM - it doesn't work otherwise.

I'm also much less importantly having issues with passing through the built in Matrox G200e on the board - that's a case where it works in Xen and VMWare, but not in KVM.

Startup, if you're bothered :

qemu-system-x86_64 -enable-kvm -M q35 -m 3184 -cpu qemu64,+ssse3,+cx16,+xtpr,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+est,+tm2,+pdcm,+lahf_lm, -smp 2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/home/peter/6950bios.rom,x-vga=on -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1   -device ahci,bus=pcie.0,id=ahci  -drive file=/home/peter/debian.img,if=virtio  -drive file=/home/peter/Downloads/ubuntu-13.04-desktop-amd64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd  -drive file=/home/peter/Downloads/virtio-win-0.1-65.iso,id=driviso -device ide-cd,bus=ahci.2,drive=driviso -usb   -boot order=cd -usb -usbdevice host:03f0:0024 -usbdevice host:046d:c051

dmesg | grep KVM

[18766.868390] kvm [3596]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
[18767.310411] kvm: zapping shadow pages for mmio generation wraparound

dmesg | grep VFIO

[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.12.0-rc4 root=/dev/mapper/vmtest--vg-root ro quiet modprobe.blacklist=radeon pci-stub.ids=1002:6719,1002:aa80 intel_iommu=on kvm-intel.emulate_invalid_guest_state=1 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream console=tty0 console=ttyS0,115200n8
[    2.959621] VFIO - User Level meta-driver version: 0.3
[18723.504293] vfio-pci 0000:03:00.0: enabling device (0140 -> 0143)
[18771.524071] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X
[18776.940059] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X

dmesg | grep iommu

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.12.0-rc4 root=/dev/mapper/vmtest--vg-root ro quiet modprobe.blacklist=radeon pci-stub.ids=1002:6719,1002:aa80 intel_iommu=on kvm-intel.emulate_invalid_guest_state=1 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream console=tty0 console=ttyS0,115200n8
[    0.000000] Intel-IOMMU: enabled
[    0.020122] dmar: IOMMU 0: reg_base_addr feb03000 ver 1:0 cap c9008020230272 ecap 1000
[    2.886000] IOMMU 0 0xfeb03000: using Register based invalidation
[    2.886004] IOMMU: Setting RMRR:
[    2.886022] IOMMU: Setting identity map for device 0000:00:1a.7 [0x9fcfc000 - 0x9fcfefff]
[    2.886046] IOMMU: Setting identity map for device 0000:00:1d.7 [0x9fcff000 - 0x9fd01fff]
[    2.886066] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9fd32000 - 0x9fd34fff]
[    2.886084] IOMMU: Setting identity map for device 0000:00:1d.1 [0x9fd32000 - 0x9fd34fff]
[    2.886103] IOMMU: Setting identity map for device 0000:00:1d.2 [0x9fd32000 - 0x9fd34fff]
[    2.886121] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9fd32000 - 0x9fd34fff]
[    2.886140] IOMMU: Setting identity map for device 0000:00:1a.1 [0x9fd32000 - 0x9fd34fff]
[    2.886150] IOMMU: Setting identity map for device 0000:00:1d.7 [0x9fd32000 - 0x9fd34fff]
[    2.886153] IOMMU: Setting identity map for device 0000:00:1a.7 [0x9fd32000 - 0x9fd34fff]
[    2.886156] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9fd02000 - 0x9fd07fff]
[    2.886159] IOMMU: Setting identity map for device 0000:00:1a.1 [0x9fd02000 - 0x9fd07fff]
[    2.886162] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9fd08000 - 0x9fd10fff]
[    2.886165] IOMMU: Setting identity map for device 0000:00:1d.1 [0x9fd08000 - 0x9fd10fff]
[    2.886168] IOMMU: Setting identity map for device 0000:00:1d.2 [0x9fd08000 - 0x9fd10fff]
[    2.886172] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    2.886182] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Yes, it is using Debian in this case. I started with Arch and it does exactly the same thing (Ubuntu ok, Windows not).

Any help appreciated!

Offline

#903 2013-12-19 10:02:51

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys, I just registered to thank you all for your work - because of this thread, I have finally got KVM/QEMU VGA passthrough working on my GT660Ti.

My setup is i7-4770 / ASRock Z87 Extreme 4 / Nvidia GT660Ti, host OS is Debian x64, guest OS is Windows 7 x64, keyboard/mouse sharing with Synergy.

I had bought the processor/mobo combo specifically to try and do VGA passthrough and had been really struggling first with Xen and latterly with KVM until I found this thread - the information in the OP and the patches it links to have been invaluable.

Thanks once again, all.

IMG_20131218_221423.jpg

Offline

#904 2013-12-20 04:28:50

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, really a GREAT thread!!!

It seems that the current generation (haswell) of Intel mobile processors supports vt-d in both cpu (i.e. i7-4700HQ) and chipset (i.e. HM87). Has anyone tried vga-passthrough on a laptop? Looking forward to seeing any successful story.

Offline

#905 2013-12-21 05:24:58

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding my initial troubles with Code 10 and BSOD 0x116: I posted here to Fedora's virt mailing list and Alex Williamson suggested I try with Kernel 3.13 release candidates as well as the Haswell i915 VGA arbritration patches. After compiling qemu.git and kernel 3.13 from Fedora rawhide with the VGA arbitration patches, I'm happy my setup is now working flawlessly and I don't get the BSOD anymore. I didn't even have to reinstall Windows!

Last edited by firewing1 (2013-12-21 08:52:50)

Offline

#906 2013-12-22 13:16:42

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

Regarding my initial troubles with Code 10 and BSOD 0x116: I posted here to Fedora's virt mailing list and Alex Williamson suggested I try with Kernel 3.13 release candidates as well as the Haswell i915 VGA arbritration patches. After compiling qemu.git and kernel 3.13 from Fedora rawhide with the VGA arbitration patches, I'm happy my setup is now working flawlessly and I don't get the BSOD anymore. I didn't even have to reinstall Windows!

This is good news, as I am seeing exactly the same problem on my system now. Could you provide sources to the patches you applied so I can also give it a try? Thanks!

Offline

#907 2013-12-22 14:16:27

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

Regarding my initial troubles with Code 10 and BSOD 0x116: I posted here to Fedora's virt mailing list and Alex Williamson suggested I try with Kernel 3.13 release candidates as well as the Haswell i915 VGA arbritration patches. After compiling qemu.git and kernel 3.13 from Fedora rawhide with the VGA arbitration patches, I'm happy my setup is now working flawlessly and I don't get the BSOD anymore. I didn't even have to reinstall Windows!

Hello,

Can you provide your patch please.

Thanks.

Offline

#908 2013-12-23 00:09:13

revasm
Member
Registered: 2013-12-22
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i915_313rc4.patch

Applies cleanly, but 100% untested. YMMV.

The patch was generated by reverting to Linus's commit:
   cd4edf7 Merge branch 'drm-fixes' of git://people.freedesktop.org/~airlied/linux

Followed by undoing the below changes and merging onto rc4.
   e1264eb Revert "drm/i915: Delay disabling of VGA memory until vgacon->fbcon handoff is done"
   ebff5fa9 Revert "i915: Update VGA arbiter support for newer devices"

Alex wrote that these changes are necessary, but they seem intact.
   5c0f6ee vgaarb: Fix VGA decodes changes
   f22d776 vgaarb: Don't disable resources that are not owned

Further reading:

  1. https://lkml.org/lkml/2013/8/15/458

  2. https://lkml.org/lkml/2013/8/28/362

  3. http://cgit.freedesktop.org/~danvet/drm … ba24389bee

Offline

#909 2013-12-23 05:44:23

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

This is good news, as I am seeing exactly the same problem on my system now. Could you provide sources to the patches you applied so I can also give it a try? Thanks!

Val532 wrote:

Can you provide your patch please.

The poster above covers it pretty well - apply the VGA arbitration patch to 3.13rc and you should be good.

A.W. recommended using kernel 3.13rc + qemu.git OR kernel 3.12.5 + qemu 1.7.0 with the following commits patched in: https://lists.fedoraproject.org/piperma … 03916.html

I have tried both 3.13rc4 and 3.12.5, and as A.W. said either work perfectly after applying the patches. I believe the key is the 915 VGA arbiter patch, and NoSnoop if your device is listing NoSnoop+ in "lspci -vv".

For those interested in replicating my configuration exactly, I also used this kernel patch to fix an IOMMU-related traceback present in dmesg on boot: Quirk non-compliant PCIe-to-PCI bridges - fixes backtrace in dmesg on certain systems with broken PCIe-to-PCI bridges - http://lists.linux-foundation.org/piper … 05668.html

Last edited by firewing1 (2013-12-23 05:46:08)

Offline

#910 2013-12-23 06:27:52

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok thanks.

But i have one problem, with kernel 3.13-rc4 (with i915-patch and acs patch) and qemu from git, my VM (Windows 7) refuse to boot ! But with qemu 1.7 it work.

And my VGA card, an 7770HD have NoSnoop+ so i patch qemu with NoSnoop patch but the error 0x00000116 is always here !

It's crazy, because in the past i managed to work, but i don't understand what is the difference with my setup now !

Offline

#911 2013-12-23 08:54:53

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

And my VGA card, an 7770HD have NoSnoop+ so i patch qemu with NoSnoop patch but the error 0x00000116 is always here !

It's crazy, because in the past i managed to work, but i don't understand what is the difference with my setup now !

Are you passing an HDMI audio device along with your GPU? If so, try booting after changing "bus=root.1,addr=0x01" for "bus=pcie.0" on the HDMI PCI device or alternatively remove it altogether. In my case (as other Radeon users have suggested in this thread), I had to move my HDMI device to pcie.0 in order to overcome BSOD 0x116, even with all patches applied. Prior to the patches, I would get 0x116 regardless of the status of the HDMI PCI device.

Offline

#912 2013-12-23 14:15:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

A.W. recommended using kernel 3.13rc + qemu.git OR kernel 3.12.5 + qemu 1.7.0 with the following commits patched in: https://lists.fedoraproject.org/piperma … 03916.html

I have tried both 3.13rc4 and 3.12.5, and as A.W. said either work perfectly after applying the patches. I believe the key is the 915 VGA arbiter patch, and NoSnoop if your device is listing NoSnoop+ in "lspci -vv".

The commits in the above url supersede the NoSnoop patch.  The NoSnoop patch should no longer be necessary.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#913 2013-12-23 15:03:53

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

Are you passing an HDMI audio device along with your GPU? If so, try booting after changing "bus=root.1,addr=0x01" for "bus=pcie.0" on the HDMI PCI device or alternatively remove it altogether. In my case (as other Radeon users have suggested in this thread), I had to move my HDMI device to pcie.0 in order to overcome BSOD 0x116, even with all patches applied. Prior to the patches, I would get 0x116 regardless of the status of the HDMI PCI device.

I tested all possibility and i do not understand ! Is it possible that an external library causes the problem ?

I will do more test during the next hours.

Thanks.

And

aw wrote:

The commits in the above url supersede the NoSnoop patch.  The NoSnoop patch should no longer be necessary.

So you're saying with Kernel 3.13-rc4 and qemu git is no need to use NoSnoop patch, but acs patch and i915 patch is needed ?

Last edited by Val532 (2013-12-23 15:05:16)

Offline

#914 2013-12-23 15:17:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:
aw wrote:

The commits in the above url supersede the NoSnoop patch.  The NoSnoop patch should no longer be necessary.

So you say with Kernel 3.13-rc4 and qemu git is no need to use NoSnoop patch, but acs patch and i915 patch is needed ?

The ACS patch should be used with caution.  It's purpose is to allow the user to override and supplement the PCI ACS information provided by the hardware.  If used improperly this can indicate that devices are isolated from each other when they actually are not.  Lack of real isolation can mean DMA transactions can go to other devices rather than be translated through the IOMMU.  This can create very difficult to debug problems.  There's a reason why this patch was rejected upstream.

The i915 patches are necessary if you have Intel graphics and it will cause DRI to be disabled on the host X session using the Intel device.  AFAIK, these patches are correct other than inducing poor behavior in other layers.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#915 2013-12-23 16:47:14

techdude300
Member
Registered: 2012-04-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this guide! I've been wanting to do something like this for a long time, was about to go the Xen route but this seemed a little easier for a noob like me. Mostly everything works, though I'm still having issues with getting it to play nice with pulseaudio (I may go get a cheap usb soundcard and just pass that through if all else fails).

One strange thing I've noticed is that when I start a VM and then shut it down, either using system_powerdown qemu command or shutting it down from within the Windows VM (Win 8) it shuts down cleanly, but doesn't free up the memory it allocated to the VM. This means that after running and stopping a VM, I have about 3GB of memory my system is telling me is still in use. I can't find any process that's using all this memory. There's no lingering qemu processes, and top/htop doesn't show anything using up all that memory (even run as root). There also aren't any huge files in /tmp that might cause this, but free -h and similar tools insist that 3GB is being used and it certainly acts like it, since if I try to run the command to start the VM again it just steals another 3GB, and trying it a 3rd time pushes my poor 8GB machine to start swapping like a maniac, usually so bad I just have to hard reset the host, which I hate to do. Is this normal for qemu, and I'm just not understanding something right? Even if I could allocate that 3GB on boot and have it be re-used by subsequent VM launches would be nice. The command I'm using (as root):

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -M q35 -m 3072 \ -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/william/VirtualBox\ VMs/Windows\ 8/Windows\ 8.vdi,id=disk,format=vdi,cache=none \ -device ide-hd,bus=ahci.0,drive=disk \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -monitor stdio \ -device vfio-pci,host=00:12.0,bus=pcie.0 \ -device vfio-pci,host=00:12.2,bus=pcie.0

Where 04:00.* is the video card (radeon 4850), and 00:12.* is a usb controller. Guest VM is Windows 8 x64, host is running kernel 3.13.0-rc3.

(EDIT: spaced things out better in the above command for readbility)

Last edited by techdude300 (2013-12-23 16:52:23)

Offline

#916 2013-12-23 18:14:07

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

I tested all possibility and i do not understand ! Is it possible that an external library causes the problem ?

In my case it was definitely kernel + QEMU that was the cause. As soon as I upgraded to kernel-3.13rc and QEMU.git, the very same virtual machine image started working without a BSOD.

Val532 wrote:

So you're saying with Kernel 3.13-rc4 and qemu git is no need to use NoSnoop patch, but acs patch and i915 patch is needed ?

The i915 patches are needed in either case, NoSnoop is only needed if your device is showing "NoSnoop+" when you check "lspci -vv". I didn't use the ACS patch, but if you need to refer to A.W.'s post above.

techdude300 wrote:

One strange thing I've noticed is that when I start a VM and then shut it down, either using system_powerdown qemu command or shutting it down from within the Windows VM (Win 8) it shuts down cleanly, but doesn't free up the memory it allocated to the VM

What's the output of "free -m" after closing the VM? My guess is that the memory is just being cached in case you re-start the VM again, and will be freed automatically as the system demands it. If you see a large amount of memory in the "cached" column from the command, then this is the case.

Last edited by firewing1 (2013-12-23 18:15:07)

Offline

#917 2013-12-23 19:15:35

techdude300
Member
Registered: 2012-04-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:
techdude300 wrote:

One strange thing I've noticed is that when I start a VM and then shut it down, either using system_powerdown qemu command or shutting it down from within the Windows VM (Win 8) it shuts down cleanly, but doesn't free up the memory it allocated to the VM

What's the output of "free -m" after closing the VM? My guess is that the memory is just being cached in case you re-start the VM again, and will be freed automatically as the system demands it. If you see a large amount of memory in the "cached" column from the command, then this is the case.

I don't think it's cached, because the system doesn't seem to be freeing it when it's needed, because when I stop and start the VM three times (3gb vm, 8gb mem on host) it starts swapping very heavily. Here's free -m before a run:

 total used free shared buffers cached Mem: 7953 1666 6287 32 56 564 -/+ buffers/cache: 1046 6907 Swap: 8187 0 8187

While qemu is running:

 total used free shared buffers cached Mem: 7953 4837 3116 33 56 577 -/+ buffers/cache: 4202 3751 Swap: 8187 0 8187

And after a clean shutdown (shutdown via guest), qemu not running:

 total used free shared buffers cached Mem: 7953 4787 3166 28 57 579 -/+ buffers/cache: 4150 3803 Swap: 8187 0 8187

I'm not very knowledgable on Linux memory management, but if the memory doesn't appear to be used by a process, or any tmpfs using RAM, could that mean that the kernel is allocating all that memory and not releasing it? Could this be a kernel bug? Some things I've noticed after further testing that the issue is not affected by choice of guest OS (tested Ubuntu and Windows 8), and memory is successfully returned if I don't pass any devices through (both Ubuntu and Windows). However, when I pass either the USB controller, VGA card, or both to the VM (both Ubuntu and Win 8) memory is not returned on qemu exiting. This makes me think it's some kind of bug in either the kernel, qemu, or my motherboard's IOMMU stuff. Motherboard is Gigabyte 531020 GA-990FXA-UD3, just bought it a few days ago. I've been having issues with USB3 while UEFI booting, and I remember there being something about vga passthrough on Xen not working right when the host was UEFI booted, so I wouldn't be surprised if that's messing something up. I'll set up grub and do a normal bios boot and see if it changes anything.

Offline

#918 2013-12-23 19:19:23

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The ACS patch should be used with caution.  It's purpose is to allow the user to override and supplement the PCI ACS information provided by the hardware.  If used improperly this can indicate that devices are isolated from each other when they actually are not.  Lack of real isolation can mean DMA transactions can go to other devices rather than be translated through the IOMMU.  This can create very difficult to debug problems.  There's a reason why this patch was rejected upstream.

The i915 patches are necessary if you have Intel graphics and it will cause DRI to be disabled on the host X session using the Intel device.  AFAIK, these patches are correct other than inducing poor behavior in other layers.

Ok so with a 3.13-rc4 without patch (no i915 or acs) my VM do not work, but i do not see any BSOD just reboot so it's maybe an other problem in my machine. Also i see a new problem in dmesg durring the VM work  :

vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

and at the stop of the VM :

[ 663.077661] WARNING: CPU: 1 PID: 532 at drivers/gpu/drm/i915/intel_display.c:6309 hsw_enable_pc8_work+0x6c9/0x6f0 [i915]() [ 663.077662] Power well on [ 663.077663] Modules linked in: vfio_iommu_type1(F) vfio_pci(F) vfio(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) bnep(F) bridge(F) rfcomm(F) stp(F) bluetooth(F) llc(F) parport_pc(F) ppdev(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) mxm_wmi(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) snd_hda_intel(F) snd_hda_codec(F) snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) i915(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) psmouse(F) mei_me(F) drm_kms_helper(F) drm(F) mei(F) microcode(F) snd(F) soundcore(F) serio_raw(F) video(F) wmi(F) lpc_ich(F) mac_hid(F) lp(F) parport(F) hid_generic(F) usbhid(F) hid(F) igb(F) i2c_algo_bit(F) dca(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F) [ 663.077693] CPU: 1 PID: 532 Comm: kworker/1:2 Tainted: GF W 3.13.0-rc4-no-patch #1 [ 663.077694] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme6, BIOS P2.10 07/03/2013 [ 663.077702] Workqueue: events hsw_enable_pc8_work [i915] [ 663.077703] 0000000000000009 ffff88060a671d58 ffffffff81714c09 ffff88060a671da0 [ 663.077706] ffff88060a671d90 ffffffff810643ad ffff88060bf83bc8 ffff88060bf80000 [ 663.077707] ffff88060d116350 ffff88060d116358 0000000000000040 ffff88060a671df0 [ 663.077710] Call Trace: [ 663.077715] [<ffffffff81714c09>] dump_stack+0x45/0x56 [ 663.077718] [<ffffffff810643ad>] warn_slowpath_common+0x7d/0xa0 [ 663.077720] [<ffffffff8106441c>] warn_slowpath_fmt+0x4c/0x50 [ 663.077728] [<ffffffffa026d0e9>] hsw_enable_pc8_work+0x6c9/0x6f0 [i915] [ 663.077730] [<ffffffff810800f2>] process_one_work+0x182/0x450 [ 663.077732] [<ffffffff81080eb1>] worker_thread+0x121/0x410 [ 663.077734] [<ffffffff81080d90>] ? rescuer_thread+0x3e0/0x3e0 [ 663.077737] [<ffffffff81087b52>] kthread+0xd2/0xf0 [ 663.077739] [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190 [ 663.077742] [<ffffffff8172577c>] ret_from_fork+0x7c/0xb0 [ 663.077744] [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190 [ 663.077745] ---[ end trace 47811739132dc27e ]--- [ 726.106603] ------------[ cut here ]------------ [ 726.106623] WARNING: CPU: 1 PID: 532 at drivers/gpu/drm/i915/intel_display.c:6309 hsw_enable_pc8_work+0x6c9/0x6f0 [i915]() [ 726.106624] Power well on [ 726.106625] Modules linked in: vfio_iommu_type1(F) vfio_pci(F) vfio(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) bnep(F) bridge(F) rfcomm(F) stp(F) bluetooth(F) llc(F) parport_pc(F) ppdev(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) mxm_wmi(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) snd_hda_intel(F) snd_hda_codec(F) snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) i915(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) psmouse(F) mei_me(F) drm_kms_helper(F) drm(F) mei(F) microcode(F) snd(F) soundcore(F) serio_raw(F) video(F) wmi(F) lpc_ich(F) mac_hid(F) lp(F) parport(F) hid_generic(F) usbhid(F) hid(F) igb(F) i2c_algo_bit(F) dca(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F) [ 726.106654] CPU: 1 PID: 532 Comm: kworker/1:2 Tainted: GF W 3.13.0-rc4-no-patch #1 [ 726.106655] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme6, BIOS P2.10 07/03/2013 [ 726.106663] Workqueue: events hsw_enable_pc8_work [i915] [ 726.106664] 0000000000000009 ffff88060a671d58 ffffffff81714c09 ffff88060a671da0 [ 726.106667] ffff88060a671d90 ffffffff810643ad ffff88060bf83bc8 ffff88060bf80000 [ 726.106669] ffff88060d116350 ffff88060d116358 0000000000000040 ffff88060a671df0 [ 726.106671] Call Trace: [ 726.106676] [<ffffffff81714c09>] dump_stack+0x45/0x56 [ 726.106679] [<ffffffff810643ad>] warn_slowpath_common+0x7d/0xa0 [ 726.106681] [<ffffffff8106441c>] warn_slowpath_fmt+0x4c/0x50 [ 726.106690] [<ffffffffa026d0e9>] hsw_enable_pc8_work+0x6c9/0x6f0 [i915] [ 726.106693] [<ffffffff810800f2>] process_one_work+0x182/0x450 [ 726.106694] [<ffffffff81080eb1>] worker_thread+0x121/0x410 [ 726.106696] [<ffffffff81080d90>] ? rescuer_thread+0x3e0/0x3e0 [ 726.106699] [<ffffffff81087b52>] kthread+0xd2/0xf0 [ 726.106701] [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190 [ 726.106703] [<ffffffff8172577c>] ret_from_fork+0x7c/0xb0 [ 726.106706] [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190 [ 726.106707] ---[ end trace 47811739132dc27f ]--- [ 741.101725] ------------[ cut here ]------------ [ 741.101745] WARNING: CPU: 0 PID: 46 at drivers/gpu/drm/i915/intel_display.c:6309 hsw_enable_pc8_work+0x6c9/0x6f0 [i915]() [ 741.101746] Power well on [ 741.101747] Modules linked in: vfio_iommu_type1(F) vfio_pci(F) vfio(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) bnep(F) bridge(F) rfcomm(F) stp(F) bluetooth(F) llc(F) parport_pc(F) ppdev(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) mxm_wmi(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) snd_hda_intel(F) snd_hda_codec(F) snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) i915(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) psmouse(F) mei_me(F) drm_kms_helper(F) drm(F) mei(F) microcode(F) snd(F) soundcore(F) serio_raw(F) video(F) wmi(F) lpc_ich(F) mac_hid(F) lp(F) parport(F) hid_generic(F) usbhid(F) hid(F) igb(F) i2c_algo_bit(F) dca(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F) [ 741.101776] CPU: 0 PID: 46 Comm: kworker/0:1 Tainted: GF W 3.13.0-rc4-no-patch #1 [ 741.101777] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme6, BIOS P2.10 07/03/2013 [ 741.101785] Workqueue: events hsw_enable_pc8_work [i915] [ 741.101787] 0000000000000009 ffff88060d3efd58 ffffffff81714c09 ffff88060d3efda0 [ 741.101789] ffff88060d3efd90 ffffffff810643ad ffff88060bf83bc8 ffff88060bf80000 [ 741.101791] ffff88060d116350 ffff88060d116358 0000000000000000 ffff88060d3efdf0 [ 741.101793] Call Trace: [ 741.101797] [<ffffffff81714c09>] dump_stack+0x45/0x56 [ 741.101800] [<ffffffff810643ad>] warn_slowpath_common+0x7d/0xa0 [ 741.101802] [<ffffffff8106441c>] warn_slowpath_fmt+0x4c/0x50 [ 741.101810] [<ffffffffa026d0e9>] hsw_enable_pc8_work+0x6c9/0x6f0 [i915] [ 741.101813] [<ffffffff810800f2>] process_one_work+0x182/0x450 [ 741.101815] [<ffffffff81080eb1>] worker_thread+0x121/0x410 [ 741.101817] [<ffffffff81080d90>] ? rescuer_thread+0x3e0/0x3e0 [ 741.101820] [<ffffffff81087b52>] kthread+0xd2/0xf0 [ 741.101822] [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190 [ 741.101825] [<ffffffff8172577c>] ret_from_fork+0x7c/0xb0 [ 741.101827] [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190 [ 741.101828] ---[ end trace 47811739132dc280 ]---

Offline

#919 2013-12-23 19:29:41

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@techdude300: try to uprade your kernel to latest -rc5. -rc3 was very flaky here, too.

Offline

#920 2013-12-23 21:36:06

revasm
Member
Registered: 2013-12-22
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Code 43 on i5-2500 IGP and Nvidia 560 TI. VGA works well (1200x800). Device/bus reset seems broken: QEMU issues "Invalid ROM" error on restart.

Z68 was not supposed to have full support for an IOMMU anyway, so I'm surprised that VFIO works at all.

Linux 3.13-rc4
qemu.git 2013-12-20
i915 driver arbiter patches
qemu-vfio NoSnoop v1 patch (unpatched/patched, didn't make a difference, except to flip the NoSnoop bit)


# cat run-kvm.sh #!/bin/sh ./pcistub-bind "8086 1c2d" "0000:00:1a.0" # USB 2.0 controller ./vfio-group 1 qemu-system-x86_64 -nodefaults \ -enable-kvm -M q35 -m 2048 -cpu SandyBridge \ -smp cores=2,threads=1,sockets=1 \ -bios /usr/share/qemu/bios.bin -rtc base=localtime \ -vga none -nographic -monitor stdio -boot menu=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=ich9-pcie-port-1 \ -device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1,addr=00.0,multifunction=on,x-vga=on,romfile=/export/misc/GF114.bin \ -device vfio-pci,host=01:00.1,bus=ich9-pcie-port-1,addr=00.1 \ -device pci-assign,host=00:1a.0,addr=1a.0,multifunction=on,id=ehci-1 \ -device ide-cd,drive=cd1,bus=ide.1 \ -device ide-cd,drive=cd2,bus=ide.2 \ -drive id=cd1,media=cdrom,index=1,file=/export/iso/en_windows_7*.iso \ -drive id=cd2,media=cdrom,index=2,file=/export/iso/virtio-win-0.1-74.iso \ -drive if=virtio,index=0,cache=none,format=qcow2,file=/export/vm/win7sp1.qcow2 \ -net bridge,br=br0 -net nic,model=virtio
# cat /proc/cmdline initrd=\initramfs-linux-mainline.img root=PARTUUID="667c0274-455e-46a4-8f37-cab6d73269a6" rw acpi_osi=Linux intel_iommu=on pci-stub.ids=8086:1c2d,10de:1200,10de:0e0c
# cat /proc/version Linux version 3.13.0-1-mainline (gcc version 4.8.2 (GCC) ) #1 SMP PREEMPT Sun Dec 22 14:50:44 PST 2013
# lsgroup | grep "Group 1" -A4 ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1) ### Group 2 ###
# lspci | egrep "VGA|Audio" 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05) 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)
# head -n1 /dev/vga_arbiter count:2,PCI:0000:00:02.0,decodes=io,owns=none,locks=none(0:0)
# dmesg | grep -i iommu [ 0.000000] Intel-IOMMU: enabled [ 0.286043] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.286048] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.286118] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.567671] IOMMU 0 0xfed90000: using Queued invalidation [ 0.567672] IOMMU 1 0xfed91000: using Queued invalidation [ 0.567673] IOMMU: Setting RMRR: [ 0.567683] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff] [ 0.567983] IOMMU: Setting identity map for device 0000:00:1d.0 [0xca8c1000 - 0xca8d2fff] [ 0.567999] IOMMU: Setting identity map for device 0000:00:1a.0 [0xca8c1000 - 0xca8d2fff] [ 0.568008] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.568015] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
# dmesg | tail -n12 [ 70.786246] pci-stub 0000:00:1a.0: claimed by stub [ 70.831198] VFIO - User Level meta-driver version: 0.3 [ 71.564129] tun: Universal TUN/TAP device driver, 1.6 [ 71.564132] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 71.564688] device tap0 entered promiscuous mode [ 71.564776] br0: port 2(tap0) entered forwarding state [ 71.564781] br0: port 2(tap0) entered forwarding state [ 72.140304] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 72.232772] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002) [ 72.675966] pci-stub 0000:00:1a.0: kvm assign device [ 86.574242] br0: port 2(tap0) entered forwarding state [ 87.229914] kvm: zapping shadow pages for mmio generation wraparound
# dmesg | grep -i vga [ 0.482563] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.482566] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.482567] vgaarb: loaded [ 0.482568] vgaarb: bridge control possible 0000:01:00.0 [ 0.482568] vgaarb: no bridge control possible 0000:00:02.0 [ 0.582479] fb0: EFI VGA frame buffer device [ 4.441728] fb: conflicting fb hw usage inteldrmfb vs EFI VGA - removing generic driver [ 4.572183] [drm] GMBUS [i915 gmbus vga] timed out, falling back to bit banging on pin 2 [ 4.852049] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io
# lspci -vvv | grep GTX -A48 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 1563 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=32M] Region 1: Memory at e0000000 (64-bit, prefetchable) [size=128M] Region 3: Memory at e8000000 (64-bit, prefetchable) [size=64M] Region 5: I/O ports at e000 [size=128] Expansion ROM at f6000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci Kernel modules: nouveau

Offline

#921 2013-12-24 02:11:06

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

My first sucess, now i have a working setup, with 3.12.0 kernel and i915 patch, and qemu git + NoSnoop patch (probably v2 but i'm not sure).
Now i will try with 3.12.6 kernel.

Edit : And only with hdmi audio on pcie.0 insted of root.1

Edit 2nd : Now it work for me with 3.13-rc5 + i915 patch and qemu git without any patch. But only with hdmi audio as bus=pcie.0 and not root.1

Last edited by Val532 (2013-12-24 04:10:16)

Offline

#922 2013-12-24 02:19:54

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This thread has been great. Thanks for the all the info so far. Also I have hardware now that does work and is supported for IOMMU. However now I am running into an issue when I run the qemu commands. Here is my commands and output:

qemu-system-x86_64 -enable-kvm -M q35 -m 16024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 17 qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What am I missing here? It seems like its almost there but I am sure I am issing something here. The error seems to indicate to put this in an iommu_group which I was under the impression I did.

Edit: Specs. Im running on an TYAN S8225 board. Running 32Gb of memory with 2x 4234 Opterons, OS: Linuxmint 16 with 3.12.6 kernel. I have an nvidia 660ti as the host card and my intended card is an AMD R9 270X.

Thanks

Last edited by paradexes (2013-12-24 04:48:01)

Offline

#923 2013-12-24 10:37:07

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@techdude300: i have exactly same problem with memory, pls let me/us know if you find a fix/workaround;

@paradexes: i had same output, but i saw it works prefix sudo ...  so looking into it i found this :

...The final step is to provide the user with access to the group if unprivileged operation is desired (note that /dev/vfio/vfio provides no capabilities on its own and is therefore expected to be set to mode 0666 by the system)...

# chown user:user /dev/vfio/17

Afterwards i think should work. This is like a default and needs a change, i would vote for this to be mentioned by OP because it was not obvious for me either.

Offline

#924 2013-12-24 18:53:02

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well i'm back with a "Crosshair V Formula" motherboard and FX CPU. Upon booting the VM, my system freezes up when the BIOS screen loads and says no bootable device. The text is also strangly spaced out wide.

Anyone know how to fix the crash or is this because the BIOS is resetting after failing to boot?

Offline

#925 2013-12-25 11:57:40

xani
Member
Registered: 2013-12-24
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, first of all big thanks to nbhs and Alex for your awesome job!

I managed to make this work with pci-assign back in July, but I couldn't reboot even with ejecting gpu on shutdown. I was using windows 8 and it gave BSOD on shutdown everytime and gpu wouldn't reset.
Now I want to do this using vfio. I managed to get it working with acs override and i915 patches for rc3. I'm using qemu-git, seabios-git and linux-mainline provided by nbhs at first page.

My setup:
cpu: i7-4770
mobo: asrock z87 extreme6
gpu: radeon hd7870 GHz edition

Anyway, I had the same problem as noobman and techdude300, memory wouldn't free after closing qemu.
I bisected problem to commit

0c44c2d0f459cd7e275242b72f500137c4fa834d

I also reverted this one, I think it would'nt build without reverting this but now I'm not sure.

b021fe3e25094fbec22d0eff846d2adeee1b9736

Now it does free memory, but I cannot reboot anyway because seabios is hanging at '_' and after a while it shows that it cannot find hard drive to boot from.
Host reboot is necessary to fire up vm again.

My config:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=a63d1c94-4682-431c-9788-0cccd29ee598 rw intel_iommu=on pci-stub.ids=1002:6818,1002:aab0,8086:0 c0c,8086:153b pcie_acs_override=downstream
qemu-system-x86_64 -M q35 \ -m 8196 -enable-kvm -cpu host -monitor stdio \ -smp 4,sockets=1,cores=4,threads=1 \ -rtc clock=host,base=localtime \ -vga none -net none \ -parallel none -serial none \ -usb -usbdevice host:05ac:0221 -usbdevice host:1532:0037 \ -device ahci,bus=pcie.0,id=ahci \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -device vfio-pci,host=00:19.0,bus=pcie.0 \ -device vfio-pci,host=05:00.0,bus=pcie.0 

I use nbhs script from first page to bind devices.

Here's my directory from which you can test that memory leak issues.
linux-mainline.tar.gz

Provide me with feedback if you have the same problem with seabios after reverting those patches.

EDIT:
Further investigation showed that this problem can lie within gcc itself. Here's link that can give you explanation. I will report on my findings asap.

EDIT2:
OK, so after disabling asm goto with this little patch, it works just like reversing commits. But this bug should be fixed by commit

88f182dd77

Maybe it's another thing then. I still have few options to try.

Last edited by xani (2013-12-25 14:49:06)

Offline

#926 2013-12-25 13:48:52

peaquino
Member
Registered: 2013-12-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I would like to report partial success with my APU setup (A85X, A10-5800K and 7870XT). With latest mainline kernel and qemu-git (from AUR, not the patched versions from first page) I was able to start Windows 8.1 with latest Catalyst and even play few minutes of Metro 2033. Unfortunatelly, the game is hanging up after a while (host stays responsive). It looks like it hangs during intense disk activity (ex. loading textures when entering new area). I tried booting the system with nohz=off and voluntary preemption, it doesn't seem to make any difference. Do you have any idea what else can I check?

Offline

#927 2013-12-25 16:24:14

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

@techdude300: i have exactly same problem with memory, pls let me/us know if you find a fix/workaround;

@paradexes: i had same output, but i saw it works prefix sudo ...  so looking into it i found this :

...The final step is to provide the user with access to the group if unprivileged operation is desired (note that /dev/vfio/vfio provides no capabilities on its own and is therefore expected to be set to mode 0666 by the system)...

# chown user:user /dev/vfio/17

Afterwards i think should work. This is like a default and needs a change, i would vote for this to be mentioned by OP because it was not obvious for me either.

Thanks for the advice. It seems obvious however it still didnt work. I am running these commands as root. Im going in sudo -s instead of just running sudo + command as it gets tedious after hours of that. At any rate seeing as its being run as root and I am still getting this error Im trying to figure out what else it could be. Thanks for the help so far big_smile

Offline

#928 2013-12-25 19:10:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:

This thread has been great. Thanks for the all the info so far. Also I have hardware now that does work and is supported for IOMMU. However now I am running into an issue when I run the qemu commands. Here is my commands and output:

qemu-system-x86_64 -enable-kvm -M q35 -m 16024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 17 qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What am I missing here? It seems like its almost there but I am sure I am issing something here. The error seems to indicate to put this in an iommu_group which I was under the impression I did.

Edit: Specs. Im running on an TYAN S8225 board. Running 32Gb of memory with 2x 4234 Opterons, OS: Linuxmint 16 with 3.12.6 kernel. I have an nvidia 660ti as the host card and my intended card is an AMD R9 270X.

Thanks

The error message is trying really hard to tell you what the problem is: "please ensure all devices within the iommu_group are bound to their vfio bus driver".  So, run lsgroup, look at everything in group 17.  If it's an endpoint it must be bound to vfio-pci (or pci-stub if you don't plan to assign it).  If it's a root port, leave it alone.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#929 2013-12-25 20:04:32

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
paradexes wrote:

This thread has been great. Thanks for the all the info so far. Also I have hardware now that does work and is supported for IOMMU. However now I am running into an issue when I run the qemu commands. Here is my commands and output:

qemu-system-x86_64 -enable-kvm -M q35 -m 16024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 17 qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What am I missing here? It seems like its almost there but I am sure I am issing something here. The error seems to indicate to put this in an iommu_group which I was under the impression I did.

Edit: Specs. Im running on an TYAN S8225 board. Running 32Gb of memory with 2x 4234 Opterons, OS: Linuxmint 16 with 3.12.6 kernel. I have an nvidia 660ti as the host card and my intended card is an AMD R9 270X.

Thanks

The error message is trying really hard to tell you what the problem is: "please ensure all devices within the iommu_group are bound to their vfio bus driver".  So, run lsgroup, look at everything in group 17.  If it's an endpoint it must be bound to vfio-pci (or pci-stub if you don't plan to assign it).  If it's a root port, leave it alone.

aw thanks for the input. This device is an AMD R9 270X. The host card is an nvidia 660ti. The output appears to be correct. Forgive the n00bness in the question but how do I know if its a root port vs not? right now the card is coming up with the usual lspci and grep commands as 0000:03:00.0 and 0000:03:00.1 (hdmi part) They appear to be bound correctly using the vfio-pci script in the OP. This seems like the last piece of the puzzle. I suppose a step by step on what would be needed to fix this error? I have done some reading but nothing is really clearing this part up here. Thanks.

Offline

#930 2013-12-25 20:28:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:
aw wrote:

The error message is trying really hard to tell you what the problem is: "please ensure all devices within the iommu_group are bound to their vfio bus driver".  So, run lsgroup, look at everything in group 17.  If it's an endpoint it must be bound to vfio-pci (or pci-stub if you don't plan to assign it).  If it's a root port, leave it alone.

aw thanks for the input. This device is an AMD R9 270X. The host card is an nvidia 660ti. The output appears to be correct. Forgive the n00bness in the question but how do I know if its a root port vs not? right now the card is coming up with the usual lspci and grep commands as 0000:03:00.0 and 0000:03:00.1 (hdmi part) They appear to be bound correctly using the vfio-pci script in the OP. This seems like the last piece of the puzzle. I suppose a step by step on what would be needed to fix this error? I have done some reading but nothing is really clearing this part up here. Thanks.

lsgroup will give you the lspci description of the device, you can interrogate further with lspci -vvv -s <pci device>, or just paste the lsgroup output for the group and we can tell you.  Chances are that there are more devices in the group than just the GPU and audio device if you're getting this error.  You can also use 'find /sys/kernel/iommu_groups' to understand the grouping if you don't have lsgroup (it's pasted somewhere in this forum thread).  If you want to change the grouping beyond what the hardware tells us is safe, you can use the ACS override patch to supply boot options to do this, but note the risks I mentioned a page or two back about using this.  I hope to be able to release a patch that will make this unnecessary for a bunch of Intel root ports soon.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#931 2013-12-25 20:42:36

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I hope to be able to release a patch that will make this unnecessary for a bunch of Intel root ports soon.

So you work on new proper patch for ACS on Intel ??? Or it's just a wish ?


And is someone has managed to use a Sata card with vfio ?

I have two AsMedia 1062 (2*sata3) and it does not work correctly.

Thanks

Offline

#932 2013-12-25 21:32:39

paradexes
Member
Registered: 2013-11-30
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
paradexes wrote:
aw wrote:

The error message is trying really hard to tell you what the problem is: "please ensure all devices within the iommu_group are bound to their vfio bus driver".  So, run lsgroup, look at everything in group 17.  If it's an endpoint it must be bound to vfio-pci (or pci-stub if you don't plan to assign it).  If it's a root port, leave it alone.

aw thanks for the input. This device is an AMD R9 270X. The host card is an nvidia 660ti. The output appears to be correct. Forgive the n00bness in the question but how do I know if its a root port vs not? right now the card is coming up with the usual lspci and grep commands as 0000:03:00.0 and 0000:03:00.1 (hdmi part) They appear to be bound correctly using the vfio-pci script in the OP. This seems like the last piece of the puzzle. I suppose a step by step on what would be needed to fix this error? I have done some reading but nothing is really clearing this part up here. Thanks.

lsgroup will give you the lspci description of the device, you can interrogate further with lspci -vvv -s <pci device>, or just paste the lsgroup output for the group and we can tell you.  Chances are that there are more devices in the group than just the GPU and audio device if you're getting this error.  You can also use 'find /sys/kernel/iommu_groups' to understand the grouping if you don't have lsgroup (it's pasted somewhere in this forum thread).  If you want to change the grouping beyond what the hardware tells us is safe, you can use the ACS override patch to supply boot options to do this, but note the risks I mentioned a page or two back about using this.  I hope to be able to release a patch that will make this unnecessary for a bunch of Intel root ports soon.


Ok here is the output I have from the lsgroup script. (applicable to group 17 since thats where I am having the issue. also below is the output to the lspci and find commands you specified thanks again.

~#sh lsgroup.sh ### Group 17 ### 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ~#find /sys/kernel/iommu_groups/ /sys/kernel/iommu_groups/17 /sys/kernel/iommu_groups/17/devices /sys/kernel/iommu_groups/17/devices/0000:03:00.0 /sys/kernel/iommu_groups/17/devices/0000:03:00.1 ~ # lspci -vvv -s 0000:03:00.0 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 2272 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 32 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at fcc80000 (64-bit, non-prefetchable) [size=256K] Region 4: I/O ports at b000 [size=256] Expansion ROM at fcc60000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: vfio-pci ~ # lspci -vvv -s 0000:03:00.1 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: Gigabyte Technology Co., Ltd Device aab0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 124 Region 0: Memory at fccfc000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: snd_hda_intel

I think from what I am seeing its actually the HDMI 03.00.1 that is actually causing my problem? I presume it would need to have had the same vfio-pci driver as well. That seems tobe the one thing that stood out to me on this. That said I have already done what I needed in order to pass that as well as the card itself. Anything I did to the 03.00.0 I also did to 03.00.1 as well.

Offline

#933 2013-12-25 23:34:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paradexes wrote:

Ok here is the output I have from the lsgroup script. (applicable to group 17 since thats where I am having the issue. also below is the output to the lspci and find commands you specified thanks again.

~#sh lsgroup.sh ### Group 17 ### 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ~ # lspci -vvv -s 0000:03:00.0 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] (prog-if 00 [VGA controller]) Kernel driver in use: vfio-pci ~ # lspci -vvv -s 0000:03:00.1 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Kernel driver in use: snd_hda_intel

I think from what I am seeing its actually the HDMI 03.00.1 that is actually causing my problem? I presume it would need to have had the same vfio-pci driver as well. That seems tobe the one thing that stood out to me on this. That said I have already done what I needed in order to pass that as well as the card itself. Anything I did to the 03.00.0 I also did to 03.00.1 as well.

Yes, you can see from lspci that 3:00.1 is bound to snd_hda_intel, not vfio_pci.  Check your setup procedure.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#934 2013-12-26 06:01:31

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For those of us experiencing the memory allocation bug (guest VM memory is not freed after shutdown), are you passing through USB devices from the host? I just noticed that I encountered the memory issue only after attempting to add a USB mouse attached to the host to my VM,

Offline

#935 2013-12-26 08:20:42

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

For those of us experiencing the memory allocation bug (guest VM memory is not freed after shutdown), are you passing through USB devices from the host? I just noticed that I encountered the memory issue only after attempting to add a USB mouse attached to the host to my VM,

Nope, I've got the memory allocation bug, but the only thing I'm passing through is the VGA card.  This is my invocation:

sudo QEMU_AUDIO_DRV=alsa \ QEMU_AUDIO_TIMER_PERIOD=0 \ qemu-system-x86_64 -enable-kvm -M q35 -m 3072 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -monitor stdio -bios /usr/share/qemu/bios.bin -vga none \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/tuoni/kvm/win7.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -soundhw hda,ac97

I'm pretty sure some of those switches are sub-optimal, but I'm just sticking with the first thing I got working smile

EDIT: One thing I have noticed is there has been a marked decrease in my host (i915) graphics performance since using mainline-rc3 + i915 patch.  I'm presuming it's the kernel rather than the patch...?

Last edited by tuoni (2013-12-26 08:29:08)

Offline

#936 2013-12-26 10:21:21

xani
Member
Registered: 2013-12-24
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

And is someone has managed to use a Sata card with vfio ?

I have two AsMedia 1062 (2*sata3) and it does not work correctly.

I don't have any problems with this controller.

05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])

I am passing it like this

-device vfio-pci,host=05:00.0,bus=pcie.0

Windows 7 is recognising it without any hassle.

firewing1 wrote:

For those of us experiencing the memory allocation bug (guest VM memory is not freed after shutdown), are you passing through USB devices from the host? I just noticed that I encountered the memory issue only after attempting to add a USB mouse attached to the host to my VM,

For me it doesn't matter if I'm using usb passthrough or not. It happens when using pci-assing or vfio-pci. Did you try my kernel package? Here's new one. It includes acs override and i915 arbiter fix patches.

tuoni wrote:

Nope, I've got the memory allocation bug, but the only thing I'm passing through is the VGA card.

@tuoni you can test if my package helps on mem leaks too.

Offline

#937 2013-12-26 14:26:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

xani wrote:
Val532 wrote:

And is someone has managed to use a Sata card with vfio ?

I have two AsMedia 1062 (2*sata3) and it does not work correctly.

I don't have any problems with this controller.

05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])

I am passing it like this

-device vfio-pci,host=05:00.0,bus=pcie.0

Windows 7 is recognising it without any hassle.

firewing1 wrote:

For those of us experiencing the memory allocation bug (guest VM memory is not freed after shutdown), are you passing through USB devices from the host? I just noticed that I encountered the memory issue only after attempting to add a USB mouse attached to the host to my VM,

For me it doesn't matter if I'm using usb passthrough or not. It happens when using pci-assing or vfio-pci. Did you try my kernel package? Here's new one. It includes acs override and i915 arbiter fix patches.

tuoni wrote:

Nope, I've got the memory allocation bug, but the only thing I'm passing through is the VGA card.

@tuoni you can test if my package helps on mem leaks too.

Thanks a lot for the kernel package, i have linked it at the front page.

Offline

#938 2013-12-26 15:02:49

xani
Member
Registered: 2013-12-24
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Thanks a lot for the kernel package, i have linked it at the front page.

No problem, it is you that started this little vga passthrough movement in here wink

Package that I posted also includes patch that disables gcc asm goto instructions, I think that only folks that have problem with memory leaks should use it. It is really low level x86 and gcc stuff that I don't fully understand.
Anyway, I made new one with all those fixes except asm goto: linux-mainline.tar.gz.
This is ofc yours updated to rc5 and patches so I don't want to take all the credit smile

Offline

#939 2013-12-26 15:51:51

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

xani wrote:
Val532 wrote:

And is someone has managed to use a Sata card with vfio ?

I have two AsMedia 1062 (2*sata3) and it does not work correctly.

I don't have any problems with this controller.

05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])

I am passing it like this

-device vfio-pci,host=05:00.0,bus=pcie.0

Windows 7 is recognising it without any hassle.

It's strange for me it's work but only once, i can start my VM only once, if my VM reboot (cold reboot or hot reboot) it's bug on VM bios screen. And i have some dmesg :

[ 404.206866] dmar: DRHD: handling fault status reg 2 [ 404.206870] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr 2affdf000 [ 404.206870] DMAR:[fault reason 05] PTE Write access is not set [ 404.206877] dmar: DRHD: handling fault status reg 2 [ 404.206879] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 2affda000 [ 404.206879] DMAR:[fault reason 06] PTE Read access is not set

What is your config ?

Last edited by Val532 (2013-12-26 15:55:26)

Offline

#940 2013-12-26 16:10:38

xani
Member
Registered: 2013-12-24
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:
xani wrote:
Val532 wrote:

And is someone has managed to use a Sata card with vfio ?

I have two AsMedia 1062 (2*sata3) and it does not work correctly.

I don't have any problems with this controller.

05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) (prog-if 01 [AHCI 1.0])

I am passing it like this

-device vfio-pci,host=05:00.0,bus=pcie.0

Windows 7 is recognising it without any hassle.

It's strange for me it's work but only once, i can start my VM only once, if my VM reboot (cold reboot or hot reboot) it's bug on VM bios screen. And i have some dmesg :

[ 404.206866] dmar: DRHD: handling fault status reg 2 [ 404.206870] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr 2affdf000 [ 404.206870] DMAR:[fault reason 05] PTE Write access is not set [ 404.206877] dmar: DRHD: handling fault status reg 2 [ 404.206879] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 2affda000 [ 404.206879] DMAR:[fault reason 06] PTE Read access is not set

What is your config ?

Sorry to misguide you, but I noticed that this controller is responsible for seabios hang after I posted, and I wanted to test more before writing about it.
I think that when I was hunting for mem leak issue I changed my config to use this controller and I forgot about it.
Anyway, I think that vfio is not properly resetting it. Seabios showed that it cannot send commands to it, but this is only hunch.
I am going back to virtio because I don't have time to debug this and my sata drive that I used to passthrough is slower than using virtio...

Here's my current config if it can help somehow:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=a63d1c94-4682-431c-9788-0cccd29ee598 rw intel_iommu=on pci-stub.ids=1002:6818,1002:aab0,8086:0 c0c,8086:153b pcie_acs_override=downstream sudo qemu-system-x86_64 -M q35 \ -m 8196 -enable-kvm -cpu host -monitor stdio \ -smp 4,sockets=1,cores=4,threads=1 \ -rtc clock=host,base=localtime \ -vga none -net none \ -parallel none -serial none -usb \ -drive cache=off,id=disk,file=/home/xani/old_home/xani/obrazy/windows/win-7.raw,format=raw \ -drive if=virtio,cache=off,id=diskc,file=/home/xani/old_home/xani/obrazy/windows/data.raw,format=raw \ -device ahci,bus=pcie.0,id=ahci \ -device ide-hd,bus=ahci.0,drive=disk \ -drive id=isocd,file=/home/xani/downloads/virtio-win-0.1-74.iso \ -device ide-cd,bus=ahci.1,drive=isocd \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -device vfio-pci,host=00:19.0,bus=pcie.0 \ -chardev file,id=seabios,path=/home/xani/seabios.log -device isa-debugcon,iobase=0x402,chardev=seabios

Last edited by xani (2013-12-26 16:16:17)

Offline

#941 2013-12-26 18:50:42

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

xani wrote:

Here's my directory from which you can test that memory leak issues.
linux-mainline.tar.gz
Provide me with feedback if you have the same problem with seabios after reverting those patches.

Hi and tyvm. I just used your linux-mainline-x and now the memory behaves nicely: i get it all back now upon closing the vm via qemu window. So i can confirm this fixed me. I just started using it and works great.
I will also try the second link asap, just holidays and etc. Speaking of which: Happy holidays to everybody smile

Offline

#942 2013-12-26 19:35:24

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@xani :

Ok so i'm not the only one to have this probleme, the funny fact is when you lauch a linux live cd and boot on it with the AsMedia 1062 you can access to your hard drive but you can reboot on it, but you have to wait some minute when Seabios hang (it finish to boot on CD)

So an other Sata controler ?

Offline

#943 2013-12-27 01:33:50

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just registered to say thank you to every one. I managed to use vfio to passthrough my nvidia 760 GTX.

My setup:
CPU: Xeon 1230V2
MotherBoard: Tyan S5512. It has an onboard video chip AST 2150 for IPMI, which turns out to be a cause of Code 43.

Host: Gentoo linux: kernel 3.12.6 with 3 patches from this link (which has been shown in this thread): https://lists.fedoraproject.org/piperma … 03916.html NoSnoop patch in top post is no longer necessary.
QEMU: 1.7 with 2 patches also provided in the link.

Note:
if you have an intel cpu with intel video gpu, you need to include the i915 video arbitration patches or you will have Code 43. If you don't want to patch then kernel 3.13 is the first working kernel...
But in my case it is something different, I don't think there is arbitration patch avaliable for AST chips. The only workaround is to set AST as primary video output in bios.

Note 2:
if you have intel CPU and want to use ATI card, I think it still needs some time to make them work with vfio. Black Screen is what I am facing right now. However, ATI cards works perfectly with PCI-STUB. But the last working build I have is qemu 1.5.3 with seabios 1.6.3. Any higher version results in a black screen.

Performance:
I just use windows index to check, everything looks normal except mem access. Qemu with Q35 simulation has some mem access regression. It is much slower than with 440fx:(5.5 vs 7.1). As a result, my 760 GTX doesn't play games faster than my 7790. 250 dollar is wasted...


I would say for ATI card users, stay with PCI-stub, you ready get the best performance. Everything is perfect except the reboot problem. By the way, My via usb 3 controller works flawless with 440fx emulation. With Q35 chipset, I always receive code 10...

For nvidia card users, well, no choices. I never make PCI-stub work for nvidia cards.

Last edited by doubledr (2013-12-27 01:35:07)

Offline

#944 2013-12-27 02:49:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuoni wrote:

EDIT: One thing I have noticed is there has been a marked decrease in my host (i915) graphics performance since using mainline-rc3 + i915 patch.  I'm presuming it's the kernel rather than the patch...?

This is why the i915 patches were reverted upstream, X wants to mmap the VGA mmio space and will disable DRI if the arbiter reports that more than one VGA device is present.  Therefore i915 has gone back to incorrectly reporting the VGA ranges the hardware decodes.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#945 2013-12-27 08:57:24

dismuter
Member
Registered: 2013-12-27
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying to get passthrough working for a Radeon on an Intel platform, but it seems that it just doesn't work right now (black screens or code 10 in windows).
If I may... Forums are nice to discuss subjects but if anyone is accustomed to creating wikis, one would be nice to summarize information for various cases. 38 pages is starting to be critical in terms of finding the right information, and it's never good to rely solely on the OP to update his post.

doubledr wrote:

if you have intel CPU and want to use ATI card, I think it still needs some time to make them work with vfio. Black Screen is what I am facing right now. However, ATI cards works perfectly with PCI-STUB. But the last working build I have is qemu 1.5.3 with seabios 1.6.3. Any higher version results in a black screen.
[...]
I would say for ATI card users, stay with PCI-stub, you ready get the best performance.

You mean legacy pci-assign? It's a shame that it doesn't work with the latest qemu+seabios then.

Offline

#946 2013-12-27 17:04:42

xj4reg
Member
Registered: 2013-12-27
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have two Radeon cards on a intel i7 machine, the vga pass through worked, but I meet BSoD on gust VM after install the newest ATI Catalyst driver. The BSoD told the error is on atikmpag.sys.
My HW configuration are - i7 3820, MSI x79a-gd45(8d), ATI Radeon 5400(for host),MSI Hawk ATI Radeon 6850(for gust windows).

I strictly follow the instructions, using the PKGBUILD to seabios, qemu and linux-mainline listed in page 1, but still run into this issue. Do you have any idea?

lspci outputs

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850]
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]

lspci -n

03:00.0 0300: 1002:6739
03:00.1 0403: 1002:aa88

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=a6859dd0-b64e-4c62-82ac-cbef89dce8ae rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream pci-stubs.ids=1002:6739,1002:aa88 quiet

vfio-bind.cfg

DEVICES="0000:03:00.0 0000:03:00.1"

qemu script

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-usb -usbdevice "host:7.4" -usbdevice "host:7.3" \
-net nic -net user \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/tianxj/temp/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/home/tianxj/temp/cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-boot menu=on

Last edited by xj4reg (2013-12-27 17:05:43)

Offline

#947 2013-12-27 17:59:23

techdude300
Member
Registered: 2012-04-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

xani wrote:

Here's my directory from which you can test that memory leak issues.
linux-mainline.tar.gz

Provide me with feedback if you have the same problem with seabios after reverting those patches.

Thanks so much! I can confirm that this does in fact seem to fix the issue. For some reason I was having trouble building it when i tried the first two times, but I got it to work on the 3rd try. All the memory the VM allocates is now returned back to the machine, so I no longer have to reboot the host. A very strange issue. Now all that's left on my end is to figure out sound.

Offline

#948 2013-12-28 03:46:37

sbd
Member
Registered: 2013-11-21
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

firewing1 wrote:

A.W. recommended using kernel 3.13rc + qemu.git OR kernel 3.12.5 + qemu 1.7.0 with the following commits patched in: https://lists.fedoraproject.org/piperma … 03916.html

I can confirm that the 3.12.5 kernel series will work with the method described above, if anyone would prefer not to use 3.13rc. I applied the patches to kernel 3.12.5-302 and 1.7.0-3 on Fedora 20 (qemu package is from rawhide), and I've installed the Catalyst drivers in the Windows 7 guest with no special options (control centre is included). It seems to be working normally.

I did use different i915 patches though, since the ones being linked in this thread are for the 3.13 series and don't compile on 3.12.5. I used the following instead:

81b5c7bc8de3e6f63419139c2fc91bf81dea8a7d
6e1b4fdad5157bb9e88777d525704aba24389bee

Thanks very much for all the advice guys, this thread has been great.

Offline

#949 2013-12-28 11:13:27

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The windows VM now works perfectly, except if i'm running the open source radeon driver on the host, my entire system freezes when windows boots. So if I blacklist radeon, windows boots perfectly but then I can't use Linux XD Anyone experienced this? :s

Primary GPU for Linux: AMD X1300 (R500 arch)
Secondary GPU for Windows: AMD R7950

root@josh-desktop ~]# dmesg | grep AMD-Vi [ 1.079338] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.079340] AMD-Vi: Interrupt remapping enabled [ 1.091501] AMD-Vi: Lazy IO/TLB flushing enabled [ 3.073444] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0019 address=0x0000000000009000 flags=0x0000] [ 3.073453] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0019 address=0x0000000000005000 flags=0x0020] [ 3.073461] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0019 address=0x0000000000009040 flags=0x0000] [ 3.073468] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0019 address=0x0000000000009080 flags=0x0000] [ 3.073474] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0019 address=0x00000000000090c0 flags=0x0000] [ 3.426471] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0019 address=0x0000000000005400 flags=0x0020]
[root@josh-desktop ~]# dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=7af60c5d-239a-4347-ae94-424f99fe644d rw vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:679a,1002:aaa0 quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=7af60c5d-239a-4347-ae94-424f99fe644d rw vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:679a,1002:aaa0 quiet [ 1.227222] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.227232] pci-stub 0000:09:00.0: claimed by stub [ 1.227236] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.227242] pci-stub 0000:09:00.1: claimed by stub
[root@josh-desktop ~]# lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) 00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E) 00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F) 00:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port G) 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link) 00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port B) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV516 [Radeon X1300/X1550 Series] 01:00.1 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] RV516 [Radeon X1300/X1550 Series] (Secondary) 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 03:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950] 09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

EDIT: Fixed it by updating BIOS, moving the 2nd CPU to a different PCI port, enabling a 64MB IOMMU mode in the BIOS, adding "iommu=pt iommu=1 amd_iommu=on" to kernel boot and compiling the kernel to have static modules. Not sure which one actually fixed it but it works! big_smile

By the way, the linux-mainline package provided in this topic doesn't have the kernel modules statically loaded by default. I had to change the config files manually and skip makepkg integrity check. I know you can enable them in the kernel config dialog but I can never find anything there. I don't understand why they wouldn't be enabled by default in the package since it's possible to do so?

Last edited by BulliteShot (2013-12-28 12:08:08)

Offline

#950 2013-12-28 13:32:11

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ xj4reg :
I have quite same setup, i7-3820 and dx79to (same x79 chipset), my primary is also ati (hd6570 with open source drivers), passing secondary gtx770. Perhaps not the best for this, but nonetheless i think it should work also for you.
About kernel, i had 'bus options \ pci-stub' built-in (not module) - if its module then see OP. Also ' device drivers \ iommu \ enable intel dma remmapping default ' built-in. And i did built in all virtio (except mem balooning). Perhaps not all this are necesarily like that but i am  just describing mine. For kernel, i had the memory issue with OP ones, so now i use the first link from xani (the link with fixmemory patch, not the second without). About config i used options kvm ignore_msrs=1 and about all the rest from OP. Also i am passing an entire usb controller instead of just devices, that is because i use a kvm switch. So i made the system service from OP and upon startup is good to check lspci -vvv and see kernel driver in use: vfio-pci for devices card, sound, and usb controller if passed like so. My sound works from guest to host with just the first two lines from OP, no other kinky quemu start params.

Good luck!

Last edited by noobman (2013-12-28 13:44:54)

Offline

#951 2013-12-28 13:55:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

By the way, the linux-mainline package provided in this topic doesn't have the kernel modules statically loaded by default. I had to change the config files manually and skip makepkg integrity check. I know you can enable them in the kernel config dialog but I can never find anything there. I don't understand why they wouldn't be enabled by default in the package since it's possible to do so?

They're built as modules, it shouldn't make any difference as far as i'm aware off

Last edited by nbhs (2013-12-28 13:58:44)

Offline

#952 2013-12-28 14:54:07

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

techdude300 wrote:
xani wrote:

Here's my directory from which you can test that memory leak issues.
linux-mainline.tar.gz

Provide me with feedback if you have the same problem with seabios after reverting those patches.

Thanks so much! I can confirm that this does in fact seem to fix the issue. For some reason I was having trouble building it when i tried the first two times, but I got it to work on the 3rd try. All the memory the VM allocates is now returned back to the machine, so I no longer have to reboot the host. A very strange issue. Now all that's left on my end is to figure out sound.

I have finally gotten round to testing this this morning and can confirm it also fixed the problem for me.  Thank you!

Offline

#953 2013-12-29 15:51:43

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,
I have another success story for you smile

I almost gave up on getting vga passthrough to work on my system (Intel DH87RL, i7-4771, Intel graphics + GTX 660) due to error 43, but I just found that a bios setting was the solution.
My mainboard has a "fast boot" option, which prevents the gpus from getting initialized by the bios and leaves that task to the operating system or bootloader. Previously both gpus would show the bios splash screen and the bootloader, now the only the internal graphics shows the bootloader and the linux console. I also had to use the romfile setting to get *any* output at all, which is not neccessary anymore. Now everything is running smooth and windows 8.1 works like a charm (well it's still a crappy os, but you know what I mean ;-)).
I think this setting would be worth mentioning on the first post, as it might solve the issue for others as well.

A few words about my setup (in the hope that it will be useful for others with similar hardware):
kernel cmdline: only

intel_iommu=on

is needed
qemu cmdline:

qemu-system-x86_64 -name windows8.1 -enable-kvm -watchdog i6300esb -boot menu=on -localtime -machine type=q35,accel=kvm -cpu host -smp 8,cores=4,threads=2,sockets=1 -m 6144 -k de -monitor tcp:127.0.0.1:5702,server,nowait -balloon virtio -vga none -display none -bios /root/vt-d/qemu-git/pc-bios/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -usb -device usb-host,vendorid=0x17ef,productid=0x6009 -netdev user,id=net.0 -device virtio-net-pci,netdev=net.0,mac=00:01:02:03:04:05 -drive id=disk1,file=/dev/sdd,if=none,index=0,media=disk -device virtio-blk-pci,drive=disk1,bootindex=1

software versions: qemu-1.7.50-git_e8092f7 and bundled seabios, linux 3.13-rc6 with i915 and memleak patches (3.12.6+patches did not work)

Now I can fine-tune my setup. Have you found a good solution to save power while the VM is shut down or not in use? I am running this on my 24/7 home server and having the nvidia card running without any power management permanently consumes about 20 Watts, which amounts to over 50€ electricity cost per year in my country.

Best regards,
Flyser

Last edited by Flyser (2014-01-03 23:44:57)

Offline

#954 2013-12-29 16:02:16

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuoni wrote:

Hi guys, I just registered to thank you all for your work - because of this thread, I have finally got KVM/QEMU VGA passthrough working on my GT660Ti.

My setup is i7-4770 / ASRock Z87 Extreme 4 / Nvidia GT660Ti, host OS is Debian x64, guest OS is Windows 7 x64, keyboard/mouse sharing with Synergy.

I had bought the processor/mobo combo specifically to try and do VGA passthrough and had been really struggling first with Xen and latterly with KVM until I found this thread - the information in the OP and the patches it links to have been invaluable.

Thanks once again, all.

Hi tuoni, could you please elaborate a little more on your software stack? Are you using Wheezy plus self-compiled kernel and qemu or Debian unstable or ... ?

Just thinking about buying a very similar set of gears as yours. smile

Offline

#955 2013-12-30 22:12:20

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jesse2004 wrote:
tuoni wrote:

Hi guys, I just registered to thank you all for your work - because of this thread, I have finally got KVM/QEMU VGA passthrough working on my GT660Ti.

My setup is i7-4770 / ASRock Z87 Extreme 4 / Nvidia GT660Ti, host OS is Debian x64, guest OS is Windows 7 x64, keyboard/mouse sharing with Synergy.

I had bought the processor/mobo combo specifically to try and do VGA passthrough and had been really struggling first with Xen and latterly with KVM until I found this thread - the information in the OP and the patches it links to have been invaluable.

Thanks once again, all.

Hi tuoni, could you please elaborate a little more on your software stack? Are you using Wheezy plus self-compiled kernel and qemu or Debian unstable or ... ?

Just thinking about buying a very similar set of gears as yours. smile

Sure - I'm using Jessie, self-compiled kernel (3.13-rc5 with gcc, i915, mem-leak and acs patches) and qemu from git pull (with vfio patch).  I'm pretty sure it would work just fine on Wheezy, though, as you need to build your own kernel and qemu anyway.

Offline

#956 2014-01-01 11:36:35

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I made a summary of the working hardware and software configurations mentioned in the thread:
Configs

I hope I got all of them right, but there might be some mistakes, feel free to correct them. I'm also going to try VGA passthrough with vfio in next days.

Offline

#957 2014-01-01 13:19:12

empie
Member
From: The Netherlands
Registered: 2013-06-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great job noctavian!

Offline

#958 2014-01-02 00:26:38

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

xani, I can confirm that your kernel also fixes the memory problems for me. All mem is free'd when qemu exits :-)

Offline

#959 2014-01-02 01:59:40

syllopsium
Member
Registered: 2013-12-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success! I haven't conducted a full cycle of tests yet, but it works with Windows 7 and 8 x64 once the NoSnoop patch is applied to aw's qemu-vfio.git. GPU BIOS file is required.I suspect normal qemu would be fine too. Some instability when fiddling around with configs. Please hold for a stable configuration..

The nosnoop patch is not required for a Linux guest to work correctly with passthrough. At the moment I'm using the qemu64 processor instead of 'host' inside qemu. Changing it to host creates bluescreens. This might change if installed from scratch - watch this space. I hadn't bothered applying the nosnoop patch before now as I thought it was only necessary for Nvidia cards.

Brief configuration : Debian Jessie (Arch worked for Linux passthrough too, but Arch is a little too fast moving a distro for my main system). Kernel 3.13.0-rc5 with ACS patch. qemu with nosnoop patch.
Hardware : Intel S3210SHLC server motherboard, passing through AMD HD6950 GPU. Host GPU : discrete Geforce 218.
Qemu config : qemu64 processor in use. It is absolutely necessary to supply the appropriate BIOS to the graphics card. Experimented with various ioh3420/PCIe root configurations, appears fine with both below the ioh.

The motherboard has a built in Matrox G200e, I'm hoping to pass this through to a different VM as it's a horribly slow chip. It appears in the same iommu group as the southbridge x4 PCIe slot (the only slot where a graphics card won't be slowed down to x1 speed by the 3210 chipset on this motherboard). This is why I've applied the ACS patch.

Now to experiment more and decide if I want to run a fairly full function Linux host with a Windows guest, or a barebones Linux host with passthrough for both Windows and Linux or BSD VMs..

Last edited by syllopsium (2014-01-02 02:00:31)

Offline

#960 2014-01-02 02:50:48

pascal257
Member
Registered: 2014-01-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Haha, I was about to post this, but I could fix the error by modprobe vfio_iommu_type1. So if someone has a similar problem, that might help.
But I would still like to know:
I've compiled the kernel without changing the config. Should I enable the VFIO config settings or is that not necessary with the patches?

Edit: And another question: I'd like to run a media center through the integrated graphics. Is that feasible? Can I pci-stub the integrated graphics from Arch and run it to a VM or are there any known problems?

First of all this thread has been a great read so far. I first started VGA-Passthrough with esxi and besides several other problems that didn't actually work.
Unfortunately I couldn't get it running with KVM either, but I'm certain that it'll work.

That's my hardware:
ASRock Z87 Extreme6
Intel i7 4770 (Host running on integrated graphics)
AMD Radeon R9 290 (For Windows Host)
NVidia GT 630 (For OSX Host)

At the moment I am working on getting the R9 290 to run.

On the software side:
Linux home 3.13.0-2-mainline - Compiled from first post with pci-stub.ids=1002:67b1,1002:aac8 intel_iommu=on pcie_acs_override=downstream
QEMU emulator version 1.7.50 - Compiled from first post

pci-stub is grabbing the card correctly:
[    0.920077] pci-stub: add 1002:67B1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.920083] pci-stub 0000:01:00.0: claimed by stub
[    0.920086] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.920090] pci-stub 0000:01:00.1: claimed by stub

But I'm getting this error running qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: No available IOMMU models
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 18
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 18
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Group 18 beeing the R9 290:
### Group 18 ###
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8

I've compiled the kernel without changing the config. Should I enable the VFIO config settings or is that not necessary with the patches?

Last edited by pascal257 (2014-01-02 02:57:09)

Offline

#961 2014-01-02 09:54:39

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

AUDIO EMULATION:

To hear the sound from the vm on your host speakers you'll need to add this lines:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

You might need to start qemu like this (using pulseaudio, to use a different driver change QEMU_AUDIO_DRV= variable):

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

With QEMU_PA_SAMPLES=128 the sound is horribly distorted here. Without this parameter it works fine though. @Op: What is the background of this setting?

Offline

#962 2014-01-02 13:07:22

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: Nevermind. I can't re-produce the problem.

With that patched kernel to stop the memory leak I can confirm it works unless I use hugetlbfs. Using hugetlbfs causes the memory leak again.

Last edited by BulliteShot (2014-01-05 01:42:10)

Offline

#963 2014-01-02 16:09:47

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:
nbhs wrote:

AUDIO EMULATION:

To hear the sound from the vm on your host speakers you'll need to add this lines:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

You might need to start qemu like this (using pulseaudio, to use a different driver change QEMU_AUDIO_DRV= variable):

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

With QEMU_PA_SAMPLES=128 the sound is horribly distorted here. Without this parameter it works fine though. @Op: What is the background of this setting?

Trial and error, im using alsa now instead of pulse now though

Offline

#964 2014-01-03 04:13:52

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: Upgrading qemu to the git version may have fixed this odd behavior -- but I still wonder how it was possible...
EDIT again: Nope, still broken -- it was the power cycle that made it work once.

Original post:

I've managed to get GPU passthrough to "work" on my system, but the behavior is quite odd.

Upon initial driver installation, the Radeon showed the desktop properly.
Unfortunately, at every guest boot thereafter (even after host reboot), the desktop starts with only the cursor visible.

If I start Narrator (win-R, 'narrator'), I can navigate the desktop blindly, and it seems like the desktop is working "perfectly".

Hardware:
* Xeon E3-1245v3
* SuperMicro X10SAT
* XFX HD 5750
* Asus Xonar DX (with EEPROM tweaked to call it a Xonar D1).
Video card was grabbed from e-waste pile at work; unknown condition, but works quite well in Windows native boot.

Software:
* Ubuntu 13.10
* Qemu 1.7.0 (Debian 1.7.0+dfsg-2ubuntu3)
* Kernel 3.13 + acs-override patch (set to IDs of Intel root ports). 
* Guest is Windows 8.1, 64-bit.  (Video drivers won't load with Windows 7.)

Extra notes:
* VM behaves approximately the same both in Q35 mode and in PIIX4 mode (two different VMs, same virtual disk).
* Host is booted in EFI boot mode, using Intel graphics.  This hopefully helps avoid some Intel VGA register usage.
* Guest is given GPU via VFIO, but primary VGA is Cirrus.
* I'd use EFI mode for the guest if I had a supporting video card, and if qemu didn't hang when attempting EFI mode...

The group I'm passing through is Group 1. 
I tweaked 'lsgroup.sh' to use lspci -nn (names and numbers) instead of just lspci:

### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] 02:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) 03:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 10 ### 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0a:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01) 0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01) ### Group 11 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 12 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) ### Group 13 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 14 ### 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) ### Group 15 ### 04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 16 ### 0d:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) root@server-linux:~# nano -w lsgroup.sh root@server-linux:~# ./lsgroup.sh ### Group 0 ### 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) ### Group 1 ### 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58] 02:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa) 03:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] ### Group 2 ### 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) ### Group 3 ### 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) ### Group 4 ### 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) ### Group 5 ### 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) 00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller [8086:8c3d] (rev 04) ### Group 6 ### 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 05) ### Group 8 ### 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) ### Group 9 ### 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) ### Group 10 ### 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) 0a:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) 0b:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01) 0c:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] [104c:823f] (rev 01) ### Group 11 ### 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) ### Group 12 ### 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) ### Group 13 ### 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) ### Group 14 ### 00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller [8086:8c56] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05) ### Group 15 ### 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) ### Group 16 ### 0d:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

Last edited by DanaGoyette (2014-01-03 04:42:09)

Offline

#965 2014-01-03 07:57:35

ayc
Member
Registered: 2014-01-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have VGA pass-through of an ATI card working fine. For another VM I'm trying to pass through an Intel C606 SAS/SATA controller which isn't going very well.

1) pci-stub grabs the controller on boot and vfio binds to it
2) pci-assign passthrough to an Ubuntu Linux works fine
3) vfio-pci to the same Ubuntu Linux fails with I/O errors from the driver
4) device in the guest is bound to a ioh3420 bridge in qemu
5) Neither pci-assign nor vfio-pci work under FreeBSD, which is the eventual goal for this VM

The device has FLR, so I think should work fine. From the host:

[root@vs ~]# lspci -vvvs 03:00.0 03:00.0 Serial Attached SCSI controller: Intel Corporation C606 chipset Dual 4-Port SATA/SAS Storage Control Unit (rev 06) Subsystem: Super Micro Computer Inc Device 0630 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 26 Region 0: Memory at d18f8000 (64-bit, prefetchable) [size=32K] Region 2: Memory at d1000000 (64-bit, prefetchable) [size=8M] Region 4: I/O ports at 8100 [size=256] Region 5: I/O ports at 8000 [size=256] Capabilities: [98] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [c4] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ [snip]

In looking at the differences between dmesg on pci-assign vs vfio-pci, there's this when
using vfio.

pci 0000:01:00.0: no compatible bridge window for [mem 0xd1800000-0xe0ffffff 64bit pref

and later on

PCI: max bus depth: 1 pci_try_num: 2 pci 0000:01:00.0: reg 164: [mem 0xd1800000-0xd1ffffff 64bit pref pci 0000:00:1c.0: BAR 14: assigned [mem 0xc0000000-0xcf7fffff pci 0000:01:00.0: reg 164: [mem 0xd1800000-0xd1ffffff 64bit pref pci 0000:01:00.0: reg 164: [mem 0xd1800000-0xd1ffffff 64bit pref pci 0000:01:00.0: BAR 7: assigned [mem 0xc0000000-0xcf7fffff 64bit pref pci 0000:01:00.0: BAR 7: error updating (0xc000000c != 0xd180000c) pci 0000:00:1c.0: PCI bridge to [bus 01-01 pci 0000:00:1c.0: bridge window [io 0xc000-0xcfff pci 0000:00:1c.0: bridge window [mem 0xc0000000-0xcf7fffff pci 0000:00:1c.0: bridge window [mem 0xfc000000-0xfcffffff 64bit pref pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 pci_bus 0000:00: resource 5 [io 0x0d00-0xffff pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff pci_bus 0000:01: resource 0 [io 0xc000-0xcfff pci_bus 0000:01: resource 1 [mem 0xc0000000-0xcf7fffff pci_bus 0000:01: resource 2 [mem 0xfc000000-0xfcffffff 64bit pref

vs doing a pci_assign:

pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff pci 0000:00:1c.0: res[14 pci 0000:00:1c.0: BAR 14: assigned [mem 0xc0000000-0xc03fffff pci 0000:00:1c.0: PCI bridge to [bus 01-01 pci 0000:00:1c.0: bridge window [io 0xc000-0xcfff Freeing initrd memory: 15736k freed pci 0000:00:1c.0: bridge window [mem 0xc0000000-0xc03fffff pci 0000:00:1c.0: bridge window [mem 0xfc000000-0xfcffffff 64bit pref pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 pci_bus 0000:00: resource 5 [io 0x0d00-0xffff pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff pci_bus 0000:01: resource 0 [io 0xc000-0xcfff pci_bus 0000:01: resource 1 [mem 0xc0000000-0xc03fffff pci_bus 0000:01: resource 2 [mem 0xfc000000-0xfcffffff 64bit pref

When I don't put this behind a bridge, the only difference is that when using vfio, I get this extra line:

 pci 0000:00:02.0: reg 164: [mem 0xd1800000-0xd1ffffff 64bit pref

which is not in there when using pci-assign.

Any thoughts on further debugging this?

   ...alan

Offline

#966 2014-01-03 19:18:46

syllopsium
Member
Registered: 2013-12-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's the parent device for the C606 SAS controller on the host? Looking at the chipset diagram it seems to be hanging directly off the IOH?

It may be obvious, but have you tried passing through the controller without specifying the point to hang it off? i.e. device=vfiio-pci,host=nn:nn.n,id=<friendlyname>

Offline

#967 2014-01-03 20:43:37

ayc
Member
Registered: 2014-01-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syllopsium wrote:

What's the parent device for the C606 SAS controller on the host? Looking at the chipset diagram it seems to be hanging directly off the IOH?

It's a bit weird. From Intel's data sheet:

The Intel® C606, C608 Chipset SKUs contain a PCI Express* switch. The Intel® C606, C608 Chipset SKUs contains a x4 uplink and a “virtual” switch port that serves as the connection for the integrated MFD shown in Figure 5-2. Software will discover a virtual switch port with an attached multi-function end point device. The PCI Express* switch is compliant to the PCI Express Base specification 2.0.

and earlier it gives a rationale as to why they're doing this:

The purpose of the uplink port is to provide a direct path for the SAS controllers, SGPIO used by the SAS controllers, and SMBus ports to the processor/ memory without having to be multiplexed onto the DMI bus and sharing bandwidth with the rest of the component. The uplink port is not connected to the downstream root ports

lspci gives:

 +-01.0-[01-03]----00.0-[02-03]----08.0-[03]----00.0 Intel Corporation C606 chipset Dual 4-Port SATA/SAS Storage Control Unit

so, it's the 0000:03:00 device that i'm passing through. this is not a straight-forward device passthrough, so i'm not too surprised i'm having troubles with it.

It may be obvious, but have you tried passing through the controller without specifying the point to hang it off? i.e. device=vfiio-pci,host=nn:nn.n,id=<friendlyname>

yes, no go.

Forgot to say, just upgraded to the mainline from the first post, kernel 3.13-rc5.

   ...alan

Offline

#968 2014-01-04 08:05:13

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@xani: Do you know exactly the root cause of the memory leak and how you fixed it in your kernel? Or are you unsure on the exact reason? It would be awesome if we get this fixed also in the main kernel sources :-) Or are you already in contact with kernel devs about this topic?

Last edited by TripleSpeeder (2014-01-04 08:05:28)

Offline

#969 2014-01-04 08:09:30

sbd
Member
Registered: 2013-11-21
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

Upon initial driver installation, the Radeon showed the desktop properly.
Unfortunately, at every guest boot thereafter (even after host reboot), the desktop starts with only the cursor visible.

If I start Narrator (win-R, 'narrator'), I can navigate the desktop blindly, and it seems like the desktop is working "perfectly".

Probably because you have the Cirrus VGA card as the primary. That's being treated as the primary monitor, so all the desktop content is showing up there and not on the Radeon.

Offline

#970 2014-01-04 10:40:22

peaquino
Member
Registered: 2013-12-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'd like to report sucess with my Radeon passthrough! It works flawlessly, but only on a second guest launch.

On first launch after I get screen corruption on any 3D operation in guest (even showing a search box in Chrome) and the guest becomes unusable with constant driver resets to a point, when it grinds to a halt. At that moment I get about ~1K of log entries similar to

sty 04 11:12:57 miner kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x000000007bc4c100 flags=0x0010] sty 04 11:12:57 miner kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 

When I abort first launch of the quest with a monitor quit command during during guest BIOS initialisation, on a second launch I get the following errors in Qemu monitor:

qemu-system-x86_64: vfio_dma_map(0x7f4b93fbbd40, 0x0, 0xb0000000, 0x2aaac0000000) = -16 (Device or resource busy) qemu-system-x86_64: vfio_dma_map(0x7f4b93fbbd40, 0xc0000, 0xaff40000, 0x2aaac00c0000) = -16 (Device or resource busy) qemu-system-x86_64: vfio_dma_map(0x7f4b93fbbd40, 0xc8000, 0xaff38000, 0x2aaac00c8000) = -16 (Device or resource busy) qemu-system-x86_64: vfio_dma_map(0x7f4b93fbbd40, 0xd0000, 0xaff30000, 0x2aaac00d0000) = -16 (Device or resource busy) 

and the following entries in the kernel log

sty 01 22:04:53 miner kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 sty 01 22:04:55 miner kernel: ------------[ cut here ]------------ sty 01 22:04:55 miner kernel: WARNING: CPU: 3 PID: 1012 at drivers/vfio/vfio_iommu_type1.c:685 vfio_dma_do_map+0x43c/0x838 [vfio_iommu_type1]() sty 01 22:04:55 miner kernel: Modules linked in: tun bridge stp llc bnep vfio_pci vfio_iommu_type1 vfio fuse ts2020 ds3000 dvb_usb_dw2102 kvm_amd kvm crc32_pclmul c...luetooth rc_ sty 01 22:04:55 miner kernel: CPU: 3 PID: 1012 Comm: qemu-system-x86 Not tainted 3.13.0-2-mainline #1 sty 01 22:04:55 miner kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./FM2A85X Extreme6, BIOS P2.30 07/11/2013 sty 01 22:04:55 miner kernel: 0000000000000009 ffff8804311b9d20 ffffffff814ddc72 0000000000000000 sty 01 22:04:55 miner kernel: ffff8804311b9d58 ffffffff8105f2ed 00000000fffffff0 00000000b0000000 sty 01 22:04:55 miner kernel: 00000000000b0000 00000000b0000000 0000000000000000 ffff8804311b9d68 sty 01 22:04:55 miner kernel: Call Trace: sty 01 22:04:55 miner kernel: [<ffffffff814ddc72>] dump_stack+0x4d/0x6f sty 01 22:04:55 miner kernel: [<ffffffff8105f2ed>] warn_slowpath_common+0x7d/0xa0 sty 01 22:04:55 miner kernel: [<ffffffff8105f3ca>] warn_slowpath_null+0x1a/0x20 sty 01 22:04:55 miner kernel: [<ffffffffa0ce9dec>] vfio_dma_do_map+0x43c/0x838 [vfio_iommu_type1] sty 01 22:04:55 miner kernel: [<ffffffffa0cea3f9>] vfio_iommu_type1_ioctl+0x211/0x288 [vfio_iommu_type1] sty 01 22:04:55 miner kernel: [<ffffffffa0ce0e28>] vfio_fops_unl_ioctl+0x78/0x340 [vfio] sty 01 22:04:55 miner kernel: [<ffffffff811adfb8>] do_vfs_ioctl+0x2d8/0x4b0 sty 01 22:04:55 miner kernel: [<ffffffff811ae211>] SyS_ioctl+0x81/0xa0 sty 01 22:04:55 miner kernel: [<ffffffff814ebd2d>] system_call_fastpath+0x1a/0x1f sty 01 22:04:55 miner kernel: ---[ end trace 53ac540a2e8783dc ]--- sty 01 22:04:56 miner kernel: ------------[ cut here ]------------ 

but the guest works flawlessly (even with reboots) with Catalyst drivers (Steam games work as expected).

Currently I'm using qemu-git package from AUR and linux-mainline from first page of this thread.

I hope this helps people struggling with Radeon passthrough.

Here is a link to a Qemu bug report: https://bugs.launchpad.net/qemu/+bug/1265998

Offline

#971 2014-01-04 18:22:33

MCon
Member
Registered: 2014-01-04
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm actually trying to follo this nice guide on a Debian Wheezy install.
I rebuilt a custom kernel:

$ uname -a Linux black 3.13.0-rc6-customkernel #2 SMP Sat Jan 4 16:21:06 CET 2014 x86_64 GNU/Linux

VIRTIO is currently compiled into the kernel (no modules).

I also compiled qemu-git (NOT using Your scripts, obviously tied to ArchLinux, but a "normal" .configure -> make -> sudo make install cycle; here may be the problem!) and openqrm-community (again with plain make -> sudo make install).

Launching qemu-system-x86_64 ... (it's in a script) I see:

$ sudo ./start VNC server running on `::1:5900'

Nothing happens on the GTX770 monitor and connecting to VNC I see:

compat_monitor0 console QEMU 1.7.50 monitor - type 'help' for more information (qemu)

Before I send my whole config is there something I should cross-check?
I'm pretty sure I gooofed somewhere badly.

Here comes config:

$ lspci 00:00.0 Host bridge: Intel Corporation Haswell DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Haswell PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Haswell PCI Express x8 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Haswell Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Haswell HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation Lynx Point MEI Controller #1 (rev 04) 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) 00:1a.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation Lynx Point High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation Lynx Point PCI Express Root Port #1 (rev d4) 00:1c.1 PCI bridge: Intel Corporation Lynx Point PCI Express Root Port #2 (rev d4) 00:1c.5 PCI bridge: Intel Corporation Lynx Point PCI Express Root Port #6 (rev d4) 00:1d.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Lynx Point LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation Lynx Point 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation Lynx Point SMBus Controller (rev 04) 02:00.0 VGA compatible controller: NVIDIA Corporation Device 1184 (rev a1) 02:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) 04:00.0 Network controller: Atheros Communications Inc. AR9462 Wireless Network Adapter (rev 01) 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) $ lspci -n | grep ^02 02:00.0 0300: 10de:1184 (rev a1) 02:00.1 0403: 10de:0e0a (rev a1) $ cat bind.sh sudo vfio-bind 0000:02:00.0 0000:02:00.1 $ cat start qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 4,sockets=1,cores=2,threads=2 \ -bios /usr/local/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Thanks in advance
Mauro

Offline

#972 2014-01-05 01:40:21

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone got any working packages for pci-stub only? q35 performance is pretty poor for memory demanding games.

Offline

#973 2014-01-05 02:51:46

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I have some new information and I do not know who to share.

The probleme i found for pci passthougr of sata AsMedia 1062 :

[ 404.206866] dmar: DRHD: handling fault status reg 2 [ 404.206870] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr 2affdf000 [ 404.206870] DMAR:[fault reason 05] PTE Write access is not set [ 404.206877] dmar: DRHD: handling fault status reg 2 [ 404.206879] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 2affda000 [ 404.206879] DMAR:[fault reason 06] PTE Read access is not set

Is the same for Intel Z87 sata.

I can't use the Intel Sata with vfio because of ACS, ACS patch does not work for Intel Sata [00:1f.2] and when i use with pci_stub it's work only one time, when the VM reboot i have this error :

[ 1801.280862] dmar: DRHD: handling fault status reg 2 [ 1801.280866] dmar: DMAR:[DMA Read] Request device [00:1f.2] fault addr 2affdd000 [ 1801.280866] DMAR:[fault reason 06] PTE Read access is not set

Where i can report this error in the hope of any boby correct this ?

Thanks.

Offline

#974 2014-01-05 05:21:30

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

I can't use the Intel Sata with vfio because of ACS, ACS patch does not work for Intel Sata [00:1f.2]...

If you haven't already done so, try passing through the other devices in that group, such as:
    00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller [8086:8c56] (rev 05)
    00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
    00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
    00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05)

Alternately, these devices should be overridable via 'multifunction', or better, via device ID (in my case, it's id:8086:8c02).


sbd wrote:

Probably because you have the Cirrus VGA card as the primary. That's being treated as the primary monitor, so all the desktop content is showing up there and not on the Radeon.

I actually used Narrator to get the virtual machine's desktop to appear only on the ATI card, and not at all on the Cirrus.  The ATI desktop was alive and "functional", but totally black.

I've rebuilt Qemu from git, and now it seems to be defaulting to pci-assign instead of VFIO.  This works fine, aside from needing the "eject" workaround (which kicks the GPU into high-speed-fan mode).
In fact, now I'm thinking this may have something to do with that black desktop -- perhaps the black desktop is a result of NOT ejecting.
Time for more experimenting...

EDIT:
Okay, I tried switching back to vfio.  No dice:
genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (ehci_hcd:usb1)

Adding the EHCI controller (#2, oddly enough, not #1 as 'usb1' would seem to imply), seems to perhaps fix this.

If I add my sound card to the mix:
genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1))

How the heck is '00000000' a mismatch with '00000000'?

Last edited by DanaGoyette (2014-01-05 05:58:32)

Offline

#975 2014-01-05 13:52:34

cataphract
Member
Registered: 2013-08-06
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried this with Linux 3.12.6-1-ARCH, qemu 1.7.0-1 and seabios 1.7.3.1-2 without success.

I have this motherboard:

[ 0.000000] DMI: Gigabyte Technology Co., Ltd. Z87MX-D3H/Z87MX-D3H-CF, BIOS F5 08/02/2013

a 4770 and Radeon 6950. Main display is connected to the IGP.

$ dmesg | grep pci-stub [ 0.000000] Command line: root=UUID=9dac92cf-60d1-4a6d-b9a4-066bb876c9ce ro intel_iommu=on modprobe.blacklist=radeon pci-stub.ids=1002:6719,1002:aa80 initrd=\initramfs-linux.img [ 0.000000] Kernel command line: root=UUID=9dac92cf-60d1-4a6d-b9a4-066bb876c9ce ro intel_iommu=on modprobe.blacklist=radeon pci-stub.ids=1002:6719,1002:aa80 initrd=\initramfs-linux.img [ 1.289177] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.289193] pci-stub 0000:01:00.0: claimed by stub [ 1.289197] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.289201] pci-stub 0000:01:00.1: claimed by stub

The vfio-bind also works fine. I then tried this command line to run qemu:

qemu-system-x86_64 \ -enable-kvm \ -m 4096 \ -drive file=win7.qcow2,if=ide \ -drive file=/mnt/evo/Arquivo/Windows7_Pro_SP1_English_x64.iso,if=ide,media=cdrom \ -bios /usr/share/qemu/bios.bin \ -M q35 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=XFX.HD6950.1024.110116_1.rom \ -device vfio-pci,host=01:00.1,addr=0.1,bus=root.1 \ -boot menu=on \ -cpu host \ -smp 3 \ -nographic \ -vga none

but I see nothing on the second display (connected via HDMI).

For some reason, I cannot boot linux-mainline or the modified linux-mainline posted on the front page. It can't find the root partition and I'm dropped to a shell where even my usb keyboard doesn't work. It's strange because I never had any problem with official kernel. I'm using refind.

Offline

#976 2014-01-05 14:27:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
Val532 wrote:

I can't use the Intel Sata with vfio because of ACS, ACS patch does not work for Intel Sata [00:1f.2]...

If you haven't already done so, try passing through the other devices in that group, such as:
    00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller [8086:8c56] (rev 05)
    00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
    00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
    00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05)

Does it really seem like a good idea to pass these other devices through to a guest?  I hope not.

Alternately, these devices should be overridable via 'multifunction', or better, via device ID (in my case, it's id:8086:8c02).


sbd wrote:

Probably because you have the Cirrus VGA card as the primary. That's being treated as the primary monitor, so all the desktop content is showing up there and not on the Radeon.

I actually used Narrator to get the virtual machine's desktop to appear only on the ATI card, and not at all on the Cirrus.  The ATI desktop was alive and "functional", but totally black.

I've rebuilt Qemu from git, and now it seems to be defaulting to pci-assign instead of VFIO.  This works fine, aside from needing the "eject" workaround (which kicks the GPU into high-speed-fan mode).
In fact, now I'm thinking this may have something to do with that black desktop -- perhaps the black desktop is a result of NOT ejecting.
Time for more experimenting...

EDIT:
Okay, I tried switching back to vfio.  No dice:
genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (ehci_hcd:usb1)

Adding the EHCI controller (#2, oddly enough, not #1 as 'usb1' would seem to imply), seems to perhaps fix this.

If I add my sound card to the mix:
genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1))

How the heck is '00000000' a mismatch with '00000000'?

Because neither of them include IRQF_SHARED and therefore require exclusive access to the interrupt line.  This means that neither devices supports PCI2.3 INTx disable.  You can generally fix this, but lose access to the conflicting device, by detaching the device from the host driver.

Last edited by aw (2014-01-05 15:29:05)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#977 2014-01-05 14:30:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cataphract wrote:

I tried this with Linux 3.12.6-1-ARCH, qemu 1.7.0-1 and seabios 1.7.3.1-2 without success.

That's not very surprising if your kernel doesn't include the i915 fixes and neither kernel nor qemu include the coherency fixes included by the kvm-vfio device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#978 2014-01-05 14:41:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Hi all,

I have some new information and I do not know who to share.

The probleme i found for pci passthougr of sata AsMedia 1062 :

[ 404.206866] dmar: DRHD: handling fault status reg 2 [ 404.206870] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr 2affdf000 [ 404.206870] DMAR:[fault reason 05] PTE Write access is not set [ 404.206877] dmar: DRHD: handling fault status reg 2 [ 404.206879] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 2affda000 [ 404.206879] DMAR:[fault reason 06] PTE Read access is not set

Is the same for Intel Z87 sata.

I can't use the Intel Sata with vfio because of ACS, ACS patch does not work for Intel Sata [00:1f.2] and when i use with pci_stub it's work only one time, when the VM reboot i have this error :

[ 1801.280862] dmar: DRHD: handling fault status reg 2 [ 1801.280866] dmar: DMAR:[DMA Read] Request device [00:1f.2] fault addr 2affdd000 [ 1801.280866] DMAR:[fault reason 06] PTE Read access is not set

Where i can report this error in the hope of any boby correct this ?

Thanks.

You can run 'info mtree' from the qemu monitor to check whether those fault addresses are actually something that should be mapped to the device.  There's a reasonable chance that the device just does a stray DMA access, in which case it's broken.  If it only works once and the device does not support a standard reset then it needs a device-specific reset.  You might be able to figure out how to do that by looking through the datasheet for the device.  There's no guarantee that such a mechanism exists though.  The other question is whether it's really worthwhile to assign the SATA device.  I'd guess that you're not going to get a massive performance increase by assigning it versus using virtio-block with a raw partition, disk or lvm volume.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#979 2014-01-05 14:47:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

Has anyone got any working packages for pci-stub only? q35 performance is pretty poor for memory demanding games.

pci-stub is a kernel module that does nothing but sit on a device to prevent other drivers from claiming it.  q35 is a qemu chipset model.  I think you're asking if anyone has this working with the qemu 440fx chipset model, which can still use vfio-pci.  It's possible, try it an see.  q35 is more similar in architecture to today's PCs, so it's more like how drivers expect to see the hardware.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#980 2014-01-05 14:56:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

Now I can fine-tune my setup. Have you found a good solution to save power while the VM is shut down or not in use? I am running this on my 24/7 home server and having the nvidia card running without any power management permanently consumes about 20 Watts, which amounts to over 50€ electricity cost per year in my country.

We could think about making vfio-pci put devices into the D3 power state when not opened by a userspace driver like qemu, but I worry that it might cause more problems than it solves and there's also a good chance that it won't make a desktop graphics card consume any less power.  You can use setpci to experiment whether D3 actually saves you any power.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#981 2014-01-05 15:03:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syllopsium wrote:

The motherboard has a built in Matrox G200e, I'm hoping to pass this through to a different VM as it's a horribly slow chip. It appears in the same iommu group as the southbridge x4 PCIe slot (the only slot where a graphics card won't be slowed down to x1 speed by the 3210 chipset on this motherboard). This is why I've applied the ACS patch.

FWIW, nvidia/amd require several quirks in qemu to make things work.  Nobody has done these for matrox and I don't have anything with matrox graphics.  Besides that, what's the point of attaching a matrox card to a guest?  It seems like the headache of assigning it is not worth the incremental improvement versus emulated graphics.  Assignment makes sense for high end GPUs, but I'm not sure I'm interested in maintaining code for assigning random 2d-only VGA devices.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#982 2014-01-05 15:05:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pascal257 wrote:

Edit: And another question: I'd like to run a media center through the integrated graphics. Is that feasible? Can I pci-stub the integrated graphics from Arch and run it to a VM or are there any known problems?

There are numerous problem with assigning IGD.  Please don't try it unless you plan to contribute to development.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#983 2014-01-05 17:52:43

cataphract
Member
Registered: 2013-08-06
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
cataphract wrote:

I tried this with Linux 3.12.6-1-ARCH, qemu 1.7.0-1 and seabios 1.7.3.1-2 without success.

That's not very surprising if your kernel doesn't include the i915 fixes and neither kernel nor qemu include the coherency fixes included by the kvm-vfio device.

OK, I tried with the linux package in the first post (I'd tried before but I couldn't boot it and gave up -- turns out I was loading the wrong initramfs), qemu-git and seabios-git and I was able to get the windows installer displayed on the 6950.

There were some issues with qemu though. First it wouldn't build (error in /usr/include/sys/xattr.h; it seems XATTR_CREATE was already defined when this was included) and then I'd get this error:

WARNING: failed to find q35-acpi-dsdt.aml

I solved this by copying the file into /usr/share/qemu.

Offline

#984 2014-01-05 18:44:03

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You can run 'info mtree' from the qemu monitor to check whether those fault addresses are actually something that should be mapped to the device.  There's a reasonable chance that the device just does a stray DMA access, in which case it's broken.  If it only works once and the device does not support a standard reset then it needs a device-specific reset.  You might be able to figure out how to do that by looking through the datasheet for the device.  There's no guarantee that such a mechanism exists though.  The other question is whether it's really worthwhile to assign the SATA device.  I'd guess that you're not going to get a massive performance increase by assigning it versus using virtio-block with a raw partition, disk or lvm volume.

I don't know how use "info mtree" and when ^^.

multifonction for acs overdrive does not work for intel sata, and yes is not a good idea to pass the entry bloc to the VM, and the real probleme, is it work with pci_stub (but only once) and not with vfio-pci (vfio-pci say the group is not blah blah).

And why use sata controler instead of virtio-scsi it's because i use a SSD ^^. And i don't know if the trim fuction work trough the virtio.


But it's funy to see the error is the same on Intel chipset, for Intel Sata or AsMedia, do anybody can test with a AMD motherboard ?

Offline

#985 2014-01-05 21:09:21

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DanaGoyette wrote:

If I add my sound card to the mix:
genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000000 (vfio-intx(0000:01:00.1))

How the heck is '00000000' a mismatch with '00000000'?

Because neither of them include IRQF_SHARED and therefore require exclusive access to the interrupt line.  This means that neither devices supports PCI2.3 INTx disable.  You can generally fix this, but lose access to the conflicting device, by detaching the device from the host driver.

Hmm, perhaps that message could use some tweaking.
Also, both devices function when both are attached to the host, and I'm passing both to the SAME virtual machine!
CORRECTION: If snd-virtuoso and radeon are both loaded, then the system hard-locks when trying to start X. 
With just 'radeon' loaded, no lockup occurs.  Maybe that's precisely the problem -- the shared interrupt!  I wonder why the kernel doesn't warn about this...

I'm trying to pass through (to the same VM) the following, at the moment:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58] -- if I remove this, the mismatch instead cites 01:00.0. 07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] -- this lies under 06:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa) 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) 0c:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

I can see several ways to work around this, but none seems ideal:
* Use KVM built-in audio: only stereo; I'd like 5.1 surround.
* Pass the onboard Realtek to the VM, and keep the Xonar sound card for the host: last time I tried that, the Realtek produced horribly broken audio.
* Move to second CPU slot: steals 8 lanes from primary GPU, despite only actually needing 1 lane.  Also leaves only 4 lanes for a possible second GPU.
* Try another x1 slot: the x1 slot below second CPU slot conflicts with the Serial-over-LAN port, which I am definitely using; the x1 slot below the first GPU slot, I'd need a ribbon riser to try, due to dual-slot video card.
* Get a different sound card and/or GPU:  I can't tell before purchase which ones will likely work, and the same issues about slots will apply.
* Use a USB sound card?  I tried with the Xonar sound card either unbound or physically removed, and now the Renesas USB controller reports the same "00000000" conflict.
* Try the X10SAE motherboard I also have: would lose the Thunderbolt port, but more importantly, it has this bug: https://bugzilla.kernel.org/show_bug.cgi?id=44881

Last edited by DanaGoyette (2014-01-05 22:39:57)

Offline

#986 2014-01-05 23:32:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

* Try the X10SAE motherboard I also have: would lose the Thunderbolt port, but more importantly, it has this bug: https://bugzilla.kernel.org/show_bug.cgi?id=44881

This is due to the asmedia PCIe-to-PCI bridge, it fails to expose a PCIe capability in direct violation of the PCIe spec.  It's not a trivial problem to solve, which is why it's been around so long.  Personally, I just avoid anything with an asmedia bridge.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#987 2014-01-06 00:04:32

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DanaGoyette wrote:

* Try the X10SAE motherboard I also have: would lose the Thunderbolt port, but more importantly, it has this bug: https://bugzilla.kernel.org/show_bug.cgi?id=44881

This is due to the asmedia PCIe-to-PCI bridge, it fails to expose a PCIe capability in direct violation of the PCIe spec.  It's not a trivial problem to solve, which is why it's been around so long.  Personally, I just avoid anything with an asmedia bridge.

That makes it sound like a PCI-PCI bridge sitting directly on a PCIe bus -- so yeah, that's broken.

Incidentally, the markings on the chip closest to the PCI slots are:
IDT ©
89HMPEB38
3ZBEMG
ZB1242CH

CH09493MX TWP BU

Last edited by DanaGoyette (2014-01-06 00:11:57)

Offline

#988 2014-01-06 00:30:18

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've gone back and re-tested a bunch of the slots.  With pci-assign, I can use the video card in the top x16 and the sound card in the middle x16, just fine.  With vfio, even that configuration fails due to interrupt flag conflicts.  I can add or remove the firewire controller and the ethernet controller, and I'll get varying collisions between the devices I noted above, and sometimes with the Intel chipset KT (serial-over-lan) or SMBUS controller. 
I'm going to try re-adding all those assignments with the sound card in the bottom slot, using pci-assign again.

EDIT: Okay, with sound card back in an x1 slot, and the Ethernet controller included in the VM, I get a kernel oops.  Without ethernet controller, I'm noticing that the "genirq" message mentions IRQs that aren't actually shared in lspci.  For example, genirq will mention the hdmi audio and the C-Media card as conflicting on IRQ 17 -- yet, lspci -nnvv mentions only the C-Media card as using IRQ 17.

EDIT #2:  With sound card in x1 slot, VM boots if I don't bother forwarding HDMI audio, even when including the ethernet controller.  Again, this is still with pci-assign, not vfio.

It looks like vfio is protecting me from certain conflicts -- but unfortunately, all that protection must be somewhat over-zealous.

Also, in case you're wondering why I'm passing the USB controller through, it's because the USB passthrough in Qemu wasn't working for me when I tried it.  Devices stayed attached to the host, not the guest.

Last edited by DanaGoyette (2014-01-06 00:59:48)

Offline

#989 2014-01-07 18:03:22

MichaelLong
Member
Registered: 2014-01-06
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,
thx a lot for this great thread, it helped me a lot. So I'd like to report another success story on the following hardware and software:

Hardware
Mainboard: Intel DP67BG
CPU: Intel i7-2600
GPU0 (host): PowerColor Go! Green HD5450 512MB PCIEx1
GPU1 (guest): Gainward GeForce GTX 560 Ti 2048MB
RAM: 8 GB

Software
Kernel: 3.13-rc7
QEMU: qemu-git

What works:
Using the AMD PCIe x1 card makes it convenient to avoid any ACS patches at the moment. For my tests I'm using a clone of my native Windows 7 gaming installation so it is quite easy to make comparisons. The VM runs without any general problems so far. I ran several benchmarks like Cinebench R15, 3DMark (2013), Unigine Heaven 4.0, PassMark. Performance ranges from about 85% (Cinemax) to even 99% (Unigine Heaven) compared to the native installation.

What does not work:
I can run the QEMU instance only once between a power cycle of the host system. Rebooting the VM is not a problem but shutting it down and starting it later results in:

vfio-pci 0000:01:00.0: Invalid ROM contents

Supplying a VBIOS to QEMU makes the message go away but does not change the result. QEMU hangs. This might not be any news at all but I'm not aware of the current state.

What is a problem:
My only remaining issue right now is a quite specific one. Running the benchmarks or even running Crysis 2 Maximum Edition were possible without any noticeable problems but as soon as I start Battlefield 3 or Battlefield 4 the VMs starts behaving very very sluggish. Mouse cursor moves choppy, task manager is updating slowly and  the performance is completely off.
BF3 shows around 30-40 FPS were it should 90-120, BF4 is even worse: 7-12 FPS no matter what configuration (normally around 70-90 FPS).
Sometimes I get a BSOF BUGCODE_USB_DRIVER, never seen before.

I didn't find any benchmark to reproduce this behavior yet.

So any advice to that matter is much appreciated.

Offline

#990 2014-01-08 11:23:40

xani
Member
Registered: 2013-12-24
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

@xani: Do you know exactly the root cause of the memory leak and how you fixed it in your kernel? Or are you unsure on the exact reason? It would be awesome if we get this fixed also in the main kernel sources :-) Or are you already in contact with kernel devs about this topic?

I didn't contact upstream, I thought about it but there's too much on my head right now. It would be great if you could report it.

Here's what I found:

xani wrote:

EDIT:
Further investigation showed that this problem can lie within gcc itself. Here's link that can give you explanation. I will report on my findings asap.

In short, Linus wanted to use some better asm goto instructions. It was implemented in commit

0c44c2d0f459c

But it was causing problems and they discovered that problem lies in gcc. It compiled new code in wrong way.
The patches for gcc will be released with 4.9, I think. Anyway, Ingo Molnar repaired problem they had with a hack in commit

88f182dd77

This commit is not fixing mem leak issue. I read in article that is in my qoute, that problem folks had upstream was repaired by disabling ASM_GOTO, so I did the same and problem disappeared. Patch that I included in latest version is very simple. Mem leak patch.

I found out that ASM_GOTO is a cause of the problem by bisecting kernel.

Offline

#991 2014-01-08 11:26:01

xani
Member
Registered: 2013-12-24
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

Edit: Nevermind. I can't re-produce the problem.

With that patched kernel to stop the memory leak I can confirm it works unless I use hugetlbfs. Using hugetlbfs causes the memory leak again.

I'm not using hugetlbfs. After my exams maybe I'll test it, so it will be mid February.

Offline

#992 2014-01-09 05:14:24

syllopsium
Member
Registered: 2013-12-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
syllopsium wrote:

The motherboard has a built in Matrox G200e, I'm hoping to pass this through to a different VM as it's a horribly slow chip. It appears in the same iommu group as the southbridge x4 PCIe slot (the only slot where a graphics card won't be slowed down to x1 speed by the 3210 chipset on this motherboard). This is why I've applied the ACS patch.

FWIW, nvidia/amd require several quirks in qemu to make things work.  Nobody has done these for matrox and I don't have anything with matrox graphics.  Besides that, what's the point of attaching a matrox card to a guest?  It seems like the headache of assigning it is not worth the incremental improvement versus emulated graphics.  Assignment makes sense for high end GPUs, but I'm not sure I'm interested in maintaining code for assigning random 2d-only VGA devices.

Point. Although it's currently a moot one because a) I'm stupid and forgot to turn rombar back to one - it works when correctly configured, at least in VGA mode. b) QEMU now fails with an emulation failure on the creaky old OS I'm trying to run, but that's not a vfio problem.. Thanks for all the help anyway smile

Offline

#993 2014-01-09 10:58:10

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! great stuff. This has inspired me to try this on my own. I had to buy a new cpu/motherboard as i didn't have the virtualization stuff needed in my old Intel 2600K.

My setup (once parts get delivered) will be:
AMD FX8350
Gigabyte GA-990XA-UD3, Socket-AM3+
8GB ram
Nvidia GTX 570 (host)
Nvidia GTX580 (passed through)

I will be using the kernel/qemu/seabios from OP.

Results will be posted, if anyone have any starter tips feel free to let me know.

nbhs wrote:

ISSUES:
Using NVIDIA proprietary drivers on the host requires a patch: see http://www.mail-archive.com/qemu-devel@ … 74066.html, patch (posted by andy123):  http://bpaste.net/show/109185/

One question is that i have been testing qemu on my old hardware just to get a feel of it (without passthrough) and i noticed that my Nvidia driver works without patching on the Host even with kvm modules inserted. Is this because i am not passing anything through yet or is the patch obsolete?

Offline

#994 2014-01-09 13:45:36

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbirkis wrote:

[…]

nbhs wrote:

ISSUES:
Using NVIDIA proprietary drivers on the host requires a patch: see http://www.mail-archive.com/qemu-devel@ … 74066.html, patch (posted by andy123):  http://bpaste.net/show/109185/

One question is that i have been testing qemu on my old hardware just to get a feel of it (without passthrough) and i noticed that my Nvidia driver works without patching on the Host even with kvm modules inserted. Is this because i am not passing anything through yet or is the patch obsolete?

The patch is to fix some problems with passthough (something about the vga arbiter?) and I think I'm still using it.


i'm sorry for my poor english wirting skills…

Offline

#995 2014-01-09 15:58:09

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks everybody for this great thread, I managed to get VGA passthrough working.

Motherboard: AsRock H67M-GE
CPU: Intel i7-2600
Host GPU: Intel HD 2500 (this is a headless machine, X isn't installed)
Guest GPU: AMD HD6770

I used the kernel, qemu and seabios PKBUILDs from the first post on a fresh Arch install and everything seems to be working just fine.

I created a Windows 7 VM and a Xubuntu VM, installed the drivers (the Windows Catalyst installer crashes a lot) for the card and run Uningine without any problems in either VM. I can also shutdown/restart/switch between the two VMs at will, awesome!

Offline

#996 2014-01-09 22:22:47

beardedchimp
Member
Registered: 2014-01-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been running KVM for a few months now all thanks to this thread so thanks for that.

My setup is:
archlinux host
Windows 8.1 Guest
Gigabyte 990FXA-UD3
Radeon X700 PRO for host. I didn't even know what card is was until I looked it up for this post, all I knew was that it was ati and old.
gtx 760 for guest.
amd 8350fx, 6 cores for guest, 2 for host
16GB ram, 8GB guest, 8GB host
I pass through a 200GB hard drive through /dev/sda
I also passthrough a few usb ports

So far it's been working well, I love having full access to arch while I mess about playing games but I have run into issues with what I can run on the guest.

For some reason only some games will start. Other games (such as CS:GO) start launcing from steam but crash with no error message.

Games that do work are:
Counter Strike Source
Borderlands 2
Just Cause 2
Bad company 2
F1 2012

Games that don't work:
CS:GO
Shogun 2
Tomb Raider (the new one)
Civ 5
A load of others but can't remember which


The games all crash in the same way, no error message and no error log file generated. I've tried launching them in various ways with no success.

Because I run the guest from /dev/sda, I can also boot windows directly instead of through KVM. I was really surpised that this work actually but it's fine. When windows is booted directly all the games run, no hint of crashing.

I've had this setup working since August and I was hoping by now updates to linux/kvm/windows would have sorted this issue but it's still there and I'm no closer to working out why.

Has anyone experianced anything similar or know what might be causing it?

Here's the script I've cobled together from this thread, please let me know if I've done anything stupid.

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) CPUSET=/sys/fs/cgroup/cpuset QEMU=/usr/bin/qemu-system-x86_64 USE_CPUSET=true VCPUS="6" #must be > 0 < cpu core number CONFIG="-M q35 -enable-kvm" MEM="8192" MEM_PATH="" CPU="host" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0" #NET="-net nic -net bridge,br=br0" NET="" SND=true SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128" USB_DEVICES="" PCI_DEVICES="00:16.0 00:16.2" GPU="04:00.0" GPU_SND="04:00.1" ROM="/var/lib/vm/GTX760.rom" VBIOS="" HDD="/dev/sda" EXTRA_ARGS="-boot menu=on -monitor stdio" if [ -n "$GPU" ]; then CONFIG+=" -vga none -nographic" DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1,romfile=$ROM" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d,bus=pcie.0" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,if=virtio,cache=none" fi if [ -n "$CDROM" ]; then isonum=1 for d in $CDROM; do isonum=$[$isonum +1] STORAGE+=" -drive file=$d,id=isocd$isonum -device ide-cd,bus=ahci.$isonum,drive=isocd$isonum" #STORAGE+=" -drive file=$CDROM,id=isocd$isonum -device ide-cd,bus=ahci.1,drive=isocd$isonum" done fi fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" #SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -soundhw ac97" #SND="-soundhw ac97" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if [ -n "$MEM" ]; then MEM="-m $MEM" if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi else if [ -n "$MEM_PATH" ]; then MEM+=" -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ $USE_CPUSET ]; then if [ ! -d $CPUSET/system ]; then mkdir $CPUSET/system fi if [ ! -d $CPUSET/qemu ]; then mkdir $CPUSET/qemu fi echo $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) > $CPUSET/qemu/cpuset.cpus echo 1 > $CPUSET/qemu/cpuset.cpu_exclusive echo 0 > $CPUSET/qemu/cpuset.mems echo 0-$((CPU_CORE_NR-VCPUS-1))> $CPUSET/system/cpuset.cpus echo 1 > $CPUSET/system/cpuset.cpu_exclusive echo 0 > $CPUSET/system/cpuset.mems for t in `cat $CPUSET/tasks`; do echo $t > $CPUSET/system/tasks done > /dev/null 2>&1 fi set -m #netctl start bridge QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $NET $STORAGE $EXTRA_ARGS & #$QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $STORAGE $EXTRA_ARGS & PID=$! if [ $USE_CPUSET ]; then ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then echo $t > $CPUSET/qemu/tasks taskset -pc $i $t; let i++ fi fi done break fi done ) & fg %1 wait for t in `cat $CPUSET/system/tasks`; do echo $t > $CPUSET/tasks done > /dev/null 2>&1 rmdir $CPUSET/system rmdir $CPUSET/qemu fi echo "All Done!"

Offline

#997 2014-01-10 13:15:20

pascal257
Member
Registered: 2014-01-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Because I run the guest from /dev/sda, I can also boot windows directly instead of through KVM. I was really surpised that this work actually but it's fine. When windows is booted directly all the games run, no hint of crashing.

I've had this setup working since August and I was hoping by now updates to linux/kvm/windows would have sorted this issue but it's still there and I'm no closer to working out why.

I'd try a fresh virtualisation-only installation. I suspect some driver issues or something related to that. I had no issues so far playing games.

But I haven't tried Civ 5 yet, I'll check that tonight. Also I am running Windows 7 instead of 8.

Last edited by pascal257 (2014-01-10 13:16:37)

Offline

#998 2014-01-10 15:46:44

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pascal257 wrote:

[…] I haven't tried Civ 5 yet, I'll check that tonight. Also I am running Windows 7 instead of 8.

I've played countless hours (>100) of Civ 5 on my setup, most likely even in different windows versions. My VM started out with windows 7 and was upgraded to 8 and 8.1 from there, iirc.


i'm sorry for my poor english wirting skills…

Offline

#999 2014-01-11 00:19:59

beardedchimp
Member
Registered: 2014-01-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pascal257 wrote:

Because I run the guest from /dev/sda, I can also boot windows directly instead of through KVM. I was really surpised that this work actually but it's fine. When windows is booted directly all the games run, no hint of crashing.

I've had this setup working since August and I was hoping by now updates to linux/kvm/windows would have sorted this issue but it's still there and I'm no closer to working out why.

I'd try a fresh virtualisation-only installation. I suspect some driver issues or something related to that. I had no issues so far playing games.

But I haven't tried Civ 5 yet, I'll check that tonight. Also I am running Windows 7 instead of 8.

I actually ran windows 7 first, then switched to windows 8 to see if it would fix the problem and didn't.

I've also upgraded the video drivers etc. multiple times with no effect so I don't know what drivers could be causing it.

Ahh, I might actually be wrong about Civ 5 being broken as I just checked and this win8 vm doesn't even have it installed. I'm going to work out a more complete list of what doesn't work.

Since I've already installed the VM twice with the same results I'm not confident about installing it again but I'll try any suggestions.

Cheers

Offline

#1000 2014-01-11 00:40:59

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just ordered an FX-8350 with an ASUS M5A97 R2.0. Has anyone managed to get this working on this motherboard?

Also, has anyone figured out a way around the issue with catalyst initializing both GPUs if they are the exact same model? I have 2x HD 6870, both from XFX, exact same model, that I would like to use (one in windows guest, one on arch host). I don't mind buying a new nvidia GPU for the host, but it may be a couple months until I can do that.

Offline

#1001 2014-01-11 05:31:52

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, on the mailing list, Andrew helped me track down my "genirq" issue.  Somehow, I had set "nointxmask" on vfio-pci.  However, the behavior I'm getting is still somewhat odd.

Intel video is host, but is in UEFI boot mode.  Therefore, I'd expect it to not need the legacy VGA registers...
I'm booted into a kernel WITH the Intel VGA Abiter patches, though.

Radeon is handed to the guest, as secondary video (precautions against VGA arbitration issues).
With pci-assign, the radeon needs to be manually ejected before guest session, and upon doing so, the GPU fan kicks into ultra-high-speed mode.
With radeon handed to VM via vfio, but still as secondary device (no x-vga), the VM shows garbage on the Radeon, and then reboots.

Ideally, I'd like to get a video card with native UEFI boot mode support -- with a proper EFI GOP video card, Windows 8 supposedly doesn't need the legacy VGA registers at all.
Right now, that a moot point, since the VM won't even TRY to boot if I try to use OVMF.fd as the firmware.  I don't even get the TianoCore logo, even with all VFIO usage removed!

EDIT: Okay, redid it without libvirt involved at all.
Q35 system, primary is cirrus, BIOS image not loaded: VIDEO_TR_FAILURE (atikmpag.sys)
Q35 system, primary is QXL, BIOS image not loaded: Radeon driver won't load due to unspecified resource conflict.
Q35 system, primary is cirrus or QXL, BIOS image loaded: THREAD_STUCK_IN_DEVICE_DRIVER
Q35 system, x-vga=on: some reboots with no output, some boots with THREAD_STUCK_IN_DEVICE_DRIVER.
piix4 system: tutorial doesn't say where I should attach things.

EDIT again: Okay, this time I tried removing ALL forwarded devices.  Still, black screen!
EDIT yet again: While I bang my head against this for a while longer (or perhaps just call it a night), I guess I'll leave a few coherent questions.

The biggest question: is my assumption that using EFI boot in the guest will make VGA arbitration unnnecessary, likely to be true?  Also, what can I do to get Qemu to actually START in EFI mode?

Last edited by DanaGoyette (2014-01-11 06:23:17)

Offline

#1002 2014-01-11 09:02:25

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I forget where I read this but I remember seeing that booting from UEFI causes some issues with VFIO for now. I could try without it and if it works, see if A.W. can help point you in the right direction. He posts here from time to time or you may have better luck on the qemu mailing lists.
Edit: this seems to prove me wrong about UEFI+VFIO, so maybe the issue was UEFI + VBIOS / x-vga=on...

Last edited by firewing1 (2014-01-11 09:05:04)

Offline

#1003 2014-01-11 16:49:06

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The vfio passthrough isn't even a factor in OVMF not working.  Compare these two command lines:
qemu-system-x86_64 -M q35 -m 5120 -boot menu=on -monitor stdio -debugcon file:/dev/stderr -global isa-debugcon.iobase=0x402 -bios /usr/share/seabios/bios.bin
qemu-system-x86_64 -M q35 -m 5120 -boot menu=on -monitor stdio -debugcon file:/dev/stderr -global isa-debugcon.iobase=0x402 -bios /usr/share/ovmf/OVMF.fd

The former goes on to PXE boot; the latter just hangs, before even printing any of the isa-debugcon info!

EDIT: Okay, well, at least the following tries to start:
qemu-system-x86_64 -M pc -m 5120 -boot menu=on -monitor stdio -debugcon file:/dev/stderr -global isa-debugcon.iobase=0x402 -bios /usr/share/ovmf/OVMF.fd
However, if I add enable-kvm, it again hangs instead of booting.

Are there any general guidelines about where you're supposed to attach the GPU on the i440fx virtual machine?

Last edited by DanaGoyette (2014-01-11 17:22:29)

Offline

#1004 2014-01-11 17:36:59

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Okay, it looks like UEFI is a dead-end -- it's only in a minimally-usable state.
The better option is to keep using Cirrus as primary, but I'd really like to get it working with vfio for the Radeon, instead of pci-attach.  Right now, attempting to do so results in a guest display full of checkerboarded static, and then a guest reboot.

EDIT: Interesting: I can get a somewhat more correct pc-q35 system using this config file: https://github.com/qemu/qemu/blob/maste … hipset.cfg

Then the semantics of attaching devices is somewhat different:
-readconfig q35-chipset.cfg \
-device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1,romfile=/usr/share/qemu/hd5750_vbios.bin,x-vga=on
-device vfio-pci,host=09:00.0,bus=ich9-pcie-port-2 \
-device vfio-pci,host=0c:00.0,bus=ich9-pcie-port-3 \
-device vfio-pci,host=07:04.0,bus=ich9-pci-bridge \

The last device is actually a PCI device, not a PCIe device, by the way.

With the above config, I can boot an Ubuntu VM, and get garbled colors on the host, but working 3D acceleration on an Ubuntu LiveCD.  It still BSODs in Windows, though.

EDIT: Adding some version info:

seabios: (version rel-1.7.3-117-g31b8b4e-20131206_080705-nilsson.home.kraxel.org)
QEMU emulator version 1.7.0 (Debian 1.7.0+dfsg-2ubuntu5), Copyright (c) 2003-2008 Fabrice Bellard

Last edited by DanaGoyette (2014-01-11 18:36:54)

Offline

#1005 2014-01-11 18:45:13

firewing1
Member
Registered: 2013-12-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

With the above config, I can boot an Ubuntu VM, and get garbled colors on the host, but working 3D acceleration on an Ubuntu LiveCD.  It still BSODs in Windows, though.

Your hardware is very similar to my setup - the garbled colors and BSOD I was able to fix using the Haswell i915 arbitration patches. If you haven't applied those already, try rebuilding the kernel with them.

Offline

#1006 2014-01-11 19:07:29

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Interesting... I realized I had somehow trampled on git qemu, and hence reinstalled git qemu. 

After a bit more mucking around, I finally got the damn thing to start -- even WITHOUT the VGA arbiter patches!

Here's my complete config now:

qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 5120 \
-cpu host \
-boot order=dc \
-monitor stdio \
-debugcon file:/dev/stderr -global isa-debugcon.iobase=0x402 \
-smp 8,sockets=1,cores=4,threads=2 \
-bios /usr/share/qemu/bios.bin \
-readconfig q35-chipset.cfg \
-device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1,rombar=0 \
-device vfio-pci,host=09:00.0,bus=ich9-pcie-port-3 \
-device vfio-pci,host=0c:00.0,bus=ich9-pcie-port-4 \
-device vmware-svga,bus=ich9-pci-bridge \
-device vfio-pci,host=07:04.0,bus=ich9-pci-bridge \
-drive id=dvd -device ide-cd,drive=dvd,bus=ide.0 \
-drive file=/var/lib/libvirt/images/Windows.img,id=hdd,format=raw -device ide-hd,drive=hdd,bus=ide.1 \
-device usb-tablet \
-net none

I'm not exactly sure WHAT it's using for VGA bios now (particularly considering where I stashed the vmware VGA controller), but hey, as long as it boots, it's great.

EDIT: Note that "-net none" is acceptable for me, because I'm forwarding 0c:00.0, which is a real NIC.

EDIT: Also note: the host is, again, in EFI boot mode.  With my hacked-up config above, I have no idea if that's part of why it works.

Last edited by DanaGoyette (2014-01-11 19:29:01)

Offline

#1007 2014-01-12 14:38:54

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
TripleSpeeder wrote:
nbhs wrote:

AUDIO EMULATION:

To hear the sound from the vm on your host speakers you'll need to add this lines:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

You might need to start qemu like this (using pulseaudio, to use a different driver change QEMU_AUDIO_DRV= variable):

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

With QEMU_PA_SAMPLES=128 the sound is horribly distorted here. Without this parameter it works fine though. @Op: What is the background of this setting?

Trial and error, im using alsa now instead of pulse now though

What QEMU_AUDIO_* and QEMU_ALSA_* parameters do you use? Is there something non-default in your .asoundrc? I was unable to pick up the right parameters to use with QEMU_AUDIO_DRV=alsa and Windows 7. Sound is horribly chopped and distorted each time I run the Windows 7 VM (sound works fine in a Linux VM with default parameters). Specifying usb-audio or ich9-intel-hda as emulated sound card doesn't really make any difference. Sound card used on the host system is HDA Intel (ALC1150).

Offline

#1008 2014-01-12 18:58:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cryptonymous wrote:
nbhs wrote:
TripleSpeeder wrote:

With QEMU_PA_SAMPLES=128 the sound is horribly distorted here. Without this parameter it works fine though. @Op: What is the background of this setting?

Trial and error, im using alsa now instead of pulse now though

What QEMU_AUDIO_* and QEMU_ALSA_* parameters do you use? Is there something non-default in your .asoundrc? I was unable to pick up the right parameters to use with QEMU_AUDIO_DRV=alsa and Windows 7. Sound is horribly chopped and distorted each time I run the Windows 7 VM (sound works fine in a Linux VM with default parameters). Specifying usb-audio or ich9-intel-hda as emulated sound card doesn't really make any difference. Sound card used on the host system is HDA Intel (ALC1150).

Using win 8.1

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64...

But like i said, it was just trial and error, and no its just stock arch settings on a gnome3 desktop

Last edited by nbhs (2014-01-12 19:00:49)

Offline

#1009 2014-01-12 20:46:28

dogblue
Member
Registered: 2013-06-23
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

beardedchimp wrote:

Games that don't work:
CS:GO
Shogun 2
Tomb Raider (the new one)
Civ 5
A load of others but can't remember which

I have also had problems with Shogun 2. The demo (downloaded from steam) ran fine, but after I bought it it would crash to desktop from the loading/legal screen. I assumed it was just poorly written and unrelated to running in a VM.

Offline

#1010 2014-01-13 05:41:07

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Works fine for me with a Radeon HD7770, after trying many different configurations. Have a R9 280X on the way.

Specs:

ASUS P9D-WS motherboard
Intel Xeon E3-1275 v3
32 GB RAM
LSI SAS9211-8I
XFX Radeon HD7770

I had to plug in the LSI card on the last x16 slot which forced it in x4 mode, just so I could get the GPU in its isolated IOMMU group.

I compiled 3.13-rc6 using the PKGBUILD from https://aur.archlinux.org/packages/linux-mainline/, putting in the Intel IGP arbitration patch and the memory leak patch posted earlier, with all the CONFIG_VFIO stuff built in the kernel.

Installed libvirt, et al., save for QEMU and seabios using pacman. Pulled from QEMU git recursively and compiled, installed into /usr/local.

I added these arguments to the kernel command line:

vfio_iommu_type1.allow_unsafe_interrupts=1 intel_iommu=on pci-stub.ids=xxxx:yyyy,aaaa:bbbb kvm_intel.emulate_invalid_guest_state=0 nohz=off

Replacing xxxx:yyyy and aaaa:bbbb with their appropriate respective PCI ids for the GPUs.

/etc/libvirt/qemu.conf:

user = "root" group = "kvm" cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1" ] clear_emulator_capabilities = 0

For some reason,

clear_emulator_capabilities = 1

would not work. Is this really a big deal? qemu.conf has a scary-sounding warning:

# Warning: Disabling this option means that a compromised guest can # exploit the privileges and possibly do damage to the host.

And in my libvirt XML specification I use, the relevant parts:

<os> <type arch='x86_64' machine='pc-q35-1.7'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> </cpu> ... <seclabel type='none'/> <qemu:commandline> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/local/share/qemu/bios.bin'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> </qemu:commandline>

Installed Windows 8.1 with VFIO drivers at install time. Installed Radeon drivers. Everything seems to work fine.

Now I need to figure out how to pin CPUs and stuff like that.

Does anyone know if VM guest suspend/pause works with GPU passthrough or not?

Last edited by MacCoaster (2014-01-13 05:49:35)

Offline

#1011 2014-01-13 07:16:37

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

EDIT: Also note: the host is, again, in EFI boot mode.  With my hacked-up config above, I have no idea if that's part of why it works.

When I tried to do VGA passthrough with pci-assign last summer, I had a problem, IOMMU and DMAR weren't enabled properly unless I was booting in (U)EFI mode, I think it was something related to secure boot or a buggy (U)EFI. I updated to the latest (U)EFI for my motherboard, then (re)installed Arch in (U)EFI boot mode and then IOMMU and DMAR were beeing enabled and VGA passthrough worked.

Offline

#1012 2014-01-14 12:51:13

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:

Does anyone know if VM guest suspend/pause works with GPU passthrough or not?

pause/resume vm works fine for me with 7850 OC

Offline

#1013 2014-01-15 04:14:55

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, just for comparison, I tried an NVIDIA GeForce 260 (borrowed); with my same config above that works with the Radeon, it definitely does NOT work with the nvidia card (even with the Code 43 NoSnoop patch).

If I go to device manager (audio navigation using Narrator), I can try to disable and re-enable the GeForce, and I get Qemu (on the host) aborting:

qemu-system-x86_64: hw/pci/pci.c:232: pcibus_reset: Assertion `bus->irq_count[i] == 0' failed.

Offline

#1014 2014-01-15 12:50:10

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem getting the patch for the nvidia driver to work. My setup has the kernel + qemu + seabios from OP. But i cant figure out how to get a patched nvidia proprietary driver for the host.

I tried building the driver using ABS, changing the version in the PKGBUILD to 319.23 and applying the andy123 patch from OP. But i couldn't get past some xorg dependency issue to install it.

In addition i tried changing the version numbers in the patch file and PKGBUILD applying it to the current version 331.38, but i could not load the module.

What am i doing wrong here?

Offline

#1015 2014-01-16 06:24:38

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Been studying this fine thread for quite a bit, great work, appreciate it!

Now I have things up and running, specs:

ASUS Rampage IV Extreme
intel 4930K
beta BIOS 9502 (only bios currently for this mb that supports vt-d for ivy-E)
host using AMD 7970 and radeon drivers
Windows 7 guest in VM using NVIDIA K2000 passed through
host is Slackware 14.1 but with kernel 3.12.1 downloaded from kernel.org with iommu config options compiled in kernel
qemu 1.7.50 from git per original post on this thread, with seabios as well

Things pretty much worked exactly according to instructions.  I have no sound on the guest, but haven't given up yet, suspect it's a qemu compile problem. Had the catalyst driver installed with Slackware's default kernel but for some reason it didn't play nice with kernel 3.12.1.  Couldn't run X.   It's a work in progress.

Last edited by mostlyharmless (2014-01-16 06:32:54)

Offline

#1016 2014-01-17 09:12:48

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A quick update. My attempt to run VGA_Passthrough is what i would call a success with the odd hickup here and there.

Info:
Motherboard - Gigabyte 990XA-UD3 (ps dont buy this motherboard, it is not good imo)
CPU - AMD 8350fx 8 core (6 cores reserved for guest, 2 for host)
RAM - 8GB (4gb reserved for guest)
Host VGA - Nvidia GTX 570
Guest VGA - Nvidia GTX 580
120Gb ssd for host
120Gb ssd for guest
Host is Arch linux with kernel+qemu+seabios from OP
Guest is Win7 64bit

I have only gotten as far as to getting the guest up and running on the passed through card, havent tested any games or 3d intensive apps yet. It seems to work good in guest, i get a windows experience rating of 7,3 and aero looks smooth and good. I will report back with more info on performance etc after i get some more testing done.

What i havent gotten working:
Nvidia driver for the host. I am having problems getting the driver patched using OP patch from andy123, this is not a vga passthrough/kvm/qemu problem but probably due to user error on my part.
Sound is not tested yet.
I can only run the vm one time each reboot, it is kinda annoying but i really cant find a way to debug and fix this. If someone have any tips i can try pls let me know.

Offline

#1017 2014-01-18 03:21:23

Lauri
Member
Registered: 2014-01-18
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this great thread. With all the information here I was able to get VGA passthrough working.

My setup is:
Debian testing host
Kernel 3.12.8 with i915 and acs override patches
QEMU-git with nosnoop patch
Windows 7 guest

Motherboard: Asus Z87-K
CPU: Intel i7-4770
GPU host: Intel HD4600
GPU guest: GeForce GTX560

I've only used this for gaming and so far every game I've tried has worked. Only problem I have with this is if I shutdown the VM I have to reboot the host to get VGA passthrough working again.

Offline

#1018 2014-01-18 12:12:42

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

when i try to compile your qemu-git i get:

cp: cannot stat ‘/home/user/Downloads/qemu-git/src/qemu-git-2.0.r30392.g1cf892c/libcacard.pc’: No such file or directory ==> ERROR: A failure occurred in package_libcacard-git(). Aborting...

Why would i run into problems if i compile from AUR? (not that i can, qemu requires spice, spice requires libcacard and libcacard conflicts with qemu breaking the package) hmm

Why can't we use the qemu package in the official repos?

Finally, how do i apply the patch for using the nvidia proprietary drivers on the host? i can see a patch, but i ahve no idea what file to patch or how to get my hands on it (i thought they were closed source these drivers...(

Last edited by rabcor (2014-01-18 13:53:48)

Offline

#1019 2014-01-18 14:35:34

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

[…]
Finally, how do i apply the patch for using the nvidia proprietary drivers on the host? i can see a patch, but i ahve no idea what file to patch or how to get my hands on it (i thought they were closed source these drivers...(

Since mbirkis also had a problem with this, here is my modified PKGBUILD for current nvidia-dkms.


i'm sorry for my poor english wirting skills…

Offline

#1020 2014-01-18 15:09:06

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, i tried that and it didn't work for me it seems, i.e. if i run the mainline kernel i can't start X (on host), but if i run the stock arch kernel it runs X. Do i need to run a special command or something to make the card work?

Last edited by rabcor (2014-01-18 15:09:38)

Offline

#1021 2014-01-18 15:12:11

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dkms needs to be set up and working, but besides that… maybe you don't have the headers installed for the mainline kernel?


i'm sorry for my poor english wirting skills…

Offline

#1022 2014-01-18 18:24:59

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, is there any way to get suspend/resume of the host to not crash the guest?  I suspended my host, and now the guest refuses to boot -- I'll hear my Xonar sound card click (indicating that the driver has loaded), but then the guest reboots, over and over.
I tried switching back to the Cirrus as VGA, and the damn thing still gives THREAD_STUCK_IN_DEVICE_DRIVER, and tries to reboot.  However, instead of rebooting, Qemu crashes:

pcibus_reset: Assertion `bus->irq_count[i] == 0' failed.

gdb backtrace:

#0 0x00007f72de10af77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 resultvar = 0 pid = 27419 selftid = 27419 #1 0x00007f72de10e5e8 in __GI_abort () at abort.c:90 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7fffabffe532, sa_sigaction = 0x7fffabffe532}, sa_mask = {__val = {140131329995524, 140131448300662, 232, 140127602999296, 140131328612305, 140131480475824, 140736079064797, 124136728992, 4, 140736079056528, 0, 0, 0, 140131329994867, 140131442405376, 140131330010968}}, sa_flags = -449665849, sa_restorer = 0x7f72e532aff6 <__PRETTY_FUNCTION__.24793>} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f72de103d43 in __assert_fail_base (fmt=0x7f72de25af58 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f72e532a4c7 "bus->irq_count[i] == 0", file=file@entry=0x7f72e532a476 "hw/pci/pci.c", line=line@entry=232, function=function@entry=0x7f72e532aff6 <__PRETTY_FUNCTION__.24793> "pcibus_reset") at assert.c:92 str = 0x7f72e7346790 "" total = 4096 #3 0x00007f72de103df2 in __GI___assert_fail (assertion=0x7f72e532a4c7 "bus->irq_count[i] == 0", file=0x7f72e532a476 "hw/pci/pci.c", line=232, function=0x7f72e532aff6 <__PRETTY_FUNCTION__.24793> "pcibus_reset") at assert.c:101 No locals. #4 0x00007f72e507e487 in pcibus_reset (qbus=0x7f72e704a440) at hw/pci/pci.c:232 bus = 0x7f72e704a440 i = 3 __PRETTY_FUNCTION__ = "pcibus_reset" #5 0x00007f72e4ff80c3 in qbus_reset_one (bus=0x7f72e704a440, opaque=0x0) at hw/core/qdev.c:236 bc = 0x7f72e7054d40 __func__ = "qbus_reset_one" #6 0x00007f72e4ff8698 in qbus_walk_children (bus=0x7f72e704a440, pre_devfn=0x0, pre_busfn=0x0, post_devfn=0x7f72e4ff800d <qdev_reset_one>, post_busfn=0x7f72e4ff8053 <qbus_reset_one>, opaque=0x0) at hw/core/qdev.c:365 kid = 0x0 err = 0 #7 0x00007f72e4ff8756 in qdev_walk_children (dev=0x7f72e70603c0, pre_devfn=0x0, pre_busfn=0x0, post_devfn=0x7f72e4ff800d <qdev_reset_one>, post_busfn=0x7f72e4ff8053 <qbus_reset_one>, opaque=0x0) at hw/core/qdev.c:390 bus = 0x7f72e704a440 err = 0 #8 0x00007f72e4ff865c in qbus_walk_children (bus=0x7f72e70356d0, pre_devfn=0x0, pre_busfn=0x0, post_devfn=0x7f72e4ff800d <qdev_reset_one>, post_busfn=0x7f72e4ff8053 <qbus_reset_one>, opaque=0x0) at hw/core/qdev.c:356 kid = 0x7f72e7055fa0 err = 0 #9 0x00007f72e4ff8183 in qbus_reset_all (bus=0x7f72e70356d0) at hw/core/qdev.c:248 No locals. #10 0x00007f72e4ff81c8 in qbus_reset_all_fn (opaque=0x7f72e70356d0) at hw/core/qdev.c:254 bus = 0x7f72e70356d0 #11 0x00007f72e51a0aa4 in qemu_devices_reset () at vl.c:1839 re = 0x7f72e730eed0 nre = 0x7f72e735b2e0 #12 0x00007f72e51a0b10 in qemu_system_reset (report=true) at vl.c:1848 No locals. #13 0x00007f72e51a1029 in main_loop_should_exit () at vl.c:1981 r = RUN_STATE_DEBUG #14 0x00007f72e51a1139 in main_loop () at vl.c:2021 nonblocking = false last_io = 1 #15 0x00007f72e51a8715 in main (argc=54, argv=0x7fffabffd6e8, envp=0x7fffabffd8a0) at vl.c:4382 i = 64 snapshot = 0 linux_boot = 0 icount_option = 0x0 initrd_filename = 0x0 kernel_filename = 0x0 kernel_cmdline = 0x7f72e534d0e6 "" boot_order = 0x7f72e7010300 "dc" ds = 0x7f72e70e8810 cyls = 0 heads = 0 secs = 0 translation = 0 hda_opts = 0x0 opts = 0x7f72e7010250 ---Type <return> to continue, or q <return> to quit--- machine_opts = 0x7f72e7010180 olist = 0x7f72e573abc0 <qemu_machine_opts> optind = 54 optarg = 0x7fffabffe891 "base=localtime" loadvm = 0x0 machine = 0x7f72e5746da0 <pc_q35_machine_v2_0> cpu_model = 0x7fffabffe565 "host" vga_model = 0x7fffabffe628 "cirrus" qtest_chrdev = 0x0 qtest_log = 0x0 pid_file = 0x0 incoming = 0x0 show_vnc_port = 0 defconfig = true userconfig = true log_mask = 0x0 log_file = 0x0 mem_trace = {malloc = 0x7f72e51a47f8 <malloc_and_trace>, realloc = 0x7f72e51a4850 <realloc_and_trace>, free = 0x7f72e51a48bd <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} trace_events = 0x0 trace_file = 0x0 args = {machine = 0x7f72e5746da0 <pc_q35_machine_v2_0>, ram_size = 5368709120, boot_order = 0x7f72e7010300 "dc", kernel_filename = 0x0, kernel_cmdline = 0x7f72e534d0e6 "", initrd_filename = 0x0, cpu_model = 0x7fffabffe565 "host"}

EDIT: Okay, I managed to get the thing to hang instead of BSODing.  http://msdn.microsoft.com/en-us/library … 85%29.aspx
What can I do to debug this, now?

Last edited by DanaGoyette (2014-01-18 19:01:22)

Offline

#1023 2014-01-18 18:33:47

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Dkms needs to be set up and working, but besides that… maybe you don't have the headers installed for the mainline kernel?

dkms nvidia drivers require dkms as a dependency, so naturally i have that.

linux-mainline headers should be a part of the linux-mainline package (according to AUR)

Last edited by rabcor (2014-01-18 18:34:00)

Offline

#1024 2014-01-18 18:54:21

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
andy123 wrote:

Dkms needs to be set up and working, but besides that… maybe you don't have the headers installed for the mainline kernel?

dkms nvidia drivers require dkms as a dependency, so naturally i have that.

linux-mainline headers should be a part of the linux-mainline package (according to AUR)

Sure, it installs the dkms package, but did you enable the service, start it and build the module?

I'm building the kernel from the stable git repository and using the pkgbuild from [core] with makepkg -R atm.

Moreover what do you mean by "X doesn't start"? Can you post xinit output, or output from your display manager, anything?


i'm sorry for my poor english wirting skills…

Offline

#1025 2014-01-18 19:23:47

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oh hey it works, i just had to bind the other GPU to vfio first. Otherwise it gave me a: "NVIDIA: Failed to load the NVIDIA kernel module. No Devices Detected"

Also, if i bound my 01:00.0  (Primary GPU, i.e. the one that's loaded in the first PCI-E slot on my motherboard) X would give the same error, but if i load 02:00.0 instead (the other card) X loaded the primary card.

Weird, i don't get it but well, it works. The rest (i guess) is to got a VM (and blasted QEMU which wont build!) up and running, first time using Qemu though, i'm bound to run into problems, but this is like the 3rd time i tried to follow this guide yikes first time i have any kind of success with this too (yeah i'm such a noob, sorry)

I bet it might be a pain to use my gaming keyboard though.

I have 2 questions though that bothered me a whole lot.

1. How can i update my linux-mainline kernel to the latest version without breaking the patches or the custom settings?
2. Is there nay way to passthrough a PCI or USB device (not GPU, i'm thinking soundcard this time), loading the windows drivers for it and then emulating it back to the host (Rather than the usual of emulating hardware to guest)

Update:

nbhs wrote:
myweb wrote:

warning: cannot resolve "libcacard-git", a dependency of "qemu-git"
:: The following package cannot be upgraded due to unresolvable dependencies:
      qemu-git

Aur does not provide libcacard-git

The qemu-git PKGBUILD builds 2 packages, qemu-git and libcacard-git, it should be built on the same folder as qemu-git, so you should install it first then qemu

However, when i try to build qemu git i get

cp: cannot stat ‘/home/xandis/Downloads/qemu-git/src/qemu-git-2.0.r30392.g1cf892c/.libs/libcacard.so*’: No such file or directory ==> ERROR: A failure occurred in package_libcacard-git(). Aborting...

What do i do? If i uninstall libcacard i need to uninstall spice too, and qemu-git wont compile without spice.

Update 2:

I managed to install the AUR version of qemu, but when i run the command to test the VM i get this

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Update 3:

And i seem to have messed something up and X doesn't start anymore on the linux mainline D:

That's not fair, i followed the guide to the letter, but i can't even start X. on my host, and vfio-bind always seems to fail (considering i always get that no iommu_group error) should i just give up?

Last edited by rabcor (2014-01-18 23:14:56)

Offline

#1026 2014-01-19 18:19:55

ironicbadger
Member
Registered: 2013-11-12
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauri wrote:

Thanks for this great thread. With all the information here I was able to get VGA passthrough working.

My setup is:
Debian testing host
Kernel 3.12.8 with i915 and acs override patches
QEMU-git with nosnoop patch
Windows 7 guest

Motherboard: Asus Z87-K
CPU: Intel i7-4770
GPU host: Intel HD4600
GPU guest: GeForce GTX560

I've only used this for gaming and so far every game I've tried has worked. Only problem I have with this is if I shutdown the VM I have to reboot the host to get VGA passthrough working again.

Does ejecting your GPU before reboot from the guest resolve this issue? Also I had problems with Win7 and found Win8 much more 'friendly' in this regard. I used Xen up until recently (and havn't tried this under KVM yet but I automated the process using devmanview and a tiny script I wrote. I wrote about it in my blog post http://blog.ktz.me/?p=219. (Sorry if linking isn't allowed I'll remove it, just PM me).


I have just purchased a Supermicro X10SAT board and using the information found in the thread thought I'd give KVM a try, having used Xen for months it would be nice to use something without having to install a package. Anyway, my issue is this. I have vt-d / passthrough of my RAID controller working great, except for one thing. It stops HDMI audio working as soon as I add intel_iommu=on to my grub_default bit. Here is the relevant part of my grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipv6.disable=1 intel_iommu=on vfio_iommu_type"

Here is my mkinitcpio.conf

MODULES="i915 kvm kvm_intel pci-stub"

If iommu isn't present in the default line audio via HDMI works great, as soon as I switch it on and redo grub and reboot (regardless of whether the host is actually running a guest of not) it doesn't play audio. So weird.

Is this likely to be a motherboard issue? I'm thinking so as I have an identical software setup on another machine (in my sig) and that works flawlessly.


Hardware - Asus P8H77-V LE, BIOS 0237 | Intel i5 3470 3.2ghz | 8GB RAM | TX750W Corsair PSU | 60GB OCZ SSD | 500GB data drive
ArchX (dom0) - Arch Linux | Xen 4.3 | XBMC 12.2 output to TV
unRAID Server Pro v5.0(domU) - passed through Supermicro SAS-AOC-SASLP-MV8 - 16.3TB of protected storage.
Windows 8(domU) - with VGA passthrough // Arch Linux(domU) - development VM

Offline

#1027 2014-01-20 09:06:50

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
rabcor wrote:

[…]
Finally, how do i apply the patch for using the nvidia proprietary drivers on the host? i can see a patch, but i ahve no idea what file to patch or how to get my hands on it (i thought they were closed source these drivers...(

Since mbirkis also had a problem with this, here is my modified PKGBUILD for current nvidia-dkms.

Thanks alot for the PKGBUILD, i was able to build the driver using that one. One slight problem is it does not work using the linux-mainline kernel (works fine with stock arch).
Errors could have something to do with this:

% uname -r 3.13.0-2-mainline
% dmesg | grep nvidia [ 2.642180] nvidia: module license 'NVIDIA' taints kernel. [ 2.652591] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 2.653150] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 3.475852] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 3.586826] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 3.683180] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 3.706485] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 3.815154] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 3.922719] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0) [ 4.037577] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
% systemctl status gdm.service gdm.service - GNOME Display Manager Loaded: loaded (/usr/lib/systemd/system/gdm.service; enabled) Active: active (running) since Mon 2014-01-20 09:33:58 CET; 1h 14min ago Main PID: 325 (gdm) CGroup: /system.slice/gdm.service 325 /usr/bin/gdm Jan 20 09:33:59 archxen gdm[325]: GdmDisplay: display lasted 0,107528 seconds Jan 20 09:33:59 archxen gdm[325]: Child process 678 was already dead. Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_ref: assertion 'object->ref_count > 0' failed Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_unref: assertion 'object->ref_count > 0' failed Jan 20 09:33:59 archxen gdm[325]: GdmDisplay: display lasted 0,106267 seconds Jan 20 09:33:59 archxen gdm[325]: Child process 690 was already dead. Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_ref: assertion 'object->ref_count > 0' failed Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_unref: assertion 'object->ref_count > 0' failed Jan 20 09:33:59 archxen gdm[325]: GdmDisplay: display lasted 0,114289 seconds Jan 20 09:33:59 archxen gdm[325]: GdmLocalDisplayFactory: maximum number of X display failures reached: ch...rrors Hint: Some lines were ellipsized, use -l to show in full.

Edit: Found similar issue here: https://devtalk.nvidia.com/default/topi … c1-kernel/ testing that patch now.

Edit2: The patch worked, i now have working nvidia driver on the host.

Last edited by mbirkis (2014-01-20 10:16:51)

Offline

#1028 2014-01-20 10:09:04

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oops, yeah. That makes sense. I forgot to tell you that I use this with 3.12.x not 3.13, sorry for the confusion.


i'm sorry for my poor english wirting skills…

Offline

#1029 2014-01-20 11:55:13

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How would one get 2x of the same GPU working with this? I have to pass 1002:6738 into pci-stub, but that matches *both* of my video cards.

Edit: Here's my graphics adapters in lspci -nn.

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] [1002:6738] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88] 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] [1002:6738] 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88]

I am using the open source radeon driver on the host, so the issue mentioned in OP shouldn't effect me?

Edit: nevermind, looks like it isn't an issue at all. I just put the above IDs into pci-stub in bootloader, started it, ran the steps for vfio-pci and it's working fine. However, now I'm getting stuck in the windows 7 installation. the first step, "copying windows files", goes fine, but the next step (expanding windows files) has been sitting at 0% for 20 minutes now. pretty sure it's stalled and not actually working, as monitoring disk IO on the host shows there is practically no IO coming from qemu. Looking at CPU usage, qemu is using 10% of one core.

I know I should expect it to be kinda slow, because I'm using a file as raw disk image, but this seems wrong.

-device ahci,bus=pcie.0,id=ahci -drive file=/mnt/oldarch/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -drive file=/mnt/2tb/win7/Windows7Ultimate.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd

Last edited by alexis_evo (2014-01-20 16:04:46)

Offline

#1030 2014-01-20 20:18:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

However, now I'm getting stuck in the windows 7 installation. the first step, "copying windows files", goes fine, but the next step (expanding windows files) has been sitting at 0% for 20 minutes now. pretty sure it's stalled and not actually working, as monitoring disk IO on the host shows there is practically no IO coming from qemu. Looking at CPU usage, qemu is using 10% of one core.

I know I should expect it to be kinda slow, because I'm using a file as raw disk image, but this seems wrong.

QEMU AHCI is barely functional, wait it out and switch to virtio once installed (or maybe you can do it during install with a virtual driver disk)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1031 2014-01-20 22:28:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For your testing pleasure (Intel root port ACS):

https://lkml.org/lkml/2014/1/20/453

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1032 2014-01-21 05:16:12

Lauri
Member
Registered: 2014-01-18
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ironicbadger wrote:

Does ejecting your GPU before reboot from the guest resolve this issue? Also I had problems with Win7 and found Win8 much more 'friendly' in this regard. I used Xen up until recently (and havn't tried this under KVM yet but I automated the process using devmanview and a tiny script I wrote. I wrote about it in my blog post http://blog.ktz.me/?p=219. (Sorry if linking isn't allowed I'll remove it, just PM me).

Ejecting the GPU from the guest doesn't help. If I eject the GPU from the guest and shut down and try start it again the host freezes.

aw wrote:

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

I'm using kernel 3.12.8 with i915 and acs override patches, latest development version of QEMU with nosnoop patch. I also tried it with kernel 3.13.0 and QEMU without nosnoop patch but trying to start the guest for a 2nd time make the host completely freeze. Can't even ssh to it or use sysrq key combinations to reboot.

01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 83b5 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=32M] Region 1: Memory at e0000000 (64-bit, prefetchable) [size=128M] Region 3: Memory at e8000000 (64-bit, prefetchable) [size=64M] Region 5: I/O ports at e000 [size=128] Expansion ROM at f6000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci

Last edited by Lauri (2014-01-21 05:43:19)

Offline

#1033 2014-01-21 08:18:41

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After letting it sit for like 3 hours it finally installed. Got virtio up and working. Games run, but I get random stalling. Lowering graphics settings seems to help but I can manage 60 FPS with highest settings in some games so it shouldn't be necessary.

Trying to run Mirror's Edge, get a solid framerate, highest settings on everything but 2xAA. Runs fine for about 5 minutes, will then stall for 5-10 seconds. Sometimes it recovers and lets me play a bit further. Sometimes it corrupts the display and I have to forcibly close qemu. In one situation after a stall, I got some weird screen tearing that persisted even after exiting the game. http://i.imgur.com/fuD5HkB.jpg

ASUS M5A99FX PRO R2.0, latest BIOS
AMD FX-8350
8GB of RAM (4GB given to the VM)
HD 6870 on both host and guest. Open source radeon driver on host, latest stable catalyst driver on guest.
Guest is Win7 64bit.

qemu incantation:

qemu-system-x86_64 \ -enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device ahci,bus=pcie.0,id=ahci -drive file=/mnt/oldarch/windows.img,id=disk,format=raw -device virtio-blk-pci,drive=disk \ -usb -usbdevice host:04d9:0167 -usbdevice host:0461:4d0f -boot menu=on \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Offline

#1034 2014-01-21 08:31:39

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Games run, but I get random stalling.

this happens for me as well, but only when the host is swapping (heavily?). I will upgrade to 16GB RAM in the next weeks.

Last edited by Flyser (2014-01-21 08:34:20)

Offline

#1035 2014-01-21 08:40:34

manu.lima
Member
Registered: 2014-01-10
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your post. It helped me to succesfully passthrough my vga on a windows 7 x86 guest. My hardware specs are:

AMD FX8320
GIGABYTE GA970-D3 (No issues at all, IOMMU works out of the box)
8 GB RAM
ATI RADEON HD 5570 (on primary pcie slot x16) > attached to windows guest
NVIDIA 7300LE (on secondary pcie x4) > on host using propietary nvidia drivers
TWO SATA DRIVES (One for the host and the other for the guest)

On the software side I'm running stock arch kernel, stock qemu, stock nvidia-304xx drivers, and pulseaudio server (very important, you'll see later).

I managed to setup my vm using virt-manager which uses the old pc-i440fx and pci-assign to passthrough the vga. The only modifications I did, is to add the vga's id to pci-stub on kernel parameters, enable the vfio-pci systemd service (which I think it is not really necessary) and disabled nested page tables. I can restart the VM as many times as I like without having any performance issue. The windows guest evens boots after forcedly killing qemu.

I'm using the VIRTIO drivers on windows which greatly improve performance and set disk cache to "none" on virt-manager.

Now here comes the biggest trouble I had while setting up this vm. The audio was terribly bad. Even after disabling nested pages tables, trying the preempt_voluntary=y and hz=1000 options in the kernel, trying the BFQ schuelder  on linux-ck, setting qemu's priority to realtime and trying both "alsa" and "pa" audio variables.

My solution was to use the PULSEAUDIO SERVER and use this audio variables "QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa". Now the audio works flawlessly.

Thank you for reading my story and hope this helps you solve your audio problems.

PD: Sorry for my english. I'm from Argentina.

Offline

#1036 2014-01-21 10:17:45

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fixed the stalling problems, at least mostly. Noticed a couple, but that may be because of slow disk IO.

Added to modprobe.d confs: options kvm-amd npt=0
Added to bootloader line: iommu=pt iommu=1

Enabled hugepages (I think this is what did it) (the number is amount of RAM given to VM divided by 2 plus a bit extra):
echo 2700 > /proc/sys/vm/nr_hugepages
add "-mem-path /dev/hugepages" to qemu parameters

However... after 20 minutes of playing, I get the same tearing issue I did before. A random amount of the side of the screen is moved to the opposite side of the screen. I had it randomly be fixed once in the past 3-4 hours of playing. A menu popped up in the game telling me what to do next and it snapped back into place. Have not been able to reproduce it, even after playing with the screen torn for an hour.

Would really, really appreciate any help in fixing this problem. The game is running pretty smooth on max settings, really impressive after so many years of barely getting games to run on lowest settings in wine, but... this issue makes games unplayable. sad

Offline

#1037 2014-01-21 12:52:43

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Would really, really appreciate any help in fixing this problem. The game is running pretty smooth on max settings, really impressive after so many years of barely getting games to run on lowest settings in wine, but... this issue makes games unplayable. sad

You could try to change the screen resolution (ingame or in the operating system).

Last edited by Flyser (2014-01-21 12:54:00)

Offline

#1038 2014-01-21 14:29:06

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

Here is some info on my setup. I have it working but i need to reboot the host to make the guest boot a 2nd time. Let me know if i can get you more info and i will sort it.

OP kernel pkgbuild

 % uname -r 3.13.0-2-mainline

Using the qemu from OP along with seabios from OP

 % pacman -Q qemu-git qemu-git 2.0.r30022.gf976b09-1

My vfio bind devices, the 0000:00:14.2 is onboard sound and works good.

cat /etc/vfio-pci.cfg DEVICES="0000:02:00.0 0000:02:00.1 0000:00:14.2"

The 01:00.0 is used on the host, with nvidia proprietary driver and is working. Pci-stub grabs the 01:00.1 because it has the same pci-stub parameter as the 02:00.1 but it is no issue for me as i dont use it for anything.

 % lspci -vv 01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 570 Rev. 2] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 1571 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 76 Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at d0000000 (64-bit, prefetchable) [size=128M] Region 3: Memory at d8000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at e000 [size=128] [virtual] Expansion ROM at f7000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: nvidia Kernel modules: nouveau, nvidia 01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1) Subsystem: eVga.com. Corp. Device 1571 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 7 Region 0: Memory at f7080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Kernel driver in use: pci-stub Kernel modules: snd_hda_intel 02:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. Device 2563 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 28 Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at d000 [size=128] Expansion ROM at f5000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia 02:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1) Subsystem: Micro-Star International Co., Ltd. Device 2563 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 11 Region 0: Memory at f5080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

Edit: did lspci -vv as root to get more info.

Last edited by mbirkis (2014-01-21 14:57:28)

Offline

#1039 2014-01-21 14:48:46

iamtakingiteasy
Member
Registered: 2014-01-21
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, i am trying to passthrough a GTX 780 Nvidia card. I had my success with Radeon HD 7870 in same setup, but it is way too much overheating, making me worry that it will flood my computer with melting soldering iron :)

(problematic setup)
CPU: Core i7 3770
Motherboard: Asus P8Z77-V LX-2
Guest GPU: Asus GTX 780 in primary (upper, one that closer to CPU) PCI-E slot.
Kernel: 3.13.0-ARCH with SRC and patches from archive in first post, built with supplied x86_64 config (with few changes to support my initrd's lvm-related setup (built-in scsi and sata drivers)).
Kernel params: pci-stub.ids=10de:1004,10de:0e1a intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1
qemu: commit 1cf892ca2689c84960b4ce4d2723b6bee453711c from git://git.qemu.org/qemu.git (last available from master branch, also i've failed to see any specific patches or check-outs in qemu-git from the first post, so i am guessing it is untouched upstream)
seabios: commit 19f789bdfd58eba2ed8fe604bbabf8df0fcc0771 from git://git.seabios.org/seabios.git (last available from master branch, built with build function sourced from PKGBUILD with minor change (not to cd to source directory, as i am already standing in there and also setting srcdir=.))

Host is currently operating without own GPU (everything is done over SSH), though i have plans in using either embedded Intel GPU or HD 7870 in secondary PCI-E slot.

I am using this script to launch my Win7 guest:

#!/bin/sh # some helper function move_device () { BDF=$1; shift DRV=$1; shift BND=$1; shift # Unbind a PCI function from its driver as necessary [ -e /sys/bus/pci/devices/$BDF/driver/unbind ] && \ echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind # Add a new slot to the PCI Backend's list vendor=$(sed 's/^0x//' /sys/bus/pci/devices/$BDF/vendor) device=$(sed 's/^0x//' /sys/bus/pci/devices/$BDF/device); echo -n $vendor $device > /sys/bus/pci/drivers/$DRV/new_id 2>/dev/null # Now that the backend is watching for the slot, bind to it [ "x$BND" = "xtrue" ] && \ echo -n $BDF > /sys/bus/pci/drivers/$DRV/bind } dmesg -C modprobe kvm_intel modprobe pci-stub modprobe vfio-pci # primary videocard -- 780 GTX move_device "0000:01:00.0" "vfio-pci" # hdmi audio card move_device "0000:01:00.1" "vfio-pci" # some crap in the same iommu group as my primary GPU move_device "0000:00:01.0" "vfio-pci" qemu-system-x86_64 \ -enable-kvm -machine type=pc,accel=kvm -enable-kvm -m 8192 -M q35 \ -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,bus=pcie.0,id=ahci \ -device ide-hd,bus=ahci.0,drive=disk \ -bios /usr/share/qemu/bios.bin \ -vnc :1 \ -drive file=/dev/virt/win7,id=disk,format=raw \ -net nic,model=rtl8139,macaddr=52:54:00:08:d9:75 \ -net tap,script=/home/user/bin/asset/qemu-ifup \ -boot menu=on -vga none

But upon launch this script, screen is just going blank and nothing happens (but win7 is actually booting, when it finished, i could login via RDP and see 43 error on GPU)

I am getting the following output in my dmesg log:

[ 35.886341] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 35.886353] pci-stub 0000:01:00.0: claimed by stub [ 35.886360] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 35.907811] VFIO - User Level meta-driver version: 0.3 [ 36.091437] tun: Universal TUN/TAP device driver, 1.6 [ 36.091440] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 36.127293] device tap0 entered promiscuous mode [ 36.127986] br0: port 1(tap0) entered forwarding state [ 36.127998] br0: port 1(tap0) entered forwarding state [ 39.077328] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 40.044509] type=1006 audit(1390315308.533:5): pid=2076 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=4 res=1 [ 41.497683] vfio-pci 0000:01:00.0: Invalid ROM contents [ 41.497799] vfio-pci 0000:01:00.0: Invalid ROM contents

Interesting thing: when i simply replace nvidia with radeon in primary PCI-E slot, everything is launching up just fine without a single change, so i guess it is nvidia-specific problem.

Any hints why it might not work with nvidia and how to possibly fix this?

Thanks.

Offline

#1040 2014-01-21 16:28:49

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
alexis_evo wrote:

Would really, really appreciate any help in fixing this problem. The game is running pretty smooth on max settings, really impressive after so many years of barely getting games to run on lowest settings in wine, but... this issue makes games unplayable. :(

You could try to change the screen resolution (ingame or in the operating system).

Nope... doesn't work. Maybe relevant, though: when capturing desktop video via OBS (Open Broadcast Software), it doesn't include the tearing. Tried changing resolution, switching monitors, switching HDMI cables, nothing will fix it short of restarting the VM.

Also, the GPU fails to 'eject' whenever I try to do so to 'safely' shut down the VM. I hear the windows eject sound, and then the entire VM locks up with very corrupted video output.

Offline

#1041 2014-01-21 16:42:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauri, mbirkis, what exactly doesn't work that requires you to reboot the host?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1042 2014-01-21 17:15:06

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

New problem since restarting after the last screen tear... now the monitor will randomly blank for 2-3 seconds. Seems like the GPU is trying to adjust something on the display, since the On Screen Menu on my monitor will not work/disappears whenever the screen blanks. It happens very often, every 5-10 seconds. After waiting 10-15 minutes, it occasionally stops doing it for 20 minutes, then starts again. Only occurs when a 3D game is at the forefront. Alt tabbing out of the game to the desktop causes it to immediately stop, and it resumes when I tab back in.

Really don't know what could be causing this...

Offline

#1043 2014-01-21 17:33:54

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

New problem since restarting after the last screen tear... now the monitor will randomly blank for 2-3 seconds. Seems like the GPU is trying to adjust something on the display, since the On Screen Menu on my monitor will not work/disappears whenever the screen blanks. It happens very often, every 5-10 seconds. After waiting 10-15 minutes, it occasionally stops doing it for 20 minutes, then starts again. Only occurs when a 3D game is at the forefront. Alt tabbing out of the game to the desktop causes it to immediately stop, and it resumes when I tab back in.

Really don't know what could be causing this...

I had a kinda similar issue, but I don't think it was reproducible.
My primary screen (1920x1080 hdmi) "blanked" like you described it and said something about "wrong input/resolution" with Civ5 up, but worked fine when on the desktop. The second screen (1280x1024 vga) still worked on the desktop at the same time.
I think I fixed the issue by messing with the settings and restarting the game. Something like, starting with 1600x900 and setting to 1920x1080 ingame…
In general, I had far less issues in Windows 8(.1) with resolution changes, multiple monitors and so on.


i'm sorry for my poor english wirting skills…

Offline

#1044 2014-01-21 19:28:15

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Lauri, mbirkis, what exactly doesn't work that requires you to reboot the host?

If i shutdown the guest, and try to start it back up i cant get the display to work.

here is some relevant dmesg from the failed guest boot:

[41945.924908] vfio-pci 0000:00:14.2: enabling device (0000 -> 0002) [41949.495172] vfio-pci 0000:02:00.0: Invalid ROM contents

Edit: I actually read the error message, and loaded the rom file as described in the OP. Guest reboot now works for me.

Last edited by mbirkis (2014-01-21 19:38:21)

Offline

#1045 2014-01-21 19:30:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbirkis wrote:
aw wrote:

Lauri, mbirkis, what exactly doesn't work that requires you to reboot the host?

If i shutdown the guest, and try to start it back up i cant get the display to work.

here is some relevant dmesg from the failed guest boot:

[41945.924908] vfio-pci 0000:00:14.2: enabling device (0000 -> 0002) [41949.495172] vfio-pci 0000:02:00.0: Invalid ROM contents

So it's not actually a guest reboot, it's a guest shutdown and restart?  Why don't you just save off the ROM to a file and use the romfile=option to the vfio-pci device?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1046 2014-01-21 19:43:02

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You are absolutely right, just as i posted the reply i thought about the romfile thing. It workes flawlesly now. Sorry about that, Guess i needed to post forum replys to get my head working.

Offline

#1047 2014-01-21 19:52:38

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iamtakingiteasy wrote:

Host is currently operating without own GPU (everything is done over SSH), though i have plans in using either embedded Intel GPU or HD 7870 in secondary PCI-E slot.

How do you prevent the GPU from becoming the primary GPU of the host (meaning: showing bios, bootloader and linux boot output)? Because if you don't, that's likely the reason for error 43.

Offline

#1048 2014-01-21 20:14:13

iamtakingiteasy
Member
Registered: 2014-01-21
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
iamtakingiteasy wrote:

Host is currently operating without own GPU (everything is done over SSH), though i have plans in using either embedded Intel GPU or HD 7870 in secondary PCI-E slot.

How do you prevent the GPU from becoming the primary GPU of the host (meaning: showing bios, bootloader and linux boot output)? Because if you don't, that's likely the reason for error 43.

Shouldn't binding to pci-stub/vfio-pci (notice kernel argument line and beginning of the script) deal with it? Besides, Radeon card passes through fine in same setup.

Last edited by iamtakingiteasy (2014-01-21 20:16:11)

Offline

#1049 2014-01-21 20:17:55

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iamtakingiteasy wrote:

Shouldn't binding to pci-stub/vfio-pci (notice kernel argument line and beginning of the script) deal with it? Besides, Radeon card passes through fine in same setup.

As far as I know, no. It definitely wouldn't work on my system.

Offline

#1050 2014-01-21 23:50:28

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

For your testing pleasure (Intel root port ACS):

https://lkml.org/lkml/2014/1/20/453

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

Yeah a new acs patch !!!

But why no x58 or x79 for the moment ?

Offline

#1051 2014-01-22 04:46:19

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ironicbadger wrote:

If iommu isn't present in the default line audio via HDMI works great, as soon as I switch it on and redo grub and reboot (regardless of whether the host is actually running a guest of not) it doesn't play audio. So weird.

Is this likely to be a motherboard issue? I'm thinking so as I have an identical software setup on another machine (in my sig) and that works flawlessly.

Hmm, wild guess, but I wonder if intel_iommu=on,igfx_off would fix the issues?

I have the same board, and for me, the DisplayPort audio doesn't work even without IOMMU enabled.

-----------

I downloaded the 3.13 final kernel, and applied those two patches (note that 'patch' reported "patch ends in middle of line"). 

Now my groups are as they were with ACS override, I think.

(It is really quite lame that SuperMicro wasted a few ports -- they could've at least given that switch a 2x upstream link...)

### Group 0 ### 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) ### Group 1 ### 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper PRO [Radeon HD 5750] [1002:68be] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58] ### Group 2 ### 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller [8086:041a] (rev 06) ### Group 3 ### 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) ### Group 4 ### 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) ### Group 5 ### 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) 00:16.3 Serial controller [0700]: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller [8086:8c3d] (rev 04) ### Group 6 ### 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-LM [8086:153a] (rev 05) ### Group 7 ### 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) ### Group 8 ### 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) ### Group 9 ### 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) ### Group 10 ### 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) ### Group 11 ### 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) ### Group 12 ### 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) ### Group 13 ### 00:1f.0 ISA bridge [0601]: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller [8086:8c56] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) 00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05) ### Group 14 ### 02:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) ### Group 15 ### 03:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) ### Group 16 ### 04:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) ### Group 17 ### 04:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) ### Group 18 ### 04:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) ### Group 19 ### 04:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) ### Group 20 ### 04:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) ### Group 21 ### 06:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa) 07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] ### Group 22 ### 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) ### Group 23 ### 0a:00.0 PCI bridge [0604]: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] [104c:823e] (rev 01) 0b:00.0 FireWire (IEEE 1394) [0c00]: Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] [104c:823f] (rev 01) ### Group 24 ### 0c:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

Now, I do still have a few remaining issues:
* Can't seem to suspend host while guest is inactive.  The system locks up, until the watchdog resets the computer.  Either that, or the VM crashes, and the GPU becomes unusable until host reboot.
* Can't get qemu to start with q35 in EFI mode.  I'll need that if I want to use the GOP mode on the new video card I've ordered.
* I really don't want to have to kill GL on the host and use VGA mode in the guest.  Fixing EFI mode will address this.
* Can't get video card to fire up as secondary; instead, I see garbage on the Radeon, and then the guest reboots. 
* Currently, Windows 8.1 boots "blind" due to being the only VGA device, but lacking x-vga=on.  Interestingly enough, the desktop works perfectly.

Last edited by DanaGoyette (2014-01-22 06:56:53)

Offline

#1052 2014-01-22 08:26:44

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
alexis_evo wrote:

New problem since restarting after the last screen tear... now the monitor will randomly blank for 2-3 seconds. Seems like the GPU is trying to adjust something on the display, since the On Screen Menu on my monitor will not work/disappears whenever the screen blanks. It happens very often, every 5-10 seconds. After waiting 10-15 minutes, it occasionally stops doing it for 20 minutes, then starts again. Only occurs when a 3D game is at the forefront. Alt tabbing out of the game to the desktop causes it to immediately stop, and it resumes when I tab back in.

Really don't know what could be causing this...

I had a kinda similar issue, but I don't think it was reproducible.
My primary screen (1920x1080 hdmi) "blanked" like you described it and said something about "wrong input/resolution" with Civ5 up, but worked fine when on the desktop. The second screen (1280x1024 vga) still worked on the desktop at the same time.
I think I fixed the issue by messing with the settings and restarting the game. Something like, starting with 1600x900 and setting to 1920x1080 ingame…
In general, I had far less issues in Windows 8(.1) with resolution changes, multiple monitors and so on.

Nope... changing resolution doesn't fix it. Haven't made any progress on fixing this.

Offline

#1053 2014-01-22 09:24:50

iamtakingiteasy
Member
Registered: 2014-01-21
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

As far as I know, no. It definitely wouldn't work on my system.

Many thanks to you! After making my integrated video primary GPU i have successfully passed through 780 GTX!

One note, i have observed, though: nvidia GPU still has to be in primary PCI-E slot, otherwise guest OS will freeze whole host upon booting.

But in primary slot everything is working great, even rebooting goes flawlessly, without any previously-reported host crashes.

Last edited by iamtakingiteasy (2014-01-22 09:25:00)

Offline

#1054 2014-01-22 17:31:20

roms
Member
Registered: 2014-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I would like to thank everybody contributing to this incredible usefull thread. I got VGA-Passthrough working with the following setup:

Hardware
Processor: Intel Core i5 4670
Mainboard: DH87RL (BIOS-Version 0323) with HD4600 onboard (used for Linux-Host)
VGA: AMD Radeon HD5450 (used for Windows-Guest)

Software
Fedora 20
Kernel 3.13.0.rc8 with Intel vag arbiter patch
Qemu from git
Seabios from git
BIOS: Using Legacy-Boot (aka. BIOS-Mode), Fastboot (minimal hardware initialisation) is off
Guest: Win7 with current Radeon driver and Virtio-driver for HDD

Kernel-Cmdline: added "pci-stub.ids=1002:68e1,1002:aa68 intel_iommu=on"
Qemu-Startup (without USB stuff):
qemu-system-x86_64 -bios bios.bin -cpu host -smp 2,sockets=1,cores=2,threads=1 -m 8192 -mem-prealloc -mem-path /dev/hugepages -M q35 -enable-kvm -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -device vfio-pci,host=01:00.1,bus=pcie.0  -drive file=/dev/md0,format=raw,if=virtio -drive file=/dev/md1,format=raw,if=virtio -parallel none -serial none -rtc clock=host,base=localtime -monitor stdio -vga none

I had to attach the VGA-Sound-Device to pcie.0, not root.1, as someone mentioned earlier in this thread. Otherwise the guest will lock up with a bluescreen (crashing in atikmpag.sys) during startup. Except the memory leak (I will try patch later), everything is working really great.

I use an USB-Switch (Delock USB 2.0 Sharing Switch) together with an USB-Hub to share one set of Mouse/Keyboard between host and guest. The USB-devices are attached to qemu with "-readconfig docs/ich9-ehci-uhci.cfg -device usb-host,bus=ehci.0,hostbus=3,hostport=2.1 -device usb-host,bus=ehci.0,hostbus=3,hostport=2.2".

Also, I use my normal "Multiboot-Windows" as the guest-OS in the VM. See here for details. I had to reactivate my Windows-License when booted in the VM because of hardware changes. Since that, I can boot the Windows installation on hardware or as VM without any problems related to license/hardware-changes. :-)

TNX everybody

Last edited by roms (2014-01-22 19:05:07)

Offline

#1055 2014-01-22 20:55:17

roms
Member
Registered: 2014-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, now I have a problem. (For my setup, please see my post above.)
I also want to pass a USB3.0-Card (VIA VL800 Chipset) to the guest.

With the ACS-patch from yesterday, the card has its own IOMMU-group. I attached pci-stub to the card and passed it to the VM with "-device vfio-pci,host=03:00.0,bus=pcie.0".
The card gets recognized by Windows but the driver won't start up (Error 10 in Device Manager).
On the host, dmesg gets spammed with:

DMAR:[fault reason 06] PTE Read access is not set [ 163.401213] dmar: DRHD: handling fault status reg 2 [ 163.401215] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 3f000f000 

I would like to understand whats probably causing the problem. The card gets initialised by the BIOS (attached keyboard does work). I tried turning on the "Fastboot-Mode" which should let the card uninitialised as I understand it. But the problem persists. So its probably not a "previous initialisation" issue?

Is it likely that a diffrent USB-Card (with different controller-chip) suffers from the same problem?

Again, TNX everybody

Some more info:
dmesg (startup only)

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.13.0-1.kvm_pt.fc20.x86_64 (roms@kang.fritz.box) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7) (GCC) ) #1 SMP Wed Jan 22 20:31:00 CET 2014 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.13.0-1.kvm_pt.fc20.x86_64 root=UUID=b7aad659-aaf7-44d7-a6a7-7458c6271719 ro vconsole.font=latarcyrheb-sun16 quiet selinux=0 pci-stub.ids=1002:68e1,1002:aa68,1106:3432 intel_iommu=on LANG=de_DE.UTF-8 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000b7f60fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000b7f61000-0x00000000b7f67fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000b7f68000-0x00000000b8dcffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000b8dd0000-0x00000000b9396fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000b9397000-0x00000000cb0fcfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000cb0fd000-0x00000000cb304fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000cb305000-0x00000000cb31afff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000cb31b000-0x00000000cb489fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000cb48a000-0x00000000cbffefff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000cbfff000-0x00000000cbffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000cd000000-0x00000000cf1fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000042fdfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: /DH87RL, BIOS RLH8710H.86A.0323.2013.1204.1726 12/04/2013 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x42fe00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-CFFFF write-protect [ 0.000000] D0000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000000000 mask 7C00000000 write-back [ 0.000000] 1 base 0400000000 mask 7FE0000000 write-back [ 0.000000] 2 base 0420000000 mask 7FF0000000 write-back [ 0.000000] 3 base 00E0000000 mask 7FE0000000 uncachable [ 0.000000] 4 base 00D0000000 mask 7FF0000000 uncachable [ 0.000000] 5 base 00CE000000 mask 7FFE000000 uncachable [ 0.000000] 6 base 00CD000000 mask 7FFF000000 uncachable [ 0.000000] 7 base 042FE00000 mask 7FFFE00000 uncachable [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: update [mem 0xcd000000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0xcc000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd720-0x000fd72f] mapped at [ffff8800000fd720] [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x0200f000, 0x0200ffff] PGTABLE [ 0.000000] BRK [0x02010000, 0x02010fff] PGTABLE [ 0.000000] BRK [0x02011000, 0x02011fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x42fc00000-0x42fdfffff] [ 0.000000] [mem 0x42fc00000-0x42fdfffff] page 2M [ 0.000000] BRK [0x02012000, 0x02012fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x42c000000-0x42fbfffff] [ 0.000000] [mem 0x42c000000-0x42fbfffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x400000000-0x42bffffff] [ 0.000000] [mem 0x400000000-0x42bffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0xb7f60fff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0xb7dfffff] page 2M [ 0.000000] [mem 0xb7e00000-0xb7f60fff] page 4k [ 0.000000] init_memory_mapping: [mem 0xb7f68000-0xb8dcffff] [ 0.000000] [mem 0xb7f68000-0xb7ffffff] page 4k [ 0.000000] [mem 0xb8000000-0xb8bfffff] page 2M [ 0.000000] [mem 0xb8c00000-0xb8dcffff] page 4k [ 0.000000] BRK [0x02013000, 0x02013fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0xb9397000-0xcb0fcfff] [ 0.000000] [mem 0xb9397000-0xb93fffff] page 4k [ 0.000000] [mem 0xb9400000-0xcaffffff] page 2M [ 0.000000] [mem 0xcb000000-0xcb0fcfff] page 4k [ 0.000000] BRK [0x02014000, 0x02014fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0xcbfff000-0xcbffffff] [ 0.000000] [mem 0xcbfff000-0xcbffffff] page 4k [ 0.000000] init_memory_mapping: [mem 0x100000000-0x3ffffffff] [ 0.000000] [mem 0x100000000-0x3ffffffff] page 1G [ 0.000000] RAMDISK: [mem 0x36ffe000-0x377f6fff] [ 0.000000] ACPI: RSDP 00000000000f0490 000024 (v02 INTEL) [ 0.000000] ACPI: XSDT 00000000cb309080 000084 (v01 INTEL DH87RL 00000143 AMI 00010013) [ 0.000000] ACPI: FACP 00000000cb315cb8 00010C (v05 INTEL DH87RL 00000143 AMI 00010013) [ 0.000000] ACPI: DSDT 00000000cb309198 00CB1F (v02 INTEL DH87RL 00000143 INTL 20120711) [ 0.000000] ACPI: FACS 00000000cb488080 000040 [ 0.000000] ACPI: APIC 00000000cb315dc8 000072 (v03 INTEL DH87RL 00000143 AMI 00010013) [ 0.000000] ACPI: FPDT 00000000cb315e40 000044 (v01 INTEL DH87RL 00000143 AMI 00010013) [ 0.000000] ACPI: LPIT 00000000cb315e88 00005C (v01 INTEL DH87RL 00000143 AMI. 00000005) [ 0.000000] ACPI: SSDT 00000000cb315ee8 000539 (v01 INTEL DH87RL 00000143 INTL 20120711) [ 0.000000] ACPI: SSDT 00000000cb316428 000AD8 (v01 INTEL DH87RL 00000143 INTL 20120711) [ 0.000000] ACPI: SSDT 00000000cb316f00 0001C7 (v01 INTEL DH87RL 00000143 INTL 20120711) [ 0.000000] ACPI: MCFG 00000000cb3170c8 00003C (v01 INTEL DH87RL 00000143 MSFT 00000097) [ 0.000000] ACPI: HPET 00000000cb317108 000038 (v01 INTEL DH87RL 00000143 AMI. 00000005) [ 0.000000] ACPI: SSDT 00000000cb317140 00036D (v01 INTEL DH87RL 00000143 INTL 20120711) [ 0.000000] ACPI: SSDT 00000000cb3174b0 003004 (v01 INTEL DH87RL 00000143 INTL 20091112) [ 0.000000] ACPI: DMAR 00000000cb31a4b8 0000B8 (v01 INTEL DH87RL 00000143 INTL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000042fdfffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x42fdfffff] [ 0.000000] NODE_DATA [mem 0x42fde7000-0x42fdfafff] [ 0.000000] [ffffea0000000000-ffffea0010bfffff] PMD -> [ffff88041f400000-ffff88042f3fffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x42fdfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0xb7f60fff] [ 0.000000] node 0: [mem 0xb7f68000-0xb8dcffff] [ 0.000000] node 0: [mem 0xb9397000-0xcb0fcfff] [ 0.000000] node 0: [mem 0xcbfff000-0xcbffffff] [ 0.000000] node 0: [mem 0x100000000-0x42fdfffff] [ 0.000000] On node 0 totalpages: 4171980 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12909 pages used for memmap [ 0.000000] DMA32 zone: 826160 pages, LIFO batch:31 [ 0.000000] Normal zone: 52216 pages used for memmap [ 0.000000] Normal zone: 3341824 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x1808 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ2 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] nr_irqs_gsi: 40 [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xb7f61000-0xb7f67fff] [ 0.000000] PM: Registered nosave memory: [mem 0xb8dd0000-0xb9396fff] [ 0.000000] PM: Registered nosave memory: [mem 0xcb0fd000-0xcb304fff] [ 0.000000] PM: Registered nosave memory: [mem 0xcb305000-0xcb31afff] [ 0.000000] PM: Registered nosave memory: [mem 0xcb31b000-0xcb489fff] [ 0.000000] PM: Registered nosave memory: [mem 0xcb48a000-0xcbffefff] [ 0.000000] PM: Registered nosave memory: [mem 0xcc000000-0xccffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xcd000000-0xcf1fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xcf200000-0xf7ffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfecfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed00000-0xfed03fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed04000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff] [ 0.000000] e820: [mem 0xcf200000-0xf7ffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:1024 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 29 pages/cpu @ffff88042fa00000 s86592 r8192 d24000 u524288 [ 0.000000] pcpu-alloc: s86592 r8192 d24000 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 4106770 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.13.0-1.kvm_pt.fc20.x86_64 root=UUID=b7aad659-aaf7-44d7-a6a7-7458c6271719 ro vconsole.font=latarcyrheb-sun16 quiet selinux=0 pci-stub.ids=1002:68e1,1002:aa68,1106:3432 intel_iommu=on LANG=de_DE.UTF-8 [ 0.000000] Intel-IOMMU: enabled [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Memory: 16334188K/16687920K available (6756K kernel code, 1067K rwdata, 2980K rodata, 1424K init, 1628K bss, 353732K reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=1024 to nr_cpu_ids=4. [ 0.000000] NR_IRQS:65792 nr_irqs:712 16 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] allocated 67108864 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.001000] tsc: Detected 3392.357 MHz processor [ 0.000001] Calibrating delay loop (skipped), value calculated using timer frequency.. 6784.71 BogoMIPS (lpj=3392357) [ 0.000003] pid_max: default: 32768 minimum: 301 [ 0.000020] Security Framework initialized [ 0.000025] SELinux: Disabled at boot. [ 0.000843] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.003137] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.004081] Mount-cache hash table entries: 256 [ 0.004197] Initializing cgroup subsys memory [ 0.004201] Initializing cgroup subsys devices [ 0.004202] Initializing cgroup subsys freezer [ 0.004203] Initializing cgroup subsys net_cls [ 0.004204] Initializing cgroup subsys blkio [ 0.004205] Initializing cgroup subsys perf_event [ 0.004206] Initializing cgroup subsys hugetlb [ 0.004223] CPU: Physical Processor ID: 0 [ 0.004223] CPU: Processor Core ID: 0 [ 0.004977] mce: CPU supports 9 MCE banks [ 0.004988] CPU0: Thermal monitoring enabled (TM1) [ 0.004996] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 tlb_flushall_shift: 6 [ 0.005064] Freeing SMP alternatives memory: 24K (ffffffff81e70000 - ffffffff81e76000) [ 0.005748] ACPI: Core revision 20131115 [ 0.010836] ACPI: All ACPI Tables successfully acquired [ 0.010985] ftrace: allocating 25894 entries in 102 pages [ 0.018761] dmar: Host address width 39 [ 0.018762] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.018767] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.018768] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.018771] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.018771] dmar: RMRR base: 0x000000cbeb6000 end: 0x000000cbec4fff [ 0.018772] dmar: RMRR base: 0x000000cd000000 end: 0x000000cf1fffff [ 0.018835] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.018836] HPET id 0 under DRHD base 0xfed91000 [ 0.018946] Enabled IRQ remapping in x2apic mode [ 0.018946] Enabling x2apic [ 0.018947] Enabled x2apic [ 0.018960] Switched APIC routing to cluster x2apic. [ 0.019369] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.029375] smpboot: CPU0: Intel(R) Core(TM) i5-4670 CPU @ 3.40GHz (fam: 06, model: 3c, stepping: 03) [ 0.029380] TSC deadline timer enabled [ 0.029386] Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. [ 0.029391] ... version: 3 [ 0.029392] ... bit width: 48 [ 0.029392] ... generic registers: 8 [ 0.029393] ... value mask: 0000ffffffffffff [ 0.029393] ... max period: 0000ffffffffffff [ 0.029394] ... fixed-purpose events: 3 [ 0.029394] ... event mask: 00000007000000ff [ 0.030256] x86: Booting SMP configuration: [ 0.044123] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.030257] .... node #0, CPUs: #1 #2 #3 [ 0.071993] x86: Booted up 1 node, 4 CPUs [ 0.071995] smpboot: Total of 4 processors activated (27138.85 BogoMIPS) [ 0.074577] devtmpfs: initialized [ 0.076236] PM: Registering ACPI NVS region [mem 0xb7f61000-0xb7f67fff] (28672 bytes) [ 0.076238] PM: Registering ACPI NVS region [mem 0xcb31b000-0xcb489fff] (1503232 bytes) [ 0.076707] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.076708] pinctrl core: initialized pinctrl subsystem [ 0.076736] RTC time: 21:09:31, date: 01/22/14 [ 0.076759] NET: Registered protocol family 16 [ 0.076806] cpuidle: using governor menu [ 0.076836] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.076837] ACPI: bus type PCI registered [ 0.076838] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.076872] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000) [ 0.076874] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820 [ 0.080985] PCI: Using configuration type 1 for base access [ 0.081586] bio: create slab <bio-0> at 0 [ 0.081657] ACPI: Added _OSI(Module Device) [ 0.081658] ACPI: Added _OSI(Processor Device) [ 0.081659] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.081659] ACPI: Added _OSI(Processor Aggregator Device) [ 0.084038] ACPI: Executed 1 blocks of module-level executable AML code [ 0.085599] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored [ 0.089522] ACPI: SSDT 00000000cb2fac18 0003D3 (v01 PmRef Cpu0Cst 00003001 INTL 20120711) [ 0.089803] ACPI: Dynamic OEM Table Load: [ 0.089804] ACPI: SSDT (null) 0003D3 (v01 PmRef Cpu0Cst 00003001 INTL 20120711) [ 0.094611] ACPI: SSDT 00000000cb2fa618 0005AA (v01 PmRef ApIst 00003000 INTL 20120711) [ 0.094933] ACPI: Dynamic OEM Table Load: [ 0.094934] ACPI: SSDT (null) 0005AA (v01 PmRef ApIst 00003000 INTL 20120711) [ 0.099544] ACPI: SSDT 00000000cb2f9d98 000119 (v01 PmRef ApCst 00003000 INTL 20120711) [ 0.099822] ACPI: Dynamic OEM Table Load: [ 0.099823] ACPI: SSDT (null) 000119 (v01 PmRef ApCst 00003000 INTL 20120711) [ 0.104858] ACPI: Interpreter enabled [ 0.104862] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20131115/hwxface-580) [ 0.104866] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20131115/hwxface-580) [ 0.104876] ACPI: (supports S0 S3 S4 S5) [ 0.104877] ACPI: Using IOAPIC for interrupt routing [ 0.104892] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.105002] ACPI: No dock devices found. [ 0.109437] ACPI: Power Resource [FN00] (off) [ 0.109480] ACPI: Power Resource [FN01] (off) [ 0.109519] ACPI: Power Resource [FN02] (off) [ 0.109561] ACPI: Power Resource [FN03] (off) [ 0.109602] ACPI: Power Resource [FN04] (off) [ 0.110047] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e]) [ 0.110051] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.110420] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 0.110747] PCI host bridge to bus 0000:00 [ 0.110749] pci_bus 0000:00: root bus resource [bus 00-3e] [ 0.110750] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] [ 0.110751] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.110752] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.110753] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff] [ 0.110754] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] [ 0.110755] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] [ 0.110756] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff] [ 0.110757] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff] [ 0.110758] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff] [ 0.110759] pci_bus 0000:00: root bus resource [mem 0xcf200000-0xfeafffff] [ 0.110764] pci 0000:00:00.0: [8086:0c00] type 00 class 0x060000 [ 0.110817] pci 0000:00:01.0: [8086:0c01] type 01 class 0x060400 [ 0.110839] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 0.110870] pci 0000:00:01.0: System wakeup disabled by ACPI [ 0.110890] pci 0000:00:02.0: [8086:0412] type 00 class 0x030000 [ 0.110898] pci 0000:00:02.0: reg 0x10: [mem 0xf7800000-0xf7bfffff 64bit] [ 0.110902] pci 0000:00:02.0: reg 0x18: [mem 0xd0000000-0xdfffffff 64bit pref] [ 0.110905] pci 0000:00:02.0: reg 0x20: [io 0xf000-0xf03f] [ 0.110950] pci 0000:00:03.0: [8086:0c0c] type 00 class 0x040300 [ 0.110955] pci 0000:00:03.0: reg 0x10: [mem 0xf7e34000-0xf7e37fff 64bit] [ 0.111019] pci 0000:00:14.0: [8086:8c31] type 00 class 0x0c0330 [ 0.111035] pci 0000:00:14.0: reg 0x10: [mem 0xf7e20000-0xf7e2ffff 64bit] [ 0.111084] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 0.111108] pci 0000:00:14.0: System wakeup disabled by ACPI [ 0.111129] pci 0000:00:16.0: [8086:8c3a] type 00 class 0x078000 [ 0.111145] pci 0000:00:16.0: reg 0x10: [mem 0xf7e3f000-0xf7e3f00f 64bit] [ 0.111198] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 0.111244] pci 0000:00:19.0: [8086:153b] type 00 class 0x020000 [ 0.111258] pci 0000:00:19.0: reg 0x10: [mem 0xf7e00000-0xf7e1ffff] [ 0.111265] pci 0000:00:19.0: reg 0x14: [mem 0xf7e3d000-0xf7e3dfff] [ 0.111272] pci 0000:00:19.0: reg 0x18: [io 0xf080-0xf09f] [ 0.111321] pci 0000:00:19.0: PME# supported from D0 D3hot D3cold [ 0.111346] pci 0000:00:19.0: System wakeup disabled by ACPI [ 0.111367] pci 0000:00:1a.0: [8086:8c2d] type 00 class 0x0c0320 [ 0.111384] pci 0000:00:1a.0: reg 0x10: [mem 0xf7e3c000-0xf7e3c3ff] [ 0.111458] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 0.111492] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 0.111513] pci 0000:00:1b.0: [8086:8c20] type 00 class 0x040300 [ 0.111525] pci 0000:00:1b.0: reg 0x10: [mem 0xf7e30000-0xf7e33fff 64bit] [ 0.111580] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold [ 0.111607] pci 0000:00:1b.0: System wakeup disabled by ACPI [ 0.111625] pci 0000:00:1c.0: [8086:8c10] type 01 class 0x060400 [ 0.111677] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 0.111695] pci 0000:00:1c.0: Enabling MPC IRBNCE [ 0.111714] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 0.111734] pci 0000:00:1c.5: [8086:8c1a] type 01 class 0x060400 [ 0.111786] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold [ 0.111803] pci 0000:00:1c.5: Enabling MPC IRBNCE [ 0.111823] pci 0000:00:1c.5: System wakeup disabled by ACPI [ 0.111846] pci 0000:00:1d.0: [8086:8c26] type 00 class 0x0c0320 [ 0.111863] pci 0000:00:1d.0: reg 0x10: [mem 0xf7e3b000-0xf7e3b3ff] [ 0.111937] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 0.111971] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 0.111992] pci 0000:00:1f.0: [8086:8c4a] type 00 class 0x060100 [ 0.112119] pci 0000:00:1f.2: [8086:8c02] type 00 class 0x010601 [ 0.112132] pci 0000:00:1f.2: reg 0x10: [io 0xf0d0-0xf0d7] [ 0.112138] pci 0000:00:1f.2: reg 0x14: [io 0xf0c0-0xf0c3] [ 0.112144] pci 0000:00:1f.2: reg 0x18: [io 0xf0b0-0xf0b7] [ 0.112150] pci 0000:00:1f.2: reg 0x1c: [io 0xf0a0-0xf0a3] [ 0.112156] pci 0000:00:1f.2: reg 0x20: [io 0xf060-0xf07f] [ 0.112162] pci 0000:00:1f.2: reg 0x24: [mem 0xf7e3a000-0xf7e3a7ff] [ 0.112192] pci 0000:00:1f.2: PME# supported from D3hot [ 0.112229] pci 0000:00:1f.3: [8086:8c22] type 00 class 0x0c0500 [ 0.112240] pci 0000:00:1f.3: reg 0x10: [mem 0xf7e39000-0xf7e390ff 64bit] [ 0.112257] pci 0000:00:1f.3: reg 0x20: [io 0xf040-0xf05f] [ 0.112334] pci 0000:01:00.0: [1002:68e1] type 00 class 0x030000 [ 0.112345] pci 0000:01:00.0: reg 0x10: [mem 0xe0000000-0xefffffff 64bit pref] [ 0.112353] pci 0000:01:00.0: reg 0x18: [mem 0xf7d20000-0xf7d3ffff 64bit] [ 0.112358] pci 0000:01:00.0: reg 0x20: [io 0xe000-0xe0ff] [ 0.112367] pci 0000:01:00.0: reg 0x30: [mem 0xf7d00000-0xf7d1ffff pref] [ 0.112396] pci 0000:01:00.0: supports D1 D2 [ 0.112410] pci 0000:01:00.0: System wakeup disabled by ACPI [ 0.112430] pci 0000:01:00.1: [1002:aa68] type 00 class 0x040300 [ 0.112441] pci 0000:01:00.1: reg 0x10: [mem 0xf7d40000-0xf7d43fff 64bit] [ 0.112496] pci 0000:01:00.1: supports D1 D2 [ 0.113542] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.113544] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.113545] pci 0000:00:01.0: bridge window [mem 0xf7d00000-0xf7dfffff] [ 0.113548] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xefffffff 64bit pref] [ 0.113594] acpiphp: Slot [1] registered [ 0.113599] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.113665] pci 0000:03:00.0: [1106:3432] type 00 class 0x0c0330 [ 0.113677] pci 0000:03:00.0: reg 0x10: [mem 0xf7c00000-0xf7c00fff] [ 0.113764] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.113782] pci 0000:03:00.0: System wakeup disabled by ACPI [ 0.115547] pci 0000:00:1c.5: PCI bridge to [bus 03] [ 0.115552] pci 0000:00:1c.5: bridge window [mem 0xf7c00000-0xf7cfffff] [ 0.115570] acpi PNP0A08:00: Disabling ASPM (FADT indicates it is unsupported) [ 0.116007] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.116038] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.116067] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.116096] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 10 11 12 14 15) [ 0.116124] ACPI: PCI Interrupt Link [LNKE] (IRQs *3 4 5 6 10 11 12 14 15) [ 0.116152] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 0.116181] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 *4 5 6 10 11 12 14 15) [ 0.116209] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.116438] ACPI: Enabled 6 GPEs in block 00 to 3F [ 0.116442] ACPI: \_SB_.PCI0: notify handler is installed [ 0.116482] Found 1 acpi root devices [ 0.116523] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.116526] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.116527] vgaarb: loaded [ 0.116528] vgaarb: bridge control possible 0000:01:00.0 [ 0.116528] vgaarb: no bridge control possible 0000:00:02.0 [ 0.116564] SCSI subsystem initialized [ 0.116584] libata version 3.00 loaded. [ 0.116593] ACPI: bus type USB registered [ 0.116601] usbcore: registered new interface driver usbfs [ 0.116605] usbcore: registered new interface driver hub [ 0.116616] usbcore: registered new device driver usb [ 0.116645] PCI: Using ACPI for IRQ routing [ 0.117798] PCI: pci_cache_line_size set to 64 bytes [ 0.117832] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff] [ 0.117833] e820: reserve RAM buffer [mem 0xb7f61000-0xb7ffffff] [ 0.117834] e820: reserve RAM buffer [mem 0xb8dd0000-0xbbffffff] [ 0.117835] e820: reserve RAM buffer [mem 0xcb0fd000-0xcbffffff] [ 0.117836] e820: reserve RAM buffer [mem 0x42fe00000-0x42fffffff] [ 0.117879] NetLabel: Initializing [ 0.117880] NetLabel: domain hash size = 128 [ 0.117880] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.117888] NetLabel: unlabeled traffic allowed by default [ 0.117912] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 0.117915] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 0.119933] Switched to clocksource hpet [ 0.122570] pnp: PnP ACPI init [ 0.122577] ACPI: bus type PNP registered [ 0.122618] system 00:00: [mem 0xfed40000-0xfed44fff] has been reserved [ 0.122620] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.122627] pnp 00:01: [dma 4] [ 0.122635] pnp 00:01: Plug and Play ACPI device, IDs PNP0200 (active) [ 0.122644] pnp 00:02: Plug and Play ACPI device, IDs INT0800 (active) [ 0.122700] pnp 00:03: Plug and Play ACPI device, IDs PNP0103 (active) [ 0.122728] pnp 00:04: Plug and Play ACPI device, IDs PNP0c04 (active) [ 0.122789] system 00:05: [io 0x0680-0x069f] has been reserved [ 0.122790] system 00:05: [io 0xffff] has been reserved [ 0.122791] system 00:05: [io 0xffff] has been reserved [ 0.122792] system 00:05: [io 0xffff] has been reserved [ 0.122793] system 00:05: [io 0x1c00-0x1cfe] has been reserved [ 0.122794] system 00:05: [io 0x1d00-0x1dfe] has been reserved [ 0.122795] system 00:05: [io 0x1e00-0x1efe] has been reserved [ 0.122796] system 00:05: [io 0x1f00-0x1ffe] has been reserved [ 0.122797] system 00:05: [io 0x1800-0x18fe] could not be reserved [ 0.122798] system 00:05: [io 0x164e-0x164f] has been reserved [ 0.122799] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.122813] pnp 00:06: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.122836] system 00:07: [io 0x1854-0x1857] has been reserved [ 0.122837] system 00:07: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 0.122888] system 00:08: [io 0x0a00-0x0a1f] has been reserved [ 0.122889] system 00:08: [io 0x0a20-0x0a23] has been reserved [ 0.122890] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.122940] system 00:09: [io 0x04d0-0x04d1] has been reserved [ 0.122941] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.123202] system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 0.123203] system 00:0a: [mem 0xfed10000-0xfed17fff] has been reserved [ 0.123204] system 00:0a: [mem 0xfed18000-0xfed18fff] has been reserved [ 0.123205] system 00:0a: [mem 0xfed19000-0xfed19fff] has been reserved [ 0.123206] system 00:0a: [mem 0xf8000000-0xfbffffff] has been reserved [ 0.123207] system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved [ 0.123209] system 00:0a: [mem 0xfed90000-0xfed93fff] could not be reserved [ 0.123210] system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved [ 0.123211] system 00:0a: [mem 0xff000000-0xffffffff] has been reserved [ 0.123212] system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved [ 0.123213] system 00:0a: [mem 0xf7fdf000-0xf7fdffff] has been reserved [ 0.123214] system 00:0a: [mem 0xf7fe0000-0xf7feffff] has been reserved [ 0.123215] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.123343] pnp: PnP ACPI: found 11 devices [ 0.123344] ACPI: bus type PNP unregistered [ 0.128997] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.128999] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.129001] pci 0000:00:01.0: bridge window [mem 0xf7d00000-0xf7dfffff] [ 0.129003] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xefffffff 64bit pref] [ 0.129005] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.129013] pci 0000:00:1c.5: PCI bridge to [bus 03] [ 0.129017] pci 0000:00:1c.5: bridge window [mem 0xf7c00000-0xf7cfffff] [ 0.129023] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7] [ 0.129024] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff] [ 0.129025] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff] [ 0.129026] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.129027] pci_bus 0000:00: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.129028] pci_bus 0000:00: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.129029] pci_bus 0000:00: resource 10 [mem 0x000dc000-0x000dffff] [ 0.129030] pci_bus 0000:00: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.129031] pci_bus 0000:00: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.129032] pci_bus 0000:00: resource 13 [mem 0xcf200000-0xfeafffff] [ 0.129033] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.129034] pci_bus 0000:01: resource 1 [mem 0xf7d00000-0xf7dfffff] [ 0.129035] pci_bus 0000:01: resource 2 [mem 0xe0000000-0xefffffff 64bit pref] [ 0.129036] pci_bus 0000:03: resource 1 [mem 0xf7c00000-0xf7cfffff] [ 0.129055] NET: Registered protocol family 2 [ 0.129179] TCP established hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.129331] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.129445] TCP: Hash tables configured (established 131072 bind 65536) [ 0.129456] TCP: reno registered [ 0.129469] UDP hash table entries: 8192 (order: 6, 262144 bytes) [ 0.129513] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes) [ 0.129573] NET: Registered protocol family 1 [ 0.129580] pci 0000:00:02.0: Boot video device [ 0.129930] PCI: CLS 64 bytes, default 64 [ 0.129963] Unpacking initramfs... [ 0.208769] Freeing initrd memory: 8164K (ffff880036ffe000 - ffff8800377f7000) [ 0.208784] DMAR: No ATSR found [ 0.208804] IOMMU 0 0xfed90000: using Queued invalidation [ 0.208805] IOMMU 1 0xfed91000: using Queued invalidation [ 0.208806] IOMMU: Setting RMRR: [ 0.208814] IOMMU: Setting identity map for device 0000:00:02.0 [0xcd000000 - 0xcf1fffff] [ 0.209020] IOMMU: Setting identity map for device 0000:00:1d.0 [0xcbeb6000 - 0xcbec4fff] [ 0.209042] IOMMU: Setting identity map for device 0000:00:1a.0 [0xcbeb6000 - 0xcbec4fff] [ 0.209059] IOMMU: Setting identity map for device 0000:00:14.0 [0xcbeb6000 - 0xcbec4fff] [ 0.209072] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.209079] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.209206] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.212375] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 0.212595] Initialise system trusted keyring [ 0.212622] audit: initializing netlink socket (disabled) [ 0.212631] type=2000 audit(1390424971.201:1): initialized [ 0.227752] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.228640] zbud: loaded [ 0.228728] VFS: Disk quotas dquot_6.5.2 [ 0.228748] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.228932] msgmni has been set to 32047 [ 0.228960] Key type big_key registered [ 0.229276] alg: No test for stdrng (krng) [ 0.229280] NET: Registered protocol family 38 [ 0.229283] Key type asymmetric registered [ 0.229284] Asymmetric key parser 'x509' registered [ 0.229308] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.229329] io scheduler noop registered [ 0.229330] io scheduler deadline registered [ 0.229343] io scheduler cfq registered (default) [ 0.229463] pcieport 0000:00:01.0: irq 42 for MSI/MSI-X [ 0.229545] pcieport 0000:00:1c.0: irq 43 for MSI/MSI-X [ 0.229595] pcieport 0000:00:1c.5: irq 44 for MSI/MSI-X [ 0.229647] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 0.229648] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 0.229649] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt [ 0.229651] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded [ 0.229660] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 0.229663] pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded [ 0.229671] pcieport 0000:00:1c.5: Signaling PME through PCIe PME interrupt [ 0.229672] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 0.229675] pcie_pme 0000:00:1c.5:pcie01: service driver pcie_pme loaded [ 0.229681] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.229688] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.229694] pci-stub: add 1002:68E1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.229697] pci-stub 0000:01:00.0: claimed by stub [ 0.229699] pci-stub: add 1002:AA68 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.229701] pci-stub 0000:01:00.1: claimed by stub [ 0.229703] pci-stub: add 1106:3432 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.229706] pci-stub 0000:03:00.0: claimed by stub [ 0.229719] intel_idle: MWAIT substates: 0x42120 [ 0.229720] intel_idle: v0.4 model 0x3C [ 0.229721] intel_idle: lapic_timer_reliable_states 0xffffffff [ 0.229812] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0 [ 0.229815] ACPI: Power Button [PWRB] [ 0.229835] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 0.229836] ACPI: Power Button [PWRF] [ 0.229870] ACPI: Fan [FAN0] (off) [ 0.229883] ACPI: Fan [FAN1] (off) [ 0.229897] ACPI: Fan [FAN2] (off) [ 0.229909] ACPI: Fan [FAN3] (off) [ 0.229920] ACPI: Fan [FAN4] (off) [ 0.230144] thermal LNXTHERM:00: registered as thermal_zone0 [ 0.230145] ACPI: Thermal Zone [TZ00] (28 C) [ 0.230250] thermal LNXTHERM:01: registered as thermal_zone1 [ 0.230251] ACPI: Thermal Zone [TZ01] (30 C) [ 0.230269] GHES: HEST is not enabled! [ 0.230295] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.230562] Non-volatile memory driver v1.3 [ 0.230563] Linux agpgart interface v0.103 [ 0.230620] ahci 0000:00:1f.2: version 3.0 [ 0.230678] ahci 0000:00:1f.2: irq 45 for MSI/MSI-X [ 0.230715] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 5 ports 6 Gbps 0xb impl SATA mode [ 0.230717] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst [ 0.234373] scsi0 : ahci [ 0.234532] scsi1 : ahci [ 0.234627] scsi2 : ahci [ 0.234744] scsi3 : ahci [ 0.234783] scsi4 : ahci [ 0.234798] ata1: SATA max UDMA/133 abar m2048@0xf7e3a000 port 0xf7e3a100 irq 45 [ 0.234801] ata2: SATA max UDMA/133 abar m2048@0xf7e3a000 port 0xf7e3a180 irq 45 [ 0.234802] ata3: DUMMY [ 0.234804] ata4: SATA max UDMA/133 abar m2048@0xf7e3a000 port 0xf7e3a280 irq 45 [ 0.234804] ata5: DUMMY [ 0.234847] libphy: Fixed MDIO Bus: probed [ 0.234890] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.234891] ehci-pci: EHCI PCI platform driver [ 0.234994] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 0.235016] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 0.235026] ehci-pci 0000:00:1a.0: debug port 2 [ 0.238923] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 0.238933] ehci-pci 0000:00:1a.0: irq 16, io mem 0xf7e3c000 [ 0.244012] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 0.244037] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.244038] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.244039] usb usb1: Product: EHCI Host Controller [ 0.244040] usb usb1: Manufacturer: Linux 3.13.0-1.kvm_pt.fc20.x86_64 ehci_hcd [ 0.244041] usb usb1: SerialNumber: 0000:00:1a.0 [ 0.244179] hub 1-0:1.0: USB hub found [ 0.244183] hub 1-0:1.0: 2 ports detected [ 0.244295] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 0.244402] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 0.244412] ehci-pci 0000:00:1d.0: debug port 2 [ 0.248327] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 0.248336] ehci-pci 0000:00:1d.0: irq 23, io mem 0xf7e3b000 [ 0.254048] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 0.254093] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.254095] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.254096] usb usb2: Product: EHCI Host Controller [ 0.254097] usb usb2: Manufacturer: Linux 3.13.0-1.kvm_pt.fc20.x86_64 ehci_hcd [ 0.254098] usb usb2: SerialNumber: 0000:00:1d.0 [ 0.254235] hub 2-0:1.0: USB hub found [ 0.254239] hub 2-0:1.0: 2 ports detected [ 0.254320] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.254321] ohci-pci: OHCI PCI platform driver [ 0.254328] uhci_hcd: USB Universal Host Controller Interface driver [ 0.254419] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 0.254523] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 3 [ 0.254621] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported [ 0.254643] xhci_hcd 0000:00:14.0: irq 46 for MSI/MSI-X [ 0.254690] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.254691] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.254692] usb usb3: Product: xHCI Host Controller [ 0.254693] usb usb3: Manufacturer: Linux 3.13.0-1.kvm_pt.fc20.x86_64 xhci_hcd [ 0.254694] usb usb3: SerialNumber: 0000:00:14.0 [ 0.254758] hub 3-0:1.0: USB hub found [ 0.254772] hub 3-0:1.0: 14 ports detected [ 0.256973] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 0.257014] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 4 [ 0.257038] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003 [ 0.257039] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.257040] usb usb4: Product: xHCI Host Controller [ 0.257041] usb usb4: Manufacturer: Linux 3.13.0-1.kvm_pt.fc20.x86_64 xhci_hcd [ 0.257042] usb usb4: SerialNumber: 0000:00:14.0 [ 0.257103] hub 4-0:1.0: USB hub found [ 0.257113] hub 4-0:1.0: 6 ports detected [ 0.262089] usbcore: registered new interface driver usbserial [ 0.262097] usbcore: registered new interface driver usbserial_generic [ 0.262100] usbserial: USB Serial support registered for generic [ 0.262118] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 0.264373] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.264377] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.264518] mousedev: PS/2 mouse device common for all mice [ 0.264795] rtc_cmos 00:06: RTC can wake from S4 [ 0.264899] rtc_cmos 00:06: rtc core: registered rtc_cmos as rtc0 [ 0.264924] rtc_cmos 00:06: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 0.264961] device-mapper: uevent: version 1.0.3 [ 0.265002] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com [ 0.265011] Intel P-state driver initializing. [ 0.265018] Intel pstate controlling: cpu 0 [ 0.265027] Intel pstate controlling: cpu 1 [ 0.265033] Intel pstate controlling: cpu 2 [ 0.265041] Intel pstate controlling: cpu 3 [ 0.265180] hidraw: raw HID events driver (C) Jiri Kosina [ 0.265221] usbcore: registered new interface driver usbhid [ 0.265222] usbhid: USB HID core driver [ 0.265246] drop_monitor: Initializing network drop monitor service [ 0.265286] ip_tables: (C) 2000-2006 Netfilter Core Team [ 0.265409] TCP: cubic registered [ 0.265410] Initializing XFRM netlink socket [ 0.265457] NET: Registered protocol family 10 [ 0.265541] mip6: Mobile IPv6 [ 0.265542] NET: Registered protocol family 17 [ 0.265687] Loading compiled-in X.509 certificates [ 0.266218] Loaded X.509 cert 'Fedora kernel signing key: c0aca5f079cf43ba420c1ae14c73a70e0b4bc12e' [ 0.266225] registered taskstats version 1 [ 0.266887] Magic number: 14:466:197 [ 0.266912] acpi device:52: hash matches [ 0.266923] acpi PNP0100:00: hash matches [ 0.266967] rtc_cmos 00:06: setting system clock to 2014-01-22 21:09:31 UTC (1390424971) [ 0.266989] PM: Hibernation image not present or could not be loaded. [ 0.539175] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.539193] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.539209] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 0.539660] ata1.00: ATA-9: M4-CT128M4SSD2, 070H, max UDMA/100 [ 0.539663] ata1.00: 250069680 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 0.539838] ata4.00: ATAPI: TSSTcorp CDDVDW SH-S203P, SB01, max UDMA/100 [ 0.540020] ata2.00: ATA-9: ST2000DM001-1CH164, CC29, max UDMA/133 [ 0.540022] ata2.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 0.540155] ata1.00: configured for UDMA/100 [ 0.540267] scsi 0:0:0:0: Direct-Access ATA M4-CT128M4SSD2 070H PQ: 0 ANSI: 5 [ 0.540352] sd 0:0:0:0: [sda] 250069680 512-byte logical blocks: (128 GB/119 GiB) [ 0.540372] sd 0:0:0:0: [sda] Write Protect is off [ 0.540373] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 0.540377] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 0.540380] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.540640] sda: sda1 sda2 sda3 [ 0.540729] ata4.00: configured for UDMA/100 [ 0.540772] sd 0:0:0:0: [sda] Attached SCSI disk [ 0.540844] ata2.00: configured for UDMA/133 [ 0.540940] scsi 1:0:0:0: Direct-Access ATA ST2000DM001-1CH1 CC29 PQ: 0 ANSI: 5 [ 0.540984] sd 1:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [ 0.540985] sd 1:0:0:0: [sdb] 4096-byte physical blocks [ 0.540999] sd 1:0:0:0: [sdb] Write Protect is off [ 0.541000] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 0.541005] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.541013] sd 1:0:0:0: Attached scsi generic sg1 type 0 [ 0.541678] scsi 3:0:0:0: CD-ROM TSSTcorp CDDVDW SH-S203P SB01 PQ: 0 ANSI: 5 [ 0.543979] sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray [ 0.543981] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 0.544059] sr 3:0:0:0: Attached scsi CD-ROM sr0 [ 0.544128] sr 3:0:0:0: Attached scsi generic sg2 type 5 [ 0.546190] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 0.586653] sdb: sdb1 sdb2 sdb3 sdb4 sdb5 [ 0.586948] sd 1:0:0:0: [sdb] Attached SCSI disk [ 0.587501] Freeing unused kernel memory: 1424K (ffffffff81d0c000 - ffffffff81e70000) [ 0.587502] Write protecting the kernel read-only data: 12288k [ 0.589531] Freeing unused kernel memory: 1424K (ffff88000169c000 - ffff880001800000) [ 0.591005] Freeing unused kernel memory: 1116K (ffff880001ae9000 - ffff880001c00000) [ 0.592682] systemd[1]: systemd 208 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ) [ 0.592770] systemd[1]: Running in initial RAM disk. [ 0.592797] systemd[1]: Set hostname to <kang.fritz.box>. [ 0.593142] random: systemd urandom read with 30 bits of entropy available [ 0.595345] systemd[1]: Expecting device dev-disk-by\x2duuid-b7aad659\x2daaf7\x2d44d7\x2da6a7\x2d7458c6271719.device... [ 0.595354] systemd[1]: Starting -.slice. [ 0.595426] systemd[1]: Created slice -.slice. [ 0.595434] systemd[1]: Starting System Slice. [ 0.595465] systemd[1]: Created slice System Slice. [ 0.595470] systemd[1]: Starting Slices. [ 0.595475] systemd[1]: Reached target Slices. [ 0.595480] systemd[1]: Starting Timers. [ 0.595486] systemd[1]: Reached target Timers. [ 0.595492] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch. [ 0.595511] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 0.595516] systemd[1]: Starting Paths. [ 0.595522] systemd[1]: Reached target Paths. [ 0.595527] systemd[1]: Starting Journal Socket. [ 0.595555] systemd[1]: Listening on Journal Socket. [ 0.595628] systemd[1]: Starting dracut cmdline hook... [ 0.595782] systemd[1]: Starting Journal Service... [ 0.595947] systemd[1]: Started Journal Service. [ 0.599122] systemd-journald[94]: Vacuuming done, freed 0 bytes [ 0.631447] systemd-udevd[152]: starting version 208 [ 0.660525] usb 1-1: New USB device found, idVendor=8087, idProduct=8008 [ 0.660527] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 0.660671] hub 1-1:1.0: USB hub found [ 0.660771] hub 1-1:1.0: 6 ports detected [ 0.763233] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 0.877668] usb 2-1: New USB device found, idVendor=8087, idProduct=8000 [ 0.877671] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 0.877886] hub 2-1:1.0: USB hub found [ 0.877998] hub 2-1:1.0: 8 ports detected [ 1.032409] usb 3-2: new low-speed USB device number 2 using xhci_hcd [ 1.039814] random: nonblocking pool is initialized [ 1.047621] usb 3-2: New USB device found, idVendor=2101, idProduct=0201 [ 1.047623] usb 3-2: New USB device strings: Mfr=1, Product=0, SerialNumber=0 [ 1.047625] usb 3-2: Manufacturer: USB Device [ 1.047715] usb 3-2: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.050757] hid-generic 0003:2101:0201.0001: hiddev0,hidraw0: USB HID v1.00 Device [USB Device] on usb-0000:00:14.0-2/input0 [ 1.171598] PM: Starting manual resume from disk [ 1.171601] PM: Hibernation image partition 8:17 present [ 1.171602] PM: Looking for hibernation image. [ 1.171852] PM: Image not found (code -22) [ 1.171854] PM: Hibernation image not present or could not be loaded. [ 1.178678] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null) [ 1.203430] usb 3-10: new high-speed USB device number 3 using xhci_hcd [ 1.213413] tsc: Refined TSC clocksource calibration: 3392.145 MHz [ 1.215638] usb 3-10: New USB device found, idVendor=05e3, idProduct=0606 [ 1.215640] usb 3-10: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 1.215642] usb 3-10: Product: USB2.0 Hub [ 1.216078] hub 3-10:1.0: USB hub found [ 1.216312] hub 3-10:1.0: 4 ports detected [ 1.218768] systemd-journald[94]: Received SIGTERM [ 1.262032] systemd[1]: RTC configured in localtime, applying delta of 60 minutes to system time. [ 1.320018] systemd-readahead[258]: Bumped block_nr parameter of 8:0 to 20480. This is a temporary hack and should be removed one day. [ 1.334157] systemd-journald[266]: Vacuuming done, freed 0 bytes [ 1.366121] EXT4-fs (sda3): re-mounted. Opts: (null) [ 1.384640] systemd-udevd[310]: starting version 208 [ 1.413552] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 1.418043] [drm] Initialized drm 1.1.0 20060810 [ 1.419236] pps_core: LinuxPPS API ver. 1 registered [ 1.419237] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 1.419932] PTP clock support registered [ 1.421305] mei_me 0000:00:16.0: irq 47 for MSI/MSI-X [ 1.422566] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k [ 1.422567] e1000e: Copyright(c) 1999 - 2013 Intel Corporation. [ 1.423164] e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 1.423180] e1000e 0000:00:19.0: irq 48 for MSI/MSI-X [ 1.432277] microcode: CPU0 sig=0x306c3, pf=0x2, revision=0x16 [ 1.433379] microcode: CPU1 sig=0x306c3, pf=0x2, revision=0x16 [ 1.433396] microcode: CPU2 sig=0x306c3, pf=0x2, revision=0x16 [ 1.433411] microcode: CPU3 sig=0x306c3, pf=0x2, revision=0x16 [ 1.434289] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 1.439274] alg: No test for crc32 (crc32-pclmul) [ 1.481722] usb 3-10.1: new low-speed USB device number 4 using xhci_hcd [ 1.487657] Adding 4095996k swap on /dev/sdb1. Priority:-1 extents:1 across:4095996k FS [ 1.496518] usb 3-10.1: New USB device found, idVendor=046a, idProduct=0011 [ 1.496521] usb 3-10.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 1.496585] usb 3-10.1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.499848] input: HID 046a:0011 as /devices/pci0000:00/0000:00:14.0/usb3/3-10/3-10.1/3-10.1:1.0/input/input5 [ 1.499898] hid-generic 0003:046A:0011.0002: input,hidraw1: USB HID v1.10 Keyboard [HID 046a:0011] on usb-0000:00:14.0-10.1/input0 [ 1.562747] usb 3-10.2: new full-speed USB device number 5 using xhci_hcd [ 1.576796] usb 3-10.2: New USB device found, idVendor=046d, idProduct=c049 [ 1.576798] usb 3-10.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.576800] usb 3-10.2: Product: USB Gaming Mouse [ 1.576800] usb 3-10.2: Manufacturer: Logitech [ 1.576873] usb 3-10.2: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.579061] input: Logitech USB Gaming Mouse as /devices/pci0000:00/0000:00:14.0/usb3/3-10/3-10.2/3-10.2:1.0/input/input6 [ 1.579140] hid-generic 0003:046D:C049.0003: input,hidraw2: USB HID v1.11 Mouse [Logitech USB Gaming Mouse] on usb-0000:00:14.0-10.2/input0 [ 1.580822] hid-generic 0003:046D:C049.0004: hiddev0,hidraw3: USB HID v1.11 Device [Logitech USB Gaming Mouse] on usb-0000:00:14.0-10.2/input1 [ 1.592721] e1000e 0000:00:19.0 eth0: registered PHC clock [ 1.592724] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) 00:22:4d:a9:d9:80 [ 1.592726] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection [ 1.592764] e1000e 0000:00:19.0 eth0: MAC: 11, PHY: 12, PBA No: FFFFFF-0FF [ 1.593063] ACPI Warning: 0x0000000000001828-0x000000000000182f SystemIO conflicts with Region \PMIO 1 (20131115/utaddress-251) [ 1.593067] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 1.593070] ACPI Warning: 0x0000000000001c30-0x0000000000001c3f SystemIO conflicts with Region \GPRL 1 (20131115/utaddress-251) [ 1.593071] ACPI Warning: 0x0000000000001c30-0x0000000000001c3f SystemIO conflicts with Region \GPR_ 2 (20131115/utaddress-251) [ 1.593073] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 1.593073] ACPI Warning: 0x0000000000001c00-0x0000000000001c2f SystemIO conflicts with Region \GPRL 1 (20131115/utaddress-251) [ 1.593075] ACPI Warning: 0x0000000000001c00-0x0000000000001c2f SystemIO conflicts with Region \GPR_ 2 (20131115/utaddress-251) [ 1.593076] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 1.593077] lpc_ich: Resource conflict(s) found affecting gpio_ich [ 1.593226] ACPI Warning: 0x000000000000f040-0x000000000000f05f SystemIO conflicts with Region \_SB_.PCI0.SBUS.SMBI 1 (20131115/utaddress-251) [ 1.593229] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 1.594019] [drm] Memory usable by graphics device = 2048M [ 1.594322] iTCO_vendor_support: vendor-support=0 [ 1.595009] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.10 [ 1.595038] iTCO_wdt: Found a Lynx Point TCO device (Version=2, TCOBASE=0x1860) [ 1.595134] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) [ 1.618628] i915 0000:00:02.0: irq 49 for MSI/MSI-X [ 1.618632] systemd-udevd[312]: renamed network interface eth0 to em1 [ 1.618635] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 1.618636] [drm] Driver supports precise vblank timestamp query. [ 1.622329] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) [ 1.714705] fbcon: inteldrmfb (fb0) is primary device [ 1.811304] Console: switching to colour frame buffer device 240x75 [ 1.814843] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device [ 1.814845] i915 0000:00:02.0: registered panic notifier [ 1.814847] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io [ 1.821426] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 1.821708] acpi device:53: registered as cooling_device9 [ 1.821750] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/LNXVIDEO:00/input/input7 [ 1.821815] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0 [ 1.821899] ALSA sound/pci/hda/hda_i915.c:58 HDA driver get symbol successfully from i915 module [ 1.821920] snd_hda_intel 0000:00:03.0: irq 50 for MSI/MSI-X [ 1.822030] snd_hda_intel 0000:00:1b.0: irq 51 for MSI/MSI-X [ 1.828628] input: HDA Intel HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.0/sound/card0/input9 [ 1.828755] input: HDA Intel HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.0/sound/card0/input8 [ 1.831579] ALSA sound/pci/hda/patch_realtek.c:419 SKU: Nid=0x1d sku_cfg=0x4016f601 [ 1.831580] ALSA sound/pci/hda/patch_realtek.c:421 SKU: port_connectivity=0x1 [ 1.831581] ALSA sound/pci/hda/patch_realtek.c:422 SKU: enable_pcbeep=0x1 [ 1.831582] ALSA sound/pci/hda/patch_realtek.c:423 SKU: check_sum=0x00000006 [ 1.831583] ALSA sound/pci/hda/patch_realtek.c:424 SKU: customization=0x000000f6 [ 1.831583] ALSA sound/pci/hda/patch_realtek.c:425 SKU: external_amp=0x0 [ 1.831584] ALSA sound/pci/hda/patch_realtek.c:426 SKU: platform_type=0x0 [ 1.831585] ALSA sound/pci/hda/patch_realtek.c:427 SKU: swap=0x0 [ 1.831585] ALSA sound/pci/hda/patch_realtek.c:428 SKU: override=0x1 [ 1.831992] ALSA sound/pci/hda/hda_auto_parser.c:393 autoconfig: line_outs=4 (0x14/0x15/0x16/0x18/0x0) type:line [ 1.831993] ALSA sound/pci/hda/hda_auto_parser.c:397 speaker_outs=0 (0x0/0x0/0x0/0x0/0x0) [ 1.831994] ALSA sound/pci/hda/hda_auto_parser.c:401 hp_outs=1 (0x1b/0x0/0x0/0x0/0x0) [ 1.831995] ALSA sound/pci/hda/hda_auto_parser.c:402 mono: mono_out=0x0 [ 1.831996] ALSA sound/pci/hda/hda_auto_parser.c:405 dig-out=0x11/0x1e [ 1.831996] ALSA sound/pci/hda/hda_auto_parser.c:406 inputs: [ 1.831998] ALSA sound/pci/hda/hda_auto_parser.c:410 Mic=0x19 [ 1.831998] ALSA sound/pci/hda/hda_auto_parser.c:410 Line=0x1a [ 1.831999] ALSA sound/pci/hda/patch_realtek.c:491 realtek: No valid SSID, checking pincfg 0x4016f601 for NID 0x1d [ 1.832000] ALSA sound/pci/hda/patch_realtek.c:507 realtek: Enabling init ASM_ID=0xf601 CODEC_ID=10ec0892 [ 1.843751] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card1/input14 [ 1.843793] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card1/input13 [ 1.843822] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card1/input12 [ 1.843851] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card1/input11 [ 1.843882] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card1/input10 [ 1.947632] EXT4-fs (sdb3): mounted filesystem with ordered data mode. Opts: (null) [ 1.950794] systemd-journald[266]: Received request to flush runtime journal from PID 1 [ 2.004338] type=1305 audit(1390421373.235:2): audit_pid=409 old=0 auid=4294967295 ses=4294967295 res=1 [ 2.214030] Switched to clocksource tsc [ 2.714323] [drm] Enabling RC6 states: RC6 on, RC6p off, RC6pp off [ 5.619000] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 5.625750] ip6_tables: (C) 2000-2006 Netfilter Core Team [ 5.641712] Ebtables v2.0 registered [ 5.647645] Bridge firewalling registered [ 5.859069] cfg80211: Calling CRDA to update world regulatory domain [ 5.860002] cfg80211: World regulatory domain updated: [ 5.860004] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp) [ 5.860005] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 5.860006] cfg80211: (2457000 KHz - 2482000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 5.860007] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm) [ 5.860007] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 5.860008] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 6.221398] e1000e 0000:00:19.0: irq 48 for MSI/MSI-X [ 6.321607] e1000e 0000:00:19.0: irq 48 for MSI/MSI-X [ 6.321694] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready [ 6.489112] Bluetooth: Core ver 2.17 [ 6.489123] NET: Registered protocol family 31 [ 6.489124] Bluetooth: HCI device and connection manager initialized [ 6.489130] Bluetooth: HCI socket layer initialized [ 6.489132] Bluetooth: L2CAP socket layer initialized [ 6.489134] Bluetooth: SCO socket layer initialized [ 6.490888] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 [ 6.490890] Bluetooth: BNEP filters: protocol multicast [ 6.490893] Bluetooth: BNEP socket layer initialized [ 7.830807] e1000e: em1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx [ 7.830811] e1000e 0000:00:19.0 em1: 10/100 speed: disabling TSO [ 7.830839] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [ 8.007471] e1000e 0000:00:19.0 em1: changing MTU from 1500 to 1420 [ 9.734615] e1000e: em1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx [ 9.734618] e1000e 0000:00:19.0 em1: 10/100 speed: disabling TSO

IOMMU-Groups

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 10 ### 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) ### Group 12 ### 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) ### Group 13 ### 03:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03)

lspci -vv -d 1106:3432

03:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 17 Region 0: Memory at f7c00000 (32-bit, non-prefetchable) [size=4K] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D3 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c4] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <16us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <2us, L1 <16us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP+ Rollover+ Timeout+ NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: vfio-pci

Offline

#1056 2014-01-22 21:10:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

roms wrote:

OK, now I have a problem. (For my setup, please see my post above.)
I also want to pass a USB3.0-Card (VIA VL800 Chipset) to the guest.

With the ACS-patch from yesterday, the card has its own IOMMU-group. I attached pci-stub to the card and passed it to the VM with "-device vfio-pci,host=03:00.0,bus=pcie.0".
The card gets recognized by Windows but the driver won't start up (Error 10 in Device Manager).
On the host, dmesg gets spammed with:

DMAR:[fault reason 06] PTE Read access is not set [ 163.401213] dmar: DRHD: handling fault status reg 2 [ 163.401215] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 3f000f000 

I would like to understand whats probably causing the problem. The card gets initialised by the BIOS (attached keyboard does work). I tried turning on the "Fastboot-Mode" which should let the card uninitialised as I understand it. But the problem persists. So its probably not a "previous initialisation" issue?

Is it likely that a diffrent USB-Card (with different controller-chip) suffers from the same problem?

It looks like the IOMMU did its job. For some reason the USB card is trying to read from an address just below 16G.  Since you've got a 16G host, you're likely not creating a VM large enough that it should be accessing those kinds of addresses.  I've tried both Etron and Renesas USB3 controllers.  See back at post #876 for the models.

Last edited by aw (2014-01-22 21:10:50)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1057 2014-01-22 22:28:44

roms
Member
Registered: 2014-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It looks like the IOMMU did its job. For some reason the USB card is trying to read from an address just below 16G.  Since you've got a 16G host, you're likely not creating a VM large enough that it should be accessing those kinds of addresses.  I've tried both Etron and Renesas USB3 controllers.  See back at post #876 for the models.

OK, I will try to get one of these then and report the result here. Thank you for your work, aw!

Offline

#1058 2014-01-23 05:36:54

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Speaking of "iommu doing its job", using VT-d is an awesome way to protect against Firewire DMA attacks.  If I pass the Firewire controller to a guest, firewire memory dumps get the guest's memory.  If I leave it on the host, with VT-d enabled, the host nearly locks up (spewing DMAR errors), and the dumper gets 4 gigabytes of 0xff.

Offline

#1059 2014-01-23 18:44:09

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

For your testing pleasure (Intel root port ACS):

https://lkml.org/lkml/2014/1/20/453

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

I experience the same since i upgraded from kernel 3.11 to 3.12 or 3.13.
Host freezes (no more ssh login possible - ping still working; at least MagicSysRq-Keys still worked) as soon as the guest starts up the second time (complete guest shutdown, i.e. virsh shutdown, then virsh start again).

kernel is 3.13.0, qemu is git from 16.1.2014 (same qemu compile works flawless with 3.11).

lspci -vvv is here: http://www.pastebin.ca/2579449

My guess is, some of these patches are missing upstream: http://www.pastebin.ca/2579452
They were the ones needed to get it working with 3.11, and when i try to apply them to 3.13 (see here: http://pastebin.ca/2579454) they of course hunk, but _some_ apply, so i guess something is missing ?

Thanks for your support!

Greetings

K.

Last edited by kaeptnb (2014-01-23 18:45:16)

Offline

#1060 2014-01-24 04:33:56

dalingrin
Member
Registered: 2009-03-18
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have two NVIDIA GTX 670 cards and they have the same ids. If I pass this id to pci-stub then both cards are inaccessible. What is the best way to get around this so I can keep one card for the host?

Offline

#1061 2014-01-24 13:17:11

kugel
Member
Registered: 2014-01-24
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
mastersplinter777 wrote:

Hi! I'm also interested in passing thru my VGA to VM. But I get an error 12 (Not enough system resources) and can't solve this problem. My HW is: ASRock Z77 Extreme 6 (BIOS 2.70), Intel Core i5 3470, GPUS: Intel HD 2500 (host), AMD Sapphire Radeon 6750 (68bf), ATI Radeon 4850 (9442).

Thx for your help! smile

A lot of people have reported problems using the intel igp as the host boot device you might want try to disable it and use one of your radeon cards for the host

I get error 12 as well. Is there no other workaround available? Because it won't work for me.

My setup: Thinkpad x230 laptop with integrated HD 4000 and NVIDIA eGPU (GeForce 650 Ti) attached via ExpressCard. I try to pass the eGPU to a Windows VM. The eGPU does not get enabled because of error 12. When booting with -vga none I see now output (how can I tell it actually boots?). When omitting this option Windows displays on the virtual GPU but the device manager shows the aforementioned error 12 for the nvidia card.

I followed your first post into detail and run Linux 3.12.6 with the pcie_acs_override patch applied.

Offline

#1062 2014-01-25 15:38:36

roms
Member
Registered: 2014-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It looks like the IOMMU did its job. For some reason the USB card is trying to read from an address just below 16G.  Since you've got a 16G host, you're likely not creating a VM large enough that it should be accessing those kinds of addresses.  I've tried both Etron and Renesas USB3 controllers.  See back at post #876 for the models.

I replaced the VIA-based card with one that has a NEC/Renasas controller. Now everything works as expected. :-)

lspci -nn:
03:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)

Offline

#1063 2014-01-25 18:55:25

toma222
Member
From: France
Registered: 2007-05-17
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I used my Radeon 7850 in a Windows 7/8/8.1 guest with pci-assign for months (with the only issue of the host crash in case of guest reboot).

Now, I want to use it with a Linux guest (Ubuntu) with vfio-pci, to make a kind of virtual Steam Machine + XBMC.

I have tried :
- secondary pasthrough (with a cirrus emulated graphic card) with vfio-pci : the guest start, the Radeon is detected and I have the display on the HDMI output, Unity start, I can play video, but when I try to use 3D (for example with fglrxinfo or Steam), I have only segmentation fault.
- primary passtrough with x-vga=on : I have built the 3.13 kernel (with options and patches from first post) and qemu from git, but I have exactly the same behaviour.

I have also tried with an old Radeon 6850 in secondary passthrough and vfio-pci, and it work quite well (except  some strange performance in Serious Sam 3).

Do you have an idea why I can't get the Radeon 7850 working normally ?

My host use a Core i7 4770S and is on Debian.

Thanks.

Offline

#1064 2014-01-25 22:08:45

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Here's some information that might be useful to someone; I'd post it as a bug but I suspect that it's either an ASUS bios bug, which they will cheerfully ignore (particularly as vt-d support is not yet supported in an official bios for the RIVE with an IVy-E preocessor), or an AMD catalyst bug which only appears when using intel IOMMU, which I suspect might not get much attention either...  Any suggestions on where to post it or what to do would be most welcome.

My passthrough is an Nvidia K2000 to Windows, which works fine, no errors in Device Manager, my host has an AMD HD 7970.  Using the radeon drivers, everything works fine, other than the fact that 3D acceleration is software only and slow as there is no "Southern Island" hardware support yet in the radeon driver.  Using the latest beta ati-catalyst 13.11beta9.95 on kernel 3.12.1, the driver builds OK, but running X leads to a black screen and hung keyboard. One can ssh into the system.  Here is the syslog:

Jan 23 07:09:36 animals smbd[1150]: [2014/01/23 07:09:36.503256,  0] ../lib/util/pidfile.c:153(pidfile_unlink)
Jan 23 07:09:36 animals smbd[1150]:   Failed to delete pidfile /var/run/smbd.pid. Error was No such file or directory
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: RSDP 00000000ac450000 00024 (v02 ALASKA)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: XSDT 00000000ac450078 00064 (v01 ALASKA    A M I 01072009 AMI  00010013)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: FACP 00000000ac45a1f8 0010C (v05 ALASKA    A M I 01072009 AMI  00010013)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: DSDT 00000000ac450170 0A088 (v02 ALASKA    A M I 00000016 INTL 20051117)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: FACS 00000000ac746080 00040
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: APIC 00000000ac45a308 00100 (v03 ALASKA    A M I 01072009 AMI  00010013)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: FPDT 00000000ac45a408 00044 (v01 ALASKA    A M I 01072009 AMI  00010013)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: MCFG 00000000ac45a450 0003C (v01 ALASKA OEMMCFG. 01072009 MSFT 00000097)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: HPET 00000000ac45a490 00038 (v01 ALASKA    A M I 01072009 AMI. 00000005)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: DMAR 00000000ac527680 000B4 (v01 A M I   OEMDMAR 00000001 INTL 00000001)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: SSDT 00000000ac45a520 CD128 (v02  INTEL    CpuPm 00004000 INTL 20051117)
Jan 23 07:10:41 animals kernel: [    0.000000] ACPI: BGRT 00000000ac527648 00038 (v00 ALASKA    A M I 01072009 AMI  00010013)
Jan 23 07:10:41 animals kernel: [    0.000000] Zone ranges:
Jan 23 07:10:41 animals kernel: [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
Jan 23 07:10:41 animals kernel: [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
Jan 23 07:10:41 animals kernel: [    0.000000]   Normal   [mem 0x100000000-0x84fffffff]
Jan 23 07:10:41 animals kernel: [    0.000000] Movable zone start for each node
Jan 23 07:10:41 animals kernel: [    0.000000] Early memory node ranges
Jan 23 07:10:41 animals kernel: [    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
Jan 23 07:10:41 animals kernel: [    0.000000]   node   0: [mem 0x00100000-0xabe82fff]
Jan 23 07:10:41 animals kernel: [    0.000000]   node   0: [mem 0xad76f000-0xad76ffff]
Jan 23 07:10:41 animals kernel: [    0.000000]   node   0: [mem 0xad7f6000-0xadc37fff]
Jan 23 07:10:41 animals kernel: [    0.000000]   node   0: [mem 0xadff4000-0xadffffff]
Jan 23 07:10:41 animals kernel: [    0.000000]   node   0: [mem 0x100000000-0x84fffffff]
Jan 23 07:10:41 animals kernel: [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8241995
Jan 23 07:10:41 animals kernel: [    0.000000] Memory: 32607824K/33491396K available (6322K kernel code, 734K rwdata, 2264K rodata, 1184K init, 968K bss, 883572K reserved)
Jan 23 07:10:41 animals kernel: [    0.016691] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
Jan 23 07:10:41 animals kernel: [    0.016691] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
Jan 23 07:10:41 animals kernel: [    0.036478] ACPI: All ACPI Tables successfully acquired
Jan 23 07:10:41 animals kernel: [    0.841857] raid6: sse2x1    9308 MB/s
Jan 23 07:10:41 animals kernel: [    0.858859] raid6: sse2x2   11554 MB/s
Jan 23 07:10:41 animals kernel: [    0.875862] raid6: sse2x4   13386 MB/s
Jan 23 07:10:41 animals kernel: [    0.875864] raid6: using algorithm sse2x4 (13386 MB/s)
Jan 23 07:10:41 animals kernel: [    0.875865] raid6: using ssse3x2 recovery algorithm
Jan 23 07:10:41 animals kernel: [    0.888491] ACPI: Executed 1 blocks of module-level executable AML code
Jan 23 07:10:41 animals kernel: [    0.971269] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130725/hwxface-571)
Jan 23 07:10:41 animals kernel: [    0.971273] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130725/hwxface-571)
Jan 23 07:10:41 animals kernel: [    0.978047] ACPI: \_SB_.PCI0.IOH_: can't evaluate _ADR (0x5)
Jan 23 07:10:41 animals kernel: [    0.978049] ACPI: \_SB_.PCI0.VTDR: can't evaluate _ADR (0x5)
Jan 23 07:10:41 animals kernel: [    0.978056] ACPI: \_SB_.PCI0.PCH_: can't evaluate _ADR (0x5)
Jan 23 07:10:41 animals kernel: [    0.978058] ACPI: \_SB_.PCI0.CWDT: can't evaluate _ADR (0x5)
Jan 23 07:10:41 animals kernel: [    0.978060] ACPI: \_SB_.PCI0.WMI1: can't evaluate _ADR (0x5)
Jan 23 07:10:41 animals kernel: [    0.978134] ACPI: \_SB_.PCI0.HPET: can't evaluate _ADR (0x5)
Jan 23 07:10:41 animals kernel: [    1.001344] ACPI: Enabled 3 GPEs in block 00 to 3F
Jan 23 07:10:41 animals kernel: [    1.129785] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 23 07:10:41 animals kernel: [    1.131680] ioapic: probe of 0000:00:05.4 failed with error -22
Jan 23 07:10:41 animals kernel: [    1.132291] acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
Jan 23 07:10:41 animals kernel: [    1.158969] pcieport 0000:00:1c.5: driver skip pci_set_master, fix it!
Jan 23 07:10:41 animals kernel: [    1.164052] pcieport 0000:00:1c.7: driver skip pci_set_master, fix it!
Jan 23 07:10:41 animals kernel: [    1.467125] ata1.00: supports DRM functions and may not be fully accessible
Jan 23 07:10:41 animals kernel: [    1.470184] ata1.00: supports DRM functions and may not be fully accessible
Jan 23 07:10:41 animals kernel: [    4.159271] sr0: scsi3-mmc drive: 8x/48x writer dvd-ram cd/rw xa/form2 cdda tray
Jan 23 07:10:41 animals kernel: [    4.923868] i8042: PNP: PS/2 controller doesn't have KBD irq; using default 1
Jan 23 07:10:41 animals kernel: [    7.588797] usb 7-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
Jan 23 07:10:41 animals kernel: [    7.589592] usb 7-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
Jan 23 07:10:41 animals kernel: [    8.326286] sd 12:0:0:0: [sdc] No Caching mode page found
Jan 23 07:10:41 animals kernel: [    8.326962] sd 12:0:0:0: [sdc] Assuming drive cache: write through
Jan 23 07:10:41 animals kernel: [    8.328430] sd 12:0:0:0: [sdc] No Caching mode page found
Jan 23 07:10:41 animals kernel: [    8.329120] sd 12:0:0:0: [sdc] Assuming drive cache: write through
Jan 23 07:10:41 animals kernel: [    8.336334] sd 12:0:0:0: [sdc] No Caching mode page found
Jan 23 07:10:41 animals kernel: [    8.337031] sd 12:0:0:0: [sdc] Assuming drive cache: write through
Jan 23 07:10:41 animals kernel: [   17.765553] ACPI Warning: 0x0000000000000530-0x000000000000053f SystemIO conflicts with Region \_SB_.PCI0.SBRG.GPBX 1 (20130725/utaddress-251)
Jan 23 07:10:41 animals kernel: [   17.767594] ACPI Warning: 0x0000000000000500-0x000000000000052f SystemIO conflicts with Region \_SB_.PCI0.SBRG.GPBX 1 (20130725/utaddress-251)
Jan 23 07:10:41 animals kernel: [   17.769377] lpc_ich: Resource conflict(s) found affecting gpio_ich
Jan 23 07:10:41 animals kernel: [   17.789424] fglrx: module license 'Proprietary. (C) 2002 - ATI Technologies, Starnberg, GERMANY' taints kernel.
Jan 23 07:10:41 animals kernel: [   17.791310] Disabling lock debugging due to kernel taint
Jan 23 07:10:41 animals kernel: [   17.796356] <6>[fglrx] Maximum main memory to use for locked dma buffers: 31477 MBytes.
Jan 23 07:10:41 animals kernel: [   17.797293] <6>[fglrx]   vendor: 1002 device: 6798 count: 1
Jan 23 07:10:41 animals kernel: [   17.798484] <6>[fglrx] ioport: bar 4, base 0xe000, size: 0x100
Jan 23 07:10:41 animals kernel: [   17.799057] pcieport 0000:00:02.0: driver skip pci_set_master, fix it!
Jan 23 07:10:41 animals kernel: [   17.799950] <6>[fglrx] Kernel PAT support is enabled
Jan 23 07:10:41 animals kernel: [   17.800537] <6>[fglrx] module loaded - fglrx 13.25.5 [Jan  8 2014] with 1 minors
Jan 23 07:10:41 animals kernel: [   22.570415] EXT4-fs (dm-6): mounting with "discard" option, but the device does not support discard
Jan 23 07:10:41 animals kernel: [   22.587119] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Jan 23 07:10:53 animals acpid: Too many connections.
Jan 23 07:10:54 animals smbd[1143]: [2014/01/23 07:10:54.330959,  0] ../source3/param/loadparm.c:3067(lp_set_enum_parm)
Jan 23 07:10:54 animals smbd[1143]:   WARNING: Ignoring invalid value 'share' for parameter 'security'
Jan 23 07:10:59 animals dhcpcd[902]: br0: no IPv6 Routers available
Jan 23 07:11:26 animals kernel: [   67.948886] <6>[fglrx] Firegl kernel thread PID: 1366
Jan 23 07:11:26 animals kernel: [   67.948919] <6>[fglrx] Firegl kernel thread PID: 1367
Jan 23 07:11:26 animals kernel: [   67.948950] <6>[fglrx] Firegl kernel thread PID: 1368
Jan 23 07:11:26 animals kernel: [   67.949004] <6>[fglrx] IRQ 104 Enabled
Jan 23 07:11:26 animals kernel: [   67.955156] <6>[fglrx] Reserved FB block: Shared offset:0, size:1000000
Jan 23 07:11:26 animals kernel: [   67.955158] <6>[fglrx] Reserved FB block: Unshared offset:f878000, size:4000
Jan 23 07:11:26 animals kernel: [   67.955159] <6>[fglrx] Reserved FB block: Unshared offset:f87c000, size:484000
Jan 23 07:11:26 animals kernel: [   67.955160] <6>[fglrx] Reserved FB block: Unshared offset:fff8000, size:8000
Jan 23 07:11:26 animals kernel: [   67.955161] <6>[fglrx] Reserved FB block: Unshared offset:bfff4000, size:c000
Jan 23 07:11:26 animals kernel: [   67.968943] dmar: DRHD: handling fault status reg 2
Jan 23 07:11:26 animals kernel: [   67.968949] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 812330000
Jan 23 07:11:26 animals kernel: [   67.968949] DMAR:[fault reason 02] Present bit in context entry is clear
Jan 23 07:14:26 animals kernel: [  247.861893] <6>[fglrx] ASIC hang happened
Jan 23 07:14:26 animals kernel: [  247.861897] CPU: 10 PID: 1363 Comm: X Tainted: P           O 3.12.1 #1
Jan 23 07:14:26 animals kernel: [  247.861898] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 9502 11/07/2013
Jan 23 07:14:26 animals kernel: [  247.861899]  00000000ffff328e ffff880828de58d0 ffffffff8161dcb7 0000000000000000
Jan 23 07:14:26 animals kernel: [  247.861901]  ffff880828de58e0 ffffffffa06667ae 00000000ffff328d ffffffffa067553c
Jan 23 07:14:26 animals kernel: [  247.861902]  0000000000000000 ffffffffa07293de ffff880828de5940 ffffffffa0729349
Jan 23 07:14:26 animals kernel: [  247.861904] Call Trace:
Jan 23 07:14:26 animals kernel: [  247.861908]  [<ffffffff8161dcb7>] dump_stack+0x45/0x56
Jan 23 07:14:26 animals kernel: [  247.861944]  [<ffffffffa06667ae>] KCL_DEBUG_OsDump+0xe/0x10 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.861967]  [<ffffffffa067553c>] firegl_hardwareHangRecovery+0x1c/0x30 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862009]  [<ffffffffa07293de>] ? _ZN4Asic9WaitUntil15ResetASICIfHungEv+0x1e/0x30 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862050]  [<ffffffffa0729349>] ? _ZN4Asic9WaitUntil15WaitForCompleteEv+0xb9/0x130 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862093]  [<ffffffffa0733614>] ? _ZN15ExecutableUnits10CPRingIdleE15idle_WaitMethod12_QS_CP_RING_+0x124/0x1e0 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862134]  [<ffffffffa0730001>] ? _ZNK18asic_configuration16GetIndexedAsicIDEv+0x1/0x80 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862175]  [<ffffffffa0733ffa>] ? _ZN21ExecutableUnitsCayman14AllCPRingsIdleE15idle_WaitMethod+0x1a/0x90 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862216]  [<ffffffffa073346b>] ? _ZN15ExecutableUnits7PM4idleE15idle_WaitMethod+0x4b/0x90 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862256]  [<ffffffffa073451d>] ? _ZN17ExecutableUnitsSI9assertPM4Eb+0x6d/0x80 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862304]  [<ffffffffa077b589>] ? _ZN8AsicR6009assertPM4Eb+0x39/0x80 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862339]  [<ffffffffa06f2e3c>] ? _ZN9CMMlegacy19CMMQS_Initialize_WAEP20_CMMQS_INIT_INPUT_WAP18_CMMQS_INIT_OUTPUT+0x17c/0x280 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862373]  [<ffffffffa06f2af5>] ? CMMQS_Initialize_WA+0x65/0xb0 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862399]  [<ffffffffa069df04>] ? firegl_cmmqs_init+0x684/0xc00 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862422]  [<ffffffffa0678494>] ? firegl_init_iommu+0x94/0x170 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862447]  [<ffffffffa06954ba>] ? hal_init_gpu+0x20a/0x480 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862469]  [<ffffffffa066fa7b>] ? firegl_open+0x2db/0x310 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862484]  [<ffffffffa065e087>] ? ip_firegl_open+0x17/0x20 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862499]  [<ffffffffa065f8f8>] ? firegl_stub_open+0x98/0x100 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862502]  [<ffffffff811d0832>] ? chrdev_open+0x92/0x1e0
Jan 23 07:14:26 animals kernel: [  247.862503]  [<ffffffff811c9aeb>] ? do_dentry_open+0x1eb/0x280
Jan 23 07:14:26 animals kernel: [  247.862505]  [<ffffffff811d74f4>] ? __inode_permission+0x54/0xc0
Jan 23 07:14:26 animals kernel: [  247.862506]  [<ffffffff811d07a0>] ? cdev_put+0x30/0x30
Jan 23 07:14:26 animals kernel: [  247.862508]  [<ffffffff811c9bb1>] ? finish_open+0x31/0x40
Jan 23 07:14:26 animals kernel: [  247.862509]  [<ffffffff811dacec>] ? do_last+0x5cc/0x1230
Jan 23 07:14:26 animals kernel: [  247.862511]  [<ffffffff811dba0b>] ? path_openat+0xbb/0x670
Jan 23 07:14:26 animals kernel: [  247.862532]  [<ffffffffa066ddd9>] ? firegl_release_helper+0x419/0x700 [fglrx]
Jan 23 07:14:26 animals kernel: [  247.862534]  [<ffffffff811dc6ea>] ? do_filp_open+0x3a/0x90
Jan 23 07:14:26 animals kernel: [  247.862536]  [<ffffffff811e8707>] ? __alloc_fd+0xa7/0x130
Jan 23 07:14:26 animals kernel: [  247.862537]  [<ffffffff811cafbe>] ? do_sys_open+0x12e/0x210
Jan 23 07:14:26 animals kernel: [  247.862539]  [<ffffffff811cb0be>] ? SyS_open+0x1e/0x20
Jan 23 07:14:26 animals kernel: [  247.862541]  [<ffffffff8162722d>] ? system_call_fastpath+0x1a/0x1f
Jan 23 07:14:26 animals kernel: [  247.862543] pubdev:0xffffffffa0ca9180, num of device:1 , name:fglrx, major 13, minor 25.
Jan 23 07:14:26 animals kernel: [  247.862544] device 0 : 0xffff88082a3dc000 .
Jan 23 07:14:26 animals kernel: [  247.862545] Asic ID:0x6798, revision:0x5, MMIOReg:0xffffc90022500000.
Jan 23 07:14:26 animals kernel: [  247.862546] FB phys addr: 0xd0000000, MC :0xf400000000, Total FB size :0xc0000000.
Jan 23 07:14:26 animals kernel: [  247.862547] gart table MC:0xf40f87c000, Physical:0xdf87c000, size:0x483000.
Jan 23 07:14:26 animals kernel: [  247.862548] mc_node :FB, total 1 zones
Jan 23 07:14:26 animals kernel: [  247.862549]     MC start:0xf400000000, Physical:0xd0000000, size:0xfd00000.
Jan 23 07:14:26 animals kernel: [  247.862550]     Mapped heap -- Offset:0x0, size:0xf878000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862551]     Mapped heap -- Offset:0x0, size:0x1000000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862552]     Mapped heap -- Offset:0xf878000, size:0x4000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862553]     Mapped heap -- Offset:0xf87c000, size:0x484000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862554] mc_node :INV_FB, total 1 zones
Jan 23 07:14:26 animals kernel: [  247.862555]     MC start:0xf40fd00000, Physical:0xdfd00000, size:0xb0300000.
Jan 23 07:14:26 animals kernel: [  247.862556]     Mapped heap -- Offset:0x2f8000, size:0x8000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862557]     Mapped heap -- Offset:0xb02f4000, size:0xc000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862558] mc_node :GART_USWC, total 3 zones
Jan 23 07:14:26 animals kernel: [  247.862559]     MC start:0xff90100000, Physical:0x0, size:0x60000000.
Jan 23 07:14:26 animals kernel: [  247.862559]     Mapped heap -- Offset:0x0, size:0x2000000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862560] mc_node :GART_CACHEABLE, total 3 zones
Jan 23 07:14:26 animals kernel: [  247.862561]     MC start:0xff60400000, Physical:0x0, size:0x2fd00000.
Jan 23 07:14:26 animals kernel: [  247.862562]     Mapped heap -- Offset:0x0, size:0x200000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862563]     Mapped heap -- Offset:0xef000, size:0x11000, reference count:1, mapping count:0,
Jan 23 07:14:26 animals kernel: [  247.862564] mc_node :PEER_FB_GART, total 1 zones
Jan 23 07:14:26 animals kernel: [  247.862565]     MC start:0xfff0100000, Physical:0x0, size:0x1000.
Jan 23 07:14:26 animals kernel: [  247.862567] GRBM : 0xa0003028, SRBM : 0x20000ac0 .
Jan 23 07:14:26 animals kernel: [  247.862568] CP_RB_BASE : 0xff901000, CP_RB_RPTR : 0x40 , CP_RB_WPTR :0x40.
Jan 23 07:14:26 animals kernel: [  247.862570] CP_IB1_BUFSZ:0x0, CP_IB1_BASE_HI:0x0, CP_IB1_BASE_LO:0x0.
Jan 23 07:14:26 animals kernel: [  247.862571] last submit IB buffer -- MC :0x0. Can't found mapped physical page for this MC .
Jan 23 07:14:26 animals kernel: [  247.862572] Dump the trace queue.
Jan 23 07:14:26 animals kernel: [  247.862573] End of dump

whereas with the radeon driver, which seems to work OK, despite the "Invalid ROM contents" and ACPI warnings

an 25 14:38:35 animals kernel: [    0.000000] ACPI: RSDP 00000000ac450000 00024 (v02 ALASKA)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: XSDT 00000000ac450078 00064 (v01 ALASKA    A M I 01072009 AMI  00010013)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: FACP 00000000ac45a1f8 0010C (v05 ALASKA    A M I 01072009 AMI  00010013)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: DSDT 00000000ac450170 0A088 (v02 ALASKA    A M I 00000016 INTL 20051117)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: FACS 00000000ac746080 00040
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: APIC 00000000ac45a308 00100 (v03 ALASKA    A M I 01072009 AMI  00010013)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: FPDT 00000000ac45a408 00044 (v01 ALASKA    A M I 01072009 AMI  00010013)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: MCFG 00000000ac45a450 0003C (v01 ALASKA OEMMCFG. 01072009 MSFT 00000097)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: HPET 00000000ac45a490 00038 (v01 ALASKA    A M I 01072009 AMI. 00000005)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: DMAR 00000000ac527680 000B4 (v01 A M I   OEMDMAR 00000001 INTL 00000001)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: SSDT 00000000ac45a520 CD128 (v02  INTEL    CpuPm 00004000 INTL 20051117)
Jan 25 14:38:35 animals kernel: [    0.000000] ACPI: BGRT 00000000ac527648 00038 (v00 ALASKA    A M I 01072009 AMI  00010013)
Jan 25 14:38:35 animals kernel: [    0.000000] Zone ranges:
Jan 25 14:38:35 animals kernel: [    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
Jan 25 14:38:35 animals kernel: [    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
Jan 25 14:38:35 animals kernel: [    0.000000]   Normal   [mem 0x100000000-0x84fffffff]
Jan 25 14:38:35 animals kernel: [    0.000000] Movable zone start for each node
Jan 25 14:38:35 animals kernel: [    0.000000] Early memory node ranges
Jan 25 14:38:35 animals kernel: [    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
Jan 25 14:38:35 animals kernel: [    0.000000]   node   0: [mem 0x00100000-0xabe82fff]
Jan 25 14:38:35 animals kernel: [    0.000000]   node   0: [mem 0xad76f000-0xad76ffff]
Jan 25 14:38:35 animals kernel: [    0.000000]   node   0: [mem 0xad7f6000-0xadc37fff]
Jan 25 14:38:35 animals kernel: [    0.000000]   node   0: [mem 0xadff4000-0xadffffff]
Jan 25 14:38:35 animals kernel: [    0.000000]   node   0: [mem 0x100000000-0x84fffffff]
Jan 25 14:38:35 animals kernel: [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8241979
Jan 25 14:38:35 animals kernel: [    0.000000] Memory: 32601000K/33491396K available (6322K kernel code, 734K rwdata, 2264K rodata, 1184K init, 968K bss, 890396K reserved)
Jan 25 14:38:35 animals kernel: [    0.016714] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
Jan 25 14:38:35 animals kernel: [    0.016714] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
Jan 25 14:38:35 animals kernel: [    0.036694] ACPI: All ACPI Tables successfully acquired
Jan 25 14:38:35 animals kernel: [    0.840086] raid6: sse2x1    9304 MB/s
Jan 25 14:38:35 animals kernel: [    0.857091] raid6: sse2x2   11496 MB/s
Jan 25 14:38:35 animals kernel: [    0.874091] raid6: sse2x4   13386 MB/s
Jan 25 14:38:35 animals kernel: [    0.874093] raid6: using algorithm sse2x4 (13386 MB/s)
Jan 25 14:38:35 animals kernel: [    0.874094] raid6: using ssse3x2 recovery algorithm
Jan 25 14:38:35 animals kernel: [    0.886721] ACPI: Executed 1 blocks of module-level executable AML code
Jan 25 14:38:35 animals kernel: [    0.969508] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130725/hwxface-571)
Jan 25 14:38:35 animals kernel: [    0.969512] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130725/hwxface-571)
Jan 25 14:38:35 animals kernel: [    0.976285] ACPI: \_SB_.PCI0.IOH_: can't evaluate _ADR (0x5)
Jan 25 14:38:35 animals kernel: [    0.976287] ACPI: \_SB_.PCI0.VTDR: can't evaluate _ADR (0x5)
Jan 25 14:38:35 animals kernel: [    0.976295] ACPI: \_SB_.PCI0.PCH_: can't evaluate _ADR (0x5)
Jan 25 14:38:35 animals kernel: [    0.976296] ACPI: \_SB_.PCI0.CWDT: can't evaluate _ADR (0x5)
Jan 25 14:38:35 animals kernel: [    0.976298] ACPI: \_SB_.PCI0.WMI1: can't evaluate _ADR (0x5)
Jan 25 14:38:35 animals kernel: [    0.976372] ACPI: \_SB_.PCI0.HPET: can't evaluate _ADR (0x5)
Jan 25 14:38:35 animals kernel: [    0.999581] ACPI: Enabled 3 GPEs in block 00 to 3F
Jan 25 14:38:35 animals kernel: [    1.127966] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jan 25 14:38:35 animals kernel: [    1.129868] ioapic: probe of 0000:00:05.4 failed with error -22
Jan 25 14:38:35 animals kernel: [    1.130476] acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
Jan 25 14:38:35 animals kernel: [    1.161181] pcieport 0000:00:1c.5: driver skip pci_set_master, fix it!
Jan 25 14:38:35 animals kernel: [    1.161667] pcieport 0000:00:1c.7: driver skip pci_set_master, fix it!
Jan 25 14:38:35 animals kernel: [    1.473311] ata1.00: supports DRM functions and may not be fully accessible
Jan 25 14:38:35 animals kernel: [    1.478620] ata1.00: supports DRM functions and may not be fully accessible
Jan 25 14:38:35 animals kernel: [    4.161865] sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
Jan 25 14:38:35 animals kernel: [    4.928772] i8042: PNP: PS/2 controller doesn't have KBD irq; using default 1
Jan 25 14:38:35 animals kernel: [    5.937478] usb 7-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
Jan 25 14:38:35 animals kernel: [    5.938336] usb 7-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
Jan 25 14:38:35 animals kernel: [    6.672134] sd 12:0:0:0: [sdc] No Caching mode page found
Jan 25 14:38:35 animals kernel: [    6.673024] sd 12:0:0:0: [sdc] Assuming drive cache: write through
Jan 25 14:38:35 animals kernel: [    6.674693] sd 12:0:0:0: [sdc] No Caching mode page found
Jan 25 14:38:35 animals kernel: [    6.675579] sd 12:0:0:0: [sdc] Assuming drive cache: write through
Jan 25 14:38:35 animals kernel: [    6.683232] sd 12:0:0:0: [sdc] No Caching mode page found
Jan 25 14:38:35 animals kernel: [    6.684120] sd 12:0:0:0: [sdc] Assuming drive cache: write through
Jan 25 14:38:35 animals kernel: [   15.373877] pcieport 0000:00:02.0: driver skip pci_set_master, fix it!
Jan 25 14:38:35 animals kernel: [   15.373948] ACPI Warning: 0x0000000000000530-0x000000000000053f SystemIO conflicts with Region \_SB_.PCI0.SBRG.GPBX 1 (20130725/utaddress-251)
Jan 25 14:38:35 animals kernel: [   15.373959] ACPI Warning: 0x0000000000000500-0x000000000000052f SystemIO conflicts with Region \_SB_.PCI0.SBRG.GPBX 1 (20130725/utaddress-251)
Jan 25 14:38:35 animals kernel: [   15.373967] lpc_ich: Resource conflict(s) found affecting gpio_ich
Jan 25 14:38:35 animals kernel: [   15.374214] radeon 0000:01:00.0: Invalid ROM contents


Sorry about the long post, and I'd put the syslog dumps into a code box if I knew how.. is there a tool button I'm missing somewhere?

Offline

#1065 2014-01-26 01:53:57

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did a bit more google research;  apparently this kind of catalyst dmar failure on Asus intel boards has been intermittently popping up for years, without resolution...

I see that even when apparently running with qemu, I get dmesg spammed with

WARNING: CPU: 9 PID: 2172 at drivers/vfio/vfio_iommu_type1.c:344 vfio_unmap_unpin.isra.7+0x11e/0x120()
[  388.153197] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables vhost_net vhost macvtap macvlan tun autofs4 bridge stp llc ipv6 lp parport fuse snd_hda_codec_hdmi btusb bluetooth snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul eeepc_wmi crc32c_intel asus_wmi ghash_clmulni_intel sparse_keymap i2c_dev rfkill video mxm_wmi evdev radeon joydev ttm snd_hda_intel snd_hda_codec drm_kms_helper microcode psmouse drm snd_hwdep e1000e snd_pcm serio_raw snd_page_alloc i2c_i801 agpgart snd_timer lpc_ich mei_me i2c_algo_bit snd ptp i2c_core mei soundcore pps_core ehci_pci processor thermal_sys hwmon wmi button loop msdos nls_iso8859_1 nls_cp437 uhci_hcd ehci_hcd pci_stub vfat fat ext2 ext4 jbd2 mbcache hid_generic usbhid hid xhci_hcd usb_storage
[  388.153222] CPU: 9 PID: 2172 Comm: qemu-system-x86 Tainted: G        W    3.13.0 #1
[  388.153223] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 9502 11/07/2013
[  388.153223]  0000000000000009 ffff8807e5623d10 ffffffff81644f6e 0000000000000000
[  388.153225]  ffff8807e5623d48 ffffffff810be16d 0000000000000000 0000000000000000
[  388.153226]  ffff8807e5623e30 0000000000200000 ffff880829898a80 ffff8807e5623d58
[  388.153228] Call Trace:
[  388.153229]  [<ffffffff81644f6e>] dump_stack+0x45/0x56
[  388.153231]  [<ffffffff810be16d>] warn_slowpath_common+0x7d/0xa0
[  388.153232]  [<ffffffff810be24a>] warn_slowpath_null+0x1a/0x20
[  388.153234]  [<ffffffff814a433e>] vfio_unmap_unpin.isra.7+0x11e/0x120
[  388.153236]  [<ffffffff814a43a2>] vfio_remove_dma_overlap+0x62/0x250
[  388.153238]  [<ffffffff814a5120>] vfio_iommu_type1_ioctl+0x230/0x290
[  388.153240]  [<ffffffff814a2377>] vfio_fops_unl_ioctl+0x77/0x340
[  388.153241]  [<ffffffff811e8540>] do_vfs_ioctl+0x2e0/0x4c0
[  388.153243]  [<ffffffff811d57ac>] ? vfs_write+0x15c/0x1f0
[  388.153245]  [<ffffffff811d55de>] ? vfs_read+0xee/0x160
[  388.153246]  [<ffffffff811e87a1>] SyS_ioctl+0x81/0xa0
[  388.153248]  [<ffffffff8164e5ed>] system_call_fastpath+0x1a/0x1f
[  388.153249] ---[ end trace 8731715962b54b3d ]---
[  389.569527] dmar: DRHD: handling fault status reg 2
[  389.569532] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000
[  389.569532] DMAR:[fault reason 06] PTE Read access is not set
[  395.563868] kvm: zapping shadow pages for mmio generation wraparound
[  400.086315] vfio-pci 0000:03:00.0: irq 105 for MSI/MSI-X

Does that mean anything?

Last edited by mostlyharmless (2014-01-26 23:29:02)

Offline

#1066 2014-01-27 10:04:36

Plutonium
Member
Registered: 2014-01-27
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a Dell M6600 laptop, with intel hd graphics 2000 and Nvidia Quadra 4000m, I was able to passthrough the Nvidia card to a Windows 7 VM, installed Nvidia driver in the VM and every thing seems to be fine except that the passed card only operates at the lowest 2 frequency level (Nvidia powermizer), and the performance of the VM with graphic card was even worse than the VM you can create with VirtualBox. I am not sure what is wrong, Any one with Nvidia card have similar issues?

Nvidia powermizer levels:
Level 0: 50MHz(Graphics Clock) 270MHz(Memory Transfer Rate) 101MHz(Processor Clock)
Level 1: 73MHz                            648MHz                                       147MHz
Level 2: 202MHz                          2500MHz                                     405MHz
Level 3: 475MHz                          2500MHz                                     950MHz

And my virtual machine only able to operate on Level 0 and Level 1, and thus the bad performance.

Offline

#1067 2014-01-27 15:01:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Here's some information that might be useful to someone; I'd post it as a bug but I suspect that it's either an ASUS bios bug, which they will cheerfully ignore (particularly as vt-d support is not yet supported in an official bios for the RIVE with an IVy-E preocessor), or an AMD catalyst bug which only appears when using intel IOMMU, which I suspect might not get much attention either...  Any suggestions on where to post it or what to do would be most welcome.

My passthrough is an Nvidia K2000 to Windows, which works fine, no errors in Device Manager, my host has an AMD HD 7970.  Using the radeon drivers, everything works fine, other than the fact that 3D acceleration is software only and slow as there is no "Southern Island" hardware support yet in the radeon driver.  Using the latest beta ati-catalyst 13.11beta9.95 on kernel 3.12.1, the driver builds OK, but running X leads to a black screen and hung keyboard. One can ssh into the system.  Here is the syslog:

...

Sorry about the long post, and I'd put the syslog dumps into a code box if I knew how.. is there a tool button I'm missing somewhere?

You might try the iommu=pt boot option to make the IOMMU run in passthrough mode for host devices.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1068 2014-01-27 15:02:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Did a bit more google research;  apparently this kind of catalyst dmar failure on Asus intel boards has been intermittently popping up for years, without resolution...

I see that even when apparently running with qemu, I get dmesg spammed with

WARNING: CPU: 9 PID: 2172 at drivers/vfio/vfio_iommu_type1.c:344 vfio_unmap_unpin.isra.7+0x11e/0x120()
[  388.153197] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables vhost_net vhost macvtap macvlan tun autofs4 bridge stp llc ipv6 lp parport fuse snd_hda_codec_hdmi btusb bluetooth snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul eeepc_wmi crc32c_intel asus_wmi ghash_clmulni_intel sparse_keymap i2c_dev rfkill video mxm_wmi evdev radeon joydev ttm snd_hda_intel snd_hda_codec drm_kms_helper microcode psmouse drm snd_hwdep e1000e snd_pcm serio_raw snd_page_alloc i2c_i801 agpgart snd_timer lpc_ich mei_me i2c_algo_bit snd ptp i2c_core mei soundcore pps_core ehci_pci processor thermal_sys hwmon wmi button loop msdos nls_iso8859_1 nls_cp437 uhci_hcd ehci_hcd pci_stub vfat fat ext2 ext4 jbd2 mbcache hid_generic usbhid hid xhci_hcd usb_storage
[  388.153222] CPU: 9 PID: 2172 Comm: qemu-system-x86 Tainted: G        W    3.13.0 #1
[  388.153223] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 9502 11/07/2013
[  388.153223]  0000000000000009 ffff8807e5623d10 ffffffff81644f6e 0000000000000000
[  388.153225]  ffff8807e5623d48 ffffffff810be16d 0000000000000000 0000000000000000
[  388.153226]  ffff8807e5623e30 0000000000200000 ffff880829898a80 ffff8807e5623d58
[  388.153228] Call Trace:
[  388.153229]  [<ffffffff81644f6e>] dump_stack+0x45/0x56
[  388.153231]  [<ffffffff810be16d>] warn_slowpath_common+0x7d/0xa0
[  388.153232]  [<ffffffff810be24a>] warn_slowpath_null+0x1a/0x20
[  388.153234]  [<ffffffff814a433e>] vfio_unmap_unpin.isra.7+0x11e/0x120
[  388.153236]  [<ffffffff814a43a2>] vfio_remove_dma_overlap+0x62/0x250
[  388.153238]  [<ffffffff814a5120>] vfio_iommu_type1_ioctl+0x230/0x290
[  388.153240]  [<ffffffff814a2377>] vfio_fops_unl_ioctl+0x77/0x340
[  388.153241]  [<ffffffff811e8540>] do_vfs_ioctl+0x2e0/0x4c0
[  388.153243]  [<ffffffff811d57ac>] ? vfs_write+0x15c/0x1f0
[  388.153245]  [<ffffffff811d55de>] ? vfs_read+0xee/0x160
[  388.153246]  [<ffffffff811e87a1>] SyS_ioctl+0x81/0xa0
[  388.153248]  [<ffffffff8164e5ed>] system_call_fastpath+0x1a/0x1f
[  388.153249] ---[ end trace 8731715962b54b3d ]---
[  389.569527] dmar: DRHD: handling fault status reg 2
[  389.569532] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000
[  389.569532] DMAR:[fault reason 06] PTE Read access is not set
[  395.563868] kvm: zapping shadow pages for mmio generation wraparound
[  400.086315] vfio-pci 0000:03:00.0: irq 105 for MSI/MSI-X

Does that mean anything?

Does this help: https://git.kernel.org/cgit/linux/kerne … fe8702b098


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1069 2014-01-27 15:07:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Plutonium wrote:

I have a Dell M6600 laptop, with intel hd graphics 2000 and Nvidia Quadra 4000m, I was able to passthrough the Nvidia card to a Windows 7 VM, installed Nvidia driver in the VM and every thing seems to be fine except that the passed card only operates at the lowest 2 frequency level (Nvidia powermizer), and the performance of the VM with graphic card was even worse than the VM you can create with VirtualBox. I am not sure what is wrong, Any one with Nvidia card have similar issues?

Nvidia powermizer levels:
Level 0: 50MHz(Graphics Clock) 270MHz(Memory Transfer Rate) 101MHz(Processor Clock)
Level 1: 73MHz                            648MHz                                       147MHz
Level 2: 202MHz                          2500MHz                                     405MHz
Level 3: 475MHz                          2500MHz                                     950MHz

And my virtual machine only able to operate on Level 0 and Level 1, and thus the bad performance.

Honestly, I'm surprised it got this far.  Discrete graphics in laptops are still generally very tightly tied to the platform, often requiring vendor tweaked binaries to work.  The Nvidia driver could have some sort of fail-safe in it that prevents it from running at a higher clock rate when sensor data is not available to prevent the laptop from melting.  Maybe there's an override you can enable for that (at your own risk).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1070 2014-01-27 15:19:49

Plutonium
Member
Registered: 2014-01-27
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Plutonium wrote:

I have a Dell M6600 laptop, with intel hd graphics 2000 and Nvidia Quadra 4000m, I was able to passthrough the Nvidia card to a Windows 7 VM, installed Nvidia driver in the VM and every thing seems to be fine except that the passed card only operates at the lowest 2 frequency level (Nvidia powermizer), and the performance of the VM with graphic card was even worse than the VM you can create with VirtualBox. I am not sure what is wrong, Any one with Nvidia card have similar issues?

Nvidia powermizer levels:
Level 0: 50MHz(Graphics Clock) 270MHz(Memory Transfer Rate) 101MHz(Processor Clock)
Level 1: 73MHz                            648MHz                                       147MHz
Level 2: 202MHz                          2500MHz                                     405MHz
Level 3: 475MHz                          2500MHz                                     950MHz

And my virtual machine only able to operate on Level 0 and Level 1, and thus the bad performance.

Honestly, I'm surprised it got this far.  Discrete graphics in laptops are still generally very tightly tied to the platform, often requiring vendor tweaked binaries to work.  The Nvidia driver could have some sort of fail-safe in it that prevents it from running at a higher clock rate when sensor data is not available to prevent the laptop from melting.  Maybe there's an override you can enable for that (at your own risk).

Are you suggesting that the VM is not able to know the graphic card's temperature? The temperature data is actually available in the VM, but I still have to agree with you, it might be something missing in the bios that prevent the card from functioning normally, I wish there is a fail safe that I can try to turn off.

Offline

#1071 2014-01-27 15:24:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Plutonium wrote:
aw wrote:
Plutonium wrote:

I have a Dell M6600 laptop, with intel hd graphics 2000 and Nvidia Quadra 4000m, I was able to passthrough the Nvidia card to a Windows 7 VM, installed Nvidia driver in the VM and every thing seems to be fine except that the passed card only operates at the lowest 2 frequency level (Nvidia powermizer), and the performance of the VM with graphic card was even worse than the VM you can create with VirtualBox. I am not sure what is wrong, Any one with Nvidia card have similar issues?

Nvidia powermizer levels:
Level 0: 50MHz(Graphics Clock) 270MHz(Memory Transfer Rate) 101MHz(Processor Clock)
Level 1: 73MHz                            648MHz                                       147MHz
Level 2: 202MHz                          2500MHz                                     405MHz
Level 3: 475MHz                          2500MHz                                     950MHz

And my virtual machine only able to operate on Level 0 and Level 1, and thus the bad performance.

Honestly, I'm surprised it got this far.  Discrete graphics in laptops are still generally very tightly tied to the platform, often requiring vendor tweaked binaries to work.  The Nvidia driver could have some sort of fail-safe in it that prevents it from running at a higher clock rate when sensor data is not available to prevent the laptop from melting.  Maybe there's an override you can enable for that (at your own risk).

Are you suggesting that the VM is not able to know the graphic card's temperature? The temperature data is actually available in the VM, but I still have to agree with you, it might be something missing in the bios that prevent the card from functioning normally, I wish there is a fail safe that I can try to turn off.

We likely know the GPU temperature but the driver could be looking for things like whether we're running on AC or battery or an overall system temperature.  QEMU doesn't emulate or passthrough those kinds of things.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1072 2014-01-27 23:15:07

adr
Member
Registered: 2014-01-27
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot to everyone for this thread! I can now passthrough my VGA card but still have some issues with passthrough of my other cards. Sorry for the long post, I give some details about my tests in case it helps other people who find the same kind of errors.

Hardware:

CPU: Intel Core i7-3770
Motherboard: Asrock Z77 Pro4 with

  • integrated HD4000 connected to 19" 1280x1024 screen

  • integrated Soundcard

  • integrated SATA controller connected to my hard drives

  • additional SATA controller connected to a Blu-Ray drive

VGA: AMD Radeon R9-280X connected to 27" 1920x1200 screen
PCI1: Soundcard
PCI2: DVB tuner

My goal was to keep all integrated devices for Archlinux host and passthrough VGA + PCI1 + PCI2 + SATA controller with BD drive to Win7 guest.


VGA card (device 01:00.0):
I got this working after installing qemu-git + seabios-git + linux-mainline as written in first post ingredients (actually I tried first with current versions from Archlinux => black screen during guest boot; then I installed linux-mainline but still keeping standard qemu/seabios => atikmpag.sys BSOD; then I installed everything and it was OK). I don't use HDMI device included in videocard so I don't passthrough it.


PCI cards (devices 04:00.0 & 04:01.0):
At first I could not passthrough any PCI device because IOMMU was complaining about other devices in same group. Using the lsgroup script found in a previous post:

#!/bin/sh BASE="/sys/kernel/iommu_groups" for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do GROUP=$(basename $i) echo "### Group $GROUP ###" for j in $(find $i/devices -type l); do DEV=$(basename $j) echo -n " " lspci -s $DEV done done

Result:

### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) ### Group 6 ### 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) ### Group 7 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 04:00.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] 04:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 8 ### 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) ### Group 9 ### 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)

Then I recompiled the kernel using patch found here https://lkml.org/lkml/2014/1/20/453 instead of the override_for_missing_acs_capabilities.patch provided in linux-mainline from first post, the result is better:

### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) ### Group 6 ### 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) ### Group 7 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) ### Group 8 ### 00:1c.4 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 04:00.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] 04:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) ### Group 9 ### 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) ### Group 10 ### 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) ### Group 12 ### 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) ### Group 13 ### 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) ### Group 14 ### 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

But I still cannot passthrough the PCI soundcard, now I have the following error when starting qemu:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=pcie.0: vfio: Error: Failed to setup INTx fd: Device or resource busy qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=pcie.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=pcie.0: Device 'vfio-pci' could not be initialized

I can passthrough the DVB tuner, but if I do it I also have to bind the soundcard to VFIO and cannot pass it to the guest so the soundcard becomes unusable. I tried switching physically the PCI cards, with the same result: DVB tuner works but not the PCI soundcard. I suppose it may be caused by the soundcard hardware itself. The interesting part is that one year ago I tested PCI passthrough in Xen and I could passthrough both cards at the same time to a Win7 guest (I broke my configuration during a Xen upgrade and was never able to fix it, this is why I am testing KVM+VFIO now). My problem is similar to the one listed in http://www.spinics.net/lists/kvm/msg99275.html (I have the same soundcard model as well), but the flag nointxmask is not set for me.

Is it possible to fix this somehow?

The workaround I have for now is to passthrough the integrated soundcard and keep the PCI soundcard for host. I may be able to setup a TV server with DVB tuner in host so that i can watch TV from the guest if I cannot passthrough the PCI card.


SATA controller with BD (device 06:00.0):
This is working when I first start the VM, I can access the Blu-Ray drive from my guest. But I have issues when I try to reboot the VM: sometimes the VM is frozen with message "SeaBIOS (version rel-1.7.4-27-...)" during about 2 minutes, and then it will startup normally (I can see the drive in guest after reboot). Sometimes it is definitely frozen with the same message, and the only solution is to shutdown the VM manually.

When the VM hangs for 2 minutes, I have no output for qemu but I see the following logs in dmesg:

[13707.928236] dmar: DRHD: handling fault status reg 2 [13707.928240] dmar: DMAR:[DMA Write] Request device [06:00.0] fault addr 1affde000 DMAR:[fault reason 05] PTE Write access is not set [13707.928276] dmar: DRHD: handling fault status reg 2 [13707.928278] dmar: DMAR:[DMA Read] Request device [06:00.0] fault addr 1affde000 DMAR:[fault reason 06] PTE Read access is not set [13764.520612] dmar: DRHD: handling fault status reg 2 [13764.520616] dmar: DMAR:[DMA Write] Request device [06:00.0] fault addr 1affde000 DMAR:[fault reason 05] PTE Write access is not set

When the VM hangs definitely, I have this additional message in dmesg:

[14225.129188] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (3ffc0001000)

and the following output from qemu:

qemu-system-x86_64: vfio_dma_map(0x7f9693da9bf0, 0x3ffc0000000, 0x10000000, 0x7f955c000000) = -14 (Bad address)

Any idea on how to debug this?

Offline

#1073 2014-01-28 00:04:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

adr wrote:

Thanks a lot to everyone for this thread! I can now passthrough my VGA card but still have some issues with passthrough of my other cards. Sorry for the long post, I give some details about my tests in case it helps other people who find the same kind of errors.

Hardware:

CPU: Intel Core i7-3770
Motherboard: Asrock Z77 Pro4 with

  • integrated HD4000 connected to 19" 1280x1024 screen

  • integrated Soundcard

  • integrated SATA controller connected to my hard drives

  • additional SATA controller connected to a Blu-Ray drive

VGA: AMD Radeon R9-280X connected to 27" 1920x1200 screen
PCI1: Soundcard
PCI2: DVB tuner

My goal was to keep all integrated devices for Archlinux host and passthrough VGA + PCI1 + PCI2 + SATA controller with BD drive to Win7 guest.


VGA card (device 01:00.0):
I got this working after installing qemu-git + seabios-git + linux-mainline as written in first post ingredients (actually I tried first with current versions from Archlinux => black screen during guest boot; then I installed linux-mainline but still keeping standard qemu/seabios => atikmpag.sys BSOD; then I installed everything and it was OK). I don't use HDMI device included in videocard so I don't passthrough it.

PCI cards (devices 04:00.0 & 04:01.0):
At first I could not passthrough any PCI device because IOMMU was complaining about other devices in same group. Using the lsgroup script found in a previous post:

Then I recompiled the kernel using patch found here https://lkml.org/lkml/2014/1/20/453 instead of the override_for_missing_acs_capabilities.patch provided in linux-mainline from first post, the result is better:

### Group 8 ### 00:1c.4 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 04:00.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] 04:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01)

Do these devices work on the host?  Google "ASM1083 interrupt" and note the widespread problems people have with this PCIe-to-PCI bridge.  I have one on a non-VT-d system and I can't even get a NIC to work under it with a bare-metal OS.  It also does not provide a PCIe capability, in direct violation of the PCIe spec which makes us hiccup trying to figure out the topology, which is why it gets assigned to the same group as the root port.  You probably see a WARN_ON during boot because of this.  I've written off the PCI slot on my system with this bridge, IMHO it's junk.

But I still cannot passthrough the PCI soundcard, now I have the following error when starting qemu:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=pcie.0: vfio: Error: Failed to setup INTx fd: Device or resource busy qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=pcie.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=pcie.0: Device 'vfio-pci' could not be initialized

It's entirely possible that one or both of the devices is not PCI2.3 compliant in supporting INTx disable and they share an interrupt.  I posted a script in the thread you reference below, does it say whether these devices support INTx disable?  What does lspci report for the interrupt line and what else is using it?

I can passthrough the DVB tuner, but if I do it I also have to bind the soundcard to VFIO and cannot pass it to the guest so the soundcard becomes unusable. I tried switching physically the PCI cards, with the same result: DVB tuner works but not the PCI soundcard. I suppose it may be caused by the soundcard hardware itself. The interesting part is that one year ago I tested PCI passthrough in Xen and I could passthrough both cards at the same time to a Win7 guest (I broke my configuration during a Xen upgrade and was never able to fix it, this is why I am testing KVM+VFIO now). My problem is similar to the one listed in http://www.spinics.net/lists/kvm/msg99275.html (I have the same soundcard model as well), but the flag nointxmask is not set for me.

Is it possible to fix this somehow?

The workaround I have for now is to passthrough the integrated soundcard and keep the PCI soundcard for host. I may be able to setup a TV server with DVB tuner in host so that i can watch TV from the guest if I cannot passthrough the PCI card.

You have the PCI soundcard in use by the host and the DVB card assigned to a guest?  I have a hard time believing that would work.  Splitting the IOMMU group for devices on a conventional PCI bus is in the "won't work" camp, versus normally being in the "might work" camp for PCIe devices.

SATA controller with BD (device 06:00.0):
This is working when I first start the VM, I can access the Blu-Ray drive from my guest. But I have issues when I try to reboot the VM: sometimes the VM is frozen with message "SeaBIOS (version rel-1.7.4-27-...)" during about 2 minutes, and then it will startup normally (I can see the drive in guest after reboot). Sometimes it is definitely frozen with the same message, and the only solution is to shutdown the VM manually.

When the VM hangs for 2 minutes, I have no output for qemu but I see the following logs in dmesg:

[13707.928236] dmar: DRHD: handling fault status reg 2 [13707.928240] dmar: DMAR:[DMA Write] Request device [06:00.0] fault addr 1affde000 DMAR:[fault reason 05] PTE Write access is not set [13707.928276] dmar: DRHD: handling fault status reg 2 [13707.928278] dmar: DMAR:[DMA Read] Request device [06:00.0] fault addr 1affde000 DMAR:[fault reason 06] PTE Read access is not set [13764.520612] dmar: DRHD: handling fault status reg 2 [13764.520616] dmar: DMAR:[DMA Write] Request device [06:00.0] fault addr 1affde000 DMAR:[fault reason 05] PTE Write access is not set

Others have reported similar problems with this device (note that it's also made by ASMedia...)  How much memory are you configuring for the VM?  The device is trying to do reads and writes to an address just shy of 7G, which may be off the end of guest memory.  Someone reported a similar fault with a Via USB3 card and replaced it with a Renesas controller and reported success.

When the VM hangs definitely, I have this additional message in dmesg:

[14225.129188] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (3ffc0001000)

and the following output from qemu:

qemu-system-x86_64: vfio_dma_map(0x7f9693da9bf0, 0x3ffc0000000, 0x10000000, 0x7f955c000000) = -14 (Bad address)

Any idea on how to debug this?

While the faults above are the device trying to do DMA to regions it shouldn't, this indicates QEMU trying to create an IOMMU mapping that it shouldn't.  Nothing in QEMU lives at such a high address.  If it's reproducible I could probably send you a patch to print out a little more info about where that region is coming from.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1074 2014-01-28 08:37:40

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On my AMD board and kernel 3.13 I get this IOMMU group:

### Group 14 ### 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 08:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01) 09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

I wonder if it's somehow possible to get the SATA controller in its own group so I can pass it through to the VM.

I see there are some quirks for those SB900 PCI bridge ports in drivers/pci/quirks.c, but either they do nothing or is it really impossible with these Ethernet/USB/SATA devices?

Offline

#1075 2014-01-28 15:09:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:
aw wrote:

For your testing pleasure (Intel root port ACS):

https://lkml.org/lkml/2014/1/20/453

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

I experience the same since i upgraded from kernel 3.11 to 3.12 or 3.13.
Host freezes (no more ssh login possible - ping still working; at least MagicSysRq-Keys still worked) as soon as the guest starts up the second time (complete guest shutdown, i.e. virsh shutdown, then virsh start again).

kernel is 3.13.0, qemu is git from 16.1.2014 (same qemu compile works flawless with 3.11).

lspci -vvv is here: http://www.pastebin.ca/2579449

My guess is, some of these patches are missing upstream: http://www.pastebin.ca/2579452
They were the ones needed to get it working with 3.11, and when i try to apply them to 3.13 (see here: http://pastebin.ca/2579454) they of course hunk, but _some_ apply, so i guess something is missing ?

The patch is an early implementation, the chunks that apply are no longer relevant.  What really confuses me is the implication that qemu.git works on 3.11 with this patch.  The problem is that the user API is different, which leads me to suspect that no bus reset is happening when you say it works.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1076 2014-01-28 16:16:20

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The patch is an early implementation, the chunks that apply are no longer relevant.  What really confuses me is the implication that qemu.git works on 3.11 with this patch.  The problem is that the user API is different, which leads me to suspect that no bus reset is happening when you say it works.

Yes qemu.git works perfectly with 3.11(.6) AND the necessary kernel patches. I can restart the guest as often as i like.
Is there something i can test to confirm your suspicion?

Greetings

Offline

#1077 2014-01-28 22:26:50

adr
Member
Registered: 2014-01-27
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
adr wrote:

Then I recompiled the kernel using patch found here https://lkml.org/lkml/2014/1/20/453 instead of the override_for_missing_acs_capabilities.patch provided in linux-mainline from first post, the result is better:

### Group 8 ### 00:1c.4 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 04:00.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] 04:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01)

Do these devices work on the host?  Google "ASM1083 interrupt" and note the widespread problems people have with this PCIe-to-PCI bridge.  I have one on a non-VT-d system and I can't even get a NIC to work under it with a bare-metal OS.  It also does not provide a PCIe capability, in direct violation of the PCIe spec which makes us hiccup trying to figure out the topology, which is why it gets assigned to the same group as the root port.  You probably see a WARN_ON during boot because of this.  I've written off the PCI slot on my system with this bridge, IMHO it's junk.

The sound card (04:00.0) works on the host, I didn't notice any issue with it for the moment. The DVB tuner (04:01.0) is seen by the host as I have an adapter0 in /dev/dvb. However I did not manage to tune any channel yet, but this may be due to my antenna or poor reception.

aw wrote:

It's entirely possible that one or both of the devices is not PCI2.3 compliant in supporting INTx disable and they share an interrupt.  I posted a script in the thread you reference below, does it say whether these devices support INTx disable?  What does lspci report for the interrupt line and what else is using it?

Using script:

#!/bin/sh # Usage $0 <PCI device>, ex: 9:00.0 INTX=$(( 0x400 )) ORIG=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX & $ORIG )) -ne 0 ]; then echo "INTx disable supported and enabled on $1" exit 0 fi NEW=$(printf %04x $(( $INTX | $ORIG ))) setpci -s $1 4.w=$NEW NEW=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX & $NEW )) -ne 0 ]; then echo "INTx disable support available on $1" else echo "INTx disable support NOT available on $1" fi NEW=$(printf %04x $ORIG) setpci -s $1 4.w=$NEW

Results:

# ./testINT.sh 0000:04:00.0 INTx disable support NOT available on 0000:04:00.0 # ./testINT.sh 0000:04:01.0 INTx disable support NOT available on 0000:04:01.0 # ./testINT.sh 0000:03:00.0 INTx disable support available on 0000:03:00.0 # ./testINT.sh 0000:06:00.0 INTx disable support available on 0000:06:00.0 # cat /sys/module/vfio_pci/parameters/nointxmask N # lspci -vvv (...) 04:00.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar D1) Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 32 (500ns min, 6000ns max) Interrupt: pin A routed to IRQ 16 Region 0: I/O ports at d000 [size=256] Capabilities: [c0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: snd_virtuoso Kernel modules: snd_virtuoso 04:01.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01) Subsystem: LeadTek Research Inc. WinFast DTV1000S Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 32 (21000ns min, 8000ns max) Interrupt: pin A routed to IRQ 17 Region 0: Memory at f7d00000 (32-bit, non-prefetchable) [size=1K] Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Kernel driver in use: saa7134 Kernel modules: saa7134 (...)
aw wrote:

You have the PCI soundcard in use by the host and the DVB card assigned to a guest?

No, not at all. Currently I have both the PCI soundcard and the DVB card in use by the host and I passthrough integrated soundcard "00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)" to the guest. If I reach a hardware limit with the share of interrupts for external soundcard/dvb tuner, an alternative for me would be to stream TV from the host to the guest (as far as I understand installing Mythtv on the host could do that for example). Assuming I can make the DVB tuner work properly, otherwise I can always buy another one...

aw wrote:

SATA controller with BD (device 06:00.0):
This is working when I first start the VM, I can access the Blu-Ray drive from my guest. But I have issues when I try to reboot the VM: sometimes the VM is frozen with message "SeaBIOS (version rel-1.7.4-27-...)" during about 2 minutes, and then it will startup normally (I can see the drive in guest after reboot). Sometimes it is definitely frozen with the same message, and the only solution is to shutdown the VM manually.

Others have reported similar problems with this device (note that it's also made by ASMedia...)  How much memory are you configuring for the VM?  The device is trying to do reads and writes to an address just shy of 7G, which may be off the end of guest memory.  Someone reported a similar fault with a Via USB3 card and replaced it with a Renesas controller and reported success.

I have 12GB memory in my host, and I assign only 4GB to my guest. I will try to assign 8GB to see if it is better.

aw wrote:

While the faults above are the device trying to do DMA to regions it shouldn't, this indicates QEMU trying to create an IOMMU mapping that it shouldn't.  Nothing in QEMU lives at such a high address.  If it's reproducible I could probably send you a patch to print out a little more info about where that region is coming from.

It is reproductible pretty easily (happens about 50% of the time when I reboot the guest). I will check if this is reproductible with 8GB memory assigned to the guest.

Thanks a lot for your help!


EDIT:
Still the same issue with 8GB:
- 1st boot OK
- 2min freeze during first reboot => [ 5044.282211] dmar: DMAR:[DMA Write] Request device [06:00.0] fault addr 2affde000
                                                          DMAR:[fault reason 05] PTE Write access is not set
- final freeze during second reboot => [ 5221.745854] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (3ffc0001000)

If I stop and restart QEMU at this point I have a new 2min freeze during 1st boot with same fault addr 2affde000

Last edited by adr (2014-01-28 22:40:57)

Offline

#1078 2014-01-29 03:22:33

M4L
Member
Registered: 2014-01-29
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi There

I managed to get my system working with KVM and managed to pass through my HD 7950 and Creative Sound Recon3D to my VMs!

Few problems I get:
I'm just having and wondering if anyone else is. For some apparent reason I can't install Catalyst on my Windows VM.
I got the driver working and have no issues with running games or 3D rendering. It's just handy to have if I ever want to try and crossfire another HD 7950 passed through to my VM.

Anyone else have this problem?

Also I seem to have an issue passing through my USB 3.0 controller. Getting a problem with the device not being properly binded to VFIO. (I'll print the output shortly, just in my Windows VM fixing up a few things)

Finally I decided to not really use my on board Marvell controllers as they seem to not work with IOMMU on. I used this (http://www.spinics.net/lists/linux-pci/msg28094.html) patch to get my 9172 controller working, but the other 9220 was kicking up a fuss. I've got it switched off in the BIOS.
Anyone got a work around for getting it working? At the moment I'm using a fairly old Dell SAS 5/iR controller to just hook up my storage drives and that works alright, but it isn't as fast as the Marvell controllers.

These are my specs of my machine for those interested:
CPU: i7 3960X
Motherboard: ASRock Extreme 9
GPUs: GT 220 for my Arch Host and AMD (XFX DD) HD 7950 for my Arch and Windows VMs
SSD: Samsung 840 Pro 256GB, using LVM and have a logical volume for host and each VM
Sound Card: Creative Recon3D (I have no onboard audio and this replaced dodgy sound Creative Gameblaster card which came with motherboard)
SAS controller: Dell SAS 5/iR (It's an LSI 1068 controller I believe)
HDDs: 4 x 500GB 2.5" drives hooked up to my Intel SATA 2 controller and 4 x 1TB drives hooked up to my SAS card.

Last edited by M4L (2014-01-29 03:22:53)

Offline

#1079 2014-01-29 07:11:37

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Odd... I'm not sure if it's only with the Intel Root Port patch, but I get the following when I try to unload vfio-pci while a few devices are bound to it:

[ 572.443297] kernel tried to execute NX-protected page - exploit attempt? (uid: 0) [ 572.447996] BUG: unable to handle kernel paging request at ffffea000f9fe800 [ 572.450999] IP: [<ffffea000f9fe800>] 0xffffea000f9fe800 [ 572.453592] PGD 47e5f3067 PUD 47e5f2067 PMD 800000047bc001e3 [ 572.456074] Oops: 0011 [#1] SMP [ 572.457579] Modules linked in: snd_virtuoso(F) snd_oxygen_lib(F) snd_mpu401_uart(F) xt_multiport(F) xt_conntrack(F) ipt_REJECT(F) sb_edac(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) ip6table_filter(F) ip6_tables(F) xt_CHECKSUM(F) iptable_mangle(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) iptable_filter(F) ip_tables(F) x_tables(F) autofs4(F) deflate(F) ctr(F) twofish_generic(F) twofish_avx_x86_64(F) twofish_x86_64_3way(F) twofish_x86_64(F) twofish_common(F) camellia_generic(F) camellia_aesni_avx2(F) camellia_aesni_avx_x86_64(F) camellia_x86_64(F) serpent_avx2(F) serpent_avx_x86_64(F) serpent_sse2_x86_64(F) xts(F) serpent_generic(F) blowfish_generic(F) blowfish_x86_64(F) blowfish_common(F) cast5_avx_x86_64(F) cast5_generic(F) cast_common(F) des_generic(F) cmac(F) xcbc(F) rmd160(F) crypto_null(F) af_key(F) xfrm_algo(F) parport_pc(F) ppdev(F) bnep(F) rfcomm(F) bluetooth(F) joydev(F) iTCO_wdt(F) iTCO_vendor_support(F) dm_multipath(F) scsi_dh(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) hid_logitech_dj(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) snd_hda_intel(F) lrw(F) gf128mul(F) snd_hda_codec(F) glue_helper(F) ablk_helper(F) snd_hwdep(F) cryptd(F) snd_seq_midi(F) i915(F) snd_seq_midi_event(F) snd_pcm(F) snd_rawmidi(F) shpchp(F) snd_seq(F) snd_seq_device(F) snd_page_alloc(F) snd_timer(F) drm_kms_helper(F) lpc_ich(F) drm(F) snd(F) soundcore(F) i2c_algo_bit(F) video(F) mac_hid(F) vfio_pci(F-) vfio_iommu_type1(F) vfio(F) w83627ehf(F) hwmon_vid(F) coretemp(F) msr(F) cpuid(F) jc42(F) nfsd(F) binfmt_misc(F) auth_rpcgss(F) nfs_acl(F) lp(F) nfs(F) parport(F) lockd(F) sunrpc(F) fscache(F) nls_iso8859_1(F) zfs(POF) zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) btrfs(F) xor(F) hid_generic(F) raid6_pq(F) libcrc32c(F) usbhid(F) hid(F) firewire_ohci(F) firewire_core(F) crc_itu_t(F) ahci(F) e1000e(F) libahci(F) ptp(F) pps_core(F) microcode(F) [last unloaded: snd_mpu401_uart] [ 572.547493] CPU: 7 PID: 6420 Comm: modprobe Tainted: PF O 3.13.0-acs #1 [ 572.551166] Hardware name: Supermicro X10SAT/X10SAT, BIOS 1.0 08/27/2013 [ 572.555237] task: ffff88040aa98000 ti: ffff88038e808000 task.ti: ffff88038e808000 [ 572.555268] RIP: 0010:[<ffffea000f9fe800>] [<ffffea000f9fe800>] 0xffffea000f9fe800 [ 572.555268] RSP: 0018:ffff88038e809d90 EFLAGS: 00010246 [ 572.555270] RAX: 0000000000000088 RBX: ffff880467e63098 RCX: 0000000000000000 [ 572.555270] RDX: ffff880467f7eb48 RSI: 0000000000320030 RDI: ffff880467e63098 [ 572.555271] RBP: ffff88038e809db8 R08: 0000000000000246 R09: 0000000000000000 [ 572.555271] R10: ffffea0011711700 R11: ffffffffa04d9127 R12: ffff880467e63140 [ 572.555271] R13: ffffea000f9fe800 R14: 0000000000000000 R15: 0000000000000000 [ 572.555272] FS: 00007f72a2402740(0000) GS:ffff88047ebc0000(0000) knlGS:0000000000000000 [ 572.555273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 572.555273] CR2: ffffea000f9fe800 CR3: 000000036782c000 CR4: 00000000001407e0 [ 572.555274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 572.555274] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 572.555274] Stack: [ 572.555275] ffffffff814aabf6 ffff88038e809df8 0000000000000004 ffff880467e63098 [ 572.555276] 0000000000000004 ffff88038e809de8 ffffffff814ab68d 0000000000000004 [ 572.555277] ffff880467e63098 ffff880467e63140 0000000000000246 ffff88038e809e18 [ 572.555278] Call Trace: [ 572.555281] [<ffffffff814aabf6>] ? __rpm_callback+0x36/0xc0 [ 572.555282] [<ffffffff814ab68d>] rpm_idle+0x1bd/0x2b0 [ 572.555283] [<ffffffff814ab7dd>] __pm_runtime_idle+0x5d/0x80 [ 572.555286] [<ffffffff813b95a4>] pci_device_remove+0x74/0xb0 [ 572.555290] [<ffffffff8149f09f>] __device_release_driver+0x7f/0xf0 [ 572.555292] [<ffffffff8149fa38>] driver_detach+0xb8/0xc0 [ 572.555296] [<ffffffff8149ecd5>] bus_remove_driver+0x55/0xd0 [ 572.555298] [<ffffffff814a00ac>] driver_unregister+0x2c/0x50 [ 572.555300] [<ffffffff813b8059>] pci_unregister_driver+0x29/0x90 [ 572.555304] [<ffffffffa04de061>] vfio_pci_cleanup+0x10/0x1c [vfio_pci] [ 572.555307] [<ffffffff810dd812>] SyS_delete_module+0x172/0x220 [ 572.555309] [<ffffffff817389bf>] tracesys+0xe1/0xe6 [ 572.555318] Code: 00 00 00 40 00 40 00 01 00 00 00 00 01 10 00 00 00 ad de 00 02 20 00 00 00 ad de 00 3b 00 6e 04 88 ff ff 00 00 00 00 00 00 00 00 <80> 00 00 00 00 ff ff 02 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 572.555320] RIP [<ffffea000f9fe800>] 0xffffea000f9fe800 [ 572.555320] RSP <ffff88038e809d90> [ 572.555321] CR2: ffffea000f9fe800 [ 572.561468] ---[ end trace 0e53739ef2e93df6 ]---

After this oops, lspci -nnv hangs forever.

Offline

#1080 2014-01-29 14:51:00

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I tried intel_iommu=pt instead of intel_iommu=on, and I still get dmesg with
WARNING: CPU: 2 PID: 1884 at drivers/vfio/vfio_iommu_type1.c:344 vfio_unmap_unpin.isra.7+0x11e/0x120()
[  166.948480] Modules linked in: ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables vhost_net vhost macvtap macvlan tun autofs4 bridge stp llc ipv6 lp parport fuse snd_hda_codec_hdmi btusb bluetooth snd_hda_codec_realtek x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_dev eeepc_wmi asus_wmi sparse_keymap rfkill video mxm_wmi radeon joydev evdev snd_hda_intel snd_hda_codec ttm snd_hwdep drm_kms_helper snd_pcm drm psmouse microcode snd_page_alloc snd_timer serio_raw e1000e snd i2c_i801 agpgart i2c_algo_bit mei_me i2c_core soundcore lpc_ich ptp mei pps_core ehci_pci processor thermal_sys hwmon wmi button loop msdos nls_iso8859_1 nls_cp437 uhci_hcd ehci_hcd pci_stub vfat fat ext2 ext4 jbd2 mbcache hid_generic usbhid hid xhci_hcd usb_storage
[  166.948505] CPU: 2 PID: 1884 Comm: qemu-system-x86 Tainted: G        W    3.13.0 #1
[  166.948506] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 9502 11/07/2013
[  166.948507]  0000000000000009 ffff8800aa9a7d10 ffffffff81644f6e 0000000000000000
[  166.948508]  ffff8800aa9a7d48 ffffffff810be16d 0000000000000000 0000000000000000
[  166.948510]  ffff8800aa9a7e30 0000000000200000 ffff88081ad6aba0 ffff8800aa9a7d58
[  166.948511] Call Trace:
[  166.948513]  [<ffffffff81644f6e>] dump_stack+0x45/0x56
[  166.948514]  [<ffffffff810be16d>] warn_slowpath_common+0x7d/0xa0
[  166.948516]  [<ffffffff810be24a>] warn_slowpath_null+0x1a/0x20
[  166.948518]  [<ffffffff814a433e>] vfio_unmap_unpin.isra.7+0x11e/0x120
[  166.948520]  [<ffffffff814a43a2>] vfio_remove_dma_overlap+0x62/0x250
[  166.948522]  [<ffffffff814a5120>] vfio_iommu_type1_ioctl+0x230/0x290
[  166.948524]  [<ffffffff814a2377>] vfio_fops_unl_ioctl+0x77/0x340
[  166.948525]  [<ffffffff811e8540>] do_vfs_ioctl+0x2e0/0x4c0
[  166.948527]  [<ffffffff811d57ac>] ? vfs_write+0x15c/0x1f0
[  166.948529]  [<ffffffff811d55de>] ? vfs_read+0xee/0x160
[  166.948530]  [<ffffffff811e87a1>] SyS_ioctl+0x81/0xa0
[  166.948532]  [<ffffffff8164e5ed>] system_call_fastpath+0x1a/0x1f
[  166.948532] ---[ end trace c90bd9cca026e365 ]---
[  168.059412] dmar: DRHD: handling fault status reg 102
[  168.059417] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000
[  168.059417] DMAR:[fault reason 06] PTE Read access is not set
[  177.056309] vfio-pci 0000:03:00.0: irq 105 for MSI/MSI-X

but everything seems to work OK.  Haven't tried reinstalling ati-catalyst, I'm going to work on getting 3D acceleration working with the radeon drivers instead for now.  I've tried running under 3.13.0 as well as 3.12.1 and everything seems to work fine without any issues of reboots so far. my qemu-git was downloaded on Dec 5th 2013.

Offline

#1081 2014-01-29 15:01:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

So I tried intel_iommu=pt instead of intel_iommu=on, and I still get dmesg with
WARNING: CPU: 2 PID: 1884 at drivers/vfio/vfio_iommu_type1.c:344 vfio_unmap_unpin.isra.7+0x11e/0x120()

Last time you posted this backtrace I suggested trying this patch https://git.kernel.org/cgit/linux/kerne … fe8702b098

The suggestion remains...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1082 2014-01-29 19:39:20

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah, I haven't gotten to that yet, but I'll give it a try, thanks.   
[EDIT]
So I patched 3.13.0 and now have

[8314.585792] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none
[ 8340.979240] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 8340.979363] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x900
[ 8341.003257] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
[ 8341.114083] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[ 8344.137317] dmar: DRHD: handling fault status reg 2
[ 8344.137323] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000
[ 8344.137323] DMAR:[fault reason 06] PTE Read access is not set
[ 8350.516005] kvm: zapping shadow pages for mmio generation wraparound
[ 8355.387485] vfio-pci 0000:03:00.0: irq 105 for MSI/MSI-X

and on reboot

[ 8503.680819] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[ 8506.570990] dmar: DRHD: handling fault status reg 102
[ 8506.570995] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr afffd000
[ 8506.570995] DMAR:[fault reason 06] PTE Read access is not set
[ 8512.688516] kvm: zapping shadow pages for mmio generation wraparound
[ 8519.895012] vfio-pci 0000:03:00.0: irq 105 for MSI/MSI-X

So I guess that's better than spamming dmesg with tons of repeat page faults.  Thanks again.

Last edited by mostlyharmless (2014-01-30 05:24:03)

Offline

#1083 2014-01-31 03:34:09

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've now received my new video card, and I'd really like to get EFI mode working on it.  I've gotten the thing to show a solid cursor on the Radeon, but no further than that.  With the ubuntu seabios package, I get no debug output -- same with OVMF-X64-r11337-alpha.

My command line:

qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -M q35 -chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios -smp 4,sockets=1,cores=4,threads=1 -readconfig q35-chipset.cfg -device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1 -device vfio-pci,host=09:00.0,bus=ich9-pcie-port-3 -device vfio-pci,host=0c:00.0,bus=ich9-pcie-port-4 -device vfio-pci,host=07:04.0,bus=ich9-pci-bridge -net none -rtc base=localtime -vga none -device sga

If I use an older or newer OVMF, I get:

SecCoreStartupWithStack(0xFFFCC000, 0x818000) Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000 Register PPI Notify: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Register PPI Notify: EA7CA24B-DED5-4DAD-A389-BF827E8F9B38 Install PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: DBE23AA9-A345-4B97-85B6-B226F1617389 Loading PEIM at 0x00000836D80 EntryPoint=0x00000836FE0 PcdPeim.efi Install PPI: 06E81C58-4AD7-44BC-8390-F10265F72480 Install PPI: 01F34D25-4DE2-23AD-3FF3-36353FF323F1 Loading PEIM at 0x0000083FBA0 EntryPoint=0x0000083FE00 StatusCodePei.efi Install PPI: 229832D3-7A30-4B36-B827-F40CB7D45436 Loading PEIM at 0x000008457A0 EntryPoint=0x00000845A00 PlatformPei.efi Platform PEIM Loaded CMOS: 00: 11 00 44 00 19 00 05 30 01 14 26 02 10 80 00 00 10: 40 00 00 00 07 80 02 FF FF 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: FF FF 20 00 00 07 00 20 30 00 00 00 00 12 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Install PPI: 7408D748-FC8C-4EE6-9288-C4BEC092A410 PeiInstallPeiMemory MemoryBegin 0x4000000, MemoryLength 0x4000000 MemDetect called Reserved variable store memory: 0x7FD0000; size: 128kb Platform PEI Firmware Volume Initialization Install PPI: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Notify: PPI Guid: 49EDB1C1-BF21-4761-BB12-EB0031AABB39, Peim notify entry point: 8237F7 The 1th FV start address is 0x00000900000, size is 0x00700000, handle is 0x900000 Temp Stack : BaseAddress=0x814000 Length=0x4000 Temp Heap : BaseAddress=0x810000 Length=0x870 Total temporary memory: 32768 bytes. temporary memory stack ever used: 16384 bytes. temporary memory heap used: 2160 bytes. Old Stack size 16384, New stack size 131072 Heap Offset = 0x0 Stack Offset = 0x3808000 Stack Hob: BaseAddress=0x4000000 Length=0x20000 TemporaryRamMigration(0x810000, 0x401C000, 0x8000) Loading PEIM at 0x00007FB8000 EntryPoint=0x00007FB8260 PeiCore.efi Reinstall PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Reinstall PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A Reinstall PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: F894643D-C449-42D1-8EA8-85BDD8C65BDE Loading PEIM at 0x00007FAE000 EntryPoint=0x00007FAE260 DxeIpl.efi Install PPI: 0AE8CE5D-E448-4437-A8D7-EBF5F194F731 Install PPI: 1A36E4E7-FAB6-476A-8E75-695A0576FDD7 DXE IPL Entry Loading PEIM at 0x00007F75000 EntryPoint=0x00007F75260 DxeCore.efi Loading DXE CORE at 0x00007F75000 EntryPoint=0x00007F75260 Install PPI: 605EA650-C65C-42E1-BA80-91A52AB618C6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7FA0788 HOBLIST address in DXE = 0x7D2A018 Memory Allocation 0x00000006 0x7FD0000 - 0x7FFFFFF Memory Allocation 0x00000004 0x820000 - 0x8FFFFF Memory Allocation 0x00000004 0x900000 - 0xFFFFFF Memory Allocation 0x00000004 0x7F55000 - 0x7F74FFF Memory Allocation 0x00000004 0x7FCF000 - 0x7FCFFFF Memory Allocation 0x00000004 0x7FB8000 - 0x7FCEFFF Memory Allocation 0x00000004 0x7FAE000 - 0x7FB7FFF Memory Allocation 0x00000004 0x7F75000 - 0x7FADFFF Memory Allocation 0x00000003 0x7F75000 - 0x7FADFFF Memory Allocation 0x00000004 0x7F55000 - 0x7F74FFF Memory Allocation 0x00000004 0x7F13000 - 0x7F54FFF Memory Allocation 0x00000004 0x4000000 - 0x401FFFF FV Hob 0x900000 - 0xFFFFFF InstallProtocolInterface: D8117CFE-94A6-11D4-9A3A-0090273FC14D 7FA0220 InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 7D26B30 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7D27698 InstallProtocolInterface: 220E73B6-6BDB-4413-8405-B974B108619A 7D26830 InstallProtocolInterface: EE4E5898-3914-4259-9D6E-DC7BD79403CF 7FA0448 Loading driver 9B680FCE-AD6B-4F3A-B60B-F59899003443 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3E40 Loading driver at 0x00007E9D000 EntryPoint=0x00007E9D2AF DevicePathDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7CB4B18 InstallProtocolInterface: 0379BE4E-D706-437D-B037-EDB82FB772A4 7EA8FE0 InstallProtocolInterface: 8B843E20-8132-4852-90CC-551A4E4A7F1C 7EA9020 InstallProtocolInterface: 05C99A21-C70F-4AD2-8A5F-35DF3343F51E 7EA9030 Loading driver 80CF7257-87AB-47F9-A3FE-D50B76D89541 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3C40 Loading driver at 0x00007E91000 EntryPoint=0x00007E912AF PcdDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7CB4598 InstallProtocolInterface: 11B34006-D85B-4D0A-A290-D5A571310EF7 7E994E0 InstallProtocolInterface: 13A3F0F6-264A-3EF0-F2E0-DEC512342F34 7E995E0 Loading driver 733CBAC2-B23F-4B92-BC8E-FB01CE5907B7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3A40 Loading driver at 0x00007ED4000 EntryPoint=0x00007ED42AF FvbServicesRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7CB4098 QEMU Flash: Attempting flash detection at FFE00010 QemuFlashDetected => FD behaves as ROM QemuFlashDetected => No QEMU flash was not detected. Writable FVB is not being installed. Error: Image at 00007ED4000 start failed: Write Protected Loading driver FEDE0A1B-BCA2-4A9F-BB2B-D9FD7DEC2E9F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3A40 Loading driver at 0x00007ED4000 EntryPoint=0x00007ED42AF StatusCodeRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7CB4118 InstallProtocolInterface: D2B2B828-0826-48A7-B3DF-983C006024F0 7EDA430 Loading driver B601F8C4-43B7-4784-95B1-F4226CB40CEE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3840 Loading driver at 0x00007ECD000 EntryPoint=0x00007ECD2AF RuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C99C18 InstallProtocolInterface: B7DFB4E1-052F-449F-87BE-9818FC91B733 7ED14A0 Loading driver F80697E9-7FD6-4665-8646-88E33EF71DFC InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3640 Loading driver at 0x00007DEF000 EntryPoint=0x00007DEF2AF SecurityStubDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C99898 InstallProtocolInterface: 94AB2F58-1438-4EF1-9152-18941A3A0E68 7E611B8 InstallProtocolInterface: A46423E3-4617-49F1-B9FF-D1BFA9115839 7E611B0 Loading driver 13AC6DD0-73D0-11D4-B06B-00AA00BD6DE7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3440 Loading driver at 0x00007DE3000 EntryPoint=0x00007DE32AF EbcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C99518 InstallProtocolInterface: 13AC6DD1-73D0-11D4-B06B-00AA00BD6DE7 7C99418 InstallProtocolInterface: 2755590C-6F3C-42FA-9EA4-A3BA543CDA25 7C99318 InstallProtocolInterface: AAEACCFD-F27B-4C17-B610-75CA1F2DFB52 7C99218 Loading driver 79CA4208-BBA1-4A9A-8456-E1E66A81484E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3240 Loading driver at 0x00007DDE000 EntryPoint=0x00007DDE2AF Legacy8259.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C97E98 InstallProtocolInterface: 38321DBA-4FE0-4E17-8AEC-413055EAEDC1 7DE0FC0 Loading driver A19B1FE7-C1BC-49F8-875F-54A5D542443F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7CB3040 Loading driver at 0x00007DD8000 EntryPoint=0x00007DD82AF CpuIo2Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C97A98 InstallProtocolInterface: AD61F191-AE5F-4C0E-B9FA-E869D288C64F 7DDB300 Loading driver 1A1E4886-9517-440E-9FDE-3BE44CEE2136 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96E40 Loading driver at 0x00007DC9000 EntryPoint=0x00007DC92AF CpuDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C97698 InstallProtocolInterface: 26BACCB1-6F42-11D4-BCE7-0080C73C8881 7DD2D40 Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Loading driver 6F0198AA-1F1D-426D-AE3E-39AB633FCC28 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96C40 Loading driver at 0x00007EC7000 EntryPoint=0x00007EC72AF KbcReset.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C97398 InstallProtocolInterface: 27CFAC88-46CC-11D4-9A38-0090273FC14D 0 Loading driver C8339973-A563-4561-B858-D8476F9DEFC4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96A40 Loading driver at 0x00007DC4000 EntryPoint=0x00007DC42AF Metronome.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C97018 InstallProtocolInterface: 26BACCB2-6F42-11D4-BCE7-0080C73C8881 7DC7070 Loading driver 22DC2B60-FE40-42AC-B01F-3AB1FAD9AAD8 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96840 Loading driver at 0x00007EBF000 EntryPoint=0x00007EBF2AF EmuVariableFvbRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C1FC98 EMU Variable FVB Started EMU Variable FVB: Using pre-reserved block at 7FD0000 EMU Variable FVB: Basic FV headers were invalid Installing FVB for EMU Variable support InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 7EC3A40 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7EC3A08 Loading driver 79E4A61C-ED73-4312-94FE-E3E7563362A9 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96640 Loading driver at 0x00007DBF000 EntryPoint=0x00007DBF2AF PrintDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C1F798 InstallProtocolInterface: F05976EF-83F1-4F3D-8619-F7595D41E538 7DC2040 Loading driver 348C4D62-BFBD-4882-9ECE-C80BB1C4783B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96440 Loading driver at 0x00007D98000 EntryPoint=0x00007D982AF HiiDatabase.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7C1F398 InstallProtocolInterface: E9CA4775-8657-47FC-97E7-7ED65A084324 7DB8A48 InstallProtocolInterface: 0FD96974-23AA-4CDC-B9CB-98D17750322A 7DB8A90 InstallProtocolInterface: EF9FC172-A1B2-4693-B327-6D32FC416042 7DB8AB8 InstallProtocolInterface: 587E72D7-CC50-4F79-8209-CA291FC1A10F 7DB8B10 InstallProtocolInterface: 31A6406A-6BDF-4E46-B2A2-EBAA89C40920 7DB8A68 Loading driver 96B5C032-DF4C-4B6E-8232-438DCF448D0E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96240 Loading driver at 0x00007D92000 EntryPoint=0x00007D922AF NullMemoryTestDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FED98 InstallProtocolInterface: 309DE7F1-7F5E-4ACE-B49C-531BE5AA95EF 7D95180 Loading driver F9D88642-0737-49BC-81B5-6889CD57D9EA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7C96040 Loading driver at 0x00007D8A000 EntryPoint=0x00007D8A2AF SmbiosDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FE998 InstallProtocolInterface: 03583FF6-CB36-4940-947E-B9B39F4AFAF7 7D8F390 Loading driver 9622E42C-8E38-4A08-9E8F-54F784652F6B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FDE40 Loading driver at 0x00007D7B000 EntryPoint=0x00007D7B2AF AcpiTableDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FE598 InstallProtocolInterface: FFE06BDD-6107-46A6-7BB2-5A9C7EC5275C 7D2B9A8 Loading driver A210F973-229D-4F4D-AA37-9895E6C9EABA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FDC40 Loading driver at 0x00007D75000 EntryPoint=0x00007D752AF DpcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FE198 InstallProtocolInterface: 480F8AE9-0C46-4AA9-BC89-DB9FBA619806 7D78020 Loading driver F2765DEC-6B41-11D5-8E71-00902707B35E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FDA40 Loading driver at 0x00007D70000 EntryPoint=0x00007D702AF Timer.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FBD98 InstallProtocolInterface: 26BACCB3-6F42-11D4-BCE7-0080C73C8881 7D72F40 Loading driver 2383608E-C6D0-4E3E-858D-45DFAC3543D5 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FD840 Loading driver at 0x00007D65000 EntryPoint=0x00007D652AF PciHostBridge.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FBA18 InstallProtocolInterface: CF8034BE-6768-4D8B-B739-7CCE683A9FBE 73FC548 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 7D6CB20 InstallProtocolInterface: 2F707EBB-4A1A-11D4-9A38-0090273FC14D 73CAE08 Loading driver FE5CEA76-4F72-49E8-986F-2CD899DFFE5D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FD640 Loading driver at 0x00007D59000 EntryPoint=0x00007D592AF FaultTolerantWriteDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FB498 Ftw: Remaining work space size - FE0 Ftw: Work block header check error Ftw: Work block header check error Ftw: Both are invalid, init workspace Ftw: start to reclaim work space Ftw: reclaim work space successfully InstallProtocolInterface: 3EBD9E82-2C78-4DE6-9786-8D4BFCB7C881 73B7028 Loading driver EBF342FE-B1D3-4EF8-957C-8048606FF671 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FD440 Loading driver at 0x00007D31000 EntryPoint=0x00007D312AF SetupBrowser.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 73FB098 InstallProtocolInterface: B9D4C360-BCFB-4F9B-9298-53C136982258 7D50F90 InstallProtocolInterface: A770C357-B693-4E6D-A6CF-D21C728E550B 7D50FC0 InstallProtocolInterface: 1F73B18D-4630-43C1-A1DE-6F80855D7DA4 7D50FA0 Loading driver 4110465D-5FF3-4F4B-B580-24ED0D06747A InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FD240 Loading driver at 0x00006C25000 EntryPoint=0x00006C252AF SmbiosPlatformDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 721DA18 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Select Item: 0x8001 Select Item: 0x2 Loading driver 49970331-E3FA-4637-9ABC-3B7868676970 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 73FD040 Loading driver at 0x00006C1D000 EntryPoint=0x00006C1D2AF AcpiPlatform.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 721D618 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Select Item: 0x5 Select Item: 0x19 ACPI using S3/S4 defaults ACPI PciWindow32: Base=0x08000000 End=0xFEEFFFFF Length=0xF6F00000 ACPI PciWindow64: Base=0x00000000 End=0x00000000 Length=0x00000000 Loading driver 2226F30F-3D5B-402D-9936-A97184EB4516 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721AE40 Loading driver at 0x00006B6A000 EntryPoint=0x00006B6A2AF VariableAuthRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71F9E98 InstallProtocolInterface: CD3D0A05-9E24-437C-A891-1EE053DB7638 6BE6278 InstallProtocolInterface: 1E5668E2-8481-11D4-BCF1-0080C73C8881 0 Variable PK does not exist. Variable SetupMode is 1 Variable SecureBoot is 0 Variable SecureBootEnable is 0 Variable CustomMode is 0 Variable VendorKeys is 1 InstallProtocolInterface: 6441F818-6362-4E44-B570-7DBA31DD2453 0 Loading driver F099D67F-71AE-4C36-B2A3-DCEB0EB2B7D8 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721AC40 Loading driver at 0x00006AB7000 EntryPoint=0x00006AB72AF WatchdogTimer.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71F9998 InstallProtocolInterface: 665E3FF5-46CC-11D4-9A38-0090273FC14D 6AB9E10 Loading driver E660EA85-058E-4B55-A54B-F02F83A24707 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721AA40 Loading driver at 0x00006A77000 EntryPoint=0x00006A772AF DisplayEngine.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71F9598 InstallProtocolInterface: 9BBE29E9-FDA1-41EC-AD52-452213742D2E 6A8EC50 Loading driver 378D7B65-8DA9-4773-B6E4-A47826A833E1 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71F7E40 Loading driver at 0x00007EB5000 EntryPoint=0x00007EB52AF PcRtc.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71E1918 InstallProtocolInterface: 27CFAC87-46CC-11D4-9A38-0090273FC14D 0 Loading driver 68E2BC17-198D-41F1-8213-065E3A2A7C12 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721A840 Loading driver at 0x00006A5D000 EntryPoint=0x00006A5D2AF SecureBootConfigDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71E1518 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 6A6E390 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 71E1420 InstallProtocolInterface: 68E2BC17-198D-41F1-8213-065E3A2A7C12 71E1418 Loading driver AD608272-D07F-4964-801E-7BD3B7888652 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DEE40 Loading driver at 0x00007EAF000 EntryPoint=0x00007EAF2AF MonotonicCounterRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DDA98 InstallProtocolInterface: 1DA97072-BDDC-4B30-99F1-72A0B56FFF2A 0 Loading driver 42857F0A-13F2-4B21-8A23-53D3F714B840 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721A640 Loading driver at 0x00006A52000 EntryPoint=0x00006A522AF CapsuleRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DD718 InstallProtocolInterface: 5053697E-2CBC-4819-90D9-0580DEEE5754 0 Loading driver FC5C7020-1A48-4198-9BE2-EAD5ABC8CF2F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721A440 Loading driver at 0x00006A03000 EntryPoint=0x00006A032AF BdsDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DD318 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. InstallProtocolInterface: 665E3FF6-46CC-11D4-9A38-0090273FC14D 6A40018 Loading driver 93B80004-9FB3-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 721A040 Loading driver at 0x000069E4000 EntryPoint=0x000069E42AF PciBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DBD18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69FDB20 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69FDDE0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69FDE00 InstallProtocolInterface: 19CB87AB-2CB9-4665-8360-DDCF6054F79D 69FDB78 Loading driver 33CB97AF-6C33-4C42-986B-07581FA366D4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DAE40 Loading driver at 0x000069DD000 EntryPoint=0x000069DD2AF BlockMmioToBlockIoDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DB718 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69E0D60 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69E1060 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69E1080 Loading driver 83DD3B39-7CAF-4FAC-A542-E050B767E3A7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DAC40 Loading driver at 0x000069D6000 EntryPoint=0x000069D62AF VirtioPciDeviceDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DB498 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69DA520 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69DA580 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69DA5A0 Loading driver 11D92DFB-3CA9-4F93-BA2E-4780ED3E03B5 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DAA40 Loading driver at 0x000069CE000 EntryPoint=0x000069CE2AF VirtioBlkDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71DB198 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69D2DE0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69D2E40 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69D2E60 Loading driver FAB5D4F4-83C0-4AAF-8480-442D11DF6CEA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DA840 Loading driver at 0x000069C6000 EntryPoint=0x000069C62AF VirtioScsiDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71D8E98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69CB2A0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69CB300 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69CB320 Loading driver 51CCF399-4FDF-4E55-A45B-E123F84D456A InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DA640 Loading driver at 0x000069BE000 EntryPoint=0x000069BE2AF ConPlatformDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71D8B98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69C2AA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69C2C80 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69C2CA0 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69C2AE0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69C2C80 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69C2CA0 Loading driver 408EDCEC-CF6D-477C-A5A8-B4844E3DE281 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DA440 Loading driver at 0x000069AF000 EntryPoint=0x000069AF2AF ConSplitterDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71D8718 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69BA880 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69BAE30 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69BAE50 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69BA940 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69BAE70 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69BAE90 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69BA980 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69BAEB0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69BAED0 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69BA8C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69BAEF0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69BAF10 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69BA900 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69BAF30 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69BAF50 InstallProtocolInterface: 387477C1-69C7-11D2-8E39-00A0C969723B 69BA490 InstallProtocolInterface: DD9E7534-7762-4698-8C14-F58517A625AA 69BA4C0 InstallProtocolInterface: 31878C87-0B75-11D5-9A4F-0090273FC14D 69BA518 InstallProtocolInterface: 8D59D32B-C655-4AE9-9B15-F25904992A43 69BA570 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B 69BA650 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B 69BA770 Loading driver CCCB0C28-4B24-11D5-9A5A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DA240 Loading driver at 0x000069A3000 EntryPoint=0x000069A32AF GraphicsConsoleDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71D4998 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69AA580 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69AA840 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69AA860 Loading driver 9E863906-A40F-4875-977F-5B93FF237FC6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71DA040 Loading driver at 0x00006993000 EntryPoint=0x000069932AF TerminalDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71D4498 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 699E1E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 699E6F0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 699E710 Loading driver 6B38F7B4-AD98-40E9-9093-ACA2B5A253C4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5E40 Loading driver at 0x00006989000 EntryPoint=0x000069892AF DiskIoDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71D4198 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 698F680 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 698FBE0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 698FC00 Loading driver 1FA1F39E-FEFF-4AAE-BD7B-38A070A3B609 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5C40 Loading driver at 0x0000697D000 EntryPoint=0x0000697D2AF PartitionDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71B4E98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6985360 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6985510 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6985530 Loading driver CD3BAFB6-50FB-4FE8-8E4E-AB74D2C1A600 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5A40 Loading driver at 0x00006977000 EntryPoint=0x000069772AF EnglishDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71B4B98 InstallProtocolInterface: 1D85CD7F-F43D-11D2-9A0C-0090273FC14D 697A160 InstallProtocolInterface: A4C751FC-23AE-4C3E-92E9-4964CF63F349 697A1A0 Loading driver 0167CCC4-D0F7-4F21-A3EF-9E64B7CDCE8B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5840 Loading driver at 0x0000696D000 EntryPoint=0x0000696D2AF ScsiBus.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71B4798 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69734C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69735F0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6973610 Loading driver 0A66E322-3740-4CCE-AD62-BD172CECCA35 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5640 Loading driver at 0x00006960000 EntryPoint=0x000069602AF ScsiDisk.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71B4498 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6969840 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6969C50 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6969C70 Loading driver 69FD8E47-A161-4550-B01A-5594CEB2B2B2 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5440 Loading driver at 0x0000694A000 EntryPoint=0x0000694A2AF IdeBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 71B4198 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 695B5E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 695BA10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 695BA30 InstallProtocolInterface: 0784924F-E296-11D4-9A49-0090273FC14D 695B8E0 InstallProtocolInterface: 4D330321-025F-4AAC-90D8-5ED900173B63 695B8F0 Loading driver 99549F44-49BB-4820-B9D2-901329412D67 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5240 Loading driver at 0x00006944000 EntryPoint=0x000069442AF IdeController.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7182D98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6947920 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6947B00 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6947B20 Loading driver 38A0EC22-FBE7-4911-8BC1-176E0D6C1DBD InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 71B5040 Loading driver at 0x0000693E000 EntryPoint=0x0000693E2AF IsaAcpi.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7182A98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6941C40 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6941F10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6941EF0 Loading driver 240612B5-A063-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181E40 Loading driver at 0x00006935000 EntryPoint=0x000069352AF IsaBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7182798 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 693B260 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 693B380 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 693B3A0 Loading driver 93B80003-9FB3-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181C40 Loading driver at 0x0000692A000 EntryPoint=0x0000692A2AF IsaSerialDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7182498 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6931E60 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6932180 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69321A0 Loading driver 3DC82376-637B-40A6-A8FC-A565417F2C38 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181A40 Loading driver at 0x0000691D000 EntryPoint=0x0000691D2AF Ps2KeyboardDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7182198 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 69259A0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6925BA0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6925BC0 Loading driver 0ABD8284-6DA3-4616-971A-83A5148067BA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181840 Loading driver at 0x00006911000 EntryPoint=0x000069112AF IsaFloppyDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717FE98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6918DA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6918F90 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6918FB0 Loading driver 961578FE-B6B7-44C3-AF35-6BC705CD2B1F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181640 Loading driver at 0x0000690A000 EntryPoint=0x0000690A570 InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717FB98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 690A418 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 690A448 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 690A460 Loading driver A2F436EA-A127-4EF8-957C-8048606FF670 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181440 Loading driver at 0x000068FB000 EntryPoint=0x000068FB2AF SnpDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717F898 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6905980 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 69061D0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 69061F0 Loading driver 025BBFC7-E6A9-4B8B-82AD-6815A1AEAF4A InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181240 Loading driver at 0x000068E6000 EntryPoint=0x000068E62AF MnpDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717F598 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 68F4FA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68F6480 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68F64A0 Loading driver E4F61863-FE2C-4B56-A8F4-08519BC439DF InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7181040 Loading driver at 0x000068D8000 EntryPoint=0x000068D82AF VlanConfigDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717F298 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 68E1480 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68E1AA0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68E1AC0 Loading driver 529D3F93-E8E9-4E73-B1E1-BDF6A9D50113 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717CE40 Loading driver at 0x000068CA000 EntryPoint=0x000068CA2AF ArpDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717BF98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 68D3240 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68D3580 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68D35A0 Loading driver 94734718-0BBC-47FB-96A5-EE7A5AE6A2AD InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717CC40 Loading driver at 0x000068B5000 EntryPoint=0x000068B52AF Dhcp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717BC98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 68C4700 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68C4E90 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68C4EB0 Loading driver 26841BDE-920A-4E7A-9FBE-637F477143A6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717CA40 Loading driver at 0x000068A5000 EntryPoint=0x000068A52AF Ip4ConfigDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717B998 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 68B0780 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68B0E50 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68B0E70 Loading driver 9FB1A1F3-3B71-4324-B39A-745CBB015FFF InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717C840 Loading driver at 0x0000688A000 EntryPoint=0x0000688A2AF Ip4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717B698 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 689F460 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 689FDF0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 689FE10 Loading driver DC3641B8-2FA8-4ED3-BC1F-F9962A03454B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717C640 Loading driver at 0x00006877000 EntryPoint=0x000068772AF Mtftp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717B398 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6884F40 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68850D0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68850F0 Loading driver 6D6963AB-906D-4A65-A7CA-BD40E5D6AF4D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717C440 Loading driver at 0x00006855000 EntryPoint=0x000068552AF Tcp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 717B098 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 686E8C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 686FE50 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 686FE70 Loading driver 6D6963AB-906D-4A65-A7CA-BD40E5D6AF2B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717C240 Loading driver at 0x00006841000 EntryPoint=0x000068412AF Udp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7178D98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6850240 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 68503C0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 68503E0 Loading driver 3B1DEAB5-C75D-442E-9238-8E2FFB62B0BB InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 717C040 Loading driver at 0x0000682E000 EntryPoint=0x0000682E2AF UefiPxe4BcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7178A98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 683C980 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 683CAB0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 683CAD0 Loading driver 4579B72D-7EC4-4DD4-8486-083C86B182A7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713EE40 Loading driver at 0x0000680E000 EntryPoint=0x0000680E2AF IScsi4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7178798 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 6827500 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 6827630 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6827650 InstallProtocolInterface: 59324945-EC44-4C0D-B1CD-9DB139DF070C 68282F0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 6827720 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 713DC28 Loading driver A92CDB4B-82F1-4E0B-A516-8A655D371524 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713EC40 Loading driver at 0x00006804000 EntryPoint=0x000068042AF VirtioNetDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 713CE98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 680A920 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 680A7E0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 680A800 Loading driver 2FB92EFA-2EE0-4BAE-9EB6-7464125E1EF7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713EA40 Loading driver at 0x000067F5000 EntryPoint=0x000067F52AF UhciDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 713CB98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 67FFD40 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 67FFFF0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 6800010 Loading driver BDFE430E-8F2A-4DB0-9991-6F856594777E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713E840 Loading driver at 0x000067D1000 EntryPoint=0x000067D12AF EhciDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 713C898 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 67DDCA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 67DF2B0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 67DF2D0 Loading driver 240612B7-A063-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713E640 Loading driver at 0x000067BF000 EntryPoint=0x000067BF2AF UsbBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 713C598 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 67CC1E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 67CCC30 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 67CCC50 Loading driver 2D2E62CF-9ECF-43B7-8219-94E7FC713DFE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713E440 Loading driver at 0x000067E7000 EntryPoint=0x000067E72AF UsbKbDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 713C298 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 67F03E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 67F1200 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 67F1220 Loading driver 9FB4B4A7-42C0-4BCD-8540-9BCC6711F83E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713E240 Loading driver at 0x000067B3000 EntryPoint=0x000067B32AF UsbMassStorageDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7139F98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 67BADC0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 67BBAF0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 67BBB10 Loading driver E3752948-B9A1-4770-90C4-DF41C38986BE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 713E040 Loading driver at 0x000067A3000 EntryPoint=0x000067A32AF QemuVideoDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7139C98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 67AA9E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D 67AAFF0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 67AB010 InstallProtocolInterface: 5C198761-16A8-4E69-972C-89D67954F81D 67AAC80 [BdsDxe] Locate Variable Lock protocol - Success [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:PlatformLangCodes [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:LangCodes [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:BootOptionSupport [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:HwErrRecSupport [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:OsIndicationsSupported Variable Driver Auto Update Lang, Lang:eng, PlatformLang:en InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 6A41DF0 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 6A41DA8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 6A41E10 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 6A41DC0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 6A42640 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 6A42678 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B 6A41A60 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 6A41A40 PlatformBdsInit Registered NotifyDevPath Event PlatformBdsPolicyBehavior PCI Bus First Scanning PciBus: Discovered PCI @ [00|00|00] PciBus: Discovered PCI @ [00|1A|00] BAR[4]: Type = Io32; Alignment = 0x1F; Length = 0x20; Offset = 0x20 PciBus: Discovered PCI @ [00|1A|01] BAR[4]: Type = Io32; Alignment = 0x1F; Length = 0x20; Offset = 0x20 PciBus: Discovered PCI @ [00|1A|02] BAR[4]: Type = Io32; Alignment = 0x1F; Length = 0x20; Offset = 0x20 PciBus: Discovered PCI @ [00|1A|07] BAR[0]: Type = Mem32; Alignment = 0xFFF; Length = 0x1000; Offset = 0x10 PciBus: Discovered PCI @ [00|1B|00] BAR[0]: Type = Mem32; Alignment = 0x3FFF; Length = 0x4000; Offset = 0x10 PciBus: Discovered PPB @ [00|1C|00] ASSERT /var/lib/jenkins/jobs/edk2/workspace/rpmbuild/rpm/BUILD/edk2.git/MdeModulePkg/Library/DxeReportStatusCodeLib/ReportStatusCodeLib.c(371): DevicePath != ((void *) 0)

EDIT: Also, when booting without EFI mode, I'm still getting this all the time:

qemu-system-x86_64: hw/pci/pci.c:232: pcibus_reset: Assertion `bus->irq_count[i] == 0' failed

Last edited by DanaGoyette (2014-01-31 04:45:59)

Offline

#1084 2014-02-01 04:15:21

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've given up on EFI+Q35, for now.

Now, EFI with i440fx, on the other hand, works for me.
While wrestling with Q35+EFI, I chopped a couple of assertions out of the OVMF source; no idea if they're part of why i440fx is now working.

My new video card has a GraphicsOutputProtocol ROM, so it works as the boot video device, despite not having x-vga=on or the Intel VGA Arbiter patches applied!
I'm using the Intel graphics as the primary video, and until I set my BIOS not to load the x16 slot's ROM, even a native-booted Windows would fail to load the AMD driver.

My passed-through XHCI controller doesn't work at boot-time; I had to copy the xHCI driver from the "Clover" boot loader and add it as a driver option.

Some quirks: the biggest is that rebooting seems buggy or unreliable.  I sometimes have to entirely restart qemu because the reset hangs.

My command line now (with extra debugging enabled):

qemu-system-x86_64 -enable-kvm -bios /usr/share/ovmf/OVMF-new.fd -m 5G -boot order=dc -chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=01:00.0 -device vfio-pci,host=09:00.0 -device vfio-pci,host=0c:00.0 -device vfio-pci,host=07:04.0 -usb -hda /dev/disk/by-id/ata-ST9500420AS_5VJDD6R9 -device usb-tablet -net none -rtc base=localtime -vga none -device sga

EDIT: I'll bet my issues with reboot are the same as the GPU reset issues mentioned above.  My host won't suspend (even without VMs running), so I can't just suspend to get the GPU alive again.

EDIT: I tried to run 3DMark in the guest, and the result was a hang, and a GPU rendered inoperative.  Looks like the device reset code is indeed broken.

EDIT: For once, my host actually suspended and resumed properly this time.  Result: GPU has revived.

Last edited by DanaGoyette (2014-02-02 07:38:24)

Offline

#1085 2014-02-02 14:12:33

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

Thanks for this Thread, thats what i was looking for for years.

My System is:
Intel i5 4670
Asus H87-Plus  both confirmed VT-d support.
Host Graphics: Intel HD 4000
Guest Graphics: Geforce GFX560Ti

My Host is a fresh Arch install from last week.
My Guest should be Win8 or 8.1.

I installed all the Pakages from the OP in the first Post from 22-01-14

Using his Script with cpuset and so on. My Geforce is pci-stubbed and vfio-bind is successful to. But the card wont start. Monitor is in standby. When i use virt-manager with the card passedthrough i get "Code 43" in Windows 8.

lspci 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)
ls -l /sys/bus/pci/drivers/vfio-pci/ lrwxrwxrwx 1 root root 0 2. Feb 15:08 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 2. Feb 15:08 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 --w------- 1 root root 4096 2. Feb 15:08 bind lrwxrwxrwx 1 root root 0 2. Feb 15:08 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 2. Feb 15:07 new_id --w------- 1 root root 4096 2. Feb 15:08 remove_id --w------- 1 root root 4096 2. Feb 14:35 uevent --w------- 1 root root 4096 2. Feb 15:07 unbind

My Gummiboot Commandline is as follows:

root=/dev/sda4 rw init=/usr/lib/systemd/systemd add_efi_memmap pci-stub.ids=10de:1200,10de:0e0c intel_iommu=on,igfx_on vfio_iommu_type1.allow_unsafe_interrupts=1

Any Ideas what i'am missing?

Thanks in advance

Tec

EDIT: here is my qemu start script, i removed the pci root, in testing but with no succsess

#!/bin/sh CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l) QEMU=/usr/bin/qemu-system-x86_64 CSET=/usr/bin/cset VCPUS="3" #must be > 0 < cpu core number CPU="Haswell" MEM="8192" #MEM_PATH="/dev/hugepages" BIOS="/usr/share/qemu/bios.bin" #NET="-netdev bridge,br=br0,id=hostnet0 -device virtio-net-#pci,netdev=hostnet0,id=net0" USB_DEVICES="" PCI_DEVICES="" GPU="01:00.0" GPU_SND="01:00.1" SND=true SND_DRIVER_OPTS="QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa " VBIOS="/home/alex/GTX560Ti.rom" CDROM="/home/alex/win8cd.iso" HDD="/home/alex/windows.img" QEMU_ARGS=" -M q35 -enable-kvm -monitor stdio" EXTRA_ARGS="-boot menu=on" if [ -n "$GPU" ]; then EXTRA_ARGS+=" -vga none -nographic" DEV+="" if [ -n "$VBIOS" ]; then DEV+=" -device vfio-pci,host=$GPU,x-vga=on,romfile=$VBIOS" else DEV+=" -device vfio-pci,host=$GPU,x-vga=on" fi if [ -n "$GPU_SND" ]; then DEV+=" -device vfio-pci,host=$GPU_SND" fi fi if [ -n "$PCI_DEVICES" ]; then for d in $PCI_DEVICES; do DEV+=" -device vfio-pci,host=$d" done fi if [ -n "$HDD" ] || [ -n "$CDROM" ]; then DEV+=" -device ahci,bus=pcie.0,id=ahci" if [ -n "$HDD" ]; then STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk" fi if [ -n "$CDROM" ]; then STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" fi fi if [ -n "$CPU" ]; then CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" else CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1" fi if [ -n "$BIOS" ]; then BIOS=" -bios $BIOS" fi if $SND ; then SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,cad=0" export $SND_DRIVER_OPTS else SND="" fi if [ -n "$MEM" ]; then MEMORY+="-m $MEM" if [ -n "$MEM_PATH" ]; then MEMORY+=" -mem-prealloc -mem-path $MEM_PATH" fi fi if [ -n "$USB_DEVICES" ]; then for u in $USB_DEVICES; do USB_DEV+=" -usbdevice $u" done fi if [ $VCPUS -lt $CPU_CORE_NR ]; then $CSET shield -c $((CPU_CORE_NR-VCPUS))-$((CPU_CORE_NR-1)) set -m $QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS & PID=$! ( while ( (kill -s 0 $PID > /dev/null 2>&1) ); do if [ "$(grep Threads /proc/$PID/status | awk '{print $2}' )" -ge "$((VCPUS+1))" > /dev/null 2>&1 ]; then i=$((CPU_CORE_NR-VCPUS)) for t in `ls /proc/$PID/task/`; do if [ $i -ne $((CPU_CORE_NR)) ]; then if [ $t -ne $PID ];then $CSET shield --shield --pid $t taskset -pc $i $t let i++ fi fi done break fi done ) & fg %1 wait $CSET shield --reset else echo "VCPU count $VCPUS is greater or equal than HOST CPU core number $CPU_CORE_NR" fi echo "All Done!!"

Last edited by tecnologic (2014-02-02 14:33:46)

Offline

#1086 2014-02-02 19:44:27

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Argh, I just got stuck in a loop of Windows 8.1 "Working on updates", then getting stuck upon reboot because the bus reset doesn't work.  Every single time, I have to suspend and resume the host before the guest will boot.

EDIT: I'm now resorting to running qemu with no passthrough devices -- maybe then it will finish the updates.

Last edited by DanaGoyette (2014-02-02 19:55:18)

Offline

#1087 2014-02-02 19:50:25

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi some pages before there was n win application postet to detach the PCI device, maybe this helb on reboot of the guest system.

Offline

#1088 2014-02-02 20:02:00

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It finally finished, after what must've been at least 10 reboots.

The problem with the Group Policy method is that I don't know if it will help when the system is rebooting before reaching the desktop.

I also just got the following messages (for the first time ever) upon re-adding the passthrough:

[ 2027.315800] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (c2000000c0001000)
qemu-system-x86_64: vfio_dma_map(0x7fe1a4de4e30, 0xc2000000c0000000, 0x10000000, 0x7fe028000000) = -14 (Bad address)
[ 2029.066767] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (c2000000c0001000)
qemu-system-x86_64: vfio_dma_map(0x7fe1a4de4e30, 0xc2000000c0000000, 0x10000000, 0x7fe028000000) = -14 (Bad address)

Last edited by DanaGoyette (2014-02-02 20:05:36)

Offline

#1089 2014-02-02 21:43:55

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!
I have an optimus laptop... Here somebody says passthrough isn't possible, but I'd like to give it a try anyway. Since passthrough works so well on my desktop, I just can't resist the urge to make all systems passthrough capable. smile

Laptop: Clevo w350st
CPU: i7-4800MQ (supports vt-d)
Chipset: HM87 (supports vt-d)
iGPU: Intel HD4600
dGPU: GeForce 765M
It goes without saying the main show stopper is the dGPU.

First, it has no direct video output. The plan is to use remote desktop solutions or streaming if passthrough will work.

Secondly, it's not a VGA compatible controller. lspci:
01:00.0 3D controller: NVIDIA Corporation GK106M [GeForce GTX 765M] (rev a1)
x-vga is out of question. Or is it?

Question for gurus. Is it possible to add VGA compatibility by pushing to qemu a vbios from a card that has such functions? Like rom=765MwithVBIOS.bin.
Also, how can I check whether the GPU supports UEFI GOP? Perhaps I could use it instead of vga?

If all that won't happen, maybe there's a chance it would work as a secondary GPU in the guest? Have there been any successful secondary passthroughs with NVIDIA cards?

I've tried two configurations (common parts are qemu-git from today, seabios supplied by qemu, machine q35-2.0, primary gpu cirrus, host kernel 3.13.1, no nvidia drivers on the host):
1. Connecting the GPU to pcie port as explained in the first post (sans x-vga) produces BSOD on boot "PROCESS1_INITIALIZATION_FAILED."
2. Letting virt-manager connect the GPU (-device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x6) makes Win7 load, but the GPU is dead "The device cannot start (Code 10)." Interestingly, I found mysterious "NVIDIA Virtual Audio Device (WaveExtensible) (WDM)" which is connected to "Location Unknown" and is "working properly" according to Win7. Where did it come from? 765M has no HDMI audio 01:00.1.

Offline

#1090 2014-02-02 21:53:08

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tecnologic wrote:

[…]When i use virt-manager with the card passedthrough i get "Code 43" in Windows 8.[…]

This error has popped up in this thread several times, but I think it was only with intel integrated graphics, which I don't use so I have no idea how to fix it, but there should be a solution to it somewhere around this thread.


i'm sorry for my poor english wirting skills…

Offline

#1091 2014-02-02 23:10:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
tecnologic wrote:

[…]When i use virt-manager with the card passedthrough i get "Code 43" in Windows 8.[…]

This error has popped up in this thread several times, but I think it was only with intel integrated graphics, which I don't use so I have no idea how to fix it, but there should be a solution to it somewhere around this thread.

Also try without my script, it wasnt ment for general use, it just happens to be the one i use

Last edited by nbhs (2014-02-02 23:12:14)

Offline

#1092 2014-02-02 23:36:32

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all

@andy123: i searched the thread for Code 43 and found that i had to use the qemu nosnoop patch from "aw", i downloaded his last qemu-vfio tag an installed it but i got errors on qemu start, I'll post the errors tomorrow, cause i need to sleep due to an early meeting at work sad.

@nbhs: in my opinion i used your exact setup from the first post. But on the point of testing the qemu with graphics the display was black. What will bie the minimal qemu commandline?

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

or without q35?

as next step i will examin the error Messages from qemu with "aw"'s latest tag.  According to the change logs there are many fixes made for Geforce.

Is it still useful to blacklist i915?

Greetings until tomorrow.

Tec

EDIT: i just started the commandline above an got the following dmesg output.

[ 4953.189292] ------------[ cut here ]------------ [ 4953.189299] WARNING: CPU: 0 PID: 2299 at kernel/jump_label.c:81 __static_key_slow_dec+0xa6/0xb0() [ 4953.189300] jump label: negative count! [ 4953.189300] Modules linked in: usbhid ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp bridge stp llc fuse ip6table_filter ip6_tables iptable_filter ip_tables x_tables vfio_pci vfio_iommu_type1 vfio nls_cp437 vfat fat eeepc_wmi asus_wmi sparse_keymap rfkill iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal coretemp kvm_intel nls_utf8 kvm ntfs crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd microcode joydev pcspkr serio_raw i2c_i801 r8169 mii snd_hda_codec_realtek snd_hda_codec_hdmi fan thermal snd_hda_intel snd_hda_codec snd_hwdep snd_pcm wmi snd_page_alloc evdev snd_timer lpc_ich snd mei_me shpchp [ 4953.189331] mei processor soundcore ext4 crc16 mbcache jbd2 hid_generic hid sr_mod cdrom sd_mod ahci libahci libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common pci_stub i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core [last unloaded: usbhid] [ 4953.189345] CPU: 0 PID: 2299 Comm: qemu-system-x86 Not tainted 3.12.9-2-ARCH #1 [ 4953.189346] Hardware name: ASUS All Series/H87-PLUS, BIOS 0805 01/03/2014 [ 4953.189347] 0000000000000009 ffff8803f3693a98 ffffffff814ed0d3 ffff8803f3693ae0 [ 4953.189349] ffff8803f3693ad0 ffffffff81062aed ffffffffa07a5840 000000000000012c [ 4953.189350] ffffffffa07a5860 ffff8803f378c130 ffff88040f19a1e8 ffff8803f3693b30 [ 4953.189352] Call Trace: [ 4953.189356] [<ffffffff814ed0d3>] dump_stack+0x54/0x8d [ 4953.189359] [<ffffffff81062aed>] warn_slowpath_common+0x7d/0xa0 [ 4953.189362] [<ffffffff81062b5c>] warn_slowpath_fmt+0x4c/0x50 [ 4953.189364] [<ffffffff81088575>] ? lock_hrtimer_base.isra.20+0x25/0x50 [ 4953.189372] [<ffffffffa0770360>] ? kvm_arch_vcpu_uninit+0x20/0x90 [kvm] [ 4953.189374] [<ffffffff811345b6>] __static_key_slow_dec+0xa6/0xb0 [ 4953.189375] [<ffffffff811345f6>] static_key_slow_dec_deferred+0x16/0x20 [ 4953.189380] [<ffffffffa0788a8c>] kvm_free_lapic+0x6c/0xa0 [kvm] [ 4953.189384] [<ffffffffa0770368>] kvm_arch_vcpu_uninit+0x28/0x90 [kvm] [ 4953.189387] [<ffffffffa0755c91>] kvm_vcpu_uninit+0x21/0x30 [kvm] [ 4953.189390] [<ffffffffa07df457>] vmx_free_vcpu+0x47/0x70 [kvm_intel] [ 4953.189394] [<ffffffffa076fa54>] kvm_arch_vcpu_free+0x44/0x50 [kvm] [ 4953.189398] [<ffffffffa0770592>] kvm_arch_destroy_vm+0xf2/0x1f0 [kvm] [ 4953.189400] [<ffffffff8108a2ed>] ? synchronize_srcu+0x1d/0x20 [ 4953.189404] [<ffffffffa07584ee>] kvm_put_kvm+0x10e/0x1c0 [kvm] [ 4953.189407] [<ffffffffa07585d8>] kvm_vcpu_release+0x18/0x20 [kvm] [ 4953.189409] [<ffffffff811a62c1>] __fput+0xa1/0x230 [ 4953.189410] [<ffffffff811a649e>] ____fput+0xe/0x10 [ 4953.189413] [<ffffffff81081c1c>] task_work_run+0xbc/0xe0 [ 4953.189414] [<ffffffff81064f8e>] do_exit+0x2be/0xab0 [ 4953.189416] [<ffffffff810657ff>] do_group_exit+0x3f/0xa0 [ 4953.189418] [<ffffffff810747ad>] get_signal_to_deliver+0x2ad/0x610 [ 4953.189420] [<ffffffff810144d8>] do_signal+0x48/0x940 [ 4953.189422] [<ffffffff81014e38>] do_notify_resume+0x68/0xa0 [ 4953.189424] [<ffffffff814fbeaa>] int_signal+0x12/0x17 [ 4953.189425] ---[ end trace 2f6c48f77244ce60 ]--- [ 5091.775617] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 5093.831760] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5093.831774] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5093.831785] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5093.831842] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5093.831860] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5093.831877] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5096.565189] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5096.565200] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

Whats this "Unclaimed register before interrupt" anyone an hint?

Last edited by tecnologic (2014-02-03 19:11:08)

Offline

#1093 2014-02-03 23:37:39

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So i've done further testing.

First my Bios, i set igpu as primary, shared mem with PCIE GPU is off(caused problems with x starting).

when i now run a minimal qemu cmd

vfio-bind 0000:01:00.0 0000:01:00.1 qemu-system-x86_64 -enable-kvm -M q35 -net none -cpu host -boot menu=on -monitor stdio -debugcon file:/dev/stderr -global isa-debugcon.iobase=0x402 -bios /usr/share/qemu/bios.bin -m 5G -smp 2,sockets=1,cores=2,threads=1 -vga none -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,romfile=GTX560Ti2.rom

i get plenty of those errors in dmesg:

[drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

and qemu says:

MP table addr=0x000f1ad0 MPC table addr=0x000f1ae0 size=208 SMBIOS ptr=0x000f1ab0 table=0x000f1930 size=381 Scan for VGA option rom Running option rom at c000:0003 KVM internal error. Suberror: 1 emulation failure EAX=ffff5151 EBX=00000000 ECX=00000000 EDX=00000300 ESI=00006d41 EDI=00006d5a EBP=00006d5a ESP=0000dd5c EIP=0004eaef EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =5151 00051510 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =c000 000c0000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6d38 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

so when dmesg say some thing of interrupt i thought of this "vfio_iommu_type1.allow_unsafe_interrupts=1" kernel option. But what does this option i found no documentation, only mailing list, which i do not right understand.

EDIT: i added "vfio_iommu_type1.allow_unsafe_interrupts=1" to my boot options, stil get Unclaimed register errors and qemu stops with Cpu register dum message.


EDIT2: nbhs mentions in the first post blacklisting the nvidia drivers, and blacklisting i915 was also mentioned. so i tryed this but lspic -k says, i915 is used. But reading futher i was sad that i915 and vga arbiter patches are included in 3.13 kernel build from nbhs. An this is the kernel i use actually.  Strange sad

00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) Subsystem: ASUSTeK Computer Inc. Device 8534 Kernel driver in use: i915 Kernel modules: i915 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) Subsystem: CardExpert Technology Device 0401 Kernel driver in use: vfio-pci Kernel modules: nouveau 01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1) Subsystem: CardExpert Technology Device 0401 Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

any chanches that using a second Geforce as host graphics could be more successful?

any ideas?

Thanks

Tec

Last edited by tecnologic (2014-02-03 23:48:54)

Offline

#1094 2014-02-03 23:48:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tecnologic wrote:

So i've done further testing.

First my Bios, i set igpu as primary, shared mem with PCIE GPU is off(caused problems with x starting).

when i now run a minimal qemu cmd

vfio-bind 0000:01:00.0 0000:01:00.1 qemu-system-x86_64 -enable-kvm -M q35 -net none -cpu host -boot menu=on -monitor stdio -debugcon file:/dev/stderr -global isa-debugcon.iobase=0x402 -bios /usr/share/qemu/bios.bin -m 5G -smp 2,sockets=1,cores=2,threads=1 -vga none -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,romfile=GTX560Ti2.rom

i get plenty of those errors in dmesg:

[drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

This means the VGA accesses of the Nvidia ROM are going to the iGPU and causing problems, which means you don't have the VGA arbiter patches.

and qemu says:

MP table addr=0x000f1ad0 MPC table addr=0x000f1ae0 size=208 SMBIOS ptr=0x000f1ab0 table=0x000f1930 size=381 Scan for VGA option rom Running option rom at c000:0003 KVM internal error. Suberror: 1 emulation failure EAX=ffff5151 EBX=00000000 ECX=00000000 EDX=00000300 ESI=00006d41 EDI=00006d5a EBP=00006d5a ESP=0000dd5c EIP=0004eaef EFL=00010007 [-----PC] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =5151 00051510 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =c000 000c0000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6d38 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

so when dmesg say some thing of interrupt i thought of this "vfio_iommu_type1.allow_unsafe_interrupts=1" kernel option. But what does this option i found no documentation, only mailing list, which i do not right understand.

allow_unsafe_interrupts is simply a switch that allows vfio to work when an interrupt remapper is not present.  Intel added interrupt remapping support in VT-d2 and among other things, it protects the kernel from MSI injection attacks by guests with an assigned device.  If you need it, you've already had to opt-in to the option, if you don't need it, adding the option isn't going to change anything.

Perhaps try setting the emulate_invalid_guest_state module option for kvm_intel to 0.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1095 2014-02-03 23:56:29

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi aw,

This means the VGA accesses of the Nvidia ROM are going to the iGPU and causing problems, which means you don't have the VGA arbiter patches.

How can i check that? is there an cmd?

Perhaps try setting the emulate_invalid_guest_state module option for kvm_intel to 0.


done an restarting...

[root@Alex alex]# echo "options kvm_intel emulate_invalid_guest_state=0" >> /etc/modprobe.d/kvm-intel.conf [root@Alex alex]# cat /etc/modprobe.d/kvm-intel.conf options kvm_intel emulate_invalid_guest_state=0

EDIT: Strange, same error as before adding the option, but i heard my graphicscard turn is fan on. so there is some thing going on. So i need to fix the arbiter. Can i pull the linux-vfio tag "vfio-v3.14-rc1" from your github to solve that, or should i use a dedicated patch an add it to an pkgbuild?

Tec

Last edited by tecnologic (2014-02-04 00:04:11)

Offline

#1096 2014-02-04 00:00:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tecnologic wrote:

Hi aw,

This means the VGA accesses of the Nvidia ROM are going to the iGPU and causing problems, which means you don't have the VGA arbiter patches.

How can i check that? is there an cmd?

No, but the driver message is often accompanied by seeing screen corruption on the iGPU display across the top few pixels of the screen (running in VT mode).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1097 2014-02-04 00:06:06

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

no screen corruption here, i got 2 screens in the intel card and the geforce on the digital of one of the screens so i can switch the inputs.

Offline

#1098 2014-02-04 00:11:38

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

11 days ago
vfio-v3.14-rc1 …

    3be3a07

6 days ago
vfio-pci-for-qemu-20140128.0 …

    8b6d140

seems a good combo, from your repos, are these supposed to run together?

Offline

#1099 2014-02-04 00:12:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tecnologic wrote:

EDIT: Strange, same error as before adding the option, but i heard my graphicscard turn is fan on. so there is some thing going on. So i need to fix the arbiter. Can i pull the linux-vfio tag "vfio-v3.14-rc1" from your github to solve that, or should i use a dedicated patch an add it to an pkgbuild?

No, I don't carry the patches in my tree.  They've been upstream, caused DRI to be disabled in the host and got reverted.  We need to re-architect the VGA arbiter, I have some ideas but I haven't had time to work on it.  You need these patches:

http://git.kernel.org/cgit/linux/kernel … f81dea8a7d
http://git.kernel.org/cgit/linux/kernel … ba24389bee

They don't apply cleanly, but it should be pretty easy to figure out the conflict (just more stuff being added to the header since the patch).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1100 2014-02-04 00:14:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tecnologic wrote:

11 days ago
vfio-v3.14-rc1 …

    3be3a07

6 days ago
vfio-pci-for-qemu-20140128.0 …

    8b6d140

seems a good combo, from your repos, are these supposed to run together?

They can, but you'll still need the patches in the previous post.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1101 2014-02-04 00:18:46

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok stupid question, how do i pull the patch? Clone Kernel repo and then?

sry i'am not used to such deep linux things, but wanna get in to it. i'am more at home on embedded microprocessors.

Offline

#1102 2014-02-04 00:21:30

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok found a discription for git apply.

are there any problems with the 3.14rc1? if not i'll take that to try my luck on this one

Offline

#1103 2014-02-04 00:37:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tecnologic wrote:

ok found a discription for git apply.

are there any problems with the 3.14rc1? if not i'll take that to try my luck on this one

v3.13 is likely a more stable target


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1104 2014-02-04 00:44:47

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i'll use that. but this will take some time i need to read about this patching. how to apply your links to a branch

Offline

#1105 2014-02-04 12:59:13

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone found a way to reliably rebind usb devices to the host after the VM is shut down?
I tried to

echo $bus-$port > /sys/bus/usb/drivers_probe

but it does not seem work. Any ideas?

Offline

#1106 2014-02-04 16:09:01

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe it helps to eject the devices when shutting down the VM? some pages before there was an .exe mentiont to eject GPU maybe it works for USB parts to.

Offline

#1107 2014-02-04 17:15:28

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't test it right now, but I don't think one can eject a usb keyboard?

Offline

#1108 2014-02-04 17:39:38

toma222
Member
From: France
Registered: 2007-05-17
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

toma222 wrote:

Hello,

I used my Radeon 7850 in a Windows 7/8/8.1 guest with pci-assign for months (with the only issue of the host crash in case of guest reboot).

Now, I want to use it with a Linux guest (Ubuntu) with vfio-pci, to make a kind of virtual Steam Machine + XBMC.

I have tried :
- secondary pasthrough (with a cirrus emulated graphic card) with vfio-pci : the guest start, the Radeon is detected and I have the display on the HDMI output, Unity start, I can play video, but when I try to use 3D (for example with fglrxinfo or Steam), I have only segmentation fault.
- primary passtrough with x-vga=on : I have built the 3.13 kernel (with options and patches from first post) and qemu from git, but I have exactly the same behaviour.

I have also tried with an old Radeon 6850 in secondary passthrough and vfio-pci, and it work quite well (except  some strange performance in Serious Sam 3).

Do you have an idea why I can't get the Radeon 7850 working normally ?

My host use a Core i7 4770S and is on Debian.

Thanks.

Just to let you know, with the latest commits in qemu.git, I now have my Radeon 7850 working normally in my Linux guest.
It seems that the issue was that before I can't start the guest with CPU model other than qemu64. Now it works with host model and the 3D is Ok.
I have tested secondary passthrough with unpatched kernel 3.13 from Debian and primary passthrough with patched one, and it work great for both (except for the memory not releases when the guest shutdown without the fix_memleak patch).

So thank you Alex for your work and all others for all the information in this thread.

Offline

#1109 2014-02-05 19:46:24

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've read this thread now for 8 months or so and I'm the guy who created this crude fix for the NVIDIA binary driver. Just to follow-up. I've already informed NVIDIA in their DevTalk Forum about this issue and it looks that there is some progress since January this year. You can follow it here:

https://devtalk.nvidia.com/default/topi … ation-lock

Anybody who needs this fixed should reply to it so that NVIDIA sees importance for it. Another thing that somebody here might observed and found a solution already is multi GPU passthrough to a single VM. I've asked already on qemu-devel and maybe Alex is reading it soon. I got multi GPU passthrough only working with Linux VMs for now but not with Windows. You can follow it here:

http://lists.nongnu.org/archive/html/qe … 00753.html

Any help or investigation to get it working is much appreciated. smile

--Maik

Offline

#1110 2014-02-05 20:06:13

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

Hi,

I've read this thread now for 8 months or so and I'm the guy who created this crude fix for the NVIDIA binary driver. Just to follow-up. I've already informed NVIDIA in their DevTalk Forum about this issue and it looks that there is some progress since January this year. You can follow it here:

https://devtalk.nvidia.com/default/topi … ation-lock

Anybody who needs this fixed should reply to it so that NVIDIA sees importance for it. Another thing that somebody here might observed and found a solution already is multi GPU passthrough to a single VM. I've asked already on qemu-devel and maybe Alex is reading it soon. I got multi GPU passthrough only working with Linux VMs for now but not with Windows. You can follow it here:

http://lists.nongnu.org/archive/html/qe … 00753.html

Any help or investigation to get it working is much appreciated. smile

--Maik

Hi, thanks for creating this patch. I'm ajs124 on the nvidia forum. Seems like you failed to respond to the Nvidia guy, too. They have a strange notification system over there, so it took me months to follow up on his request for the nvidia-bug-report.log.gz and steps to reproduce the issue.

More replies/traffic over at nvidia devtalk from the people using the patch seems like a good idea to increase visibilty and maybe have this fixed upstream at some point.

I think I was credited for this patch in the original post for some reason. Maybe nbhs should change my name there to yours ^^

Last edited by andy123 (2014-02-05 20:07:20)


i'm sorry for my poor english wirting skills…

Offline

#1111 2014-02-05 21:36:49

teekay
Member
Registered: 2011-10-26
Posts: 267

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mbroemme: responded there. Thanks a hundret times for the patch. W/o it my setup wouldn't work.

@andy123: something completely different. I rember you also have a Gigabyte board with funny Marvell 88SE91xx controller(s). See https://bugzilla.kernel.org/show_bug.cgi?id=42679#c30 for a patch and check if it helps. Andrew needs some more AMD testers. With that patch my controllers are fully usable on the host itself, and passthrough also works fine (though I meanwhile use a ZFS ZVOL for the VM rootfs).

Offline

#1112 2014-02-05 21:53:46

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

@mbroemme: responded there. Thanks a hundret times for the patch. W/o it my setup wouldn't work.

Thanks, but honestly most thanks go to aw who points me to the right direction on what to do. smile

Offline

#1113 2014-02-05 22:00:47

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was also able to retrieve the following output by running 'cat /proc/kmsg' during the crash

<6>[ 996.181525] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) <6>[ 996.207159] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 <6>[ 998.065072] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem <6>[ 998.065076] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:02:00.0 <3>[ 998.095398] ata3: illegal qc_active transition (00000001->97b282a8) <3>[ 998.095540] ata3.00: exception Emask 0x2 SAct 0x0 SErr 0xd0000000 action 0x6 frozen <3>[ 998.095567] ata3: SError: { } <3>[ 998.095580] ata3.00: failed command: FLUSH CACHE EXT <3>[ 998.095607] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 <3>[ 998.095607] res 40/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation) <3>[ 998.095637] ata3.00: status: { DRDY } <6>[ 998.095653] ata3: hard resetting link <4>[ 998.103575] Clocksource tsc unstable (delta = -147227892866 ns)

Thanks again for your help!

I had the exact same issue and I observed the same if I changed the NVIDIA card with an AMD using radeon driver. The solution for me was very simple but took me a couple of days to figure out.

After changing the default VGA device to use in BIOS of my GA-990FX-UD3 to the device which will be assigned to the VM later it started to work. So currently I do the following: I boot system via VGA on AMD Radeon card, bind it during boot via udev (before loading nouveau or radeon module) to VFIO and if nouveau or radeon is loaded it enables my secondary device which will be used for host system. After that I'm able to passthrough the AMD card (used for booting host) to a VM.

Offline

#1114 2014-02-05 23:54:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

...
I think I was credited for this patch in the original post for some reason. Maybe nbhs should change my name there to yours ^^

Actually i was crediting you for the link, but i also linked the original conversation on qemu-devel, and mbroemme thank you a lot for this patch smile

Offline

#1115 2014-02-06 00:02:54

Tom_B
Member
Registered: 2014-02-05
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this awesome topic, I've been following this thread for a while and and I'm planning a system build that allows VGA passthrough but I have some questions:

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise
2) Can you run gpus from the same manufacturer in both the host and the guest OS? e.g. 2 radeons or 2 geforce cards? As I understand it there is a limitation on the catalyst drivers which means you cant use the official driver in the host, which isn't ideal

Offline

#1116 2014-02-06 00:10:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tom_B wrote:

Thanks for this awesome topic, I've been following this thread for a while and and I'm planning a system build that allows VGA passthrough but I have some questions:

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise
2) Can you run gpus from the same manufacturer in both the host and the guest OS? e.g. 2 radeons or 2 geforce cards? As I understand it there is a limitation on the catalyst drivers which means you cant use the official driver in the host, which isn't ideal

1) iamtakingiteasy got a geforce 780 GTX working on a vm, also if you look back on the thread there's been a lot of nvidia cards reported working
2) I'm using 2 radeon cards myself, different models though using the open-source drivers, i havent tried the catalyst drivers but some people have reported problems using them on the host

Offline

#1117 2014-02-06 10:19:18

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tom_B wrote:

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise

The modding info is (or was?) only true for Xen VGA Passthrough as far as I know.

Tom_B wrote:

2) Can you run gpus from the same manufacturer in both the host and the guest OS? e.g. 2 radeons or 2 geforce cards? As I understand it there is a limitation on the catalyst drivers which means you cant use the official driver in the host, which isn't ideal

A friend of mine runs two different nvidia cards and it seems to work. An advantage is that you would not need the i915 patch in that case.

Offline

#1118 2014-02-06 11:40:23

tecnologic
Member
Registered: 2014-02-02
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

i made some progress, i saw my first bios screen on the passthrough Geforce. Thanks to aw for the i915 kernel patches. But when i kill qemu due to no boot device, and start it a second time the card does not respond and the host freezes. When i use vrit-manager the passthrough does not funktion, but it resets the card properly.

so I need to look back in the thread i think the reset problem was mentioned before.

Offline

#1119 2014-02-06 16:25:18

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

teekay wrote:

@mbroemme: responded there. Thanks a hundret times for the patch. W/o it my setup wouldn't work.

@andy123: something completely different. I rember you also have a Gigabyte board with funny Marvell 88SE91xx controller(s). See https://bugzilla.kernel.org/show_bug.cgi?id=42679#c30 for a patch and check if it helps. Andrew needs some more AMD testers. With that patch my controllers are fully usable on the host itself, and passthrough also works fine (though I meanwhile use a ZFS ZVOL for the VM rootfs).

Hey teekay! Thanks for the link, I have an ASRock board, but this patch works for it after adding the PCI-ID of my controller. Once I've done more testing I'll report back to the upstream report.


i'm sorry for my poor english wirting skills…

Offline

#1120 2014-02-07 14:29:54

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,
thank you for your time spent to create a working configuration.

I'm following the thread for more than 6 month and I hope, like you, to have a stable passthrough.

On my hardware Intel DQ67SW, i5 2400 I have succeeded to passthrough a radeon 7790 recently.
I also have a LSI 1068E HBA which I want to assign it to a FreeNAS VM.
I tried to load the HBA ROM from file and the card still will not be initialized in the VM.

In ESXi is working ok, but only when the device reset method is D3D0.
Can this be done using KVM ?

In ESXi possible values for the reset method include flr, d3d0, link, bridge, or default.
http://www.vmware.com/files/pdf/techpap … h_host.pdf.

The default setting is described as follows. If a device supports function level reset (FLR), ESXi always uses FLR. If the device does not support FLR, ESXi next defaults to link reset and bus reset in that order. Link reset and bus reset might prevent some devices from being assigned to different virtual machines, or from being assigned between the VMkernel and virtual machines. In the absence of FLR, it is possible to use PCI Power Management capability (D3 to D0 transitions) to trigger a reset. Most of the Intel NICs and various other HBAs support this mode.

I'm thinking to change my motherboard with an Asrock Z77 Professional Fatal1ty.
Does anyone had any luck with this board ?

Thanks.

Offline

#1121 2014-02-07 19:39:44

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tom_B wrote:

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise

noobman reported that GTX 770 works with secondary passthrough
https://bbs.archlinux.org/viewtopic.php … 4#p1364674

anickname wrote:

I'm thinking to change my motherboard with an Asrock Z77 Professional Fatal1ty.
Does anyone had any luck with this board ?

cmdr managed to passthrough a Nvidia GTX 470 on this motherboard
https://bbs.archlinux.org/viewtopic.php … 2#p1348932

Also you might want to take a look at
https://docs.google.com/spreadsheet/ccc … _web#gid=0

Offline

#1122 2014-02-09 09:56:21

ilya80
Member
Registered: 2013-11-12
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some time I wanted to give it another try, but still no luck. I`m running Core i5 3470 on Z68 chipset, Debian x64 with compiled 3.12.0 kernel, built qemu & seabios myself with few patches from original post applied ( patches seem to work as intended ). Trying to run GTX680 hardmodded into GRID K2 as a secondary adapter to an emulated vga, still getting Error code 43.

Only new thing i see in kern. log is :

Feb 9 13:45:45 cave-lin kernel: [19833.025719] vfio-pci 0000:01:00.0: irq 57 for MSI/MSI-X

Starting VM like this:

$ sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,addr=07.1 -drive file=/opt/kalujny/thrash/windows.img,if=none,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -usbdevice tablet

My next steps would be building a fresh kernel / qemu and probably give up afterwards. sad

Last edited by ilya80 (2014-02-09 10:00:37)

Offline

#1123 2014-02-09 12:58:29

Kaidax
Member
Registered: 2014-02-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any way to increase KVM performance? Right now I experience atrocious framerates in Borderlands 2 under KVM, it runs at most at 55 fps and drops badly to 25. Dropping resolution to 640x480 and disabling every graphics option doesn't affect FPS at all. But under Xen I easily get 200 fps with the same Windows image! Pinning vcpus and hugepages increases FPS by 1-2, but nowhere near playable. Benchmarks indicate no problems, Cinebench CPU score is 766, 3DMark 11 Physics 9300, WEI 7.8.

I launch mainly with this script on Arch qemu-git, linux 3.13 with memleak and arbiter patches, but I've also tried secondary passthrough with pci-assign and vfio both on Arch and on stock Ubuntu 13.04 libvirt to no avail.

#!/bin/bash # i440fx QEMU_AUDIO_DRV=alsa sudo qemu-system-x86_64 -enable-kvm \ -m 3072 -mem-prealloc -mem-path /dev/hugepages \ -cpu host -smp 8,sockets=1,cores=4,threads=8 \ -bios /usr/share/qemu/bios.bin -vga none \ -device vfio-pci,host=01:00.0,bus=pci.0,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pci.0 \ -drive file=/run/media/abc/7582fbde-8935-4dd7-a6e4-42440830672e/home/nya/w-1.img,id=hd,format=raw,index=0 \ -drive file=/dev/disk/by-label/windowsgames,id=hd2,format=raw,index=1 \ -device ich9-intel-hda,bus=pci.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -netdev bridge,br=virbr0,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f7:26:8a,bus=pci.0 \ -usb -usbdevice host:093a:2510 \ "$@"

My specs are:
Intel Core i7-4771
Asrock Z87 Extreme4
8GB RAM
AMD Radeon HD 7870
guest OS - Windows 7 x64

Can anyone post their framerates for comparison? Borderlands 2 is free on steam for today.

Last edited by Kaidax (2014-02-09 13:12:36)

Offline

#1124 2014-02-09 13:51:46

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kaidax wrote:

-cpu host -smp 8,sockets=1,cores=4,threads=8

That might be your problem. You should use -cpu host -smp 8,sockets=1,cores=4,threads=2 for your CPU.
Maybe using i440fx is slower as well? Why don't you use -machine type=q35,accel=kvm?


Kaidax wrote:

-drive file=/run/media/abc/7582fbde-8935-4dd7-a6e4-42440830672e/home/nya/w-1.img,id=hd,format=raw,index=0 \
-drive file=/dev/disk/by-label/windowsgames,id=hd2,format=raw,index=1 \
-netdev bridge,br=virbr0,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f7:26:8a,bus=pci.0 \

Any reason you don't use virtual device drivers? e.g. like so:

-netdev tap,id=net.0,vhost=on,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net.0 \ -drive id=disk1,file=/dev/sdg,if=none,index=0,media=disk -device virtio-blk-pci,drive=disk1,bootindex=1

It should not boost fps that much, but it should generally be a bit snappier.

Kaidax wrote:

Can anyone post their framerates for comparison? Borderlands 2 is free on steam for today.

I didn't measure FPS in detail, but steam in-house streaming reported 20fps at 1080p and highest details on my system.

Offline

#1125 2014-02-09 14:35:25

Kaidax
Member
Registered: 2014-02-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

That might be your problem. You should use -cpu host -smp 8,sockets=1,cores=4,threads=2 for your CPU.

No difference. I've already tried passing arbitrary number of cores, 2, 4, 3, 6, 8. No real difference.

Flyser wrote:

Maybe using i440fx is slower as well? Why don't you use -machine type=q35,accel=kvm?

q35 memory access is generally slower than i440fx. I've tried q35, no difference in FPS, except q35 doesn't even load textures properly, so they stay blurred. Graphics performance is the same between KVM and Xen, unigine heaven scores are perfectly identical, but the game still lags on KVM.

Flyser wrote:

Any reason you don't use virtual device drivers? e.g. like so:

-netdev tap,id=net.0,vhost=on,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net.0 \ -drive id=disk1,file=/dev/sdg,if=none,index=0,media=disk -device virtio-blk-pci,drive=disk1,bootindex=1

It should not boost fps that much, but it should generally be a bit snappier.

I've used virtio drivers on Ubuntu's virt-manager. No difference. sad

Flyser wrote:

I didn't measure FPS in detail, but steam in-house streaming reported 20fps at 1080p and highest details on my system.

20 fps? Ouch, that's bad. gtx 660 is capable of much-much more.

Last edited by Kaidax (2014-02-09 15:34:01)

Offline

#1126 2014-02-09 15:12:55

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kaidax wrote:

20 fps? Ouch, that's bad. gtx 660 is capable of much-much more.

Could be related to the streaming... I don't have the time to test in detail today, but I can test a different game during the week.
While I did notice a drop in performance between native and kvm in Assassins Creed 4, it wasn't huge ...

Offline

#1127 2014-02-09 15:27:19

Kaidax
Member
Registered: 2014-02-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
Kaidax wrote:

20 fps? Ouch, that's bad. gtx 660 is capable of much-much more.

Could be related to the streaming...

Not if you have >20 fps for any other game at the same resolution... Did you try streaming at 640x480 all graphics options turned off?

Flyser wrote:

I don't have the time to test in detail today, but I can test a different game during the week.
While I did notice a drop in performance between native and kvm in Assassins Creed 4, it wasn't huge ...

Well, I've noticed that I can't get more than 62 fps in Arkham City no matter what... (gameplay, not benchmark mode) Also, Crysis is jerkier and drops frames more than on native...
But Borderlands 2 is the only game I've encountered that COMPLETELY DIES on kvm, so it makes for a much better test.

Offline

#1128 2014-02-10 01:29:38

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've switched back to using libvirt, since EFI+i440fx works reliably with several devices assigned.

However, after a suspend/resume cycle (needed because the bus reset doesn't work),
the VM suddenly complains about one of the devices not being assignable!

I see this in the libvirt log:

error : virPCIDeviceIsAssignable:2260 : internal error: Device 0000:08:00.0 is behind a switch lacking ACS and cannot be assigned

08:00.0 sits behind a PLX PEX8606, which absolutely DOES have ACS.
It seems libvirt adds a second layer of checking, which somehow gets broken by suspend/resume.

Edit: "fixed" by editing /etc/libvirt/qemu.conf to set relaxed_acs_check = 1

EDIT again: What do I need to do to get PCI bus reset to work?  Right now, I still have to suspend and resume the host to get the card reset when it's in a bad state.
At least host suspend/resume finally work, since I removed the Asus sound card.

Last edited by DanaGoyette (2014-02-11 04:16:12)

Offline

#1129 2014-02-11 21:47:32

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, does any developer in QEMU notice about poor mem performance in Q35 setting?

Last edited by doubledr (2014-02-11 21:48:21)

Offline

#1130 2014-02-11 22:08:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

doubledr wrote:

Just curious, does any developer in QEMU notice about poor mem performance in Q35 setting?

I'd suggest finding a test case that shows the problem, is reproducible, and preferably does not depend on assigned devices and does use a Linux guest (or simple tool like memtest86).  Report it to kvm & qemu-devel mailing lists.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1131 2014-02-12 01:13:30

Borg8401
Member
Registered: 2014-02-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First off, I want to say thanks to nbhs and anyone else who has posted useful information or have helped others on this thread.  I've finally got passthrough working on my system and wanted to share my results and see if anyone has some suggestions as to how I could possibly improve my experience.  I'm currently running Arch Linux 2014.02.01, I haven't downloaded the patched kernel and qemu and seabios were installed from pacman.  I'm willing to give them all a shot but as my current setup is working fairly well I thought I'd ask for some input before proceeding.

Hardware Asus Rampage IV Extreme Intel i7 3930K @ 4.0GHz (4 cores, 8 threads given to guest) 16GB DDR3 2133MHz Corsair Vengeance (12288MB given to guest) AMD 4890 - Sapphire (Host - using open source drivers) Nvidia GTX 780 - Asus OC DirectCU II (Guest) Kingston HyperX 3K 120GB (Host, 16GB swap, 80GB raw image for guest) Seagate Barracuda 3TB (Given to guest as slave - games held here)

My starting script is (loosely) based on the one used by Kaidax as it improved my WEI score from 5.9 to 7.6 (based on hard disk performance, other values did not change) as compared to OP's first post.  I added vfi-bind into my starting script as it needs to be ran every time I reboot my host for some reason and this was a simple solution to not forget.   Running "export SDL_VIDEO_X11_DGAMOUSE=0" helped some weird lag/loss of input/stuttering that I'm experiencing with my mouse, however it has not completely solved the issue and I also suffer from something similar at times when using my keyboard; I added it into my script for no good reason really.  On "-machine type=pc,accel=kvm" I still have no idea if there is a difference either way when using this option, so it can stay until I discover otherwise.

/sbin/Win7 vfio-bind 0000:03:00.0 0000:03:00.1 export SDL_VIDEO_X11_DGAMOUSE=0 qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm \ -m 12288 -mem-prealloc \ -cpu host -smp 8,sockets=1,cores=4,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on \ -device vfio-pci,host=03:00.1,bus=pci.0 \ -drive file=/home/borg8401/windows.img,id=hd,format=raw,index=0 \ -drive file=/dev/sdb,id=hd2,format=raw,index=1 \ -usb -usbdevice host:1038:1200 -usbdevice host:06a3:0ccb

Results:
yBdxtB3.png

I've uploaded a video that is comprised of one three minute clips each of War Thunder, Borderlands 2, and Battlefield 4 to demonstrate my current performance.  After uploading the video I realized my FPS was only being monitored during War Thunder, but you should still be able to get a general feel for the performance that I'm getting (I will upload a fixed video and edit this post).  http://www.youtube.com/watch?v=oJB1Om6qJsg

War Thunder plays at a (nearly) constant framerate of 60 FPS with vsync on and apart from occasional twitches is very playable.  Borderlands 2 could be considered playable (by some) but is very choppy and the framerate is painfully inconsistent, definitely no where close to native performance.  Battlefield 4 was easily the worst of the three and the lighting effects seemed to destroy any hopes of it being an enjoyable experience; also Punkbuster kicks me when trying to join an online game, stating that I have no ping.

Any suggestions moving forward would be greatly appreciated, if necessary I could go ahead and attempt installing the kernel, qemu and seabios from OP's post; I just don't want to break what I currently have working in the process as I'm still fairly confused on how to go about building things in arch.

Last edited by Borg8401 (2014-02-12 04:48:38)

Offline

#1132 2014-02-12 05:09:35

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
doubledr wrote:

Just curious, does any developer in QEMU notice about poor mem performance in Q35 setting?

I'd suggest finding a test case that shows the problem, is reproducible, and preferably does not depend on assigned devices and does use a Linux guest (or simple tool like memtest86).  Report it to kvm & qemu-devel mailing lists.

It is not that easy, all mem benchmark software crash... I can't find a good benchmark software.

Offline

#1133 2014-02-12 05:19:20

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borg8401 wrote:

First off, I want to say thanks to nbhs and anyone else who has posted useful information or have helped others on this thread.  I've finally got passthrough working on my system and wanted to share my results and see if anyone has some suggestions as to how I could possibly improve my experience.  I'm currently running Arch Linux 2014.02.01, I haven't downloaded the patched kernel and qemu and seabios were installed from pacman.  I'm willing to give them all a shot but as my current setup is working fairly well I thought I'd ask for some input before proceeding.

Hardware Asus Rampage IV Extreme Intel i7 3930K @ 4.0GHz (4 cores, 8 threads given to guest) 16GB DDR3 2133MHz Corsair Vengeance (12288MB given to guest) AMD 4890 - Sapphire (Host - using open source drivers) Nvidia GTX 780 - Asus OC DirectCU II (Guest) Kingston HyperX 3K 120GB (Host, 16GB swap, 80GB raw image for guest) Seagate Barracuda 3TB (Given to guest as slave - games held here)

My starting script is (loosely) based on the one used by Kaidax as it improved my WEI score from 5.9 to 7.6 (based on hard disk performance, other values did not change) as compared to OP's first post.  I added vfi-bind into my starting script as it needs to be ran every time I reboot my host for some reason and this was a simple solution to not forget.   Running "export SDL_VIDEO_X11_DGAMOUSE=0" helped some weird lag/loss of input/stuttering that I'm experiencing with my mouse, however it has not completely solved the issue and I also suffer from something similar at times when using my keyboard; I added it into my script for no good reason really.  On "-machine type=pc,accel=kvm" I still have no idea if there is a difference either way when using this option, so it can stay until I discover otherwise.

/sbin/Win7 vfio-bind 0000:03:00.0 0000:03:00.1 export SDL_VIDEO_X11_DGAMOUSE=0 qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm \ -m 12288 -mem-prealloc \ -cpu host -smp 8,sockets=1,cores=4,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on \ -device vfio-pci,host=03:00.1,bus=pci.0 \ -drive file=/home/borg8401/windows.img,id=hd,format=raw,index=0 \ -drive file=/dev/sdb,id=hd2,format=raw,index=1 \ -usb -usbdevice host:1038:1200 -usbdevice host:06a3:0ccb

Results:
http://i.imgur.com/yBdxtB3.png

I've uploaded a video that is comprised of one three minute clips each of War Thunder, Borderlands 2, and Battlefield 4 to demonstrate my current performance.  After uploading the video I realized my FPS was only being monitored during War Thunder, but you should still be able to get a general feel for the performance that I'm getting (I will upload a fixed video and edit this post).  http://www.youtube.com/watch?v=oJB1Om6qJsg

War Thunder plays at a (nearly) constant framerate of 60 FPS with vsync on and apart from occasional twitches is very playable.  Borderlands 2 could be considered playable (by some) but is very choppy and the framerate is painfully inconsistent, definitely no where close to native performance.  Battlefield 4 was easily the worst of the three and the lighting effects seemed to destroy any hopes of it being an enjoyable experience; also Punkbuster kicks me when trying to join an online game, stating that I have no ping.

Any suggestions moving forward would be greatly appreciated, if necessary I could go ahead and attempt installing the kernel, qemu and seabios from OP's post; I just don't want to break what I currently have working in the process as I'm still fairly confused on how to go about building things in arch.

lighting is a serious problem in MS DX11 implementation. The same story happens in Dolphin WII emulator. When I am facing lighting in WII game, my fps in WII will drop from 60 to 20. It turns out that this is a CPU bounded problem. In Dolphin, I have to turn "Skip EFB access from CPU" on to avoid this problem. In general case, I guess pinning CPU to VM will help this situation.

PS: thx for your post, now I know that pcie will work with 440fx, let me try it in my box.

Offline

#1134 2014-02-12 05:40:37

Borg8401
Member
Registered: 2014-02-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

doubledr wrote:

lighting is a serious problem in MS DX11 implementation. The same story happens in Dolphin WII emulator. When I am facing lighting in WII game, my fps in WII will drop from 60 to 20. It turns out that this is a CPU bounded problem. In Dolphin, I have to turn "Skip EFB access from CPU" on to avoid this problem. In general case, I guess pinning CPU to VM will help this situation.

PS: thx for your post, now I know that pcie will work with 440fx, let me try it in my box.


Attempted googling pinning of the CPU to qemu, not finding much on the subject (just stuff regarding NUMA and libvirt, which I don't use).  Could anyone point me in the right direction?

Offline

#1135 2014-02-12 05:51:22

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borg8401 wrote:
doubledr wrote:

lighting is a serious problem in MS DX11 implementation. The same story happens in Dolphin WII emulator. When I am facing lighting in WII game, my fps in WII will drop from 60 to 20. It turns out that this is a CPU bounded problem. In Dolphin, I have to turn "Skip EFB access from CPU" on to avoid this problem. In general case, I guess pinning CPU to VM will help this situation.

PS: thx for your post, now I know that pcie will work with 440fx, let me try it in my box.


Attempted googling pinning of the CPU to qemu, not finding much on the subject (just stuff regarding NUMA and libvirt, which I don't use).  Could anyone point me in the right direction?

It is in the top post, "performance improvement" section

Offline

#1136 2014-02-12 05:52:55

Borg8401
Member
Registered: 2014-02-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

doubledr wrote:

It is in the top post, "performance improvement" section

Holy crap, I even remember staring at that earlier, thanks! lol

Offline

#1137 2014-02-12 05:55:57

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How can I get the VFIO_DEVICE_PCI_BUS_RESET changes into 3.13 and current qemu?  I tried to build the vfio-vga-reset branch of qemu on Ubuntu 13.10, and got:

hw/usb/host-libusb.c:243:5: error: ‘libusb_get_port_path’ is deprecated (declared at /usr/include/libusb-1.0/libusb.h:1358): Use libusb_get_port_numbers instead [-Werror=deprecated-declarations]

My host has been suspending and resuming reasonably reliably since removing the Asus sound card, so long as the guest is not running.  Still, it would be better to have the reset working.

I hope the bus reset won't require having x-vga=on. 
Judging by the lack of glitches, EFI mode in Windows 8.1 doesn't seem to need the VGA registers.

Offline

#1138 2014-02-12 05:56:32

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

UPDATE: I can confirm that vfio will work with 440FX. Yes, my VIA VL80X USB 3 adpater is back! All you have to do is to  remove "-M Q35" and change "pcie.0" to "pci.0" in the script.

Offline

#1139 2014-02-12 07:01:24

Borg8401
Member
Registered: 2014-02-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

Also, is there a way to deny access to the physical and virtual cores I select above to my host and limit it only to the remaining cores/threads?

EDIT:
I have tried all of the following combinations with nearly identical levels of success/performance when testing with BF4:

0-3 and 6-9 -cpu host -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu host -smp 8,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

2-5 and 8-11 -cpu host -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu host -smp 8,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

0-3 -cpu host -smp 4,sockets=1,cores=4,threads=1
0-3 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

2-5 -cpu host -smp 4,sockets=1,cores=4,threads=1
2-5 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

0-3 -smp 4,sockets=1,cores=4,threads=1

2-5 -smp 4,sockets=1,cores=4,threads=1

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?  Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?  I could see that being a definite bottleneck that may not be noticeable in less demanding games, for instance upon testing Star Wars the Old Republic I had the same performance as if I were on a native Windows install.  Something strange I have noticed however is graphics settings don't have an impact on performance either, I get the same FPS in BF4 on ultra or on low.

Last edited by Borg8401 (2014-02-12 09:49:01)

Offline

#1140 2014-02-12 10:19:55

Kaidax
Member
Registered: 2014-02-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borg8401 wrote:

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?

It's the first way. You can pin cores at runtime in virt-manager and verify that benchmarks won't like it one bit if you assign 0,6,1,7 to a 4-core VM, while 0,1,8,9 is fine. Pinning cores is a last touch to ensure performance, it won't make a huge difference; the scheduler is not as dumb as to assign two saturated vcores to a single core if it can help it, and the gains of reduced context-switching won't manifest in gaming.

Borg8401 wrote:

Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?

That would've impacted your score in benchmarks like 3DMark. I get the same score as on Windows.

Offline

#1141 2014-02-12 11:04:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borg8401 wrote:

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

Also, is there a way to deny access to the physical and virtual cores I select above to my host and limit it only to the remaining cores/threads?

EDIT:
I have tried all of the following combinations with nearly identical levels of success/performance when testing with BF4:

0-3 and 6-9 -cpu host -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu host -smp 8,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

2-5 and 8-11 -cpu host -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu host -smp 8,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

0-3 -cpu host -smp 4,sockets=1,cores=4,threads=1
0-3 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

2-5 -cpu host -smp 4,sockets=1,cores=4,threads=1
2-5 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

0-3 -smp 4,sockets=1,cores=4,threads=1

2-5 -smp 4,sockets=1,cores=4,threads=1

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?  Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?  I could see that being a definite bottleneck that may not be noticeable in less demanding games, for instance upon testing Star Wars the Old Republic I had the same performance as if I were on a native Windows install.  Something strange I have noticed however is graphics settings don't have an impact on performance either, I get the same FPS in BF4 on ultra or on low.

you could try playing arround with kvm-intel options

modinfo kvm-intel

i found out disabling npt on my amd system (modprobe kvm-amd npt=0) made a huge difference in performance, as for vcpu pinning im using cset and then using taskset to pin each vcpu to a cpu core, so on my 8 core amd system, arch gets cores 0-1 and windows 2-7

You can find my cset package here: http://www.fileswap.com/dl/yzZMZTTREg/

Last edited by nbhs (2014-02-12 11:09:27)

Offline

#1142 2014-02-12 12:48:57

Kaidax
Member
Registered: 2014-02-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

you could try playing arround with kvm-intel options

modinfo kvm-intel

i found out disabling npt on my amd system (modprobe kvm-amd npt=0) made a huge difference in performance, as for vcpu pinning im using cset and then using taskset to pin each vcpu to a cpu core, so on my 8 core amd system, arch gets cores 0-1 and windows 2-7

You can find my cset package here: http://www.fileswap.com/dl/yzZMZTTREg/

Passthrough doesn't work at all with

 modprobe kvm_intel ept=0 

and gpu-less benchmarks seem to be unaffected.

Offline

#1143 2014-02-12 17:14:55

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have some scripts for systemd or libvirt to start / poweroff the virtual machines ?
For libvirt, maybe some minimal templates (xml) ?

Last edited by anickname (2014-02-12 17:15:52)

Offline

#1144 2014-02-13 03:41:57

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borg8401 wrote:

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

Also, is there a way to deny access to the physical and virtual cores I select above to my host and limit it only to the remaining cores/threads?

EDIT:
I have tried all of the following combinations with nearly identical levels of success/performance when testing with BF4:

0-3 and 6-9 -cpu host -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu host -smp 8,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

2-5 and 8-11 -cpu host -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu host -smp 8,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

0-3 -cpu host -smp 4,sockets=1,cores=4,threads=1
0-3 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

2-5 -cpu host -smp 4,sockets=1,cores=4,threads=1
2-5 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

0-3 -smp 4,sockets=1,cores=4,threads=1

2-5 -smp 4,sockets=1,cores=4,threads=1

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?  Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?  I could see that being a definite bottleneck that may not be noticeable in less demanding games, for instance upon testing Star Wars the Old Republic I had the same performance as if I were on a native Windows install.  Something strange I have noticed however is graphics settings don't have an impact on performance either, I get the same FPS in BF4 on ultra or on low.

Interesting, adjusting graphic setting should disable some CPU bounded visual effect code paths. Unless this issue is in the basic visual effects. Have you tried to limit the number of VCPU? It might be a SMP issue.

Offline

#1145 2014-02-13 16:11:16

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just an addendum to anyone trying to run the amd binary drivers in the host:  there is clearly an IOMMU/vt-d conflict with the fglrx driver, at least in the ASUS RIVE.  Enable vt-d in bios: black screen in X with 99% CPU and no terminal response. Disable vt-d with no other change:  normal 3-D acceleration and function.  I've confirmed this with kernels 3.10.7, 3.12.1 and 3.13.0/2 with the last three beta amd drivers.  Since the radeon drivers don't support 3-D very well on the 7900 yet, I'm going to try switching the cards so that the nvidia is host and the amd is guest.

Offline

#1146 2014-02-13 16:13:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Just an addendum to anyone trying to run the amd binary drivers in the host:  there is clearly an IOMMU/vt-d conflict with the fglrx driver, at least in the ASUS RIVE.  Enable vt-d in bios: black screen in X with 99% CPU and no terminal response. Disable vt-d with no other change:  normal 3-D acceleration and function.  I've confirmed this with kernels 3.10.7, 3.12.1 and 3.13.0/2 with the last three beta amd drivers.  Since the radeon drivers don't support 3-D very well on the 7900 yet, I'm going to try switching the cards so that the nvidia is host and the amd is guest.

iommu=pt to put host devices in passthrough mode may also be an option.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1147 2014-02-13 20:22:56

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, IOMMU=pt or in fact no IOMMU on the kernel line makes no difference.  Just having vt-d activated in BIOS is sufficient to bork fglrx.

Offline

#1148 2014-02-13 20:49:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

No, IOMMU=pt or in fact no IOMMU on the kernel line makes no difference.  Just having vt-d activated in BIOS is sufficient to bork fglrx.

Perhaps interrupt remapping support is broken, try nointremap.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1149 2014-02-13 21:24:16

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've still got much of this thread to read through, but before I burn too many cycles trying various things to see if I can get this to work the way I want I have two basic questions that I would like to knock out quickly....

1) Do I need multiple video cards or can this be done with just one?  Ideally I would like to run the host as just a headless host and deal with it through ssh, so I would prefer not to have another card in there.   It looks life vfio-pci would require multiple cards, but pci-assign might?

2) Any source I should look at to set this up from a fresh install?  Reapplying the approach in the first post should work, but I'll take any other compiled info as well if someone wrote something up.

(I will update this post if I find the answers while reading the thread, but putting them out there anyhow)

Offline

#1150 2014-02-14 02:40:23

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm occasionally getting a guest video driver hang, accompanied by an oops on the host.
This also invariably results in a freeze on host shutdown.

[165349.140201] kernel tried to execute NX-protected page - exploit attempt? (uid: 0) [165349.144001] BUG: unable to handle kernel paging request at ffffea0005587340 [165349.147069] IP: [<ffffea0005587340>] 0xffffea0005587340 [165349.149381] PGD 47e5f3067 PUD 47e5f2067 PMD 80000004718001e3 [165349.152172] Oops: 0011 [#1] SMP [165349.153542] Modules linked in: usb_storage(F) xt_multiport(F) xt_conntrack(F) ipt_REJECT(F) sb_edac(F) dm_crypt(F) ip6table_filter(F) ip6_tables(F) xt_CHECKSUM(F) iptable_mangle(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) iptable_filter(F) ip_tables(F) x_tables(F) autofs4(F) deflate(F) ctr(F) twofish_generic(F) twofish_avx_x86_64(F) twofish_x86_64_3way(F) twofish_x86_64(F) twofish_common(F) camellia_generic(F) camellia_aesni_avx2(F) camellia_aesni_avx_x86_64(F) camellia_x86_64(F) serpent_avx2(F) serpent_avx_x86_64(F) serpent_sse2_x86_64(F) xts(F) serpent_generic(F) blowfish_generic(F) blowfish_x86_64(F) blowfish_common(F) cast5_avx_x86_64(F) cast5_generic(F) cast_common(F) des_generic(F) cmac(F) xcbc(F) rmd160(F) crypto_null(F) af_key(F) xfrm_algo(F) parport_pc(F) ppdev(F) bnep(F) rfcomm(F) bluetooth(F) joydev(F) iTCO_wdt(F) iTCO_vendor_support(F) dm_multipath(F) scsi_dh(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) kvm_intel(F) snd_usb_audio(F) snd_hda_codec_hdmi(F) kvm(F) snd_usbmidi_lib(F) crct10dif_pclmul(F) hid_logitech_dj(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) snd_hda_codec_realtek(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) snd_hda_intel(F) i915(F) snd_hda_codec(F) snd_seq_midi(F) shpchp(F) snd_seq_midi_event(F) snd_hwdep(F) snd_rawmidi(F) snd_pcm(F) snd_seq(F) drm_kms_helper(F) snd_page_alloc(F) snd_seq_device(F) snd_timer(F) drm(F) lpc_ich(F) snd(F) i2c_algo_bit(F) soundcore(F) video(F) mac_hid(F) vfio_pci(F) vfio_iommu_type1(F) vfio(F) w83627ehf(F) hwmon_vid(F) coretemp(F) msr(F) cpuid(F) jc42(F) nfsd(F) binfmt_misc(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lp(F) parport(F) lockd(F) sunrpc(F) fscache(F) nls_iso8859_1(F) zfs(POF) zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) btrfs(F) xor(F) hid_generic(F) raid6_pq(F) libcrc32c(F) usbhid(F) hid(F) firewire_ohci(F) firewire_core(F) crc_itu_t(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F) microcode(F) [165349.234609] CPU: 4 PID: 2092 Comm: libvirtd Tainted: PF W O 3.13.0-acs #1 [165349.237988] Hardware name: Supermicro X10SAT/X10SAT, BIOS 1.0 08/27/2013 [165349.240918] task: ffff880455471800 ti: ffff880452446000 task.ti: ffff880452446000 [165349.245214] RIP: 0010:[<ffffea0005587340>] [<ffffea0005587340>] 0xffffea0005587340 [165349.248924] RSP: 0018:ffff880452447d98 EFLAGS: 00010246 [165349.251166] RAX: 0000000000000088 RBX: ffff880467e63098 RCX: 0000000000000000 [165349.253973] RDX: ffff880067ad5748 RSI: 0000000000560054 RDI: ffff880467e63098 [165349.257448] RBP: ffff880452447dc0 R08: 0000000000000246 R09: 0000000000000000 [165349.261025] R10: ffffffff8122f679 R11: ffffea0011496c00 R12: ffff880467e63140 [165349.263848] R13: ffffea0005587340 R14: 0000000000000000 R15: ffffffffffffffed [165349.267308] FS: 00007f8dcf8de840(0000) GS:ffff88047eb00000(0000) knlGS:0000000000000000 [165349.271130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [165349.273672] CR2: ffffea0005587340 CR3: 0000000452763000 CR4: 00000000001407e0 [165349.277059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [165349.279770] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [165349.283501] Stack: [165349.284079] ffffffff814aabf6 ffffffff817342cc 0000000000000004 ffff880467e63098 [165349.287573] 0000000000000004 ffff880452447df0 ffffffff814ab68d 0000000000000004 [165349.291175] ffff880467e63098 ffff880467e63140 0000000000000246 ffff880452447e20 [165349.294208] Call Trace: [165349.295446] [<ffffffff814aabf6>] ? __rpm_callback+0x36/0xc0 [165349.298879] [<ffffffff817342cc>] ? notifier_call_chain+0x4c/0x70 [165349.304086] [<ffffffff814ab68d>] rpm_idle+0x1bd/0x2b0 [165349.307279] [<ffffffff814ab7dd>] __pm_runtime_idle+0x5d/0x80 [165349.310657] [<ffffffff8149f085>] __device_release_driver+0x65/0xf0 [165349.314992] [<ffffffff8149f133>] device_release_driver+0x23/0x30 [165349.318650] [<ffffffff8149dead>] unbind_store+0xbd/0xe0 [165349.321664] [<ffffffff8149d2d4>] drv_attr_store+0x24/0x40 [165349.324955] [<ffffffff8122e878>] sysfs_write_file+0x128/0x1c0 [165349.329196] [<ffffffff811b7934>] vfs_write+0xb4/0x1f0 [165349.332390] [<ffffffff811b8369>] SyS_write+0x49/0xa0 [165349.335567] [<ffffffff817389bf>] tracesys+0xe1/0xe6 [165349.338873] Code: 00 00 00 ff ff ff ff 01 00 00 00 00 01 10 00 00 00 ad de 00 02 20 00 00 00 ad de 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 ff ff 02 00 00 00 00 00 00 00 00 00 00 00 00 00 [165349.349115] RIP [<ffffea0005587340>] 0xffffea0005587340 [165349.352128] RSP <ffff880452447d98> [165349.354904] CR2: ffffea0005587340 [165349.363300] ---[ end trace d2abebf797ae841e ]---

Offline

#1151 2014-02-14 09:11:55

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

I'm occasionally getting a guest video driver hang, accompanied by an oops on the host.
This also invariably results in a freeze on host shutdown.

While maybe unrelated to your error I am also getting something similar on my setup. It works very well most of the time, but occasionally i can't start up my windows guest (qemu quits in the middle of the guest bootup) and i need to do a full host reboot in order to get it working again. What puzzles me is that it is not a consistent error and that it is

feb. 13 23:15:45 archxen kernel: ------------[ cut here ]------------ feb. 13 23:15:45 archxen kernel: WARNING: CPU: 1 PID: 693 at fs/sysfs/group.c:214 sysfs_remove_group+0xc4/0xc8() feb. 13 23:15:45 archxen kernel: sysfs group ffffffff8188f3a0 not found for kobject 'hidraw0' feb. 13 23:15:45 archxen kernel: Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl ctr ccm hid_generic vfio_pci vfio_iommu_type1 vfio it87 hwmon_vid arc4 kvm_amd kvm mxm_wmi crc32_pclmul crc32 feb. 13 23:15:45 archxen kernel: ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common pci_stub feb. 13 23:15:45 archxen kernel: CPU: 1 PID: 693 Comm: upowerd Tainted: P O 3.13.0-2-mainline #1 feb. 13 23:15:45 archxen kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./990XA-UD3, BIOS FD 02/04/2013 feb. 13 23:15:45 archxen kernel: 0000000000000009 ffff8800ba1f3d28 ffffffff814cef33 ffff8800ba1f3d70 feb. 13 23:15:45 archxen kernel: ffff8800ba1f3d60 ffffffff8105d0dd 0000000000000000 ffffffff8188f3a0 feb. 13 23:15:45 archxen kernel: ffff8802331cd810 ffff88023116b8e8 ffff8802332e4520 ffff8800ba1f3dc0 feb. 13 23:15:45 archxen kernel: Call Trace: feb. 13 23:15:45 archxen kernel: [<ffffffff814cef33>] dump_stack+0x45/0x56 feb. 13 23:15:45 archxen kernel: [<ffffffff8105d0dd>] warn_slowpath_common+0x7d/0xa0 feb. 13 23:15:45 archxen kernel: [<ffffffff8105d14c>] warn_slowpath_fmt+0x4c/0x50 feb. 13 23:15:45 archxen kernel: [<ffffffff814d3cfe>] ? mutex_unlock+0xe/0x10 feb. 13 23:15:45 archxen kernel: [<ffffffff81205a46>] ? sysfs_get_dirent_ns+0x4e/0x70 feb. 13 23:15:45 archxen kernel: [<ffffffff81206bbc>] sysfs_remove_group+0xc4/0xc8 feb. 13 23:15:45 archxen kernel: [<ffffffff81376bb3>] dpm_sysfs_remove+0x43/0x50 feb. 13 23:15:45 archxen kernel: [<ffffffff8136ce55>] device_del+0x45/0x1c0 feb. 13 23:15:45 archxen kernel: [<ffffffff8136cfee>] device_unregister+0x1e/0x60 feb. 13 23:15:45 archxen kernel: [<ffffffff8136d0a4>] device_destroy+0x3c/0x48 feb. 13 23:15:45 archxen kernel: [<ffffffffa02c0ae7>] drop_ref.part.2+0x57/0x70 [hid] feb. 13 23:15:45 archxen kernel: [<ffffffffa02c0f48>] hidraw_release+0xa0/0xb8 [hid] feb. 13 23:15:45 archxen kernel: [<ffffffff811959f4>] __fput+0xa4/0x228 feb. 13 23:15:45 archxen kernel: [<ffffffff81195bbe>] ____fput+0xe/0x10 feb. 13 23:15:45 archxen kernel: [<ffffffff8107aa17>] task_work_run+0x97/0xd0 feb. 13 23:15:45 archxen kernel: [<ffffffff81012d77>] do_notify_resume+0x87/0x90 feb. 13 23:15:45 archxen kernel: [<ffffffff814dda6a>] int_signal+0x12/0x17 feb. 13 23:15:45 archxen kernel: ---[ end trace 01cc04907c3ac967 ]---

Any thoughts?

Offline

#1152 2014-02-14 11:05:41

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

So I've still got much of this thread to read through, but before I burn too many cycles trying various things to see if I can get this to work the way I want I have two basic questions that I would like to knock out quickly....

1) Do I need multiple video cards or can this be done with just one?  Ideally I would like to run the host as just a headless host and deal with it through ssh, so I would prefer not to have another card in there.   It looks life vfio-pci would require multiple cards, but pci-assign might?

2) Any source I should look at to set this up from a fresh install?  Reapplying the approach in the first post should work, but I'll take any other compiled info as well if someone wrote something up.

(I will update this post if I find the answers while reading the thread, but putting them out there anyhow)

I also plan to build such a machine, with essentially headless host and Windows running on selected cores with VGA passthrough. For the time being my plan is to use the newest stable kernel and rely on top post in this thread, since it seems to be kept up-to date (thanks nbhs !)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1153 2014-02-14 13:32:41

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried intremap=off (I see nointremap is deprecated) but there is no change; X still hangs when IOMMU is activated in bios.

Offline

#1154 2014-02-14 14:57:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbirkis wrote:
DanaGoyette wrote:

I'm occasionally getting a guest video driver hang, accompanied by an oops on the host.
This also invariably results in a freeze on host shutdown.

While maybe unrelated to your error I am also getting something similar on my setup. It works very well most of the time, but occasionally i can't start up my windows guest (qemu quits in the middle of the guest bootup) and i need to do a full host reboot in order to get it working again. What puzzles me is that it is not a consistent error and that it is

feb. 13 23:15:45 archxen kernel: ------------[ cut here ]------------ feb. 13 23:15:45 archxen kernel: WARNING: CPU: 1 PID: 693 at fs/sysfs/group.c:214 sysfs_remove_group+0xc4/0xc8() feb. 13 23:15:45 archxen kernel: sysfs group ffffffff8188f3a0 not found for kobject 'hidraw0' feb. 13 23:15:45 archxen kernel: Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl ctr ccm hid_generic vfio_pci vfio_iommu_type1 vfio it87 hwmon_vid arc4 kvm_amd kvm mxm_wmi crc32_pclmul crc32 feb. 13 23:15:45 archxen kernel: ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common pci_stub feb. 13 23:15:45 archxen kernel: CPU: 1 PID: 693 Comm: upowerd Tainted: P O 3.13.0-2-mainline #1 feb. 13 23:15:45 archxen kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./990XA-UD3, BIOS FD 02/04/2013 feb. 13 23:15:45 archxen kernel: 0000000000000009 ffff8800ba1f3d28 ffffffff814cef33 ffff8800ba1f3d70 feb. 13 23:15:45 archxen kernel: ffff8800ba1f3d60 ffffffff8105d0dd 0000000000000000 ffffffff8188f3a0 feb. 13 23:15:45 archxen kernel: ffff8802331cd810 ffff88023116b8e8 ffff8802332e4520 ffff8800ba1f3dc0 feb. 13 23:15:45 archxen kernel: Call Trace: feb. 13 23:15:45 archxen kernel: [<ffffffff814cef33>] dump_stack+0x45/0x56 feb. 13 23:15:45 archxen kernel: [<ffffffff8105d0dd>] warn_slowpath_common+0x7d/0xa0 feb. 13 23:15:45 archxen kernel: [<ffffffff8105d14c>] warn_slowpath_fmt+0x4c/0x50 feb. 13 23:15:45 archxen kernel: [<ffffffff814d3cfe>] ? mutex_unlock+0xe/0x10 feb. 13 23:15:45 archxen kernel: [<ffffffff81205a46>] ? sysfs_get_dirent_ns+0x4e/0x70 feb. 13 23:15:45 archxen kernel: [<ffffffff81206bbc>] sysfs_remove_group+0xc4/0xc8 feb. 13 23:15:45 archxen kernel: [<ffffffff81376bb3>] dpm_sysfs_remove+0x43/0x50 feb. 13 23:15:45 archxen kernel: [<ffffffff8136ce55>] device_del+0x45/0x1c0 feb. 13 23:15:45 archxen kernel: [<ffffffff8136cfee>] device_unregister+0x1e/0x60 feb. 13 23:15:45 archxen kernel: [<ffffffff8136d0a4>] device_destroy+0x3c/0x48 feb. 13 23:15:45 archxen kernel: [<ffffffffa02c0ae7>] drop_ref.part.2+0x57/0x70 [hid] feb. 13 23:15:45 archxen kernel: [<ffffffffa02c0f48>] hidraw_release+0xa0/0xb8 [hid] feb. 13 23:15:45 archxen kernel: [<ffffffff811959f4>] __fput+0xa4/0x228 feb. 13 23:15:45 archxen kernel: [<ffffffff81195bbe>] ____fput+0xe/0x10 feb. 13 23:15:45 archxen kernel: [<ffffffff8107aa17>] task_work_run+0x97/0xd0 feb. 13 23:15:45 archxen kernel: [<ffffffff81012d77>] do_notify_resume+0x87/0x90 feb. 13 23:15:45 archxen kernel: [<ffffffff814dda6a>] int_signal+0x12/0x17 feb. 13 23:15:45 archxen kernel: ---[ end trace 01cc04907c3ac967 ]---

Any thoughts?

Looks usb related


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1155 2014-02-14 21:53:06

an.archy
Member
From: Zilina, Slovakia
Registered: 2014-02-14
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

just started working on this for my rig.

I got the output from my second VGA working. The VGA does not reset correctly from what I can tell but that's not my biggest issue at the moment.

When I start the VM and start to install Win7 my memory usage just keeps on increasing until the system is unusable.

I allocate 4G of memory to the VM. When I start it "top" shows 4G being used by the process but after I start the install the memory used increases until 7.9G and then the system is effectively frozen with 100MB free memory.

I installed the 3 packages from the first post.

My setup:
CPU: i7-4770
MG: GA-B85-HD3
VGA1: IGFX
VGA2: GTX560

Any pointers or help is appreciated.

Thanks!
- Matt


- Matt

Offline

#1156 2014-02-14 21:57:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

an.archy wrote:

Hello,

just started working on this for my rig.

I got the output from my second VGA working. The VGA does not reset correctly from what I can tell but that's not my biggest issue at the moment.

When I start the VM and start to install Win7 my memory usage just keeps on increasing until the system is unusable.

I allocate 4G of memory to the VM. When I start it "top" shows 4G being used by the process but after I start the install the memory used increases until 7.9G and then the system is effectively frozen with 100MB free memory.

I installed the 3 packages from the first post.

My setup:
CPU: i7-4770
MG: GA-B85-HD3
VGA1: IGFX
VGA2: GTX560

Any pointers or help is appreciated.

Thanks!
- Matt

That sounds a lot like RAM is being used for your disk.  Are you by chance using the -snapshot option?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1157 2014-02-15 11:58:19

an.archy
Member
From: Zilina, Slovakia
Registered: 2014-02-14
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It feels like that. I don't have -snapshot in the startup though. I have a SSD disk with lvm and btrfs. Also I boot using UEFI. If that is relevant.

This is how I start the VM:

qemu-system-x86_64 -enable-kvm -M q35 -balloon none \ -m 4G \ -cpu host -smp 8,cores=4,threads=8,sockets=1 \ -bios /usr/share/qemu/bios.bin \ -drive file=/home/win.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -usb -device usb-host,hostbus=3,hostaddr=4 \ -device usb-host,hostbus=1,hostaddr=4 \ -device ide-cd,drive=cdrom1,id=drive1,bus=ide.1 \ -drive file=/root/Win7-x64.iso,if=none,media=cdrom,id=cdrom1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -vga none -nographic

I used qemu-kvm alot before and never had a problem like this so im stumped. Although never on a SSD and I have a feeling it might have something to do with this.

EDIT: Ok. It's not the SSD. I'm just getting desperate grasping at any straw I can find. By the way, qemu wiki says that -snapshot saves everything to temporary files that are deleted after the VM is turned off. I guess my problem is related to the memory leaks. When I turn off the machine the memory is still occupied. Not the 4G allocated but the unidentified part.

uname -a

Linux localhost 3.13.0-1-mainline #1 SMP PREEMPT Fri Feb 14 17:55:12 CET 2014 x86_64 GNU/Linux

Thanks!
- Matt

Last edited by an.archy (2014-02-15 12:27:12)


- Matt

Offline

#1158 2014-02-15 12:42:44

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Personally I'd try upgrading kernel to most recent stable (3.13.3) as well as qemu and libvirt. I'm not implying you have old versions of those, just don't know.

Last edited by Bronek (2014-02-15 12:42:55)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1159 2014-02-15 16:17:06

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I had same issue with memory remaining occupied after close, and the patch "fix_memleak.patch" fixed this specifically and i get all my memory back since then. Are you sure you are using the kernel patches from OP.  Perhaps you are using an unpatched kernel?

Offline

#1160 2014-02-16 11:57:03

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi.
It is the best forum about kvm video passthrough.
I have configured all on Fedora 20 following instructions except getting pakages from git(kernel 3.12 kvm 1.6.1).
I still have the problem with restart guest with HD7750.
Which patch  is responsible for this feature?

Last edited by zzz3000 (2014-02-16 12:28:37)

Offline

#1161 2014-02-16 15:28:16

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I solved the reboot problem by doing like in this article :
http://blog.ktz.me/?p=219

Offline

#1162 2014-02-16 20:11:46

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm still in need of PCI bus reset on the 3.13 kernel.  While you can work around the issue by ejecting the video card in the guest before shutdown, that doesn't help the slightest bit in cases of guest crash.  I instead end up having to suspend and resume the host.
I've tried building the vfio-vga-reset branch of qemu, but I got build errors (mentioned in an earlier post).

As for my 'hidraw' error, that seems to happen when I switch from host to guest with my physical KVM switch. 

Does anyone know how to prevent hda-intel and xhci-hcdi on the host from binding to certain devices? I'd like to prevent the passed-through HDMI Audio and USB 3.0 controllers from ever becoming active on the host, but I can't just blacklist the modules.

EDIT: To be clear, booting with pci-stub.ids doesn't seem to do it.  It might prevent binding once, if I'm lucky, but then on VM stop (via libvirt), the real driver takes over.

Last edited by DanaGoyette (2014-02-16 20:30:58)

Offline

#1163 2014-02-16 20:35:35

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

I'm still in need of PCI bus reset on the 3.13 kernel.  While you can work around the issue by ejecting the video card in the guest before shutdown, that doesn't help the slightest bit in cases of guest crash.  I instead end up having to suspend and resume the host.
I've tried building the vfio-vga-reset branch of qemu, but I got build errors (mentioned in an earlier post).

As for my 'hidraw' error, that seems to happen when I switch from host to guest with my physical KVM switch. 

Does anyone know how to prevent hda-intel and xhci-hcdi on the host from binding to certain devices? I'd like to prevent the passed-through HDMI Audio and USB 3.0 controllers from ever becoming active on the host, but I can't just blacklist the modules.

I don't think the vfio-vga-reset branch is still necesarry and might even be incompatible. I'm using stable, upstream qemu at the moment, as in the arch package of 1.7.0 and seabios 1.7.3.1 and it's working perfectly fine including reset and all.


i'm sorry for my poor english wirting skills…

Offline

#1164 2014-02-16 21:45:59

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zzz3000 wrote:

Hi.
It is the best forum about kvm video passthrough.
I have configured all on Fedora 20 following instructions except getting pakages from git(kernel 3.12 kvm 1.6.1).
I still have the problem with restart guest with HD7750.
Which patch  is responsible for this feature?

You'll need the lastest stable version (1.7) or qemu-git

Offline

#1165 2014-02-16 21:50:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

I'm still in need of PCI bus reset on the 3.13 kernel.  While you can work around the issue by ejecting the video card in the guest before shutdown, that doesn't help the slightest bit in cases of guest crash.  I instead end up having to suspend and resume the host.
I've tried building the vfio-vga-reset branch of qemu, but I got build errors (mentioned in an earlier post).

As for my 'hidraw' error, that seems to happen when I switch from host to guest with my physical KVM switch. 

Does anyone know how to prevent hda-intel and xhci-hcdi on the host from binding to certain devices? I'd like to prevent the passed-through HDMI Audio and USB 3.0 controllers from ever becoming active on the host, but I can't just blacklist the modules.

EDIT: To be clear, booting with pci-stub.ids doesn't seem to do it.  It might prevent binding once, if I'm lucky, but then on VM stop (via libvirt), the real driver takes over.

Im not sure reset will work without x-vga=on

Offline

#1166 2014-02-17 02:13:06

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw:

Your new ACS patch does not work on z87 chipset, but the older work....
If you whant i can post some test.

Last edited by Val532 (2014-02-17 02:22:38)

Offline

#1167 2014-02-17 02:14:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

@aw:

Your new ACS patch does not work on z87 chipset, but the older work....
If you want i can post some test.

What are the device IDs?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1168 2014-02-17 03:15:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Val532 wrote:

@aw:

Your new ACS patch does not work on z87 chipset, but the older work....
If you want i can post some test.

What are the device IDs?

z87 would be a lynx point chipset, so should be covered by the 8c1X set of device IDs.  Note though that the new ACS patch only enables ACS for PCH-based root ports, ie. the ones at 00:1c.X.  Intel has not provided details for whether the same is possible with processor-based root ports.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1169 2014-02-17 04:43:41

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

I don't think the vfio-vga-reset branch is still necesarry and might even be incompatible. I'm using stable, upstream qemu at the moment, as in the arch package of 1.7.0 and seabios 1.7.3.1 and it's working perfectly fine including reset and all.

I checked in vfio.c, and the symbol mentioned in the original thread about the bus reset ("VFIO_DEVICE_PCI_BUS_RESET") doesn't appear anywhere in the kernel or in qemu. 
I see other mentions of device reset, but whatever is there sure isn't working.  I'd imagine it will also be similarly broken for the secondary-passthrough case.

Offline

#1170 2014-02-17 04:46:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
andy123 wrote:

I don't think the vfio-vga-reset branch is still necesarry and might even be incompatible. I'm using stable, upstream qemu at the moment, as in the arch package of 1.7.0 and seabios 1.7.3.1 and it's working perfectly fine including reset and all.

I checked in vfio.c, and the symbol mentioned in the original thread about the bus reset ("VFIO_DEVICE_PCI_BUS_RESET") doesn't appear anywhere in the kernel or in qemu. 
I see other mentions of device reset, but whatever is there sure isn't working.  I'd imagine it will also be similarly broken for the secondary-passthrough case.

Does the patch here help?  http://lists.nongnu.org/archive/html/qe … 00767.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1171 2014-02-17 05:41:34

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DanaGoyette wrote:

I checked in vfio.c, and the symbol mentioned in the original thread about the bus reset ("VFIO_DEVICE_PCI_BUS_RESET") doesn't appear anywhere in the kernel or in qemu. 
I see other mentions of device reset, but whatever is there sure isn't working.  I'd imagine it will also be similarly broken for the secondary-passthrough case.

Does the patch here help?  http://lists.nongnu.org/archive/html/qe … 00767.html

I applied the patch, and don't see any change, at least so far.  I'm going to try adding x-vga=on... as soon as I reboot my host, due to getting this again at guest shutdown:

[11277.738918] BUG: unable to handle kernel paging request at 0000000000001000 [11277.745979] IP: [<0000000000001000>] 0x1000 [11277.750192] PGD 0 [11277.752230] Oops: 0010 [#1] SMP [11277.755514] Modules linked in: xt_multiport(F) xt_conntrack(F) ipt_REJECT(F) dm_crypt(F) sb_edac(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) ip6table_filter(F) ip6_tables(F) xt_CHECKSUM(F) iptable_mangle(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) iptable_filter(F) ip_tables(F) x_tables(F) autofs4(F) deflate(F) ctr(F) twofish_generic(F) twofish_avx_x86_64(F) twofish_x86_64_3way(F) twofish_x86_64(F) twofish_common(F) camellia_generic(F) camellia_aesni_avx2(F) camellia_aesni_avx_x86_64(F) camellia_x86_64(F) serpent_avx2(F) serpent_avx_x86_64(F) serpent_sse2_x86_64(F) xts(F) serpent_generic(F) blowfish_generic(F) blowfish_x86_64(F) blowfish_common(F) cast5_avx_x86_64(F) cast5_generic(F) cast_common(F) des_generic(F) cmac(F) xcbc(F) rmd160(F) crypto_null(F) af_key(F) xfrm_algo(F) parport_pc(F) ppdev(F) bnep(F) rfcomm(F) bluetooth(F) joydev(F) iTCO_wdt(F) iTCO_vendor_support(F) dm_multipath(F) scsi_dh(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) kvm_intel(F) kvm(F) snd_usb_audio(F) snd_usbmidi_lib(F) crct10dif_pclmul(F) crc32_pclmul(F) hid_logitech_dj(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) snd_hda_intel(F) glue_helper(F) ablk_helper(F) snd_hda_codec(F) cryptd(F) snd_seq_midi(F) snd_hwdep(F) snd_seq_midi_event(F) i915(F) shpchp(F) snd_pcm(F) snd_rawmidi(F) snd_seq(F) snd_page_alloc(F) snd_seq_device(F) snd_timer(F) lpc_ich(F) drm_kms_helper(F) snd(F) drm(F) soundcore(F) i2c_algo_bit(F) mac_hid(F) video(F) vfio_pci(F) vfio_iommu_type1(F) vfio(F) w83627ehf(F) hwmon_vid(F) coretemp(F) msr(F) cpuid(F) nfsd(F) auth_rpcgss(F) jc42(F) nfs_acl(F) nfs(F) binfmt_misc(F) lp(F) lockd(F) parport(F) sunrpc(F) fscache(F) nls_iso8859_1(F) zfs(POF) zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) hid_generic(F) btrfs(F) xor(F) usbhid(F) hid(F) raid6_pq(F) libcrc32c(F) firewire_ohci(F) firewire_core(F) crc_itu_t(F) ahci(F) e1000e(F) libahci(F) ptp(F) pps_core(F) microcode(F) [11277.936682] CPU: 0 PID: 2103 Comm: libvirtd Tainted: PF W O 3.13.0-acs #1 [11277.944094] Hardware name: Supermicro X10SAT/X10SAT, BIOS 1.0 08/27/2013 [11277.950815] task: ffff8804596bb000 ti: ffff880452ce8000 task.ti: ffff880452ce8000 [11277.958315] RIP: 0010:[<0000000000001000>] [<0000000000001000>] 0x1000 [11277.964963] RSP: 0018:ffff880452ce9d70 EFLAGS: 00010246 [11277.970288] RAX: 0000000000000008 RBX: ffff880467e63098 RCX: 0000000000000000 [11277.977441] RDX: ffff880415ac7348 RSI: 00000000015c015a RDI: ffff880467e63098 [11277.984593] RBP: ffff880452ce9d98 R08: 0000000000000246 R09: 0000000000000000 [11277.991738] R10: ffffea0007ea5080 R11: ffffffffa04e3127 R12: ffff880467e63140 [11277.998891] R13: 0000000000001000 R14: 0000000000000000 R15: ffffffffffffffed [11278.006042] FS: 00007f0adcf30700(0000) GS:ffff88047ea00000(0000) knlGS:0000000000000000 [11278.014149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11278.019906] CR2: 0000000000001000 CR3: 0000000458490000 CR4: 00000000001407e0 [11278.027052] Stack: [11278.029070] ffffffff814aabf6 ffff880452ce9dd8 0000000000000004 ffff880467e63098 [11278.036561] 0000000000000004 ffff880452ce9dc8 ffffffff814ab68d 0000000000000004 [11278.044046] ffff880467e63098 ffff880467e63140 0000000000000246 ffff880452ce9df8 [11278.051546] Call Trace: [11278.054009] [<ffffffff814aabf6>] ? __rpm_callback+0x36/0xc0 [11278.059679] [<ffffffff814ab68d>] rpm_idle+0x1bd/0x2b0 [11278.064829] [<ffffffff814ab7dd>] __pm_runtime_idle+0x5d/0x80 [11278.070589] [<ffffffff813b95a4>] pci_device_remove+0x74/0xb0 [11278.076353] [<ffffffff8149f09f>] __device_release_driver+0x7f/0xf0 [11278.082647] [<ffffffff8149f133>] device_release_driver+0x23/0x30 [11278.088750] [<ffffffff8149dead>] unbind_store+0xbd/0xe0 [11278.094073] [<ffffffff8149d2d4>] drv_attr_store+0x24/0x40 [11278.099571] [<ffffffff8122e878>] sysfs_write_file+0x128/0x1c0 [11278.105416] [<ffffffff811b7934>] vfs_write+0xb4/0x1f0 [11278.110565] [<ffffffff811b8369>] SyS_write+0x49/0xa0 [11278.115631] [<ffffffff817389bf>] tracesys+0xe1/0xe6 [11278.120603] Code: Bad RIP value. [11278.123993] RIP [<0000000000001000>] 0x1000 [11278.128303] RSP <ffff880452ce9d70> [11278.131804] CR2: 0000000000001000 [11278.142523] ---[ end trace 2f92dc116cd5ac76 ]---

EDIT: Okay, I tried with x-vga=on.  At least with x-vga=on and the above mentioned change, I was able to normal-reboot the guest properly, unlike ever before.  However, when the guest BSOD'd (unrelated to video stuff), the GPU became unresponsive again.  The EFI framebuffer still works while in this state, but Windows won't boot.  Then when I tried to stop the VM, I yet again got the above kernel oops.

Last edited by DanaGoyette (2014-02-17 06:03:19)

Offline

#1172 2014-02-17 07:56:49

sephiroth1395
Member
Registered: 2014-02-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone,

First, I would like to thank everyone for their effort making VGA-Passthrough usable.  As many of us here, I'm having problems.  I'm running Fedora 20 with kernel 3.13.3 and the packages from the virt-preview, which are close to upstream.
My configuration is a Dell OptiPlex 7010, equipped with a Core i5 3470 and a Radeon HD7570 from Dell.

VT-d and vfio work fine for other devices, such as the audio controller, but I can't seem to passthrough the graphic cards.  Primary passthrough simply fails, secondary passthrough with a Cirrus VGA gives me Error 10.  In both cases, I hear the GPU fan reacting when the VM boots, but then nothing happens.

I tried blacklisting completely the i915 driver, as I can live without it.  This is for a project aimed at a seamless interaction with a Windows VM, the Linux hypervisor doing background processing, hidden from the user.  Given my situation, what are my best next shots ?

Last edited by sephiroth1395 (2014-02-17 07:57:45)

Offline

#1173 2014-02-17 16:56:03

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
Val532 wrote:

@aw:

Your new ACS patch does not work on z87 chipset, but the older work....
If you want i can post some test.

What are the device IDs?

z87 would be a lynx point chipset, so should be covered by the 8c1X set of device IDs.  Note though that the new ACS patch only enables ACS for PCH-based root ports, ie. the ones at 00:1c.X.  Intel has not provided details for whether the same is possible with processor-based root ports.


Sorry i'm late,

But yes it's a 0x8C1X but the tree root port of my mother board dans the two VGA card on it is on the same iommu group ...

Offline

#1174 2014-02-17 16:58:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:
aw wrote:
aw wrote:

What are the device IDs?

z87 would be a lynx point chipset, so should be covered by the 8c1X set of device IDs.  Note though that the new ACS patch only enables ACS for PCH-based root ports, ie. the ones at 00:1c.X.  Intel has not provided details for whether the same is possible with processor-based root ports.


Sorry i'm late,

But yes it's a 0x8C1X but the tree root port of my mother board dans the two VGA card on it is on the same iommu group ...

Are they below one of the 0x8C1X root ports or a different one?

EDIT: this would be useful: lspci -tv; lspci -n; find /sys/kernel/iommu_groups/

Last edited by aw (2014-02-17 17:23:33)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1175 2014-02-17 21:35:56

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Val532 wrote:
aw wrote:

z87 would be a lynx point chipset, so should be covered by the 8c1X set of device IDs.  Note though that the new ACS patch only enables ACS for PCH-based root ports, ie. the ones at 00:1c.X.  Intel has not provided details for whether the same is possible with processor-based root ports.


Sorry i'm late,

But yes it's a 0x8C1X but the tree root port of my mother board dans the two VGA card on it is on the same iommu group ...

Are they below one of the 0x8C1X root ports or a different one?

EDIT: this would be useful: lspci -tv; lspci -n; find /sys/kernel/iommu_groups/

So i made some test.

On my AsRock Z87 extrem 6 :
Without patch

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 9 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) ### Group 10 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)

With old ACS patch :

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) ### Group 3 ### 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) ### Group 4 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 5 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 6 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 7 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) ### Group 8 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) ### Group 9 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 10 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 11 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 12 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) ### Group 13 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 14 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) ### Group 15 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) ### Group 16 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 17 ### 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] ### Group 18 ### 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 19 ### 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) ### Group 20 ### 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

Whit the new ACS patch :

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 9 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 10 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) ### Group 11 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 12 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) ### Group 13 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) ### Group 14 ### 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 15 ### 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) ### Group 16 ### 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

On my GigaByte X58 .... :

Without any patch :

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 12) ### Group 1 ### 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 12) ### Group 2 ### 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 12) ### Group 3 ### 00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 12) 00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 12) ### Group 4 ### 00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 12) 00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 12) ### Group 5 ### 00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 12) ### Group 6 ### 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 12) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 12) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 12) ### Group 7 ### 00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 12) ### Group 8 ### 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 ### Group 9 ### 00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller ### Group 10 ### 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 04:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02) 04:00.1 IDE interface: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 ### Group 12 ### 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) ### Group 13 ### 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller ### Group 14 ### 01:00.0 VGA compatible controller: NVIDIA Corporation NV44 [GeForce 7100 GS] (rev a1) ### Group 15 ### 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] ### Group 16 ### 3f:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05) 3f:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05) ### Group 17 ### 3f:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) 3f:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) ### Group 18 ### 3f:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05) 3f:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05) 3f:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05) ### Group 19 ### 3f:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05) 3f:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05) 3f:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05) 3f:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05) ### Group 20 ### 3f:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05) 3f:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05) 3f:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05) 3f:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05) ### Group 21 ### 3f:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05) 3f:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05) 3f:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05) 3f:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)

With old ACS patch :

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 12) ### Group 1 ### 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 12) ### Group 2 ### 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 12) ### Group 3 ### 00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 12) 00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 12) ### Group 4 ### 00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 12) 00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 12) ### Group 5 ### 00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 12) ### Group 6 ### 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 12) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 12) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 12) ### Group 7 ### 00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 12) ### Group 8 ### 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 ### Group 9 ### 00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller ### Group 10 ### 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 04:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02) 04:00.1 IDE interface: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 ### Group 12 ### 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) ### Group 13 ### 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller ### Group 14 ### 01:00.0 VGA compatible controller: NVIDIA Corporation NV44 [GeForce 7100 GS] (rev a1) ### Group 15 ### 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] ### Group 16 ### 3f:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05) 3f:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05) ### Group 17 ### 3f:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) 3f:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) ### Group 18 ### 3f:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05) 3f:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05) 3f:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05) ### Group 19 ### 3f:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05) 3f:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05) 3f:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05) 3f:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05) ### Group 20 ### 3f:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05) 3f:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05) 3f:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05) 3f:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05) ### Group 21 ### 3f:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05) 3f:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05) 3f:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05) 3f:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)

With new ACS patch :

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 12) ### Group 1 ### 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 12) ### Group 2 ### 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 12) ### Group 3 ### 00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 12) 00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 12) ### Group 4 ### 00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 12) 00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 12) ### Group 5 ### 00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 12) ### Group 6 ### 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 12) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 12) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 12) ### Group 7 ### 00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 12) ### Group 8 ### 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 ### Group 9 ### 00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller ### Group 10 ### 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 04:00.0 SATA controller: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02) 04:00.1 IDE interface: JMicron Technology Corp. JMB363 SATA/IDE Controller (rev 02) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 ### Group 12 ### 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) ### Group 13 ### 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller ### Group 14 ### 01:00.0 VGA compatible controller: NVIDIA Corporation NV44 [GeForce 7100 GS] (rev a1) ### Group 15 ### 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] ### Group 16 ### 3f:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05) 3f:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05) ### Group 17 ### 3f:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) 3f:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) ### Group 18 ### 3f:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05) 3f:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05) 3f:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05) ### Group 19 ### 3f:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05) 3f:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05) 3f:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05) 3f:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05) ### Group 20 ### 3f:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05) 3f:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05) 3f:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05) 3f:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05) ### Group 21 ### 3f:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05) 3f:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05) 3f:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05) 3f:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)

If you need more information ask me ^^.

Offline

#1176 2014-02-17 21:47:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the info, it's doing exactly what it's supposed to...

Val532 wrote:

On my AsRock Z87 extrem 6 :
Without patch

### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

I've trimmed this down to just the problem caused by the PCH root ports

With old ACS patch :

Doesn't matter, trimmed

Whit the new ACS patch :

### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 9 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 10 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) ### Group 11 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 14 ### 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 15 ### 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) ### Group 16 ### 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

So all of the problems caused by lack of _PCH_ root more ACS are gone, the devices are properly separated.

On my GigaByte X58 .... :

Doesn't matter, X58 is ICH, not PCH.

So you seem to be expecting this patch to address lack of ACS on every known Intel root port.  As stated in the patch, this only supports PCH root ports.  As of yet, I cannot confirm whether it's safe to use the ACS override patch on other root ports, but I will continue to push Intel for solutions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1177 2014-02-17 22:08:08

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Thanks for the info, it's doing exactly what it's supposed to...

But no, or i do not understand, for the X58 ok it just for compare (and my report is not good).
When i use the new acs patch on my Z87 hardware my two VGA card are on the same iommu group ?!

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

But why with the old patch i have that :

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) ### Group 3 ### 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) ### Group 9 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 10 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 11 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 12 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) ### Group 13 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 16 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 17 ### 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

If i understand corectly, the root port on the CPU is to "ACS" enable ?

Last edited by Val532 (2014-02-17 22:12:17)

Offline

#1178 2014-02-17 22:22:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:
aw wrote:

Thanks for the info, it's doing exactly what it's supposed to...

But no, or i do not understand, for the X58 ok it just for compare (and my report is not good).
When i use the new acs patch on my Z87 hardware my two VGA card are on the same iommu group ?!

See this diagram:

Z87 chipset diagram

The line coming out of the top left of the processor goes to a set of boxes representing the possible root port configuration for processor-based root ports.  You have the last option, which provides 3 slots, however those root ports are sourced from the processor, not the PCH, so are unaffected by this patch that's currently going upstream.

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

But why with the old patch i have that :

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) ### Group 3 ### 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) ### Group 9 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 10 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 11 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 12 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) ### Group 13 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 16 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 17 ### 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1179 2014-02-17 22:41:57

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.

Ok so for the moment i have to use the old patch not the new even if it's probably not a safe patch.

Ok ^^.

Are your planning to make an adition to the new patch ? That add the processor-based root ports a not safe patch but just for people like me with multiple VGA passthrough ^^.

Last edited by Val532 (2014-02-17 22:46:26)

Offline

#1180 2014-02-17 22:47:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:
aw wrote:

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.

Ok so for the moment i have to use the old patch not the new because of probably not a safe patch.

Ok ^^.

Are your planning to make an adition to the new patch ? That add the processor-based root ports a not safe patch but just for people like me with multiple VGA passthrough ^^.

No, I proposed the ACS override patch upstream and it was rejected for fear that it once again places a dangerous tool in the arsenal that users are quick to jump on for functionality without fully understanding the implications (as evidenced by this forum wink).  The kernel will only accept quirks that the vendors identify as safe.  I expect that as a result of this, we'll see a resurgence of ACS support in Intel chipsets, but getting confirmation and code in place for existing devices is a tedious process.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1181 2014-02-18 00:04:54

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

No, I proposed the ACS override patch upstream and it was rejected for fear that it once again places a dangerous tool in the arsenal that users are quick to jump on for functionality without fully understanding the implications (as evidenced by this forum wink).  The kernel will only accept quirks that the vendors identify as safe.  I expect that as a result of this, we'll see a resurgence of ACS support in Intel chipsets, but getting confirmation and code in place for existing devices is a tedious process.

Ok i understand, but it's a fun fonction ^^. I understand the acs is a tools that can make some problem for some people (like me ^^), but for the problem is just i was to fast and not completly read the comment in the patch ^^, so it's my fault.

For the moment the old acs is sufficient for the use of people like my (with multiple vga card and multiple guest ^^) but i think a proper implementation of the stuf in the i915 patch is needed, and probably the correction of some bug on iommu for intel (like the AsMedia or marvell sata controler and probably other). And by the way any news for this problem ?

Thanks for the time you spent to answering to me.

PS : If some body is intersted for a Mac OSX 10.x VM with VGA passthrough i can help to make this, because i have a "working" VM with Max OSX 10.8.

Offline

#1182 2014-02-18 03:05:42

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, I'm trying to do this these days, finally works, only have one problem.
And...if this question unsuitable to post here, please help me to delete this reply.
Also, my English is bad, so apologize for my bad English.

Post my PC spec first:

CPU: Intel Core i7 3820
MB: ASUS Rampage IV Gene (Intel X79)
RAM: 4GB*2
VGA1: AMD Radeon R9 270 (Host)
VGA2: NVIDIA GeForce GTX770 (Guest)

But I'm not use Archlinux to do this, I'm using Ubuntu 12.04.4 x86_64.

And here is my installation steps

1.  Install VFIO-patched kernel as module and iommu_vfio

1.1  Add settings to kernel (CONFIG_VIFO*=y)

1.2  Add "intel_iommu=on" and "pci-stub.ids=(HID)" into /etc/default/grub

1.3  Set nouveau to blacklist

1.4. Add pci-stub to /etc/modules

When I finished in here, I checked dmesg, IOMMU and pci-stub is work.
Then.

2.  Reboot. And make vfio-bind according from 1st post, copy to /usr/bin and set permission.

2.1  Make service according from 1st post into /etc/systemd/system

2.2  Put GTX770 pci number into /etc/vfio-pci.cfg

3.  Reboot again, now bind GTX770 pci number into VFIO (need root)

4.  Start VM with this command:

# qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/ak74/test.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/home/ak74/win7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \


Now a problem is, it can run without errors. But when I typed start command, it only shows

"VNC server started at 127.0.0.1:5900"

And didn't show any VNC screen, just like nothing happen.
But if I plug DVI cable on GTX770 it can shows VM screen (as I saw is WIndows 7 setup.)

After that, I installed TigerVNC and try to connect VM's VNC server, but when I connected, it only show QEMU command.

Maybe I'll try Archlinux to do this, but how can I use VNC to show GTX770's screen?
What step that I doing wrong or I forget to do?

Many thanks.

Offline

#1183 2014-02-18 03:30:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

# qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/ak74/test.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/home/ak74/win7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \


Now a problem is, it can run without errors. But when I typed start command, it only shows

"VNC server started at 127.0.0.1:5900"

When using an assigned VGA, the VNC server doesn't have anything to connect to.  You can use the -nographic option to prevent it from being started.  If you need remote access to the accelerated desktop, you'll need to get it from inside the VM, just like you would if it was a separate physical system (ex. guest-based VNC server).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1184 2014-02-18 05:16:18

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

When using an assigned VGA, the VNC server doesn't have anything to connect to.  You can use the -nographic option to prevent it from being started.  If you need remote access to the accelerated desktop, you'll need to get it from inside the VM, just like you would if it was a separate physical system (ex. guest-based VNC server).

Thanks for your reply.

So let me make a example, if I want to install Windows first, I need to use virtual graphic at beginning
After that, install VNC server that support guest OS into Windows, then disable virtual VGA and virtual graphic, and connect to guest OS when VM is already get into Windows.

Will it work?

Offline

#1185 2014-02-18 05:30:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
aw wrote:

When using an assigned VGA, the VNC server doesn't have anything to connect to.  You can use the -nographic option to prevent it from being started.  If you need remote access to the accelerated desktop, you'll need to get it from inside the VM, just like you would if it was a separate physical system (ex. guest-based VNC server).

Thanks for your reply.

So let me make a example, if I want to install Windows first, I need to use virtual graphic at beginning
After that, install VNC server that support guest OS into Windows, then disable virtual VGA and virtual graphic, and connect to guest OS when VM is already get into Windows.

Will it work?

Yes, if you can't use a physical monitor attached to the card, that's precisely what you'll need to do.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1186 2014-02-18 08:55:34

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Video passthrought had worked for me with reboot problem when I used pci-stub metod and started VM via virt-manager.

Reboot  problem dissapeared  for me when I used vfio method and started via console, but host intel graphics becomes spoiled.

After I had installed Radeon Driver it stopped working at all, and host begun to crash.

Offline

#1187 2014-02-18 09:25:41

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yes, if you can't use a physical monitor attached to the card, that's precisely what you'll need to do.

Actually, I can. My monitor support HDMI and DVI, so I can use DVI to see GTX770's VM screen.
The problem is I can't assign my wireless keyboard and mouse into VM, even I assigned USB 2.0 controllor into VFIO.

Also I tried:

-usb -usbdevice host:046d:c534

but got error

qemu: could not add USB device 'host:046d:c534'

Offline

#1188 2014-02-18 09:28:40

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Also I tried:

-usb -usbdevice host:046d:c534

but got error

qemu: could not add USB device 'host:046d:c534'

You need to prefix them with 0x (those numbers are in hexadecimal).

Offline

#1189 2014-02-18 10:30:46

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What is the cause of this crash ?

[    0.387396] ------------[ cut here ]------------
[    0.387401] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90()
[    0.387402] Modules linked in:
[    0.387405] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.13.0-1-mainline #1
[    0.387406] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Professional, BIOS P1.70B 11/12/2013
[    0.387407]  0000000000000009 ffff880803469d20 ffffffff814f5cc4 0000000000000000
[    0.387410]  ffff880803469d58 ffffffff81061abd ffff8808032b3098 ffff8808032b3000
[    0.387412]  ffff8808032b3098 0000000000000000 ffff88080e8097e0 ffff880803469d68
[    0.387413] Call Trace:
[    0.387418]  [<ffffffff814f5cc4>] dump_stack+0x4d/0x6f
[    0.387422]  [<ffffffff81061abd>] warn_slowpath_common+0x7d/0xa0
[    0.387424]  [<ffffffff81061b9a>] warn_slowpath_null+0x1a/0x20
[    0.387426]  [<ffffffff812c9c07>] pci_find_upstream_pcie_bridge+0x77/0x90
[    0.387429]  [<ffffffff813e429d>] intel_iommu_add_device+0x4d/0x220
[    0.387431]  [<ffffffff813da620>] ? bus_set_iommu+0x50/0x50
[    0.387433]  [<ffffffff813da64a>] add_iommu_group+0x2a/0x50
[    0.387436]  [<ffffffff8138c633>] bus_for_each_dev+0x63/0xa0
[    0.387438]  [<ffffffff813da618>] bus_set_iommu+0x48/0x50
[    0.387441]  [<ffffffff8192d0cb>] intel_iommu_init+0xaad/0xba2
[    0.387443]  [<ffffffff81052355>] ? free_init_pages+0x75/0xa0
[    0.387446]  [<ffffffff818e61c5>] ? maybe_link.part.1+0x10e/0x10e
[    0.387449]  [<ffffffff818eed6a>] ? memblock_find_dma_reserve+0x124/0x124
[    0.387451]  [<ffffffff818eed7c>] pci_iommu_init+0x12/0x3c
[    0.387453]  [<ffffffff8100214a>] do_one_initcall+0xfa/0x1b0
[    0.387456]  [<ffffffff810818b5>] ? parse_args+0x225/0x3f0
[    0.387459]  [<ffffffff818e50da>] kernel_init_freeable+0x17b/0x200
[    0.387461]  [<ffffffff818e48d0>] ? do_early_param+0x88/0x88
[    0.387464]  [<ffffffff814e8bf0>] ? rest_init+0x90/0x90
[    0.387466]  [<ffffffff814e8bfe>] kernel_init+0xe/0x130
[    0.387467]  [<ffffffff8150407c>] ret_from_fork+0x7c/0xb0
[    0.387469]  [<ffffffff814e8bf0>] ? rest_init+0x90/0x90
[    0.387474] ---[ end trace a2c46857dc8df419 ]---

Offline

#1190 2014-02-18 12:12:33

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zzz3000 wrote:

Video passthrought had worked for me with reboot problem when I used pci-stub metod and started VM via virt-manager.

Reboot  problem dissapeared  for me when I used vfio method and started via console, but host intel graphics becomes spoiled.

After I had installed Radeon Driver it stopped working at all, and host begun to crash.

You'll probably need to use the i915 patch

Offline

#1191 2014-02-18 16:05:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

What is the cause of this crash ?

[    0.387396] ------------[ cut here ]------------
[    0.387401] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90()

A trivial search would find https://bugzilla.kernel.org/show_bug.cgi?id=44881

The short answer is broken hardware.  A number of PCIe-to-PCI bridges do not include a PCIe capability, in direct violation of the PCIe spec, confusing the code that tries to find the upstream PCIe bridge and suddenly finds itself on a non-bridge PCIe device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1192 2014-02-18 16:57:53

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks.

Is it possible to find the device which causes the crash in order to disable it ?

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002 02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01) 08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 09:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 0a:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller 0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 0c:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller 0c:00.1 IDE interface: VIA Technologies, Inc. VT6415 PATA IDE Host Controller (rev a0)
-[0000:00]-+-00.0 Intel Corporation 2nd Generation Core Processor Family DRAM Controller +-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device 0002 +-02.0 Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller +-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 +-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 +-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller +-1c.0-[02]----00.0 LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS +-1c.7-[03-0c]----00.0-[04-0c]--+-01.0-[05]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-04.0-[06-07]----00.0-[07]-- | +-05.0-[08]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-06.0-[09]----00.0 Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe | +-07.0-[0a]----00.0 Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller | +-08.0-[0b]----00.0 Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe | \-09.0-[0c]--+-00.0 VIA Technologies, Inc. VT6315 Series Firewire Controller | \-00.1 VIA Technologies, Inc. VT6415 PATA IDE Host Controller +-1d.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 +-1f.0 Intel Corporation Z77 Express Chipset LPC Controller +-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Offline

#1193 2014-02-18 17:04:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

Thanks.

Is it possible to find the device which causes the crash in order to disable it ?

Probably not unless you're very good with a soldering iron.

06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)

This is a known offender.

 | +-04.0-[06-07]----00.0-[07]--

You don't have anything behind it anyway, so can't you ignore it?  It's not actually a kernel crash, it's just a warning and stack trace.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1194 2014-02-18 18:30:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borderlands folks, I think this is the problem you're trying to describe:

Bare Metal

VM

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Last edited by aw (2014-02-19 17:23:21)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1195 2014-02-18 18:41:01

peedeeX21
Member
Registered: 2014-02-18
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi ppl,

tying to set up a virual machine and passing throug my gt610m.

CPU: Intel Core i5 3317
RAM: 4GB*2
VGA1: Intel HD4000
VGA2: Nvidia GeForce GT 610M
OS: Linux Mint 16

I set up everything like GizmoChicken posted.

I am using a Kernel build with the options:

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_INTEL_IOMMU_DEFAULT_ON=y

(stock kernel 1.11.0-12 does also not work...)

The problem ist now, that here are no iommu_group links in /sys/bus/pci/devices/xxxx:xx:xx.x.

Things i did / checked:

  • In BIOS the Intel Virtualisation Technology is enable

  • intel_iommu=1 is set in /etc/default/grub

  • echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Has anybody an idea why this links are missing? thanks.

Offline

#1196 2014-02-18 20:26:03

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What motherboard are you using?  Intel Virtualization technology just means Vt-x not Vt-d.

Offline

#1197 2014-02-18 21:20:24

peedeeX21
Member
Registered: 2014-02-18
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

What motherboard are you using?  Intel Virtualization technology just means Vt-x not Vt-d.

oh... i missed that. i got a Lenovo motherboard with Mobile Intel HM77 Express Chipset on it, which does not support vt-d.

thanks for your support.

EDIT:

but how is it possible

 dmesg | grep IOMMU 

tells me

 Intel-IOMMU: enabled 

?

Last edited by peedeeX21 (2014-02-18 21:25:14)

Offline

#1198 2014-02-18 21:48:20

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe that is the kernel acknowledging your intel_iommu=on

Offline

#1199 2014-02-19 01:30:38

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some test, now I can use this VM functionally.

I tried to use TigerVNC server, but I don't know why if it set to service mode, client application can't connect.
And use user-mode server, it can connect, but after less 10 sec, server will get crashes.

Finally, I use TeamViewer to my temporary VNC. Looks like plug another monitor is the best choice, or does it have better solutions to this?


Also, sound is another problem, too.
I don't know why my QEMU doesn't have ALSA or pa.
When I type -audio-help, it only shows oss and wav. And oss can't use.
How can I find other modules and how to install it?


Sorry for my 'newbie' questions.

Offline

#1200 2014-02-20 06:01:29

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just installed a second of the same video card (to forward to the same VM), and no matter what I try, qemu always aborts with an assertion failure.


With EFI and piix4 (since EFI has never worked with q35):

ASSERT /home/dana/Downloads/edk2/PcAtChipsetPkg/PciHostBridgeDxe/PciHostBridge.c(394): ((BOOLEAN)(0==1))

With BIOS mode and q35, I've tried the same settings that somewhat work with a single card:

qemu-system-x86_64: hw/pci/pci.c:232: pcibus_reset: Assertion `bus->irq_count[i] == 0' failed.

With a single video card, I still get this assertion failure, but only when I try to disable the Radeon in Device Manager.

I'd love to get the second video card working, but that's not possible, I'll have to return it.

EDIT: Hmm, after some more wrestling, I may have gotten it working.  Anyway, the "irq count" assertion seems like there must be some broken logic somewhere.

Last edit for the night: I got it to actually, totally, work (aside from likely crashing if I disable a device).  I had to make sure both video cards had the dual-ROM switch set to the GOP mode.  It would still be good to get that assertion fixed, though.

Last edited by DanaGoyette (2014-02-20 07:19:43)

Offline

#1201 2014-02-20 12:06:45

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
zzz3000 wrote:

Video passthrought had worked for me with reboot problem when I used pci-stub metod and started VM via virt-manager.

Reboot  problem dissapeared  for me when I used vfio method and started via console, but host intel graphics becomes spoiled.

After I had installed Radeon Driver it stopped working at all, and host begun to crash.

You'll probably need to use the i915 patch

Thanks for reply.

i915 patch could resolv intel spoiled graphics or  full crash with radeon driver installed on Windows?

Where  can I get this patch? Is  it a kernel patch or what?

Also I don't see bios booting in guest's video card.

I am using Fedora 20.

Last edited by zzz3000 (2014-02-21 06:38:06)

Offline

#1202 2014-02-22 16:39:38

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, I tried switching my NVIDIA K2000 and AMD HD 7970 in my ASUS RIVE with 4930K .  Previously, everything worked without problems with the NVIDIA as guest and the AMD as host, but of course I couldn't run the AMD drivers on the host, and the radeon drivers don't support 3D acceleration. So now I have the NVIDIA on the host with the HD7970 as guest.  I ave been running the same kernel/qemu/seabios; ie 3.13.2/qemu-git and seabios git downloaded 12-5-13.   I've patched the kernel with https://git.kernel.org/cgit/linux/kerne … fe8702b098 . My kernel line has vga=normal on it, but of course I still get this warning in dmesg:

fb0: EFI VGA frame buffer device [ 18.026908] hda-intel 0000:01:00.1: Handle VGA-switcheroo audio client [ 49.298091] NVRM: Your system is not currently configured to drive a VGA console [ 49.298094] NVRM: on the primary VGA device. The NVIDIA Linux graphics driver [ 49.298095] NVRM: requires the use of a text-mode VGA console. Use of other console

because I boot in efi mode.  I'm not sure this a way to boot in any other mode on this motherboard. I've tried this https://devtalk.nvidia.com/default/topi … -warning-/
but there is no framebuffer other than efifb, so i get a black screen if I add the video=efifb:off.  All that might be irrelevent.

I uninstalled the NVIDIA drivers from the guest prior to swapping.  I patched the NVIDIA driver for the VGA arbitration. Using 319.76 (or for that matter 334.16) on the host seems to work fine with functioning OpenGL. 

I removed the 03:00.1 audio passthrough on my qemu line and installed the AMD driver without CCC.  Windows boots, and seems to work OK with no errors seen in Device Manager.  However, while I can shutdown the guest, I cannot restart the guest, and in fact the host temporarily freezes up.  I tried ejecting the AMD driver prior to shutdown, didn't help and now I have a black screen on the guest too.  Here's dmesg after the crash/freeze. Seems to be a memory leak or something

[ 186.932077] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003) [ 186.953915] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270 [ 186.953920] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1b@0x2d0 [ 187.081805] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa [ 189.849067] dmar: DRHD: handling fault status reg 2 [ 189.849073] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000 [ 189.849073] DMAR:[fault reason 06] PTE Read access is not set [ 201.195616] vfio-pci 0000:03:00.0: irq 103 for MSI/MSI-X [ 297.885563] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270 [ 297.885568] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1b@0x2d0 [ 298.006473] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa [ 410.323923] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270 [ 410.323930] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1b@0x2d0 [ 410.452817] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa [ 412.892137] kvm: zapping shadow pages for mmio generation wraparound [ 774.404347] kwin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [ 774.404351] kwin cpuset=/ mems_allowed=0 [ 774.404354] CPU: 5 PID: 1539 Comm: kwin Tainted: P O 3.13.2 #2 [ 774.404355] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 9502 11/07/2013 [ 774.404356] ffff880829b4db90 ffff880816fafa60 ffffffff816338cd ffff880829b4d640 [ 774.404358] ffff880816fafac0 ffffffff81631535 0000000001320122 ffff880816fafa88 [ 774.404360] ffffffff8114daee 0000000000000286 ffffffff81c3b820 ffff8805b227e4a0 [ 774.404363] Call Trace: [ 774.404369] [<ffffffff816338cd>] dump_stack+0x45/0x56 [ 774.404372] [<ffffffff81631535>] dump_header.isra.12+0x76/0x1a7 [ 774.404376] [<ffffffff8114daee>] ? __delayacct_freepages_end+0x2e/0x30 [ 774.404379] [<ffffffff81183d2d>] oom_kill_process+0x1dd/0x340 [ 774.404382] [<ffffffff812a3835>] ? security_capable_noaudit+0x15/0x20 [ 774.404384] [<ffffffff811842a4>] out_of_memory+0x254/0x280 [ 774.404387] [<ffffffff811890c3>] __alloc_pages_nodemask+0x8b3/0x930 [ 774.404389] [<ffffffff81182450>] filemap_fault+0x1b0/0x3d0 [ 774.404393] [<ffffffff811a4449>] __do_fault+0x69/0x4e0 [ 774.404394] [<ffffffff811821cf>] ? wait_on_page_bit_killable+0x7f/0x90 [ 774.404396] [<ffffffff811a870a>] handle_mm_fault+0x30a/0xc70 [ 774.404399] [<ffffffff81095fe0>] __do_page_fault+0x170/0x500 [ 774.404403] [<ffffffff811e31fc>] ? poll_select_copy_remaining+0xdc/0x130 [ 774.404405] [<ffffffff810963ae>] do_page_fault+0xe/0x10 [ 774.404408] [<ffffffff8163c608>] page_fault+0x28/0x30 [ 774.404409] Mem-Info: [ 774.404410] DMA per-cpu: [ 774.404411] CPU 0: hi: 0, btch: 1 usd: 0 [ 774.404412] CPU 1: hi: 0, btch: 1 usd: 0 [ 774.404413] CPU 2: hi: 0, btch: 1 usd: 0 [ 774.404414] CPU 3: hi: 0, btch: 1 usd: 0 [ 774.404415] CPU 4: hi: 0, btch: 1 usd: 0 [ 774.404416] CPU 5: hi: 0, btch: 1 usd: 0 [ 774.404417] CPU 6: hi: 0, btch: 1 usd: 0 [ 774.404418] CPU 7: hi: 0, btch: 1 usd: 0 [ 774.404419] CPU 8: hi: 0, btch: 1 usd: 0 [ 774.404419] CPU 9: hi: 0, btch: 1 usd: 0 [ 774.404420] CPU 10: hi: 0, btch: 1 usd: 0 [ 774.404421] CPU 11: hi: 0, btch: 1 usd: 0 [ 774.404422] DMA32 per-cpu: [ 774.404423] CPU 0: hi: 186, btch: 31 usd: 0 [ 774.404424] CPU 1: hi: 186, btch: 31 usd: 0 [ 774.404425] CPU 2: hi: 186, btch: 31 usd: 0 [ 774.404426] CPU 3: hi: 186, btch: 31 usd: 0 [ 774.404427] CPU 4: hi: 186, btch: 31 usd: 0 [ 774.404428] CPU 5: hi: 186, btch: 31 usd: 0 [ 774.404429] CPU 6: hi: 186, btch: 31 usd: 0 [ 774.404430] CPU 7: hi: 186, btch: 31 usd: 0 [ 774.404430] CPU 8: hi: 186, btch: 31 usd: 0 [ 774.404431] CPU 9: hi: 186, btch: 31 usd: 0 [ 774.404432] CPU 10: hi: 186, btch: 31 usd: 0 [ 774.404433] CPU 11: hi: 186, btch: 31 usd: 0 [ 774.404434] Normal per-cpu: [ 774.404435] CPU 0: hi: 186, btch: 31 usd: 0 [ 774.404436] CPU 1: hi: 186, btch: 31 usd: 0 [ 774.404436] CPU 2: hi: 186, btch: 31 usd: 0 [ 774.404437] CPU 3: hi: 186, btch: 31 usd: 0 [ 774.404438] CPU 4: hi: 186, btch: 31 usd: 0 [ 774.404439] CPU 5: hi: 186, btch: 31 usd: 0 [ 774.404440] CPU 6: hi: 186, btch: 31 usd: 0 [ 774.404441] CPU 7: hi: 186, btch: 31 usd: 0 [ 774.404442] CPU 8: hi: 186, btch: 31 usd: 0 [ 774.404442] CPU 9: hi: 186, btch: 31 usd: 0 [ 774.404443] CPU 10: hi: 186, btch: 31 usd: 0 [ 774.404444] CPU 11: hi: 186, btch: 31 usd: 0 [ 774.404447] active_anon:7519336 inactive_anon:582210 isolated_anon:320 [ 774.404447] active_file:15 inactive_file:83 isolated_file:0 [ 774.404447] unevictable:0 dirty:0 writeback:0 unstable:0 [ 774.404447] free:67089 slab_reclaimable:7419 slab_unreclaimable:11764 [ 774.404447] mapped:6157 shmem:2 pagetables:9275 bounce:0 [ 774.404447] free_cma:0 [ 774.404452] DMA free:15864kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 774.404453] lowmem_reserve[]: 0 2723 32114 32114 [ 774.404457] DMA32 free:128936kB min:11456kB low:14320kB high:17184kB active_anon:2072576kB inactive_anon:576816kB active_file:60kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2804552kB managed:2794080kB mlocked:0kB dirty:0kB writeback:0kB mapped:700kB shmem:0kB slab_reclaimable:3512kB slab_unreclaimable:2280kB kernel_stack:128kB pagetables:2548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3988935 all_unreclaimable? yes [ 774.404458] lowmem_reserve[]: 0 0 29390 29390 [ 774.404463] Normal free:123556kB min:123640kB low:154548kB high:185460kB active_anon:28004768kB inactive_anon:1752024kB active_file:0kB inactive_file:280kB unevictable:0kB isolated(anon):1280kB isolated(file):0kB present:30670848kB managed:30095696kB mlocked:0kB dirty:0kB writeback:0kB mapped:23928kB shmem:8kB slab_reclaimable:26164kB slab_unreclaimable:44744kB kernel_stack:3392kB pagetables:34552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:45224968 all_unreclaimable? yes [ 774.404463] lowmem_reserve[]: 0 0 0 0 [ 774.404465] DMA: 0*4kB 1*8kB (U) 1*16kB (U) 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (MR) = 15864kB [ 774.404473] DMA32: 270*4kB (UEM) 180*8kB (UEM) 215*16kB (UEM) 333*32kB (UEM) 221*64kB (UEM) 145*128kB (UEM) 95*256kB (UEM) 42*512kB (UEM) 33*1024kB (UEM) 0*2048kB 0*4096kB = 128936kB [ 774.404481] Normal: 509*4kB (UEM) 389*8kB (UEM) 224*16kB (UEM) 242*32kB (UEM) 129*64kB (UEM) 103*128kB (UEM) 109*256kB (UEM) 73*512kB (UEM) 20*1024kB (UEM) 0*2048kB 0*4096kB = 123676kB [ 774.404488] 575775 total pagecache pages [ 774.404490] 575675 pages in swap cache [ 774.404491] Swap cache stats: add 1679802, delete 1104127, find 30940/40375 [ 774.404492] Free swap = 30715568kB [ 774.404493] Total swap = 33554428kB [ 774.404493] 8372849 pages RAM [ 774.404494] 0 pages HighMem/MovableOnly [ 774.404495] 143788 pages reserved [ 774.404496] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 774.404499] [ 467] 0 467 7610 15 21 298 -1000 udevd [ 774.404501] [ 710] 0 710 1617 0 8 44 0 syslogd [ 774.404502] [ 714] 0 714 1090 0 8 23 0 klogd [ 774.404504] [ 923] 0 923 2184 0 10 67 0 dhcpcd [ 774.404505] [ 993] 1 993 1630 0 8 29 0 rpc.portmap [ 774.404507] [ 997] 32 997 3206 1 12 41 0 rpc.statd [ 774.404508] [ 1015] 0 1015 1612 0 9 26 0 inetd [ 774.404509] [ 1018] 0 1018 6994 0 17 108 -1000 sshd [ 774.404511] [ 1029] 0 1029 94565 0 27 105 0 automount [ 774.404512] [ 1043] 0 1043 1093 0 8 42 0 acpid [ 774.404514] [ 1056] 81 1056 5096 111 13 130 0 dbus-daemon [ 774.404515] [ 1059] 0 1059 1063594 0 91 391 0 console-kit-dae [ 774.404517] [ 1128] 0 1128 53403 0 43 1121 0 polkitd [ 774.404518] [ 1157] 0 1157 13311 32 31 186 0 cupsd [ 774.404519] [ 1159] 0 1159 3198 0 12 37 0 crond [ 774.404521] [ 1161] 0 1161 3196 0 11 35 0 atd [ 774.404522] [ 1165] 0 1165 68585 13 131 409 0 smbd [ 774.404523] [ 1167] 0 1167 49046 39 92 272 0 nmbd [ 774.404524] [ 1169] 0 1169 1878 4 9 23 0 gpm [ 774.404526] [ 1179] 0 1179 68585 27 130 395 0 smbd [ 774.404527] [ 1193] 0 1193 75778 39 86 779 0 libvirtd [ 774.404528] [ 1212] 0 1212 1657 14 8 19 0 vde_switch [ 774.404530] [ 1306] 1000 1306 5143 1 16 178 0 bash [ 774.404531] [ 1307] 0 1307 3199 1 12 35 0 agetty [ 774.404533] [ 1308] 0 1308 3199 1 12 36 0 agetty [ 774.404534] [ 1309] 0 1309 3199 1 12 38 0 agetty [ 774.404536] [ 1310] 0 1310 3199 1 12 39 0 agetty [ 774.404537] [ 1311] 0 1311 3199 1 12 39 0 agetty [ 774.404538] [ 1365] 1000 1365 5153 1 16 188 0 bash [ 774.404540] [ 1381] 1000 1381 4002 0 13 50 0 xinit [ 774.404541] [ 1382] 1000 1382 44593 1696 89 9035 0 X [ 774.404543] [ 1385] 0 1385 7609 61 21 256 -1000 udevd [ 774.404544] [ 1388] 0 1388 7609 0 21 315 -1000 udevd [ 774.404545] [ 1399] 1000 1399 2971 0 11 102 0 sh [ 774.404547] [ 1400] 1000 1400 9189 0 24 70 0 ck-launch-sessi [ 774.404548] [ 1409] 1000 1409 2978 1 11 108 0 startkde [ 774.404550] [ 1443] 1000 1443 7720 0 20 82 0 dbus-launch [ 774.404551] [ 1444] 1000 1444 5268 35 14 252 0 dbus-daemon [ 774.404552] [ 1451] 1000 1451 1050 0 6 21 -300 start_kdeinit [ 774.404554] [ 1452] 1000 1452 87774 4 151 3670 -300 kdeinit4 [ 774.404555] [ 1453] 1000 1453 90179 2 132 4257 0 klauncher [ 774.404557] [ 1455] 1000 1455 431428 378 279 5744 0 kded4 [ 774.404558] [ 1457] 1000 1457 5117 1 17 103 0 gam_server [ 774.404560] [ 1461] 1000 1461 111316 13 171 4401 0 kglobalaccel [ 774.404561] [ 1475] 1000 1475 114168 4 175 4488 0 kwalletd [ 774.404562] [ 1480] 0 1480 49158 0 31 765 0 upowerd [ 774.404564] [ 1508] 0 1508 47874 120 31 655 0 udisks-daemon [ 774.404565] [ 1512] 0 1512 10848 0 25 74 0 udisks-daemon [ 774.404567] [ 1533] 1000 1533 1084 0 8 22 0 kwrapper4 [ 774.404568] [ 1534] 1000 1534 133118 0 180 4075 0 ksmserver [ 774.404569] [ 1536] 1000 1536 177296 0 156 1912 0 kactivitymanage [ 774.404571] [ 1539] 1000 1539 728405 7237 273 9453 0 kwin [ 774.404572] [ 1553] 1000 1553 249972 92 213 4052 0 knotify4 [ 774.404573] [ 1555] 1000 1555 846145 1662 334 23202 0 plasma-desktop [ 774.404575] [ 1561] 1000 1561 3036 0 11 151 0 ksysguardd [ 774.404576] [ 1564] 1000 1564 73196 0 129 1048 0 kuiserver [ 774.404578] [ 1567] 1000 1567 38367 38 38 163 0 akonadi_control [ 774.404579] [ 1586] 1000 1586 109022 0 131 4044 0 nepomukserver [ 774.404581] [ 1590] 1000 1590 203101 8 185 2631 0 nepomukservices [ 774.404582] [ 1591] 1000 1591 212223 167 261 5854 0 krunner [ 774.404584] [ 1593] 1000 1593 76019 13 130 988 0 nepomukcontroll [ 774.404585] [ 1595] 1000 1595 158076 113 191 5208 0 kmix [ 774.404587] [ 1600] 1000 1600 103051 3267 55 8842 0 virtuoso-t [ 774.404588] [ 1601] 1000 1601 670612 14 202 8803 0 kwrite [ 774.404589] [ 1604] 1000 1604 155527 0 160 1830 0 synergy [ 774.404591] [ 1606] 1000 1606 138834 608 190 4618 0 konsole [ 774.404592] [ 1612] 1000 1612 138974 350 191 4823 0 konsole [ 774.404593] [ 1616] 1000 1616 5134 0 16 169 0 bash [ 774.404595] [ 1619] 1000 1619 5134 0 15 169 0 bash [ 774.404596] [ 1625] 1000 1625 91018 0 136 4467 0 blueman-applet [ 774.404598] [ 1626] 1000 1626 123644 4 224 5510 0 python [ 774.404599] [ 1629] 1000 1629 97769 3 147 1089 0 polkit-kde-auth [ 774.404600] [ 1632] 1000 1632 39380 144 78 3083 0 python [ 774.404602] [ 1633] 1000 1633 32376 40 64 3163 0 python [ 774.404603] [ 1634] 1000 1634 98085 112 178 1200 0 korgac [ 774.404604] [ 1638] 1000 1638 183525 10028 240 14937 0 firefox [ 774.404606] [ 1641] 1000 1641 113540 122 176 4395 0 klipper [ 774.404607] [ 1649] 1000 1649 46340 0 27 131 0 gvfsd [ 774.404609] [ 1653] 1000 1653 70911 0 40 777 0 gvfsd-fuse [ 774.404610] [ 1661] 1000 1661 12515 5 30 136 0 gconfd-2 [ 774.404611] [ 1679] 1000 1679 47423 0 86 390 0 obex-data-serve [ 774.404613] [ 1720] 1000 1720 115420 151 148 1604 0 nepomukservices [ 774.404614] [ 1721] 1000 1721 97736 144 150 1348 0 nepomukservices [ 774.404615] [ 1729] 0 1729 5137 35 16 139 0 bash [ 774.404617] [ 1845] 0 1845 5137 44 15 131 0 bash [ 774.404618] [ 1851] 1000 1851 33696 56 34 166 0 synergys [ 774.404620] [ 2132] 0 2132 2976 7 11 93 0 sh [ 774.404622] [ 2133] 0 2133 2290625 488926 2188 567641 0 qemu-system-x86 [ 774.404623] [ 2146] 0 2146 49046 49 91 262 0 nmbd

My current qemu line is

QEMU_SDL_SAMPLES=2048 \ qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none\ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio \ -device piix4-ide,bus=pcie.0 \ -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk \ -net vde,sock=/var/run/kvm0.ctl,vlan=0 -net nic,vlan=0,model=virtio \ -device vfio-pci,host=00:1d.0,bus=pcie.0 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -drive file=virtio-win-0.1-74.iso,id=isocd -device ide-cd,bus=ide.2,drive=isocd \ -rtc base=localtime

The DMAR error regarding device [00:1d.0] has always been there (prior to the VGA swap) and refers to a passed through USB controller for a mouse for the guest, so I doubt it is relevent.

[EDIT] Hmm, one more reboot out of many seems to have made the problem go away. Still, if anyone knows what the crash means, let me know smile

Last edited by mostlyharmless (2014-02-22 17:27:54)

Offline

#1203 2014-02-22 20:08:39

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The nvidia guys seem to have a problem with their setup: https://devtalk.nvidia.com/default/topi … 4/#4131164

We have a number on their internal bugtracker, so maybe we should try helping them with their issues, so they can reproduce it and fix it at some point.


i'm sorry for my poor english wirting skills…

Offline

#1204 2014-02-23 06:19:16

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After I patched  fedora kernel-3.13 with  patches from main page, all  problems have disappeared except one.
I get BSOD under Windows7 guest when I am installing  radeon driver for 7750...

log:

Feb 23 17:03:42 DeltaS kernel: [  703.287069] kvm: zapping shadow pages for mmio generation wraparound

Last edited by zzz3000 (2014-02-23 13:06:17)

Offline

#1205 2014-02-24 04:51:37

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've done a good deal more experimenting with my setup (back to one GPU at the moment).  Only one variant seems at all stable.

q35, EFI: Not an option -- OVMF is hardcoded for i440fx.
q35, radeon as primary, x-vga=on, no VGA Arbiter patch: hang (and garbled host graphics) upon trying to run VGA option rom.
q35, radeon as primary, x-vga=on, i915 arbiter patches:  Bus IRQ count assertion failure every single time I try to boot the guest (except in Safe Mode).  Also, breaks host OpenGL, which is a showstopper for me.
q35, radeon as primary, no x-vga: hang upon trying to run VGA option ROM.
q35, radeon as primary, rombar=0: guest boots and can be connected to via VNC... but AMD driver doesn't load.
q35, radeon as secondary: guest boots.  AMD driver won't load, or loads but doesn't detect displays or appear as usable in CCC.

i440fx, EFI guest: Boots reliably and runs reliably... but the bus reset doesn't work!  Any guest reboot or crash requires me to suspend and resume the host!

Last edited by DanaGoyette (2014-02-24 06:42:30)

Offline

#1206 2014-02-24 05:07:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

q35, radeon as primary, x-vga=on, i915 arbiter patches:  Bus IRQ count assertion failure every single time I try to boot the guest (except in Safe Mode).

A number of other people seem to have this one working and it's certainly what I would advise for most people.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1207 2014-02-24 17:05:58

C0NPAQ
Member
Registered: 2014-02-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Suggesting that OP puts 1-2 sentences in his post on what passthrough really does.

I had to read the whole post to count 1:1 together. I expected some functionality which would let you forward the output of graphics card A to display on graphics card B within spice or something. But all you essentially get is one graphics card plainly assigned to the VM and that is that. Meaning you need another monitor (or one with two inputs one unused), another mouse and keyboard if you don't do weird tricks and don't want to steal those from the host OS and switch back and forth. Slightly better than rebooting ... if you happen to have a perfectly redundant graphics card ... but seriously: it sounded way better when you had not thought it through.

Don't know if there are any cards that eat DVI and pipe it to an application. Like TV cards... VNC is too slow for games but the same functionality for keyboard/mouse as a VM driver would work out. Then the problem would be solved.

EDIT: Arrived in the 21th century, there are cards for 100-200 USD, 200 USD with certain linux support. Seem to be made for video recording though, cheap ones even USB, probably latency issues.

Last edited by C0NPAQ (2014-02-24 17:14:22)

Offline

#1208 2014-02-24 18:05:20

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

Offline

#1209 2014-02-24 18:27:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

C0NPAQ wrote:

Suggesting that OP puts 1-2 sentences in his post on what passthrough really does.

I had to read the whole post to count 1:1 together. I expected some functionality which would let you forward the output of graphics card A to display on graphics card B within spice or something. But all you essentially get is one graphics card plainly assigned to the VM and that is that. Meaning you need another monitor (or one with two inputs one unused), another mouse and keyboard if you don't do weird tricks and don't want to steal those from the host OS and switch back and forth. Slightly better than rebooting ... if you happen to have a perfectly redundant graphics card ... but seriously: it sounded way better when you had not thought it through.

Don't know if there are any cards that eat DVI and pipe it to an application. Like TV cards... VNC is too slow for games but the same functionality for keyboard/mouse as a VM driver would work out. Then the problem would be solved.

EDIT: Arrived in the 21th century, there are cards for 100-200 USD, 200 USD with certain linux support. Seem to be made for video recording though, cheap ones even USB, probably latency issues.

It would certainly be nice if there was some way to extract the framebuffer of the card and expose it on the host, and it may be possible, but typically that's not how PCI device assignment works.  If you assign a NIC, you don't somehow expect that it obeys the routing of the host, the NIC is an isolated device with it's own data path.  That said, I can mention that the guest is just a process to the host and that framebuffer that goes out to the screen likely lives somewhere in the physical memory of the system.  If you can find it and not interfere with the guest, you could theoretically render it into a window on a host display.  I have no tips for how to do that, it's likely proprietary and requires in-depth knowledge of how the card works, but it may be possible.  Another option is to use a PCoIP card co-assigned to the guest.  These are intended to be used for low-latency, high performance remote displays.  If they have a software client, it could be run on the host.  Another project that may be interesting is the Virgil 3D GPU project.  This intends to effectively share the host GPU with a guest and allow 3D rendering onto the host display.  If non-Linux gaming is your target, drivers are likely to be an issue with that approach for some time.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1210 2014-02-24 18:33:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

What do you use as a backing store for you OS image?  If you're not using a raw disk/partition/logical volume with virtio drivers, that's an obvious place to start.  Beyond that, there are tools like kvm_stat, perf, and trace-cmd that can show where you're spending time, this is how we were able to determine debug register access is a significant performance bottleneck for games like Borderlands2 (fix coming soon - at least on Intel, no data for AMD yet).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1211 2014-02-24 20:49:40

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SpacePirate wrote:

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

What do you use as a backing store for you OS image?  If you're not using a raw disk/partition/logical volume with virtio drivers, that's an obvious place to start.  Beyond that, there are tools like kvm_stat, perf, and trace-cmd that can show where you're spending time, this is how we were able to determine debug register access is a significant performance bottleneck for games like Borderlands2 (fix coming soon - at least on Intel, no data for AMD yet).

Hi, aw

Is there any tutorial about how to use these tools? I want to help this topic too.

Offline

#1212 2014-02-24 21:51:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

doubledr wrote:

Hi, aw

Is there any tutorial about how to use these tools? I want to help this topic too.

You'll find kvm_stat in the qemu source tree, here are some instructions:

https://docs.fedoraproject.org/en-US/Fe … 20s02.html

Tracing is similarly easy, but the report generated is massive and takes some skill to analyze:

http://www.linux-kvm.org/page/Tracing

Generally you'll want to run kvm_stat first to see what the heavy hitters are, then look in the trace output to try to figure out why they're being called.

There's a bit on perf here too:

http://www.linux-kvm.org/page/Perf_events

A simple place to start, especially if you're compiling your own qemu is to uncomment #define DEBUG_VFIO from hw/misc/vfio.c, recompile and test with that binary.  You will get a ton of initial output but once the OS drivers are running it should slow down considerably.  The output will never cease entirely though, all config space access will go through qemu and any of the quirks.  So long as you don't see more than around a dozen lines per second, vfio is probably not the bottleneck.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1213 2014-02-24 22:14:59

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SpacePirate wrote:

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

What do you use as a backing store for you OS image?  If you're not using a raw disk/partition/logical volume with virtio drivers, that's an obvious place to start.  Beyond that, there are tools like kvm_stat, perf, and trace-cmd that can show where you're spending time, this is how we were able to determine debug register access is a significant performance bottleneck for games like Borderlands2 (fix coming soon - at least on Intel, no data for AMD yet).

Thanks Ill try those.
I use virtio drivers on a raw lvm partition.

Offline

#1214 2014-02-25 13:51:21

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello Andrew,

thanks for this hint. I tried "-m cpu host,hv-time" - but as soon as "hv-time" is added, the Guest (Win7 64bit) behaves aweful: IF the boot succeeds, half of the Services can't get startet, but mostly i get a BSOD with STOP 7e.
It may be this is because of the current GIT-Version of qemu - not because of "hv-time" - i cant tell exactly because i can test hv-time only with current git of qemu;
Guest behaves normal with qemu 1.7.0.
Kernel is 3.14-rc4 in both cases.

Maybe "hv-time" is only possible with a fixed cpu type, not "host" ?

Greetings!

Offline

#1215 2014-02-25 13:56:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello Andrew,

thanks for this hint. I tried "-m cpu host,hv-time" - but as soon as "hv-time" is added, the Guest (Win7 64bit) behaves aweful: IF the boot succeeds, half of the Services can't get startet, but mostly i get a BSOD with STOP 7e.
It may be this is because of the current GIT-Version of qemu - not because of "hv-time" - i cant tell exactly because i can test hv-time only with current git of qemu;
Guest behaves normal with qemu 1.7.0.
Kernel is 3.14-rc4 in both cases.

Maybe "hv-time" is only possible with a fixed cpu type, not "host" ?

Greetings!

I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1216 2014-02-25 18:36:43

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.


i tried it with "Nehalem" (only have a i7 920) and the new git-qemu - but no luck. it seems the current GIT snapshot of qemu messes up the whole virtual machine here: passed through GPU is there, but the device manager says the driver cant be loaded sad

Running with qemu 1.7.0 works perfect here, even with -cpu Nehalem - so it must be the current state of qemu GIT.

Offline

#1217 2014-02-25 19:27:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.


i tried it with "Nehalem" (only have a i7 920) and the new git-qemu - but no luck. it seems the current GIT snapshot of qemu messes up the whole virtual machine here: passed through GPU is there, but the device manager says the driver cant be loaded sad

Running with qemu 1.7.0 works perfect here, even with -cpu Nehalem - so it must be the current state of qemu GIT.

I just refreshed my git tree, head is aa0d1f448871314bfc535da97eb003fe7766d4c2, I add this this patch because otherwise multifunction devices are broken and it works fine.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1218 2014-02-25 20:11:47

neona
Member
Registered: 2014-02-25
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got things mostly working (windows boots, can play games, performs decently, though I'll probably tweak it more), though I'm having a few issues.

First, my commandline and config:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -vga none \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=Gigabyte.GTX550Ti.1024.110308.rom \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -drive file=/dev/sda,if=virtio,format=raw \ -usb -device usb-host,hostbus=3,hostaddr=7 \ -device usb-host,hostbus=6,hostaddr=2 \ -device usb-host,hostbus=6,hostaddr=3 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device virtio-balloon \ -net nic,model=virtio -net user

using the kernel, qemu, and seabios from OP, along with patched nvidia driver on host. Have all the relevant virtio drivers installed and working on guest, though it seems that the host kernel may need to be rebuilt again for me to actually enable balloon.

System specs:

FX-8350
GeForce 9500 GT (host)
GTX 550 Ti (guest)
Windows 8.1 - guest os
16gb of memory, currently allocating 8gb to guest
ASRock 970 EXTREME3

Anyway, onward to problems:

For some reason, qemu only claims to support oss, wav, and spice output, so sound won't work.  I double checked the PKGBUILD for qemu, and it's clearly being compiled with pulse and alsa support (or should be, anyway), so I have no clue what's causing this, and some googling around hasn't found anything.  I've tried various ways of wrapping oss to pulse/alsa, and none of that has worked either.  This is rather inconvienient, but at least if I can get networking to behave then I think it's possible to have windows output to pulse if nothing else works.  I might also get a usb soundcard, but I'd rather avoid spending money if I can.

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.  Right now I'm just passing the host input devices to the guest, so I lose control of the host when I have the vm running (with ssh just in case, of course).  It'd also be nice for ICMP to work. Does anyone have experience with tap/tun?  I haven't been able to find any particularly clear guides, though I've tried a variety of things.

Oh, and does anyone know if use of hugetlbfs is mutually exclusive with proper use of balloon memory? It seems likely.  Also, how much of/what sort of an improvement does hugetlbfs provide, anyway?  Does it just reduce memory/tlb thrashing - can this reduce stutter in games?

Past that, everything is working great though!  Bit of stuttering in some games, but could be way worse.

I appreciate all the research you guys have put into getting this working.

Offline

#1219 2014-02-25 20:36:53

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any quirk list fpr vfio-vga devices in the the kernel or qemu? I ask because a friend of mine has problems getting his Geforce 750 Ti to work with vga-passthrough, while his old G210 (or 8400GS, don't remember) worked.
The relevant dmesg errors seem to be:

[ 142.835457] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x258 [ 142.835472] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x900

Best regards,
Flyser

Offline

#1220 2014-02-25 20:45:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

Is there any quirk list fpr vfio-vga devices in the the kernel or qemu? I ask because a friend of mine has problems getting his Geforce 750 Ti to work with vga-passthrough, while his old G210 (or 8400GS, don't remember) worked.
The relevant dmesg errors seem to be:

[ 142.835457] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x258 [ 142.835472] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x900

These probably aren't relevant.  Extended capability 0x19 is a secondary PCIe capability, 0x1e is L1 PM substates (ie. power management states).  These messages simply mean that we're hiding the capability from the guest, either because it's not necessary, it's inappropriate for a guest, or nobody has added it yet.  It's very unlikely that these are blocking anything.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1221 2014-02-26 00:39:36

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

[I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

Which is the correct one: hv-time or hv_time? libvirt uses hv_time [1]. The long supported hv_relaxed, hv_vapic, hv_spinlocks all use undeline characters [2].

1. http://libvirt.org/git/?p=libvirt.git;a … d0;hb=HEAD
2. http://qemu-project.org/ChangeLog/1.1#x86

Offline

#1222 2014-02-26 01:23:00

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

neona wrote:

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.  Right now I'm just passing the host input devices to the guest, so I lose control of the host when I have the vm running (with ssh just in case, of course).  It'd also be nice for ICMP to work. Does anyone have experience with tap/tun?  I haven't been able to find any particularly clear guides, though I've tried a variety of things.

Do you have tun/tap enabled in kernel? CONFIG_TUN that is. The module name "tun."
Command line should look like:

-device virtio-net,netdev=t10,mac=52:54:00:00:00:00 \ -netdev tap,id=t10,vhost=on,ifname=tap10,script=no,downscript=no

I have a tun setup. When I asked people on #kvm they said the fastest setup for inter-vm communication is tun + bridges (macvtap is a tad slower). Basicly it creates just another net interface (e.g. ifname=tap10) like ethernet or wireless. Once you get the new tap interface you have to configure it like any other net interface. Setup a bridge or NAT. For example, right after the qemu command line I have:

ifconfig tap10 up brctl addif br10 tap10

That bridge connects the vm to dhcp/router.

neona wrote:

Oh, and does anyone know if use of hugetlbfs is mutually exclusive with proper use of balloon memory? It seems likely.  Also, how much of/what sort of an improvement does hugetlbfs provide, anyway?  Does it just reduce memory/tlb thrashing - can this reduce stutter in games?

AFAIK, yes. You can't balloon hugetlbfs backed memory. As for improvement, for me hugetlbfs gives about 2 points in win7 benchmark. From 5.5 to about 7.3-7.4.

neona wrote:

Bit of stuttering in some games, but could be way worse.

You can try adding hv_relaxed,hv_vapic,hv_spinlocks=0x1000 to -cpu options (there is also hv_time, try it if you're running kernel 3.14, qemu-git). Might give you a couple more percents. All these options enable kvm-supported Hyper-V enlightenments which make windows behave better in virtual environment.

Offline

#1223 2014-02-26 01:44:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
aw wrote:

[I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

Which is the correct one: hv-time or hv_time? libvirt uses hv_time [1]. The long supported hv_relaxed, hv_vapic, hv_spinlocks all use undeline characters [2].

1. http://libvirt.org/git/?p=libvirt.git;a … d0;hb=HEAD
2. http://qemu-project.org/ChangeLog/1.1#x86

http://git.qemu.org/?p=qemu.git;a=commi … 9fd1933fbc

hv-time, but the others are also '-', so it's likely that either is accepted.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1224 2014-02-26 12:58:01

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi All,
firstly thanks for all the good information in this guide

I've been working through the instructions (as a new Arch user ie. n00b) and have hit a snag. I have complied the new kernel, qmeu and seabios as supplied by NBHS (thanks smile and compiled  pci-stub in and set the VFIO flags.
When running the first test of qemu I see this message "qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed."

After a bit of hunting around I think the code responsible can be found in this post http://lists.gnu.org/archive/html/qemu- … 01183.html ... which seems to continue some code going back at least to 2010 .... which prevents hotpluging of devices with a non-zero "function" ie. 01:00.0 is ok, but 01:00.1 fails
BUT
That seems unlikely since the example quoted  includes just such a device and apparently works ok.

Some other background -
Asrock z87 extreme6 motherboard with Intel 4670 CPU and 16GB of RAM
2 add-in cards (both PCIe) AMD HD6950 GPU  and Renesas USB3 card
Arch installed as of 25-Feb-14 ie. current with sofwtare downloaded from first page on the same day

Can anybody point me in the right direction ? I'd be grateful for any advice
Thanks

PS this motherboard works fine in Opensuse KVM using virt-manager (easy setup) with GPU card as secondary ... but latency is high and frame rate low, hence trying VFIO. Furthermore the Renesas USB card passes through nicely
The motherboard also works with XEN (with XM toolset using Powerhouse's instructions for Mint) but doesn't suit me because I can't get AMD's Catalyst Control Centre to install to support the triple screen (eyefinity) facility AND there's a problem with the Renesas USB card not resetting correctly which causes untold grief. The XL toolset allows the VM to start but there's a Windows device error on the GPU which prevents its use.

qemu command and error message

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu Haswell,hv-time \ > -smp 2,sockets=1,cores=2,threads=1 \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

lspci -nn

00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller [8086:0c09] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05) 00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04) 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05) 00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05) 00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05) 00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5) 00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5) 00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5) 00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05) 00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80] 02:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)

dmesg |grep pci-stub

 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=714ebc3d-5848-4d11-9329-39c5067a8917 rw intel_iommu=on pci-stub.ids=8086:0c01,8086:0c09,1002:6719,1002:aa80,1912:0014,8086:8c10,8086:8c14,8086:8c16,8086:8c18,1b21:0612,8086:1539 quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=714ebc3d-5848-4d11-9329-39c5067a8917 rw intel_iommu=on pci-stub.ids=8086:0c01,8086:0c09,1002:6719,1002:aa80,1912:0014,8086:8c10,8086:8c14,8086:8c16,8086:8c18,1b21:0612,8086:1539 quiet [ 0.332791] pci-stub: add 8086:0C01 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332794] pci-stub: add 8086:0C09 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332796] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332800] pci-stub 0000:01:00.0: claimed by stub [ 0.332801] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332804] pci-stub 0000:01:00.1: claimed by stub [ 0.332806] pci-stub: add 1912:0014 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332809] pci-stub 0000:02:00.0: claimed by stub [ 0.332810] pci-stub: add 8086:8C10 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332813] pci-stub: add 8086:8C14 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332816] pci-stub: add 8086:8C16 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332819] pci-stub: add 8086:8C18 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332822] pci-stub: add 1B21:0612 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332826] pci-stub 0000:04:00.0: claimed by stub [ 0.332828] pci-stub 0000:06:00.0: claimed by stub [ 0.332829] pci-stub: add 8086:1539 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.332833] pci-stub 0000:05:00.0: claimed by stub

dmesg |grep -e IOMMU -e DMAR

[ 0.000000] ACPI: DMAR 00000000bc173bd8 0000B8 (v01 INTEL HSW 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.022553] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.022557] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.022623] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.319935] DMAR: No ATSR found [ 0.319952] IOMMU 0 0xfed90000: using Queued invalidation [ 0.319952] IOMMU 1 0xfed91000: using Queued invalidation [ 0.319953] IOMMU: Setting RMRR: [ 0.319961] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.321223] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc027000 - 0xbc033fff] [ 0.321244] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc027000 - 0xbc033fff] [ 0.321261] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc027000 - 0xbc033fff] [ 0.321273] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.321279] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

IOMMU groups

(for i in /sys/kernel/iommu_groups/*;do echo "IOMMU: $i";for dev in $i/devices/*;do lspci -s "${dev##*/}"; done; echo;done)
IOMMU: /sys/kernel/iommu_groups/0 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) IOMMU: /sys/kernel/iommu_groups/1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] 02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) IOMMU: /sys/kernel/iommu_groups/10 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) IOMMU: /sys/kernel/iommu_groups/11 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) IOMMU: /sys/kernel/iommu_groups/2 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) IOMMU: /sys/kernel/iommu_groups/3 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) IOMMU: /sys/kernel/iommu_groups/4 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) IOMMU: /sys/kernel/iommu_groups/5 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) IOMMU: /sys/kernel/iommu_groups/6 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) IOMMU: /sys/kernel/iommu_groups/7 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) IOMMU: /sys/kernel/iommu_groups/8 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) IOMMU: /sys/kernel/iommu_groups/9 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) 04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

contents of vfio-pci.cfg

DEVICES="0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:04:00.0 0000:05:00.0 0000:06:00.0"

ie. the GPU, Renesas USB card, Asmedia Sata controllers and 2nd Intel NIC will be passed to the guest (this works nicely on OpenSuse via Virt-Manager ... hopefully even better with VFIO)

Offline

#1225 2014-02-26 15:26:13

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

redger wrote:

Can anybody point me in the right direction ? I'd be grateful for any advice
Thanks

Try it with qemu-1.7.0 stable. and then without "hv-time" because 1.7.0 doesn't know about hv-time yet.

aw posted a patch (Post #1217) which should eliminate your error - it did here; BUT my guest produces BSOD with current qemu git even with this patch - but works fine with qemu 1.7.0...

Last edited by kaeptnb (2014-02-26 15:26:56)

Offline

#1226 2014-02-26 17:06:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Borderlands folks, I think this is the problem you're trying to describe:

Bare Metal

VM

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

VM-new

This initial implementation is Intel-only, but hopefully we can do the same for AMD.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1227 2014-02-26 17:33:12

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks kaeptnb !  Reverting to to qemu-1.7.0 and Seabios-1.7.3.1 overcame that problem
then the following "Could not initialize SDL" error was addressed by setting the -nographic option

now the test passes (text appears on the GPU smile ) ..... though for some strange reason my mouse now insists on constantly moving rapidly left .... which is making life entertaining

thanks again

Offline

#1228 2014-02-26 22:01:15

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all (again ^^)

@aw

I saw a new set of patch for qemu to use Intel IGD as a passthrough device but only with Xen, do you think with this new information you can add Intel IGD to vfio ?

Offline

#1229 2014-02-26 22:11:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Hi all (again ^^)

@aw

I saw a new set of patch for qemu to use Intel IGD as a passthrough device but only with Xen, do you think with this new information you can add Intel IGD to vfio ?

There's already someone in the community working on IGD passthrough.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1230 2014-02-26 22:19:40

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Val532 wrote:

Hi all (again ^^)

@aw

I saw a new set of patch for qemu to use Intel IGD as a passthrough device but only with Xen, do you think with this new information you can add Intel IGD to vfio ?

There's already someone in the community working on IGD passthrough.

Really ? Where can i find some information about this ?

Offline

#1231 2014-02-26 22:23:11

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

neona wrote:

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.

I don't specify a networking switch and have synergy working just fine by giving it the host's IP address (192.168.0.X) to the Win7 guest.  Saying that, I'm having similar problems getting tap/tun working, but I'm still getting plenty good enough throughput on the guest.

Offline

#1232 2014-02-27 03:05:21

neona
Member
Registered: 2014-02-25
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuoni wrote:
neona wrote:

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.

I don't specify a networking switch and have synergy working just fine by giving it the host's IP address (192.168.0.X) to the Win7 guest.  Saying that, I'm having similar problems getting tap/tun working, but I'm still getting plenty good enough throughput on the guest.

The reason that synergy isn't working for me is that I'm trying to run the server on the guest (which is behind NAT by default) so I can usb passthrough my kb/m, both for ease of use and to expose all features on them (particularly a razer naga).

Of course, you've reminded me that the opposite setup is an option for the time being, so I should probably just do that until I can properly configure networking.  I'm considering just being lazy and sticking in an extra NIC to pass through though, that may offer better performance than any virtual option anyway.

Offline

#1233 2014-02-27 09:19:56

neona
Member
Registered: 2014-02-25
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

neona wrote:

For some reason, qemu only claims to support oss, wav, and spice output, so sound won't work.  I double checked the PKGBUILD for qemu, and it's clearly being compiled with pulse and alsa support (or should be, anyway), so I have no clue what's causing this, and some googling around hasn't found anything.  I've tried various ways of wrapping oss to pulse/alsa, and none of that has worked either.  This is rather inconvienient, but at least if I can get networking to behave then I think it's possible to have windows output to pulse if nothing else works.  I might also get a usb soundcard, but I'd rather avoid spending money if I can.

Turns out I'm an idiot, and installed a different qemu at some point that was earlier in my path than the one from qemu-git.  Cleaned that out and sound works fine now.

Offline

#1234 2014-02-27 09:47:27

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Before I spend hours trying ... is it possible to pass the intel onboard sound card to the guest with vfio?

Offline

#1235 2014-02-27 10:36:05

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

Before I spend hours trying ... is it possible to pass the intel onboard sound card to the guest with vfio?

On my motherboard "Asrock Z77 Professional Fatal1ty" worked without any problems.
The sound is like on the real hardware.

Offline

#1236 2014-02-27 19:33:33

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody have a working audio on HDMI for AMD card ?
For me, i have no audio on all of my VM.

Offline

#1237 2014-02-27 19:42:54

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Is anybody have a working audio on HDMI for AMD card ?
For me, i have no audio on all of my VM.

I actually used to have my sound setup like this "AMD guest gpu HDMI → Display → audio cable output on Display → line in on onboard sound", which worked pretty well.

As far as my understanding of the stuff goes the "soundcard" on the GPU is the .1 function in most cases, so you need to make sure that it's passed through. What GPU are you using and do you have any error messages besides no output? 

Since qemu is running as root and I'm using pulseaudio I'm now doing "remote" pulseaudio over localhost.


i'm sorry for my poor english wirting skills…

Offline

#1238 2014-02-27 20:04:51

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
Val532 wrote:

Is anybody have a working audio on HDMI for AMD card ?
For me, i have no audio on all of my VM.

I actually used to have my sound setup like this "AMD guest gpu HDMI → Display → audio cable output on Display → line in on onboard sound", which worked pretty well.

As far as my understanding of the stuff goes the "soundcard" on the GPU is the .1 function in most cases, so you need to make sure that it's passed through. What GPU are you using and do you have any error messages besides no output? 

Since qemu is running as root and I'm using pulseaudio I'm now doing "remote" pulseaudio over localhost.

So i start (one of) my VM with :

sudo qemu-system-x86_64 \ -M q35 -m 8G -enable-kvm -cpu host,hv-time -vga none -boot menu=on -no-fdc \ -smp 4,sockets=1,cores=4,threads=1 \ -device ioh3420,bus=pcie.0,id=root.1,port=1,chassis=1,addr=1c.0,multifunction=on \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/val/Image\ HDD/Windows-7-Test.img,id=disk,format=qcow2 \ -device ide-hd,bus=ahci.0,drive=disk \ -net nic,model=e1000 -net tap,ifname=tap0 \ -usb -usbdevice host:05e3:0607 -usbdevice host:1532:002f -usbdevice host:046d:c229 -usbdevice host:046d:c228

And my graphique card is an AMD HD5570.

I see an audio card in the device manager of Windows, but with error code 10.


About another subject, i try hv-time and i do not understand why but with -cpu host,hv-time does not work with Windows 8.1 Entreprise, but work with Windows 7 it's ok. It works with -cpu Haswell,hv-time for both.

Last edited by Val532 (2014-02-27 20:05:48)

Offline

#1239 2014-02-27 21:37:18

MichaelLong
Member
Registered: 2014-01-06
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:

Borderlands folks, I think this is the problem you're trying to describe:

https://dl.dropboxusercontent.com/u/198 … emetal.jpg

https://dl.dropboxusercontent.com/u/198 … ds2-vm.jpg

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

https://dl.dropboxusercontent.com/u/198 … vm-new.jpg

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

Great news. I applied those patches ontop of kernel 3.14-rc4 and I can report that they also fixed my performance problems in Battlefield 4! I couldn't do any intensive tests yet but after playing about 1h or so it is clear that the performance has nearly restored. Thx a lot.

Offline

#1240 2014-02-27 22:38:51

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

Offline

#1241 2014-02-27 23:01:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1242 2014-02-28 00:04:31

Kaidax
Member
Registered: 2014-02-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

https://dl.dropboxusercontent.com/u/198 … vm-new.jpg

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

That does the trick! Except I have to disable hv-time to play; enabling it drops performance to 15 fps, introduces severe graphical corruption, and makes journald dump gigabytes of logs.

Offline

#1243 2014-02-28 00:11:05

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Val532 wrote:

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

Yeah i know it seems not relevant to the thread, but if i want to try it, it because of GPU passtrough ^^.

I have a working setup with two GPU on the same VM and i want to try the remoteFX fonction ^^.

But could you explain to me how to enable nested fonction ? (i see nested:bool in kvm_intel module).

Thanks.

Offline

#1244 2014-02-28 00:17:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:
aw wrote:
Val532 wrote:

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

Yeah i know it seems not relevant to the thread, but if i want to try it, it because of GPU passtrough ^^.

I have a working setup with two GPU on the same VM and i want to try the remoteFX fonction ^^.

But could you explain to me how to enable nested fonction ? (i see nested:bool in kvm_intel module).

Thanks.

More than I know here - http://git.kernel.org/cgit/linux/kernel … ed-vmx.txt


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1245 2014-02-28 00:30:18

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Val532 wrote:
aw wrote:

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

Yeah i know it seems not relevant to the thread, but if i want to try it, it because of GPU passtrough ^^.

I have a working setup with two GPU on the same VM and i want to try the remoteFX fonction ^^.

But could you explain to me how to enable nested fonction ? (i see nested:bool in kvm_intel module).

Thanks.

More than I know here - http://git.kernel.org/cgit/linux/kernel … ed-vmx.txt

Ok so this is what i already do.

But Windows 8.1 knows it's runing on the top of an hypervisor, so i does not let me install Hyper-V.

Probably it's because of some modification to some Hyper-V change on qemu ?

Offline

#1246 2014-02-28 06:08:18

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sigh... my system was working fine with a R9 290X, even after the new ACS patch (no downstream). But now I'm getting:

[ 310.018977] dmar: DRHD: handling fault status reg 3 [ 310.033576] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.033576] DMAR:[fault reason 06] PTE Read access is not set [ 310.218996] dmar: DRHD: handling fault status reg 3 [ 310.233609] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.233609] DMAR:[fault reason 06] PTE Read access is not set [ 310.468989] dmar: DRHD: handling fault status reg 3 [ 310.483594] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.483594] DMAR:[fault reason 06] PTE Read access is not set

1:00.0 is my R9 290X. sad Screen goes crazy once the AMD drivers load.

lsgroup:

### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 9 ### 00:1c.1 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4) 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 04:02.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) ### Group 10 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 11 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) ### Group 12 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 13 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) ### Group 14 ### 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) ### Group 15 ### 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) ### Group 16 ### 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) ### Group 17 ### 07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

GRUB_CMDLINE_LINUX:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/disk/by-uuid/442a2a16-e934-48d6-a043-ff00534c5dfe:sys vfio_iommu_type1.allow_unsafe_interrupts=1 intel_iommu=on pci-stub.ids=1002:67b0,1002:aac8 kvm_intel.emulate_invalid_guest_state=0 nohz=off"

Am I screwed?

Offline

#1247 2014-02-28 10:44:24

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:

Sigh... my system was working fine with a R9 290X, even after the new ACS patch (no downstream). But now I'm getting:

[ 310.018977] dmar: DRHD: handling fault status reg 3 [ 310.033576] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.033576] DMAR:[fault reason 06] PTE Read access is not set [ 310.218996] dmar: DRHD: handling fault status reg 3 [ 310.233609] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.233609] DMAR:[fault reason 06] PTE Read access is not set [ 310.468989] dmar: DRHD: handling fault status reg 3 [ 310.483594] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.483594] DMAR:[fault reason 06] PTE Read access is not set

1:00.0 is my R9 290X. sad Screen goes crazy once the AMD drivers load.


Am I screwed?

This is not your primary GPU now, is it? These things may change depending on your computer BIOS setup and PCIe slot location it is attached to.

Basically what I learned from reading aw posts above is that if you use pci-stub on a GPU device (as you should) this will prevent device driver from attaching to it, but will not prevent vgacon from using the same device. And vgacon is not smart enough to care about hypervisor arbitration etc, it will simply corrupt your device state when the GPU gets attached to VM.

So the trick here is either to ensure that some other card is primary on boot, or let the smarter driver (rather than vgacon) use the device prior to VM startup, by not assigning it to the pci-stub module on kernel boot.

At least, that's my understanding, but I'm very much beginner in such setup

Last edited by Bronek (2014-02-28 10:45:41)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1248 2014-02-28 15:09:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
MacCoaster wrote:

Sigh... my system was working fine with a R9 290X, even after the new ACS patch (no downstream). But now I'm getting:

[ 310.018977] dmar: DRHD: handling fault status reg 3 [ 310.033576] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.033576] DMAR:[fault reason 06] PTE Read access is not set [ 310.218996] dmar: DRHD: handling fault status reg 3 [ 310.233609] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.233609] DMAR:[fault reason 06] PTE Read access is not set [ 310.468989] dmar: DRHD: handling fault status reg 3 [ 310.483594] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 [ 310.483594] DMAR:[fault reason 06] PTE Read access is not set

1:00.0 is my R9 290X. sad Screen goes crazy once the AMD drivers load.


Am I screwed?

This is not your primary GPU now, is it? These things may change depending on your computer BIOS setup and PCIe slot location it is attached to.

Basically what I learned from reading aw posts above is that if you use pci-stub on a GPU device (as you should) this will prevent device driver from attaching to it, but will not prevent vgacon from using the same device. And vgacon is not smart enough to care about hypervisor arbitration etc, it will simply corrupt your device state when the GPU gets attached to VM.

So the trick here is either to ensure that some other card is primary on boot, or let the smarter driver (rather than vgacon) use the device prior to VM startup, by not assigning it to the pci-stub module on kernel boot.

At least, that's my understanding, but I'm very much beginner in such setup

That's true, but I'm not sure it would cause this problem.  The VGA region is accessed by the CPU and therefore not subject to IOMMU translation.  The IOMMU only handles accesses initiated by the device.  So if vgacon is being used we'll get corrupted screens, as you indicate, but we won't get DMAR faults.

The DMAR faults indicate they are blocking the 8970 from reading memory just below 128GB.  It seems like a safe assumption that you don't have a 128GB VM, so the IOMMU is probably doing it's job by preventing these.  Does the card still work on bare metal?  It is possible that a failed memory controller on the card could be generating these accesses.  Does the address remotely resemble the host or guest address of the device?  (lspci in the host, info pci/info mtree in the qemu monitor)  If you haven't already, a hard power cycle might be a good idea.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1249 2014-02-28 17:23:52

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

The nvidia guys seem to have a problem with their setup: https://devtalk.nvidia.com/default/topi … 4/#4131164

We have a number on their internal bugtracker, so maybe we should try helping them with their issues, so they can reproduce it and fix it at some point.

I've told the guy at Nvidia already in a private message that I will follow-up next week as I was on a conference.

Offline

#1250 2014-02-28 22:15:41

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am new to passthrough with KVM and have read this thread from beginning to end with the intention of setting up a similar system but I have a some basic / "superficial" questions before I begin.

#1: How is the Windows guest visualized (what/where is its display)?
My only experience with "graphical access" to KVM guests is via VNC, but that will not do it for gaming. So how is the guest visualized / what is its display? As far as I can understand from the thread people simply have a monitor connected directly to the graphics card which is passed through to the guest, is this understood correctly? This directly connected monitor can then have multiple inputs which can be changed on the monitor itself or one could use a display switch box.

#2: Any alternatives to #1?
In the thread I see a couple of discussions about copying the frame buffer of one graphics card to another, but all of them end with the poster saying it is just speculation and that it should be theoretically possible. Anyone who have succeeded with getting the video output from the guest into a window on the host via whatever method?

#3: Are two powerful graphics cards needed?
Quite a few posters have two quite powerful graphics cards installed. Isn't one powerful graphics card enough (one to pass to the guest used for playing games) and a less powerful one for running the host displays?

#4: Is a dedicated sound card needed?
Do I need a dedicated sound card or can the host and guest share the onboard sound card?

#5: Requirements for CPU and mother board?
Is it (more or less) sufficient to select a CPU with Intel VT-d or AMD-Vi, and a matching mother board (supporting the CPU and Intel VT-d / AMD-Vi)?

Offline

#1251 2014-02-28 22:37:00

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jonascj wrote:

#1: How is the Windows guest visualized (what/where is its display)?
My only experience with "graphical access" to KVM guests is via VNC, but that will not do it for gaming. So how is the guest visualized / what is its display? As far as I can understand from the thread people simply have a monitor connected directly to the graphics card which is passed through to the guest, is this understood correctly?

Correct.

jonascj wrote:

This directly connected monitor can then have multiple inputs which can be changed on the monitor itself or one could use a display switch box.

I lost you there ... could you clarify?

jonascj wrote:

#2: Any alternatives to #1?
In the thread I see a couple of discussions about copying the frame buffer of one graphics card to another, but all of them end with the poster saying it is just speculation and that it should be theoretically possible. Anyone who have succeeded with getting the video output from the guest into a window on the host via whatever method?

Afaik there are no generic solutions. I use steam in-house streaming every now and then, but it only works for some games and is not very fast.

jonascj wrote:

#3: Are two powerful graphics cards needed?
Quite a few posters have two quite powerful graphics cards installed. Isn't one powerful graphics card enough (one to pass to the guest used for playing games) and a less powerful one for running the host displays?

One is enough. I use the integrated Intel GPU for the host and a powerful nvidia card for the guest.

jonascj wrote:

#4: Is a dedicated sound card needed?
Do I need a dedicated sound card or can the host and guest share the onboard sound card?

You can share your onboard sound card. Either with passthrough (and thus only usable by the guest) or shared between host and guest.

jonascj wrote:

#5: Requirements for CPU and mother board?
Is it (more or less) sufficient to select a CPU with Intel VT-d or AMD-Vi, and a matching mother board (supporting the CPU and Intel VT-d / AMD-Vi)?

In theory yes. In practice, there seem to be some mainboards that work better than others. see https://docs.google.com/spreadsheet/ccc … _web#gid=0

Last edited by Flyser (2014-02-28 22:39:31)

Offline

#1252 2014-02-28 23:10:26

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

This is not your primary GPU now, is it? These things may change depending on your computer BIOS setup and PCIe slot location it is attached to.

Nope. Never was my primary. Intel IGP is my primary, used for Linux console.

aw wrote:

The DMAR faults indicate they are blocking the 8970 from reading memory just below 128GB.  It seems like a safe assumption that you don't have a 128GB VM, so the IOMMU is probably doing it's job by preventing these.  Does the card still work on bare metal?  It is possible that a failed memory controller on the card could be generating these accesses.  Does the address remotely resemble the host or guest address of the device?  (lspci in the host, info pci/info mtree in the qemu monitor)  If you haven't already, a hard power cycle might be a good idea.

I haven't tested bare metal, will do. Hard power cycles do nothing. sad I'll get the lspci (what do you need specifically?, -vv?) and info pci/mtree.

Thanks,

Will

Offline

#1253 2014-03-01 03:17:27

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
doubledr wrote:

Hi, aw

Is there any tutorial about how to use these tools? I want to help this topic too.

You'll find kvm_stat in the qemu source tree, here are some instructions:

https://docs.fedoraproject.org/en-US/Fe … 20s02.html

Tracing is similarly easy, but the report generated is massive and takes some skill to analyze:

http://www.linux-kvm.org/page/Tracing

Generally you'll want to run kvm_stat first to see what the heavy hitters are, then look in the trace output to try to figure out why they're being called.

There's a bit on perf here too:

http://www.linux-kvm.org/page/Perf_events

A simple place to start, especially if you're compiling your own qemu is to uncomment #define DEBUG_VFIO from hw/misc/vfio.c, recompile and test with that binary.  You will get a ton of initial output but once the OS drivers are running it should slow down considerably.  The output will never cease entirely though, all config space access will go through qemu and any of the quirks.  So long as you don't see more than around a dozen lines per second, vfio is probably not the bottleneck.

Thx, aw

I took several days to read throught these documents. But I think I am missing something. kvm_stat displays two columns of number, but what do they mean? The first document doesn't explain it. And I don't know what is the typical value for each counter... So when I type kvm_stat, I have no idea which counter is "abnormal". I think the knowledge gap is pretty large here... sad

Offline

#1254 2014-03-01 09:45:11

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
jonascj wrote:

This directly connected monitor can then have multiple inputs which can be changed on the monitor itself or one could use a display switch box.

I lost you there ... could you clarify?

I just meant that most people are looking for the guest to feel like it is just another application window on the host machine. They are trying to get as far away from dual boot as possible. So most people will probably choose a monitor with two inputs, one which can be connected to the host and one to the guest. That way they do not have a monitor whose sole purpose is to display the output of the guest; by selecting source on the monitor they can turn it into a secondary monitor for the host. Or if their monitor do not have two inputs maybe they invest in an vga/dvi switch which will let them switch between the input of the guest and host.

Flyser wrote:
jonascj wrote:

jonascj wrote:
#2: Any alternatives to #1?
In the thread I see a couple of discussions about copying the frame buffer of one graphics card to another, but all of them end with the poster saying it is just speculation and that it should be theoretically possible. Anyone who have succeeded with getting the video output from the guest into a window on the host via whatever method?

Afaik there are no generic solutions. I use steam in-house streaming every now and then, but it only works for some games and is not very fast.

I have a Thinkpad T420 laptop with Nvidia optimus technology (not intended to be used for this setup, just some perspective). There is a heap of trouble with having output on the display port and powersaving from the Optimus technology. One solution to get output on the display port is to "setup a virtual display on the Intel GPU (so it now have the internal display and one virtual), render all output on the Intel GPU and then copy the contents of the virtual display to the Nvidia GPU for output via the display port" (http://zachstechnotes.blogspot.co.at/2012/04/post-title.html , https://github.com/liskin/hybrid-screenclone).

Of course the Nvidia GPU and Intel GPU of the T420 is "connected" through the Optimus technology, but if it is possible to copy display content between them in an Optimus setup maybe setups exist for desktop computers which would allow the same thing. Of course this would not be a generic solution (like you said: no generic solution exist).

Offline

#1255 2014-03-01 13:20:53

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

should i expect vga passthrough to work out of box with kernel 3.14 and qemu 2.0 or xen 4.4?

i have a 2x geforce 770 cards intel 4670 and asrock z87 extreme4

i'm new to linux and the closest i have got to success is with xenserver 6.2 and esxi 5.5 i get yellow triangle in device manager with error code 43 under windows 8.1 guest.
i did this guide step by step in arch linux with gnome and linux mint 16, and both times virt-manager window doesn't respond after starting guest with vga passthrough

Offline

#1256 2014-03-01 17:46:05

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

Patches were just posted upstream to do lazy tracking of the debug registers.

Presumably you have to be running 3.14-rc4 or linux-next for those patches to apply cleanly?  I tried them on 3.13.2, and most applied, with a few failures I was able to fix manually, but I haven't tried compiling a kernel with them yet...  [EDIT] and that wouldn't work anyway.

Anyway, I'm not sure whether they apply to me in any case, as I don't have a Haswell processor, I have an IvyBridge. (4930K) 

Also, Is that hv-time option relevant at all? I currently use -cpu host.

Last edited by mostlyharmless (2014-03-01 22:16:06)

Offline

#1257 2014-03-01 21:59:49

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@neona

Turns out I'm an idiot, and installed a different qemu at some point that was earlier in my path than the one from qemu-git.  Cleaned that out and sound works fine now.

Hey, thanks, turns out I'm the same sub-species of idiot; now my sound is working too!

Offline

#1258 2014-03-01 22:49:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

@aw

Patches were just posted upstream to do lazy tracking of the debug registers.

Presumably you have to be running 3.14-rc4 or linux-next for those patches to apply cleanly?  I tried them on 3.13.2, and most applied, with a few failures I was able to fix manually, but I haven't tried compiling a kernel with them yet...  [EDIT] and that wouldn't work anyway.

Anyway, I'm not sure whether they apply to me in any case, as I don't have a Haswell processor, I have an IvyBridge. (4930K) 

Also, Is that hv-time option relevant at all? I currently use -cpu host.

It should help all Intel processors when the guest makes use of debug registers, it's not Haswell specific.  hv-time is a Hyper-V enlightened time source which is more efficient than other emulated time sources.  I saw an improvement of several FPS when using hv-time.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1259 2014-03-02 13:43:36

arny91
Member
Registered: 2014-01-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone

I installed every package suggested in the 1st post by nbhs in order to get qemu working with vfio-pci option
and edited every config file.
Till now i compiled the kernel twice, the second time i did i took care that the following options were selected
as builtin and not module:

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

Vt-d is enabled in the efi setup.

This is my configuration:

MB:MSI Z77A-GD80
CPU: Intel Core I7-2600
RAM:16GB ddr3 1866
DISK:128GB SSD Sata 3
Primary GPU: Intel® HD Graphics 2000
Secondary GPU: Nvidia GTX 280 <-- Windows display

But when i try to launch qemu (after device passthrough with vfio-bind) i get this error:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

this is the output of

dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 000000009947a598 0000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.065036] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.065041] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.065112] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1

lspci:

lspci 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation GT200b [GeForce GTX 285] (rev a1) 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

here's the full dmesg
http://pastebin.com/SZ9aMVhG

I also added intel_iommu=on, igfx=on, but this changed nothing.

Is there something else to adjust?
Thanks

Last edited by arny91 (2014-03-02 13:44:17)

Offline

#1260 2014-03-02 14:22:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

arny91 wrote:

Hi everyone

here's the full dmesg
http://pastebin.com/SZ9aMVhG

I also added intel_iommu=on, igfx=on, but this changed nothing.

You have:

[ 0.000000] Command line: initrd=\initramfs-linux-mainline.img root=/dev/sda2 rw,intel_iommu=on,igfx_on

Change the comma between "rw" and "intel_iommu" to a space.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1261 2014-03-02 22:46:57

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The DMAR faults indicate they are blocking the 8970 from reading memory just below 128GB.  It seems like a safe assumption that you don't have a 128GB VM, so the IOMMU is probably doing it's job by preventing these.  Does the card still work on bare metal?  It is possible that a failed memory controller on the card could be generating these accesses.  Does the address remotely resemble the host or guest address of the device?  (lspci in the host, info pci/info mtree in the qemu monitor)  If you haven't already, a hard power cycle might be a good idea.

Alright, finally had the time to get around to it.

lspci -vvv

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: e0000000-f08fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: ASUSTeK Computer Inc. Device 8534 Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <8us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Via WAKE# ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Capabilities: [d94 v1] #19 Kernel driver in use: pcieport Kernel modules: shpchp 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 43 Region 0: Memory at f0c00000 (64-bit, non-prefetchable) [size=4M] Region 2: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at f000 [disabled] [size=64] Expansion ROM at <unassigned> [disabled] Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00018 Data: 0000 Capabilities: [d0] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a4] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: i915 Kernel modules: i915 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 48 Region 0: Memory at f1414000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee003f8 Data: 0000 Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- RBE- FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) (prog-if 30 [XHCI]) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 45 Region 0: Memory at f1400000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Address: 00000000fee00398 Data: 0000 Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 47 Region 0: Memory at f141f000 (64-bit, non-prefetchable) [size=16] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee003d8 Data: 0000 Kernel driver in use: mei_me Kernel modules: mei_me 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) (prog-if 20 [EHCI]) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 20 Region 0: Memory at f141c000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) Subsystem: ASUSTeK Computer Inc. Device 8573 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 49 Region 0: Memory at f1410000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00438 Data: 0000 Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE- FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=2 ArbSelect=Fixed TC/VC=04 Status: NegoPending- InProgress- Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 00002000-00002fff Memory behind bridge: cf200000-cf3fffff Prefetchable memory behind bridge: 00000000cf400000-00000000cf5fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train+ SlotClk+ DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Slot #0, PowerLimit 0.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8534 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.1 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=03, subordinate=04, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f1300000-f13fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #2, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8534 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: f1200000-f12fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #2, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8534 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 0000b000-0000bfff Memory behind bridge: f1100000-f11fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #3, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8534 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: cf600000-cfafffff Prefetchable memory behind bridge: 00000000cfb00000-00000000cfbfffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #5, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <512ns, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #4, PowerLimit 25.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Device 8534 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) (prog-if 20 [EHCI]) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 23 Region 0: Memory at f141b000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 04) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0]) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin B routed to IRQ 46 Region 0: I/O ports at f0b0 [size=8] Region 1: I/O ports at f0a0 [size=4] Region 2: I/O ports at f090 [size=8] Region 3: I/O ports at f080 [size=4] Region 4: I/O ports at f060 [size=32] Region 5: Memory at f141a000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee003b8 Data: 0000 Capabilities: [70] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004 Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) Subsystem: ASUSTeK Computer Inc. Device 8534 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 18 Region 0: Memory at f1419000 (64-bit, non-prefetchable) [size=256] Region 4: I/O ports at f040 [size=32] Kernel modules: i2c_i801 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Device e285 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at f0000000 (64-bit, prefetchable) [size=8M] Region 4: I/O ports at e000 [size=256] Region 5: Memory at f0800000 (32-bit, non-prefetchable) [size=256K] Expansion ROM at f0840000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 00000000fee00458 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 Subsystem: PC Partner Limited / Sapphire Technology Device aac8 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 17 Region 0: Memory at f0860000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=03, secondary=04, subordinate=04, sec-latency=32 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f1300000-f13fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [c0] Subsystem: ASUSTeK Computer Inc. Device 8489 04:02.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) (prog-if 10 [OHCI]) Subsystem: ASUSTeK Computer Inc. Device 82eb Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 32 (8000ns max), Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 19 Region 0: Memory at f1300000 (32-bit, non-prefetchable) [size=2K] Region 1: I/O ports at d000 [size=128] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: firewire_ohci Kernel modules: firewire_ohci 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) Subsystem: ASUSTeK Computer Inc. Device 8557 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 18 Region 0: Memory at f1200000 (32-bit, non-prefetchable) [size=512K] Region 2: I/O ports at c000 [size=32] Region 3: Memory at f1280000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 74-d0-2b-ff-ff-97-9d-0c Capabilities: [1a0 v1] Transaction Processing Hints Device specific mode supported Steering table in TPH capability structure Kernel driver in use: igb Kernel modules: igb 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) Subsystem: ASUSTeK Computer Inc. Device 8557 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 21 Region 0: Memory at f1100000 (32-bit, non-prefetchable) [size=512K] Region 2: I/O ports at b000 [size=32] Region 3: Memory at f1180000 (32-bit, non-prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <2us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 74-d0-2b-ff-ff-97-9d-0d Capabilities: [1a0 v1] Transaction Processing Hints Device specific mode supported Steering table in TPH capability structure Kernel driver in use: igb Kernel modules: igb 07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) Subsystem: LSI Logic / Symbios Logic Device 3020 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: I/O ports at a000 [size=256] Region 1: Memory at cfa40000 (64-bit, non-prefetchable) [size=16K] Region 3: Memory at cf600000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at cfb00000 [disabled] [size=512K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [d0] Vital Product Data Unknown small resource type 00, will not decode more. Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=15 Masked- Vector table: BAR=1 offset=00002000 PBA: BAR=1 offset=00003800 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- IOVSta: Migration- Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00 VF offset: 1, stride: 1, Device ID: 0072 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000cfa44000 (64-bit, non-prefetchable) Region 2: Memory at 00000000cf640000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt2sas Kernel modules: mpt2sas

info mtree and pci

INFO MTREE: memory 0000000000000000-ffffffffffffffff (prio 0, RW): system 0000000000000000-00000000afffffff (prio 0, RW): alias ram-below-4g @pc.ram 0000000000000000-00000000afffffff 0000000000000000-ffffffffffffffff (prio -1, RW): pci 00000000000a0000-00000000000bffff (prio 1, RW): alias pci_bridge_vga_mem @pci_bridge_pci 00000000000a0000-00000000000bffff 00000000000c0000-00000000000dffff (prio 1, RW): pc.rom 00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios @pc.bios 0000000000000000-000000000001ffff 00000000d0000000-00000000efffffff (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000d0000000-00000000efffffff 00000000f0000000-00000000f0000000 (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000f0000000-00000000f0000000 00000000fe400000-00000000fe7fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe400000-00000000fe7fffff 00000000fe800000-00000000fe9fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe800000-00000000fe9fffff 00000000fea00000-00000000fea03fff (prio 1, RW): VFIO 0000:01:00.1 BAR 0 00000000fea00000-00000000fea03fff (prio 0, RW): VFIO 0000:01:00.1 BAR 0 mmap 00000000fea04000-00000000fea04fff (prio 1, RW): ahci 00000000fffe0000-00000000ffffffff (prio 0, R-): pc.bios 00000000000a0000-00000000000bffff (prio 1, RW): alias smram-region @pci 00000000000a0000-00000000000bffff 00000000000c0000-00000000000c3fff (prio 1, R-): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff 00000000000c4000-00000000000c7fff (prio 1, R-): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff 00000000000c8000-00000000000cbfff (prio 1, R-): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff 00000000000cc000-00000000000cffff (prio 1, R-): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff 00000000000d0000-00000000000d3fff (prio 1, R-): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff 00000000000d1000-00000000000d3fff (prio 1000, RW): alias kvmvapic-rom @pc.ram 00000000000d1000-00000000000d3fff 00000000000d4000-00000000000d7fff (prio 1, R-): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff 00000000000d8000-00000000000dbfff (prio 1, R-): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff 00000000000dc000-00000000000dffff (prio 1, R-): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff 00000000000e0000-00000000000e3fff (prio 1, R-): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff 00000000000e4000-00000000000e7fff (prio 1, R-): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff 00000000000e8000-00000000000ebfff (prio 1, R-): alias pam-rom @pc.ram 00000000000e8000-00000000000ebfff 00000000000ec000-00000000000effff (prio 1, RW): alias pam-ram @pc.ram 00000000000ec000-00000000000effff 00000000000f0000-00000000000fffff (prio 1, R-): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff 00000000b0000000-00000000bfffffff (prio 0, RW): pcie-mmcfg 00000000fec00000-00000000fec00fff (prio 0, RW): kvm-ioapic 00000000fee00000-00000000feefffff (prio 4096, RW): icc-apic-container 00000000fee00000-00000000feefffff (prio 0, RW): kvm-apic-msi 0000000100000000-000000064fffffff (prio 0, RW): alias ram-above-4g @pc.ram 00000000b0000000-00000005ffffffff I/O 0000000000000000-000000000000ffff (prio 0, RW): io 0000000000000000-0000000000000007 (prio 0, RW): dma-chan 0000000000000008-000000000000000f (prio 0, RW): dma-cont 0000000000000020-0000000000000021 (prio 0, RW): kvm-pic 0000000000000040-0000000000000043 (prio 0, RW): kvm-pit 0000000000000060-0000000000000060 (prio 0, RW): i8042-data 0000000000000061-0000000000000061 (prio 0, RW): elcr 0000000000000064-0000000000000064 (prio 0, RW): i8042-cmd 0000000000000070-0000000000000071 (prio 0, RW): rtc 000000000000007e-000000000000007f (prio 0, RW): kvmvapic 0000000000000080-0000000000000080 (prio 0, RW): ioport80 0000000000000081-0000000000000083 (prio 0, RW): dma-page 0000000000000087-0000000000000087 (prio 0, RW): dma-page 0000000000000089-000000000000008b (prio 0, RW): dma-page 000000000000008f-000000000000008f (prio 0, RW): dma-page 0000000000000092-0000000000000092 (prio 0, RW): port92 00000000000000a0-00000000000000a1 (prio 0, RW): kvm-pic 00000000000000b2-00000000000000b3 (prio 0, RW): apm-io 00000000000000c0-00000000000000cf (prio 0, RW): dma-chan 00000000000000d0-00000000000000df (prio 0, RW): dma-cont 00000000000000f0-00000000000000f0 (prio 0, RW): ioportF0 00000000000003b0-00000000000003bb (prio 1, RW): alias pci_bridge_vga_io_lo @pci_bridge_io 00000000000003b0-00000000000003bb 00000000000003c0-00000000000003df (prio 1, RW): alias pci_bridge_vga_io_hi @pci_bridge_io 00000000000003c0-00000000000003df 00000000000003f1-00000000000003f5 (prio 0, RW): fdc 00000000000003f7-00000000000003f7 (prio 0, RW): fdc 00000000000004d0-00000000000004d0 (prio 0, RW): kvm-elcr 00000000000004d1-00000000000004d1 (prio 0, RW): kvm-elcr 0000000000000510-0000000000000511 (prio 0, RW): fwcfg 0000000000000cf8-0000000000000cfb (prio 0, RW): pci-conf-idx 0000000000000cf9-0000000000000cf9 (prio 1, RW): lpc-reset-control 0000000000000cfc-0000000000000cff (prio 0, RW): pci-conf-data 0000000000005658-0000000000005658 (prio 0, RW): vmport 000000000000b000-000000000000b07f (prio 0, RW): ich9-pm 000000000000b000-000000000000b003 (prio 0, RW): acpi-evt 000000000000b004-000000000000b005 (prio 0, RW): acpi-cnt 000000000000b008-000000000000b00b (prio 0, RW): acpi-tmr 000000000000b020-000000000000b02f (prio 0, RW): apci-gpe0 000000000000b030-000000000000b037 (prio 0, RW): apci-smi 000000000000b100-000000000000b13f (prio 1, RW): pm-smbus 000000000000c000-000000000000cfff (prio 1, RW): alias pci_bridge_io @pci_bridge_io 000000000000c000-000000000000cfff 000000000000d000-000000000000dfff (prio 1, RW): alias pci_bridge_io @pci_bridge_io 000000000000d000-000000000000dfff 000000000000e040-000000000000e05f (prio 1, RW): ahci-idp mch ICH9 LPC ich9-ahci 0000000000000000-ffffffffffffffff (prio 0, RW): alias bus master @system 0000000000000000-ffffffffffffffff ICH9 SMB i82801b11-bridge pci-bridge ich9-usb-ehci1 0000000000000000-ffffffffffffffff (prio 0, RW): alias bus master @system 0000000000000000-ffffffffffffffff ich9-usb-uhci1 0000000000000000-ffffffffffffffff (prio 0, RW): alias bus master @system 0000000000000000-ffffffffffffffff ich9-usb-uhci2 0000000000000000-ffffffffffffffff (prio 0, RW): alias bus master @system 0000000000000000-ffffffffffffffff ich9-usb-uhci3 0000000000000000-ffffffffffffffff (prio 0, RW): alias bus master @system 0000000000000000-ffffffffffffffff virtio-blk-pci virtio-net-pci virtio-balloon-pci ioh3420 vfio-pci vfio-pci aliases pc.ram 0000000000000000-00000005ffffffff (prio 0, RW): pc.ram pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, RW): pci_bridge_pci 00000000000a0000-00000000000bffff (prio 1, RW): vfio-vga-mmio@0xa0000 00000000d0000000-00000000dfffffff (prio 1, RW): VFIO 0000:01:00.0 BAR 0 00000000d0000000-00000000dfffffff (prio 0, RW): VFIO 0000:01:00.0 BAR 0 mmap 00000000e0000000-00000000e07fffff (prio 1, RW): VFIO 0000:01:00.0 BAR 2 00000000e0000000-00000000e07fffff (prio 0, RW): VFIO 0000:01:00.0 BAR 2 mmap 00000000e0004000-00000000e0004fff (prio 1, RW): vfio-ati-bar2-4000-quirk 00000000fe800000-00000000fe83ffff (prio 1, RW): VFIO 0000:01:00.0 BAR 5 00000000fe800000-00000000fe83ffff (prio 0, RW): VFIO 0000:01:00.0 BAR 5 mmap pc.bios 00000000fffe0000-00000000ffffffff (prio 0, R-): pc.bios pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, RW): pci_bridge_pci 00000000f0000000-00000000f0000000 (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000f0000000-00000000f0000000 00000000fe400000-00000000fe5fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe400000-00000000fe5fffff 00000000fe600000-00000000fe6000ff (prio 1, RW): shpc-bar 00000000fe600000-00000000fe60009f (prio 0, RW): shpc-mmio pci 0000000000000000-ffffffffffffffff (prio -1, RW): pci 00000000000a0000-00000000000bffff (prio 1, RW): alias pci_bridge_vga_mem @pci_bridge_pci 00000000000a0000-00000000000bffff 00000000000c0000-00000000000dffff (prio 1, RW): pc.rom 00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios @pc.bios 0000000000000000-000000000001ffff 00000000d0000000-00000000efffffff (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000d0000000-00000000efffffff 00000000f0000000-00000000f0000000 (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000f0000000-00000000f0000000 00000000fe400000-00000000fe7fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe400000-00000000fe7fffff 00000000fe800000-00000000fe9fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe800000-00000000fe9fffff 00000000fea00000-00000000fea03fff (prio 1, RW): VFIO 0000:01:00.1 BAR 0 00000000fea00000-00000000fea03fff (prio 0, RW): VFIO 0000:01:00.1 BAR 0 mmap 00000000fea04000-00000000fea04fff (prio 1, RW): ahci 00000000fffe0000-00000000ffffffff (prio 0, R-): pc.bios pci_bridge_io 0000000000000000-000000000000ffff (prio 0, RW): pci_bridge_io 00000000000003b0-00000000000003bb (prio 1, RW): vfio-vga-io@0x3b0 00000000000003c0-00000000000003df (prio 1, RW): vfio-vga-io@0x3c0 00000000000003c3-00000000000003c3 (prio 0, RW): vfio-ati-3c3-quirk 000000000000d000-000000000000d0ff (prio 1, RW): VFIO 0000:01:00.0 BAR 4 000000000000d000-000000000000d007 (prio 1, RW): vfio-ati-bar4-window-quirk 000000000000d000-000000000000d000 (prio 0, RW): VFIO 0000:01:00.0 BAR 4 mmap pci_bridge_io 0000000000000000-000000000000ffff (prio 0, RW): pci_bridge_io 000000000000c000-000000000000cfff (prio 1, RW): alias pci_bridge_io @pci_bridge_io 000000000000c000-000000000000cfff system 0000000000000000-ffffffffffffffff (prio 0, RW): system 0000000000000000-00000000afffffff (prio 0, RW): alias ram-below-4g @pc.ram 0000000000000000-00000000afffffff 0000000000000000-ffffffffffffffff (prio -1, RW): pci 00000000000a0000-00000000000bffff (prio 1, RW): alias pci_bridge_vga_mem @pci_bridge_pci 00000000000a0000-00000000000bffff 00000000000c0000-00000000000dffff (prio 1, RW): pc.rom 00000000000e0000-00000000000fffff (prio 1, R-): alias isa-bios @pc.bios 0000000000000000-000000000001ffff 00000000d0000000-00000000efffffff (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000d0000000-00000000efffffff 00000000f0000000-00000000f0000000 (prio 1, RW): alias pci_bridge_pref_mem @pci_bridge_pci 00000000f0000000-00000000f0000000 00000000fe400000-00000000fe7fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe400000-00000000fe7fffff 00000000fe800000-00000000fe9fffff (prio 1, RW): alias pci_bridge_mem @pci_bridge_pci 00000000fe800000-00000000fe9fffff 00000000fea00000-00000000fea03fff (prio 1, RW): VFIO 0000:01:00.1 BAR 0 00000000fea00000-00000000fea03fff (prio 0, RW): VFIO 0000:01:00.1 BAR 0 mmap 00000000fea04000-00000000fea04fff (prio 1, RW): ahci 00000000fffe0000-00000000ffffffff (prio 0, R-): pc.bios 00000000000a0000-00000000000bffff (prio 1, RW): alias smram-region @pci 00000000000a0000-00000000000bffff 00000000000c0000-00000000000c3fff (prio 1, R-): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff 00000000000c4000-00000000000c7fff (prio 1, R-): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff 00000000000c8000-00000000000cbfff (prio 1, R-): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff 00000000000cc000-00000000000cffff (prio 1, R-): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff 00000000000d0000-00000000000d3fff (prio 1, R-): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff 00000000000d1000-00000000000d3fff (prio 1000, RW): alias kvmvapic-rom @pc.ram 00000000000d1000-00000000000d3fff 00000000000d4000-00000000000d7fff (prio 1, R-): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff 00000000000d8000-00000000000dbfff (prio 1, R-): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff 00000000000dc000-00000000000dffff (prio 1, R-): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff 00000000000e0000-00000000000e3fff (prio 1, R-): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff 00000000000e4000-00000000000e7fff (prio 1, R-): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff 00000000000e8000-00000000000ebfff (prio 1, R-): alias pam-rom @pc.ram 00000000000e8000-00000000000ebfff 00000000000ec000-00000000000effff (prio 1, RW): alias pam-ram @pc.ram 00000000000ec000-00000000000effff 00000000000f0000-00000000000fffff (prio 1, R-): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff 00000000b0000000-00000000bfffffff (prio 0, RW): pcie-mmcfg 00000000fec00000-00000000fec00fff (prio 0, RW): kvm-ioapic 00000000fee00000-00000000feefffff (prio 4096, RW): icc-apic-container 00000000fee00000-00000000feefffff (prio 0, RW): kvm-apic-msi 0000000100000000-000000064fffffff (prio 0, RW): alias ram-above-4g @pc.ram 00000000b0000000-00000005ffffffff pci_bridge_pci 0000000000000000-ffffffffffffffff (prio 0, RW): pci_bridge_pci 00000000fe440000-00000000fe440fff (prio 1, RW): virtio-net-pci-msix 00000000fe440000-00000000fe44002f (prio 0, RW): msix-table 00000000fe440800-00000000fe440807 (prio 0, RW): msix-pba 00000000fe441000-00000000fe441fff (prio 1, RW): ehci 00000000fe441000-00000000fe44100f (prio 0, RW): capabilities 00000000fe441020-00000000fe441063 (prio 0, RW): operational 00000000fe441064-00000000fe44107b (prio 0, RW): ports 00000000fe442000-00000000fe442fff (prio 1, RW): virtio-blk-pci-msix 00000000fe442000-00000000fe44201f (prio 0, RW): msix-table 00000000fe442800-00000000fe442807 (prio 0, RW): msix-pba pci_bridge_io 0000000000000000-000000000000ffff (prio 0, RW): pci_bridge_io 000000000000c000-000000000000c03f (prio 1, RW): virtio-pci 000000000000c040-000000000000c05f (prio 1, RW): virtio-pci 000000000000c060-000000000000c07f (prio 1, RW): uhci 000000000000c080-000000000000c09f (prio 1, RW): uhci 000000000000c0a0-000000000000c0bf (prio 1, RW): uhci 000000000000c0c0-000000000000c0df (prio 1, RW): virtio-pci INFO PCI: Bus 0, device 0, function 0: Host bridge: PCI device 8086:29c0 id "" Bus 0, device 1, function 0: Audio controller: PCI device 1002:aac8 IRQ 11. BAR0: 64 bit memory at 0xfea00000 [0xfea03fff]. id "" Bus 0, device 28, function 0: PCI bridge: PCI device 8086:3420 BUS 0. secondary bus 1. subordinate bus 1. IO range [0xd000, 0xdfff] memory range [0xfe800000, 0xfe9fffff] prefetchable memory range [0xd0000000, 0xefffffff] id "root" Bus 1, device 0, function 0: VGA controller: PCI device 1002:67b0 IRQ 10. BAR0: 64 bit prefetchable memory at 0xd0000000 [0xdfffffff]. BAR2: 64 bit prefetchable memory at 0xe0000000 [0xe07fffff]. BAR4: I/O at 0xd000 [0xd0ff]. BAR5: 32 bit memory at 0xfe800000 [0xfe83ffff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe]. id "" Bus 0, device 30, function 0: PCI bridge: PCI device 8086:244e BUS 0. secondary bus 2. subordinate bus 3. IO range [0xc000, 0xcfff] memory range [0xfe400000, 0xfe7fffff] prefetchable memory range [0xf0000000, 0xefffffff] id "pci.1" Bus 2, device 1, function 0: PCI bridge: PCI device 1b36:0001 IRQ 10. BUS 2. secondary bus 3. subordinate bus 3. IO range [0xc000, 0xcfff] memory range [0xfe400000, 0xfe5fffff] prefetchable memory range [0xf0000000, 0xefffffff] BAR0: 64 bit memory at 0xfe600000 [0xfe6000ff]. id "pci.2" Bus 3, device 1, function 0: Ethernet controller: PCI device 1af4:1000 IRQ 11. BAR0: I/O at 0xc040 [0xc05f]. BAR1: 32 bit memory at 0xfe440000 [0xfe440fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe]. id "net0" Bus 3, device 2, function 0: USB controller: PCI device 8086:2934 IRQ 11. BAR4: I/O at 0xc060 [0xc07f]. id "" Bus 3, device 2, function 1: USB controller: PCI device 8086:2935 IRQ 10. BAR4: I/O at 0xc080 [0xc09f]. id "" Bus 3, device 2, function 2: USB controller: PCI device 8086:2936 IRQ 10. BAR4: I/O at 0xc0a0 [0xc0bf]. id "" Bus 3, device 2, function 7: USB controller: PCI device 8086:293a IRQ 11. BAR0: 32 bit memory at 0xfe441000 [0xfe441fff]. id "usb" Bus 3, device 3, function 0: SCSI controller: PCI device 1af4:1001 IRQ 10. BAR0: I/O at 0xc000 [0xc03f]. BAR1: 32 bit memory at 0xfe442000 [0xfe442fff]. id "virtio-disk0" Bus 3, device 4, function 0: Class 0255: PCI device 1af4:1002 IRQ 10. BAR0: I/O at 0xc0c0 [0xc0df]. id "balloon0" Bus 0, device 31, function 0: ISA bridge: PCI device 8086:2918 id "" Bus 0, device 31, function 2: SATA controller: PCI device 8086:2922 IRQ 10. BAR4: I/O at 0xe040 [0xe05f]. BAR5: 32 bit memory at 0xfea04000 [0xfea04fff]. id "" Bus 0, device 31, function 3: SMBus: PCI device 8086:2930 IRQ 10. BAR4: I/O at 0xb100 [0xb13f]. id ""

Sorry for the big dump. I don't see anything out of the ordinary, unless I'm being blind? I'm going to try and see if I can get the card to run in native Linux with the opensource drivers.

Offline

#1262 2014-03-02 23:25:15

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For the moment, I'm actually giving up on using VT-d, and have switched to Windows as the host and Linux as a storage-only guest.  I'm tired of all those assertion failures and the need to suspend and resume the host for the guest to start.

Incidentally, before Borderlands patch, the VM gave a mere 25% of the bare-hardware performance (30FPS vs 120FPS). I haven't tried with the patches.

As for rendering to a window on the host, you'd essentially need something like Lucid Virtu MVP, with the emulated Cirrus card as the output device, rather than an Intel card.
On my system, the "Star Swarm" mantle demo can use the Radeon for rendering, while displaying the output on the Intel GPU.  Who knows... maybe you can do the same with the Cirrus!

For switching between host and guest, I'd actually recommend using a KVM switch, and passing through an actual USB controller.  I tried passing through a hub instead, but the only thing that got forwarded was the hub itself.

Offline

#1263 2014-03-02 23:32:34

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:

I'm going to try and see if I can get the card to run in native Linux with the opensource drivers.

Well, crap:

[ 1902.089496] [drm] radeon kernel modesetting enabled. [ 1902.089542] radeon 0000:01:00.0: enabling device (0000 -> 0003) [ 1902.089732] [drm] initializing kernel modesetting (HAWAII 0x1002:0x67B0 0x174B:0xE285). [ 1902.089760] [drm] register mmio base: 0xF0800000 [ 1902.089761] [drm] register mmio size: 262144 [ 1902.089769] [drm] doorbell mmio base: 0xF0000000 [ 1902.089769] [drm] doorbell mmio size: 8388608 [ 1902.566774] ATOM BIOS: C67101 [ 1902.566789] [drm] GPU not posted. posting now... [ 1902.587575] radeon 0000:01:00.0: VRAM: 4096M 0x0000000000000000 - 0x00000000FFFFFFFF (4096M used) [ 1902.587576] radeon 0000:01:00.0: GTT: 1024M 0x0000000100000000 - 0x000000013FFFFFFF [ 1902.587577] [drm] Detected VRAM RAM=4096M, BAR=256M [ 1902.587578] [drm] RAM width 512bits DDR [ 1902.587701] [TTM] Zone kernel: Available graphics memory: 16461126 kiB [ 1902.587714] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 1902.587714] [TTM] Initializing pool allocator [ 1902.587717] [TTM] Initializing DMA pool allocator [ 1902.587726] [drm] radeon: 4096M of VRAM memory ready [ 1902.587726] [drm] radeon: 1024M of GTT memory ready. [ 1902.587948] [drm] GART: num cpu pages 262144, num gpu pages 262144 [ 1902.588381] [drm] probing gen 2 caps for device 8086:c01 = 261ad03/e [ 1902.588383] [drm] PCIE gen 3 link speeds already enabled [ 1902.589539] [drm] Loading HAWAII Microcode [ 1902.590805] [drm] PCIE GART of 1024M enabled (table at 0x0000000000277000). [ 1902.590897] radeon 0000:01:00.0: WB enabled [ 1902.590906] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000100000c00 and cpu addr 0xffff8807e9679c00 [ 1902.590907] radeon 0000:01:00.0: fence driver on ring 1 use gpu addr 0x0000000100000c04 and cpu addr 0xffff8807e9679c04 [ 1902.590908] radeon 0000:01:00.0: fence driver on ring 2 use gpu addr 0x0000000100000c08 and cpu addr 0xffff8807e9679c08 [ 1902.590908] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000100000c0c and cpu addr 0xffff8807e9679c0c [ 1902.590909] radeon 0000:01:00.0: fence driver on ring 4 use gpu addr 0x0000000100000c10 and cpu addr 0xffff8807e9679c10 [ 1902.591283] radeon 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000000076c98 and cpu addr 0xffffc90022836c98 [ 1902.591284] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 1902.591284] [drm] Driver supports precise vblank timestamp query. [ 1902.591298] radeon 0000:01:00.0: irq 61 for MSI/MSI-X [ 1902.591304] radeon 0000:01:00.0: radeon: using MSI. [ 1902.591323] [drm] radeon: irq initialized. [ 1902.593780] [drm] ring test on 0 succeeded in 4 usecs [ 1902.593845] [drm] ring test on 1 succeeded in 3 usecs [ 1902.593857] [drm] ring test on 2 succeeded in 3 usecs [ 1902.594025] [drm] ring test on 3 succeeded in 2 usecs [ 1902.594035] [drm] ring test on 4 succeeded in 2 usecs [ 1902.650012] [drm] ring test on 5 succeeded in 2 usecs [ 1902.670004] [drm] UVD initialized successfully. [ 1902.671134] [drm] Enabling audio 0 support [ 1902.671134] [drm] Enabling audio 1 support [ 1902.671135] [drm] Enabling audio 2 support [ 1902.671136] [drm] Enabling audio 3 support [ 1902.671137] [drm] Enabling audio 4 support [ 1902.671137] [drm] Enabling audio 5 support [ 1902.671280] [drm] ib test on ring 0 succeeded in 0 usecs [ 1902.671420] [drm] ib test on ring 1 succeeded in 0 usecs [ 1902.671592] [drm] ib test on ring 2 succeeded in 0 usecs [ 1902.671728] [drm] ib test on ring 3 succeeded in 0 usecs [ 1902.671864] [drm] ib test on ring 4 succeeded in 1 usecs [ 1902.692772] [drm] ib test on ring 5 succeeded [ 1902.713505] [drm] Radeon Display Connectors [ 1902.713507] [drm] Connector 0: [ 1902.713508] [drm] DP-2 [ 1902.713509] [drm] HPD2 [ 1902.713511] [drm] DDC: 0x6530 0x6530 0x6534 0x6534 0x6538 0x6538 0x653c 0x653c [ 1902.713511] [drm] Encoders: [ 1902.713512] [drm] DFP1: INTERNAL_UNIPHY2 [ 1902.713513] [drm] Connector 1: [ 1902.713514] [drm] HDMI-A-4 [ 1902.713515] [drm] HPD3 [ 1902.713516] [drm] DDC: 0x6550 0x6550 0x6554 0x6554 0x6558 0x6558 0x655c 0x655c [ 1902.713517] [drm] Encoders: [ 1902.713518] [drm] DFP2: INTERNAL_UNIPHY2 [ 1902.713519] [drm] Connector 2: [ 1902.713520] [drm] DVI-D-1 [ 1902.713521] [drm] HPD1 [ 1902.713522] [drm] DDC: 0x6560 0x6560 0x6564 0x6564 0x6568 0x6568 0x656c 0x656c [ 1902.713523] [drm] Encoders: [ 1902.713524] [drm] DFP3: INTERNAL_UNIPHY1 [ 1902.713525] [drm] Connector 3: [ 1902.713526] [drm] DVI-D-2 [ 1902.713526] [drm] HPD6 [ 1902.713528] [drm] DDC: 0x6580 0x6580 0x6584 0x6584 0x6588 0x6588 0x658c 0x658c [ 1902.713529] [drm] Encoders: [ 1902.713529] [drm] DFP4: INTERNAL_UNIPHY [ 1902.713636] [drm] Internal thermal controller with fan control [ 1902.713661] [drm] radeon: power management initialized [ 1902.868919] [drm] fb mappable at 0xE1488000 [ 1902.868921] [drm] vram apper at 0xE0000000 [ 1902.868921] [drm] size 9216000 [ 1902.868922] [drm] fb depth is 24 [ 1902.868922] [drm] pitch is 7680 [ 1902.868953] radeon 0000:01:00.0: fb1: radeondrmfb frame buffer device [ 1902.868977] [drm] Initialized radeon 2.35.0 20080528 for 0000:01:00.0 on minor 1 [ 1907.863768] [drm] Disabling audio 0 support [ 1907.863770] [drm] Disabling audio 1 support [ 1907.863770] [drm] Disabling audio 2 support [ 1907.863771] [drm] Disabling audio 3 support [ 1907.863772] [drm] Disabling audio 4 support [ 1907.863772] [drm] Disabling audio 5 support [ 1907.874010] dmar: DRHD: handling fault status reg 3 [ 1907.889233] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 100315000 DMAR:[fault reason 06] PTE Read access is not set [ 1917.918561] radeon 0000:01:00.0: GPU lockup CP stall for more than 10053msec [ 1917.940335] radeon 0000:01:00.0: GPU lockup (waiting for 0x0000000000000002 last fence id 0x0000000000000001 on ring 3) [ 1929.846660] [drm] probing gen 2 caps for device 8086:c01 = 261ad03/e [ 1929.846663] [drm] PCIE gen 3 link speeds already enabled [ 1929.847829] radeon 0000:01:00.0: ffff8807ee59c400 pin failed [ 1929.869772] [drm:cik_resume] *ERROR* cik startup failed on resume [ 1929.892186] dmar: DRHD: handling fault status reg 3 [ 1929.892207] [drm:cik_ib_test] *ERROR* radeon: fence wait failed (-35). [ 1929.892208] [drm:radeon_ib_ring_tests] *ERROR* radeon: failed testing IB on ring 1 (-35). [ 1929.892236] [drm:cik_ib_test] *ERROR* radeon: fence wait failed (-35). [ 1929.892236] [drm:radeon_ib_ring_tests] *ERROR* radeon: failed testing IB on ring 2 (-35). [ 1929.892275] [drm:cik_sdma_ib_test] *ERROR* radeon: fence wait failed (-35). [ 1929.892275] [drm:radeon_ib_ring_tests] *ERROR* radeon: failed testing IB on ring 3 (-35). [ 1929.892303] [drm:cik_sdma_ib_test] *ERROR* radeon: fence wait failed (-35). [ 1929.892303] [drm:radeon_ib_ring_tests] *ERROR* radeon: failed testing IB on ring 4 (-35). [ 1930.116523] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 100115000 DMAR:[fault reason 06] PTE Read access is not set [ 1938.674784] radeon 0000:01:00.0: ffff8807ee59c400 unpin not necessary [ 1966.369162] type=1006 audit(1393802873.962:6): pid=7164 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=5 res=1

Edit: this seems to happen only if the Radeon isn't assigned as the primary video card (over the IGP). Once I told my motherboard's BIOS to set the PCI-E as the primary VGA, it seems to work fine natively. Going to see if I can translate this to a working VM again.

Edit 2: nope

[ 256.508640] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 0xe0000000-0xefffffff 64bit pref]

Last edited by MacCoaster (2014-03-03 00:15:08)

Offline

#1264 2014-03-03 02:15:39

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm able to get it to work again by using pci-assign, not vfio anymore.

WTF.

Offline

#1265 2014-03-03 03:18:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:

I'm able to get it to work again by using pci-assign, not vfio anymore.

WTF.

Your graphics card has Address Translation Services (ATS), aka a device IOMMU.  That usually doesn't cause problems, but since you're seeing stray addresses and since pci-assign works, what happens if you hide it, using something like this:

--- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -89,7 +89,7 @@ static u16 pci_ext_cap_length[] = { [PCI_EXT_CAP_ID_CAC] = 0, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup obsolete */ [PCI_EXT_CAP_ID_ACS] = 0xFF, [PCI_EXT_CAP_ID_ARI] = PCI_EXT_CAP_ARI_SIZEOF, - [PCI_EXT_CAP_ID_ATS] = PCI_EXT_CAP_ATS_SIZEOF, + [PCI_EXT_CAP_ID_ATS] = 0, [PCI_EXT_CAP_ID_SRIOV] = PCI_EXT_CAP_SRIOV_SIZEOF, [PCI_EXT_CAP_ID_MRIOV] = 0, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup not yet */ [PCI_EXT_CAP_ID_MCAST] = PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF,

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1266 2014-03-03 05:06:17

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Your graphics card has Address Translation Services (ATS), aka a device IOMMU.  That usually doesn't cause problems, but since you're seeing stray addresses and since pci-assign works, what happens if you hide it, using something like this: ...

First boot, VFIO works again. Rebooted the VM itself and seems to work fine, something it couldn't do reliably before. This one patch might be working well for the Radeon R9 290X. Anything else I should be testing?

Edit Spoke too soon. Looks like I can't reboot the VM reliably anymore. But real machine reboot seems to work fine and the Radeon keeps coming back up reliably in the VM.

Edit 2: the USB card I got (Renesas) still doesn't work. Code 10 in the VM, @333 (when it first boots) and @374 when I disable it and re-enable it in the VM using Device Manager.

[ 333.211539] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.238128] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.238134] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.251441] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.251447] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.251450] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.264767] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.264772] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.264776] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.264780] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.278093] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.278098] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.278102] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.278106] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.278110] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.291419] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.291424] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.291428] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.291431] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.291435] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.291439] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 333.304754] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.304760] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.304764] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.304767] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.304771] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.304775] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 333.304778] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 333.318068] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.318073] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.318077] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.318081] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.318085] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.318088] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 333.318092] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 333.318096] vfio-pci 0000:03:00.0: irq 70 for MSI/MSI-X [ 374.563010] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.596290] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.596296] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.612944] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.612949] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.612953] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.629600] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.629606] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.629610] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.629613] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.646259] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.646264] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.646268] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.646272] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.646275] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.666245] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.666250] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.666254] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.666258] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.666261] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.666265] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 374.686231] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.686237] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.686240] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.686244] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.686248] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.686251] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 374.686255] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 374.702888] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.702893] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.702897] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.702901] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.702904] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.702908] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 374.702911] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 374.702915] vfio-pci 0000:03:00.0: irq 70 for MSI/MSI-X

and the lspci -vv -s 3: output:

03:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI]) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 18 Region 0: Memory at f1400000 (64-bit, non-prefetchable) [disabled] [size=8K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [90] MSI-X: Enable- Count=8 Masked- Vector table: BAR=0 offset=00001000 PBA: BAR=0 offset=00001080 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [150 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: vfio-pci Kernel modules: xhci_hcd

Anything I can do?

Last edited by MacCoaster (2014-03-03 05:53:26)

Offline

#1267 2014-03-03 08:26:05

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For a long time i have had issues when trying to start the guest (win 8.1).

My problems where that the guest boot process would stop before it could finish, then it would hang there for a bit until the host qemu process stopped. In order to boot my guest i had to restart the host 1-2 times and it would work until next time i powered down my host. And then the reboot process had to be repeated.

I think i may have found a solution which i wanted to share in case other people was having this problem.

What i did was disable the "fast boot" option in the windows 8.1 guest, after that it has worked flawlessly for me (3 host reboots now).

Offline

#1268 2014-03-03 09:09:28

arny91
Member
Registered: 2014-01-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, were just the commas to prevent IOMMU from being enabled.

I attached a 2nd hdd with a windows install and it boots! This is the command line i use

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device ahci,bus=pcie.0,id=ahci -drive file=/dev/sdb,id=disk,cache=none,if=virtio -usb -usbdevice host:1532:001c -usbdevice host:0e6a:6001 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

The GPU just gets passed-through finely, but i'm sure there is something else to tweak.

Offline

#1269 2014-03-03 15:36:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:
aw wrote:

Your graphics card has Address Translation Services (ATS), aka a device IOMMU.  That usually doesn't cause problems, but since you're seeing stray addresses and since pci-assign works, what happens if you hide it, using something like this: ...

First boot, VFIO works again. Rebooted the VM itself and seems to work fine, something it couldn't do reliably before. This one patch might be working well for the Radeon R9 290X. Anything else I should be testing?

Edit Spoke too soon. Looks like I can't reboot the VM reliably anymore. But real machine reboot seems to work fine and the Radeon keeps coming back up reliably in the VM.

Have you tried this:

--- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -3209,7 +3209,7 @@ static void vfio_pci_reset_handler(void *opaque) QLIST_FOREACH(group, &group_list, next) { QLIST_FOREACH(vdev, &group->device_list, next) { - if (!vdev->reset_works || (!vdev->has_flr && vdev->has_pm_reset)) { + if (!vdev->reset_works || !vdev->has_flr) { vdev->needs_reset = true; } }

Edit 2: the USB card I got (Renesas) still doesn't work. Code 10 in the VM, @333 (when it first boots) and @374 when I disable it and re-enable it in the VM using Device Manager.

[ 333.211539] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.238128] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.238134] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.251441] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.251447] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.251450] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.264767] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.264772] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.264776] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.264780] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.278093] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.278098] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.278102] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.278106] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.278110] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.291419] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.291424] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.291428] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.291431] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.291435] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.291439] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 333.304754] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.304760] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.304764] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.304767] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.304771] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.304775] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 333.304778] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 333.318068] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 333.318073] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 333.318077] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 333.318081] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 333.318085] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 333.318088] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 333.318092] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 333.318096] vfio-pci 0000:03:00.0: irq 70 for MSI/MSI-X [ 374.563010] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.596290] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.596296] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.612944] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.612949] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.612953] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.629600] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.629606] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.629610] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.629613] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.646259] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.646264] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.646268] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.646272] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.646275] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.666245] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.666250] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.666254] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.666258] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.666261] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.666265] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 374.686231] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.686237] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.686240] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.686244] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.686248] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.686251] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 374.686255] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 374.702888] vfio-pci 0000:03:00.0: irq 63 for MSI/MSI-X [ 374.702893] vfio-pci 0000:03:00.0: irq 64 for MSI/MSI-X [ 374.702897] vfio-pci 0000:03:00.0: irq 65 for MSI/MSI-X [ 374.702901] vfio-pci 0000:03:00.0: irq 66 for MSI/MSI-X [ 374.702904] vfio-pci 0000:03:00.0: irq 67 for MSI/MSI-X [ 374.702908] vfio-pci 0000:03:00.0: irq 68 for MSI/MSI-X [ 374.702911] vfio-pci 0000:03:00.0: irq 69 for MSI/MSI-X [ 374.702915] vfio-pci 0000:03:00.0: irq 70 for MSI/MSI-X

Nothing unusual there, just an artifact of how interrupts are setup.

and the lspci -vv -s 3: output:

03:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI]) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 18 Region 0: Memory at f1400000 (64-bit, non-prefetchable) [disabled] [size=8K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [90] MSI-X: Enable- Count=8 Masked- Vector table: BAR=0 offset=00001000 PBA: BAR=0 offset=00001080 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [150 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: vfio-pci Kernel modules: xhci_hcd

Anything I can do?

Dumb question, have you actually installed the driver for the device?  Coming from Linux I'm spoiled by adding a device and having it "just work".  Windows isn't that way and I forget that sometimes.  In fact, just last week I though there might be some problem assigning the XHCI controller from the B85 chipset to a Win7 guest, but it was just a matter of going to the Intel website and installing the driver.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1270 2014-03-04 21:28:52

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Have you tried this:

--- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -3209,7 +3209,7 @@ static void vfio_pci_reset_handler(void *opaque) QLIST_FOREACH(group, &group_list, next) { QLIST_FOREACH(vdev, &group->device_list, next) { - if (!vdev->reset_works || (!vdev->has_flr && vdev->has_pm_reset)) { + if (!vdev->reset_works || !vdev->has_flr) { vdev->needs_reset = true; } }

I'll try this when I can.

aw wrote:

Dumb question, have you actually installed the driver for the device?  Coming from Linux I'm spoiled by adding a device and having it "just work".  Windows isn't that way and I forget that sometimes.  In fact, just last week I though there might be some problem assigning the XHCI controller from the B85 chipset to a Win7 guest, but it was just a matter of going to the Intel website and installing the driver.

Yep. Windows 8 has the driver which makes most if not all USB3 controllers work out of the box. I just tried a Windows native install and it seems like the USB3 card refuses to initialize there either (similarly to native Linux drivers), so I guess the card is bad. sad

Offline

#1271 2014-03-05 00:00:19

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I`ve got problems with my sound again - it just randomly stopped working....
On the Log I get these entries:

/etc/gdm/Xsession[915]: pulseaudio: pa_context_connect() failed /etc/gdm/Xsession[915]: pulseaudio: Reason: Connection refused /etc/gdm/Xsession[915]: pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' /etc/gdm/Xsession[915]: pulseaudio: pa_context_connect() failed /etc/gdm/Xsession[915]: pulseaudio: Reason: Connection refused /etc/gdm/Xsession[915]: pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' /etc/gdm/Xsession[915]: ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave /etc/gdm/Xsession[915]: alsa: Could not initialize DAC /etc/gdm/Xsession[915]: alsa: Failed to open `default': /etc/gdm/Xsession[915]: alsa: Reason: Device or resource busy /etc/gdm/Xsession[915]: ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave /etc/gdm/Xsession[915]: alsa: Could not initialize DAC /etc/gdm/Xsession[915]: alsa: Failed to open `default': /etc/gdm/Xsession[915]: alsa: Reason: Device or resource busy /etc/gdm/Xsession[915]: audio: Failed to create voice `dac'

I created a different VM with ubuntu where the sound worked. After I executed it as root it showed the same behaviour as the windows vm... Do you know what could be going wrong?

Offline

#1272 2014-03-05 04:27:35

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using latest kernel/qemu from OP. Getting this error.

qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

It boots fine after downgrading to qemu-git-2.0.r30691.g91abb80-1-x86_64.pkg.tar.xz.

Note that it errors when downgrading, because of the custom SeaBIOS file. I deleted /usr/share/qemu/bios.bin, downgraded, then reinstalled my SeaBIOS package.

Last edited by alexis_evo (2014-03-05 04:33:20)

Offline

#1273 2014-03-05 07:24:56

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Using latest kernel/qemu from OP. Getting this error.

qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

It boots fine after downgrading to qemu-git-2.0.r30691.g91abb80-1-x86_64.pkg.tar.xz.

Note that it errors when downgrading, because of the custom SeaBIOS file. I deleted /usr/share/qemu/bios.bin, downgraded, then reinstalled my SeaBIOS package.

You need this fix for the assertion :

https://lists.gnu.org/archive/html/qemu … 02774.html

Last edited by anickname (2014-03-05 07:25:12)

Offline

#1274 2014-03-05 11:48:37

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

I put in a second graphicscard and tested it again.
The good thing is. I had no problems to pass it to the VM... have seen the bios of the Card + seabios on the screen + Windows was booting (was booting to system repair, but I think that's because of other storage type)

The bad thing is, that both graphicscards are in the same iommu-group + 2 PCI-Controller 01.0 and 01.1. Think that is because my 2 PCIe Ports share share the 16 lanes, that are normally used by the first port.
So I think I will be abled to pass the card to the VM, but can't really use the primary card for Linux and I have to wait to get the Intel as Primary working ...

personally I would look to move primary (i.e. used by hardware own BIOS) graphics card to slot which does not share IOMMU group with the other one .... if that excludes all 16 lanes slots, there is at least one which will fit into X1 slot


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1275 2014-03-05 11:55:19

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
apoapo wrote:

I put in a second graphicscard and tested it again.
The good thing is. I had no problems to pass it to the VM... have seen the bios of the Card + seabios on the screen + Windows was booting (was booting to system repair, but I think that's because of other storage type)

The bad thing is, that both graphicscards are in the same iommu-group + 2 PCI-Controller 01.0 and 01.1. Think that is because my 2 PCIe Ports share share the 16 lanes, that are normally used by the first port.
So I think I will be abled to pass the card to the VM, but can't really use the primary card for Linux and I have to wait to get the Intel as Primary working ...

personally I would look to move primary (i.e. used by hardware own BIOS) graphics card to slot which does not share IOMMU group with the other one .... if that excludes all 16 lanes slots, there is at least one which will fit into X1 slot

RIght now im passing through my primary card (radeon 6950) what i did was, use pci-stub to bind it at boot and i disabled the efifb framebuffer with video=efifb:off on my grub cmdline

Offline

#1276 2014-03-05 12:53:20

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

RIght now im passing through my primary card (radeon 6950) what i did was, use pci-stub to bind it at boot and i disabled the efifb framebuffer with video=efifb:off on my grub cmdline

that's neat, I didn't know this option smile


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1277 2014-03-05 13:19:03

stedaniels
Member
From: UK
Registered: 2013-11-21
Posts: 2
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've just set up a bare arch linux install, and I'd like to start experimenting with KVM VGA-passthrough using vfio-vga.

I see there are source tarballs on the first page with patches pre-applied, and mentions of AUR, but I'd rather put together a script that I can run every time there's a kernel update.

There are three patches mentioned:

  1. acs override patch

  2. i935 vga arbiter fixes

  3. memleak fix by xani

1. Is the ACS override patch on the kvm mailing list still the latest one?
2. For the i935 vga arbiter fixes patch is version 3 the latest and greatest?
3. Xani posted the first xani memory leak fix but that link is now dead, I read through and saw that xani posted a second memory leak fix but that link si also dead, does anyone have a current source?

Thanks!

I don't doubt the quality of the packages already provided, I just want a quick and easy way to rebuild and reproduce on a whim.

Thank you for all your time and energy already exerted on this adventure!

Steve

Last edited by stedaniels (2014-03-05 13:19:22)

Offline

#1278 2014-03-05 13:51:11

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

stedaniels wrote:

Hi,

I've just set up a bare arch linux install, and I'd like to start experimenting with KVM VGA-passthrough using vfio-vga.

I see there are source tarballs on the first page with patches pre-applied, and mentions of AUR, but I'd rather put together a script that I can run every time there's a kernel update.

There are three patches mentioned:

  1. acs override patch

  2. i935 vga arbiter fixes

  3. memleak fix by xani

1. Is the ACS override patch on the kvm mailing list still the latest one?
2. For the i935 vga arbiter fixes patch is version 3 the latest and greatest?
3. Xani posted the first xani memory leak fix but that link is now dead, I read through and saw that xani posted a second memory leak fix but that link si also dead, does anyone have a current source?

Thanks!

I don't doubt the quality of the packages already provided, I just want a quick and easy way to rebuild and reproduce on a whim.

Thank you for all your time and energy already exerted on this adventure!

Steve

Those are not source packages, and all the patches you mention are there, you can build it yourself using makepkg

https://wiki.archlinux.org wrote:

makepkg is used for compiling and building packages suitable for installation with pacman, Arch Linux's package manager. makepkg is a script that automates the building of packages; it can download and validate source files, check dependencies, configure build-time settings, compile the sources, install into a temporary root, make customizations, generate meta-info, and package everything together.

Last edited by nbhs (2014-03-05 13:55:04)

Offline

#1279 2014-03-05 14:21:10

stedaniels
Member
From: UK
Registered: 2013-11-21
Posts: 2
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi nbhs & others,

Sorry for my ignorance and thank you for not calling me out on it!  You learn something new everyday it seems.

Reading through PKGBUILD was enlightening.  I'd previously got this all mostly running on Ubuntu so arch is a whole new experience.

Cheers,

Steve

Offline

#1280 2014-03-05 14:35:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

stedaniels wrote:

Hi nbhs & others,

Sorry for my ignorance and thank you for not calling me out on it!  You learn something new everyday it seems.

Reading through PKGBUILD was enlightening.  I'd previously got this all mostly running on Ubuntu so arch is a whole new experience.

Cheers,

Steve

Here's how you build and install these packages.

For example the kernel, unpack the linux-mainline.tar.gz file somewhere then:

cd path/to/linux-mainline makepkg -s sudo pacman -U linux-mainline-*

EDIT: Dont forget to set MAKEFLAGS to the number of cores you have on your /etc/makepkg.conf file

Last edited by nbhs (2014-03-05 14:41:30)

Offline

#1281 2014-03-05 21:59:42

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA passthrough is working great with a Radeon 5850. I have tested with a few games and performance is impressive. Tested guests were Windows 7, Fedora 20, Ubuntu 12.04 and 14.04 and Debian 7.

Using Debian Sid as host. Linux 3.13.5 with patches and QEMU from the first post in this topic.

Current script :

${BIN_PATH}qemu-system-x86_64 --enable-kvm \ -M q35 -m ${RAM} -cpu ${CPU} -name ${NAME} \ -smp ${CORES},sockets=1,cores=${CORES},threads=1 \ -bios ${BIOS} \ -vga none -rtc base=localtime,clock=host \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device piix4-ide,bus=pcie.0,id=ide \ -device vfio-pci,host=${GPU_RADEON},bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=${GPU_BIOS} \ -device vfio-pci,host=${GPU_AUDIO},bus=root.1 \ -device vfio-pci,host=${USB_PORTS},bus=pcie.0 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -net nic,macaddr=${MAC_ADDR},model=virtio -net tap,ifname=tap0,script=no,downscript=no \ -drive file=${HD_PATH},id=disk,if=virtio \ -boot order=cd,menu=on

What I'm wondering is how can I rebind some devices so the host after unbinding them from VFIO-PCI?

echo 0000:00:1d.0 > /sys/bus/pci/drivers/vfio-pci/unbind

But I can't seem to find where to reattach it to the host. It's a USB controller hub from the motherboard. If I unbind it from host, then I'm losing at least 6 USB ports. I'm guessing I need to echo $vendor and $device to a new_id but I can't seem to find the right one, all I get is "echo: write error: Invalid argument".

Last edited by mv0lnicky (2014-03-06 15:22:04)

Offline

#1282 2014-03-06 03:00:16

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A quick update - another success story ... thanks for all your help.

CPU = Intel 4670, MBoard = Asrock Z87 Extreme6, RAM=16GB 1600, GPU = Radeon HD6950, Arch Host Kernel 3.13, Window 7 guest, Renesas based PCIe add-on card passed to the guest

I get about 70-80% native performance under my test conditions (rfacvtor v1, using 11 AI drivers offline). Most importantly there is no micro-stutter problem and no significant latency issues (any more)
I found that allocating multiple CPUs made a noticeable difference to frame rate and subjective experience (surprisingly since I thought this app was GPU limited not CPU)
Test results reported below are expressed as frames / sec as reported in the application (all a bit subjective, but good enough to indicate what to expect). The test was deliberately set up to present a reasonable maximum load. In practice most situations present a lower load than the test case so "normal" performance will be better than reported here
                         Qemu     Qemu          Native        Native
Track Position    2 CPU    1 CPU          4 CPU       1 CPU
---------------------    ---------     --------           ---------        ---------
Pits                     30+         20+             40+           40+
Start Grid             40+         25+             60+           50+
StartF Straight      70+         30+           100+           90+

2 lap MAX            70+         50+                            100+

Note that Native mode is based on a full 4 CPUs on the Intel 4670 ..... CPU constraints are applied at the App level (there's a parameter with options all-CPUs or 1-CPU) so there's no obvious way to reproduce the guest CPU count

Online racing against other (real) people I can see frame rates over 200 fps depending on track and mod (cars) ie. quite acceptable and very useable. Furthermore there are no significant lag issues which means that it is practical to follow closely without crashing eg. if they brake suddenly you have time to react because there is no more detectable lag than a native installation.

After applying this patch to fix the Qemu assertion error (I had to modify this PCIE: fix regression with coldplugged multifunction device  to get it to apply)

diff --git a/qemu/hw/pci/pcie.c b/qemu/hw/pci/pcie.c index 8ecd11e..02cde6f 100644 --- a/qemu/hw/pci/pcie.c +++ b/qemu/hw/pci/pcie.c @@ -221,29 +221,23 @@ DeviceState *dev, uint8_t **exp_cap, Error **errp) { - PCIDevice *pci_dev = PCI_DEVICE(dev); *exp_cap = hotplug_dev->config + hotplug_dev->exp.exp_cap; uint16_t sltsta = pci_get_word(*exp_cap + PCI_EXP_SLTSTA); - PCIE_DEV_PRINTF(pci_dev, "hotplug state: %d\n", state); + PCIE_DEV_PRINTF(PCI_DEVICE(dev), "hotplug state: %d\n", state); if (sltsta & PCI_EXP_SLTSTA_EIS) { /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup the slot is electromechanically locked. 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html This error is propagated up to qdev and then to HMP/QMP. */ error_setg_errno(errp, -EBUSY, "slot is electromechanically locked"); } - - /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup TODO: multifunction hot-plug. - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html Right now, only a device of function = 0 is allowed to be - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html hot plugged/unplugged. - */ - assert(PCI_FUNC(pci_dev->devfn) == 0); } void pcie_cap_slot_hotplug_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { uint8_t *exp_cap; + PCIDevice *pci_dev = PCI_DEVICE(dev); pcie_cap_slot_hotplug_common(PCI_DEVICE(hotplug_dev), dev, &exp_cap, errp); @@ -255,6 +249,11 @@ PCI_EXP_SLTSTA_PDS); return; } + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup TODO: multifunction hot-plug. + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html Right now, only a device of function = 0 is allowed to be + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html hot plugged/unplugged. + */ + assert(PCI_FUNC(pci_dev->devfn) == 0); pci_word_test_and_set_mask(exp_cap + PCI_EXP_SLTSTA, PCI_EXP_SLTSTA_PDS);

And then away it went (using Qemu 1.7 from the repos failed)

Note that i did apply the following tuning factors -
-  hugetlbfs  (mine set to 3200 - for a 6GB guest)
-  -cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000
-  kernel compiled with CONFIG_PCI_STUB=y CONFIG_PREEMPT_VOLUNTARY=y   and  CONFIG_HZ_1000=y    and  CONFIG_HZ=1000    (in addition to the CONFIG_VFIO and parameters set to y)

final command now looks like this

qemu-system-x86_64 -name win7 -enable-kvm -M q35 -m 6144 -mem-path /dev/hugepages \ -cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -boot menu=on \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -usbdevice tablet \ -spice port=5902,disable-ticketing \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device ahci,bus=pcie.0,id=ahci \ -net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \ -net user \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=02:00.0,bus=pcie.0 \ -device vfio-pci,host=04:00.0,bus=pcie.0 \ -device vfio-pci,host=06:00.0,bus=pcie.0 \ -device vfio-pci,host=05:00.0,bus=pcie.0 \ -drive file=/dev/virt-test-xen/lvwin7a_kvm,id=disk1,format=raw,if=virtio \ -drive file=/dev/sda3,id=disk2,format=raw,if=virtio \ -drive file=/dev/sda2,id=disk3,format=raw,if=virtio \ -drive file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2 \ -device ide-cd,bus=ahci.2,drive=isocd2 \ -drive file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3 \ -device ide-cd,bus=ahci.3,drive=isocd3

You can see above I'm passing through (a) GPUat 1:00:0 (b) Renesas USB card at 2:00.0 (c) NIC at 04:00.0 (d) Asmedia disk controllers at 05:00.0 and 06:00.0 (e) windows install from LVM partition (f) Virtio drivers on iso image as CD-Rom (g) Asrock motherboard drivers and Renesas drivers on iso image as CD-Rom                 plus a virtual NIC which I disable as soon as the passed through device is running. I also use Spice for initial keyboard communications until the USB card drivers are loaded and working.

Grub command line -

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on nohz=off pci-stub.ids=8086:0c01,8086:0c09,1002:6719,1002:aa80,1912:0014,8086:8c10,8086:8c14,8086:8c16,8086:8c18,1b21:0612,8086:1539 quiet"

Things to do -
- obtain kernel 3.14 with debug register patches (in addition to existing patches, most of which are presumably included in 3.14)
- install Windows on a dedicated disk (requires Asrock Asmedia drivers incorporated into initial install, hence std command includes "Winfiles" "CD-Rom" which makes those drivers available)
- Review sound - the GPU sound device is not working (I use USB wireless headphones so not a current issue)

Thanks again for all your help

Offline

#1283 2014-03-06 19:39:45

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

RIght now im passing through my primary card (radeon 6950) what i did was, use pci-stub to bind it at boot and i disabled the efifb framebuffer with video=efifb:off on my grub cmdline

Are you booting using UEFI or BIOS mode ?
The onboard video is it enabled or disabled ?
What about "IGPU Multi Monitor" and "Render standby" options ?

I tried booting using BIOS, onboard GPU, both enabled and disabled; It didn't worked for me.
I see the seabios screen but after it only random pixels. I think that both Windows and Linux are
trying to write to the GPU.

I used pci-stub and video=vesafb:off in the command line.

Thanks.

Offline

#1284 2014-03-06 20:12:55

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:

What I'm wondering is how can I rebind some devices so the host after unbinding them from VFIO-PCI?

echo 0000:00:1d.0 > /sys/bus/pci/drivers/vfio-pci/unbind

But I can't seem to find where to reattach it to the host. It's a USB controller hub from the motherboard. If I unbind it from host, then I'm losing at least 6 USB ports. I'm guessing I need to echo $vendor and $device to a new_id but I can't seem to find the right one, all I get is "echo: write error: Invalid argument".

Ok I found out that rebind to host, I just need to bind to the driver. So in my case

echo "0000:00:1d.0" > /sys/bus/pci/drivers/ehci-pci/bind 

Offline

#1285 2014-03-06 20:19:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

RIght now im passing through my primary card (radeon 6950) what i did was, use pci-stub to bind it at boot and i disabled the efifb framebuffer with video=efifb:off on my grub cmdline

Are you booting using UEFI or BIOS mode ?
The onboard video is it enabled or disabled ?
What about "IGPU Multi Monitor" and "Render standby" options ?

I tried booting using BIOS, onboard GPU, both enabled and disabled; It didn't worked for me.
I see the seabios screen but after it only random pixels. I think that both Windows and Linux are
trying to write to the GPU.

I used pci-stub and video=vesafb:off in the command line.

Thanks.

Im booting in EFI mode, and i dont have an onboard gpu, my primary card is a radeon 6950, and my secondary card is a radeon 5450, im passing though the radeon 6950

Last edited by nbhs (2014-03-06 20:19:55)

Offline

#1286 2014-03-06 23:11:48

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone. First time user here and I am really interested in doing something similar to this. I have read a couple of pages, but its starting to get overwhelming, considering all the new patches, kernels and other fixes. Is it perhaps possible to write an up to date guide, perhaps a bit more in depth on how we can set this up? (What do I need if I boot with UEFI or BIOS, with a NVIDIA or AMD card etc) I am perhaps speaking for myself here, but I know I would appreciate it alot.

Either way, I will definitely try this out, hopefully I wont pull out all my hairs in the process!

Offline

#1287 2014-03-07 02:02:23

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Some of the patches depend on your hardware. Also, if you're using an up to date distribution, you can get some results with just the compiled kernel +  acs override patch,  memleak fix and the patch for your host graphic adapter.

Last edited by mv0lnicky (2014-03-07 02:12:57)

Offline

#1288 2014-03-07 13:10:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1289 2014-03-08 02:49:38

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have been trying with a clean install of Arch Linux, fully updated(per 08 March 2014) and I think Im getting there, but I havent quite made it yet.
i7 processor (Haswell), Kernel 3.13.5-1
Host GPU: Intel HD 4600
Guest GPU: NVIDIA GTX 670

Nouveau and nvidia drivers are blacklisted.

This is the only thing I got in my kernel boot options: intel_iommu=on

This is my dmesg |grep IOMMU:
[    0.000000] Intel-IOMMU: enabled
[    0.036230] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.036235] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.036300] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.428676] IOMMU 0 0xfed90000: using Queued invalidation
[    0.428676] IOMMU 1 0xfed91000: using Queued invalidation
[    0.428678] IOMMU: Setting RMRR:
[    0.428686] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.429887] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc0cf000 - 0xbc0dbfff]
[    0.429907] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc0cf000 - 0xbc0dbfff]
[    0.429924] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc0cf000 - 0xbc0dbfff]
[    0.429936] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.429942] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

vfio-bind 0000:01:00.0 0000:01:00.1 gives me this in journalctl:
Mar 08 04:26:11 arch kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
Mar 08 04:26:11 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

However, when trying to run this command:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get the message: Could not initialize SDL(No available video device) - exiting.
Nothing in the journalctl is logged when I try to run qemu-system, besides this message: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Any help is greatly appreciated, as this is something I really would like to have.

Offline

#1290 2014-03-08 03:10:47

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

I have been trying with a clean install of Arch Linux, fully updated(per 08 March 2014) and I think Im getting there, but I havent quite made it yet.
i7 processor (Haswell), Kernel 3.13.5-1
Host GPU: Intel HD 4600
Guest GPU: NVIDIA GTX 670

Nouveau and nvidia drivers are blacklisted.

This is the only thing I got in my kernel boot options: intel_iommu=on

This is my dmesg |grep IOMMU:
[    0.000000] Intel-IOMMU: enabled
[    0.036230] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.036235] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.036300] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.428676] IOMMU 0 0xfed90000: using Queued invalidation
[    0.428676] IOMMU 1 0xfed91000: using Queued invalidation
[    0.428678] IOMMU: Setting RMRR:
[    0.428686] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.429887] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc0cf000 - 0xbc0dbfff]
[    0.429907] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc0cf000 - 0xbc0dbfff]
[    0.429924] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc0cf000 - 0xbc0dbfff]
[    0.429936] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.429942] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

vfio-bind 0000:01:00.0 0000:01:00.1 gives me this in journalctl:
Mar 08 04:26:11 arch kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
Mar 08 04:26:11 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

However, when trying to run this command:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get the message: Could not initialize SDL(No available video device) - exiting.
Nothing in the journalctl is logged when I try to run qemu-system, besides this message: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Any help is greatly appreciated, as this is something I really would like to have.

Use the qemu -nographic arg

Last edited by nbhs (2014-03-08 03:15:33)

Offline

#1291 2014-03-08 10:20:09

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Im just going to add the -nographic arg without removing anything from the qemu command?

This gives me the following in journalctl:

Mar 08 12:11:33 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:35 arch kernel: [drm:hsw_unclaimed_reg_check] *ERROR* Unclaimed write to 4400c
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:37 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

Looking at the processes with htop I can see two qemu-system processes running at above 90%. But nothing else is happening.

Offline

#1292 2014-03-08 10:26:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

Im just going to add the -nographic arg without removing anything from the qemu command?

This gives me the following in journalctl:

Mar 08 12:11:33 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:35 arch kernel: [drm:hsw_unclaimed_reg_check] *ERROR* Unclaimed write to 4400c
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:35 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 12:11:37 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

Looking at the processes with htop I can see two qemu-system processes running at above 90%. But nothing else is happening.

Are you using the kernel i provided?

Offline

#1293 2014-03-08 11:30:27

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Of course, I forgot that. I have compiled it now, installed it and rebooted.
Journalctl gives me:

Mar 08 13:27:09 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 13:27:09 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 13:27:09 arch kernel: [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to 44040
Mar 08 13:27:09 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 13:27:11 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

Tested with both -nographic and without - nothing happens.

(Will these patches be implemented upstream in the future, so we dont have to compile our own kernel all the time?)

As before, Im grateful for your help.

Offline

#1294 2014-03-08 12:40:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

Of course, I forgot that. I have compiled it now, installed it and rebooted.
Journalctl gives me:

Mar 08 13:27:09 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 13:27:09 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 13:27:09 arch kernel: [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to 44040
Mar 08 13:27:09 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Mar 08 13:27:11 arch kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

Tested with both -nographic and without - nothing happens.

(Will these patches be implemented upstream in the future, so we dont have to compile our own kernel all the time?)

As before, Im grateful for your help.

Missing i915 patches


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1295 2014-03-08 12:50:30

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I downloaded this: linux-mainline.tar.gz (3.13.5 includes acs override patch, i935 vga arbiter fixes, memleak fix by xani)
tar xfvz
makepkg -s
pacman -U
reboot

Is there something extra I should have done? By i915 patch you mean the i935 vga arbiter fix?

Offline

#1296 2014-03-08 13:54:52

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

I downloaded this: linux-mainline.tar.gz (3.13.5 includes acs override patch, i935 vga arbiter fixes, memleak fix by xani)
tar xfvz
makepkg -s
pacman -U
reboot

Is there something extra I should have done? By i915 patch you mean the i935 vga arbiter fix?

That patch is included in my kernel package, make sure you have installed it and you are running it

uname -r

3.13.5-1-mainline

Offline

#1297 2014-03-08 14:28:53

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

uname -r
3.13.5-1-ARCH

What am I doing wrong? (sudo pacman -U linux-mainline-3.13.5-1-x86_64.pkg.tar.xz)

Offline

#1298 2014-03-08 14:39:09

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

uname -r
3.13.5-1-ARCH

What am I doing wrong? (sudo pacman -U linux-mainline-3.13.5-1-x86_64.pkg.tar.xz)

sudo pacman -U linux-mainline-* sudo grub-mkconfig -o /boot/grub/grub.cfg

You should see a linux mainline boot option on the grub menu now

Offline

#1299 2014-03-08 15:03:36

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh stupid, stupid me! I forgot to put the new kernel in my Gummiboot.
Loading into X-session takes a while now, before it was rather instant. But lo and behold my GTX 670 has been passthroughed to my Windows 8.1WM!
Thank you so much, I am now one (major) step closer to getting this just how I want it to be smile

Offline

#1300 2014-03-08 17:33:43

zoop2
Member
Registered: 2014-03-08
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello all,  first thanks for all the work people are putting into this.  It has been a fun project.

I wanted to share my experience, so that it might help others as well.

My system is a Mac Pro (early 2008)  has 8 cores from Xeon E5462's

The Hosts Video card:
02:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce 8800 GT] (rev a2)
Passthrough Video card:
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks XT [Radeon HD 6670/7670]

I am using Gentoo with Gentoo's moded kernel 3.12.13  I was using QEMU 1.6 from portage,  Now I have changed to the GIT for some new features.  And I have Seabios 1.7.3 also from Portage

So I first had this VM using libvirt.  I tried playing with it to get it to pass the card to windows without success.  I may try to play wi
th it again at a later date.  But I think getting it going without it is a little easier.  I had a lot of problems getting Windows to Boot
when passing the Vfio stuff.  I tried the various things with -M and other config settings.  I ran into various issues.  Like tracebacks fr
om the host  QEMU crashing on start.  and windows crashing on start with just a blank blue screen etc.  I ended up getting around this by b
ooting the system with the original config again and converting everything to only use virtio drivers.   From what I saw debuging all of th
e different ide/sata controllers I tried to setup was causing these problems.  I am sure I could have been making mistakes, but this is wha
t I did.

After converting to that I had a bit of problems with the Ati driver in windows crashing it, but I think that adding these kernel options to the host stopped that from happening. intel_iommu=on,igfx_off,pass-through

I am still having a problem with audio going through the Hosts audio device.  This seems to be a known issue of using ALSA with windows gue
sts.  I dislike pulseaudio so I won't install it.

Here is the line I am using to start the system for now.

/home/zoop/tmp/qemu-git/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -M q35 -cpu qemu64,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1000 -mem-path /dev/hugepages -device virtio-balloon -m 6144 -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/seabios/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -boot menu=on -vga none -drive if=virtio,file=/mnt/backup/vm/win8-Mar2014snap.qcow2,id=disk,format=qcow2 -usb -usbdevice host:047d:1048 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no

From here i am looking to implement the Debug registers patch that was mentioned by @aw a while back to improve performance.  I am wondering what kernel that Patch was for, I am assuming it was for 3.13.x.

To help imporove performance a littl more i also plan on assigning the system to use specific CPUs.

Offline

#1301 2014-03-08 20:37:57

GNA
Member
Registered: 2011-02-15
Posts: 57

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/sys/kernel/iommu_groups is empty.
My system is supporting vt-d and it should be enabled. I dont get it.
Did i miss something obvious? Thanks for help!

Offline

#1302 2014-03-08 20:43:24

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thanks for this thread. Tried to follow the instructions, can boot into windows 7, though when installing AMD catalyst drivers the host is hanging.

relevant info:

lspci:

... (for usb keyboard for guest) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller ... (NVIDIA for host) 01:00.0 VGA compatible controller: NVIDIA Corporation G71 [GeForce 7900 GT/GTO] (rev a1) ... (ATI for guest) 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] ...

with the script mentioned @beginning of thread:

sudo vfio-bind 0000:04:00.0 0000:04:00.1 0000:00:12.0 0000:00:12.2

and my qemu command is:

/usr/local/bin/qemu-system-x86_64 \ -M q35 \ -vga none \ -nographic \ -cpu host \ -machine accel=kvm \ -bios /usr/local/share/qemu/bios.bin \ -m 4096 \ -smp 4,cores=4 \ -drive file=win8.img,if=virtio \ -netdev type=tap,ifname=tap0,script=no,id=net0 \ -device virtio-net-pci,netdev=net0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on,addr=0.0 \ -device vfio-pci,host=04:00.1,bus=root.1,addr=0.1 \ -device vfio-pci,host=00:12.0,bus=root.1 \ -boot d

kernel: 3.13.5-031305-generic (ubuntu)

tried to unload nouveau from host card (or boot without it, by blacklisting), closed X, but whatsoever way I am doing, do not manage to have
the catalyst installed. Without catalyst it works, but need the powerfull graphics under windows with driver.

Any hints how I could narrow down the cause of host freeze?

thanks a lot in advance for your help

Offline

#1303 2014-03-09 08:10:44

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys, now I have another problem here.

I successfully pass-through my GTX770 into my KVM, and it works perfectly.
But I'm using Ubuntu to do this I'll try Archlinux in these days.

Here is my question.

If I want to pass-through 2 GPUs into 2 VMs (each VM have 1 GPU), and these GPU is same (Ex. Both GTX480)
Is VFIO or pci-stub will got 'conflict' when I entered hardware ID into pci-stub or VFIO?
Because same GPU have same hardware ID, and pci-stub.ids is input hardware ID only. (In bootloader cfg)

Is someone do this before?

Last edited by AKSN74 (2014-03-09 08:13:23)

Offline

#1304 2014-03-09 08:45:43

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

to pereczes
I have the same problem with intel host and Windows7 hd7750 guest...

Offline

#1305 2014-03-09 13:01:43

FruitieX
Member
Registered: 2008-07-03
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

KVM VGA passthrough confirmed working with an AMD Radeon R9 280X graphics card, host is Arch and guest Windows 8.1. More relevant specs:

Mainboard: Asrock Z87M Pro4 Socket LGA 1150
CPU: Intel Core i5 4670 3.4 GHz LGA 1150
Graphics card: Asus Radeon R9 280X DirectCU II TOP 3 GB GDDR5

To pereczes and zzz3000: I initially had issues with crashing (and refusing to boot again) right after Windows would either automatically install the video drivers or after I'd do so manually (Catalyst). To fix this I:

1. Compiled latest qemu and seabios from git, used these. Had to comment out an assertion otherwise the code wouldn't compile. (probably not a great idea smile)
2. Passed through only the video card, not the HDMI audio interface on it. I'm not sure if this actually helped the cause, I'll do some further testing once I get back home tonight. In the qemu version found in Arch repos this would cause the passed through display to show garbled output but it works in the git version.

Offline

#1306 2014-03-09 13:23:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Here is my question.

If I want to pass-through 2 GPUs into 2 VMs (each VM have 1 GPU), and these GPU is same (Ex. Both GTX480)
Is VFIO or pci-stub will got 'conflict' when I entered hardware ID into pci-stub or VFIO?
Because same GPU have same hardware ID, and pci-stub.ids is input hardware ID only. (In bootloader cfg)

Is someone do this before?

If you intend to pass both through to guests, it shouldn't matter.  If you use pci-stub.ids= with the single vendor:device ID, both devices will be claimed.  You can then individually more them over to vfio-pci.  Multiple identical devices is usually only a problem if you intend one to be used by the host.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1307 2014-03-09 23:37:34

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! I managed to successfully pass thru my Sparkle GTS450 1024M GPU to Windows 8.1 VM. But now I'm trying to do the same with OSX 10.9 guest:

spicec --host 127.0.0.1 --port 5908 & sudo qemu-system-x86_64 \ -enable-kvm \ -bios '/opt/Qemu/OSX/bios-1.7.4-osx.bin' \ -smp 2,sockets=1,cores=2,threads=1 \ -spice port=5908,disable-ticketing \ -chardev spicevmc,id=vdagent,name=vdagent \ -device virtio-serial \ -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \ -M q35 \ -m 4096 \ -cpu core2duo \ -rtc base=localtime \ -vga none \ -device ich9-usb-uhci1,id=usb1,bus=pcie.0 \ -usb -usbdevice host:15ca:00c3 \ -net nic,macaddr=00:00:00:8a:2b:2c,model=virtio,id=netkvm \ -net bridge \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,addr=1a.0,bus=pcie.0 \ -drive file='/opt/Qemu/OSX/osx.img',id=hd,format=raw -device ide-hd,bus=ide.0,drive=hd

and get message (after connecting to VM thru ssh):

NVDA,Display-A: Not usable NVDA,Display-B: Not usable

Also, when I try to do it using AMD 7770 GPU - Chameleon just hangs with a lot of unreadable symbols.
Could you please help me to solve this problem or maybe try to do the same youself?

P.S.: In the past I used pci-assign and successfully passed my AMD GPU to VM: https://www.linux.org.ru/gallery/screen … 6#comments

Last edited by mastersplinter777 (2014-03-09 23:40:32)

Offline

#1308 2014-03-10 01:49:09

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
AKSN74 wrote:

Here is my question.

If I want to pass-through 2 GPUs into 2 VMs (each VM have 1 GPU), and these GPU is same (Ex. Both GTX480)
Is VFIO or pci-stub will got 'conflict' when I entered hardware ID into pci-stub or VFIO?
Because same GPU have same hardware ID, and pci-stub.ids is input hardware ID only. (In bootloader cfg)

Is someone do this before?

If you intend to pass both through to guests, it shouldn't matter.  If you use pci-stub.ids= with the single vendor:device ID, both devices will be claimed.  You can then individually more them over to vfio-pci.  Multiple identical devices is usually only a problem if you intend one to be used by the host.

Roger that! I'll try in these days, thanks for ur answers.

Offline

#1309 2014-03-10 07:21:44

swex
Member
Registered: 2014-03-09
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yea! Second day and I've got it working.
Asrock Z77 Extreme3
Intel Core i7-3770
Nvidia Geforce GT640 - host
AMD Radeon HD 7750
git qemu - no patches
patched ACS kubuntu 14.04  kernel
patched nvidia host driver.

Offline

#1310 2014-03-10 10:11:10

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to get 3.14rc6 work with dirty debug registers patches (http://www.spinics.net/lists/kvm/msg100840.html). A minor annoyance is that the first patch doesn't apply cleanly, I have to do it manually. But the biggest issue is that kernel fails to compile:

arch/x86/kvm/vmx.c: In function ‘vmx_sync_dirty_debug_regs’: arch/x86/kvm/vmx.c:5155:2: error: implicit declaration of function ‘get_debugreg’ [-Werror=implicit-function-declaration] get_debugreg(vcpu->arch.db[0], 0); ^

Can somebody help me?

Offline

#1311 2014-03-10 20:22:18

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
keatsfonam wrote:

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.


I was perhaps a tad to fast saying that my GTX670 was pass through perfectly. I also get the dreaded error code 43 error people have been having.
I keep see references to a patch of yours aw, but is this already implemented upstream or do I have to compile "qemu-vfio-vfio-pci-for-qemu-20131003.0" for my arch install? (Quoting a post you made a couple of weeks ago)

When I boot my Win8.1 install I get the following in journalctl:

Mar 10 22:10:45 arch kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Mar 10 22:10:45 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Mar 10 22:10:54 arch kernel: kvm: zapping shadow pages for mmio generation wraparound Mar 10 22:10:56 arch kernel: vfio-pci 0000:00:19.0: irq 48 for MSI/MSI-X

I use the following script to start my VM:

#!/bin/bash # bind gtx 670, gpu audio and secondary network card echo -n "Binding devices..." sudo vfio-bind 0000:01:00.0 0000:01:00.1 0000:00:19.0 echo "done" echo "Starting Synergy" sudo systemctl start synergys@trimm echo "Done" echo -n "Disabling display VGA1..." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --off # use pulseaudio #export QEMU_AUDIO_DRV=pa echo -n "Starting VIRTUAL machine..." sudo qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 16384 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:19.0,bus=pcie.0 echo "Done" sudo systemctl stop synergys@trimm echo -n "Restoring display VGA1.." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --mode 1920x1080 --pos 0x0 --rotate normal echo "done"

My setup at the moment is perfect for me, the only (small?) problem is the error code 43 issue.
Any help is greatly appreciated.

Offline

#1312 2014-03-10 20:39:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:
aw wrote:
keatsfonam wrote:

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.


I was perhaps a tad to fast saying that my GTX670 was pass through perfectly. I also get the dreaded error code 43 error people have been having.
I keep see references to a patch of yours aw, but is this already implemented upstream or do I have to compile "qemu-vfio-vfio-pci-for-qemu-20131003.0" for my arch install? (Quoting a post you made a couple of weeks ago)

When I boot my Win8.1 install I get the following in journalctl:

Mar 10 22:10:45 arch kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Mar 10 22:10:45 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Mar 10 22:10:54 arch kernel: kvm: zapping shadow pages for mmio generation wraparound Mar 10 22:10:56 arch kernel: vfio-pci 0000:00:19.0: irq 48 for MSI/MSI-X

I use the following script to start my VM:

#!/bin/bash # bind gtx 670, gpu audio and secondary network card echo -n "Binding devices..." sudo vfio-bind 0000:01:00.0 0000:01:00.1 0000:00:19.0 echo "done" echo "Starting Synergy" sudo systemctl start synergys@trimm echo "Done" echo -n "Disabling display VGA1..." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --off # use pulseaudio #export QEMU_AUDIO_DRV=pa echo -n "Starting VIRTUAL machine..." sudo qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 16384 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:19.0,bus=pcie.0 echo "Done" sudo systemctl stop synergys@trimm echo -n "Restoring display VGA1.." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --mode 1920x1080 --pos 0x0 --rotate normal echo "done"

My setup at the moment is perfect for me, the only (small?) problem is the error code 43 issue.
Any help is greatly appreciated.

are you using my qemu-git and seabios-git packages?

Offline

#1313 2014-03-10 21:07:27

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Blind Tree Frog wrote:

1) Do I need multiple video cards or can this be done with just one? ...

2) Any source I should look at to set this up from a fresh install? ...

I also plan to build such a machine, with essentially headless host and Windows running on selected cores with VGA passthrough. For the time being my plan is to use the newest stable kernel and rely on top post in this thread, since it seems to be kept up-to date (thanks nbhs !)

I've been swamped lately and I'm only now getting back to this thread, but I noticed that it looks like my first question might be doable.  So 'yay!' for that.  Skimming also seems to suggest that powering through the first post is going to be my best option for the second.

I'm planning on reinstalling my machine this week and give this another go.  Hopefully I'll get it working and be able to report back with great success.  I'll be sure to take notes.

Offline

#1314 2014-03-10 21:23:09

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

RIght now im passing through my primary card (radeon 6950) what i did was, use pci-stub to bind it at boot and i disabled the efifb framebuffer with video=efifb:off on my grub cmdline

While using VFIO or no?

Just checking since you said this earlier...

nbhs wrote:

You should set you primary boot device as IGP instead of pcie, vfio-vga wont work on you primary vga device.

... and I wasn't sure if you've learned a different approach or if you found a setting that you didn't know about then (I'm assuming you just found the 'efifb' setting you mention above)

EDIT:
and how did i never notice post 40 about you doing this in XEN.... bah... reading is hard apparently.

Last edited by Blind Tree Frog (2014-03-10 21:45:57)

Offline

#1315 2014-03-10 21:23:37

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
trimm wrote:
aw wrote:

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.


I was perhaps a tad to fast saying that my GTX670 was pass through perfectly. I also get the dreaded error code 43 error people have been having.
I keep see references to a patch of yours aw, but is this already implemented upstream or do I have to compile "qemu-vfio-vfio-pci-for-qemu-20131003.0" for my arch install? (Quoting a post you made a couple of weeks ago)

When I boot my Win8.1 install I get the following in journalctl:

Mar 10 22:10:45 arch kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Mar 10 22:10:45 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Mar 10 22:10:54 arch kernel: kvm: zapping shadow pages for mmio generation wraparound Mar 10 22:10:56 arch kernel: vfio-pci 0000:00:19.0: irq 48 for MSI/MSI-X

I use the following script to start my VM:

#!/bin/bash # bind gtx 670, gpu audio and secondary network card echo -n "Binding devices..." sudo vfio-bind 0000:01:00.0 0000:01:00.1 0000:00:19.0 echo "done" echo "Starting Synergy" sudo systemctl start synergys@trimm echo "Done" echo -n "Disabling display VGA1..." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --off # use pulseaudio #export QEMU_AUDIO_DRV=pa echo -n "Starting VIRTUAL machine..." sudo qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 16384 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:19.0,bus=pcie.0 echo "Done" sudo systemctl stop synergys@trimm echo -n "Restoring display VGA1.." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --mode 1920x1080 --pos 0x0 --rotate normal echo "done"

My setup at the moment is perfect for me, the only (small?) problem is the error code 43 issue.
Any help is greatly appreciated.

are you using my qemu-git and seabios-git packages?


I am afraid not, this is the result while using qemu-git and seabios-git: qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

Last edited by trimm (2014-03-10 21:24:42)

Offline

#1316 2014-03-10 21:26:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:
nbhs wrote:
trimm wrote:

I was perhaps a tad to fast saying that my GTX670 was pass through perfectly. I also get the dreaded error code 43 error people have been having.
I keep see references to a patch of yours aw, but is this already implemented upstream or do I have to compile "qemu-vfio-vfio-pci-for-qemu-20131003.0" for my arch install? (Quoting a post you made a couple of weeks ago)

When I boot my Win8.1 install I get the following in journalctl:

Mar 10 22:10:45 arch kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Mar 10 22:10:45 arch kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Mar 10 22:10:54 arch kernel: kvm: zapping shadow pages for mmio generation wraparound Mar 10 22:10:56 arch kernel: vfio-pci 0000:00:19.0: irq 48 for MSI/MSI-X

I use the following script to start my VM:

#!/bin/bash # bind gtx 670, gpu audio and secondary network card echo -n "Binding devices..." sudo vfio-bind 0000:01:00.0 0000:01:00.1 0000:00:19.0 echo "done" echo "Starting Synergy" sudo systemctl start synergys@trimm echo "Done" echo -n "Disabling display VGA1..." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --off # use pulseaudio #export QEMU_AUDIO_DRV=pa echo -n "Starting VIRTUAL machine..." sudo qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 16384 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:19.0,bus=pcie.0 echo "Done" sudo systemctl stop synergys@trimm echo -n "Restoring display VGA1.." xrandr --output VIRTUAL1 --off --output DP1 --off --output HDMI3 --mode 1920x1080 --pos 1920x0 --rotate normal --output HDMI2 --off --output HDMI1 --off --output VGA1 --mode 1920x1080 --pos 0x0 --rotate normal echo "done"

My setup at the moment is perfect for me, the only (small?) problem is the error code 43 issue.
Any help is greatly appreciated.

are you using my qemu-git and seabios-git packages?


I am afraid not, this is the result while using qemu-git and seabios-git: qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

Oh yeah my bad, i forgot to update it on the first post, try this one qemu-git.tar.gz
Sorry again its this one: qemu-git.tar.gz

Last edited by nbhs (2014-03-10 21:33:17)

Offline

#1317 2014-03-10 21:43:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GNA wrote:

/sys/kernel/iommu_groups is empty.
My system is supporting vt-d and it should be enabled. I dont get it.
Did i miss something obvious? Thanks for help!

perhaps you forgot to boot with intel_iommu=on

Offline

#1318 2014-03-10 21:53:09

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
trimm wrote:
nbhs wrote:

are you using my qemu-git and seabios-git packages?


I am afraid not, this is the result while using qemu-git and seabios-git: qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

Oh yeah my bad, i forgot to update it on the first post, try this one qemu-git.tar.gz
Sorry again its this one: qemu-git.tar.gz

Tried with your second qemu-git and get the following:

cp: cannot stat '/home/thor/Downloads/qemu-git/src/qemu-git-2.0.r31488.gb304bf0/.libs/libcacard.so*': No such file or directory
==> ERROR: A failure occurred in package_libcacard-git().
    Aborting...

Edit: I tried starting my virtual machine(after I installed the newest qemu-git and ignoring libcacard-git) but now I got issues with my hard drive. "Booting from Hard Disk: boot failed, could not read the boot disk"

Last edited by trimm (2014-03-10 22:14:25)

Offline

#1319 2014-03-10 22:14:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:
nbhs wrote:
trimm wrote:

I am afraid not, this is the result while using qemu-git and seabios-git: qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

Oh yeah my bad, i forgot to update it on the first post, try this one qemu-git.tar.gz
Sorry again its this one: qemu-git.tar.gz

Tried with your second qemu-git and get the following:

cp: cannot stat '/home/thor/Downloads/qemu-git/src/qemu-git-2.0.r31488.gb304bf0/.libs/libcacard.so*': No such file or directory
==> ERROR: A failure occurred in package_libcacard-git().
    Aborting...

I downloaded the second package i linked and it builds and installs fine ( qemu-git.tar.gz ), and since you also managed to build the first package i posted, perhaps you're working on an unclean folder, try deleting the qemu-git folder, unpack my package and build it again

Offline

#1320 2014-03-10 22:36:08

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
trimm wrote:
nbhs wrote:

Oh yeah my bad, i forgot to update it on the first post, try this one qemu-git.tar.gz
Sorry again its this one: qemu-git.tar.gz

Tried with your second qemu-git and get the following:

cp: cannot stat '/home/thor/Downloads/qemu-git/src/qemu-git-2.0.r31488.gb304bf0/.libs/libcacard.so*': No such file or directory
==> ERROR: A failure occurred in package_libcacard-git().
    Aborting...

I downloaded the second package i linked and it builds and installs fine ( qemu-git.tar.gz ), and since you also managed to build the first package i posted, perhaps you're working on an unclean folder, try deleting the qemu-git folder, unpack my package and build it again

Weird, I tried to delete the old folder and build it again but the issue persists.
However, I installed the qemu-git package, rebooted and now my GPU is working in WIndows 8.1VM without any errors. Thank you so much nbhs!

Offline

#1321 2014-03-11 10:12:25

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zoop2 wrote:

From here i am looking to implement the Debug registers patch that was mentioned by @aw a while back to improve performance.  I am wondering what kernel that Patch was for, I am assuming it was for 3.13.x.

The second debug register patchset that @aw posted (the generic one that works on AMD as well) is definitely for a newer kernel than 3.13.x. I used linux-mainline from AUR (3.14rc6 I think) and managed to get it compiling, but you'll need to fix a couple of the patches.

Actually, here... I've uploaded my changes to the PKGBUILD/patches here. https://gist.github.com/anonymous/9482833

You'll need to download linux-mainline from the AUR, extract it, copy all of the .patch files from linux-mainline.tar.gz (the one in OP, not in AUR) into the new folder, replace the PKGBUILD with the one supplied, download all of the patches from that gist, then it should compile fine.

I'm not quite sure what version that patchset was made for. I compared the diffs against the latest git and it still didn't match up. The diff files had lines/changes that didn't exist in the current git. But I don't know much about the kernel release/cvs structure, so maybe there's a newer version than the main git.

Offline

#1322 2014-03-11 11:35:53

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FruitieX wrote:

To pereczes and zzz3000: I initially had issues with crashing (and refusing to boot again) right after Windows would either automatically install the video drivers or after I'd do so manually (Catalyst). To fix this I:
1. Compiled latest qemu and seabios from git, used these. Had to comment out an assertion otherwise the code wouldn't compile. (probably not a great idea smile)
2. Passed through only the video card, not the HDMI audio interface on it. I'm not sure if this actually helped the cause, I'll do some further testing once I get back home tonight. In the qemu version found in Arch repos this would cause the passed through display to show garbled output but it works in the git version.

@FruitieX: not clear from your post: crashing of host or guest? I assume host.

cloned git repo, built it (version 1.7.50)... BTW would be lovely if qemu people would embedd some extra commit version info into the version string
removed the

-device vfio-pci,host=04:00.1,bus=root.1,addr=0.1

line (HDMI sound)

but no changes... host is freezing once windows is trying to activate the catalyst driver (I assume that is the moment)

would be interesting to know what could be traced at this moment for easier investigation

Offline

#1323 2014-03-11 15:32:04

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:

I'm trying to get 3.14rc6 work with dirty debug registers patches (http://www.spinics.net/lists/kvm/msg100840.html). A minor annoyance is that the first patch doesn't apply cleanly, I have to do it manually. But the biggest issue is that kernel fails to compile:

arch/x86/kvm/vmx.c: In function ‘vmx_sync_dirty_debug_regs’: arch/x86/kvm/vmx.c:5155:2: error: implicit declaration of function ‘get_debugreg’ [-Werror=implicit-function-declaration] get_debugreg(vcpu->arch.db[0], 0); ^

Can somebody help me?

I just uploaded a new kernel package on the front page (3.13.6) that includes this patch

Offline

#1324 2014-03-11 18:15:43

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I just uploaded a new kernel package on the front page (3.13.6) that includes this patch

Since I am not using arch, I tried to apply fix_reading_of_DR6.patch and debug_registers.patch from your linux-mainline tarball on top of 3.13.6, but if fails with

 CC [M] arch/x86/kvm/../../../virt/kvm/vfio.o arch/x86/kvm/vmx.c: In function 'vmx_sync_dirty_debug_regs': arch/x86/kvm/vmx.c:5124:2: error: implicit declaration of function 'get_debugreg' [-Werror=implicit-function-declaration] make[3]: `arch/x86/realmode/rm/realmode.bin' is up to date. CC [M] arch/x86/kvm/../../../virt/kvm/assigned-dev.o cc1: some warnings being treated as errors make[2]: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html [arch/x86/kvm/vmx.o] Error 1 make[2]: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html Waiting for unfinished jobs.... CHK kernel/config_data.h make[1]: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html [arch/x86/kvm] Error 2 make: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html [arch/x86] Error 2 make: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_5.html topic_6.html topic_7.html topic_8.html topic_9.html Waiting for unfinished jobs....

Anything I am missing?

EDIT: Adding "#include <asm/debugreg.h>" to arch/x86/kvm/vmx.c fixed the error. Maybe you could update the patch?

Last edited by Flyser (2014-03-11 18:27:53)

Offline

#1325 2014-03-11 18:22:38

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I just uploaded a new kernel package on the front page (3.13.6) that includes this patch

B-but, but! 3.13 doesn't support hv-time. sad

Offline

#1326 2014-03-11 21:13:09

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

trying to pass through a Radeon HD 5770, but can't get it to work

Using this as the only GPU (i.e. start qemu with -vga none) completely freezes the host some time during the boot of the guest. If I get a glimpse of dmesg on the host, it shows errors of some PCI device. Sometimes the AHCI controller, sometimes the USB controller. Sometimes I get "[R600] flush TLB failed".

If I pass the GPU as the secondary device (-vga std), it works with a Linux guest. I can start X and run glxgears hardware accelerated.

It does not work, however, with a Windows 7 guest. Device Manager shows error 10 (can not start device). If I disable the device then scan for hardware changes, Windows BSODs with bugcheck 0x000000F4 (some system thread/process stopped). (Edit: this has nothing to do with GPU passthrough. It also happens, if I don't pass through the GPU.)

Chipset: AMD 990FX
Kernel: 3.14.0-rc5
Qemu: qemu-git as of today
GPU:
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]

I run qemu as root with the following command line :

~/dev/kvm-gpu-passthrough/qemu/x86_64-softmmu/qemu-system-x86_64 \ -nodefaults \ -runas spielzeug \ -monitor unix:monitor,server,nowait \ -L ~/dev/kvm-gpu-passthrough/qemu/pc-bios \ -drive file=spielzeug.img,if=virtio,cache=none,media=disk \ -boot order=c \ -smp 4 \ -cpu host \ -m 4096M \ -net nic,model=virtio,macaddr=52:54:00:12:34:57 \ -net tap,ifname=tap0,script=no,downscript=no \ -localtime \ -enable-kvm \ -M q35 \ -sdl \ -vga std \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=0000:06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=0000:06:00.1,bus=root.1,addr=00.1 \ -usbdevice tablet

I've already tried different things, like assigning the audio device to pcie.0, using other cpu types, providing a biosfile, but none of these fixed the problems.

Edit
I've now switched around my GPUs and pass through a Radeon HD 6450. This works! I can boot the guest with this GPU as the only/primary GPU and install the AMD Catalyst driver in the guest. So it might be that the HD 5770 just does some naughty stuff. I have to use different PCI ports, however, because of space reasons. So this might also be a factor.

I still can't use the passed-through GPU as secondary GPU. It now says Code 12 (not enough free resources that it can use).

I'd still really like to pass through the HD 5770. So my original problem remains.

Last edited by rawuza (2014-03-14 11:39:36)

Offline

#1327 2014-03-11 23:57:32

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rawuza wrote:

Using this as the only GPU (i.e. start qemu with -vga none) completely freezes the host some time during the boot of the guest. If I get a glimpse of dmesg on the host, it shows errors of some PCI device. Sometimes the AHCI controller, sometimes the USB controller. Sometimes I get "[R600] flush TLB failed".

This might sound random, but I had similar problems with 3.14-rc4, so you might try 3.13.6 and see if it works better.

Offline

#1328 2014-03-12 02:53:55

zoop2
Member
Registered: 2014-03-08
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
nbhs wrote:

I just uploaded a new kernel package on the front page (3.13.6) that includes this patch

B-but, but! 3.13 doesn't support hv-time. sad

I thought that the hv-time was part of qemu Vs the kernel.  I know when I used the qemu-git version it became available and improved performance for my system.  I am currently using a 3.12 kernel.

Offline

#1329 2014-03-12 04:36:46

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GizmoChicken wrote:
nbhs wrote:

That is good news! i wonder if they'll release an updated bios for the rest of their boards with this issue

I recently learned that, in addition to the fix for my M5A99FX PRO R2.0 motherboard, ASUS issued a similar "test BIOS" for a Sabertooth 990FX R2.0.  I don't know for certain what ASUS has planned for other motherboards, but I'm under the impression ASUS will at least make an effort to add the fix to future BIOS releases... perhaps not the next stable BIOS release, but eventually.

Late update on this, but I just updated my bios to the latest (late fall of last year. v2103) which came after his post.  If I'm reading dmesg correctly it looks like the test bios rolled into the official.

[Pandora]:: dmesg | grep -i AMD ... [ 1.108304] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.108305] AMD-Vi: Interrupt remapping enabled [ 1.118270] AMD-Vi: Lazy IO/TLB flushing enabled [ 1.199374] perf: AMD IBS detected (0x0000001f) [ 1.271386] QUIRK: Enable AMD PLL fix [ 1.840492] ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 2.512930] ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 2.635756] ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround [ 5.197053] AMD64 EDAC driver v3.4.0 [ 5.206768] EDAC amd64: DRAM ECC disabled. [ 5.206777] EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.

So yeah, those ASUS people who don't update their BIOS's often might want to look into updating.

Last edited by Blind Tree Frog (2014-03-12 04:47:58)

Offline

#1330 2014-03-12 05:10:41

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zoop2 wrote:

I thought that the hv-time was part of qemu Vs the kernel.  I know when I used the qemu-git version it became available and improved performance for my system.  I am currently using a 3.12 kernel.

Actually it needs both qemu and kernel bits. Some of kernel bits can be found here: https://git.kernel.org/cgit/virt/kvm/kv … 7421370f00 (dunno if it's all)

Offline

#1331 2014-03-12 10:36:07

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
rawuza wrote:

Using this as the only GPU (i.e. start qemu with -vga none) completely freezes the host some time during the boot of the guest. If I get a glimpse of dmesg on the host, it shows errors of some PCI device. Sometimes the AHCI controller, sometimes the USB controller. Sometimes I get "[R600] flush TLB failed".

This might sound random, but I had similar problems with 3.14-rc4, so you might try 3.13.6 and see if it works better.

I've already tried with 3.12 and 3.13. Doesn't make a difference.

Offline

#1332 2014-03-12 14:22:25

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@pereczes, zzz3000, rawuza

To those passing the radeon HD series and getting the freeze or BSOD; try not passing the audio vfio device ie. 06:00.1 Audio device: Advanced Micro Device or 04:00.1 Audio device: Advanced Micro Device. just leave out the -device vfio-pci,host=0000:06:00.1,bus=root.1,addr=00.1 line out.  I had the same problem with my HD7970.  There are some comments about this earlier in the thread; perhaps they need to be added to the first page on hints/problems?

I still get acceptable audio through alsa; though I do get

audio: Failed to create voice `adc' ALSA lib pcm.c:7843:(snd_pcm_recover) underrun occurred

in the console, with the ALSA error occuring every time there's sound.

Also, FYI all, ASUS has released an official bios for the RIVE supporting vt-d for IvyBridge, bios 4804, replacing the beta bios 9502.  Seems to work fine.

Last edited by mostlyharmless (2014-03-12 14:28:26)

Offline

#1333 2014-03-12 20:41:43

neona
Member
Registered: 2014-02-25
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since I built the new kernel/qemu from the OP, I've had issues with system-wide lag spikes.

This affects everything from games to audio playback (though happens more often and is more annoying in games).  Some spikes are short and cause fps dips/freezes, some are long, multiple seconds, and sometimes cause guest graphics driver crashes, and in one instance a bluescreen (I would guess because it causes things to not respond quickly enough).

How bad it is varies by game/activity, but it's making Titanfall a lot more difficult to play for me tongue

Anyone know what might be causing it?  My guess is that the debug register patches aren't entirely stable, at least on amd systems.

Offline

#1334 2014-03-12 20:53:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

neona wrote:

Since I built the new kernel/qemu from the OP, I've had issues with system-wide lag spikes.

This affects everything from games to audio playback (though happens more often and is more annoying in games).  Some spikes are short and cause fps dips/freezes, some are long, multiple seconds, and sometimes cause guest graphics driver crashes, and in one instance a bluescreen (I would guess because it causes things to not respond quickly enough).

How bad it is varies by game/activity, but it's making Titanfall a lot more difficult to play for me tongue

Anyone know what might be causing it?  My guess is that the debug register patches aren't entirely stable, at least on amd systems.

have you tried:

echo "options kvm-amd npt=0" > /etc/modprobe.d/kvm-amd.conf

Offline

#1335 2014-03-12 21:24:09

neona
Member
Registered: 2014-02-25
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

have you tried:

echo "options kvm-amd npt=0" > /etc/modprobe.d/kvm-amd.conf

Already running with it.

Offline

#1336 2014-03-12 21:47:51

neona
Member
Registered: 2014-02-25
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, it looks like my setup may have completely fallen apart.

Since bluescreening, I get *constant* microstutter.  Previously, I got this when qemu used alsa instead of pulse, but only while sounds were playing.. and of course, pulse has been misbehaving, so I just run it as root and deal with the host effectively not having sound.

Considering trying removing sound from the equation entirely, though I'm getting stutter even during os load animations, so i'm not hopeful. sad

Offline

#1337 2014-03-12 22:19:35

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

@pereczes, zzz3000, rawuza

To those passing the radeon HD series and getting the freeze or BSOD; try not passing the audio vfio device ie. 06:00.1 Audio device: Advanced Micro Device or 04:00.1 Audio device: Advanced Micro Device. just leave out the -device vfio-pci,host=0000:06:00.1,bus=root.1,addr=00.1 line out.  I had the same problem with my HD7970.  There are some comments about this earlier in the thread; perhaps they need to be added to the first page on hints/problems?

it was already suggested, tried already without tho audio, but the same (no) results wink. BTW: I have gigabyte 990fxa ud7

Offline

#1338 2014-03-12 22:34:03

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do any of you sometimes experience a total freeze of a VM when inside a game on the guest(seemingly at random, sometimes after a couple of seconds, other times after a couple of hours)? Host is ok and useable while guest is frozen.
Im not that bothered about it (at this time), however I cant seem to kill the qemu processes when this happens. Is it somehow possible to clean or reset the vm without completely restart the computer, just so that I can start it up again straight away? As of this moment, I have to reboot the whole computer and the frozen picture on the screen of the virtual machine will be the last thing that gets cleared before the reboot is completed.

Kernel: x86_64 Linux 3.13.5-1-mainline
Latest Qemu-git and Seabios-git
Nvidia GTX670, latest Nvidia driver on Windows 8.1
Synergy used between Arch Linux(server) and Windows(client)

Edit: Just tested Battlefield 4 and it froze(Everything ultra setting with 50FPS, at test range).
Ping against the VM stopped aswell, so its not only the GPU which froze.
This is the following journalctl:

Mar 13 00:40:09 arch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 13 00:40:09 arch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe780 fff99920 00000000 0d00000a 07058000
Mar 13 00:40:09 arch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 13 00:40:09 arch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe7a0 fff99930 00000000 0d00000c 07058000
Mar 13 00:40:09 arch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 13 00:40:09 arch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe7c0 fff99940 00000000 0d00000c 07058000

lspci -v
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) (prog-if 30 [XHCI])

Last edited by trimm (2014-03-12 22:56:36)

Offline

#1339 2014-03-13 00:15:40

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@pereczes
Have you tried installing the ati driver without the CCC? You'll still get 3D hardware acceleration.

Offline

#1340 2014-03-13 07:46:54

shweng
Member
Registered: 2014-03-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,all
Could any body tell me how to access the VM's desktop when successfully VGA-Passthroughed,
from virt-manager console on host, or other remote desktop access tools such as vncviewer, remote-viewer, rdesktop, freerdp etc?

Offline

#1341 2014-03-13 09:18:51

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shweng wrote:

Could any body tell me how to access the VM's desktop when successfully VGA-Passthroughed,

How about attaching a display to the graphics card you passed through?


I’m back after a few months with a new problem. I initially got this working with my good ol’ 4850 as a proof of concept while waiting for a viable custom design of the R290X. Now I finally acted and put a R290X Tri-X in there. Getting the new card passed through was straight forward and works great except for when Catalyst is supposed to kick in. Booting to safe mode, which uses the VESA drivers, works but as soon as I boot up normally the last thing I see is the Windows logo. After that the display simply shows black. It does *not* go into standby but shows a black screen. I tried both the stable and the Beta Catalyst so far; updated to latest kernel, qemu-git and seabios-git. Any ideas?

Offline

#1342 2014-03-13 12:37:56

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

@pereczes
Have you tried installing the ati driver without the CCC? You'll still get 3D hardware acceleration.

you mean Catalyst Control Center? ... well how would I do that... I mean other driver than the original one?
I assume the problem is with the driver, not with CCC, or?

Offline

#1343 2014-03-13 13:15:36

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@peterczes there were reported problem with using CCC (on GPU passthrough card) rather than with the driver


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1344 2014-03-13 14:42:20

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

@peterczes there were reported problem with using CCC (on GPU passthrough card) rather than with the driver

thanks... well little step further: uninstalled CCC, but it seems that the drivers remained installed under C:/Program Files/Amd

good news: started to work... wanted to install synergy (nice software btw if you do not know it), and:

bod news: bang... host froze again ...

will try to uninstall the driver and reinstall it... maybe helps
btw.. this experiment was without hdmi audio

I passed through a USB PCI controller as well for keyboard and mouse... after synergy will remove them, maybe it makes the system more stable

Offline

#1345 2014-03-14 19:24:12

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@op: Just downloaded the PKGBUILDs from page 1. The vfio_multifunction_regression.patch seems to be already applied upstream in quemu-git :-)

Edit: Was committed on March 9th: http://git.qemu.org/?p=qemu.git;a=commi … e4b2617c74

Last edited by TripleSpeeder (2014-03-14 19:27:44)

Offline

#1346 2014-03-14 19:32:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

@op: Just downloaded the PKGBUILDs from page 1. The vfio_multifunction_regression.patch seems to be already applied upstream in quemu-git :-)

Edit: Was committed on March 9th: http://git.qemu.org/?p=qemu.git;a=commi … e4b2617c74

Thanks!

Offline

#1347 2014-03-14 20:37:40

arny91
Member
Registered: 2014-01-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, is the MSI 990FXA-GD80 a good motherboard for use with iommu?

Offline

#1348 2014-03-14 21:12:39

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Somewhere in this very long thread there is a list of hardware with success or failure saved as a spreadsheet; perhaps a link on page 1 might be helpful.

Offline

#1349 2014-03-15 00:50:16

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I cant for the life of me sort out my audio for my VM.

I use this:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

I have also tried with -soundhw all and -soundhw ac97.

This is what I get when I fire up the computer:

pulseaudio: pa_context_connect() failed pulseaudio: Reason: Connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: No such file or directory ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: No such file or directory audio: Failed to create voice `dac' ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave alsa: Could not initialize ADC alsa: Failed to open `default': alsa: Reason: No such file or directory ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave alsa: Could not initialize ADC alsa: Failed to open `default': alsa: Reason: No such file or directory audio: Failed to create voice `adc'

In the first post its stated that I may have to use the QEMU_AUDIO_DRV variable. I have tried this (export QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_DRV=pa as root) without success. Anyone in the same boat as me, and/or found a solution?(except passing a USB headset? smile)

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted... hmm

Last edited by trimm (2014-03-15 01:39:05)

Offline

#1350 2014-03-15 09:04:09

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One more success story here. This is all done on mint 16 by the way.
Hardware: Sabertooth FX990 R2.0 motherboard, AMD FX(tm)-8350 CPU, host GPU geforce gtx 550ti, VM GPU Radeon R9 270X.
qemu: from git
kernel: 3.13.6

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
~/VM/src/qemu/build/release/x86_64-softmmu/qemu-system-x86_64 \ -enable-kvm -m 4096 -cpu host -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -bios /usr/share/qemu/bios.bin \ -drive file=/dev/disk/by-id/ata-WDC_WD3200BEVT-22ZCT0_WD-WXE708J50240,if=none,id=virtio-disk0,format=raw,cache=none,aio$ -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/disk/by-id/ata-WDC_WD7502AAEX-00Y9A0_WD-WCAW31075836,if=none,id=virtio-disk1,format=raw,cache=none,aio$ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=20:64:32:c1:f9:22 -netdev tap,id=vnet0 \ -smb user,smbserver=10.0.2.4 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -vga none

Some observations i made:
To correctly pass-through USB controller i had to include both 00:13.0 and 00:13.2, othervise qemu threw an error about failing to bind vfio or something.
To find correct address of USB ports i wanted i used stupid method: plugged in mouse to target port and then go through pci addresses and vfio-bind them. When mouse lights went out i knew usb port was no longer working so that was last one i bound. Then had to reboot to make them come back. There probably is better way..
I could use -vga sdl along with passed-through GPU. Using stock 1.5 qemu from repos did not quite work. Device manager reported radeon had some kind of malfunction. Compiling qemu from git made it work. Both emulated vga and passed-through one side by side. 3d apps did not run on radeon vga in that mode though and i did not care to find out how to make that happen.
Everything else went pretty much as described in original guide.
Well and bit offtopic - to use virtio for disk i had to boot OS while disk with windows was attached with -hda and then i had to attach dummy few meg image in virtio mode. That way windows picked up new device and i could install drivers. After that all disks can be attached in virtio.

Now this brings me to problems.. I am too having this issue where after reboot vga will not reset and VM with passed-through vga can not be started again. As far as i read this thread did not see a solution. Correct me if im blind and i missed it.
Also performance issue led me to believe i should use these patches:

For what kernel version are they? I fail to apply them because latest tree does not quite match code in these patches. Not entirely sure if its a good idea to apply these patches in this case.


EDIT:
After some more digging i figured out patches are for latest 3.14-rc6 kernel. Thing is after kernel update i still can not see reasonable speed improvement. I used passmarkt to test and only dx11 test seems to run at ~80% of speed compared to other users. dx9 is especially slow. Basically all other 3d tests are somewhere around 20-30% of other people's results. Really slow.. If anyone knows possible cure - hint me please!

Also i forgot to mention that if i run VM then host will not shut down properly. It hangs after bunch of "synchronizing SCISI cache" messages.
Also had to flib-back to nouveau driver for host as latest nvidia beta driver (which is supposed to work with rc6) wont install (hangs without displaying anything upon start). Anyone happen to install that beast?

Last edited by novist (2014-03-15 15:45:39)

Offline

#1351 2014-03-15 18:45:17

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On my end the VGA passthrough is working quite fine for some time now. I am thinking how to integrate this with libvirt and virtmanager instead of configuring/running quemu manually. So i'd like to use libvirtd and virtmanager to setup, configure, start, stop etc. my VMs with VGA passthrough using vfio. Anyone here have experience? Is this already possible?

Offline

#1352 2014-03-15 18:52:00

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Should be possible already.  There might be a custom line you need to add to the XML (or two) but I don't remember where in the thread it was discussed.  I know that people are doing what you ask though.

Offline

#1353 2014-03-15 20:12:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

One more success story here. This is all done on mint 16 by the way.
Hardware: Sabertooth FX990 R2.0 motherboard, AMD FX(tm)-8350 CPU, host GPU geforce gtx 550ti, VM GPU Radeon R9 270X.
qemu: from git
kernel: 3.13.6

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
~/VM/src/qemu/build/release/x86_64-softmmu/qemu-system-x86_64 \ -enable-kvm -m 4096 -cpu host -machine q35,accel=kvm \ -smp 8,sockets=1,cores=8,threads=1 \ -bios /usr/share/qemu/bios.bin \ -drive file=/dev/disk/by-id/ata-WDC_WD3200BEVT-22ZCT0_WD-WXE708J50240,if=none,id=virtio-disk0,format=raw,cache=none,aio$ -device virtio-blk-pci,scsi=off,drive=virtio-disk0,id=disk0 \ -drive file=/dev/disk/by-id/ata-WDC_WD7502AAEX-00Y9A0_WD-WCAW31075836,if=none,id=virtio-disk1,format=raw,cache=none,aio$ -device virtio-blk-pci,scsi=off,drive=virtio-disk1,id=disk1 \ -device e1000,netdev=vnet0,mac=20:64:32:c1:f9:22 -netdev tap,id=vnet0 \ -smb user,smbserver=10.0.2.4 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -vga none

Some observations i made:
To correctly pass-through USB controller i had to include both 00:13.0 and 00:13.2, othervise qemu threw an error about failing to bind vfio or something.
To find correct address of USB ports i wanted i used stupid method: plugged in mouse to target port and then go through pci addresses and vfio-bind them. When mouse lights went out i knew usb port was no longer working so that was last one i bound. Then had to reboot to make them come back. There probably is better way..
I could use -vga sdl along with passed-through GPU. Using stock 1.5 qemu from repos did not quite work. Device manager reported radeon had some kind of malfunction. Compiling qemu from git made it work. Both emulated vga and passed-through one side by side. 3d apps did not run on radeon vga in that mode though and i did not care to find out how to make that happen.
Everything else went pretty much as described in original guide.
Well and bit offtopic - to use virtio for disk i had to boot OS while disk with windows was attached with -hda and then i had to attach dummy few meg image in virtio mode. That way windows picked up new device and i could install drivers. After that all disks can be attached in virtio.

Now this brings me to problems.. I am too having this issue where after reboot vga will not reset and VM with passed-through vga can not be started again. As far as i read this thread did not see a solution. Correct me if im blind and i missed it.
Also performance issue led me to believe i should use these patches:

For what kernel version are they? I fail to apply them because latest tree does not quite match code in these patches. Not entirely sure if its a good idea to apply these patches in this case.


EDIT:
After some more digging i figured out patches are for latest 3.14-rc6 kernel. Thing is after kernel update i still can not see reasonable speed improvement. I used passmarkt to test and only dx11 test seems to run at ~80% of speed compared to other users. dx9 is especially slow. Basically all other 3d tests are somewhere around 20-30% of other people's results. Really slow.. If anyone knows possible cure - hint me please!

Also i forgot to mention that if i run VM then host will not shut down properly. It hangs after bunch of "synchronizing SCISI cache" messages.
Also had to flib-back to nouveau driver for host as latest nvidia beta driver (which is supposed to work with rc6) wont install (hangs without displaying anything upon start). Anyone happen to install that beast?

Have you tried this?

echo "options kvm-amd npt=0" > /etc/modprobe.d/kvm-amd.conf

Offline

#1354 2014-03-16 09:17:41

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes it is there already. Maybe i will make summary of my modprobe.d configs and grub params:
kvm-amd.conf

options kvm-amd npt=0

kvm.conf

#options kvm ignore_msrs=1 options kvm allow_unsafe_assigned_interrupts=1

qemu-system-x86.conf

options kvm_intel nested=1

vfio_iommu_type1.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

GRUB:

GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 radeonhd.blacklist=1 nouveau.blacklist=0 nouveau.modeset=1 nvidia.blacklist=1 nvidia.modeset=0 iommu=pt iommu=1 pci-stub.ids=1002:6810,1002:aab0 vfio_iommu_type1.allow_unsafe_interrupts=1"

blacklist.conf

blacklist radeon blacklist radeonhd #blacklist nouveau blacklist nvidia

I probably overstretched it a bit. Some double-configs probably are not really needed but im not sure. Not an expert on linux kernel things here. I maybe even would want to move all modprobe configs related to kvm to same file but again idk if its good idea.

EDIT:
Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
ictrgo.png

Last edited by novist (2014-03-16 10:04:57)

Offline

#1355 2014-03-16 11:54:20

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for this great guide. I immediatelly got VGA passthrough working perfectly with Debian Jessie guest.

Setup:
AMD A10-5700 APU
Asus F2A85-M PRO, BIOS 6204
Host: Radeon HD 7660D IGP (Gallium, Northern Islands)
Guest: Radeon HD 3870 (Gallium, R600)
Linux 3.12.14-gentoo
Qemu 1.7.0 (Gentoo portage)
Seabios 1.7.3.2 (Gentoo portage)

I did not apply patches of any kind.

These are needed for running as non-root (numbers may vary and I choose that 3GiB arbitrarily):
chgrp kvm /dev/vfio/2
chmod g+rw /dev/vfio/2
echo "@kvm - memlock 3145728" >> /etc/security/limits.conf


Unfortunately I ran into a problem unrelated to passthrough: I can't get WinXP 32-bit (SP3) to boot or install with "-M q35" no matter what I do! Is everyone else using newer Windows? Or what am I doing wrong?

Here is one example how to reproduce the problem:
1. qemu-system-x86_64 -machine accel=kvm -m 1024 -M q35 -drive file=/path/to/winxp_sp3.iso,if=none,id=dr0,media=cdrom -device ide-cd,bus=ide.0,drive=dr0 -net none -daemonize -vga cirrus
2. Wait ~10 seconds
3. BSOD with STOP 0xA5 appears

Above works fine without "-M q35".

Identical looking problems:
https://bugzilla.redhat.com/show_bug.cgi?id=902662
https://bugzilla.redhat.com/show_bug.cgi?id=959815

To solve the problem I have tried:
- Qemu 1.4.2, 1.5.3, 1.7.0 and git
- Seabios 1.7.2.1, 1.7.2.2, 1.7.3.2, 1.7.4, git and also the bios that came with qemu-git and also 1.7.4 with something resempling commit 2114f50148c42e374586359d23b522483ca10e8d reverted
- qemu-system-i386 and qemu-system-x86_64
- Quite a lot of different qemu options including: with and without kvm, -no-hpet, -no-acpi, various -cpu and -smp, various -vga, various memory sizes, various emulated SATA, IDE and AHCI controllers, some random -acpitables

UPDATE: I managed to get past this one ordeal. The trick is to use this virtual hd controller: -device piix4-ide,bus=pcie.0,id=piix4-ide
AND then press F7 during Windows install to disable ACPI. Any other combo won't work. So now I have WinXP working just fine with "-M q35".

Unfortunately attempts to actually boot the same WinXP with Radeon card enabled seem to lead to horrible host crashes of data corruption. Interestingly, Debian guest still works just fine.

UPDATE 2: Now using Asus BIOS 6308 and git kernel + fix_memleak.patch and qit qemu.

Still no luck with either Win7 or WinXP. Win7 works without AMD drivers but once the drivers have been installed, it crashes right after "Starting Windows". Host usually also crashes. If I start qemu from console without any graphics drivers, host seems not to crash(?), but it won't make the guest work any better. And if I accidentally do modprobe radeon after guest crash the whole system crashes? I used to get ATA-errors on host crashes, but now after some changes I don't get any error messages.

(I also found out there was no need for allow_unsafe_interrupts=1. I simply didn't have Interrupt Mapping compiled in kernel.)

Last edited by ahl (2014-03-18 05:34:27)

Offline

#1356 2014-03-16 15:38:34

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

Thank you for this great guide. I immediatelly got VGA passthrough working perfectly with Debian Jessie guest.

Setup:
AMD A10-5700 APU
Asus F2A85-M PRO, BIOS 6204
Host: Radeon HD 7660D IGP (Gallium, Northern Islands)
Guest: Radeon HD 3870 (Gallium, R600)
Linux 3.12.14-gentoo
Qemu 1.7.0 (Gentoo portage)
Seabios 1.7.3.2 (Gentoo portage)

I did not apply patches of any kind.

Required hacks:
vfio_iommu_type1.allow_unsafe_interrupts=1
chgrp kvm /dev/vfio/2
chmod g+rw /dev/vfio/2
echo "@kvm - memlock 3145728" >> /etc/security/limits.conf

I wonder what are the implications of that interrupt hack? "unsafe" doesn't sound good.


Unfortunately I ran into a problem unrelated to passthrough: I can't get WinXP 32-bit (SP3) to boot or install with "-M q35" no matter what I do! Is everyone else using newer Windows? Or what am I doing wrong?

Here is one example how to reproduce the problem:
1. qemu-system-x86_64 -machine accel=kvm -m 1024 -M q35 -drive file=/path/to/winxp_sp3.iso,if=none,id=dr0,media=cdrom -device ide-cd,bus=ide.0,drive=dr0 -net none -daemonize -vga cirrus
2. Wait ~10 seconds
3. BSOD with STOP 0xA5 appears

Above works fine without "-M q35".

Identical looking problems:
https://bugzilla.redhat.com/show_bug.cgi?id=902662
https://bugzilla.redhat.com/show_bug.cgi?id=959815

To solve the problem I have tried:
- Qemu 1.5.3, 1.7.0 and git
- Seabios 1.7.1, 1.7.2, 1.7.2.2, 1.7.3.2, 1.7.4 and also the bios that came with qemu-git and also 1.7.4 with something resempling commit 2114f50148c42e374586359d23b522483ca10e8d reverted
- qemu-system-i386 and qemu-system-x86_64
- Quite a lot of different qemu options including: with and without kvm, -no-hpet, -no-acpi, various -cpu and -smp, various -vga, various memory sizes, various emulated SATA, IDE and AHCI controllers, some random -acpitables

Any ideas?

but why would you want that old OS? i mean seriously.. its done. move on.. also q35 is still in development and kind of experimental. i suppose noone cared to test obsolete OS with it.. i suggest picking up windows7. its fastest way to get you up and running.


Also anyone having stability issues with windows VMs?  Mine for some reason restarts some time after boot. Still no idea what could be causing it. Not even BSOD, just reboots instantly. I compiled qemu with --enable-debug but it does not output any more error messages. there is lots of following spam:

qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism.

but i suppose its fine since 0000:00:13.0 is passed-through usb port and im switching between host and guest with kvm switch. this is real bummer..

Offline

#1357 2014-03-16 15:43:50

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

this thread is getting really long... would take pb weeks for someone to resume the most valuable info wink

IMPORTANT: there were to things that were making freezing my host:
-> if i wast trying to run CCC
-> but also the fact that I had the HDMI cable attached

details about the second:
I was booting windows 7.1 guest on primary monitor attached to ATI/RADEN DIV connector.
Once I was attaching the HDMI, the guest was causing the host to freeze. Crazy...

Now tried to boot with DIV removed, but HDMI attached. Though there was no boot screen,
probably the card was sending the signal only to DIV, the guest booted and worked no crash.
Now re-installed CCC, and indeed once started CCC it crashed. At high level I can conclude that
the crash is related to manipulation of VIDEO output.
So there is a huge progress for me, but I still cannot live with the situation as the hdmi signal
someway does not show full screen. I had the same problem on linux as well and only CCC could
stretch the screen to the full size (though windows reports it is at max resolution)

Offline

#1358 2014-03-16 17:20:58

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

but why would you want that old OS? i mean seriously.. its done. move on.. also q35 is still in development and kind of experimental. i suppose noone cared to test obsolete OS with it.. i suggest picking up windows7. its fastest way to get you up and running.

Thanks but I have "moved on" to Linux. Giving more money to Microsoft for them to harass me more just doesn't feel right. And more importantly, there is nothing wrong with running old software in restricted environment. It's not like I'd be surfing the net with it.

I also tried Windows 2000 Pro now. Different looking BSoD but exact same error and message.

More luck with Windows 98 SE, perhaps? (Honestly, I'm out of ideas now...)

pereczes wrote:

So there is a huge progress for me, but I still cannot live with the situation as the hdmi signal
someway does not show full screen. I had the same problem on linux as well and only CCC could
stretch the screen to the full size (though windows reports it is at max resolution)

I believe that's called underscanning and overscanning of video -- a ridiculous legacy feature of the HDMI protocol. A most reliable way to get rid of it, is to use good old VGA when possible.

Offline

#1359 2014-03-16 17:53:05

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

Thanks but I have "moved on" to Linux. Giving more money to Microsoft for them to harass me more just doesn't feel right. And more importantly, there is nothing wrong with running old software in restricted environment. It's not like I'd be surfing the net with it.

Poor support is exactly what is wrong with running old software. Suppose you survive pains of getting XP to run. It wont be long until you need software <insert name here> but it does not run on that legacy OS so upgrade is inevitable. You might try XP x64, it has rather different kernel from 32 bit counterpart (newer one). Maybe that would do the trick. By the way you can rearm Windows7 few times to extend trial to 120 days. Or you can do some other things. Im not even sure what happens when trial expires. I use one VM with expired 2008 r2 server trial and all i get is nag screen and no updates.



Now my update: i made interesting observation. I played FEAR3 for a period of time. GPU was working really loud and there was no lag whatsoever. I had no FPS counter visible so i dont know numbers but it was totally smooth on highest settings. Right after i started arma3 which lags quite a lot. Now GPU worked rather silently. Not sure what to make of it. Also this entire session lasted longer than what it took for system to go down earlier. Interesting fact is that when i had stability issues i was also running vmware with powered on VM. this time i had no VM on. So i suspect vmware might make qemu VM unstable. Will test this hypothesis in following days.

Offline

#1360 2014-03-16 19:21:53

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

everything has a cost wink ... removed the NVIDIA card, and left only powercolor 6950

not behaving anymore as used to be:
-> HDMI no signal (though not 100% sure if cable is not broken... next thing to check)
-> on HDMI port both on BIOS setup strange characters, more precisely roughly 30%
of the text is displayed with wrong characters. "Loading operating system" is displayed
correctly, though the linux grub menu is the same messed up... Radeon driver would
just freeze the system (host!!), without driver managed to log in and write this...

as I was not sure if the latest gigabyte bios would have caused this (never used booting with
the radeon since I started the experiment), I reverted it to earlier version, but no changes.

Question: as I suspect that I might have corrupted the bios with all the experiments,
is there a way to reflash the bios from linux?

thanks

Offline

#1361 2014-03-17 02:15:59

fukhda
Member
Registered: 2014-02-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I have been working on VGA passthrough for a while but no success yet. The prbolem I got is code12 for VGA card assigned to Win7 VM. Please check my work here and any comments are welcome. Thanks in advance, guys.

[Somethings to be noted first]
I am a newbie to linux and kvm. So please correct me if anything wrong.
I am working on Fedora 20, not archlinux.
The ingredients nbhs provided in post1. I do not know how to apply them properly in Fedora. (Will explain how I did it later)

[Env. setup]
CPU: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
Motherboard: ASUS B85M-E
RAM: 4GB DDR3 1600 MHz
DISK:
VGA1: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
VGA2: AMD Radeon HD 7700


[Patches]
Following is my way to apply tngredients (nbhs provided in post1) in Fedora.

1. linux-mainline.tar.gz
The base is kernel linux-3.13.3 (https://www.kernel.org/pub/linux/kernel … 3.3.tar.gz)
I manually merge the diff code (six *.patch files, such as override_for_missing_acs_capabilities.patch) into linux-3.13.3. And add CONFIG_VIFO*=y in menuconfig. Then rebuild the kernel and install.

2. qemu-git.tar.gz
The base is qemu-1.7.0 (http://wiki.qemu-project.org/download/q … .0.tar.bz2)
Just follow the configure switch according to PKGBUILD

 ./configure --prefix=/usr --sysconfdir=/etc --audio-drv-list='pa alsa sdl oss' \ --python=/usr/bin/python2 --smbd=/usr/bin/smbd \ --enable-docs --libexecdir=/usr/lib/qemu \ --disable-gtk --enable-linux-aio --enable-seccomp \ --enable-spice --localstatedir=/var make make install

3. seabios-git.tar.gz
I am not doing any change here. Just run make in qemu-1.7.0/roms/seabios

[Grub.cfg]
Add below switches in /boot/grub2/grub.cfg
pcie_acs_override=downstream intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1

So, cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.13.3 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root pcie_acs_override=downstream intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 rhgb quiet LANG=en_US.UTF-8

[Before launching VM]
1. blacklist radeon in /etc/modprobe.d/blacklist.conf

2. make vfio-bind.service according from 1st post into /etc/systemd/system
   add DEVICES="0000:04:00.0 0000:04:00.1" in /etc/vfio-pci.cfg
   /*
   || This service is not launched automatically while machine reboots.
   || Instead, I manually run vfio-bind 0000:04:00.0 0000:04:00.1 before launching VM
   */

3.

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

4.
/* vfio-bind is from 1st post */

vfio-bind 0000:04:00.0 0000:04:00.1

 

[Launch VM]
I did a clean installation using the Win7x32.iso downloaded from MSDN. Once installation completed, "-drive file=/home/test/fuhsiao/win7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd" is removed from the below script.

qemu-kvm --enable-kvm -M q35 -m 2048 \ -cpu host -smp 1,sockets=1,cores=1,threads=1 \ -bios /usr/local/src/qemu-1.7.0/roms/seabios/out/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/test/fuhsiao/test.img,id=disk,format=raw \ -drive file=/home/test/fuhsiao/win7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \ -device ide-hd,bus=ahci.0,drive=disk \ -boot c \

[The problem is?]
In win7 device manager, it showed that passed AMD card has got code 12 "The device can not find enough free resources that it can use"

[What I have tried to solve the problem]
1. Instead of passing through AMD Radeon HD 7700, I also tried NVIDIA EGVA GTX650. Still code12.

2. Diable iGPU
VGA1: iGPU is disabled from BIOS
VGA2: AMD Radeon HD 7700
VGA3: NVIDIA EGVA GTX650
Either passing AMD or NVIDIA, still code12.

3. Use the different qemu source
git://github.com/awilliam/qemu-vfio.git tags/vfio-pci-for-qemu-20131003.0
NOT the earlier http://wiki.qemu-project.org/download/q … .0.tar.bz2. Still code12.


[FYI]

lspci 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation B85 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde LE [Radeon HD 7730/8730] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] lsgroup ### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) ### Group 9 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) ### Group 10 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) ### Group 12 ### 00:1f.0 ISA bridge: Intel Corporation B85 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) ### Group 13 ### 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) ### Group 14 ### 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde LE [Radeon HD 7730/8730] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

Offline

#1362 2014-03-17 12:08:25

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, guys.
Now I got a another problem.

I'm not sure it's GTX480 problem or KVM problem.

When I entered code, KVM can start normally and it can pass my GTX480 into VM.
But when I plug DVI cable to GTX480, it got nothing but no signal. Even tried another DVI, same result.

I know KVM is working because I'm already installed TeamViewer in guest OS and it can connect, also I can see OS screen.
Does someone tried before?

Offline

#1363 2014-03-17 12:14:45

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Hello, guys.
Now I got a another problem.

I'm not sure it's GTX480 problem or KVM problem.

When I entered code, KVM can start normally and it can pass my GTX480 into VM.
But when I plug DVI cable to GTX480, it got nothing but no signal. Even tried another DVI, same result.

I know KVM is working because I'm already installed TeamViewer in guest OS and it can connect, also I can see OS screen.
Does someone tried before?

no signal sounds like your card is not passed through.

Offline

#1364 2014-03-17 13:17:50

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:
AKSN74 wrote:

Hello, guys.
Now I got a another problem.

I'm not sure it's GTX480 problem or KVM problem.

When I entered code, KVM can start normally and it can pass my GTX480 into VM.
But when I plug DVI cable to GTX480, it got nothing but no signal. Even tried another DVI, same result.

I know KVM is working because I'm already installed TeamViewer in guest OS and it can connect, also I can see OS screen.
Does someone tried before?

no signal sounds like your card is not passed through.

But it pass-through I'm sure, because Windows 7 detected and driver installation is successful.

Last edited by AKSN74 (2014-03-17 13:18:24)

Offline

#1365 2014-03-17 13:57:52

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Okay, I am out of ideas now. I got everything running perfectly when starting qemu manual. Now i want to move to a libvirt xml file, so i can start/stop/manage my VMs with virtmanager. But no matter what I do, i don't get the card passed through.
Here is a minimal xml config to reproduce:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7</name> <uuid>87e41de8-b3a5-4d76-a048-5d8d206aacc2</uuid> <memory unit='KiB'>6144000</memory> <currentMemory unit='KiB'>6144000</currentMemory> <vcpu placement='static' current='2'>4</vcpu> <os> <type arch='x86_64' machine='pc-q35-1.5'>hvm</type> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pcid'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='smx'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='tm2'/> <feature policy='require' name='ss'/> <feature policy='require' name='pbe'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <memballoon model='none'/> </devices> <seclabel type='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> </qemu:commandline> </domain>

When starting this instance e.g. through "virsh start win7" i get the following error:

# virsh start win7
error: Failed to start domain win7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What I did:

  • Make sure that AMD-card is assigned to vfio

  • Edit /etc/libvirt/qemu.conf and set user/group qemu runs as to "root"

  • Getting desperate: chmod a+rw /dev/vfio/13

But no luck. Everytime the same error :-(

Does anyone have a clue what I am missing? As I said, when I launch qemu manually (as root) everything works fine...

Offline

#1366 2014-03-17 14:12:20

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
Bronek wrote:
Blind Tree Frog wrote:

1) Do I need multiple video cards or can this be done with just one? ...

2) Any source I should look at to set this up from a fresh install? ...

I also plan to build such a machine, with essentially headless host and Windows running on selected cores with VGA passthrough. For the time being my plan is to use the newest stable kernel and rely on top post in this thread, since it seems to be kept up-to date (thanks nbhs !)

I've been swamped lately and I'm only now getting back to this thread, but I noticed that it looks like my first question might be doable.  So 'yay!' for that.  Skimming also seems to suggest that powering through the first post is going to be my best option for the second.

I'm planning on reinstalling my machine this week and give this another go.  Hopefully I'll get it working and be able to report back with great success.  I'll be sure to take notes.

I got portions of this to work last night without trouble, but proceeding much further ran into difficulty.

Using the mainline, bios, and qemu code from the first post everything looked to build correctly.  I switched kernels and passed "quiet iommu video=efifb:off pci-stub.ids=...." as the cmd line (where '....' would be the id's for my one video card and my USB devices).  I also blacklisted 'radeon' as well as enabling the service described in post one.  Reboot.  I then SSH'd in for the rest of the steps.

The good news is that I was able to boot the seagate BIOS and see the output on my screen.   It came up and said that it couldn't find any boot devices.

The bad news is that I got no further than that even if I passed a LiveCD ISO in with the '-cdrom' option; The BIOS still claimed that it couldn't find a bootable device.  Also, I had to dial back some of the USB ID's I was passing to PCI stub as they didn't seem to work when passed to qemu.  And if I didn't blacklist the radeon driver qemu wouldn't pick it up regardless of if it was in the pci_stub.ids list or not. And the vfio-bind service does seem to properly bind the USB devices, but fails to bind the video card so it needs to be run twice (Once by the system as per enabling, and once by the user after boot.  The first time might not be required honestly but the second time is).

Hopefully I'll be able to investigate some more sometime this week and get meaningful questions.

Hardware:
ASUS Sabertooth 990FX/Gen3 R2
Asus Radeon HD7770 GHz Ed.

Offline

#1367 2014-03-17 14:45:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

EDIT:
Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

What is this benchmark?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1368 2014-03-17 15:20:46

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
novist wrote:

EDIT:
Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

What is this benchmark?

passmark performance test ( https://www.passmark.com/products/pt.htm )

Offline

#1369 2014-03-17 15:46:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:
aw wrote:
novist wrote:

EDIT:
Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

What is this benchmark?

passmark performance test ( https://www.passmark.com/products/pt.htm )

Ok, last time I tried passmark the VM blue screened regardless of having an assigned GPU.  I'll try it again.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1370 2014-03-17 18:59:02

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello again guys.

I wanted to give you an quick update on my setup.
I had a 1TB raw image partitioned on a BTRFS drive and it was unbearable slow. It took ages loading stuff (however, writing did not seem that bad).

I fixed it by following Arch guide(disabling CoW): https://wiki.archlinux.org/index.php/Btrfs

$ mv /path/to/dir /path/to/dir_old $ mkdir /path/to/dir $ chattr +C /path/to/dir $ cp /path/to/dir_old/* /path/to/dir $ rm -rf /path/to/dir_old

Now its as fast as it can be and games load much, much faster.

Another thing though, and its my sound. I gave up on passing through my USB headset(crackling noise) and sharing sound between my host and guest. (I tried following the first post without success)
Im thinking I can passthrough my integrated sound on the motherboard, and I tried doing it like this:

Binding the sound:

vfio-pci 0000:00:1b.0

In my dmesg I get the following:

vfio-pci 0000:00:1b.0: kvm assign device

And in my script I have this:

-device vfio-pci,host=00:1b.0

No errors in Linux, but I get a red mark over my audio icon in Windows. Going into "sound" tab inside audio properties and the whole VM totally freezes.

I passthroughed with vfio-pci, a drive and my secondary network card without any issues, however I have not had any success with audio.
I have also passthroughed my Xbox Controller and Rocksmith USB adapter.

Any help is appreciated!

Offline

#1371 2014-03-17 20:48:31

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anybody tried and succeeded with ATIWinflash to reflash radeon card under qemu?

I tried ... but completely corrupted bios. Now booted with qxl, TechPowerUp GPU-Z can see the card but BIos Version is just a messy string.

Another strange thing is that my 6950 was showing 768 Shaders, though everybody is reporting double the ammount. It is like I would have only one core.

Offline

#1372 2014-03-17 20:54:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pereczes wrote:

anybody tried and succeeded with ATIWinflash to reflash radeon card under qemu?

I would strongly discourage this.  The ROM BAR is entirely emulated to the guest, we enable and read the ROM on the first guest access, from then on the guest only reads from the cached copy.  Any sort of ROM update utility is likely to get very, very confused by this.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1373 2014-03-17 20:59:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

Another thing though, and its my sound. I gave up on passing through my USB headset(crackling noise) and sharing sound between my host and guest. (I tried following the first post without success)
Im thinking I can passthrough my integrated sound on the motherboard, and I tried doing it like this:

Binding the sound:

vfio-pci 0000:00:1b.0

In my dmesg I get the following:

vfio-pci 0000:00:1b.0: kvm assign device

And in my script I have this:

-device vfio-pci,host=00:1b.0

No errors in Linux, but I get a red mark over my audio icon in Windows. Going into "sound" tab inside audio properties and the whole VM totally freezes.

I passthroughed with vfio-pci, a drive and my secondary network card without any issues, however I have not had any success with audio.
I have also passthroughed my Xbox Controller and Rocksmith USB adapter.

Any help is appreciated!

I haven't had any trouble passing HDA audio from the chipset through to the guest.  I notice a discrepancy in your process though, the string "kvm assign device" is only printed out by virt/kvm/iommu.c:kvm_assign_device() which is only called if using pci-assign instead of vfio-pci.  AFAIK either will work, but I don't know what to trust in your report...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1374 2014-03-17 21:18:10

stefan_vgapass
Member
Registered: 2014-03-17
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi KVM vga passthrough enthusiasts,

   Thank you so much for your posts, they were invaluable for a KVM vga passthrough beginner like myself to be able to find my way through this unknown teritory.
   
   I can confirm that I've finally got KVM vga passthrough working with an windows 7 x64 guest.
   What finally helped is something from FruitieX's post that made me realize I was still trying to get vga passthrough to work with a 3 month old qemu, so I tried qemu git which worked.
   

FruitieX wrote:

1. Compiled latest qemu and seabios from git, used these.

   FruitieX's post - https://bbs.archlinux.org/viewtopic.php … 6#p1390326
   
Specs:

Mainboard: intel DX79TO http://ark.intel.com/products/55800/Int … ard-DX79TO
CPU: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
Host graphics card: Asus Nvidia GeForce GT 620 2GB mem
Guest graphics card ( passed to the windows guest ): XFX Radeon HD 7850 2GB mem
Host OS: headless Ubuntu 12.04.3 LTS x64
Host kernel: Linux engine 3.14.0-031400rc5-generic #201403022235 SMP Mon Mar 3 03:36:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Guest OS: windows 7 ultimate x64
Guest OS disk: virtio ( running 2 virtual HDDs - win7 is a hog after installation ~ 20GB of diskspace used so I've added a second disk )
Guest Network: virtio bridged network

I'm passing to the windows guest the ATI VGA ( without HDMI ), an usb port, and the onboard Intel HDA sound card therefore getting video, usb and audio inside my windows guest.


   Initially I thought that I need to compile all VFIO kernel module in the kernel ( not as modules ) so I spent time compiling 3.14.0 with .*VFIO=y .
   Then I booted accidentally back into the original ubuntu built 3.14 and KVM vga passthrough works fine, with these kernel modules

# grep VFIO /boot/config-3.14.0-031400rc5-generic CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y

   I used latest qemu-git and seabios from git.
   I've configured pci_stub to bind to the 3 components that I'm passing to the VM by re-building initramfs like this:

vim /etc/initramfs-tools/modules pci_stub ids=1002:6819,1002:aab0,8086:1d20 sudo update-initramfs -u

   This is how I boot the windows guest ( using vfio-pci to bind before this qemu command - see original post #1 for vfio-pci ) :

/opt/qemu-git/bin/qemu-system-x86_64 -enable-kvm -M q35 \ -m 8196 -mem-prealloc -mem-path /mnt/hugepages \ -cpu SandyBridge \ -smp 4,sockets=1,cores=2,threads=2 \ -bios /root/kvm-vga-passthrough/vfio-post3.13/seabios/seabios/out/bios.bin -vga none \ -boot order=cd,menu=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1d.0,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=root.1,addr=00.2 \ -device ahci,bus=pcie.0,id=ahci \ -netdev bridge,br=br0,id=hostnet0 \ -device virtio-net-pci,netdev=hostnet0,id=net0 \ -drive file=/var/lib/libvirt/images/wind764test02.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/var/lib/libvirt/images/wind764test02-disk02.img,id=disk02,format=raw -device ide-hd,bus=ahci.1,drive=disk02 \ -drive file=/var/lib/libvirt/images/Windows.7.SP1.ENG.x86-x64.iso,id=isocd -device ide-cd,bus=ahci.2,drive=isocd

Tip:
In order to be able to use bridged network I had to tell qemu-git to allow it in:

# cat /opt/qemu-git/conf/bridge.conf allow br0

Tests:
I've ran unigine heaven and valley tests ( don't know how they compare with running with windows 7 as host os )
Unigine valley https://unigine.com/products/valley/

Unigine Valley Benchmark 1.0 FPS: 33.4 Score: 1396 Min FPS: 18.3 Max FPS: 60.9 System Platform: Windows 7 (build 7601, Service Pack 1) 64bit CPU model: Intel Xeon E312xx (Sandy Bridge) (3599MHz) x2 GPU model: AMD Radeon HD 7800 Series 13.251.0.0 (2048MB) x1 Settings Render: Direct3D11 Mode: 1920x1200 2xAA fullscreen Preset Custom Quality High

Unigine heaven https://unigine.com/products/heaven/

Unigine Heaven Benchmark 4.0 FPS: 31.7 Score: 798 Min FPS: 7.3 Max FPS: 62.7 System Platform: Windows 7 (build 7601, Service Pack 1) 64bit CPU model: Intel Xeon E312xx (Sandy Bridge) (3599MHz) x2 GPU model: AMD Radeon HD 7800 Series 13.251.0.0 (2048MB) x1 Settings Render: Direct3D11 Mode: 1920x1200 2xAA fullscreen Preset Custom Quality High Tessellation: Moderate

I've ran crysis 3 demo with 1920x1200 antialiasing 2x smooth no issues.
I've ran starcraft 2 max settings 1920x1200 no issues.

Next things to try: KVM vga passthrough with a linux guest and Steam client running on the ATI HD7850 !

Offline

#1375 2014-03-17 21:27:15

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
pereczes wrote:

anybody tried and succeeded with ATIWinflash to reflash radeon card under qemu?

I would strongly discourage this.  The ROM BAR is entirely emulated to the guest, we enable and read the ROM on the first guest access, from then on the guest only reads from the cached copy.  Any sort of ROM update utility is likely to get very, very confused by this.

tried it already... probably could not damage.. saw afterwards following output on my terminal

Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0

added though

-device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on,addr=0.0,romfile=$ROMBIOSPATH/Powercolor.HD6970.2048.110221.rom

but strange enough GPU-Z, reports the increase of shaders (6950 versus 6970) and increase of freq 800 versus 880, but
the number of shaders is half, as one of the cores would not be able...

-> is there any other way to check if both cores are active or GPU-Z is right?
-> is there any way from linux to reload the ROM?

Offline

#1376 2014-03-17 22:23:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've seen it in several posts recently, so I'll mention again, if you're running qemu.git you should not need to build and run a separate seabios.  Only long, long ago before qemu updated their seabios snapshot was this necessary.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1377 2014-03-17 22:39:19

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
trimm wrote:

Another thing though, and its my sound. I gave up on passing through my USB headset(crackling noise) and sharing sound between my host and guest. (I tried following the first post without success)
Im thinking I can passthrough my integrated sound on the motherboard, and I tried doing it like this:

Binding the sound:

vfio-pci 0000:00:1b.0

In my dmesg I get the following:

vfio-pci 0000:00:1b.0: kvm assign device

And in my script I have this:

-device vfio-pci,host=00:1b.0

No errors in Linux, but I get a red mark over my audio icon in Windows. Going into "sound" tab inside audio properties and the whole VM totally freezes.

I passthroughed with vfio-pci, a drive and my secondary network card without any issues, however I have not had any success with audio.
I have also passthroughed my Xbox Controller and Rocksmith USB adapter.

Any help is appreciated!

I haven't had any trouble passing HDA audio from the chipset through to the guest.  I notice a discrepancy in your process though, the string "kvm assign device" is only printed out by virt/kvm/iommu.c:kvm_assign_device() which is only called if using pci-assign instead of vfio-pci.  AFAIK either will work, but I don't know what to trust in your report...

You are quite observant aw, I posted my journalctl when I tried pci-assign.
This is the message I get when I use vfio-pci:

vfio-pci 0000:00:1b.0: enabling device (0000 -> 0002)

I just tested my VM via Virtual Manager GUI, passed through my audio(0000:00:1b.0) and I got sound working.
However the following command is not giving me any audio in Windows. What am I doing wrong? (The last line is my audio..)

sudo -E qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 16384 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw,cache=none \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:19.0,bus=pcie.0 \ -drive file=/dev/sdb1,id=mmo,format=raw,cache=none \ -device ide-hd,bus=ahci.1,drive=mmo \ -drive file=/home/thor/Windows/Windows-Steam.img,id=steam,format=raw,cache=none \ -device ide-hd,bus=ahci.2,drive=steam \ -device vfio-pci,host=00:1b.0

Offline

#1378 2014-03-17 22:43:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:
sudo -E qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 16384 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw,cache=none \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=00:19.0,bus=pcie.0 \ -drive file=/dev/sdb1,id=mmo,format=raw,cache=none \ -device ide-hd,bus=ahci.1,drive=mmo \ -drive file=/home/thor/Windows/Windows-Steam.img,id=steam,format=raw,cache=none \ -device ide-hd,bus=ahci.2,drive=steam \ -device vfio-pci,host=00:1b.0

Why are you passing through both the GPU audio device 1:00.1 and the motherboard device 00:1b.0 if you only plan to use the latter?  Simplify the problem and drop GPU audio.  It's possible Windows is confused about which device to use.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1379 2014-03-17 23:19:31

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw: I just found out the weirdest thing.
If I put the audio pass through down at the bottom, it wont work.
If I put it right beneath -nographic, it works!

I cant see the logic, perhaps because of the disks are passed through and it takes too much time or something.

May be good to know if any of you experience issues with something not working and is in a situation similar to mine.

Offline

#1380 2014-03-18 01:09:31

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I would strongly discourage this.  The ROM BAR is entirely emulated to the guest, we enable and read the ROM on the first guest access, from then on the guest only reads from the cached copy.  Any sort of ROM update utility is likely to get very, very confused by this.

Uhm ... now I am *very* glad, that flashing the bios of my gtx 660 worked smile

Offline

#1381 2014-03-18 02:53:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
aw wrote:

I would strongly discourage this.  The ROM BAR is entirely emulated to the guest, we enable and read the ROM on the first guest access, from then on the guest only reads from the cached copy.  Any sort of ROM update utility is likely to get very, very confused by this.

Uhm ... now I am *very* glad, that flashing the bios of my gtx 660 worked smile

You people are nuts wink


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1382 2014-03-18 03:30:08

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all.

I have a newbie's question: why using two scsi-cds in virtio-scsi-pci does not work?

That is, when using this commandline,

qemu-system-x86_64  ......  -device virtio-scsi-pci,id=scsi0  \
        -drive file=win8.iso,id=cd1  -device scsi-cd,drive=cd1  \
                -drive file=virtio_driver.iso,id=cd2  -device scsi-cd,drive=cd2

during windows installation, the drive containing "virtio_driver.iso" will not show up (while on the boot screen, press f12, there are two cdroms there). Why is that? (I'm using qemu 1.7.0 in the official repository and I know using a ide-cd will work, just want to know more about how virtio-scsi-pci works).

Is there any limitation on the number of device on virto-scsi-pci? These commandlines of virto-scsi-pci are just mysterious to me, can't find anything about it in the manpage and  `qemu-system-x86_64 -device virtio-scsi-pci,help`  just shows some succinct words. Can anyone please point me to some documentation?

Offline

#1383 2014-03-18 05:39:29

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get some older Radeon (like HD 3870) to work with Windows? Is there some known reason why only 5000+ Radeons are reported as working? Am I trying to do something hopeless? (If so I'd immediately go buy a new card.)

I'm getting qemu shutdowns/assertion failures or total host crashes when running Windows 7 guest. Debian guest works fine. I have updated some of my findings on page 55.

Edit: I just tried WinXP after a while and with radeon module on host loaded. Still getting ATA-errors:

[ 9367.516194] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403) [ 9389.765260] ata2: illegal qc_active transition (0000000f->93e80875) [ 9389.774688] ata2.00: exception Emask 0x2 SAct 0xf SErr 0x0 action 0x6 frozen [ 9389.774695] ata2.00: failed command: READ FPDMA QUEUED [ 9389.774704] ata2.00: cmd 60/10:00:80:b3:08/00:00:fa:00:00/40 tag 0 ncq 8192 in res 40/00:18:a8:b4:08/00:00:fa:00:00/40 Emask 0x2 (HSM violation) [ 9389.774707] ata2.00: status: { DRDY }

I have seen identical messages with Win 7 too. This time I managed to kill qemu myself. It seems those errors might go away if launching from console without graphics, but it won't improve anything...

Last edited by ahl (2014-03-18 06:23:17)

Offline

#1384 2014-03-18 09:37:30

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Offline

#1385 2014-03-18 10:24:16

fukhda
Member
Registered: 2014-02-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

Good news here, a success VGA passthrough in Fedora 20. In my very first post , I swamped in code 12 issue. Finally it is solved. Le tme explain the process in the below.

  1. When the assigned AMD card and QXL vga card coexists in VM (Win7x32), there would be a conflict causing AMD card a code12. (AMD driver to be installed here)

  2. By removing the QXL vga (add the swithc -vga none in the script launching VM), there would be a black screen in QEMU window.

  3. Attach another monitor to the AMD card, the VM Win7 is succesfully displayed.

  4. Pass through (via vfio) a Renesas USB3.0 Controller with Keyboard/mouse connected, otherwise we can not operate VM. (Renesas driver to be installed here)

  5. Check the device manager for AMD card, no more code 12. Now, enjoy the VGA passthrough.

Passthrough NVIDIA EGVA GTX650 works too. Thanks for everyone, and especially for nbhs.

Offline

#1386 2014-03-18 10:49:36

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fukhda wrote:

Hi all,

Good news here, a success VGA passthrough in Fedora 20. In my very first post , I swamped in code 12 issue. Finally it is solved. Le tme explain the process in the below.

  1. When the assigned AMD card and QXL vga card coexists in VM (Win7x32), there would be a conflict causing AMD card a code12. (AMD driver to be installed here)

  2. By removing the QXL vga (add the swithc -vga none in the script launching VM), there would be a black screen in QEMU window.

  3. Attach another monitor to the AMD card, the VM Win7 is succesfully displayed.

  4. Pass through (via vfio) a Renesas USB3.0 Controller with Keyboard/mouse connected, otherwise we can not operate VM. (Renesas driver to be installed here)

  5. Check the device manager for AMD card, no more code 12. Now, enjoy the VGA passthrough.

Passthrough NVIDIA EGVA GTX650 works too. Thanks for everyone, and especially for nbhs.

oh you tried emulated vga adapter side by side with dedicated gpu.. interesting. i tried that too and with stock qemu from repositories i also had some error code in task manager. not sure if it was 12. after i compiled qemu from git repo that code was gone (this is still with -vga std -sdl). then qemu window displays everything just fine and dedicated gpu appears to be fine. however i was not able to run anything with full 3d acceleration. not that i tried much anyway. idk how useful this info could be to you... but yeah for full 3d acceleration i have to use -vga none and control VM with keyboard/mouse plugged into passed-through usb port just like you said.

Offline

#1387 2014-03-18 11:05:50

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

Has anyone managed to get some older Radeon (like HD 3870) to work with Windows? Is there some known reason why only 5000+ Radeons are reported as working? Am I trying to do something hopeless? (If so I'd immediately go buy a new card.)

I'm getting qemu shutdowns/assertion failures or total host crashes when running Windows 7 guest. Debian guest works fine. I have updated some of my findings on page 55.

Edit: I just tried WinXP after a while and with radeon module on host loaded. Still getting ATA-errors:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

Offline

#1388 2014-03-18 12:10:44

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

Offline

#1389 2014-03-18 12:53:13

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

Offline

#1390 2014-03-18 13:29:30

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

Offline

#1391 2014-03-18 14:00:26

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
AKSN74 wrote:

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1392 2014-03-18 14:10:47

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Blind Tree Frog wrote:
AKSN74 wrote:

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

That might be the case, except the script in the first post does unbind first (if I am reading it correctly) and it was hanging on my video card when it ran (without the driver being blacklisted).  Whether or not pci-stub was grabbing the USB devices I don't know, but the posted script was unbinding them as expected (Confirmed when I redid my boot line and forgot to disable the service only to boot to a system without a working mouse/keyboard).

Regardless, I may have not done the best kernel config when I built mine.  Blacklisting definitely works and pci-stub is still magic to me.  And none of this conversation is probably of much use to AKSN74

Offline

#1393 2014-03-18 14:29:24

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

I cant for the life of me sort out my audio for my VM.

I use this:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

I have also tried with -soundhw all and -soundhw ac97.

This is what I get when I fire up the computer:

pulseaudio: pa_context_connect() failed pulseaudio: Reason: Connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: No such file or directory ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: No such file or directory audio: Failed to create voice `dac' ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave alsa: Could not initialize ADC alsa: Failed to open `default': alsa: Reason: No such file or directory ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave alsa: Could not initialize ADC alsa: Failed to open `default': alsa: Reason: No such file or directory audio: Failed to create voice `adc'

In the first post its stated that I may have to use the QEMU_AUDIO_DRV variable. I have tried this (export QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_DRV=pa as root) without success. Anyone in the same boat as me, and/or found a solution?(except passing a USB headset? smile)

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted... hmm

I experience the exact same behaviour on my machine. Its working now with ALSA but the quality isnt good. Has anyone fixed this issue or got it working with Pulse Audio (which I would prefer)

Offline

#1394 2014-03-18 15:07:53

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Blind Tree Frog wrote:
AKSN74 wrote:

After some tests in these days, I found that I can't use the same card between Host and Guest.

I re-installed Ubuntu for zero start, and following steps that I done before.  I just plug 2 x GTX480 on my motherboard and want to pass-through one of GTX480 into VM.

According the 1st post guide, I didn't write blacklist for Nouveau, and using pci-stub to do this. Write GTX480's HID into grub.
And then, doing the Systemd service, write pci number into /etc/vfio-pci.cfg, then reboot.

But when I rebooted, my screen turned to 1280X1024, and check dmesg, there is nothing about pci-stub worked log. But I still bind 2nd GTX480 into VFIO.
And when I tried to start VM, It gets error say that GTX480 can't initialze.

Looks like it's impossible that make the same product for host and guest, you need to use different product (Ex. GTX480 for Host, and GTX470 for Guest.)
Is that correct? Or where I do wrong?

I've got to double check my notes (read as, redo everything anyhow), but I couldn't seem to get pci-stub to prevent anything from being loaded.  If I wanted to block my video card I had to blacklist it.   I fear that when I built my kernel I might not have had pci-assign built in and wonder if I also didn't include pci-stub correctly.

I'm pretty sure the first post uses two matching cards though.

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

Hmm... that's interesting, I'm still a Linux newbie so these settings to me may can't understand. But

I saw the usage about softdep, maybe this is a best choice to replace the grub function. Only a problem is can softdep support pci-stub command (like echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind)

Because if I use grub, it will according from HID, that means both of GTX480 will bind into pci-stub and host will not have any GPU to display.

So my idea is, if I can bind 2nd GTX480 at first, then Nouveru will not attach 2nd GTX480. After that everything will be easy.

Is that possible?

Last edited by AKSN74 (2014-03-18 15:09:05)

Offline

#1395 2014-03-18 16:26:07

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
Bronek wrote:

This might be a matter of driver ordering. If pci-stub is not loaded before your GPU driver, it will not be able to bind to GPU card because GPU driver would have grabbed them both already. To ensure that pci-stub is loaded first you can build it with kernel i.e. CONFIG_PCI_STUB=y (whilst ensuring that GPU driver is just a regular module) or, if using it as a module, use softdep command line option to enforce driver ordering.

Alternatively you might let the driver grab any GPU it wants, but then unbind it before starting qemu, and bind to vfio-pci . The risk here is that the driver might not do it cleanly or that GPU might not reset its state, but I have no experience with nvidia/nouveau drivers.

Another gotcha is order of cards in PCI, since your computer's own BIOS will grab primary card, and then pass it to kernel as VESA frame buffer. It should be still possible to unbind it before starting qemu, though.

Personally I would build kernel with pci-stub, ensure that the GPU passed to pci-stub in kernel parameters is not the primary one, and use that for qemu, because maintaining such setup would probably require the smallest amount of work. Also, you might just blacklist GPU drivers if you actually do not use those on the host.

Hmm... that's interesting, I'm still a Linux newbie so these settings to me may can't understand. But

I saw the usage about softdep, maybe this is a best choice to replace the grub function. Only a problem is can softdep support pci-stub command (like echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind)

Because if I use grub, it will according from HID, that means both of GTX480 will bind into pci-stub and host will not have any GPU to display.

So my idea is, if I can bind 2nd GTX480 at first, then Nouveru will not attach 2nd GTX480. After that everything will be easy.

Is that possible?

The point of softdep is that it just orders the drivers, so you can have both pci-stub and your regular GPU drivers. You order it such as to load GPU driver second because they are "greedy" (like all other drivers) and will grab all matching devices, thus preventing pci-stub from doing its work. So, you understand it right : set softdep to ensure pci-stub "grabs" 2nd GFX480 first, and if Nouveau is loaded after this, it will not use this card. It might be easier to build kernel with CONFIG_PCI_STUB=y (but without Nouveau) in which case pci_stub will be guaranteed to be loaded first without use of softdep.

See also post #93, using modprobe/*conf rather than kernel command line parameter . I am not sure which one is better.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1396 2014-03-18 23:57:09

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

The point of softdep is that it just orders the drivers, so you can have both pci-stub and your regular GPU drivers. You order it such as to load GPU driver second because they are "greedy" (like all other drivers) and will grab all matching devices, thus preventing pci-stub from doing its work. So, you understand it right : set softdep to ensure pci-stub "grabs" 2nd GFX480 first, and if Nouveau is loaded after this, it will not use this card. It might be easier to build kernel with CONFIG_PCI_STUB=y (but without Nouveau) in which case pci_stub will be guaranteed to be loaded first without use of softdep.

See also post #93, using modprobe/*conf rather than kernel command line parameter . I am not sure which one is better.

Thanks for your hint, now I know how to do.

I thought that it still a big challenge to do my build, because HID is a big problem, I need to do pci-stub command after pci-stub load up and Nouveau not load up yet.

I'm thought about rc.local to do this (because GRUB can't do bash code) , but I found some information, rc.local launched after driver loaded so it's not an option.

Although I can't use both GTX480 for host and guest, I still can pass-through them into VM with softdep.

I'll try for this, and hope it work, thanks for your help.

Offline

#1397 2014-03-19 11:05:39

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested 3.14-rc7 kernel. Everything works good. There is even performance increase for that passmark directx9 complex test. rc6 - 16 FPS, rc7 - 21 FPS.

P.S. In case noone noticed dr6 patch is already upstream this no longer needed. Adding my compilation of patches should anyone need. Usage - extract them to parent dir of kernel source code and from kernel source code dir run ../patch_kernel.sh

http://paste2box.com/5/#/3422361273/PsZ … hes.tar.gz

Offline

#1398 2014-03-19 19:21:55

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

Offline

#1399 2014-03-19 20:32:19

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rawuza wrote:
ahl wrote:

I'm getting qemu shutdowns/assertion failures or total host crashes when running Windows 7 guest. Debian guest works fine. I have updated some of my findings on page 55.

Edit: I just tried WinXP after a while and with radeon module on host loaded. Still getting ATA-errors:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

You are right. The behavior is pretty random. Here's an incomplete list of problems that I remember occurring: ATA-errors on all connected devices, single port usb failures, full failure of all usb-devices, display port link failure (loss of picture), complete freeze of system, freeze + automatic reboot, and failure of SATA controller with IO_PAGE_FAULTS that resulted in corrupted root file system. Just about anything can happen.

So I replaced my HD 3870 with a HD 6670. Nothing changed. Random errors continued. I wonder if HD 6670 is broken too?

Next I tried physically using different slots for all cards. Nothing changed.

And then I finally made some progress:
Removal of the cheap PCI-E network interface cards that I had on host resulted in tremendous improvement on system stability! Win 7 started working seemingly perfectly. I got graphics working for the first time and was able to run the system benchmark with good results.

For a while I thought it fixed everything, but on about 10th identical(snapshot=on) reboot of Windows 7 guest, I got some minor ATA errors on host again. However, all of the other randomness is gone. I wonder if the SATA controller is bad too (AMD A85X FCH)?

Next I tried WinXP. It seems to be much harder to boot than Win 7. It took a while and a whole lot of minor ATA errors until it finished. But it booted! So for the first time I also got WinXP running. Then I noticed a connection between qemu and those ATA errors: If I run qemu with snapshot=on I only get READ errors. Without snapshot I also get WRITE errors. So I moved WinXP completely in ramdisk and booted it: The exact same snapshot of WinXP that took a long time to boot from disk, now booted in just a few seconds. All of the ATA errors were gone too! Makes sense.

Any hints on what to try next? (I set my SATA controller to IDE mode now for testing. It used to be AHCI.)

Edit: By the way I wouldn't be surprised if a lot of people had this problem I have now. It could easily go unnoticed with Win 7. Also my HD is pretty slow, which could increase the chance of errors.

### Group 0 ### 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Complex ### Group 1 ### 00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Trinity [Radeon HD 7660D] 00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Trinity HDMI Audio Controller ### Group 2 ### 00:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks XT [Radeon HD 6670/7670] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Turks/Whistler HDMI Audio [Radeon HD 6000 Series] ### Group 3 ### 00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03) 00:10.1 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03) ### Group 4 ### 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] (rev 40) ### Group 5 ### 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11) 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11) ### Group 6 ### 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11) 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11) ### Group 7 ### 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 14) 00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD] FCH IDE Controller (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40) ### Group 8 ### 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 2) 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) ### Group 9 ### 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 5

Last edited by ahl (2014-03-19 21:17:03)

Offline

#1400 2014-03-19 21:35:19

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:
Norcoen wrote:

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

This is my virt-manager config XML for a Windows 7 VM with AMD HD6770 GPU + HDMI, Audio controller and an usb controller passed to it. I made the vm using the virt-manager gui, I manually configured the number of sockets, cores and threads and after installing windows i used the gui to add the video card and various controllers to the vm. Also I have nested virtualization enabled if this makes a difference.

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit Windows-7-Main or other application using the libvirt API. --> <domain type='kvm'> <name>Windows-7-Main</name> <uuid>fbc03842-e6ea-d94d-eeba-e1dc4da0e1dd</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='6' threads='6'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='ht'/> <feature policy='require' name='ds'/> <feature policy='require' name='pcid'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/windows_7_main.img'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:fe:3e:b0'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='vga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> </domain>

Offline

#1401 2014-03-19 21:39:04

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
rawuza wrote:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

You are right. The behavior is pretty random. Here's an incomplete list of problems that I remember occurring: ATA-errors on all connected devices, single port usb failures, full failure of all usb-devices, display port link failure (loss of picture), complete freeze of system, freeze + automatic reboot, and failure of SATA controller with IO_PAGE_FAULTS that resulted in corrupted root file system. Just about anything can happen.

To me this seems as if the GPU touches other devices' memory. My question about this is:

  • Is it possible for one PCIe device to directly access another one? If it's done through DMA, shouldn't the IOMMU prevent exactly this?

  • Is there any way to debug this?

Offline

#1402 2014-03-19 22:06:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rawuza wrote:
ahl wrote:
rawuza wrote:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

You are right. The behavior is pretty random. Here's an incomplete list of problems that I remember occurring: ATA-errors on all connected devices, single port usb failures, full failure of all usb-devices, display port link failure (loss of picture), complete freeze of system, freeze + automatic reboot, and failure of SATA controller with IO_PAGE_FAULTS that resulted in corrupted root file system. Just about anything can happen.

To me this seems as if the GPU touches other devices' memory. My question about this is:

  • Is it possible for one PCIe device to directly access another one? If it's done through DMA, shouldn't the IOMMU prevent exactly this?

  • Is there any way to debug this?

The IOMMU can only provide isolation and translation for transactions that it sees.  If there's a multifunction device that allows peer-to-peer DMA within the component, then the IOMMU may never see the transaction.  This is the purpose of Access Control Services (ACS) to allow configuration and enforcement of whether all transactions are forwarded upstream.  This is why IOMMU groups put such devices together, so we can at least have a single entity manage them rather than allowing a guest to exploit the host.  Many folks here abuse the ACS override patch which was rejected upstream exactly because such peer-to-peer interactions can be unpredictable and difficult to debug.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1403 2014-03-19 22:18:41

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The IOMMU can only provide isolation and translation for transactions that it sees.  If there's a multifunction device that allows peer-to-peer DMA within the component, then the IOMMU may never see the transaction.  This is the purpose of Access Control Services (ACS) to allow configuration and enforcement of whether all transactions are forwarded upstream.  This is why IOMMU groups put such devices together, so we can at least have a single entity manage them rather than allowing a guest to exploit the host.  Many folks here abuse the ACS override patch which was rejected upstream exactly because such peer-to-peer interactions can be unpredictable and difficult to debug.

But I have never even applied that patch. I'm using 3.14.0-rc7-00004-g9f8b483-dirty + fix_memleaks.patch.

Err, wait. I actually have it applied, but not activated. Surely it does not enable something automatically? I could try with clean source too.

Edit: I meant to test the patch, but I didn't find any documentation on how to evaluate when it might be needed. Like which devices should I try activating?

Last edited by ahl (2014-03-19 22:32:32)

Offline

#1404 2014-03-19 22:31:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
aw wrote:

The IOMMU can only provide isolation and translation for transactions that it sees.  If there's a multifunction device that allows peer-to-peer DMA within the component, then the IOMMU may never see the transaction.  This is the purpose of Access Control Services (ACS) to allow configuration and enforcement of whether all transactions are forwarded upstream.  This is why IOMMU groups put such devices together, so we can at least have a single entity manage them rather than allowing a guest to exploit the host.  Many folks here abuse the ACS override patch which was rejected upstream exactly because such peer-to-peer interactions can be unpredictable and difficult to debug.

But I have never even applied that patch. I'm using 3.14.0-rc7-00004-g9f8b483-dirty + fix_memleaks.patch.

Err, wait. I actually have it applied, but not activated. Surely it does not enable something automatically? I could try with clean source too.

As I say, multifunction devices without ACS can do it too and there's not much we can do about it.  We're dealing with lots of consumer grade hardware in more of a precision operating environment.  It's really just pure speculation that this might be playing a factor.  The only way I can think to test would be to assign only one device from any IOMMU group, the other device can be disabled by writing zero to the PCI command register (setpci -s bus:dev.fn COMMAND=0).  That should prevent them from claiming any peer-to-peer traffic.  Test and incrementally add one device at a time.  Again, this is all just speculation, assuming peer-to-peer is occurring, how would you test for it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1405 2014-03-19 22:34:24

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noctavian wrote:
TripleSpeeder wrote:
Norcoen wrote:

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

This is my virt-manager config XML for a Windows 7 VM with AMD HD6770 GPU + HDMI, Audio controller and an usb controller passed to it. I made the vm using the virt-manager gui, I manually configured the number of sockets, cores and threads and after installing windows i used the gui to add the video card and various controllers to the vm. Also I have nested virtualization enabled if this makes a difference.

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit Windows-7-Main or other application using the libvirt API. --> <domain type='kvm'> <name>Windows-7-Main</name> <uuid>fbc03842-e6ea-d94d-eeba-e1dc4da0e1dd</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='6' threads='6'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='ht'/> <feature policy='require' name='ds'/> <feature policy='require' name='pcid'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/windows_7_main.img'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:fe:3e:b0'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='vga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> </domain>

Thanks noctavian! However this is not exactly what I want to achieve. You are passing your GPU as secondary card using standard pci-assign method. But I'd like to use the GPU exclusively, so that SeaBios is already showing the boot screen on the real GPU (Using x-vga=on) and no emulated graphics adapter.

Offline

#1406 2014-03-19 23:00:42

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

As I say, multifunction devices without ACS can do it too and there's not much we can do about it.

We're dealing with lots of consumer grade hardware in more of a precision operating environment.  It's really just pure speculation that this might be playing a factor.  The only way I can think to test would be to assign only one device from any IOMMU group, the other device can be disabled by writing zero to the PCI command register (setpci -s bus:dev.fn COMMAND=0).  That should prevent them from claiming any peer-to-peer traffic.  Test and incrementally add one device at a time.  Again, this is all just speculation, assuming peer-to-peer is occurring, how would you test for it.

ACS is implemented in the bridge, right? So if my PCIe chipset support ACS (it's an AMD 990FX; how to check?) and I don't use the ACS override patch (which I don't), then this should not be possible, or am I missing something?

I don't understand how passing through the GPU can freeze the host, when the IOMMU should enforce isolation.

Offline

#1407 2014-03-20 00:12:48

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:
noctavian wrote:
TripleSpeeder wrote:

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

This is my virt-manager config XML for a Windows 7 VM with AMD HD6770 GPU + HDMI, Audio controller and an usb controller passed to it. I made the vm using the virt-manager gui, I manually configured the number of sockets, cores and threads and after installing windows i used the gui to add the video card and various controllers to the vm. Also I have nested virtualization enabled if this makes a difference.

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit Windows-7-Main or other application using the libvirt API. --> <domain type='kvm'> <name>Windows-7-Main</name> <uuid>fbc03842-e6ea-d94d-eeba-e1dc4da0e1dd</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='6' threads='6'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='ht'/> <feature policy='require' name='ds'/> <feature policy='require' name='pcid'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/windows_7_main.img'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:fe:3e:b0'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='vga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> </domain>

Thanks noctavian! However this is not exactly what I want to achieve. You are passing your GPU as secondary card using standard pci-assign method. But I'd like to use the GPU exclusively, so that SeaBios is already showing the boot screen on the real GPU (Using x-vga=on) and no emulated graphics adapter.

Hi i see you whant to use Virt-Manager to stop and start your VM,
I think i can help you a bit.

This is an old working config xml for Virt-Manager, but i dont know if it work at this moment (i test a lot of different way to passthroug GPU to a VM)

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit Windows-7 or other application using the libvirt API. --> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows-7</name> <uuid>652e4382-4dc8-7349-cdbf-359d333aba08</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-q35-1.7'>hvm</type> <boot dev='hd'/> <bootmenu enable='yes'/> <smbios mode='host'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <vendor>Intel</vendor> <topology sockets='1' cores='4' threads='1'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='abm'/> <feature policy='require' name='ht'/> <feature policy='require' name='acpi'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='f16c'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='monitor'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writethrough'/> <source file='/home/val/Image HDD/Windows-7-Test.img'/> <target dev='sda' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:f7:8e:2b'/> <source bridge='br0'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05e3'/> <product id='0x0607'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc228'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc229'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x002f'/> </source> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

This work for passthrough your graphique card as a primary gpu and use vfio

And it's just an example !!!!

Offline

#1408 2014-03-20 01:55:26

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, I'm having a problem and hopefully someone might be able to help....

As my earlier post said, I've got the 3 links from the first post built and installed (linux mainline, seabios, qemu..  done in that order with linux being installed again later to make sure I had pci-stub installed correctly... and I know I don't need seabios anymore, but I didn't know then).

I have my sole video card blacklisted.  On boot I see two or four lines of text and then no more updates.   I then SSH in, bind the video card to vfio and run qemu with this command:

qemu-system-x86_64 -nographic -enable-kvm -M q35 -m 1024 -cpu host \
  -cdrom cb.iso \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

where 'cb.iso' is a crunchbang live cd and '03:00.0' is my video card. I then hear the video card fan spin up and the screen flashes black and I get text dunmping to the screen from the VM.   Every single time it comes up with no bootable device however.   It should be loading the CDROM but it doesn't seem to be checking it.   I tried adding the '-boot order=d' option, but that didn't take and the menu option blows by before I get a chance to attempt to interact.

It will boot off an image for a floppy disk though.  So why can't I read a CDROM? (note, it also failed to read a fedora image and the physical bluray drive)

Offline

#1409 2014-03-20 02:30:18

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

It will boot off an image for a floppy disk though.  So why can't I read a CDROM? (note, it also failed to read a fedora image and the physical bluray drive)

Try the formatting listed in OP.

-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Also try hitting the F12 key while starting to get into the boot menu. If the problem is the time it takes your monitor to activate/display the output (I had this same problem), that should open the boot menu and pause long enough for you to see it.

Edit: #! installer is a dual USB/CD ISO, that may be confusing kvm. Maybe you can try passing it as a raw disk image.

Last edited by alexis_evo (2014-03-20 02:37:48)

Offline

#1410 2014-03-20 02:34:38

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone figured out the issue where the right half of the screen is shifted to the left half (or vice versa)? This has plagued every game I've tried to play in the VM. Doesn't happen as often as it originally did, but it still happens at least once every 4-5 hours of usage. It's very annoying when it happens in dota, as I'm playing with a gigantic handicap for the rest of the game sad.

Screenshot of it occurring in Mirror's Edge http://i.imgur.com/fuD5HkB.jpg

Offline

#1411 2014-03-20 03:09:01

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:
Blind Tree Frog wrote:

It will boot off an image for a floppy disk though.  So why can't I read a CDROM? (note, it also failed to read a fedora image and the physical bluray drive)

Try the formatting listed in OP.

-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Also try hitting the F12 key while starting to get into the boot menu. If the problem is the time it takes your monitor to activate/display the output (I had this same problem), that should open the boot menu and pause long enough for you to see it.

Edit: #! installer is a dual USB/CD ISO, that may be confusing kvm. Maybe you can try passing it as a raw disk image.

The fedora live cd is a regular CD iso though.

The line from the OP works and I can boot in now.  I'm so used to using the '-cdrom' option that I forgot to check the first post for suggestions.  Thanks.


My next step is to figure out why I can't get the usb devices to pass through and this reset error I'm getting on my non-video card devices.  I'm going to post this here in case the answer jumps out to anybody and so I have a reference to check when I'm looking up answers away from my computer.

My qemu command:

qemu-system-x86_64 -nographic -enable-kvm -M q35 -m 1024 -cpu host \ -drive file=/root/cb.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:12.0,bus=pcie.0 \ -device vfio-pci,host=00:12.2,bus=pcie.0 \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -device vfio-pci,host=00:14.0,bus=pcie.0 \ -device vfio-pci,host=00:14.2,bus=pcie.0 \ -device vfio-pci,host=00:14.3,bus=pcie.0 \ -device vfio-pci,host=00:14.4,bus=pcie.0 \ -device vfio-pci,host=00:14.5,bus=pcie.0 \ -device vfio-pci,host=00:16.0,bus=pcie.0 \ -device vfio-pci,host=00:16.2,bus=pcie.0 \ -device vfio-pci,host=07:00.0,bus=pcie.0 \

Output of it being run:

root@Pandora ~# sh qemu-test.sh qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism.

lspci output:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) 00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) 00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E) 00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H) 00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A) 00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 3) 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control 01:00.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) 02:08.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) 02:10.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition] 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 07:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 0c:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) 0d:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 0e:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

vfio-pci.cfg

DEVICES="0000:00:12.0 0000:00:12.2 0000:00:13.0 0000:00:13.2 0000:00:14.0 0000:00:14.2 0000:00:14.3 0000:00:14.4 0000:00:14.5 0000:00:16.0 0000:00:16.2 0000:03:00.0 0000:03:00.1 0000:07:00.0"

/proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=6a991f03-8fd6-42f2-b7be-06eacf457e76 rw quiet iommu=on video=efifb:off pci-stub.ids=1002:4397,1002:4396,1002:4397,1002:4396,1002:4385,1002:4383,1002:4395,1002:4384,1002:4399,1002:4397,1002:4396,1002:683d,1002:aab0

interesting dmesg output that is probably a huge clue glaring at me:

] [ 189.799749] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.2 domain=0x0024 address=0x0000000100000000 flags=0x0000] [ 189.800493] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.2 domain=0x0024 address=0x00000000f000ff40 flags=0x0000] [ 189.801242] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.2 domain=0x0024 address=0x00000000f000ff40 flags=0x0000]

I'm probably missing something obvious... now to find it.

Offline

#1412 2014-03-20 07:42:42

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

Offline

#1413 2014-03-20 12:16:43

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

I wasn't too worried about them if USB came up, but since my keyboard isn't working in the guest I figure I need to investigate them somewhat.

I'm more concerned about the dmesg error as I am seeing the same three strings repeated a lot.

edit:

i wonder if I'm setting the wrong bus in the qemu command

Last edited by Blind Tree Frog (2014-03-20 12:24:53)

Offline

#1414 2014-03-20 13:28:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
novist wrote:

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

I wasn't too worried about them if USB came up, but since my keyboard isn't working in the guest I figure I need to investigate them somewhat.

I'm more concerned about the dmesg error as I am seeing the same three strings repeated a lot.

edit:

i wonder if I'm setting the wrong bus in the qemu command

You need to passthrough both ehci and ohci controllers

lspci

... 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller ...

Offline

#1415 2014-03-20 13:49:53

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Blind Tree Frog wrote:
novist wrote:

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

...
i wonder if I'm setting the wrong bus in the qemu command

You need to passthrough both ehci and ohci controllers

lspci

... 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller ...

I thought I was:
(excerpt from above with newlines added to emphasis grouping)

 -device vfio-pci,host=00:12.0,bus=pcie.0 \ -device vfio-pci,host=00:12.2,bus=pcie.0 \ -device vfio-pci,host=00:13.0,bus=pcie.0 \ -device vfio-pci,host=00:13.2,bus=pcie.0 \ -device vfio-pci,host=00:14.0,bus=pcie.0 \ -device vfio-pci,host=00:14.2,bus=pcie.0 \ -device vfio-pci,host=00:14.3,bus=pcie.0 \ -device vfio-pci,host=00:14.4,bus=pcie.0 \ -device vfio-pci,host=00:14.5,bus=pcie.0 \ -device vfio-pci,host=00:16.0,bus=pcie.0 \ -device vfio-pci,host=00:16.2,bus=pcie.0 \

Except for device one device I stubbed everything out that was under the same root number and passed them all in to qemu at the same time.  The exception is 0000:07:00.0 which didn't have another.  I originally tried 0e:00.0 and 0d:00.0 as well but that caused all sorts of errors so I figured I'd come back to them.

Edit:

looking at my earlier post...

 -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

How do I determine what the bus needs to be set to and when do I set the addr field?  In this case I copied the pattern that the OP used, but I figured that there must be a deterministic way.

Off to the man pages I guess.

Last edited by Blind Tree Frog (2014-03-20 14:02:37)

Offline

#1416 2014-03-21 02:07:55

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GNA wrote:

/sys/kernel/iommu_groups is empty.
My system is supporting vt-d and it should be enabled. I dont get it.
Did i miss something obvious? Thanks for help!

I encountered this problem before, and adding intel_iommu=on to the kernel cmdline solved the problem.

Offline

#1417 2014-03-21 12:25:29

ahom
Member
Registered: 2014-03-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First off I would like to thank everybody contributing to this, it sure is exciting to see all the possibilities of these tools, I'm new to these forums but not new to archlinux and have been lurking around the vga passthrough for a longtime now.

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money tongue). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Here is a summary of the spec I want to buy and what I intend to do with it:

- CPU: AMD FX 8350 Black Edition
- RAM: 32 GB
- MB: I'm not sure yet, but one of
    - ASUS SABERTOOTH 990FX R2.0
    - MSI 990FXA-GD80
    - ASUS CROSSHAIR V FORMULA-Z
- GC: 2 AMD ones for the windows VMs (can't remember the model names), one more very small fanless one for the host
- Other parts are irrelevant to the issue I guess.

The setup of the VMs would be:

- Each Windows VM:
    - 3 cores
    - 12GB ram
    - One discrete GC
- Host:
    - 2 cores
    - 8 GB ram
    - One discrete GC

Here are my thoughts:

    - Will this certainly fail ? wrt passing 2 cards to 2 VM
    - Is there one of these motherboards that will not work properly? (BIOS wise or w/e ?)
    - Does having 16 lines PCIe x16 mandatory for performances on newest games/cards ? (I'm saying this because when you start adding up cards PCI lines satrts to be spread and there are not so much of them in the end for 3 cards, the second card usually gets 8x, it seems that the MSI one can work with 16x/16x/4x but I fear that the slots are not aligned in a way I can have a 2nd 2slot card)

Thanks!

Offline

#1418 2014-03-21 13:52:56

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahom wrote:

    - Does having 16 lines PCIe x16 mandatory for performances on newest games/cards ? (I'm saying this because when you start adding up cards PCI lines satrts to be spread and there are not so much of them in the end for 3 cards, the second card usually gets 8x, it seems that the MSI one can work with 16x/16x/4x but I fear that the slots are not aligned in a way I can have a 2nd 2slot card)
Thanks!

http://www.techpowerup.com/reviews/AMD/ … ng/25.html
http://www.techpowerup.com/reviews/Inte … ng/23.html
(seen few more in other places)

If you ask me, differences are negligible.

Offline

#1419 2014-03-21 14:02:01

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahom wrote:

Here is a summary of the spec I want to buy and what I intend to do with it:

- CPU: AMD FX 8350 Black Edition
- RAM: 32 GB
- MB: I'm not sure yet, but one of
    - ASUS SABERTOOTH 990FX R2.0
    - MSI 990FXA-GD80
    - ASUS CROSSHAIR V FORMULA-Z
- GC: 2 AMD ones for the windows VMs (can't remember the model names), one more very small fanless one for the host
- Other parts are irrelevant to the issue I guess.

I believe there are some preferences up thread to avoid ASUS mobo's and interest towards Gigabyte.  That said, that was also before ASUS's bios updates last year that seem to address the issues that people were concerned about.

Offline

#1420 2014-03-21 15:27:11

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
ahom wrote:

Here is a summary of the spec I want to buy and what I intend to do with it:

- CPU: AMD FX 8350 Black Edition
- RAM: 32 GB
- MB: I'm not sure yet, but one of
    - ASUS SABERTOOTH 990FX R2.0
    - MSI 990FXA-GD80
    - ASUS CROSSHAIR V FORMULA-Z
- GC: 2 AMD ones for the windows VMs (can't remember the model names), one more very small fanless one for the host
- Other parts are irrelevant to the issue I guess.

I believe there are some preferences up thread to avoid ASUS mobo's and interest towards Gigabyte.  That said, that was also before ASUS's bios updates last year that seem to address the issues that people were concerned about.

ASUS has been wonderful in terms of updating their BIOS to fix support. @GizmoChicken managed to get through to a tech that could get a fix working, gave us test BIOS with the fix, then eventually pushed out the update to everyone.

@ahom: I know the Sabertooth 990FX has a fixed BIOS, and I personally use an ASUS M5A99FX PRO R2.0 that works perfectly with the latest BIOS.

I'd suggest you be careful with how you limit 2 cores to a Windows VM with the FX-8350. Or avoid restricting it all together. The FX-8350 is actually four pairs of cores, with each pair sharing an FPU. In the worst case scenario, the 2 cores will perform as 1 core when the FPU bottlenecks the application in question (common for games). I find it's fine to overcommit CPUs to guests -- I run 6x cores on each guest for two Windows VMs, and use KVM to assign CPU priority to one of them. This gives the most out of the hardware to both VMs, as it isn't that often you'll 100% all four cores in both VMs.

Last edited by alexis_evo (2014-03-21 15:32:25)

Offline

#1421 2014-03-21 17:23:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

Further testing required, but an Nvidia Quadro K4000 (passed as a secondary GPU, no VGA) gives me a Passmark 3D Graphics mark of 2796.  The average for this card is 2797, so that's as good as we can hope for this card (this is without the debug register performance patches, so apparently not an issue on this test).  I'll update with results from Nvidia and Radeon in VGA mode as I'm able to run them.  Unfortunately I still saw the BSOD with passmark, but when it does it the installation gets broken and I'm unable to run it again even after uninstall-reinstall.  It seemed to run ok on a fresh win7x64 install with only the Nvidia driver installed.  No idea how to debug that.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1422 2014-03-21 20:59:13

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
novist wrote:

Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

Further testing required, but an Nvidia Quadro K4000 (passed as a secondary GPU, no VGA) gives me a Passmark 3D Graphics mark of 2796.  The average for this card is 2797, so that's as good as we can hope for this card (this is without the debug register performance patches, so apparently not an issue on this test).  I'll update with results from Nvidia and Radeon in VGA mode as I'm able to run them.  Unfortunately I still saw the BSOD with passmark, but when it does it the installation gets broken and I'm unable to run it again even after uninstall-reinstall.  It seemed to run ok on a fresh win7x64 install with only the Nvidia driver installed.  No idea how to debug that.

The benchmark worked fine for me with my AMD Radeon HD5670 on Windows 8.1 (64). Since I have a strange/disproportional GPU+CPU config, I'm not sure if my results are comparable with what other people upload to the passmark service. In case you need more results, I can paste mine, but I'd like to ensure comparability.


i'm sorry for my poor english wirting skills…

Offline

#1423 2014-03-22 08:34:10

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahom wrote:

First off I would like to thank everybody contributing to this, it sure is exciting to see all the possibilities of these tools, I'm new to these forums but not new to archlinux and have been lurking around the vga passthrough for a longtime now.

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money tongue). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Certainly an interesting project! How do you plan to assign peripherals like keyboard, mouse, etc. to the hosts? You might want to look for a motherboard that has at least 3 distinct USB controllers, so you can pass through one chip to each VM and have one left for the host.

Offline

#1424 2014-03-22 08:46:47

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
aw wrote:
novist wrote:

Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

Further testing required, but an Nvidia Quadro K4000 (passed as a secondary GPU, no VGA) gives me a Passmark 3D Graphics mark of 2796.  The average for this card is 2797, so that's as good as we can hope for this card (this is without the debug register performance patches, so apparently not an issue on this test).  I'll update with results from Nvidia and Radeon in VGA mode as I'm able to run them.  Unfortunately I still saw the BSOD with passmark, but when it does it the installation gets broken and I'm unable to run it again even after uninstall-reinstall.  It seemed to run ok on a fresh win7x64 install with only the Nvidia driver installed.  No idea how to debug that.

The benchmark worked fine for me with my AMD Radeon HD5670 on Windows 8.1 (64). Since I have a strange/disproportional GPU+CPU config, I'm not sure if my results are comparable with what other people upload to the passmark service. In case you need more results, I can paste mine, but I'd like to ensure comparability.

in passmark options i picked other people with same gpu+cpu to compare with. i figured thats best to get a perspective of virtualized performance relative to bare metal. Would be really interesting to see your results.

Offline

#1425 2014-03-22 09:25:50

ahom
Member
Registered: 2014-03-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

http://www.techpowerup.com/reviews/AMD/ … ng/25.html
http://www.techpowerup.com/reviews/Inte … ng/23.html
(seen few more in other places)

If you ask me, differences are negligible.

That's very good news! That was actually my main concern as both the VMs need to be performant.

alexis_evo wrote:

I'd suggest you be careful with how you limit 2 cores to a Windows VM with the FX-8350. Or avoid restricting it all together. The FX-8350 is actually four pairs of cores, with each pair sharing an FPU. In the worst case scenario, the 2 cores will perform as 1 core when the FPU bottlenecks the application in question (common for games). I find it's fine to overcommit CPUs to guests -- I run 6x cores on each guest for two Windows VMs, and use KVM to assign CPU priority to one of them. This gives the most out of the hardware to both VMs, as it isn't that often you'll 100% all four cores in both VMs.

I think I'll try first what you are suggesting about overcommiting CPUs, that seems like a right balance, if I perceive some issues I'll post here about my findings.

Offline

#1426 2014-03-22 09:33:33

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For these days test, I got a strange problem.

First, I have HD5450 and double GTX480 on my MB, and set both GTX480 into pci-stub, lspci -k shows that both of GTX480 has bind into pci-stub.

Second, I run 2 VMs with these GTX480 passthrough by VFIO. And lspci -k shows all of them bind into VFIO-PCI.

1st VM can work normal, it can entered Windows 7. But 2nd VM has no screen when I plug DVI cable. And terminal didn't show any error.

I tried shut down the 1st VM, And only run 2nd VM, no use. Even I tried 1st VM's storage file, still give me no screen.


I know X79 only serves 2 PCI-E x16 and 1 PCI-E x8, so 2nd GTX480 is PCI-E x8,

I'm using Rampage IV Gene. It can only plug 2 video card at the same time, but I use PCI-E cable so I can plug 3 video cards at the same time.

Now I'm thinking that maybe this MB can't do this at real. But when I switch back to Windows 7, it can catch all of 3 cards.

Is anyone have an idea?

Last edited by AKSN74 (2014-03-22 11:51:45)

Offline

#1427 2014-03-22 12:26:55

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:
trimm wrote:

I cant for the life of me sort out my audio for my VM.

I use this:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

I have also tried with -soundhw all and -soundhw ac97.

This is what I get when I fire up the computer:

pulseaudio: pa_context_connect() failed pulseaudio: Reason: Connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: No such file or directory ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: No such file or directory audio: Failed to create voice `dac' ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave alsa: Could not initialize ADC alsa: Failed to open `default': alsa: Reason: No such file or directory ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave alsa: Could not initialize ADC alsa: Failed to open `default': alsa: Reason: No such file or directory audio: Failed to create voice `adc'

In the first post its stated that I may have to use the QEMU_AUDIO_DRV variable. I have tried this (export QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_DRV=pa as root) without success. Anyone in the same boat as me, and/or found a solution?(except passing a USB headset? smile)

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted... hmm

I experience the exact same behaviour on my machine. Its working now with ALSA but the quality isnt good. Has anyone fixed this issue or got it working with Pulse Audio (which I would prefer)

Similar problem here. From my reading, the pulseaudio "Connection refused" errors above might be due to the fact that normally pulseaudio is configured in single user mode, which means only a specific session of some user can use pulseaudio in a specific time(unless you are root). If I start a pulseaudio instance as root and then run qemu as root, these errors are gone. But yes, the sound is really harsh and distorted.

Configure pulseaudio in system mode may help eliminate the errors. However I have no idea regarding the ultra-low sound quality. Hope someone can shed light on this issue.

Last edited by Jesse2004 (2014-03-22 12:41:08)

Offline

#1428 2014-03-23 05:33:50

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've at least made some progress...

I think part of my earlier problem was that pci-stub wasn't grabbing the USB devices correctly.  Thursday I built another kernel and made sure that pci-stub was built in.  Today I finally installed it.

Booting a Fedora liveCD from iso and it comes up (as it did prior to the new kernel) but mouse and keyboard work.  It still complains about little things and it locked up on me once, but it works and sound and video work.

I then switched back to the crunchbang liveCD.  It booted and got to the menu,  but the keyboard never got power and the system wouldn't respond to the inputs.  So I'm closer.

Still getting the reset errors:

qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.

And still seeing page faults in dmesg

[ 311.887722] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887725] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887728] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887731] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887734] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887737] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887740] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887743] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887746] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887749] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887754] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887757] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887760] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887763] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050] [ 311.887766] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]

Though that is a different device this time (compared to last)

I'm not sure if anyone sees anything that I should take note of in the last few dmesg messages, but just in case:

[ 357.076479] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270 [ 357.312658] vfio_cap_init: 0000:00:12.2 hiding cap 0xa [ 357.372555] vfio_cap_init: 0000:00:13.2 hiding cap 0xa [ 357.425797] vfio_cap_init: 0000:00:16.2 hiding cap 0xa [ 485.323872] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270 [ 485.553417] vfio_cap_init: 0000:00:12.2 hiding cap 0xa [ 485.606661] vfio_cap_init: 0000:00:13.2 hiding cap 0xa [ 485.659901] vfio_cap_init: 0000:00:16.2 hiding cap 0xa [ 517.644634] kvm [893]: vcpu0 unhandled rdmsr: 0xc0010112 [ 517.644646] kvm [893]: vcpu0 unhandled rdmsr: 0xc0010048 [ 517.766789] kvm [893]: vcpu0 unhandled rdmsr: 0xc0010001 [ 517.778451] kvm [893]: vcpu1 unhandled rdmsr: 0xc0010048 [ 517.789778] kvm [893]: vcpu2 unhandled rdmsr: 0xc0010048 [ 517.801080] kvm [893]: vcpu3 unhandled rdmsr: 0xc0010048 [ 517.812380] kvm [893]: vcpu4 unhandled rdmsr: 0xc0010048 [ 517.823688] kvm [893]: vcpu5 unhandled rdmsr: 0xc0010048 [ 517.909455] kvm: zapping shadow pages for mmio generation wraparound [ 519.701768] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.741639] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.741647] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X [ 519.748301] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.748309] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X [ 519.748316] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X [ 519.768267] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.768276] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X [ 519.768283] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X [ 519.768289] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X [ 519.778929] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.778938] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X [ 519.778945] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X [ 519.778951] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X [ 519.778957] vfio-pci 0000:07:00.0: irq 80 for MSI/MSI-X [ 519.808216] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.808223] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X [ 519.808231] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X [ 519.808237] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X [ 519.808243] vfio-pci 0000:07:00.0: irq 80 for MSI/MSI-X [ 519.808250] vfio-pci 0000:07:00.0: irq 81 for MSI/MSI-X [ 519.824860] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X [ 519.824868] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X [ 519.824874] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X [ 519.824880] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X [ 519.824887] vfio-pci 0000:07:00.0: irq 80 for MSI/MSI-X [ 519.824893] vfio-pci 0000:07:00.0: irq 81 for MSI/MSI-X [ 519.824899] vfio-pci 0000:07:00.0: irq 82 for MSI/MSI-X [ 523.395384] vfio-pci 0000:03:00.1: irq 100 for MSI/MSI-X

So 'yay!!' for getting closer, but still not quite there.

Offline

#1429 2014-03-23 22:50:45

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jesse2004 wrote:
SpacePirate wrote:
trimm wrote:

<SNIP>

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted... hmm

I experience the exact same behaviour on my machine. Its working now with ALSA but the quality isnt good. Has anyone fixed this issue or got it working with Pulse Audio (which I would prefer)

Similar problem here. From my reading, the pulseaudio "Connection refused" errors above might be due to the fact that normally pulseaudio is configured in single user mode, which means only a specific session of some user can use pulseaudio in a specific time(unless you are root). If I start a pulseaudio instance as root and then run qemu as root, these errors are gone. But yes, the sound is really harsh and distorted.

Configure pulseaudio in system mode may help eliminate the errors. However I have no idea regarding the ultra-low sound quality. Hope someone can shed light on this issue.

I use the following invocation for a Win7 x64 guest:

sudo QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 QEMU_AUDIO_DAC_FIXED_FREQ=48000 QEMU_AUDIO_ADC_FIXED_FREQ=48000 QEMU_ALSA_DAC_BUFFER_SIZE=16384 qemu-system-x86_64 -enable-kvm -M q35 <snip> -soundhw hda,ac97

In my guest, I have disabled the high-definition audio and have installed the x64 AC'97 drivers from Realtek's website.

My audio is now clear, though there is the tiniest bit of lag occasionally.  Your mileage may vary and I'm sure there is a better way to do it, but I stopped fiddling once I'd gotten it running big_smile

Offline

#1430 2014-03-24 01:38:31

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I still can't bootup 2nd VM with another GTX480. Now I focus on some setting problem.

Strange is, I can't start with just 2nd VM. And just start 1st VM is OK.

If I start 1st VM then start 2nd VM, 1st VM worked, 2nd still can't.

But if I converse the steps (start 2nd VM then start 1st VM), 1st VM will got KVM error.

KVM internal error. Suberror: 1 emulation failure EAX=ffff2b56 EBX=00000000 ECX=00000000 EDX=00000300 ESI=00006d0a EDI=00006d25 EBP=000041f7 ESP=00000014 EIP=00032dcf EFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =2b2b 0002b2b0 ffffffff 00809300 CS =6d23 0006d230 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =c000 000c0000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f68c0 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

And here is my dmesg when I start 2nd VM only.

[ 259.375335] VFIO - User Level meta-driver version: 0.3 [ 259.392054] ehci-pci 0000:00:1d.0: remove, state 1 [ 259.392062] usb usb2: USB disconnect, device number 1 [ 259.392063] usb 2-1: USB disconnect, device number 2 [ 259.392065] usb 2-1.1: USB disconnect, device number 3 [ 259.399330] hid-generic 0003:04D9:2011.0005: can't resubmit intr, 0000:00:1d.0-1.2/input0, status -19 [ 259.414950] usb 2-1.2: USB disconnect, device number 4 [ 259.486485] usb 2-1.5: USB disconnect, device number 5 [ 259.486487] usb 2-1.5.1: USB disconnect, device number 7 [ 259.488913] usb 2-1.6: USB disconnect, device number 6 [ 259.495437] ehci-pci 0000:00:1d.0: USB bus 2 deregistered [ 259.598585] device tap1 entered promiscuous mode [ 259.599065] br0: port 3(tap1) entered forwarding state [ 259.599070] br0: port 3(tap1) entered forwarding state [ 259.898723] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003) [ 260.030186] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa [ 262.975682] dmar: DRHD: handling fault status reg 2 [ 262.975689] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000 [ 262.975689] DMAR:[fault reason 06] PTE Read access is not set [ 274.634766] br0: port 3(tap1) entered forwarding state [ 322.927939] audit: type=1400 audit(1395624287.279:64): apparmor="DENIED" operation="capable" profile="/usr/sbin/cupsd" pid=1705 comm="cupsd" capability=36 capname="block_suspend"

Also, I don't know why it got this message in dmesg every boot, and it's a lot of then.

[ 12.903046] vgaarb: this pci device is not a vga device

And here is my QEMU start command, I thought that maybe it's anothe problem, too.

(1st VM code) #!/bin/bash vfio-bind 0000:02:00.0 0000:02:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \ -smp 2,sockets=1,cores=1,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/ak/test.img,format=raw,if=virtio \ -net nic,model=virtio,vlan=0,macaddr=52-54-00-f5-44-04 -net tap,vlan=0,ifname=tap0,script=no
(2nd VM code) #!/bin/bash vfio-bind 0000:03:00.0 0000:03:00.1 0000:00:1d.0 qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \ -smp 2,sockets=1,cores=1,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1d.0,bus=pcie.0 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/ak/test2.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/home/ak/ubuntu.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \ -net nic,model=virtio,vlan=0,macaddr=52-54-00-f5-44-08 -net tap,vlan=0,ifname=tap1,script=no

Still, I'm apologize my poor English.

Many thanks

Last edited by AKSN74 (2014-03-24 01:41:44)

Offline

#1431 2014-03-24 22:09:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FYI, if you're trying Passmark Performance Test on Intel, avoid -cpu host or any of the models above Penryn, they seem to result in a BSOD.  I've reported the problem to Passmark, so hopefully they'll resolve it soon.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1432 2014-03-24 22:37:19

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.
However as soon as I quit(!) the game it will freeze the whole machine up. I cant ping the guest from the host so it is totally frozen. The screen is showing whatever was on it the second it froze.
I do not receive any error messages in journalctl, and the Windows guest does not log anything to the event viewer.

The only way to unfreeze my Windows screen is to reboot the host, killing the qemu processes is not possible.

Playing Guild Wars 2 at Ultra Settings does work without any issues though.
(Got a Nvidia GTX 670, i7 CPU(6 dedicated to the guest), 32GB RAM(16GB to the guest). I am not running processor pinning or huge pages, but I do have cache=none set to my hard drives.

Anyone having experience to something similar like this?

Offline

#1433 2014-03-25 02:01:53

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have finally found the real source of those random hardware failures I was experiencing on host: The HD 7660D IGP of AMD A10-5700. Use of that GPU on host makes the host system very unreliable. Further, there appears to be nothing wrong with the cheap PCI-E NICs I mentioned earlier. It was a simple coincidence that their removal made my system slightly more stable.

Here's a summary of all the hardware I have tested with VGA vfio-pci:

System 1: AMD A10-5700 APU + Asus F2A85-M PRO (BIOS 6308)
- OK (must disable integrated graphics)
- passthrough of the integrated audio device also tested to work with Win 7

System 2: AMD FX 8350 + Asus Sabretooth 990FX R2.0 (BIOS 2104)
- OK
- passthrough of the integrated audio device also tested to work with Win 7

GPU 1: Radeon HD 7660D IGP of AMD A10-5700 APU
- If used on host, crashes the host horribly
- If used on guest, does not crash the host, but does not work either (no picture)
- If booted on headless Linux guest, kernel complains about invalid rom contents (or booting with romfile= extracted earlier stucks the VM in busy loop)
==> USELESS

GPU 2: Asus Radeon HD 3870
- On host: OK
- On Linux guest: OK (glxgears tested)
- On Win 7 guest: Device Manager reports Code 43 (or BSoD STOP 0x3B after forced device reinstall)
- On WinXP guest: Might work, but I have trouble installing drivers
==> For host or Linux guest only

GPU 3: Sapphire Ultimate Radeon HD 6670 (11192-06-20G)
- On host: OK
- On Linux guest: OK (glxgears tested)
- On Win 7 guest: OK (Graphics/Gaming score 6.9)
- On WinXP guest: Might work, but I have trouble installing drivers
==> Very good, well supported GPU

GPU 4: GeForce 9600 GT
GPU 5: Radeon X300
GPU 6: Radeon X1600
- None of these older cards appear to work as these refuse to initialize together with other PCI-E GPUs (lspci on host shows up to 1 card only)
==> Useless???

Offline

#1434 2014-03-25 02:11:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

GPU 4: GeForce 9600 GT
GPU 5: Radeon X300
GPU 6: Radeon X1600
- None of these older cards appear to work as these refuse to initialize together with other PCI-E GPUs (lspci on host shows up to 1 card only)
==> Useless???

The Radeon X-series are useless.  I have an X500 that I spent some time reverse engineering quirks for, but I never got it working and decided I didn't want to set a precedence for maintaining code for devices that really aren't useful as a modern, high-performance card.  The 9600GT should work.  I think the Nvidia quirks we have work down to the 8-series, maybe even the 7-series, but obviously if the platform won't initialize the slot it's a non-starter.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1435 2014-03-25 05:53:21

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The 9600GT should work.  I think the Nvidia quirks we have work down to the 8-series, maybe even the 7-series, but obviously if the platform won't initialize the slot it's a non-starter.

Thanks for the info. I did some more testing with my 9600GT. No luck with vfio-pci. All I get is blank picture even if it's the only GPU in the system.

And now it froze the host system (Sabretooth) completely...! I can't even SSH.

Here's how I tried to use it from console:

GRUB: pci-stub.ids=10de:0622 vfio-bind 0000:01:00.0 qemu-system-x86_64 -machine accel=kvm -M q35 -nographic -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

It would have been nice, had this worked, but after a host crash I don't think I'll keep trying unless there is some known trick.

Has anyone tested a brand new Radeon R7 260? It might be my next purchace.

Offline

#1436 2014-03-25 07:33:13

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:
ahom wrote:

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money tongue). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Certainly an interesting project! How do you plan to assign peripherals like keyboard, mouse, etc. to the hosts? You might want to look for a motherboard that has at least 3 distinct USB controllers, so you can pass through one chip to each VM and have one left for the host.

An addon PCI-E usb controller could also do the trick.
But is it really necessary to passthrough USB controllers? I'm asking this because so far I have experienced absolutely no reason to do that. Not even for light gaming use.

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

Offline

#1437 2014-03-25 07:47:44

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuoni wrote:

I use the following invocation for a Win7 x64 guest:

sudo QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 QEMU_AUDIO_DAC_FIXED_FREQ=48000 QEMU_AUDIO_ADC_FIXED_FREQ=48000 QEMU_ALSA_DAC_BUFFER_SIZE=16384 qemu-system-x86_64 -enable-kvm -M q35 <snip> -soundhw hda,ac97

Thanks for sharing!

Offline

#1438 2014-03-25 07:50:25

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.

May I ask your audio configuration?

Offline

#1439 2014-03-25 08:24:01

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl or someone - please comment on AMD setup performance. ahl, my setup is similar to your AMD FX 8350 + Asus Sabretooth 990FX R2.0. Does that rig run fine? im not sure why but i have terrible performance. to get a perspective - arma3 playable is at normal settings, better yet low. i run git qemu and kernel is 3.14-rc6 with patches from first post.

One more thing - anyone happen to test 3.14-rc7 kernel? Any stability issues? Because i have some that pop up quite fast after boot. Wondering if im only one unlucky,

Offline

#1440 2014-03-25 08:40:13

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

ahl or someone - please comment on AMD setup performance. ahl, my setup is similar to your AMD FX 8350 + Asus Sabretooth 990FX R2.0. Does that rig run fine? im not sure why but i have terrible performance. to get a perspective - arma3 playable is at normal settings, better yet low. i run git qemu and kernel is 3.14-rc6 with patches from first post.

One more thing - anyone happen to test 3.14-rc7 kernel? Any stability issues? Because i have some that pop up quite fast after boot. Wondering if im only one unlucky,

I can't really comment on performance just yet, because I'm not even trying to get top speeds. I only need something that works.
I use a kernel a bit newer than 3.14-rc7. And looks like -rc8 is already out so I suggest trying that.

Edit: Upgraded to 3.14-rc8 + fix_memleak.patch

Last edited by ahl (2014-03-25 12:15:30)

Offline

#1441 2014-03-25 09:54:18

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.
However as soon as I quit(!) the game it will freeze the whole machine up. I cant ping the guest from the host so it is totally frozen. The screen is showing whatever was on it the second it froze.
I do not receive any error messages in journalctl, and the Windows guest does not log anything to the event viewer.

The only way to unfreeze my Windows screen is to reboot the host, killing the qemu processes is not possible.
Anyone having experience to something similar like this?

I had a similar issue once and I think it was due to a qemu bug that has already been fixed. I am not 100% sure, but iirc just rebuilding the latest  qemu-git fixed it.

Offline

#1442 2014-03-25 11:59:45

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
novist wrote:

ahl or someone - please comment on AMD setup performance. ahl, my setup is similar to your AMD FX 8350 + Asus Sabretooth 990FX R2.0. Does that rig run fine? im not sure why but i have terrible performance. to get a perspective - arma3 playable is at normal settings, better yet low. i run git qemu and kernel is 3.14-rc6 with patches from first post.

One more thing - anyone happen to test 3.14-rc7 kernel? Any stability issues? Because i have some that pop up quite fast after boot. Wondering if im only one unlucky,

I can't really comment on performance just yet, because I'm not even trying to get top speeds. I only need something that works.
I use a kernel a bit newer than 3.14-rc7. And looks like -rc8 is already out so I suggest trying that.

thanks for heads-up! i tried new kernel. so far system is running for over an hour. keeping fingers crossed it stays that way. there is like 30% increase in fps (from 10 to 15) from rc6 kernel in directx9 complex passmark test. has little play-session, its still laggy. this time i kept gpu-z on checking some stats. apparently gpu most of the time runs at 20% load. that is really wrong... fan speed is reported at 33%. temperature at ~60 degrees under max load. no idea what to make of it.

Offline

#1443 2014-03-25 12:28:40

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
TripleSpeeder wrote:
ahom wrote:

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money tongue). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Certainly an interesting project! How do you plan to assign peripherals like keyboard, mouse, etc. to the hosts? You might want to look for a motherboard that has at least 3 distinct USB controllers, so you can pass through one chip to each VM and have one left for the host.

An addon PCI-E usb controller could also do the trick.
But is it really necessary to passthrough USB controllers? I'm asking this because so far I have experienced absolutely no reason to do that. Not even for light gaming use.

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

How does this work? Currently I pass through the USB devices (not the entire controller, just specific devices based on vendorid/productid) which means I can only use keyboard/mouse on the guest or the host. I can switch between them, but it requires using qemu console to detach/reattach. If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

Offline

#1444 2014-03-25 12:38:43

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:
ahl wrote:

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

How does this work? Currently I pass through the USB devices (not the entire controller, just specific devices based on vendorid/productid) which means I can only use keyboard/mouse on the guest or the host. I can switch between them, but it requires using qemu console to detach/reattach. If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

Qemu should open a blank window by default. You can use both mouse and keyboard on that. Any input is automatically sent to the guest.

Offline

#1445 2014-03-25 14:14:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
alexis_evo wrote:
ahl wrote:

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

How does this work? Currently I pass through the USB devices (not the entire controller, just specific devices based on vendorid/productid) which means I can only use keyboard/mouse on the guest or the host. I can switch between them, but it requires using qemu console to detach/reattach. If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

Qemu should open a blank window by default. You can use both mouse and keyboard on that. Any input is automatically sent to the guest.

I use this for quick testing as well, but with the -vnc option to the VM.  It doesn't work well for gaming though, Borderlands2 at least is completely unusable with this mouse input.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1446 2014-03-25 14:27:11

tuoni
Member
Registered: 2013-12-19
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

http://synergy-foss.org

Offline

#1447 2014-03-25 15:22:29

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

Edit: Upgraded to 3.14-rc8 + fix_memleak.patch

erm ... you mean memleak fix didnt make it to 3.14 yet ?? ouch.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1448 2014-03-25 16:38:47

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jesse2004 wrote:
trimm wrote:

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.

May I ask your audio configuration?

I am passthroughing my onboard audio card.
-device vfio-pci,host=00:1b.0

And the USB rocksmith cable:
-usb -usbdevice host:12ba:00ff


Flyser wrote:

I had a similar issue once and I think it was due to a qemu bug that has already been fixed. I am not 100% sure, but iirc just rebuilding the latest  qemu-git fixed it.

Thank you so much Flyser, I will definately try to use/reinstall the latest qemu-git!

Edit: is there a reason for why we are not using the AUR package of qemu-git? What is so special with the one OP has posted?
Edit2: Still got the same issue, the following is the only thing in my journalctl while the VM froze:

Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe590 fff97410 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5a0 fff97420 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5b0 fff97430 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5c0 fff97440 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5d0 fff97450 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5e0 fff97460 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5f0 fff97470 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe600 fff97480 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe610 fff97490 00000000 0d00000a 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe620 fff974a0 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe630 fff974b0 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe640 fff974c0 00000000 0d00000c 07058000 Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe650 fff974d0 00000000 0d00000c 07058000

Last edited by trimm (2014-03-25 20:56:42)

Offline

#1449 2014-03-27 12:37:30

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

anyone tried the new QEMU 2.0 rc0 ?

As soon as i startup my win7 guest with it instead of good old qemu 1.7.0 - i get a BSOD with STOP 0x00..01e
No Config Changes whatsover - i just replaced the qemu binary...

Perhaps the Memory layout Changes in Qemu 2.0 (see http://wiki.qemu.org/ChangeLog/2.0 ) require a Guest-Reinstall from Scratch? i dont hope so, but...

Thanks for your thouhts on that

Greetings
K.

Offline

#1450 2014-03-27 21:16:18

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FYI: All my adventure with Powercolor 6950 ends here... I think I had a faulty card from the beginning.
It was crashing linux with the original driver when I was using opencl, and now with pass-trough.
The GPU-Z was showing only half of the shaders in win7 guest. Once connecting in guest the DVI, system was
freezing. Was reading in different forums that Powercolor is just crap... better spend few more $$$ and go for better
brand. Managed though to reflash the original Bios, though is showing strange characters in bios and in text mode.
Removed now the card, power dropped from 120W to 80W in iddle mod... Was using 40W for nothing... wink

Offline

#1451 2014-03-28 17:18:36

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Okay, I am out of ideas now. I got everything running perfectly when starting qemu manual. Now i want to move to a libvirt xml file, so i can start/stop/manage my VMs with virtmanager. But no matter what I do, i don't get the card passed through.
Here is a minimal xml config to reproduce:

When starting this instance e.g. through "virsh start win7" i get the following error:

# virsh start win7
error: Failed to start domain win7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What I did:

  • Make sure that AMD-card is assigned to vfio

  • Edit /etc/libvirt/qemu.conf and set user/group qemu runs as to "root"

  • Getting desperate: chmod a+rw /dev/vfio/13

But no luck. Everytime the same error :-(

Does anyone have a clue what I am missing? As I said, when I launch qemu manually (as root) everything works fine...

Problem solved!

I missed one part of the virt-manager configuration to fix the permissions issue: You need to edit /etc/libvirt/qemu.conf and explicitly allow access to the /dev/vfio/<group>:

... cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "dev/vfio/3", "dev/vfio/13", "dev/vfio/16", ] ...

As you can see I added three groups that i am passing through.

With this change I everything was fine and I can now happily start/stop/restart etc. my VM through virt-manager. By the way, there is a tool included with libvirt that supports you with converting a qemu-command-line to xml config: virsh domxml-from-native. It even takes care of adding the extra namespace to the config file to support the quemu-commandline directive. I had to remove one or two items from my commandline to get it running, but still it was a great help.

Another big hurdle was getting sound via pulseaudio to work. As a vm started by virt-manager is not aware of the current login session it has no access to your normal pulseaudio instance. In order to get this working you need to have pulseaudio running in system mode. Note that this is in general considered a security risk and not recomended, however it is the only way (to my knowledege) to get the pulseaudio integration to work.
You need to do the following steps:

  • Create a user "pulse" with homedir "/var/run/pulse"

  • Create a group "pulse", add user pulse to this group

  • Add user pulse to group "audio" so it is allowed to access the audio hardware

Now you can already try to start pulseaudio in system mode (make sure to stop any running pulseaudio instance from your login session). As root, type:

# pulseaudio --system -v

If you see some error about problems connecting to DBus: Check https://bbs.archlinux.org/viewtopic.php … 81#p563481

Now you should already be able to startup your vm and the sound should work. If everything is working you can create a systemd service to start pulseaudio on boot in system mode:

# systemd service spec for pulseaudio running in system mode -- not recommended though! # on arch, put it under /etc/systemd/system/pulseaudio.service # start with: systemctl start pulseaudio.service # enable on boot: systemctl enable pulseaudio.service [Unit] Description=Pulseaudio sound server After=avahi-daemon.service network.target [Service] ExecStart=/usr/bin/pulseaudio --system --disallow-exit --disallow-module-loading ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target

(Source: https://gist.github.com/awidegreen/6003640)

And finally you need to prevent autostarting a pulseaudio instance at login. I achieved this by removing /etc/xdg/autostart/pulseaudio-kde.desktop.

My current config:

# virsh dumpxml windows <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>windows</name> <uuid>f702c5eb-83d4-4b50-b909-8da878e08d36</uuid> <memory unit='KiB'>8290304</memory> <currentMemory unit='KiB'>4145152</currentMemory> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <loader>/usr/share/qemu/bios.bin</loader> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pcid'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='smx'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='tm2'/> <feature policy='require' name='ss'/> <feature policy='require' name='pbe'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/sdc'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/michael/hdd/isos/virtio-win-0.1-74.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='52:54:00:29:ae:89'/> <source bridge='xenbr0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/> <qemu:arg value='-device'/> <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1a.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> </qemu:commandline> </domain>

Everything is working perfectly, I can restart the VM as often as I like, played several hours iRacing with framerates in the range of 180-220 FPS (full HD with max quality) and shortly tried several other games which all seem to run fine with top performance. Also for the sound there is no noticeable delay using pulseaudio.

I finally deleted my windows partition and removed all dualboot settings from grub :-)

Only thing to keep in mind is you need to manually set up the vfio groups before trying to start the VM. But somewhere in this thread there was already information how to set this up to happen automatically at boot time, so this should also be solvable smile

Offline

#1452 2014-03-29 23:26:19

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi together,
I`m trying to setup a Windows 7 guest and was following this thread. I successfully created a q35 guest with kvm enabled using

-vga std \ -mem-path /dev/hugepages \ -m 4096 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \

When I finished the initial guest setup, which included installing nvidia drivers for my GTX 760, I tried to switch to

-vga none

Doing so I can see the initial bios screen and the OS is starting to boot up. However during that, my mouse cursor gets laggy and after a couple of seconds the whole host crashes. I read that the nouveau module could cause such things, so that module is already blacklisted, which does not solve the problem. I`m using an Intel IGP as primary graphics device and the compiled the kernel, qemu and seabios from the packages provided here.

At the moment I`m out of ideas.
My current start script is:

qemu-system-x86_64 \ -nodefaults \ -name win7 \ -enable-kvm \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -machine type=q35,accel=kvm \ -m 4096 \ -mem-path /dev/hugepages \ -mem-prealloc \ -k de \ -bios /usr/share/qemu/bios.bin \ -vga none \ -drive if=none,file=drive/win_drive.img,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1 \ -boot order=c \ -rtc base=utc \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -usb \ -device ich9-usb-uhci3,id=uhci \ -device usb-ehci,id=ihci \ -usbdevice tablet \

Is there any way, how I could track down, what I may have forgotten to configure, or whats going wrong? Reading through the whole thread didn't help me so far. Its rather confusing, because there is so much information here.
Any hints would be helpful.
Thanks in advance
apex

Offline

#1453 2014-03-29 23:55:31

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What motherboard and CPU do you have? I've got i7-4771, ASRock Z87E-ITX with Intel as host's graphics and GTX 760 for Windows 8.1 and it's working flawlessly.

Last edited by dwe11er (2014-03-31 17:32:10)

Offline

#1454 2014-03-30 00:45:19

etcet
Member
Registered: 2010-02-15
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to nbhs and everyone else involved in this thread, I've got VGA passthrough working using this guide with the following setup:

Intel 4770
Asus Z87-Plus
Gigabyte "Windforce" GTX 770 GV-N770OC-2GD

I'm starting X with the onboard Intel GPU and getting near native Windows performance on a passthroughed 770.

For others and my future self, here are the steps involved. I think this is everything:

Install qemu-git
Install seabios-git

The i915 kernel patch is required. I couldn't get the provided mainline kernel to work. I built the latest kernel and added the i915 patch with the following:

pacman -S abs base-devel ABSROOT=. abs core/linux cd core/linux/ wget https://gist.githubusercontent.com/anonymous/732acf9551136a78e91b/raw/5b97d50e7fb7faced258ad8948b7a82127d25f31/i915_313rc4.patch wget https://gist.githubusercontent.com/etcet/9864922/raw/e49d204ddaf2d6130c91d94d0ce6a9123c6197cb/i915-pkgbuild.patch patch -p3 < i915-pkgbuild.patch makepkg pacman -U linux-i915-headers-3.13.7-1-x86_64.pkg.tar.xz pacman -U linux-i915-3.13.7-1-x86_64.pkg.tar.xz

More: https://wiki.archlinux.org/index.php/Ke … ild_System

Get PCI stub ids for passthrough devices:

[chris@archserver linux]$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) [chris@archserver src]$ lspci -n | grep "01:00.0" 01:00.0 0300: 10de:1184 (rev a1)

Add new kernel to grub, /etc/grub.d/40_custom:

menuentry "Arch Linux i915 kernel w/ iommu" { set gfxpayload=keep insmod ext2 echo 'Loading Arch Linux i915 kernel w/ iommu ...' linux /boot/vmlinuz-linux-i915 root=UUID=f4af663b-5407-498c-b46d-d3e2b9b6bc64 rw intel_iommu=on pci-stub.ids=10de:1184 echo 'Loading Arch Linux i915 kernel w/ iommu initramfs ...' initrd /boot/initramfs-linux-i915.img }

Build grub conf:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Miscellaneous:

[chris@archserver src]$ cat /etc/modprobe.d/blacklist.conf blacklist nouveau [chris@archserver src]$ cat /etc/modprobe.d/vfio_iommu_type1.conf options vfio_iommu_type1 allow_unsafe_interrupts=1

Simple xorg.conf for running host with onboard Intel GPU: https://gist.github.com/etcet/9865391

Reboot into new kernel and take the oppurtunity to check/change BIOS settings:

Advanced -- System Agent Configuration ---- Vt-d: Enabled ---- Graphics Configuration ------ Primary Display: iGPU

Test it:

sudo /usr/bin/vfio-bind 0000:01:00.0 sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

Last edited by etcet (2014-03-30 00:46:12)

Offline

#1455 2014-03-30 07:28:04

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Preparing the GPU so we can bind it to vfio

For the next step we need to:

  1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

    Example, blacklisting the opensource radeon module:

    echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 
  2. Use pci-stub

    In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub.

    NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

    mkinitcpio -p linux-mainline

    lspci

    07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

    lspci -n

    07:00.0 0300: 1002:6719 <-- radeon 6950 07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

    Now add this to your grub cfg file:

    pci-stub.ids=1002:6719,1002:aa80

    dmesg | grep pci-stub

    [ 2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.096160] pci-stub 0000:07:00.0: claimed by stub [ 2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.096174] pci-stub 0000:07:00.1: claimed by stub [ 2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

Hi all.

I am confused about whether I should do both 1 and 2, or only 1 if I am okay with blacklisting the Radeon module.

The wording "For the next step we need to: 1)    2)" and "In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub." suggests it is an choice between the two, but I am not so sure. Everywhere I see people looking at the pci-stub grep's. So it seems everybody is doing pci-stub...

My hoped setup is with a single R9 280x (kernel calls it Radeon 79something) for the Windows guest and Intel built-in for Linux host. So as far as I can figure out I would be fine with black listing the radeon module.

Last edited by jonascj (2014-03-30 07:36:00)

Offline

#1456 2014-03-30 07:59:38

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

What motherboard and CPU do you have? I've got i7-4771, ASRock Z86E-ITX with Intel as host's graphics and GTX 760 for Windows 8.1 and it's working flawlessly.

I`ve got an Core i7-3770 and an Asrock Z68 Pro3 Gen3
I`m booting with

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=0bf391df-8b9d-4a36-9961-9cc54bcd24cd rw intel_iommu=on hugepagesz=2MB hugepages=2100 pci-stub.ids=10de:1187,10de:0e0a,1102:0012,8086:1c2d

binding the graphics adapter works without reporting any errors

EDIT

dmesg | grep IOMMU ⏎ [ 0.000000] Intel-IOMMU: enabled [ 0.089982] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.089986] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.090057] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.470765] IOMMU 0 0xfed90000: using Queued invalidation [ 0.470766] IOMMU 1 0xfed91000: using Queued invalidation [ 0.470767] IOMMU: Setting RMRR: [ 0.470776] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.471865] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbddcd000 - 0xbddf7fff] [ 0.471880] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbddcd000 - 0xbddf7fff] [ 0.471889] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.471894] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Last edited by apex8 (2014-03-30 08:07:56)

Offline

#1457 2014-03-30 13:21:20

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jonascj wrote:

Hi all.

I am confused about whether I should do both 1 and 2, or only 1 if I am okay with blacklisting the Radeon module.

The wording "For the next step we need to: 1)    2)" and "In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub." suggests it is an choice between the two, but I am not so sure. Everywhere I see people looking at the pci-stub grep's. So it seems everybody is doing pci-stub...

My hoped setup is with a single R9 280x (kernel calls it Radeon 79something) for the Windows guest and Intel built-in for Linux host. So as far as I can figure out I would be fine with black listing the radeon module.

If blacklisting works for you, then you only need to use step 1 and can skip step 2.

Offline

#1458 2014-03-30 19:01:43

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I went through my qemu configuration and gave it a try with a whole new guest installation. I used the following script

sudo qemu-system-x86_64 \ -nodefaults \ -name win7 \ -enable-kvm \ -machine type=q35,accel=kvm \ -m 4096 \ -mem-path /dev/hugepages \ -mem-prealloc \ -k de \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive if=none,file=drive/testdrive.img,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1 \ -boot order=c \ -drive file=drive/win_7_sp1.iso,id=winimg \ -device ide-cd,bus=ide.0,drive=winimg \ -drive file=drive/virtio_drv.iso,id=isocd \ -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on \ -usb -usbdevice tablet \ -spice port=5930,disable-ticketing \ -net nic -net bridge,br=kvmbr0 \

So, from the very first moment I used my gtx760. The setup was totally fine, but once I installed the recent nvidia drivers I end up in host a crash during guest boot.
Any ideas what could be the problem? I read I shouldn't run qemu using sudo, but without it I get errors when starting qemu. Anyway, until the nvidia installation the guest runs fine.
So would compiling a newer qemu version a possible solution?

Offline

#1459 2014-04-01 14:03:43

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Online

#1460 2014-04-01 14:17:09

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Here's a list. Seems like there are some people with that board on there.


i'm sorry for my poor english wirting skills…

Offline

#1461 2014-04-01 16:42:26

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Here's a list. Seems like there are some people with that board on there.

Thanks a lot for link, this one is really useful.

One more question. As I understand from reading forums at moment it's impossible to use PCIe device on host again it's assigned to guest even after guest shutdown. E.g you have to restart host to use GPU on host again. Is it hardware or software limitation? Is there any information about that?

Just want to understand what hardware I should get to avoid as many problems as possible.

Last edited by SXX (2014-04-01 16:44:28)

Online

#1462 2014-04-01 17:02:50

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:
andy123 wrote:

Here's a list. Seems like there are some people with that board on there.

Thanks a lot for link, this one is really useful.

One more question. As I understand from reading forums at moment it's impossible to use PCIe device on host again it's assigned to guest even after guest shutdown. E.g you have to restart host to use GPU on host again. Is it hardware or software limitation? Is there any information about that?

Just want to understand what hardware I should get to avoid as many problems as possible.

Yeah, that list should probably linked to in the first post, if it isn't already.

Using devices without a host reboot, after shutdown of the guest works with PCI devices for me. I simply unbind the device from vfio-pci and "assign" it to another driver. (tested with my Marvell (SATA/IDE) and AMD USB controllers)

And by "assign the device to another driver" I mean something like this:

cd /sys/bus/pci/drivers/vfio-pci echo 0000:02:00.0 > unbind cd ../ahci echo 1b4b 91a0 > new_id

As for doing this with GPUs: I have no idea. Maybe some other people in this thread?

PS: I have no idea if any of this should be done the way I do it or if it's a good idea in any way, but it WORKSFORME™


i'm sorry for my poor english wirting skills…

Offline

#1463 2014-04-01 17:27:17

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for more information.

andy123 wrote:

As for doing this with GPUs: I have no idea. Maybe some other people in this thread?

GPU was my main concern because I want to use discrete GPU on host (Radeon + DRI_PRIME) when guest is off.

After many hours of search in internet I suppose it's not possible at moment, but I wonder if this software or hardware limitation.

Last edited by SXX (2014-04-01 17:28:03)

Online

#1464 2014-04-01 18:02:48

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

GPU was my main concern because I want to use discrete GPU on host (Radeon + DRI_PRIME) when guest is off.

After many hours of search in internet I suppose it's not possible at moment, but I wonder if this software or hardware limitation.

Actually it is possible. I do that on my system (dh87rl, haswell i7, nvidia gtx660). No idea if it works with amd gpus though.

Last edited by Flyser (2014-04-01 18:02:59)

Offline

#1465 2014-04-01 18:08:40

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

Actually it is possible. I do that on my system (dh87rl, haswell i7, nvidia gtx660). No idea if it works with amd gpus though.

Interesting... What driver you using on host for Nvidia? It's loading kernel module after GPU used in guest?

Last edited by SXX (2014-04-01 18:12:18)

Online

#1466 2014-04-01 19:11:42

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
SXX wrote:

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Here's a list. Seems like there are some people with that board on there.

Heya, this list (and my entry on it) reminded me I wanted to tell you a funny story about my R290X.

The most important thing first: It’s working. Kinda.

First of all I see to have some bus reset issue because it’s only working after a reboot or stand-by. But what’s *really* astonishing is that it *only* works as long as the old HD4850 is present in the Arch host system and assigned to vfio-pci (IOMMU group), too. I tried it multiple times: Removing the HD4850 (including kernel boot parameters and vfio-pci.cfg) results in a BSoD of atikmpag.sys (I think). A similar problem occurs if I actually assign the HD4850 as *secondary* device to the VM; putting it as the *primary* device gives me a Error 43 for the 290X. The only way to make my 290X work properly is to have the HD4850 present on the host system and assigned to vfio-pci but not actually hand it to the VM.

Oh and I load the vbios image.

PS:

@SXX: Yes.

But you may hit some bumps on the road. E.g. I reverted to a specific git revision of qemu-git and libcacard multiple times already because newer versions cause weird graphic bugs on the host on launch (and on launch of the VM only).

Last edited by blacky (2014-04-01 19:14:15)

Offline

#1467 2014-04-01 19:37:14

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

Interesting... What driver you using on host for Nvidia? It's loading kernel module after GPU used in guest?

I am using the nvidia binary driver and haven't tried nouveau so far. As long as the device is no longer bound to vfio *and* I execute the following script after the actual qemu command, it loads fine:

qemu-system-x86_64 -name bus-reset-only -enable-kvm -boot order=c,menu=off,reboot-timeout=0 -machine type=q35,accel=kvm -cpu host -smp 1 -m 32 -k de -vga none -display none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -net none -no-reboot

This simply issues a bus reset (afaik).

Offline

#1468 2014-04-01 21:14:32

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

@SXX: Yes.

But you may hit some bumps on the road. E.g. I reverted to a specific git revision of qemu-git and libcacard multiple times already because newer versions cause weird graphic bugs on the host on launch (and on launch of the VM only).

I think it's fine. Didn't you tried to return GPU to host and use it with fglrx?

Flyser wrote:

I am using the nvidia binary driver and haven't tried nouveau so far. As long as the device is no longer bound to vfio *and* I execute the following script after the actual qemu command, it loads fine

Thanks for your answers. About two years ago I already tried to setup VGA passthough using XEN and in result I wasted money and time with no success so now I want to choose hardware carefully.

I'll have some hardware in few weeks and post here about results.

Last edited by SXX (2014-04-01 21:15:54)

Online

#1469 2014-04-01 21:38:04

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:
blacky wrote:

@SXX: Yes.

But you may hit some bumps on the road. E.g. I reverted to a specific git revision of qemu-git and libcacard multiple times already because newer versions cause weird graphic bugs on the host on launch (and on launch of the VM only).

I think it's fine. Didn't you tried to return GPU to host and use it with fglrx?

I meant the Intel on-board graphic I use for the host system. I don’t use the dedicated GPU for the host system, also because I hope I can basically completely shut it down (and save power) unless the related VM is running.

Flyser wrote:

This simply issues a bus reset (afaik).

Ahh … thanks, I was too lazy so far to look for this command in the thread. Maybe now I can start my VM without reboot / standby.

Offline

#1470 2014-04-02 00:57:37

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
SXX wrote:

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Here's a list. Seems like there are some people with that board on there.

AFAIK every ASRock mobo based on Z87 (H87 should have too) has VT-d support.

Offline

#1471 2014-04-02 03:14:04

lildigiman
Member
Registered: 2014-04-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone! Just today I finally got my Asrock Z77 Extreme4 to replace my P8Z77-V LK because I wanted IOMMU support.

I'm reading back on an earlier post made by GizmoChicken who has successfully utilized VGA passthrough on a stock Ubuntu Kernel and stock qemu 1.5.

My setup:
Intel 3570 (non-k)
Asrock Z77 Extreme4
GTX 660 ti for Guest and iGPU for Host

Software (which should/could work according to the post linked above):
Ubuntu 13.10 (stock kernel + latest updates)
Qemu 1.5 from repositories

My problem occurs when I try to start the VM and get:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Google searches for "vfio: error no iommu_group for device" hasn't brought me anywhere except one user's post that had never properly answered what this error suggests.

My dmesg | grep -e DMAR -e IOMMU seems to suggest that VT-d is properly enabled (I think):

[    0.000000] ACPI: DMAR 00000000bd969280 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.086615] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.086619] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.086690] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[   21.194509] vboxpci: IOMMU not found (not registered)

I also added the following to /etc/modules

pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_amd

So basically my question is what can I do from here with that error? I'm happy to provide more information if necessary.

Offline

#1472 2014-04-02 19:50:03

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

#1473 2014-04-02 19:57:52

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lildigiman wrote:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Google searches for "vfio: error no iommu_group for device" hasn't brought me anywhere except one user's post that had never properly answered what this error suggests.

So basically my question is what can I do from here with that error? I'm happy to provide more information if necessary.

Check if your device is bound to vfio-pci.

ls /sys/bus/pci/drivers/vfio-pci | grep 0000

Offline

#1474 2014-04-02 20:22:59

lildigiman
Member
Registered: 2014-04-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:

Check if your device is bound to vfio-pci.

ls /sys/bus/pci/drivers/vfio-pci | grep 0000

Thanks for the response, mv0lnicky. The output is as follows:
0000:00:1d.0
0000:01:00.0
0000:01:00.1

Device 1d.0 is a USB 2.0 Controller where I have a keyboard and mouse connected.

I might also add that it appears that my Kernel was slightly older than I had thought, so I updated to 3.12.6-031206-generic.

When I now try to start the VM my error is no longer vfio: error no iommu_group for device, but this:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset

A QEMU window then opens, but the VM stops instantly only outputting what is written above. The guest display (a third monitor) never displays anything anything at all.

On a side note (and maybe irrelevant?):
I also have display issues on the HOST when I try to start the VM (possibly compiz) causing my Host's secondary screen to display artifacts and inverted colors... not sure why or how this is happening. Note that this does not happen every time I try to start the VM, only maybe a third of the time.

Offline

#1475 2014-04-02 21:33:55

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad



gneville wrote:

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

#1476 2014-04-03 00:32:31

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As much as I appreciate the original post for sharing information about this great setup, it is, to say the least, very confusing.

So I've decided to do only what made sense and what could be confirmed elsewhere. See the below short story for a quick overview of my setup and my problem. See the long story for more in depth description:

#### Short story
I am trying to make this work with a minimum of configuration and alterations to different things. This is my setup:

Kernel: Stock 3.13.6 from Arch base install
QEMU: 1.7.0 installed via 'pacman -S qemu'
Seabios:  1.7.3.1-2 (already installed, maybe a dependency of qemu)

PCI-stub or blacklisting: Neither (according to #kvm@irc.freenode.net it is not necessary, unbind is sufficient)

BOOT options: I have only added "intel_iommu=on" to my boot line in grub.cfg

I can sucessfully unbind the Radeon card and attach it to vfio-pci:

[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver lrwxrwxrwx 1 root root 0 Apr 3 03:52 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/radeon [jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver lrwxrwxrwx 1 root root 0 Apr 3 03:51 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/snd_hda_intel [jonas@jonascj-pc ~]$ sudo vfio-bind 0000:01:00.1 0000:01:00.0 [sudo] password for jonas: [jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver lrwxrwxrwx 1 root root 0 Apr 3 03:53 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci [jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver lrwxrwxrwx 1 root root 0 Apr 3 03:53 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci [jonas@jonascj-pc ~]$

I am testing with this script:

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get a blank window on the host and the monitor connected to my radeon card goes from its standby-mode (blinking led and pitch black screen / no backlight) to its there-is-signal-mode (constant led and lighter black / backlight on). The image on my host monitor (displaying my terminal, blank qemu window etc.) gets miscolored. If I change to tty2 and back to tty1 (ctrl+alt+f2 -> ctrl+alt+f1) the miscoloration disappears but the Radeon screen stays blank and so does the qemu window on the host.

If I use the kernel from the original post I get output on the Radeon monitor. So the difference seems to be solely with the kernel. Which of those provided patches might be the needed one and where can I find more information about it? I do not like to blindly accept this preconfigured kernel with patches from who knows where. I think I've already shown (see long story for details) that quite a few things listd in the original post is not necessary and so have other people posting replies. Maybe only a single patch to the kernel is necessary?

#### Long story
## Software requirements

nbhs wrote:

Ingredients

You can also use these packages from AUR but you might run into problems:

  1. A kernel >= 3.9 (consider using kernel 3.12.0 or later)  is required  or if you're building it yourself it must be configured with the following:

    • CONFIG_VFIO_IOMMU_TYPE1=y

    • CONFIG_VFIO=y

    • CONFIG_VFIO_PCI=y

    • CONFIG_VFIO_PCI_VGA=y

  2. You'll need qemu >= 1.5.0 (consider using qemu 1.7 or later)  or qemu-git

  3. And seabios

Why is there two or three lists here without any clear separation?

I have chosen to use the stock kernel 3.13.6 which came with my base installation of Arch.

This kernel is / was configured as (according to 'zless /proc/config.gz')

CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y

(Note the three m's, not y's)

For qemu I've just installed the one from the official repo ('pacman -S qemu'). This is qemu 1.7.0 according to 'qemu-system-x86_64 -version'.

For seabios I've just used the version which was already installed (maybe it was installed as a requirement of qemu): seabios-1.7.3.1-2.


## Preparing the GPU for vfio binding

nbhs wrote:

Preparing the GPU so we can bind it to vfio

For the next step we need to:

  1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

    Example, blacklisting the opensource radeon module:

    echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 
  2. Use pci-stub

    In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub.

    NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

    mkinitcpio -p linux-mainline

    lspci

    07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

    lspci -n

    07:00.0 0300: 1002:6719 <-- radeon 6950 07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

    Now add this to your grub cfg file:

    pci-stub.ids=1002:6719,1002:aa80

    dmesg | grep pci-stub

    [ 2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.096160] pci-stub 0000:07:00.0: claimed by stub [ 2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.096174] pci-stub 0000:07:00.1: claimed by stub [ 2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

According to to people in #kvm@irc.freenode.net it is not necessary to blacklist modules or attach the device (gfx card) to pci-stub before attaching it to vfio-pci. It should be sufficient to unbind it which is already done in/by the vfio-bind script (see next section).

Do you guys still think this is necessary? I haven't been able to confirm this anywhere else.

## Setting up vfio and kvm modules

nbhs wrote:

Setting up vfio and kvm modules

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

I don't know about this. I have no idea how to verify if my board enables interrupt remapping or not. According to http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM ('dmesg | grep -e DMAR -e IOMMU') IOMMU is enabled.

Anyone who knows how to find out if these options are needed (i.e. how to verify if interrupt remapping is enabled)?

## Binding a device to vfio-pci

nbhs wrote:

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

Bind the GPU:

vfio-bind 0000:07:00.0 0000:07:00.1

This here makes really good sense. The script unbinds the device (gfx card) if it is already attached to a driver and afterwards it attaches it to vfio-pci.

I would say this works perfectly for me. This is my lspci output:

[jonas@jonascj-pc ~]$ lspci | grep -i radeon 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

After booting my stock kernel without any blacklisting or pci-stub'ing I can see the Radeon devices attached to one driver and after doing vfio-bind they are attached to vfio-pci:

[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver lrwxrwxrwx 1 root root 0 Apr 3 03:52 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/radeon [jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver lrwxrwxrwx 1 root root 0 Apr 3 03:51 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/snd_hda_intel [jonas@jonascj-pc ~]$ sudo vfio-bind 0000:01:00.1 0000:01:00.0 [sudo] password for jonas: [jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver lrwxrwxrwx 1 root root 0 Apr 3 03:53 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci [jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver lrwxrwxrwx 1 root root 0 Apr 3 03:53 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci [jonas@jonascj-pc ~]$

## Testing if its working out

nbhs wrote:

Testing if its working out

Lets test if its working, as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

NOTE: -m | VM memory
NOTE: -cpu host | Expose host cpu features.
NOTE: -smp 6,sockets=1,cores=6,threads=1 | Expose 1 cpu with 6 cores and 1 thread each.
NOTE: -bios | Bios image to load.
NOTE: -vga none | Disable cirrus emulated vga.
NOTE: -m q35 | Enable the new q35 emulated chipset, qemu defaults to the old i440fx chipset.

You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,
"-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" creates a pcie root port and we attach our gpu to it.
We must also use x-vga=on option on our gpu (and -vga none) otherwise it wont work

If you're using an intel cpu and nothing happens try this:

modprobe -r kvm_intel modprobe kvm_intel emulate_invalid_guest_state=0

NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this.

I am testing with this script:

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get a blank window on the host and the monitor connected to my radeon card goes from its standby-mode (blinking led and pitch black screen / no backlight) to its there-is-signal-mode (constant led and lighter black / backlight on). The image on my host monitor (displaying my terminal, blank qemu window etc.) gets miscolored. If I change to tty2 and back to tty1 (ctrl+alt+f2 -> ctrl+alt+f1) the miscoloration disappears but the Radeon screen stays blank and so does the qemu window on the host.

Last edited by jonascj (2014-04-03 01:10:42)

Offline

#1477 2014-04-03 10:19:42

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

etcet wrote:

Thanks to nbhs and everyone else involved in this thread, I've got VGA passthrough working using this guide with the following setup:

Intel 4770
Asus Z87-Plus
Gigabyte "Windforce" GTX 770 GV-N770OC-2GD

I'm starting X with the onboard Intel GPU and getting near native Windows performance on a passthroughed 770.

For others and my future self, here are the steps involved. I think this is everything:

Install qemu-git
Install seabios-git

The i915 kernel patch is required. I couldn't get the provided mainline kernel to work. I built the latest kernel and added the i915 patch with the following:

pacman -S abs base-devel ABSROOT=. abs core/linux cd core/linux/ wget https://gist.githubusercontent.com/anonymous/732acf9551136a78e91b/raw/5b97d50e7fb7faced258ad8948b7a82127d25f31/i915_313rc4.patch wget https://gist.githubusercontent.com/etcet/9864922/raw/e49d204ddaf2d6130c91d94d0ce6a9123c6197cb/i915-pkgbuild.patch patch -p3 < i915-pkgbuild.patch makepkg pacman -U linux-i915-headers-3.13.7-1-x86_64.pkg.tar.xz pacman -U linux-i915-3.13.7-1-x86_64.pkg.tar.xz

More: https://wiki.archlinux.org/index.php/Ke … ild_System

Hi etcet

I am trying to figure out what the minimum requirements in terms of patches and configuration is (for my setup, I cannot test other setups).

My setup is very similar to yours:
Z87 chipset (Asrock Z87M Extreme 4)
Intel Core i7 4770,
Intel integrated gpu for the host
A single dedicated card (Radeon R9 280X / HD7970)

So far I've gotten output on my monitor connected to the dedicated card with these steps:

  1. Use main line kernel provided by OP

  2. Use qemu and seabios from offical repo ('pacman -S qemu seabios) which results in qemu 1.7.0 and seabios 1.7.3.1-2

  3. vfio-bind 0000:01:00.0 0000:01:00.1    (no pci-stub or blacklisting)

  4. Test with this script and I get seabios output on the monitor connected to the dedicated card:

    #!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Now I just wonder what the minimum requirements for the kernel is. The stock 3.13.6 kernel (from an Arch base-install) does not work. It gives me blank screen on the monitor connected to the dedicated card and miscolors my host display (the Intel integrated gpu). Then  saw your post about "the i915 patch is needed" and thought maybe that is my minimum requirement.

But how did you know what the "i915 kernel patch" is and where to obtain it? You use these patches (one for the kernel code itself and one for the pkgbuild file):
https://gist.githubusercontent.com/anon … 3rc4.patch
https://gist.githubusercontent.com/etce … uild.patch

How did you find those patches, who made them, where do you plan on finding updates to those if you suddenly decide to upgrade your kernel in a few months?

Edit: I have patched the 3.13.8 core kernel with the two patches you linked in your post, build it, booted it and it works (i.e. gives me seabios output on the monitor connected to the dedicated Radeon card). So this seems to be the only patching to the kernel I need (I of course haven't gone any further in the process - installing windows, drivers, played games etc. - maybe I need more patches later on due to performance etc.). The question remains the same though - where do these patches come from etc.?

Best regards
Jonas

Last edited by jonascj (2014-04-03 12:05:06)

Offline

#1478 2014-04-03 11:59:30

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

headsup - some kvm improvements at http://lkml.iu.edu/hypermail/linux/kern … 03070.html

For x86 there are optimizations for debug registers, which trigger on
some Windows games, and other important fixes for Windows guests. We now
expose to the guest Broadwell instruction set extensions and also Intel
MPX. There's also a fix/workaround for OS X guests, nested virtualization
features (preemption timer), and a couple kvmclock refinements.

Cant test myself just yet

Offline

#1479 2014-04-03 15:00:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jonascj wrote:

Edit: I have patched the 3.13.8 core kernel with the two patches you linked in your post, build it, booted it and it works (i.e. gives me seabios output on the monitor connected to the dedicated Radeon card). So this seems to be the only patching to the kernel I need (I of course haven't gone any further in the process - installing windows, drivers, played games etc. - maybe I need more patches later on due to performance etc.). The question remains the same though - where do these patches come from etc.?

They come from here:

http://git.kernel.org/cgit/linux/kernel … f81dea8a7d
http://git.kernel.org/cgit/linux/kernel … ba24389bee

And I agree, they should really be the only patches required and hopefully none are required if you don't have Intel graphics in your system.  Some people are experiencing problems with memory being freed and apply another patch, I suspect that has something to do with the toolchain they're using to build.  The other popular patch is the ACS override, which allows devices to be separated out of IOMMU groups.  There's code going into 3.15 to help many Intel chipsets with this problem and I just posted a patch this week that adds IDs for X79 that will hopefully also make 3.15.

Why are the "i915 patches" needed and why aren't they upstream?  VGA is a very old standard and was an evolution of things like EGA and CGA.  It was developed back when you were lucky to have one graphics card and ROMs were a new thing.  Like many legacy things, it stuck because it was widely supported and there was nothing better.  One of the problems with VGA is that it lives at a fixed address and only one device can occupy that address space at a time.  PCI attempts to solve this problem by having a VGA enable bit in bridges allowing the chipset to select which single device receives VGA accesses.  There are still complications when two VGA devices live on the same bus, but I'll skip as it's not important here.  To manage where VGA is routed we have something in the kernel called the VGA arbiter.  The arbiter requires that any access to VGA space be wrapped in locking that allows the arbiter to switch chipset routing.  When you specify x-vga=on for a device then the vfio-pci driver will claim accesses to the VGA space and forward them to the vfio-pci kernel module, which does this locking.  If everyone plays nice with the VGA arbiter, things work.

The trouble is that not everyone place nice, some drivers don't play at all.  The vgacon driver in the kernel doesn't participate in VGA arbitration.  The i915 driver lies and claims that it doesn't claim VGA accesses, effectively an opt-out of arbitration, while actually it does claim VGA accesses.  This means that our VM comes along and does a VGA access, it gets forwarded to the kernel and vfio takes the VGA arbiter lock, the arbiter sees that i915 doesn't claim VGA and the VGA enable bit is set on the bridge for our device and proceeds to give us the lock.  Our access then goes straight to i915.  This is the reason you see both the corruption on the host display and lack of anything happening on the guest display, the accesses are going to the wrong card.

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI.  The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

On the subject of whether you need to blacklist or otherwise avoid allowing the graphics device to attach to host drivers before you unbind it for the guest, it's a matter of personal preference and host driver capabilities.  In the non-graphics case of PCI device assignment, it's not needed because Linux has pretty good support for hotplug and thus the bugs have been worked out for detaching and re-attaching devices from drivers.  The hotplug story for graphics cards on Linux has not been as good, so those drivers tend to lag in reliably and cleanly releasing the device.  The nvidia proprietary driver for instance will allow you to unbind the driver with X up and running on the device and X continues to run after unbind.  I filed a bug with Nvidia about this, but they already closed it with no code change.  If you have no intention of using the device in the host, it's often the safest and easiest path just to blacklist the driver or force it to bind to pci-stub.  If unbinding works for you, do it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1480 2014-04-03 18:18:15

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kernel 3.14 is up on the front page

Offline

#1481 2014-04-03 19:57:02

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just tried the linux-mainline 3.14 Kernel on the OP as posted by nbhs, still hasn't resolved my issue however I now see some interesting messages in dmesg. I still get video output and can install windows etc however drivers won't install and even while booting windows I get these messages. Tested with multiple qemu versions as well:

VM starts fine, video displays, I see these messages as always:

[  351.920675] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  351.920684] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

Then all of a sudden dmesg gets flooded with this:

[  377.186286] ------------[ cut here ]------------
[  377.186288] WARNING: CPU: 3 PID: 618 at arch/x86/kvm/x86.c:1114 kvm_get_msr_common+0x793/0x960 [kvm]()
[  377.186288] Modules linked in: tun vfio_pci vfio_iommu_type1 vfio bridge stp llc snd_hda_codec_realtek snd_hda_codec_generic dm_mod coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel joydev ghash_clmulni_intel mousedev iTCO_wdt aesni_intel btusb iTCO_vendor_support bluetooth aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd 6lowpan_iphc xpad ff_memless rfkill microcode snd_hda_intel led_class hid_generic snd_hda_codec igb snd_hwdep e1000e snd_pcm hwmon psmouse dca serio_raw snd_timer mei_me pcspkr lpc_ich i2c_i801 ptp snd pps_core mei soundcore fan thermal battery evdev shpchp mac_hid processor nfs lockd sunrpc fscache usbhid hid sd_mod crc_t10dif crct10dif_common ata_generic pata_acpi atkbd libps2 ata_piix libata ehci_pci xhci_hcd ehci_hcd
[  377.186305]  scsi_mod usbcore usb_common i8042 serio pci_stub vfat fat ext4 crc16 mbcache jbd2 i915 video button intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core
[  377.186309] CPU: 3 PID: 618 Comm: qemu-system-x86 Tainted: G        W    3.14.0-2-mainline #1
[  377.186310] Hardware name:    / , BIOS 4.6.5 07/26/2013
[  377.186310]  0000000000000009 ffff880415a81c10 ffffffff814d563c 0000000000000000
[  377.186312]  ffff880415a81c48 ffffffff81064f1d 0000000040000020 ffff880415a81cd8
[  377.186313]  ffff880413bd3ec0 ffff8803378a4000 ffff8803378a4040 ffff880415a81c58
[  377.186314] Call Trace:
[  377.186315]  [<ffffffff814d563c>] dump_stack+0x4d/0x6f
[  377.186316]  [<ffffffff81064f1d>] warn_slowpath_common+0x7d/0xa0
[  377.186317]  [<ffffffff81064ffa>] warn_slowpath_null+0x1a/0x20
[  377.186320]  [<ffffffffa0efa453>] kvm_get_msr_common+0x793/0x960 [kvm]
[  377.186322]  [<ffffffffa13170c2>] vmx_get_msr+0x102/0x370 [kvm_intel]
[  377.186323]  [<ffffffffa131735a>] handle_rdmsr+0x2a/0x140 [kvm_intel]
[  377.186325]  [<ffffffffa131c9a5>] vmx_handle_exit+0xb5/0xa20 [kvm_intel]
[  377.186326]  [<ffffffff81074c02>] ? __set_task_blocked+0x32/0x70
[  377.186328]  [<ffffffffa1314360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[  377.186330]  [<ffffffffa0effb60>] kvm_arch_vcpu_ioctl_run+0xc70/0x11c0 [kvm]
[  377.186333]  [<ffffffffa0efb85e>] ? kvm_arch_vcpu_load+0x4e/0x1e0 [kvm]
[  377.186336]  [<ffffffffa0ee85c2>] kvm_vcpu_ioctl+0x2b2/0x5b0 [kvm]
[  377.186336] systemd-journald[168]: /dev/kmsg buffer overrun, some messages lost.
[  377.186338]  [<ffffffff811e4ab8>] ? fsnotify+0x228/0x2f0
[  377.186340]  [<ffffffff810a9bb8>] ? __wake_up_locked_key+0x18/0x20
[  377.186341]  [<ffffffff811b6180>] do_vfs_ioctl+0x2e0/0x4c0
[  377.186342]  [<ffffffff811bff9e>] ? __fget+0x6e/0xb0
[  377.186344]  [<ffffffff811b63e1>] SyS_ioctl+0x81/0xa0
[  377.186345]  [<ffffffff814e3069>] system_call_fastpath+0x16/0x1b
[  377.186346] ---[ end trace 73327e2f3359a7b9 ]---
[  377.186413] systemd-journald[168]: /dev/kmsg buffer overrun, some messages lost.





gneville wrote:

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad



gneville wrote:

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

#1482 2014-04-03 21:37:24

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, aw, for that very insightful reply.

I have a couple of follow-up questions and/or need a few clarifications.

aw wrote:

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI. The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

Regarding what is marked in red:

So applying the the first patch (which makes i915 work correctly with VGA arbitration) kills overall performance (because it makes Xorg  disable DRI which forces everyone to make indirect rendering) and because of that that patch was removed / reverted upstream?

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

Regarding the text marked in blue:
Solutions to which problem? Solutions to the i915 arbiter problem - i.e. an alternative to the current patch?

Offline

#1483 2014-04-03 21:49:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jonascj wrote:

Thank you, aw, for that very insightful reply.

I have a couple of follow-up questions and/or need a few clarifications.

aw wrote:

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI. The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

Regarding what is marked in red:

So applying the the first patch (which makes i915 work correctly with VGA arbitration) kills overall performance (because it makes Xorg  disable DRI which forces everyone to make indirect rendering) and because of that that patch was removed / reverted upstream?

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Regarding the text marked in blue:
Solutions to which problem? Solutions to the i915 arbiter problem - i.e. an alternative to the current patch?

Solutions to either fix or avoid the problem that we can't have multiple VGA devices participate in arbitration without losing host DRI, which means fixing the VGA arbiter and/or fixing Xorg's use of the VGA arbiter, or avoiding VGA space in the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1484 2014-04-03 22:11:01

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
jonascj wrote:

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Sorry for being so thick, but this means that everyone using the i915 module/driver suffer from disabled DRI while they use kernels patched with this patch? I.e. everyone using Intel IGPU for their host while having a dedicated gfx passed through to a KVM guest have DRI disabled on their host.

Offline

#1485 2014-04-03 22:12:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jonascj wrote:
aw wrote:
jonascj wrote:

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Sorry for being so thick, but this means that everyone using the i915 module/driver suffer from disabled DRI while they use kernels patched with this patch? I.e. everyone using Intel IGPU for their host while having a dedicated gfx passed through to a KVM guest have DRI disabled on their host.

Yes


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1486 2014-04-04 07:49:23

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The trouble is that not everyone place nice, some drivers don't play at all.  The vgacon driver in the kernel doesn't participate in VGA arbitration.  The i915 driver lies and claims that it doesn't claim VGA accesses, effectively an opt-out of arbitration, while actually it does claim VGA accesses.  This means that our VM comes along and does a VGA access, it gets forwarded to the
Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here.

Does wayland or mir solve this problem?

Offline

#1487 2014-04-04 08:06:11

MattSpe
Member
Registered: 2014-04-04
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Everyone
I've spent the last few days fighting with my VGA passthrough.
No matter what I try, I always get the same result : passthrough is successful only on the first launch after host reset. Any following guest launch fail (I see the screen flicking but it stays black).
My last attempt was with content from first post (qemu-git seabios-git linux-mainline 3.14) on Arch. Same result.

My config is : X5680 + Sapphire X58 Pure Black + GT610 + GTX560Ti.
I tried both combinations 610 primary/560 secondary and 560 primary/610 secondary, no difference.
If I try to pass the secondary, only first attempt is successful. If I try to pass the primary not even the first attempt is working (stays black after flickering).
I tried to pass a romfile (I get an error msg on second launch if I don't pass, no need on the first launch), no difference.

Could someone point me in the right direction ?
Do I miss some patch/config ? Is it a hardware issue ?

Thanks !
Matthieu

Offline

#1488 2014-04-04 09:08:44

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed 3.14 from the first page and it didn't work well. The log is full of :

------------[ cut here ]------------ WARNING: CPU: 0 PID: 1367 at arch/x86/kvm/x86.c:1114 kvm_get_msr_common+0x793/0x960 [kvm]() Modules linked in: nls_utf8 cifs dns_resolver fscache ebt_ip ebtable_filter bridge stp llc ebtables x_tables nct6775 hwmon_vid coretemp iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl raid1 md_mod sd_mod crc_t10dif atkbd libps2 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helpe ehci_pci ehci_hcd r8169 mii usbcore usb_common i8042 serio pci_stub CPU: 0 PID: 1367 Comm: windows_81 Tainted: G W 3.14.0-2-mainline #1 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Professional, BIOS P1.70B 11/12/2013 0000000000000009 ffff8802571afc10 ffffffff814d563c 0000000000000000 ffff8802571afc48 ffffffff81064f1d 0000000040000020 ffff8802571afcd8 ffff88025726bec0 ffff88025717c000 ffff88025717c040 ffff8802571afc58 Call Trace: [<ffffffff814d563c>] dump_stack+0x4d/0x6f [<ffffffff81064f1d>] warn_slowpath_common+0x7d/0xa0 [<ffffffff81064ffa>] warn_slowpath_null+0x1a/0x20 [<ffffffffa03da453>] kvm_get_msr_common+0x793/0x960 [kvm] [<ffffffffa00740c2>] vmx_get_msr+0x102/0x370 [kvm_intel] [<ffffffffa007435a>] handle_rdmsr+0x2a/0x140 [kvm_intel] [<ffffffffa00799a5>] vmx_handle_exit+0xb5/0xa20 [kvm_intel] [<ffffffffa0071360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel] [<ffffffffa03dfb60>] kvm_arch_vcpu_ioctl_run+0xc70/0x11c0 [kvm] [<ffffffff8101e9d6>] ? ___preempt_schedule+0x56/0xb0 [<ffffffffa03c85c2>] kvm_vcpu_ioctl+0x2b2/0x5b0 [kvm] [<ffffffff814d7909>] ? __schedule+0x3c9/0x900 [<ffffffff811b6180>] do_vfs_ioctl+0x2e0/0x4c0 [<ffffffff811bff9e>] ? __fget+0x6e/0xb0 [<ffffffff811b63e1>] SyS_ioctl+0x81/0xa0 [<ffffffff814e3069>] system_call_fastpath+0x16/0x1b ---[ end trace 1bf431db809123d0 ]---

Before, I had 3.13.6 also from the first page and worked without any issues.

My config is :

qemu-system-x86_64 \ -daemonize \ -pidfile "/run/qemu-$VM.pid" \ -nodefconfig -nodefaults \ -name "$VM",process="$VM" \ -enable-kvm \ -boot menu=on \ -rtc base=localtime \ -machine q35 \ -balloon none \ -m 8192 -mem-path /dev/hugepages -mem-prealloc \ -cpu Haswell,hv-time \ -smp sockets=1,cores=4,threads=1 \ -bios bios-256k.bin \ -chardev file,path="$VM-seabios.log",id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \ -vga none \ -nographic \ -monitor unix:/run/qemu-$VM.monitor,server,nowait \ -serial none \ -parallel none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$KVM_ROOT/roms/Sapphire_RadeonHD_7790.rom \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -device vfio-pci,host=06:00.0,bus=pcie.0 \ -device vfio-pci,host=09:00.0,bus=pcie.0 \ -device vfio-pci,host=0b:00.0,bus=pcie.0

Offline

#1489 2014-04-05 14:50:21

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

I installed 3.14 from the first page and it didn't work well. The log is full of :

------------[ cut here ]------------ ... ---[ end trace 1bf431db809123d0 ]---

Before, I had 3.13.6 also from the first page and worked without any issues.

My config is :

qemu-system-x86_64 \ -daemonize \ ... -device vfio-pci,host=0b:00.0,bus=pcie.0

I think hv-time might be doing bad things. It's working great without tho.

Last edited by dwe11er (2014-04-05 14:51:04)

Offline

#1490 2014-04-05 16:32:44

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I will try, thanks.
I upgraded to 3.14 hoping for some speed improvement from hv-time.

My processor is Sandy Bridge and in the config I have Haswell. Does this creates any problems ?
I have Haswell because with Sandy Bridge it wasn't stable; crashes from time to time.

EDIT: Yes, without hv-time is working with 3.14. The problems is that now, the cpu load on the host when
running Windows 8.1 idle is with ~20-25% higher than when running 3.13.6 with hv-time. With 3.13.6 was
~45-50%.
I don't know if it is because of hv-time or because of the new kernel.

Last edited by anickname (2014-04-05 18:50:10)

Offline

#1491 2014-04-05 18:45:24

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using the provided packages with Kernel 3.14 I got rid of the host crashes during VM startup. Graphics pass through is working, as far as I can tell (Windows performance indicator).
Now I`m trying to pass through my creative recon3d sound card. Originally I tried XEN where this was working without problems, but now I`m getting the following error during startup:

vfio: error, group 6 is not viable

The corresponding part of my script is:

-device vfio-pci,host=03:00.0 \
lspci -k | grep 03 03:00.0 Audio device: Creative Labs Device 0012 (rev 01) lspci -nnn | grep 03:00.0 03:00.0 Audio device [0403]: Creative Labs Device [1102:0012] (rev 01) cat /proc/cmdline ... pci-stub.ids=10de:1187,10de:0e0a,1102:0012,8086:1c2d

Does someone have an idea, whats wrong here?

BR

Offline

#1492 2014-04-05 21:28:47

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://pastebin.com/KWD4DVcS

So, if anyone interested, I did my own launching script with ability to unbind devices and rebind back to their previous driver (useful if passing hd audio) after vm is down.

Offline

#1493 2014-04-05 23:26:11

eliwallace
Member
Registered: 2014-04-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I've been trying to get this working with a GTX 770 on the host passing a GTX 660 through to the guest, but I ran into

qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: vfio: Device does not support requested feature x-vga qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: vfio: failed to get device 0000:08:00.0 qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: Device 'vfio-pci' could not be initialized

Using kernel 3.13, qemu 1.7, FX-8350 cpu, Crosshair V F-Z mobo, proprietary Nvidia drivers on host

launching qemu with

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on -device vfio-pci,host=08:00.1,bus=root.1,addr=00.1

lspci -vvvv:

08:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 2662 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 32 Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at a000 [size=128] Expansion ROM at f5000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900 v1] #19 Kernel driver in use: vfio-pci 08:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1) Subsystem: eVga.com. Corp. Device 2662 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 33 Region 0: Memory at f5080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Kernel driver in use: vfio-pci

dmesg | egrep -i "amd-vi|pci-stub|vfio|iommu"

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.13.7 root=UUID=5e9c875a-9b8a-418a-b461-b1e08d8a1791 ro quiet iommu=pt iommu=1 amd_iommu=on [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.13.7 root=UUID=5e9c875a-9b8a-418a-b461-b1e08d8a1791 ro quiet iommu=pt iommu=1 amd_iommu=on [ 0.273684] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [ 0.273757] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [ 0.273806] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found [ 0.273853] AMD-Vi: Disabling interrupt remapping [ 1.220732] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 1.220863] AMD-Vi: Initialized for Passthrough Mode [ 1.306447] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 1.306448] AMD IOMMUv2 functionality not available on this system [ 3.703453] pci-stub: add 10DE:11C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 3.703463] pci-stub 0000:08:00.0: claimed by stub [ 3.703468] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 3.703474] pci-stub 0000:08:00.1: claimed by stub [ 9.215347] pci-stub 0000:08:00.0: enabling device (0000 -> 0003) [ 38.196490] vboxpci: IOMMU found [ 344.289349] VFIO - User Level meta-driver version: 0.3 [ 464.944491] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 664.101872] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 743.049314] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 5881.675217] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 5906.191754] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 5945.493057] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 8566.737047] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900 [ 9020.289413] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900

cat /boot/config-`uname -r` | grep -i "vfio"

CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y

EDIT: Solved by bios update

Last edited by eliwallace (2014-04-06 18:23:45)

Offline

#1494 2014-04-05 23:39:23

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey want to say thanks because I successfully set VM on my hardware:
i7 4771, ASRock Z87 Extreme6, HD 6950.
Though I use Kubuntu 14.04 with:
Vanilla seabios and QEMU (version 1.7.91 (Debian 2.0.0~rc1+dfsg-0ubuntu1)) and seabios (1.7.4-4).
And kernel 3.14 with OP patches. I not yet sure what options I actually need and which are useless.

Just wonder about that problem with i915. Is this why I only have LLVMPipe on host when boot with patched kernel?

Last edited by SXX (2014-04-05 23:44:16)

Online

#1495 2014-04-06 03:31:09

Nex7
Member
Registered: 2014-04-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've pored through this thread and Google, and can't find anyone even claiming to be seeing the symptoms I'm seeing. Maybe I've missed something, or someone has an idea?

Motherboard: SuperMicro X10SAE-O
CPU: E3-1275v3
1st VGA: i915
2nd VGA (in top PCI-e x18 slot): GeForce GTX 770 "Windforce"

After much trial and error, using the latest (3.14) kernel with CONFIG_VFIO stuff compiled in & qemu-git from AUR (OP's fail to compile), plus OP's vfio-bind stuff and pci-stub set in kernel cmdline for the GTX770, I have almost full success. The Windows guest boots fine, reboots fine, can even be shut down and restarted without rebooting host.

However, I can't install the damn nVidia drivers in the Windows 7 x64 guest. They install fine, but upon reboot, Windows boots past the initial logo and then into a blank screen, I see my USB devices flicker a few times during a rather long pause, and then the VM reboots and sticks you at the 'Windows failed to boot' menu asking what to do.

I've tried a rather large number of things to resolve the issue based on every Google search I could craft, too many to list here, and I'm stumped. Anybody seen this/have an idea?

kernel cmdline:

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=f4e70249-85fe-430b-9039-fcd9d7f89327 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1184,10de:0e0a quiet

run line:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu qemu64 -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin \ -rtc base=localtime -nodefaults \ -vga none -nographic -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -net nic,model=virtio,macaddr=52:54:01:01:01:01 -net bridge,br=br0 \ -device virtio-scsi-pci,id=scsi \ -usb -usbdevice host:1532:0016 -usbdevice host:24f0:0137 \ -usbdevice host:1b1c:0a0a -usbdevice host:04a9:1909 \ -drive file=/mnt/homesan/vm-win7-game/os_disk.raw,id=disk,format=raw,cache=none -device scsi-hd,drive=disk \ -drive file=/mnt/homesan/vm-win7-game/steam_disk.raw,id=game-disk,format=raw,cache=none -device scsi-hd,drive=game-disk \ -drive file=/usr/share/qemu/win7.iso,id=installcd -device ide-cd,bus=ide.1,drive=installcd \ -drive file=/usr/share/qemu/virtio-win-0.1-74.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd

There's no output from qemu when I run it, and nothing in dmesg beyond some crap about USB that seems unimportant (as it's always there, nvidia driver or not), I'm at the point of buying a Radeon, but I see people posting just days ago that they had a working GTX 770, so I have slight hope left.  Help?

Last edited by Nex7 (2014-04-06 03:31:39)

Offline

#1496 2014-04-06 16:03:18

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Nex7, I am issuing the same problems, everything works until I install the nvidia driver on Win7, I posted the problem in awilliam's github but I guess that he didn't see or something, anyways this is the link https://github.com/awilliam/qemu-vfio/issues/2

My 770 is a Gigabyte 770 WindForce (4GB) so we are in similar situation here...
CPU: i7 4770 (without any letter)
MB: Asrock Extreme 6 Z87
GPU for host: Nvidia 430 at PCIE-3 x16 slot 1
GPU for qemu: Nvidia 770 at PCIE-3 x8 slot 2

I want to add something extra, the 430 succefully works (when I swich their roles and PCIE slots) and nvidia driver installs and works correctly (nvidia panel recognizes the card and the attached monitor) so it's something within Gigabyte 770 VBIOS maybe? maybe because card has UEFI VBIOS? I'm looking for help too...

Last edited by Cubex (2014-04-06 16:04:35)

Offline

#1497 2014-04-06 16:57:06

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

eliwallace wrote:

dmesg | egrep -i "amd-vi|pci-stub|vfio|iommu"

[ 0.273684] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [ 0.273757] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [ 0.273806] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found

Updating your BIOS should fix this issue I expect.  May not help with anything else, but FYI.

Offline

#1498 2014-04-06 17:41:47

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got the same problem but with my AMD R9 290 card, everythings fine until the drivers install. So I don't think this is exclusive to Nvidia, could be because of UEFI VBIOS......


Cubex wrote:

Hello Nex7, I am issuing the same problems, everything works until I install the nvidia driver on Win7, I posted the problem in awilliam's github but I guess that he didn't see or something, anyways this is the link https://github.com/awilliam/qemu-vfio/issues/2

My 770 is a Gigabyte 770 WindForce (4GB) so we are in similar situation here...
CPU: i7 4770 (without any letter)
MB: Asrock Extreme 6 Z87
GPU for host: Nvidia 430 at PCIE-3 x16 slot 1
GPU for qemu: Nvidia 770 at PCIE-3 x8 slot 2

I want to add something extra, the 430 succefully works (when I swich their roles and PCIE slots) and nvidia driver installs and works correctly (nvidia panel recognizes the card and the attached monitor) so it's something within Gigabyte 770 VBIOS maybe? maybe because card has UEFI VBIOS? I'm looking for help too...

Offline

#1499 2014-04-06 18:21:30

eliwallace
Member
Registered: 2014-04-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
eliwallace wrote:

dmesg | egrep -i "amd-vi|pci-stub|vfio|iommu"

[ 0.273684] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [ 0.273757] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [ 0.273806] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found

Updating your BIOS should fix this issue I expect.  May not help with anything else, but FYI.

Thanks, that fixed all the immediate problems

Offline

#1500 2014-04-06 21:34:24

Nex7
Member
Registered: 2014-04-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So if I go into Safe Mode after one of these failed boots, the popup says:

Problem signature:
  Problem Event Name:    BlueScreen
  OS Version:    6.1.7601.2.1.0.256.4
  Locale ID:    1033

Additional information about the problem:
  BCCode:    116
  BCP1:    FFFFFA8009E404E0
  BCP2:    FFFFF8800F18EE20
  BCP3:    0000000000000000
  BCP4:    0000000000000002
  OS Version:    6_1_7601
  Service Pack:    1_0
  Product:    256_1

Google for this is not helpful - lots of 'update your drivers' and 'the card might be broken'. sad I did find one thing of some interest, this quote:

"It's not a true crash, in the sense that the bluescreen was initiated only because the combination of video driver and video hardware was being unresponsive, and not because of any synchronous processing exception".

Since Vista, the "Timeout Detection and Recovery" (TDR) components of the OS video subsystem have been capable of doing some truly impressive things to try to recover from issues which would have caused earlier OSs like XP to crash.

As a last resort, the TDR subsystem sends the video driver a "please restart yourself now!" command and waits a few seconds.

If there's no response, the OS concludes that the video driver/hardware combo has truly collapsed in a heap, and it fires off that stop 0x116 BSOD.

If playing with video driver versions hasn't helped, make sure the box is not overheating.

Try removing a side panel and aiming a big mains fan straight at the motherboard and GPU.

Obviously I do not believe my card is overheating, but perhaps that explanation of how you get to a BCC code 116 might spark something in one of the local expert's minds about what's going on?

I'm going to try playing with BIOS settings some more..

Offline

#1501 2014-04-07 01:23:31

Nex7
Member
Registered: 2014-04-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Hello Nex7, I am issuing the same problems, everything works until I install the nvidia driver on Win7

Hey Cubex, I haven't yet gotten this to work, BUT, I have good news for you, IF you have qemu-1.7.4 or less (you probably have 1.7.0 like I did, from 'pacman -S qemu'). You need 1.7.5+. I can confirm that with the latest 'qemu-git' (and 'seabios-git'), my VM was able to boot into Windows with graphics, but a bunch of other crap was screwed up. I am currently attributing this to TOO NEW a qemu, since the qemu-git package seems to be grabbing the absolute bleeding edge. After dinner I will try to get the package to pull down something post 1.7.5 but not bleeding edge.

Offline

#1502 2014-04-07 09:08:41

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs: Could you perhaps tell us when you update qemu-git and seabios-git aswell? Or are these the same packages we get from AUR?

Offline

#1503 2014-04-08 03:56:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

nbhs: Could you perhaps tell us when you update qemu-git and seabios-git aswell? Or are these the same packages we get from AUR?

These are git versions, so unless something breaks or fails to build theres no reason to update the package, you can use the AUR versions if you want since there are no extra patches needed

Offline

#1504 2014-04-08 18:40:26

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I have a small probleme, on my VM i want to start OCCT and i got a BSOD "driver_irql_not_less_or_equal" i have not any other information on the BSOD screen.

I run on qemu 2.0.0-RC1 and kernel 3.14, i have two graphique card AMD HD 6870 (crossfire not enable) on the VM (just for testing).

On dmesg i have :

[ 8704.000634] kvm_get_msr_common: 82 callbacks suppressed [ 8704.000643] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.000686] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.004952] kvm [6821]: vcpu0 unhandled rdmsr: 0x637 [ 8704.004976] kvm [6821]: vcpu0 unhandled rdmsr: 0xe8 [ 8704.004989] kvm [6821]: vcpu0 unhandled rdmsr: 0xe7 [ 8704.005013] kvm [6821]: vcpu0 unhandled rdmsr: 0x31 [ 8704.005030] kvm [6821]: vcpu0 unhandled rdmsr: 0x35 [ 8704.005050] kvm [6821]: vcpu0 unhandled rdmsr: 0x39 [ 8704.005095] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.005108] kvm [6821]: vcpu0 unhandled rdmsr: 0x194

Any help ?
Thanks

Last edited by Val532 (2014-04-08 18:40:56)

Offline

#1505 2014-04-08 19:03:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Hi all,

I have a small probleme, on my VM i want to start OCCT and i got a BSOD "driver_irql_not_less_or_equal" i have not any other information on the BSOD screen.

I run on qemu 2.0.0-RC1 and kernel 3.14, i have two graphique card AMD HD 6870 (crossfire not enable) on the VM (just for testing).

On dmesg i have :

[ 8704.000634] kvm_get_msr_common: 82 callbacks suppressed [ 8704.000643] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.000686] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.004952] kvm [6821]: vcpu0 unhandled rdmsr: 0x637 [ 8704.004976] kvm [6821]: vcpu0 unhandled rdmsr: 0xe8 [ 8704.004989] kvm [6821]: vcpu0 unhandled rdmsr: 0xe7 [ 8704.005013] kvm [6821]: vcpu0 unhandled rdmsr: 0x31 [ 8704.005030] kvm [6821]: vcpu0 unhandled rdmsr: 0x35 [ 8704.005050] kvm [6821]: vcpu0 unhandled rdmsr: 0x39 [ 8704.005095] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.005108] kvm [6821]: vcpu0 unhandled rdmsr: 0x194

Any help ?
Thanks

try

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Offline

#1506 2014-04-08 20:45:39

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hours of tests, 3.14 posted on OP for me was SLOW to a crawl, 3.13 old kernel posted before gets me almost native speed.

Offline

#1507 2014-04-09 10:30:49

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

beckend wrote:

Hours of tests, 3.14 posted on OP for me was SLOW to a crawl, 3.13 old kernel posted before gets me almost native speed.

i am experiencing unexpected slowdown too. could you please specify your hardware, exact kernel version (if its 3.13.something) and patches?


For others if anyone cares i managed to get my R9 270X gpu running as secondary on libvirt i440fx (default) chipset. here is xml:

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit win or other application using the libvirt API. --> <domain type='kvm'> <name>win</name> <uuid>9d99f9e4-c66f-427e-89a2-0285358ce620</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>8</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/local/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/mapper/lvm-vm'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <interface type='network'> <mac address='50:64:11:7c:b6:85'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/> <video> <model type='vga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x13' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x13' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

Offline

#1508 2014-04-09 11:45:21

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3.14 In the OP when he posted it the first time, and I dont see that changed.
If u want to know without decompressing the posted PKGBUILD file, here it is:
        'change-default-console-loglevel.patch'
        '0001-Bluetooth-allocate-static-minor-for-vhci.patch'
        '0002-module-allow-multiple-calls-to-MODULE_DEVICE_TABLE-p.patch'
        '0003-module-remove-MODULE_GENERIC_TABLE.patch'
        '0004-Revert-syscalls.h-use-gcc-alias-instead-of-assembler.patch'
        'http://repo-ck.com/source/gcc_patch/enable_additional_cpu_optimizations_for_gcc.patch$
        'radeon_load_vbios_from_file.patch'
        'acs.patch'
        'i915_314.patch'
        'debug_registers.patch'

Kernel version is pkgver=3.14 pkgrel=2 all from PKGBUILD. I dont use the 'override_for_missing_acs_capabilities.patch' since it did not work for me, however I edited the the the PKGBUILD to include a newer patch to fix the IOMMU group isolation from:  http://linux-kernel.2935.n7.nabble.com/ … 96884.html
pasted the patch into 'acs.patch'. The same thing was done for 3.13.6 (posted earlier from OP).

Hardware is a Lenovo W530, Ivy Bridge 3740QM, 8GB Ram. What was passthoughed is an external GPU, 680GTX to the virtual machine as mostly instructed from OP except I replaced all to virtio drivers. Performance of 3.13 is quite astounding, I still can't belive it.

Offline

#1509 2014-04-09 16:25:53

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm unable to get qemu to boot the guest os. I've followed the instructions and used the packages in the main post, and have looked around the thread for a solution but I'm still clueless as to what could be wrong. Below is everything I could think of that will be helpful to troubleshoot, sorry that it's a bit much. Let me know if any more information would be helpful.

Command Run:

sudo qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 1024 \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

Error:

qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Setup:

Asrock z87 Extreme9
Intel Integrated Graphics (not in use, aside for post and bios)
Radeon R9 270x (guest) [03:00.0 and 03:00.1, pci-stubbed]
Radeon HD 6670 (host)  [04:00.0 and 04:00.1]



Drivers Tried:

radeon
fglrx (current)

Please note that both drivers have the same issue.



Boot Options:

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:6810,1002:aab0

IOMMU Group 1:

ls -l /sys/kernel/iommu_groups/1/devices total 0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:02:08.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:02:10.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:02:11.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:11.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:04:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1

Details For each device:


00.01.0:

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=05, sec-latency=0 I/O behind bridge: 0000a000-0000bfff Memory behind bridge: dc900000-dcbfffff Prefetchable memory behind bridge: 00000000a0000000-00000000bfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: ASRock Incorporation Device 0c01 Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00298 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Capabilities: [d94 v1] #19 Kernel driver in use: pcieport Kernel modules: shpchp

   
01:00.0:

01:00.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Region 0: Memory at dcb00000 (32-bit, non-prefetchable) [size=256K] Bus: primary=01, secondary=02, subordinate=05, sec-latency=0 I/O behind bridge: 0000a000-0000bfff Memory behind bridge: dc900000-dcafffff Prefetchable memory behind bridge: 00000000a0000000-00000000bfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fee002f8 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747 Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending- InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] #12 Capabilities: [b00 v1] Latency Tolerance Reporting Max snoop latency: 71680ns Max no snoop latency: 71680ns Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp

02:08.0

02:08.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000b000-0000bfff Memory behind bridge: dca00000-dcafffff Prefetchable memory behind bridge: 00000000b0000000-00000000bfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fee00318 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #72, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747 Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending- InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp

02:10.0

02:10.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: dc900000-dc9fffff Prefetchable memory behind bridge: 00000000a0000000-00000000afffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fee00338 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #16, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #80, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747 Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending- InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp

02:11.0

02:11.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fee00358 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #17, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #81, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747 Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending+ InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp

03:00.0

02:11.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fee00358 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #17, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #81, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747 Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=8 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending+ InProgress- Port Arbitration Table <?> Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp

03.00.1

03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: Gigabyte Technology Co., Ltd Device aab0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 10 Region 0: Memory at dca60000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

04.00.0

04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks XT [Radeon HD 6670/7670] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 0443 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 66 Region 0: Memory at a0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at dc920000 (64-bit, non-prefetchable) [size=128K] Region 4: I/O ports at a000 [size=256] Expansion ROM at dc900000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00678 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: fglrx_pci Kernel modules: radeon, fglrx

04.00.1

04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Turks/Whistler HDMI Audio [Radeon HD 6000 Series] Subsystem: ASUSTeK Computer Inc. Device aa90 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 65 Region 0: Memory at dc940000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00658 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel

I'm on week two of trying to get this set up, so any help at all is greatly appreciated!

Offline

#1510 2014-04-09 17:45:39

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

Offline

#1511 2014-04-09 17:49:58

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

Offline

#1512 2014-04-09 18:02:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
novist wrote:

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1513 2014-04-09 18:02:07

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just bought a PCIe USB 3.0 card for my machine. I was going to use it solely for passthroughing but Im having difficulty getting it to work.

Error Im receiving:

(qemu) qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: vfio: failed to get group 9 qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: Device 'vfio-pci' could not be initialized Done

lsgroup:

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) 03:00.0 Network controller: Broadcom Corporation BCM4352 802.11ac Wireless Network Adapter (rev 03) 04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

Im running with acs override patch included with nbhs kernel(3.13.6-1-mainline) at the moment.
I can see there are just too many devices in group 9, many of them I do not want to pass through. (It seems Intel didnt follow some standards here)

I see beckend found a new acs override patch written by awilliams, but this would be the first time I wouldve manually copy a patch for myself.

I get the following while trying to compile:

patching file drivers/pci/quirks.c patch: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_6.html topic_7.html topic_8.html topic_9.html malformed patch at line 67: u16 device;

Any help is appreciated, still got a lot to learn smile

Update: Managed to implement the patch myself. The pci devices are now divided much more than before and I got my USB PCie card passthrough working!

Last edited by trimm (2014-04-10 08:41:28)

Offline

#1514 2014-04-09 18:02:59

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Still check if they are bound to vfio driver. For example my passed-through usb port 0000:00:13.0:

ls -la /sys/devices/pci0000:00/0000:00:13.0/driver/module/drivers

prints

pci:vfio-pci

Hope you can get it working. We have same GPU and im dying to find out if terrible performance im having is fault of GPU or what (hope its not discouraging!).

Offline

#1515 2014-04-09 18:05:20

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Eb wrote:
novist wrote:

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).

Could of sworn I already applied the patch, I'll look into it when i'm back at my system. Thanks.

Offline

#1516 2014-04-09 20:07:57

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah 770 works great under new qemu and using ubuntu kernel +  pci acs patch, what problem where you facing Nex7?

I got net working and now im looking at audio, seems that is somewhat laggy and cuts often... tried both nbhs way of adding soundcard and using -soundhw parametere but result is same, all tried with alsa and pulseaudio as driver and still the same hmm

EDIT: Also I changed a bit the vfio-bind script to ignore already binded pci devices, so I can add vfio-bind on the qemu script

#!/bin/bash if [ ! $USER = "root" ]; then echo "Run as root!" exit fi modprobe vfio-pci base="/tmp/vfio-bind" mkdir $base -p for dev in "$@"; do path="$base/$dev" if [ ! -f $path ]; then touch $path vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id fi done

Last edited by Cubex (2014-04-09 22:06:50)

Offline

#1517 2014-04-09 20:41:35

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
aw wrote:
Eb wrote:

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).

Could of sworn I already applied the patch, I'll look into it when i'm back at my system. Thanks.

I've re-installed the patched linux-mainline package listed on the main page, but I'm still having the same issue of all my graphics devices being in the same IOMMU group. I'm running a Z87 chipset, would that cause any issues? Is there a specific patch I should apply for this chipset? Thanks a ton for the help.

Edit:

I've also read that this can potentially disable dri, which is something I need. Am I looking at a lost cause here?

Last edited by Eb (2014-04-09 20:42:42)

Offline

#1518 2014-04-09 21:06:09

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try the V2 of the IOMMU patch: http://linux-kernel.2935.n7.nabble.com/ … 96884.html
Do not apply the old acs patch.
Edit: only works on Intel.

Last edited by beckend (2014-04-09 21:08:06)

Offline

#1519 2014-04-09 21:37:12

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Val532 wrote:

Hi all,

I have a small probleme, on my VM i want to start OCCT and i got a BSOD "driver_irql_not_less_or_equal" i have not any other information on the BSOD screen.

I run on qemu 2.0.0-RC1 and kernel 3.14, i have two graphique card AMD HD 6870 (crossfire not enable) on the VM (just for testing).

On dmesg i have :

[ 8704.000634] kvm_get_msr_common: 82 callbacks suppressed [ 8704.000643] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.000686] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.004952] kvm [6821]: vcpu0 unhandled rdmsr: 0x637 [ 8704.004976] kvm [6821]: vcpu0 unhandled rdmsr: 0xe8 [ 8704.004989] kvm [6821]: vcpu0 unhandled rdmsr: 0xe7 [ 8704.005013] kvm [6821]: vcpu0 unhandled rdmsr: 0x31 [ 8704.005030] kvm [6821]: vcpu0 unhandled rdmsr: 0x35 [ 8704.005050] kvm [6821]: vcpu0 unhandled rdmsr: 0x39 [ 8704.005095] kvm [6821]: vcpu0 unhandled rdmsr: 0xce [ 8704.005108] kvm [6821]: vcpu0 unhandled rdmsr: 0x194

Any help ?
Thanks

try

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Nop, i test it but nothing change, and i remove one card, so now i have only one GPU on my VM.

[17229.256218] vfio-pci 0000:01:00.0: irq 56 for MSI/MSI-X [17311.585365] kvm [11122]: vcpu0 ignored rdmsr: 0xce [17311.585373] kvm [11122]: vcpu0 ignored rdmsr: 0xce [17311.589590] kvm [11122]: vcpu0 ignored rdmsr: 0x637 [17311.593788] kvm [11122]: vcpu0 ignored rdmsr: 0x637 [17311.593804] kvm [11122]: vcpu0 ignored rdmsr: 0xe8 [17311.598002] kvm [11122]: vcpu0 ignored rdmsr: 0xe8 [17311.598013] kvm [11122]: vcpu0 ignored rdmsr: 0xe7 [17311.602225] kvm [11122]: vcpu0 ignored rdmsr: 0xe7 [17311.602252] kvm [11122]: vcpu0 ignored rdmsr: 0x31 [17311.602257] kvm [11122]: vcpu0 ignored rdmsr: 0x35

Offline

#1520 2014-04-09 21:58:29

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just to get it right:
The IOMMU patch is meant to "rearrange" devices in the groups? At least this is what I understood from here
In my case the USB3 Controller and the sound card are in the same group as a pci bridge and several pci exporess root ports. Would be great if I could change that.

Offline

#1521 2014-04-09 22:12:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
Eb wrote:
aw wrote:

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).

Could of sworn I already applied the patch, I'll look into it when i'm back at my system. Thanks.

I've re-installed the patched linux-mainline package listed on the main page, but I'm still having the same issue of all my graphics devices being in the same IOMMU group. I'm running a Z87 chipset, would that cause any issues? Is there a specific patch I should apply for this chipset? Thanks a ton for the help.

Edit:

I've also read that this can potentially disable dri, which is something I need. Am I looking at a lost cause here?

You also need correct cmdline option in your bootloader otherwise the patch does nothing

Offline

#1522 2014-04-10 12:35:46

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You also need correct cmdline option in your bootloader otherwise the patch does nothing

I'm currently using:

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:6810,1002:aab0

Am I missing anything?

Also, does the linux-mainline package on the first page contain the second version of the patch?

beckend wrote:

Try the V2 of the IOMMU patch: http://linux-kernel.2935.n7.nabble.com/ … 96884.html
Do not apply the old acs patch.
Edit: only works on Intel.

I've tried to apply all three parts of the patch, however I'm getting "patch unexpectedly ends in middle of line" for 2/3 and 3/3. After doing this I realized that the patch may not work if there's an older version of the patch installed, or the same one is currently installed. Would I be correct or am I making some sort of silly mistake here?

Edit:

I've failed to mention that I'm fairly new with linux, sorry for any common questions that I may ask.

Last edited by Eb (2014-04-10 12:37:46)

Offline

#1523 2014-04-10 20:24:55

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe they are conflicting patch (that was the case for me), skip the "override_for_missing_acs_capabilities.patch" if you plan to use V2 patch, start fresh undo all patches and go from there.

Offline

#1524 2014-04-10 20:27:59

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Still not having much luck with vfio vga passthrough on my AMD R9 290 sad

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.


gneville wrote:

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad



gneville wrote:

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

#1525 2014-04-10 21:35:39

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

beckend wrote:

I believe they are conflicting patch (that was the case for me), skip the "override_for_missing_acs_capabilities.patch" if you plan to use V2 patch, start fresh undo all patches and go from there.

I've started fresh from the package on the main page, and have removed "override_for_missing_acs_capabilities.patch" from the pkgbuild and added in all three parts (in seperate files) of the ACS override patch. When I try to makepkg, I'm getting this:

==> Validating source files with md5sums... linux-3.14.tar.xz ... Passed config ... Passed config.x86_64 ... Passed linux.preset ... Passed change-default-console-loglevel.patch ... Passed 0001-Bluetooth-allocate-static-minor-for-vhci.patch ... Passed 0002-module-allow-multiple-calls-to-MODULE_DEVICE_TABLE-p.patch ... Passed 0003-module-remove-MODULE_GENERIC_TABLE.patch ... Passed 0004-Revert-syscalls.h-use-gcc-alias-instead-of-assembler.patch ... Passed enable_additional_cpu_optimizations_for_gcc.patch.gz ... Passed radeon_load_vbios_from_file.patch ... Passed aw_acs_1.patch ... Passed aw_acs_2.patch ... Passed aw_acs_3.patch ... Passed i915_314.patch ... Passed ==> Extracting sources... -> Extracting linux-3.14.tar.xz with bsdtar -> Extracting enable_additional_cpu_optimizations_for_gcc.patch.gz with gzip ==> Starting prepare()... patching file kernel/printk/printk.c Hunk #1 succeeded at 59 with fuzz 2 (offset 1 line). patching file Documentation/devices.txt patching file drivers/bluetooth/hci_vhci.c patching file include/linux/miscdevice.h patching file include/linux/module.h patching file scripts/mod/file2alias.c patching file include/linux/isapnp.h patching file include/linux/module.h patching file include/linux/compat.h patching file include/linux/syscalls.h patching file drivers/gpu/drm/radeon/radeon_bios.c patching file drivers/pci/pci.c Hunk #1 FAILED at 2180. Hunk #2 FAILED at 2212. 2 out of 2 hunks FAILED -- saving rejects to file drivers/pci/pci.c.rej patching file drivers/pci/quirks.c Hunk #1 succeeded at 3461 with fuzz 2. patching file include/linux/pci.h Hunk #1 succeeded at 1530 with fuzz 1 (offset 20 lines). Hunk #2 succeeded at 1543 with fuzz 2 (offset 20 lines). ==> ERROR: A failure occurred in prepare(). Aborting...

I just know I'm missing something simple here, anyone see what it is? Do I need to edit the patches in some way?

Offline

#1526 2014-04-10 21:56:15

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Weee got Crysis 3 working ^^, now im trying to hunt the audio issue, nbhs, there is any equivalent of disabling nested page table thing under Intel?

PD: also found this, those who want to optimize might found interesting http://pic.dhe.ibm.com/infocenter/lnxin … ng_pdf.pdf

Offline

#1527 2014-04-10 22:19:29

tswindell
Member
Registered: 2014-04-10
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Managed to get this working out-of-the-box with Ubuntu 14.04 RC

Steps:
  - Added intel_iommu=on to /etc/default/grub
  - Added i915, nouveau, nvidia to /etc/modprobe.d/blacklist.conf
  - sudo update-initramfs -u
  - sudo update-grub
  - rebooted and installed qemu-kvm
  - Used vfio-bind script from OP
  - Executed qemu as OP

Hardware:
  - Intel i7 4771
  - ASUS Z87 Plus
  - 16GB DDR3 RAM
  - 2x nVidia GeForce 680 GTX

I have only one issue now, and that is that both nVidia cards are in the same iommu group, I'm guessing this is an issue with my motherboard, and it means I can't run two guests and dedicate a card to each, which is my end goal. hmm

I guess this is what the ACS override patch is for, so I'm going to apply V2 of that and see if that works.

Last edited by tswindell (2014-04-10 22:38:16)

Offline

#1528 2014-04-11 00:46:12

lildigiman
Member
Registered: 2014-04-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tswindell wrote:

Managed to get this working out-of-the-box with Ubuntu 14.04 RC

Steps:
  - Added intel_iommu=on to /etc/default/grub
  - Added i915, nouveau, nvidia to /etc/modprobe.d/blacklist.conf
  - sudo update-initramfs -u
  - sudo update-grub
  - rebooted and installed qemu-kvm
  - Used vfio-bind script from OP
  - Executed qemu as OP

Hardware:
  - Intel i7 4771
  - ASUS Z87 Plus
  - 16GB DDR3 RAM
  - 2x nVidia GeForce 680 GTX

I have only one issue now, and that is that both nVidia cards are in the same iommu group, I'm guessing this is an issue with my motherboard, and it means I can't run two guests and dedicate a card to each, which is my end goal. hmm

I guess this is what the ACS override patch is for, so I'm going to apply V2 of that and see if that works.

Are you using the Intel GPU for the host?

Offline

#1529 2014-04-11 06:13:36

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb, I have no idea what your missing, however, here is my PKGBUILD and acsv2 patch in pastebin.
PKGBUILD: http://pastebin.com/G7huLq6p
acs.patch: http://pastebin.com/gpTZiuXv

Offline

#1530 2014-04-11 07:40:15

tswindell
Member
Registered: 2014-04-10
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lildigiman wrote:

Are you using the Intel GPU for the host?

Yes, indeed I am, though blacklisting i915 is the only way I can get the VM to boot, presumably because of the VGA arbitration issues. Not a problem though as I'm running the host practically headless. Only issue for me like I said is separating the two 680 GTX cards from the iommu group they're in to be able to use each in a separate VM container.

Offline

#1531 2014-04-11 14:47:20

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sadly the V2 patch doesn't work for me, it groups the first and second PCI-E slots on same group >.<

/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@ /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@ /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@ /sys/bus/pci/devices/0000:02:00.1/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@

I will unapply the v2 patch and put again the old acs override, that in my case works great...

EDIT:

I also post the dmesg showing that the patch was active...

dmesg | grep ACS [ 0.181833] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled [ 0.181949] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled [ 0.182066] pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled

Last edited by Cubex (2014-04-11 14:49:39)

Offline

#1532 2014-04-11 15:11:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Sadly the V2 patch doesn't work for me, it groups the first and second PCI-E slots on same group >.<

/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@ /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@ /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@ /sys/bus/pci/devices/0000:02:00.1/iommu_group/devices: 0000:00:01.0@ 0000:00:01.1@ 0000:01:00.0@ 0000:01:00.1@ 0000:02:00.0@ 0000:02:00.1@

I will unapply the v2 patch and put again the old acs override, that in my case works great...

EDIT:

I also post the dmesg showing that the patch was active...

dmesg | grep ACS [ 0.181833] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled [ 0.181949] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled [ 0.182066] pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled

The upstream patch is for PCH root ports only, 00:1c.*.  Complain to Intel about processor based root ports.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1533 2014-04-11 17:04:14

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

beckend wrote:

Eb, I have no idea what your missing, however, here is my PKGBUILD and acsv2 patch in pastebin.
PKGBUILD: http://pastebin.com/G7huLq6p
acs.patch: http://pastebin.com/gpTZiuXv

Thanks, rebuilt with what you provided with no issues.

Unfortunately, with v2 of the patch I'm getting the same problem where I have an overly large IOMMU group containing all my video devices and then some (See: https://bbs.archlinux.org/viewtopic.php … #p1402357).

I have the downstream option enabled for pcie_acs_override in my boot options, as well as intel_iommu.

I had this working before, but that was before installing another video card to try to fix some terrible graphical artifacts on the host when starting the KVM. I believe I was using linux-mainline 3.13 at that point as well.

Is there anything else I can try?

Offline

#1534 2014-04-11 17:17:25

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to share my solution to those horrible directx9 performance issues i had. I think it was just me overlooking some things as first post did not put them entirely in detail. Basically what i did was:
1. Added kvm-amd.npt=0 to grub command line. i had config in /etc/modprobe.d/.. but it did not work because kvm stuff is compiled in kernel and not as module that.
2. Did not use any patches i most likely dont need (like intel fixes since i have amd cpu/chipset):

patch -p1 -i "${srcdir}/0001-Bluetooth-allocate-static-minor-for-vhci.patch" patch -p1 -i "${srcdir}/0002-module-allow-multiple-calls-to-MODULE_DEVICE_TABLE-p.patch" patch -p1 -i "${srcdir}/0003-module-remove-MODULE_GENERIC_TABLE.patch" patch -p1 -i "${srcdir}/radeon_load_vbios_from_file.patch" patch -p1 -i "${srcdir}/override_for_missing_acs_capabilities.patch"

3. Copied timer subsystem configs from config suplied in first post:

CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set CONFIG_NO_HZ_IDLE=y # CONFIG_NO_HZ_FULL is not set # CONFIG_NO_HZ is not set CONFIG_HIGH_RES_TIMERS=y

after recompiling kernel performance is great. passmark still scores about 90% of bare metal speed on directx9 complex benchmark but hell i am ready to sacrifice that much since gpu is good enough to leave some room for penalties. directx10/11 benchmarks score in ~98% of bare metal performance though.

I suppose i missed all this stuff because of being rather new to linux and working on linux mint instead of arch. So newbies - make sure you put configs in right spot (modprobe.d or grub) and be aware of additional kernel build options!

@nbhs: it would be great if you could update first post with details of any other important kernel build options because not everyone might wish to use those configs supplied. Also all this information you shared made me and i bet lots of other people very happy pandas. Thanks bro.

Offline

#1535 2014-04-11 19:08:54

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
beckend wrote:

Eb, I have no idea what your missing, however, here is my PKGBUILD and acsv2 patch in pastebin.
PKGBUILD: http://pastebin.com/G7huLq6p
acs.patch: http://pastebin.com/gpTZiuXv

Thanks, rebuilt with what you provided with no issues.

Unfortunately, with v2 of the patch I'm getting the same problem where I have an overly large IOMMU group containing all my video devices and then some (See: https://bbs.archlinux.org/viewtopic.php … #p1402357).

I have the downstream option enabled for pcie_acs_override in my boot options, as well as intel_iommu.

I had this working before, but that was before installing another video card to try to fix some terrible graphical artifacts on the host when starting the KVM. I believe I was using linux-mainline 3.13 at that point as well.

Is there anything else I can try?

I've removed my secondary video card, and got this:

ls -l /sys/kernel/iommu_groups/1/devices total 0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:02:08.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:02:10.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:02:11.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:11.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0 lrwxrwxrwx 1 root root 0 Apr 9 11:33 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1

The IOMMU group is still quite large, but when I try to run the KVM, it runs.

After putting my secondary video card back in, I'm running into the same old errors. I've tried messing with my boot line a bit, running pcie_acs_override=downstream seems to not make any changes. Perhaps I patched it in wrong? Maybe I'm not installing the new kernel correctly? I'm patching (via PKGBUILD), then building via makepkg, then using pacman -U <local_package_name> to install it. Am I missing something here? I'm using the PKGBUILD and ACS patch as provided in the quote at the top of this post.

When I check dmesg for anything mentioning acs, I only get the following:

dmesg | grep acs [ 0.000000] Command line: BOOT_IMAGE=../vmlinuz-linux root=UUID=30b74524-9b44-46bc-bcca-6f7ebafdbf0a rw intel_iommu=on pci-stub.ids=1002:6810,1002:aab0 pcie_acs_override=downstream initrd=../initramfs-linux.img [ 0.000000] Kernel command line: BOOT_IMAGE=../vmlinuz-linux root=UUID=30b74524-9b44-46bc-bcca-6f7ebafdbf0a rw intel_iommu=on pci-stub.ids=1002:6810,1002:aab0 pcie_acs_override=downstream initrd=../initramfs-linux.img

Checking for ACS instead of acs gives:

dmesg | grep ACS [ 0.000000] ACPI: FACS 000000007c172080 000040

Looking for mentions of IOMMU gives:

dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.015176] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.015180] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.015245] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.498206] IOMMU 0 0xfed90000: using Queued invalidation [ 0.498207] IOMMU 1 0xfed91000: using Queued invalidation [ 0.498208] IOMMU: Setting RMRR: [ 0.498217] IOMMU: Setting identity map for device 0000:00:02.0 [0x7f800000 - 0x8f9fffff] [ 0.499450] IOMMU: Setting identity map for device 0000:00:1d.0 [0x7c00d000 - 0x7c01afff] [ 0.499470] IOMMU: Setting identity map for device 0000:00:1a.0 [0x7c00d000 - 0x7c01afff] [ 0.499488] IOMMU: Setting identity map for device 0000:00:14.0 [0x7c00d000 - 0x7c01afff] [ 0.499500] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.499505] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 1.889788] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 1.889790] AMD IOMMUv2 functionality not available on this system

I'm running an Intel core, not AMD, so I don't think that the AMD IOMMUv2 messages at the end matter.

Edit

D'oh, was never actually booting into the new kernel. However, still getting the same issue of IOMMU group containing the video cards being too large. The patch seems to be working however, as I have many, many more groups.

Last edited by Eb (2014-04-11 20:04:23)

Offline

#1536 2014-04-11 19:28:49

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've just found something interesting with my setup.

I can get around the BSOD on Windows boot after installing the drivers, providing I do a fresh reload of the Host. I also had to remove the HDMI Audio from the ioh3420 device.

Edit: Confirmed this is the same for Win7 & Win8 as a Guest

But provided I reboot the Host everytime I boot the Windows Guest with the below qemu commands it will start up and use the AMD drivers correctly, if I then reboot the Guest or Shutdown and restart the Qemu again from Command line then Windows will BSOD on boot with the "PAGE_FAULT_IN_NONPAGED_AREA" error.

Any ideas what can be causing this please?

/usr/local/qemu-xen-4-4-git-patched/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/xen/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio



gneville wrote:

Still not having much luck with vfio vga passthrough on my AMD R9 290 sad

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.


gneville wrote:

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad



gneville wrote:

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:





After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:




This is my QEMU config:



If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Last edited by gneville (2014-04-11 19:55:10)

Offline

#1537 2014-04-11 23:43:57

Nex7
Member
Registered: 2014-04-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've discovered what I believe to be my problem, but I'm confused why nobody else seems to be hitting it.

https://lists.gnu.org/archive/html/qemu … 01858.html

That's what's hitting me - I'm getting c0000221 stop's on boot on my VM, at this point, and seemingly this is due to virtio-scsi. I wonder why nobody else is hitting this?

EDIT: Of course, starting over with a fresh Win7 x64 install using ide instead of virtio, it works fine up until I try to install Windows updates, even only a few of them, at which point after reboot it dies with a different BSOD, code c000021a. -sigh-

Last edited by Nex7 (2014-04-12 02:55:53)

Offline

#1538 2014-04-12 03:29:55

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nex7 wrote:

So I've discovered what I believe to be my problem, but I'm confused why nobody else seems to be hitting it.

https://lists.gnu.org/archive/html/qemu … 01858.html

That's what's hitting me - I'm getting c0000221 stop's on boot on my VM, at this point, and seemingly this is due to virtio-scsi. I wonder why nobody else is hitting this?

EDIT: Of course, starting over with a fresh Win7 x64 install using ide instead of virtio, it works fine up until I try to install Windows updates, even only a few of them, at which point after reboot it dies with a different BSOD, code c000021a. -sigh-

you are not alone here. i made attempt to use virtio-scsi too. however i had OS installed on IDE drive and later on switched to virtio-scsi. what i experienced was various files being seemingly corrupted (software throwing errors due to crc32 missmatch, applications failing to read files etc). switching back to virtio made it all work fine again. seems like virtio-scsi does cause some reading problems and what not. so just stick to virtio i guess. its really fast anyway.

Last edited by novist (2014-04-12 03:30:48)

Offline

#1539 2014-04-12 04:36:29

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success! ..kind of. I've compiled with the old patch and everything seems to be working proper. After installing Windows 8.1 I decided to install the drivers for the video card. Windows seems to be running smoothly, but now I have no video output. Looking into exactly what may be happening here now..

Edit:

After trying both beta and non beta drivers, I'm consistantly getting a black screen as soon as I finish installing the drivers and the VM switches from basic output to using the card. Wiping the VM fixed this until installing the drivers once more.  Any ideas how I can get video back?

Last edited by Eb (2014-04-12 07:16:00)

Offline

#1540 2014-04-12 14:31:25

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Adding -curses to the qemu-system options fixed this for me.

rabcor wrote:
nbhs wrote:

Try:

-cpu qemu64

and remove the -smp line

If that worked you might have built qemu without kvm support

#qemu-system-x86_64 -enable-kvm -M q35 -cpu qemu64 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -cpu qemu64: could not open disk image -bios: No such file or directory

I'm looking into reinstalling qemu but like i said, you removed the link from your original post and i don't have any different source to build from than my current one except the git one, which is v 1.6 and has seabios intergrated, and i don't know how to patch the intergrated seabios, i couldn't find out how (but otherwise that is what i would do).

i tried removing the backslashes and the VM will start if i run it from X.

qemu-system-x86_64 -enable-kvm -M q35 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

if not running from x (terminal) then i get

Could not initialize SDL(No available video device) - exiting

instead of

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device). VGA may not work.

Offline

#1541 2014-04-13 02:53:58

Nex7
Member
Registered: 2014-04-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

you are not alone here. i made attempt to use virtio-scsi too. however i had OS installed on IDE drive and later on switched to virtio-scsi. what i experienced was various files being seemingly corrupted (software throwing errors due to crc32 missmatch, applications failing to read files etc). switching back to virtio made it all work fine again. seems like virtio-scsi does cause some reading problems and what not. so just stick to virtio i guess. its really fast anyway.

Maybe I'm confused - can you show me your commandline, that is using 'virtio' but not 'virtio-scsi'?

In other news, I did get this working on IDE, but have not yet retried virtio-scsi. The symptom was every time I installed the Windows 7 64-bit OS it was fine UNTIL I did ANY Windows Updates. I tried a few times, with different packages, always the same. This got me thinking it isn't the updates themselves, but something they have in common - going down that list, I hit upon 'Restore Points', as those happen any time you do any Windows Updates. So I turned off Restore Points on the OS disk entirely, and after that, did Windows Update, and it came back up after reboot w/o BSOD. It is now completely stable. I have no idea why Restore Points screw the pooch, but they seem to, at least on IDE (haven't retried with virtio).

Offline

#1542 2014-04-13 04:29:12

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:

Success! ..kind of. I've compiled with the old patch and everything seems to be working proper. After installing Windows 8.1 I decided to install the drivers for the video card. Windows seems to be running smoothly, but now I have no video output. Looking into exactly what may be happening here now..

Edit:

After trying both beta and non beta drivers, I'm consistantly getting a black screen as soon as I finish installing the drivers and the VM switches from basic output to using the card. Wiping the VM fixed this until installing the drivers once more.  Any ideas how I can get video back?

I've booted directly into the Windows system (8.1) and the card works perfectly.  Back to KVM and it doesn't work. I've even tried to install manufacturer drivers, no dice there either. I'm not sure completely if this is an issue with drivers or my hardware, but since it's only having trouble with QEMU I'm hoping that there's some options I can change when booting the KVM. Are there any QEMU boot options I can experiment with? Should just return the card and try my luck with a different model?

Eagerly waiting for a reply, I don't have much of a window left to return the card.

Offline

#1543 2014-04-13 06:26:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
Eb wrote:

Success! ..kind of. I've compiled with the old patch and everything seems to be working proper. After installing Windows 8.1 I decided to install the drivers for the video card. Windows seems to be running smoothly, but now I have no video output. Looking into exactly what may be happening here now..

Edit:

After trying both beta and non beta drivers, I'm consistantly getting a black screen as soon as I finish installing the drivers and the VM switches from basic output to using the card. Wiping the VM fixed this until installing the drivers once more.  Any ideas how I can get video back?

I've booted directly into the Windows system (8.1) and the card works perfectly.  Back to KVM and it doesn't work. I've even tried to install manufacturer drivers, no dice there either. I'm not sure completely if this is an issue with drivers or my hardware, but since it's only having trouble with QEMU I'm hoping that there's some options I can change when booting the KVM. Are there any QEMU boot options I can experiment with? Should just return the card and try my luck with a different model?

Eagerly waiting for a reply, I don't have much of a window left to return the card.

Have you tried swapping card to check if you get the same result? Have you tried to passthrough only the gpu without the audio device?

Offline

#1544 2014-04-13 08:48:48

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I have a question regarding disk performance. Originally I was using a file on a XFS partition as hard disk image:

-drive if=none,file=/home/vms/drives/winimg.img,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1 \

The physical hard disk is not an SSD

With this I can't use the virtio option x-data-plane=on.
However in the Windows Performance Indicator the Harddisk had a score of 6.5

Then I switched to using an physical hard disk partition with the following configuration:

-drive if=none,file=/dev/sdb7,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=on \

I thought performance would increase a bit, but the performance indicator dropped to 5.9.

I there some configuarion on the host, I missed?

BR

Offline

#1545 2014-04-13 19:48:22

spam
Member
Registered: 2013-06-15
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have been trying vga-passthroug but I have managed to get it properly yet.

My hardware config is:
ASRock Z87 Extreme 4
i7 4771
GTX 660 (ASUS DC2O)

I have used the OP packages and with and without this patch  http://lists.nongnu.org/archive/html/qe … 00597.html 

I have not read all thread yet and it is a bit confusing with patches here and there. I can only get a code 43 in windows. I believe I have to enable acs  but atm do I need a patch or only a kernel parameter for linux-mainline ?

I boot in my MB in UEFI mode (at least I believe so), does it matter ?

./lsgroup added ### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 04:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

I add a bit of information on my software configuration:

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=bc42943b-cfbd-47df-931f-40a88e0a4ef9 rw quiet intel_iommu=on pci-stub.ids=10de:11c0,10de:0e0b
$ cat /etc/modprobe.d/alsa-base.conf /etc/modprobe.d/blacklist-ath_pci.conf /etc/modprobe.d/blacklist.conf /etc/modprobe.d/blacklist-firewire.conf /etc/modprobe.d/blacklist-framebuffer.conf /etc/modprobe.d/blacklist-gpu.conf /etc/modprobe.d/blacklist-modem.conf /etc/modprobe.d/blacklist-oss.conf /etc/modprobe.d/blacklist-rare-network.conf /etc/modprobe.d/blacklist-watchdog.conf /etc/modprobe.d/dkms.conf /etc/modprobe.d/fbdev-blacklist.conf /etc/modprobe.d/iwlwifi.conf /etc/modprobe.d/mlx4.conf /etc/modprobe.d/modprobe.conf /etc/modprobe.d/pci-stub.conf /etc/modprobe.d/qemu-system-x86.conf /etc/modprobe.d/vfio_iommu_type1.conf /etc/modprobe.d/vmwgfx-fbdev.conf blacklist nouveau blacklist nvidia options kvm ignore_msrs=1 options pci-stub ids=10de:11c0,10de:0e0b options vfio_iommu_type1 allow_unsafe_interrupts=1
$ cat /etc/modules-load.d/cups-filters.conf pci-stub

And to finish the qemu command I have been trying:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/home/spam/windows.img,id=disk,format=raw \ -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/spam/windows.iso,id=isocd \ -device ide-cd,bus=ide.1,drive=isocd \ -usb -usbdevice host:003.006

And to finish, the last dmesg

[ 5663.995723] VFIO - User Level meta-driver version: 0.3 [ 6337.160714] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 6337.160846] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 6376.195200] kvm: zapping shadow pages for mmio generation wraparound

Last edited by spam (2014-04-13 19:51:17)

Offline

#1546 2014-04-14 04:45:57

Torello
Member
Registered: 2014-04-01
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
Finally, vga-passthrough work. ( Using kernel with debug registers patch, qemu and seabios from first post ).
But I have a strange issue, which makes keyboard and mouse in guest os - unusable.

My specs:
Core i7 4771, GA-Z87X-OC , Radeon 6950@6970 for guest and integrated intel video for host.
Keyboard Microsoft Natural 4000 ( usb ), mouse Logitech Mx510 ( usb ).

My startup script is:
qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/home/torello/virt_images/windows7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/torello/distr/win7_msdn.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on

If I boot vm without "-vga none".
I can switch to qemu window, and it grabs my keyboard and mouse input.

If I boot with "-vga none", I can't use keyboard in guest at all, and with my mouse I can do some operations like scrolling the window using the thumb, or change window place ( using mouse ), but If I want to click a button I must click 20-30 times, trying different corners, before I was able to press any onscreen button.

Do you have any suggestions, how I can debug/solve that issue?

Last edited by Torello (2014-04-14 05:23:42)

Offline

#1547 2014-04-14 08:16:41

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Torello: pass-through usb ports too.

Offline

#1548 2014-04-14 10:54:09

FruitieX
Member
Registered: 2008-07-03
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

After updating to 3.14 kernel with all patches as posted in the OP, I'm getting a *massive* amount of spam in dmesg while my VM is running. (so much that the systemd journal can't keep up and uses 100% CPU)
As a result I get terrible performance in the VM, whereas with the 3.13 kernel I get very good performance and can play Windows games at near native speeds.
Here's the dmesg output that gets repeated:

[ 99.118043] WARNING: CPU: 2 PID: 1519 at arch/x86/kvm/x86.c:1114 kvm_get_msr_common+0x793/0x960 [kvm]() [ 99.118047] Modules linked in: vfio_pci vfio_iommu_type1 vfio ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_ CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 nls_cp437 vfat fat ext4 mbcache jbd2 snd_hda_codec_hdmi iTCO _wdt iTCO_vendor_support ppdev hid_logitech_dj coretemp hwmon x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_ 64 lrw gf128mul glue_helper ablk_helper cryptd btusb microcode bluetooth pcspkr 6lowpan_iphc psmouse serio_raw i2c_i801 e1000e ptp pps_core lpc_ich rfkill crc16 mei_me mei snd_usb_audi o parport_pc tpm_tis snd_usbmidi_lib mousedev evdev joydev snd_rawmidi snd_hda_codec_realtek [ 99.118064] snd_seq_device tpm mac_hid snd_hda_codec_generic i915 nuvoton_cir parport battery rc_core snd_hda_intel snd_hda_codec drm_kms_helper snd_hwdep video drm snd_pcm button intel_gtt i2c_algo_bit snd_timer i2c_core snd soundcore shpchp processor btrfs xor raid6_pq hid_generic usbhid hid bcache sd_mod crc_t10dif crct10dif_common atkbd libps2 crc32c_intel a hci libahci libata scsi_mod ehci_pci xhci_hcd ehci_hcd usbcore usb_common i8042 serio pci_stub [ 99.118078] CPU: 2 PID: 1519 Comm: qemu-system-x86 Tainted: G W 3.14.0-ARCH #1 [ 99.118079] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87M Pro4, BIOS P2.00 11/07/2013 [ 99.118080] 0000000000000009 ffff8800b9505c08 ffffffff814cea8a 0000000000000000 [ 99.118081] ffff8800b9505c40 ffffffff81064b1d 0000000040000020 ffff8800b9505cd0 [ 99.118083] ffff8804132e3ec0 ffff8803fbaf0000 ffff8803fbaf0040 ffff8800b9505c50 [ 99.118084] Call Trace: [ 99.118085] [<ffffffff814cea8a>] dump_stack+0x4d/0x6f [ 99.118086] [<ffffffff81064b1d>] warn_slowpath_common+0x7d/0xa0 [ 99.118087] [<ffffffff81064bfa>] warn_slowpath_null+0x1a/0x20 [ 99.118091] [<ffffffffa0952233>] kvm_get_msr_common+0x793/0x960 [kvm] [ 99.118093] [<ffffffffa0ad1310>] ? vmx_get_msr+0x370/0x370 [kvm_intel] [ 99.118095] [<ffffffffa0ad10a2>] vmx_get_msr+0x102/0x370 [kvm_intel] [ 99.118097] [<ffffffffa0ad133a>] handle_rdmsr+0x2a/0x140 [kvm_intel] [ 99.118099] [<ffffffffa0ad1b8f>] ? handle_exception+0x16f/0x3e0 [kvm_intel] [ 99.118101] [<ffffffffa0ad6975>] vmx_handle_exit+0xb5/0xa20 [kvm_intel] [ 99.118103] [<ffffffffa0ace360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel] [ 99.118107] [<ffffffffa09577a0>] kvm_arch_vcpu_ioctl_run+0xc70/0x11c0 [kvm] [ 99.118110] [<ffffffffa095363e>] ? kvm_arch_vcpu_load+0x4e/0x1e0 [kvm] [ 99.118113] [<ffffffffa09405c2>] kvm_vcpu_ioctl+0x2b2/0x5b0 [kvm] [ 99.118115] [<ffffffff814d11fd>] ? __schedule+0x86d/0x900 [ 99.118116] [<ffffffff8101e966>] ? ___preempt_schedule+0x56/0xb0 [ 99.118118] [<ffffffff811b3de0>] do_vfs_ioctl+0x2e0/0x4c0 [ 99.118119] [<ffffffff811bdace>] ? __fget+0x6e/0xb0 [ 99.118121] [<ffffffff811b4041>] SyS_ioctl+0x81/0xa0 [ 99.118122] [<ffffffff814dc469>] system_call_fastpath+0x16/0x1b [ 99.118126] ---[ end trace 8d24f63038d6e9d0 ]---

I'm noticing that if I remove the hv-time flag from my qemu launch options I don't get this dmesg spam anymore, but VM performance is still really bad up to a point where even moving the mouse cursor is laggy.
Is anyone else experiencing this issue, and any ideas how to fix this? Again, the 3.13 kernel was fine.

System specs:
CPU: Intel i5-4670
GPU: AMD Radeon 280X
Mobo: ASRock Z87M Pro4

FruitieX

Last edited by FruitieX (2014-04-14 10:55:31)

Offline

#1549 2014-04-14 11:46:28

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FruitieX wrote:

Hi,

After updating to 3.14 kernel with all patches as posted in the OP, I'm getting a *massive* amount of spam in dmesg while my VM is running. (so much that the systemd journal can't keep up and uses 100% CPU)
As a result I get terrible performance in the VM, whereas with the 3.13 kernel I get very good performance and can play Windows games at near native speeds.
Here's the dmesg output that gets repeated:


I'm noticing that if I remove the hv-time flag from my qemu launch options I don't get this dmesg spam anymore, but VM performance is still really bad up to a point where even moving the mouse cursor is laggy.
Is anyone else experiencing this issue, and any ideas how to fix this? Again, the 3.13 kernel was fine.

System specs:
CPU: Intel i5-4670
GPU: AMD Radeon 280X
Mobo: ASRock Z87M Pro4

FruitieX

https://bugzilla.kernel.org/show_bug.cgi?id=73721
Still waiting for an answer.

Offline

#1550 2014-04-14 12:57:43

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:
FruitieX wrote:

Hi,

After updating to 3.14 kernel with all patches as posted in the OP, I'm getting a *massive* amount of spam in dmesg while my VM is running. (so much that the systemd journal can't keep up and uses 100% CPU)
As a result I get terrible performance in the VM, whereas with the 3.13 kernel I get very good performance and can play Windows games at near native speeds.
Here's the dmesg output that gets repeated:


I'm noticing that if I remove the hv-time flag from my qemu launch options I don't get this dmesg spam anymore, but VM performance is still really bad up to a point where even moving the mouse cursor is laggy.
Is anyone else experiencing this issue, and any ideas how to fix this? Again, the 3.13 kernel was fine.

System specs:
CPU: Intel i5-4670
GPU: AMD Radeon 280X
Mobo: ASRock Z87M Pro4

FruitieX

https://bugzilla.kernel.org/show_bug.cgi?id=73721
Still waiting for an answer.

Hi,

I do not understand your problem because i do not have it !!

i use

qemu-system-x86_64 \ -M q35 -m 8G -enable-kvm -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -no-user-config -nodefaults -vga none -boot d -nographic \ -smp 4,sockets=1,cores=4,threads=1 \ ...

And i have not any problem.

Qemu 2.0.0-RC2 + original kernel 3.14 from kernel.org + i915, acs (and other litle patch on KVM side but without it, it works the same).

On a Intel Core i5-4670S + Asrock Z87 Extrem 6 and two HD 6870.

Offline

#1551 2014-04-14 14:24:57

FruitieX
Member
Registered: 2008-07-03
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

And i have not any problem.

Qemu 2.0.0-RC2 + original kernel 3.14 from kernel.org + i915, acs (and other litle patch on KVM side but without it, it works the same).

Strange. I tried compiling the 3.14 kernel with the i915 and acs patches from the first post but am still having this issue. I'm using qemu from latest git.

I investigated this a bit more and found that this error message is generated when I have both compiled my kernel with pre-emption on and am using the hv-time qemu cpu flag. You didn't happen to change the kernel configuration from the one in the first post?

FruitieX

Offline

#1552 2014-04-14 16:20:55

scoobydog
Member
Registered: 2014-04-14
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Torello wrote:

Hello,
Finally, vga-passthrough work. ( Using kernel with debug registers patch, qemu and seabios from first post ).
But I have a strange issue, which makes keyboard and mouse in guest os - unusable.

My specs:
Core i7 4771, GA-Z87X-OC , Radeon 6950@6970 for guest and integrated intel video for host.
Keyboard Microsoft Natural 4000 ( usb ), mouse Logitech Mx510 ( usb ).

My startup script is:
qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/home/torello/virt_images/windows7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/torello/distr/win7_msdn.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on

If I boot vm without "-vga none".
I can switch to qemu window, and it grabs my keyboard and mouse input.

If I boot with "-vga none", I can't use keyboard in guest at all, and with my mouse I can do some operations like scrolling the window using the thumb, or change window place ( using mouse ), but If I want to click a button I must click 20-30 times, trying different corners, before I was able to press any onscreen button.

Do you have any suggestions, how I can debug/solve that issue?

Use a dummy console with -vga none -device qxl

So you don't need to passthru USB devices for testings.

Scoobydog

Offline

#1553 2014-04-14 16:42:53

Torello
Member
Registered: 2014-04-01
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

@Torello: pass-through usb ports too.

Thanks, but I want to use one keyboard and mouse on my system. I have already done that with some old version of qemu.

scoobydog wrote:

@Torello: Use a dummy console with -vga none -device qxl
So you don't need to passthru USB devices for testings.
Scoobydog

Thanks! That's exactly what I need, now mouse and keyboard fully functional. Hurray!

Offline

#1554 2014-04-14 17:14:31

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

after my precious time went wasted with a crap PowerColor card, that turned out to be faulty (lot of freezes also before with opencl/darktable), received my GTX TI 750, works like charm on the guest with nvidia drivers! finally an old dream became true... can do photography on virtual machine with dedicated graphic card!

Offline

#1555 2014-04-14 18:36:44

Torello
Member
Registered: 2014-04-01
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now there is only one problem left - how to pass my 2tb gpt disk to qemu?
If I use that command:
-drive file=/dev/sdc2,id=diske,format=raw -device ide-hd,bus=ide.2,drive=diske
I got 5 or 6 partitions in windows disk manager( but actually there is only one partition ).
That's how I see my disk using lsblk -f

sdc    promise_fasttrack_raid_member                                           
├─sdc1 promise_fasttrack_raid_member                                           
└─sdc2 ntfs                          2 Tb

If I mount sdc2 under linux, using ntfs-3g, all works fine.

Offline

#1556 2014-04-14 20:19:21

scoobydog
Member
Registered: 2014-04-14
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Torello wrote:

Now there is only one problem left - how to pass my 2tb gpt disk to qemu?
If I use that command:
-drive file=/dev/sdc2,id=diske,format=raw -device ide-hd,bus=ide.2,drive=diske
I got 5 or 6 partitions in windows disk manager( but actually there is only one partition ).
That's how I see my disk using lsblk -f

sdc    promise_fasttrack_raid_member                                           
├─sdc1 promise_fasttrack_raid_member                                           
└─sdc2 ntfs                          2 Tb

If I mount sdc2 under linux, using ntfs-3g, all works fine.

The sdc2 is a patition, in your gast PC it's a HDD with Partition Table, the best way is to use LVM oder Image Files, not single Partions.

User -- qemu-img crate -f raw fileneme.img sizeG -- to create a new file in the filesystem, but keep in mind, ntfs mountet to Linux is very slow.

ScoobyDog

Offline

#1557 2014-04-14 22:13:17

FruitieX
Member
Registered: 2008-07-03
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

Okay I managed to restore VM performance to what it was on 3.13. As a temporary fix to get rid of the dmesg spam, here's a very ugly hack (the warning is probably there for a reason):

Just comment out the line that generates these warnings, in the kernel source file arch/x86/kvm/x86.c, line 1120:

Change:

 WARN_ON(preemptible());

into:

 //WARN_ON(preemptible());

and recompile your kernel.

Now the dmesg spam is gone, but I found performance is still terrible especially in the VM, but I noticed it's not as good as it used to be in the host OS either. Turns out that for some reason, even though I used the default "powersave" governor in both kernels, in 3.14 the governor would actually *do* something and thus my CPU frequencies were crazy low all the time. (in 3.13 the CPU frequency was maxed out all the time)

To fix this I set the default governor to "performance" in /etc/default/cpupower, and enabled the cpupower systemd service. (install the cpupower package first!) Now my VM performance is as good as on 3.13!

Hope this helps!
FruitieX

Offline

#1558 2014-04-15 00:02:41

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Those who have problems with keyboard missing key press/releases and similiar, try:

-usbdevice mouse -usbdevice keyboard

This worked as good as passing usb device, with the advantage that I can release/grab anytime

Offline

#1559 2014-04-16 03:06:37

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can this work with the current stable arch kernel (3.14.0-5-ARCH) without recompiling it?

Those are it's default settings

$ zcat /proc/config.gz | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y $ zcat /proc/config.gz | grep STUB CONFIG_EFI_STUB=y CONFIG_PCI_STUB=m

Offline

#1560 2014-04-16 03:57:44

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Can this work with the current stable arch kernel (3.14.0-5-ARCH) without recompiling it?

Those are it's default settings

$ zcat /proc/config.gz | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y $ zcat /proc/config.gz | grep STUB CONFIG_EFI_STUB=y CONFIG_PCI_STUB=m

Depends on your hardware, if you're planning on using the intel IGP then no

Offline

#1561 2014-04-16 06:58:33

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What about Nvidia GTX-670(VM) and Nvidia GTX-550-Ti(Host)?

Offline

#1562 2014-04-16 14:46:18

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Have you tried swapping card to check if you get the same result? Have you tried to passthrough only the gpu without the audio device?

I've removed the audio device, and it works! Been playing around with it for a few hours and no issues! Thanks a bunch.

I really appreciate all the help everyone has given me, keep being cool guys. =]

Last edited by Eb (2014-04-16 14:46:54)

Offline

#1563 2014-04-16 17:27:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

What about Nvidia GTX-670(VM) and Nvidia GTX-550-Ti(Host)?

Yeah that will work, though you'll need to patch the nvidia drivers

Offline

#1564 2014-04-16 17:46:45

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
rabcor wrote:

What about Nvidia GTX-670(VM) and Nvidia GTX-550-Ti(Host)?

Yeah that will work, though you'll need to patch the nvidia drivers

Or use that other unsupported method, if you can. Yes, I'm talking about UEFI boot/efifb.


i'm sorry for my poor english wirting skills…

Offline

#1565 2014-04-17 15:32:58

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:
rabcor wrote:

What about Nvidia GTX-670(VM) and Nvidia GTX-550-Ti(Host)?

Yeah that will work, though you'll need to patch the nvidia drivers

Or use that other unsupported method, if you can. Yes, I'm talking about UEFI boot/efifb.


donno if helps, as stated before for me GTX 750 works on guest (windows + original nvidia driver) and with GeForce 7900 on host (nouveau)

Offline

#1566 2014-04-18 11:24:47

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi All,

I'm using Vanilla packages and found I can always start a VM with a Geforce 620 in VFIO pass through but never restart it, the VM looks to have restarted on the host but no display appears. Libvirt gave stability while running the VM but did not help with the restart problem. Any ideas or do I have to run a custom kernel / packages?

Hardware:
Opteron 4256EE
Asus KCMR-D12

Software:
Kernel 3.14.0-4-ARCH
Qemu V1.7.0

Offline

#1567 2014-04-18 16:45:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Krobar wrote:

Hi All,

I'm using Vanilla packages and found I can always start a VM with a Geforce 620 in VFIO pass through but never restart it, the VM looks to have restarted on the host but no display appears. Libvirt gave stability while running the VM but did not help with the restart problem. Any ideas or do I have to run a custom kernel / packages?

Hardware:
Opteron 4256EE
Asus KCMR-D12

Software:
Kernel 3.14.0-4-ARCH
Qemu V1.7.0

You need qemu =>2.0 or qemu-git

Offline

#1568 2014-04-18 18:39:53

spam
Member
Registered: 2013-06-15
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying adding acs to my config but still had no luck my guest works only as a basic video card, I get a code 43 in windows. I am gonna try a linux guest to get traces. When I try to load a vbios with the romfile option my guest has no outputs at all. Is someone has an idea of what else could I try ? My gtx 660 is not UEFI compatible and I start my computer in UEFI mode, could it be a problem ?

Last edited by spam (2014-04-18 18:40:39)

Offline

#1569 2014-04-19 11:40:15

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Krobar wrote:

Hi All,

I'm using Vanilla packages and found I can always start a VM with a Geforce 620 in VFIO pass through but never restart it, the VM looks to have restarted on the host but no display appears. Libvirt gave stability while running the VM but did not help with the restart problem. Any ideas or do I have to run a custom kernel / packages?

Hardware:
Opteron 4256EE
Asus KCMR-D12

Software:
Kernel 3.14.0-4-ARCH
Qemu V1.7.0

You need qemu =>2.0 or qemu-git

Thanks for the reply. I enabled the testing repo and now have this QEMU version:

pacman -Q qemu
qemu 2.0.0-2
qemu-x86_64 -version
qemu-x86_64 version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

I'm still having the same problem though. VM destroy and create again or reboot of the VM means no display. Its strange because otherwise it works well, XBMC Gotham is running with HD video decode and HDMI audio passthrough.

Offline

#1570 2014-04-19 19:18:04

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
rabcor wrote:

What about Nvidia GTX-670(VM) and Nvidia GTX-550-Ti(Host)?

Yeah that will work, though you'll need to patch the nvidia drivers

Cool! patching the drivers is a lot less of a pain than patching the kernel i think, since I would need to reconfigure the kernel every time i update it but with nvidia drivers i can always just patch them again. I may want to start by trying what pereczes said and just use noveau on the linux host at first. I guess this means I'll give this whole thing a third attempt pretty soon smile

Did anyone try asking around on the nvidia forums yet?

Last edited by rabcor (2014-04-19 19:25:34)

Offline

#1571 2014-04-19 19:28:03

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
nbhs wrote:
rabcor wrote:

What about Nvidia GTX-670(VM) and Nvidia GTX-550-Ti(Host)?

Yeah that will work, though you'll need to patch the nvidia drivers

Cool! patching the drivers is a lot less of a pain than patching the kernel i think, since I would need to reconfigure the kernel every time i update it but with nvidia drivers i can always just patch them again. I may want to start by trying what pereczes said and just use noveau on the linux host at first. I guess this means I'll give this whole thing a third attempt pretty soon smile

Did anyone try asking around on the nvidia forums yet?

There's a thread on there asking about the inclusion of the patch: https://devtalk.nvidia.com/default/topi … tion-lock/


i'm sorry for my poor english wirting skills…

Offline

#1572 2014-04-19 20:13:52

dismuter
Member
Registered: 2013-12-27
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I'm trying to get VGA passthrough working on the following setup:
- Asus Q87 motherboard
- Intel Core i7 4771
- For host: Intel HD
- For guest: AMD Radeon 7770 OR NVIDIA GeForce GTX 660 (not both, whichever I can get to work first)

IOMMU is enabled through kernel switch.
I disabled all the modules which would interfere with guest graphics cards (radeon, nouveau, nvidia)
I did the vfio-bind.

Here's my command line:

qemu-kvm \ -machine pc-q35-2.0 -enable-kvm -cpu host -m 4096 \ -smp 2,sockets=1,cores=2,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none -display none \ -usb -usbdevice host:046d:c52b \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device virtio-scsi-pci,id=scsi0,bus=pcie.0,addr=0x3 \ -drive file=/var/lib/libvirt/images/vm.img,if=none,id=drive-scsi0-0-0-0,format=raw,discard=unmap,aio=threads \ -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

(-display none is because there is no X server; otherwise I get "Gtk-WARNING **: cannot open display")

Results:
- Radeon on guest guests monitor out of standby, displays fine until guest drivers are loaded under Windows
- GeForce throttles fan, but monitor stays in standby, nothing gets displayed (see edit)

On the software side, I'm running Fedora 20 with kernel 3.15-rc1 (from rawhide) and qemu 2.0.
The kernel was compiled with the following config:

CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y

I read that those versions contained sufficient logic for getting this to work. But is that really the case, do they require any patch, must I go through recompiling them?
Does using the integrated graphics for the host have any influence?
What am I missing?

Thanks for any help


EDIT: I got the NVIDIA card working completely apparently by blacklisting i915.
I can even use libvirt by adding this:

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline>

Replace "01:00.x" with your output from lspci:

01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)

(for those interested, don't forget to add the namespace at the beginning of the domain xml: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>)
I did also face this error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=pcie.0,addr=07.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/1: Operation not permitted

and fixed it by adding /dev/vfio/1 to cgroup_device_acl in /etc/libvirt/qemu.conf, setting both user and group to "root" and clear_emulator_capabilities to 0

EDIT 2: Well, thanks to this post from MacCoaster, who had a similar configuration (Intel Haswell + Asus MB + R7770), I added the following to the kernel command line:

vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off

I don't know which of these got it working with the Radeon HD 7770, but it seems quite stable, I can even run the latest 3D Mark smoothly (BSOD with the NVIDIA, but I should try with the new kernel command line).

Last edited by dismuter (2014-04-20 01:01:58)

Offline

#1573 2014-04-20 01:09:38

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dismuter, does your VM reboot OK? I had to disable an Intel HDMI Audio driver to get my GF620 to passthrough, might be related to I915.

Offline

#1574 2014-04-20 01:27:19

dismuter
Member
Registered: 2013-12-27
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Krobar wrote:

Dismuter, does your VM reboot OK? I had to disable an Intel HDMI Audio driver to get my GF620 to passthrough, might be related to I915.

Both with the Radeon and the GeForce, it reboots without any problem.

Offline

#1575 2014-04-21 08:58:29

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone know if VFIO can be used with SR-IOV/VNICs? Have an Intel 1000ET dual card and it looks like Intel Virtual Function NICs might be easier to setup in Arch than a bridge.

Offline

#1576 2014-04-21 10:16:39

PunisherHD
Member
Registered: 2014-04-21
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I'm trying to passthrough my Nvidia GTX 760 card to Windows 7 KVM guest.
I got through all steps in the first post and now I have Windows 7 booting and displaying on my monitor, but nvidia driver says "error code 43".
Everything else works fine (USB, Sata drives ...), only the Nvidia driver has problem.

I'm using ArchLinux, kernel 3.14 from OP, Qemu 2.0.0 from "testing" repository.

HW setup:
ASUS H81I-Plus
Core i5 4670
Geforce GTX 760

Qemu command line:

qemu-system-x86_64 \
  -enable-kvm \
  -M q35 \
  -m 4096 \
  -cpu host \
  -smp 2,sockets=1,cores=2,threads=1 \
  -bios /usr/share/qemu/bios.bin \
  -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
  -device ahci,bus=pcie.0,id=ahci \
  -drive file=/dev/sda,id=disk1,format=raw -device ide-hd,bus=ahci.0,drive=disk1 \
  -drive file=/dev/sdc,id=disk2,format=raw -device ide-hd,bus=ahci.1,drive=disk2 \
  -device vfio-pci,host=00:14.0,bus=pcie.0 \
  -device vfio-pci,host=00:1a.0,bus=pcie.0

I tried to rebuild Qemu 2.0.0 adding this patch: http://lists.nongnu.org/archive/html/qe … 00597.html but no luck, still code 43.

Any hint to solve this mysterious code 43 ?

Offline

#1577 2014-04-21 10:35:08

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PunisherHD wrote:

Hello,
I'm trying to passthrough my Nvidia GTX 760 card to Windows 7 KVM guest.
I got through all steps in the first post and now I have Windows 7 booting and displaying on my monitor, but nvidia driver says "error code 43".
Everything else works fine (USB, Sata drives ...), only the Nvidia driver has problem.

I'm using ArchLinux, kernel 3.14 from OP, Qemu 2.0.0 from "testing" repository.

HW setup:
ASUS H81I-Plus
Core i5 4670
Geforce GTX 760

Qemu command line:

qemu-system-x86_64 \
  -enable-kvm \
  -M q35 \
  -m 4096 \
  -cpu host \
  -smp 2,sockets=1,cores=2,threads=1 \
  -bios /usr/share/qemu/bios.bin \
  -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
  -device ahci,bus=pcie.0,id=ahci \
  -drive file=/dev/sda,id=disk1,format=raw -device ide-hd,bus=ahci.0,drive=disk1 \
  -drive file=/dev/sdc,id=disk2,format=raw -device ide-hd,bus=ahci.1,drive=disk2 \
  -device vfio-pci,host=00:14.0,bus=pcie.0 \
  -device vfio-pci,host=00:1a.0,bus=pcie.0

I tried to rebuild Qemu 2.0.0 adding this patch: http://lists.nongnu.org/archive/html/qe … 00597.html but no luck, still code 43.

Any hint to solve this mysterious code 43 ?


Have you tried these tweaks?

EDIT 2: Well, thanks to this post from MacCoaster, who had a similar configuration (Intel Haswell + Asus MB + R7770), I added the following to the kernel command line:




vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off

I don't know which of these got it working with the Radeon HD 7770, but it seems quite stable, I can even run the latest 3D Mark smoothly (BSOD with the NVIDIA, but I should try with the new kernel command line).

Offline

#1578 2014-04-21 10:51:38

spam
Member
Registered: 2013-06-15
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have been abble to make some tries, with an Ubuntu host I had hda-codec errors but gfx was working both with nouveau and nvidia default configuration. I tried to disable i915 module and any sound module yet I still got code 43 under windows. Do you believe I'll had better luck with an AMD card ?  Is there a way to do pci-assign with kvm and a GTX 660 ?

Btw, i did not test nohz=off.

My last attempt was with

$ cat /etc/modprobe.d/alsa-base.conf /etc/modprobe.d/blacklist-ath_pci.conf /etc/modprobe.d/blacklist.conf /etc/modprobe.d/blacklist-firewire.conf /etc/modprobe.d/blacklist-framebuffer.conf /etc/modprobe.d/blacklist-gpu.conf /etc/modprobe.d/blacklist-modem.conf /etc/modprobe.d/blacklist-oss.conf /etc/modprobe.d/blacklist-rare-network.conf /etc/modprobe.d/blacklist-watchdog.conf /etc/modprobe.d/dkms.conf /etc/modprobe.d/fbdev-blacklist.conf /etc/modprobe.d/iwlwifi.conf /etc/modprobe.d/mlx4.conf /etc/modprobe.d/modprobe.conf /etc/modprobe.d/pci-stub.conf /etc/modprobe.d/qemu-system-x86.conf /etc/modprobe.d/vfio_iommu_type1.conf /etc/modprobe.d/vmwgfx-fbdev.conf blacklist nouveau blacklist nvidia blacklist i915 blacklist snd blacklist snd_hda_intel blacklist snd_hda_codec_realtek blacklist snd_hda_codec_generic blacklist snd_hda_intel blacklist snd_hda_codec blacklist snd_hwdep blacklist snd_pcm blacklist snd_timer blacklist snd blacklist soundcore #options kvm ignore_msrs=1 options kvm_intel emulate_invalid_guest_state=0 options pci-stub ids=10de:11c0,10de:0e0b options vfio_iommu_type1 allow_unsafe_interrupts=1
$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=0f8e3d34-5e38-4276-969f-c64aaf7f3245 rw quiet intel_iommu=on pci-stub.ids=10de:11c0,10de:0e0b

Offline

#1579 2014-04-21 14:23:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Krobar wrote:

Anyone know if VFIO can be used with SR-IOV/VNICs? Have an Intel 1000ET dual card and it looks like Intel Virtual Function NICs might be easier to setup in Arch than a bridge.

Yes


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1580 2014-04-22 05:45:25

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, is there any kernel option to active ACS patch v2? Right now I applied the patch but it is not activated. My PCH ports are still stucked in a big group...

Offline

#1581 2014-04-22 08:27:32

shweng
Member
Registered: 2014-03-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello nbhs and all,
I have tried serval days to pass-through my Nvidia GT640 and AMD HD7750 GPU to Windows7 KVM guest but with no succeed.
I followed all steps in the first post and now I could pass-through GPU device to Windows 7 VM but still has the follow issues.
1.VM with AMD HD7750 GPU, win7 start up BSOD with 0x00000116 on atikmpag.sys
    same as ilya80 post at https://bbs.archlinux.org/viewtopic.php?pid=1351243,but the solution of change "clear_emulator_capabilities = 0" on qemu.conf take no effect.
2.VM with Nvidia GT640 GPU, both vm and host hangup when vm OS booting to the twinkling mircorsoft windows logo.
3.there is no display on Qemu's Window,need to connect pass-throughed GPU's DVI port to display VM desktop.
for issue 1 and 2,enter VM on "Safe Mode" will be OK, but without graphics driver thus no hardware acceleration.
could you please help me to check my issue.
Hardware:
CPU:Intel Core i5-3470
MB:Gigabyte Z77P-D3, BIOS F7 08/24/2012 (VT-d enabled in BIOS)
RAM:4GB
Host:Intel HD IGP
Guest nVidia GeForce GT 640 or AMD RADEON HD7750
 
On the software side, I'm running Fedora 20 with kernel 3.14.1 which apply the patchset of linux-mainline.tar.gz except kernel-38-gcc48-2.patch
fix_reading_of_DR6.patch and debug_registers.patch
The kernel was compiled with the following config: 
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

IOMMU is enabled through kernel switch.
disabled all the modules which would interfere with guest graphics cards (radeon, nouveau, nvidia,fglrx)
latest qemu:qemu-2.0.0-1(also tried NoSnoop patch from http://lists.nongnu.org/archive/html/qe … 00597.html but no effect)
latest seabios:seabios-1.7.4-5

GPU
[root@localhost home]# lspci |grep VGA
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750]
06:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GT 640] (rev a1)

kernel param
[root@localhost home]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.14.1 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root rhgb quiet intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 LANG=en_US.UTF-8

update initramfs
/etc/modprobe.d/
[root@localhost modprobe.d]# cat blacklist.conf
blacklist radeon
blacklist redeonhd
blacklist nvidia
blacklist nouveau
blacklist fglrx
[root@localhost modprobe.d]# cat vfio_iommu_type1.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

[root@localhost modprobe.d]#dracut -f -v /boot/initramfs-$(uname -r).img $(uname -r)

qemu command:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 1,sockets=1,cores=1,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-drive file=/home/images/nv.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk \
-drive file=/ISO/win7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -boot order=cd,menu=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1                                      \
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \
-usbdevice tablet
(host=01:00.0 for AMD gpu).
(host=06:00.0 for Nvidia gpu).

dmesg:
[root@localhost modprobe.d]# dmesg |grep IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] Intel-IOMMU: enabled
[    0.018151] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.018155] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.018226] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.783685] IOMMU 0 0xfed90000: using Queued invalidation
[    0.783686] IOMMU 1 0xfed91000: using Queued invalidation
[    0.783688] IOMMU: Setting RMRR:
[    0.783696] IOMMU: Setting identity map for device 0000:00:02.0 [0x9f800000 - 0xaf9fffff]
[    0.784850] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9d5c9000 - 0x9d5f3fff]
[    0.784865] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9d5c9000 - 0x9d5f3fff]
[    0.784878] IOMMU: Setting identity map for device 0000:00:14.0 [0x9d5c9000 - 0x9d5f3fff]
[    0.784887] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.784892] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

[root@localhost modprobe.d]# dmesg |grep DMAR
[    0.000000] ACPI: DMAR 000000009d775558 0000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.783661] DMAR: No ATSR found
[   24.261510] [drm] DMAR active, disabling use of stolen memory

Offline

#1582 2014-04-22 17:10:44

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@shweng
Don't know about the NVIDIA problem, you might want to try the qemu/seabios from git posted on page 1

However for the AMD problem

.VM with AMD HD7750 GPU, win7 start up BSOD with 0x00000116 on atikmpag.sys
    same as ilya80 post at https://bbs.archlinux.org/viewtopic.php?pid=1351243,but the solution of change "clear_emulator_capabilities = 0" on qemu.conf take no effect.

I would recommend (1) don't pass the audio part 1:00.1, (2) in safe mode uninstall the AMD drivers; then after that presumably you can boot windows with the generic VGA driver (and you won't have the atimpag.sys error because there won't be an ati driver).  (3) then reinstall only the graphics driver, not the CCC. When you install the drivers, there is an option to do that. You will still have accelerated graphics, just not the tuning available with the CCC.  Anyway, that worked for me with my HD7970.

Offline

#1583 2014-04-23 07:35:10

ohetfi
Member
Registered: 2014-04-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, nbhs and all!

I would like to share my configuration of my sort of wannabe "All-in-One" KVM VFIO on Arch Linux. Using all nbhs' ingredients (kernel 3.14), I have successfully passed through my GeForce GTX 770 and IBM M1015 to my KVM guests. Using this configuration I can also natively boot to both Linux Mint Debian Edition (for whole disk backup of Arch Linux disk) and to Windows 7 when I need the true bare metal performance (Metro: Last Light +/-4 FPS increase on Windows 7 native).

3DMark Results
Bare Metal: http://www.3dmark.com/fs/1945959
KVM Guest: http://www.3dmark.com/fs/2040597
Any idea on how to make my CPU recognized by 3DMark when on KVM Guest? big_smile

Hardware Specification
CPU: Intel Core i7-4770
Memory: 4 x 8 GB Patriot Signature Line Kit
Motherboard: ASRock Z87 Extreme6
GPU:
    - Intel HD Graphics 4600 (host)
    - Gigabyte GeForce GTX 770 Windforce (guest1)
    - Cirrus (guest2)
Disks:
    - SanDisk FDD 32 GB -> Arch Linux (host)
    - Hitachi HDD 500 GB -> Windows 7 (guest1)
    - Seagate HDD 160 GB -> Linux Mint Debian Edition (guest2)
    - 4 x Hitachi HDD 2 TB -> ZFS (guest2)
HBA: IBM M1015 RAID Card flashed with IT-mode (guest2)

Boot Parameters (/proc/cmdline)

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d6205fde-e945-4ded-b4e1-fe3bd98b1f84 rw quiet intel_iommu=on pcie_acs_override=downstream

I need to add the pcie_acs_override parameter to force PCIe devices to be on separate IOMMU groups, and of cource intel_iommu parameter to make all this magic come true. smile

Kernel Module Parameters (/etc/modprobe.d/*.conf)

# Blacklist Gigabyte GeForce GTX 770 Windforce blacklist nouveau blacklist nvidia # Bonding Modes: balance-rr, active-backup, balance-xor, broadcast, 802.3ad, balance-tlb, balance-alb options bonding mode=balance-alb miimon=100 updelay=200 # Needed to avoid crashing KVM guest when running Passmark Performance Test and SiSoftware Sandra options kvm ignore_msrs=1 # 01:00.0 10de:1184 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) # 01:00.1 10de:0e0a Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) # 02:00.0 1000:0072 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) options pci-stub ids=10de:1184,10de:0e0a,1000:0072 # Enable interrupt remapping options vfio_iommu_type1 allow_unsafe_interrupts=1

I bond 2 of my NICs and then setup a bridge device for all my guests.

VFIO Bind Configuration (/etc/vfio.cfg)

# 01:00.0 10de:1184 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) # 01:00.1 10de:0e0a Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) # 02:00.0 1000:0072 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) DEVICES="0000:01:00.0 0000:01:00.1 0000:02:00.0"

Hugepages Parameter (/etc/sysctl.d/40-vm.conf)

# Allocate 26 GB of RAM for Hugepages vm.nr_hugepages = 13312

The host memory is 32 GB, I allocated around 26 GB for all my KVM guests. 6 GB for the host.

Windows 7 Guest Parameters (kvm-windows7.sh)

#!/bin/bash # Use PulseAudio as the audio driver QEMU_PA_SAMPLES=128 export QEMU_AUDIO_DRV=pa # Startup Windows 7 guest sudo qemu-system-x86_64 \ -name Windows-7 \ -cpu host \ -smp 8,sockets=1,cores=4,threads=2 \ -enable-kvm \ -m 8G \ -mem-path /dev/hugepages \ -mem-prealloc \ -balloon none \ -rtc clock=host \ -vga none \ -serial null \ -parallel null \ -monitor none \ -display none \ -daemonize \ -soundhw hda \ -k en-us \ -machine type=q35,accel=kvm \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -boot order=c \ -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/dev/disk/by-id/ata-Hitachi_HDS721050CLA660_JP1570FR1ZWP7K \ -device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -net nic,model=virtio,macaddr=52:54:00:6b:03:05 -net bridge,br=br0 \ -usb \ -device ich9-usb-uhci3,id=uhci \ -device usb-ehci,id=ehci \ -usbdevice host:093a:2510 \ -usbdevice host:04d9:1603 \ -usbdevice host:046d:c21e \ -usbdevice host:0bc2:2300

I use host clock (UTC), since I sometimes accidentally boot the Windows 7 natively and wanted to avoid my hardware clock confusion. So far no need to eject or no issues on rebooting the guest, thank God. big_smile

Linux Mint Debian Edition Guest Parameters (kvm-lmde.sh)

#!/bin/bash # Use PulseAudio as the audio driver export QEMU_PA_SAMPLES=128 export QEMU_AUDIO_DRV=pa # Starts up Linux Mint Debian Edition guest sudo qemu-system-x86_64 \ -name LMDE \ -cpu host \ -smp 8,sockets=1,cores=4,threads=2 \ -enable-kvm \ -m 8G \ -mem-path /dev/hugepages \ -mem-prealloc \ -balloon none \ -rtc clock=host \ -vga std \ -vnc localhost:1 \ -daemonize \ -soundhw hda \ -k en-us \ -machine type=q35,accel=kvm \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -boot order=c \ -drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/dev/disk/by-id/ata-ST3160815AS_5RA7CPYR \ -device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,romfile=/opt/sas2008/x64sas2.rom \ -net nic,model=virtio,macaddr=52:54:00:3b:cf:09 -net bridge,br=br0 \ -usb \ -device ich9-usb-uhci3,id=uhci \ -device usb-ehci,id=ehci

My ZFS storage server. smile

THANKS A LOT, Guys!

Offline

#1584 2014-04-23 07:59:59

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ohetfi wrote:

Hi, nbhs and all!

I would like to share my configuration of my sort of wannabe "All-in-One" KVM VFIO on Arch Linux. Using all nbhs' ingredients (kernel 3.14), I have successfully passed through my GeForce GTX 770 and IBM M1015 to my KVM guests. Using this configuration I can also natively boot to both Linux Mint Debian Edition (for whole disk backup of Arch Linux disk) and to Windows 7 when I need the true bare metal performance (Metro: Last Light +/-4 FPS increase on Windows 7 native).

THANKS A LOT, Guys!

You will also get some network speed improvement using vhost=on.

Eg.:

-netdev tap,vhost=on,ifname=$VM,script=$KVM_ROOT/scripts/vm_ifup_brlan,id=brlan -device virtio-net-pci,mac=52:54:00:00:00:21,netdev=brlan \

where vm_ifup_brlan is :

#!/bin/sh BRIDGE=brlan echo "Executing $0" echo "Bringing up $1 for bridged mode..." ip link set $1 up promisc on echo "Disabling STP for bridge $BRIDGE" brctl stp $BRIDGE off echo "Adding $1 to $BRIDGE ..." brctl addif $BRIDGE $1 sleep 2

Offline

#1585 2014-04-23 08:26:44

ohetfi
Member
Registered: 2014-04-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:
ohetfi wrote:

Hi, nbhs and all!

I would like to share my configuration of my sort of wannabe "All-in-One" KVM VFIO on Arch Linux. Using all nbhs' ingredients (kernel 3.14), I have successfully passed through my GeForce GTX 770 and IBM M1015 to my KVM guests. Using this configuration I can also natively boot to both Linux Mint Debian Edition (for whole disk backup of Arch Linux disk) and to Windows 7 when I need the true bare metal performance (Metro: Last Light +/-4 FPS increase on Windows 7 native).

THANKS A LOT, Guys!

You will also get some network speed improvement using vhost=on.

Eg.:

-netdev tap,vhost=on,ifname=$VM,script=$KVM_ROOT/scripts/vm_ifup_brlan,id=brlan -device virtio-net-pci,mac=52:54:00:00:00:21,netdev=brlan \

where vm_ifup_brlan is :

#!/bin/sh BRIDGE=brlan echo "Executing $0" echo "Bringing up $1 for bridged mode..." ip link set $1 up promisc on echo "Disabling STP for bridge $BRIDGE" brctl stp $BRIDGE off echo "Adding $1 to $BRIDGE ..." brctl addif $BRIDGE $1 sleep 2

Hi, anickname!

Thanks for pointing that, I was having difficulties on applying the new QEMU network parameters thus ended up using the ones that I have mentioned in my post. Will definitely try that. In my case, my bridge device is called br0, so I assume your brlan is my br0?

I also found this slide http://www.linux-kvm.org/wiki/images/c/ … 008_11.pdf, probably will try to enable/disable the kvm_intel module parameters and see if it would increase my benchmarks: vpid, flexpriority, and ept.

Thanks!

Offline

#1586 2014-04-25 11:28:40

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

I am trying to follow guide http://fds-team.de/cms/articles/2013-12 … yer-u.html , but faced with issue:

A disk read error occurred Press Ctrl+Alt+Del to restart

I think is due to different types of booting: I use UEFI (and GPT) during real booting, but VM uses seabios. Could you please point me to right direction: what should I do to boot windows in VM?
BTW, I have installed Windows on one partition (recovery is also placed on the C drive, I have no 200MB hide partition)

Offline

#1587 2014-04-26 00:00:19

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone successfully passed through a discrete GPU on notebook with nvidia optimus technology? I have already spent a whole day on this, but all the time Windows reports code 10 (Device cannot start) in Device Manager.

I use vanilla kernel 3.14.1, qemu 2.0 and seabios 1.7.3.2. Do you have any ideas?

Offline

#1588 2014-04-26 00:00:34

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A disk read error occurred

The message has been fixed, but I faced another:

missing operating system

Even if I run Windows installation from VM - installation process can not perform first reboot

Original drive:

Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors Units: sectors of 1 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_6.html topic_7.html topic_8.html topic_9.html 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: E0A01D64-268D-4491-88AB-B4498B9F4600 Device Start End Size Type /dev/sdb1 8192 1952571391 931.1G Microsoft basic data /dev/sdb2 1952571392 3905134591 931.1G Microsoft basic data /dev/sdb3 3905134592 5857697791 931.1G Microsoft basic data /dev/sdb4 5857697792 5860532223 1.4G Microsoft basic data

sdb2 - windows partition

Creation procedures:

dd if=/dev/zero of=boot.mbr count=0 seek=1952571392 sudo losetup /dev/loop0 boot.mgr sudo mdadm --build /dev/md0 --level=linear --raid-devices=2 /dev/loop0 /dev/sdb2
fdisk -l /dev/md0 Disk /dev/md0: 1.8 TiB, 1999428911104 bytes, 3905134592 sectors Units: sectors of 1 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_6.html topic_7.html topic_8.html topic_9.html 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0xf057e88f Device Boot Start End Blocks Id System /dev/md0p1 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_6.html topic_7.html topic_8.html topic_9.html 1952571392 3905134591 976281600 7 HPFS/NTFS/exFAT

Could you please help me to fix this issue.
Thanks in advance!

Last edited by myweb (2014-04-26 00:03:01)

Offline

#1589 2014-04-26 13:43:26

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:

Has anyone successfully passed through a discrete GPU on notebook with nvidia optimus technology? I have already spent a whole day on this, but all the time Windows reports code 10 (Device cannot start) in Device Manager.

I use vanilla kernel 3.14.1, qemu 2.0 and seabios 1.7.3.2. Do you have any ideas?


Now i got Error 43, like others using nvidia cards. Is there any way to fix this?

Offline

#1590 2014-04-26 16:57:03

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was able to get it work on Fedora rawhide(with qemu 2.0).
It works without patches but with corrupted host graphics.

There are first time it works I think because of new fedora rawhide(qemu 2.0 may be)

The guest windows show black screen with while lines and host crashed before(qemu 1.6).

My config
MB: MSI h67ma-e45
CPU Intel i5-2400
host GPU HD2000
guest GPU HD7750 (sapphire 1gb ddr5)

Last edited by zzz3000 (2014-04-27 13:00:05)

Offline

#1591 2014-04-27 11:10:42

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@zzz3000
Can you tell, what you mean with corrupted graphics?

Offline

#1592 2014-04-27 13:03:05

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

@zzz3000
Can you tell, what you mean with corrupted graphics?

Host image become spoiled(I was able to solve this host problem with i915 patch before but guest did not work)
Now with qemu 2.0 quest works fine I think  kernel patching will solve host image problem

Offline

#1593 2014-04-27 16:40:36

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zzz3000 wrote:
apex8 wrote:

@zzz3000
Can you tell, what you mean with corrupted graphics?

Host image become spoiled(I was able to solve this host problem with i915 patch before but guest did not work)
Now with qemu 2.0 quest works fine I think  kernel patching will solve host image problem

Ok Thanks. So that means, I still need the i915 patch, despite using intel exclusively on the host and nvidia exclusively in the VM.

Offline

#1594 2014-04-28 07:48:45

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Ok Thanks. So that means, I still need the i915 patch, despite using intel exclusively on the host and nvidia exclusively in the VM.

It is true in my case with fedora rawhide (kernel 3.15 qemu 2.0)

Offline

#1595 2014-04-28 08:24:45

ohetfi
Member
Registered: 2014-04-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

...

You will also get some network speed improvement using vhost=on.

Eg.:

-netdev tap,vhost=on,ifname=$VM,script=$KVM_ROOT/scripts/vm_ifup_brlan,id=brlan -device virtio-net-pci,mac=52:54:00:00:00:21,netdev=brlan \

...

Hi, anickname! Your suggestion worked. When I ps -ef | grep vhost, I can see the vhost processes on my host Arch Linux. Don't know if it's only my feeling or not, but I can now download Steam games at my full Internet bandwidth speed and very much stable just like when I passthrough one of my Intel NIC to a VM. Before using vhost, I experienced the bandwidth speed were very fluctuate and almost never full speed. Thanks!

myweb wrote:

...
The message has been fixed, but I faced another:
...
Could you please help me to fix this issue.
Thanks in advance!

Don't know if my experience will help you in any way, but I have different disk for each VM. My Arch Linux host resides in a 32 GB flash disk, which is very sufficient because I want my Arch Linux lean and simple, in addition I will be periodically do full offline disk backup, so 32 GB is enough for me. The flash disk is GPT, and I use EFI since my motherboard supports it and Arch Linux Beginner's Guide recommend it to use EFI/GPT for new installation if possible.

My Windows VM resides in a separate dedicated 500 GB hard disk, which I supply by giving a direct access to /dev/sdb (for instance). So the Windows VM have full access to the whole disk, but still via Arch Linux host drivers. I then install the Windows VM from Seabios and follow the installation as usual. After installing drivers etc inside the VM, I tried rebooting to the 500 GB disk (native reboot not VM reboot). And to my surprise, the Windows installation still runs, although needs to reinstall drivers for most part of the hardware. I didn't setup the grub to probe for the Windows partition, I switch manually to different boot drive via EFI/BIOS. So I think if you have spare disk, it would be easier to setup. Hope it helps.

Offline

#1596 2014-04-28 10:25:33

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying this once again. I'm using the nouveau drivers with a GTX 550-Ti this time around and then I plan to passthrough a GTX 670.

So far, I installed qemu-git from AUR. I added pci-stub to the module section of mkinitcpio.conf and then ran the mkinitcpio command, now here's where I get stuck.

nbhs wrote:

Now add this to your grub cfg file:

pci-stub.ids=1002:6719,1002:aa80

those instructions aren't specific enough for me; which grub cfg file? the one in /etc/default/ or the one in /boot/grub? and more importantly, where in that file?

Then there's this step that's confusing me as well

nbhs wrote:

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

I've got a board that supports interrupt remapping I think, but i'm also using the default arch kernel so vfio-pci is built as a module, the part that's confusing me here is that in the former example we have:

vfio_iommu_type1.allow_unsafe_interrupts=1

but in the latter we have:

options vfio_iommu_type1 allow_unsafe_interrupts=1

The part that's confusing me is that one has a period between "type1" and "allow" and the other has a space, shouldn't both have a period? I.e. shouldn't it be:

options vfio_iommu_type1.allow_unsafe_interrupts=1

Sorry for being a noob tongue

Last edited by rabcor (2014-04-28 10:30:01)

Offline

#1597 2014-04-28 14:09:24

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I'm trying this once again. I'm using the nouveau drivers with a GTX 550-Ti this time around and then I plan to passthrough a GTX 670.

So far, I installed qemu-git from AUR. I added pci-stub to the module section of mkinitcpio.conf and then ran the mkinitcpio command, now here's where I get stuck.

nbhs wrote:

Now add this to your grub cfg file:

pci-stub.ids=1002:6719,1002:aa80

those instructions aren't specific enough for me; which grub cfg file? the one in /etc/default/ or the one in /boot/grub? and more importantly, where in that file?

Then there's this step that's confusing me as well

nbhs wrote:

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

I've got a board that supports interrupt remapping I think, but i'm also using the default arch kernel so vfio-pci is built as a module, the part that's confusing me here is that in the former example we have:

vfio_iommu_type1.allow_unsafe_interrupts=1

but in the latter we have:

options vfio_iommu_type1 allow_unsafe_interrupts=1

The part that's confusing me is that one has a period between "type1" and "allow" and the other has a space, shouldn't both have a period? I.e. shouldn't it be:

options vfio_iommu_type1.allow_unsafe_interrupts=1

Sorry for being a noob tongue

The one with period is kernel parameter in grub.cfg (or /etc/default/grub), the other one put inside the file (with .conf extension) in /etc/modprobe.d/. Then add modconf hook to HOOKS in mkinicpio.conf. For best result choose latter method

Offline

#1598 2014-04-28 19:15:34

tswindell
Member
Registered: 2014-04-10
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, when passing through the HD Audio and Video card of my nVidia 680GTX, under windows Audio works great, 3D is a bit buggy. But when using Ubuntu as the guest, with both audio and video passed through, the 3D accel works great, better than under Windows, but Audio doesn't work at all hmm Get a weird feedback loop of some kind.

Has anyone managed to get an nVidia card's audio parts to work using VFIO passthrough to a Linux guest?

Offline

#1599 2014-04-28 20:32:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tswindell wrote:

So, when passing through the HD Audio and Video card of my nVidia 680GTX, under windows Audio works great, 3D is a bit buggy. But when using Ubuntu as the guest, with both audio and video passed through, the 3D accel works great, better than under Windows, but Audio doesn't work at all hmm Get a weird feedback loop of some kind.

Has anyone managed to get an nVidia card's audio parts to work using VFIO passthrough to a Linux guest?

You might try enabling MSI on the snd_hda_intel driver.  You can do this on the kernel command line with snd_hda_intel.enable_msi=1 or via modprobe.d with "options snd_hda_intel enable_msi=1".  Nvidia apparently has a poor track record with MSI on the audio card as there's code specifically to disable it on Nvidia devices, but they also have a poor track record of making device that properly re-assert INTx if we mask it using the PCI 2.3 spec defined disable bit.  MSI is more efficient in a virt environment anyway, so it it works, it's better to use it.  I haven't had any trouble with audio on a GTX660, but on a Quadro K4000 (which uses the same device ID for the audio function), it's a complete fail.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1600 2014-04-28 21:13:16

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks @dwe11er!

dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet pci-stub.ids=10de:1189,10de:0e0a [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet pci-stub.ids=10de:1189,10de:0e0a [ 0.625505] pci-stub: add 10DE:1189 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.625513] pci-stub 0000:02:00.0: claimed by stub [ 0.625518] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.625522] pci-stub 0000:02:00.1: claimed by stub

Now I've reached the qemu testing stage.

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \ -smp 6,sockets=1,cores=4,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Results in

WARNING: failed to find q35-acpi-dsdt.aml qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What did I do wrong? I googled around and found this. That suggests I should copy the file to /usr/share/qemu/ but copy it from where?

Edit: I found the file here.

but I still get

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

how do I fix that? yikes

Digging deeper:

$journalctl -b | grep vfio systemd[1]: Starting Binds devices to vfio-pci... apr 28 21:38:20 Necembel kernel: vfio-pci: probe of 0000:02:00.0 failed with error -22 apr 28 21:38:20 Necembel kernel: vfio-pci: probe of 0000:02:00.0 failed with error -22 apr 28 21:38:20 Necembel kernel: vfio-pci: probe of 0000:02:00.1 failed with error -22

"ls /sys/bus/pci/devices/0000\:02\:00.0/" also lists no "iommu_group" link.

Last edited by rabcor (2014-04-28 21:51:33)

Offline

#1601 2014-04-28 21:36:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

how do I fix that? yikes

Enable the IOMMU!


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1602 2014-04-28 21:53:09

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
rabcor wrote:

how do I fix that? yikes

Enable the IOMMU!

Like this?

Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet intel_iommu=1 pci-stub.ids=10de:1189,10de:0e0a

That isn't enough it would seem.
What part did I miss?

Last edited by rabcor (2014-04-28 21:56:07)

Offline

#1603 2014-04-28 22:19:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
aw wrote:
rabcor wrote:

how do I fix that? yikes

Enable the IOMMU!

Like this?

Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet intel_iommu=1 pci-stub.ids=10de:1189,10de:0e0a

That isn't enough it would seem.
What part did I miss?

intel_iommu=on and make sure it's enabled in your BIOS and supported by your processor (ark.intel.com)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1604 2014-04-28 22:27:22

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks! it was the "intel_iommu=on" part instead of 1 smile however:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 5 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 5 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I read a post (by you) saying I need the ACS override patch. I installed the linux mainline kernel from the OP which should have this patch, but still no luck. I am also using the nouveau drivers as is rather than the proprietary drivers. So it's not that I need to patch my gpu drivers I would think.

Last edited by rabcor (2014-04-28 23:13:09)

Offline

#1605 2014-04-29 00:53:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Thanks! it was the "intel_iommu=on" part instead of 1 smile however:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 5 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 5 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I read a post (by you) saying I need the ACS override patch. I installed the linux mainline kernel from the OP which should have this patch, but still no luck. I am also using the nouveau drivers as is rather than the proprietary drivers. So it's not that I need to patch my gpu drivers I would think.

And notice how the patch in that link requires kernel command line options to enable:

--- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2349,6 +2349,16 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nomsi Do not use MSI for native PCIe PME signaling (this makes all PCIe root ports use INTx for all services). + pcie_acs_override = + [PCIE] Override missing PCIe ACS support for: + downstream + All downstream ports - full ACS capabilties + multifunction + All multifunction devices - multifunction ACS subset + id:nnnn:nnnn + Specfic device - full ACS capabilities + Specified as vid:did (vendor/device ID) in hex +

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1606 2014-04-29 01:17:51

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ah, so essentially add "pcie_acs_override=downstream" to command line?  this seems to have worked, thanks smile

Qemu now opens as a blank window. However, the passthroughed card is giving no signals. (Monitor says "No Video Input")

Offline

#1607 2014-04-29 01:19:03

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

Is it possible to fix:

... Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3 Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000 DMAR:[fault reason 02] Present bit in context entry is clear Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3 Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000 DMAR:[fault reason 02] Present bit in context entry is clear ...

The root of the issue is DVB card

04:02.0 Multimedia controller [0480]: Philips Semiconductors SAA7146 [1131:7146] (rev 01)

PCI tree:

-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller +-01.0-[01]--+-00.0 NVIDIA Corporation GK107 [GeForce GT 640] | \-00.1 NVIDIA Corporation GK107 HDMI Audio Controller +-02.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller +-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 +-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 +-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller +-1c.0-[02]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] +-1c.4-[03-04]----00.0-[04]----02.0 Philips Semiconductors SAA7146

The messages aboves occurs if DVB card is passed to VM via:

-device vfio-pci,host=03:00.0,bus=pcie.0 \ -device vfio-pci,host=04:02.0,bus=pcie.0 \

If try to use DVB card on the host - host freezes.
Could you please help me to find a way how to use a dvb card (in VM or in Host - it does not matter)
Thank you in advance.

Offline

#1608 2014-04-29 01:21:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear All,

Is it possible to fix:

... Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3 Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000 DMAR:[fault reason 02] Present bit in context entry is clear Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3 Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000 DMAR:[fault reason 02] Present bit in context entry is clear ...

The root of the issue is DVB card

04:02.0 Multimedia controller [0480]: Philips Semiconductors SAA7146 [1131:7146] (rev 01)

PCI tree:

-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller +-01.0-[01]--+-00.0 NVIDIA Corporation GK107 [GeForce GT 640] | \-00.1 NVIDIA Corporation GK107 HDMI Audio Controller +-02.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller +-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 +-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 +-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller +-1c.0-[02]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] +-1c.4-[03-04]----00.0-[04]----02.0 Philips Semiconductors SAA7146

The messages aboves occurs if DVB card is passed to VM via:

-device vfio-pci,host=03:00.0,bus=pcie.0 \ -device vfio-pci,host=04:02.0,bus=pcie.0 \

If try to use DVB card on the host - host freezes.
Could you please help me to find a way how to use a dvb card (in VM or in Host - it does not matter)
Thank you in advance.

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

EDIT: Oh, you're passing 3:00.0 to the guest, never pass the bridge, this shouldn't even be allowed

Last edited by aw (2014-04-29 01:22:49)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1609 2014-04-29 01:47:40

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=03, secondary=04, subordinate=04, sec-latency=32 I/O behind bridge: 0000f000-00000fff Memory behind bridge: f7900000-f79fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard

Offline

#1610 2014-04-29 03:21:38

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm seeing someone had a similar problem to my own with unpatched nvidia drivers, I want to take a shot at patching the nvidia drivers instead of using nouveau, but I have no idea how to actually apply the patch to the drivers, can somebody point me in the right direction please?

Last edited by rabcor (2014-04-29 03:28:18)

Offline

#1611 2014-04-29 03:34:39

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I'm seeing someone had a similar problem to my own with unpatched nvidia drivers, I want to take a shot at patching the nvidia drivers instead of using nouveau, but I have no idea how to actually apply the patch to the drivers, can somebody point me in the right direction please?

I think I have a old gist somewhere with a pkgbuild… lets see. Here you go!.


i'm sorry for my poor english wirting skills…

Offline

#1612 2014-04-29 04:29:11

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, download nvidia-dkms.install from here, and update pkgver to 337.12 and md5sum to 'c610d5bff8f7f758dc25fdd51b1b65eb' right? Thanks smile would be nice to have something similar setup for the kernel ACS patch, too bad I don't know how to set that up.

Last edited by rabcor (2014-04-29 04:30:53)

Offline

#1613 2014-04-29 12:56:01

adejour
Member
Registered: 2014-04-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Any Idea?
have you find a solution gneville?

My last tried configuration is :

MB : ASUS H87M-PLUS
GPU : Radeon R7 260X
RAM : 8Go
CPU : i5 4570
Qemu : http://wiki.qemu-project.org/download/q … .0.tar.bz2
Kernel : https://www.kernel.org/pub/linux/kernel … 4.1.tar.xz
patch -Np1 -i override_for_missing_acs_capabilities.patch
patch -Np1 -i i915_314.patch

/usr/local/bin/qemu-system-x86_64 \
-enable-kvm -M q35 -m 4G -cpu host \
-balloon none  -rtc clock=host  -serial null  -parallel null  -monitor none  -display none \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-drive file=/home/ja/VM/windows2.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-net nic -net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

gneville wrote:

I've just found something interesting with my setup.

I can get around the BSOD on Windows boot after installing the drivers, providing I do a fresh reload of the Host. I also had to remove the HDMI Audio from the ioh3420 device.

Edit: Confirmed this is the same for Win7 & Win8 as a Guest

But provided I reboot the Host everytime I boot the Windows Guest with the below qemu commands it will start up and use the AMD drivers correctly, if I then reboot the Guest or Shutdown and restart the Qemu again from Command line then Windows will BSOD on boot with the "PAGE_FAULT_IN_NONPAGED_AREA" error.

Any ideas what can be causing this please?

/usr/local/qemu-xen-4-4-git-patched/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/xen/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio



gneville wrote:

Still not having much luck with vfio vga passthrough on my AMD R9 290 sad

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.


gneville wrote:

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad

Offline

#1614 2014-04-29 21:22:29

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.



adejour wrote:

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Any Idea?
have you find a solution gneville?

My last tried configuration is :

MB : ASUS H87M-PLUS
GPU : Radeon R7 260X
RAM : 8Go
CPU : i5 4570
Qemu : http://wiki.qemu-project.org/download/q … .0.tar.bz2
Kernel : https://www.kernel.org/pub/linux/kernel … 4.1.tar.xz
patch -Np1 -i override_for_missing_acs_capabilities.patch
patch -Np1 -i i915_314.patch

/usr/local/bin/qemu-system-x86_64 \
-enable-kvm -M q35 -m 4G -cpu host \
-balloon none  -rtc clock=host  -serial null  -parallel null  -monitor none  -display none \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-drive file=/home/ja/VM/windows2.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-net nic -net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

gneville wrote:

I've just found something interesting with my setup.

I can get around the BSOD on Windows boot after installing the drivers, providing I do a fresh reload of the Host. I also had to remove the HDMI Audio from the ioh3420 device.

Edit: Confirmed this is the same for Win7 & Win8 as a Guest

But provided I reboot the Host everytime I boot the Windows Guest with the below qemu commands it will start up and use the AMD drivers correctly, if I then reboot the Guest or Shutdown and restart the Qemu again from Command line then Windows will BSOD on boot with the "PAGE_FAULT_IN_NONPAGED_AREA" error.

Any ideas what can be causing this please?

/usr/local/qemu-xen-4-4-git-patched/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/xen/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio



gneville wrote:

Still not having much luck with vfio vga passthrough on my AMD R9 290 sad

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.

Offline

#1615 2014-04-29 23:06:34

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=03, secondary=04, subordinate=04, sec-latency=32 I/O behind bridge: 0000f000-00000fff Memory behind bridge: f7900000-f79fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard

Alex,
Is it correct that ASMedia pci bridge is root cause of "DMAR:[fault reason 02] Present bit in context entry is clear" issue?
Is it correct that it is a known issue and there is no way to fix it and it's impossible to use any PCI devices while IOMMU is turned on?

Thank you!

Offline

#1616 2014-04-30 10:02:02

adejour
Member
Registered: 2014-04-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

adejour wrote:

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

Last edited by adejour (2014-04-30 10:02:21)

Offline

#1617 2014-04-30 13:07:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
myweb wrote:
aw wrote:

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=03, secondary=04, subordinate=04, sec-latency=32 I/O behind bridge: 0000f000-00000fff Memory behind bridge: f7900000-f79fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard

Alex,
Is it correct that ASMedia pci bridge is root cause of "DMAR:[fault reason 02] Present bit in context entry is clear" issue?
Is it correct that it is a known issue and there is no way to fix it and it's impossible to use any PCI devices while IOMMU is turned on?

Thank you!

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1618 2014-04-30 13:43:36

adejour
Member
Registered: 2014-04-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Gneville, this guy had exactly the same pb reported to a qemu developper : https://lists.gnu.org/archive/html/qemu … 00905.html

Patch the vfio.c file was the solution for him. He has the R9 290X and I'm using the R7 260X. I'll try.

I looked on the code, and it is may be linked to the Power Management Flag NoSoftRst (when you run lspci -vvv).
Can somebody who succeed in multiple reboots tell we what is the result of this command for him?

this part :

Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Thanks!

adejour wrote:

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

gneville wrote:

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

adejour wrote:

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Offline

#1619 2014-04-30 14:41:41

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

adejour, I did spot that a while back and tried it but it didn't fix the BSOD issue when you reboot a guest or shutdown the guest and power it back on. I do have it applied to my current version of Qemu I'm using.

From memory however, I believe it does kill the video output when the guest is killed so it may fix your issue there.

edit: FWIW this is my output of Power Management from "lspci -vvv" from my R290, with the guest having the card passed through running:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO (prog-if 00 [VGA controller])
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Cheers

adejour wrote:

Gneville, this guy had exactly the same pb reported to a qemu developper : https://lists.gnu.org/archive/html/qemu … 00905.html

Patch the vfio.c file was the solution for him. He has the R9 290X and I'm using the R7 260X. I'll try.

I looked on the code, and it is may be linked to the Power Management Flag NoSoftRst (when you run lspci -vvv).
Can somebody who succeed in multiple reboots tell we what is the result of this command for him?

this part :

Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Thanks!

Last edited by gneville (2014-04-30 14:47:38)

Offline

#1620 2014-05-01 10:28:48

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey together,

there`s something that bothers me and I hope somebody can give me an explaination for it:
My mainboard is a Asrock Z68 Pro3 Gen3.

lspci | grep USB ⏎ 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) 07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

As you can see I have two usb 2.0 controllers. Initially I planned using a KVM switch to switch my keyboard/mouse between my VM an the host. So I added 00:1a.0 to pci-stub and binded it to vfio-pci, but I was still able to use the related ports on the host whenever I clicked the button on the KVM switch. Of course passthrough of 00:1a.0 to the VM didn't work either. Keyboard and mouse simply was usable on the host AND the VM (using parameter vga=std)

So I would like to know, if there is a reason why this controller behaves in this strange way. In the end, are these two controllers maybe the same device?
Anyway, I found Synergy being totally sufficient, but that issue still bugs me smile

BR
apex

Offline

#1621 2014-05-01 10:53:49

maduraa
Member
Registered: 2014-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I followed the instructions with a stock kernel (3.14.1). When I run qemu(for testing using the commands given in the first post) I get no output from the passed through vga (nVidia 750GTX). But I do get a blank screen on qemu on the main display, I do not own another monitor so I ran the command and plugged the cable to the nVidia card to check whether it has the BIOS output as mentioned but that port is not on the monitor goes to standby when I plug it. I am using IGP as my primary VGA.

# ls -l /sys/bus/pci/drivers/vfio-pci/ total 0 lrwxrwxrwx 1 root root 0 May 1 15:31 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/ lrwxrwxrwx 1 root root 0 May 1 15:31 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1/ --w------- 1 root root 4096 May 1 15:31 bind lrwxrwxrwx 1 root root 0 May 1 15:31 module -> ../../../../module/vfio_pci/ --w------- 1 root root 4096 May 1 14:54 new_id --w------- 1 root root 4096 May 1 15:31 remove_id --w------- 1 root root 4096 May 1 14:54 uevent --w------- 1 root root 4096 May 1 15:31 unbind

I installed Windows 7 on a VM and Windows is able to detect the card, I installed the drivers and then it got disabled by Windows(it says the devive didn't function as expected so it's disabled (code:43))

What can be the problem?

Thanks

Last edited by maduraa (2014-05-01 10:54:25)

Offline

#1622 2014-05-01 11:34:08

maduraa
Member
Registered: 2014-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

spam wrote:

I've been trying adding acs to my config but still had no luck my guest works only as a basic video card, I get a code 43 in windows. I am gonna try a linux guest to get traces. When I try to load a vbios with the romfile option my guest has no outputs at all. Is someone has an idea of what else could I try ? My gtx 660 is not UEFI compatible and I start my computer in UEFI mode, could it be a problem ?

Are you using IGP and nvidia? I saw in an earlier post that the stock kernel wouldn't work this way, I'm currently building the linux-mainline kernel will be back with updates. I get the same result with stock kernel see above post by me.

EDIT: I also tried the given kernel, seabios-git, qemu-git I get nothing on the passthru'd VGA's output
EDIT2: I also tried with another monitor, I can confirm that the passthru'd card doesn't display anything.

Last edited by maduraa (2014-05-01 14:22:50)

Offline

#1623 2014-05-01 11:57:31

adejour
Member
Registered: 2014-04-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I had the same problem.

If you have xHCI enabled in you BIOS, this controler is used for both usb 3 and usb 2 ports.
If you disable xHCI / USB3 in the BIOS, you can send one controler as you did before,
else you have to send the USB 3 conroler to the VM

I think you can also send the "device" with "-usb" as explained in the first post of this thread.

apex8 wrote:

Hey together,

there`s something that bothers me and I hope somebody can give me an explaination for it:
My mainboard is a Asrock Z68 Pro3 Gen3.

lspci | grep USB ⏎ 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) 07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

As you can see I have two usb 2.0 controllers. Initially I planned using a KVM switch to switch my keyboard/mouse between my VM an the host. So I added 00:1a.0 to pci-stub and binded it to vfio-pci, but I was still able to use the related ports on the host whenever I clicked the button on the KVM switch. Of course passthrough of 00:1a.0 to the VM didn't work either. Keyboard and mouse simply was usable on the host AND the VM (using parameter vga=std)

So I would like to know, if there is a reason why this controller behaves in this strange way. In the end, are these two controllers maybe the same device?
Anyway, I found Synergy being totally sufficient, but that issue still bugs me smile

BR
apex

Offline

#1624 2014-05-01 20:26:18

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.

This is amazing news!
Could you please specify where are you going to upload/write info about fix (twitter, github, ... - how to follow news)?

Dear All,
Could you please help to setup VM via libvirt? I have created the following xml:

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit X or other application using the libvirt API. --> <domain type='kvm'> <name>X</name> <uuid>9efa8a71-b193-c50b-c770-9ce6dad02390</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/md1'/> <target dev='sda' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='52:54:00:4b:be:6d'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1a.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1b.0,bus=pcie.0'/> </qemu:commandline> </domain>

VM started, but there is no VGA.

I use libvirt-git.

Offline

#1625 2014-05-01 20:53:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.

This is amazing news!
Could you please specify where are you going to upload/write info about fix (twitter, github, ... - how to follow news)?

I posted the patches today, here's a link to the header with a description:

http://www.spinics.net/lists/linux-pci/msg30941.html

For your convenience, I've also pushed a branch out to github that is 3.15-rc3 + these changes, you'll need to add whatever other patches (i915/acs) on top:

git://github.com/awilliam/linux-vfio.git

branch: dma-alias

Let me know how it goes.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1626 2014-05-01 21:07:31

dismuter
Member
Registered: 2013-12-27
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nex7 wrote:

So I've discovered what I believe to be my problem, but I'm confused why nobody else seems to be hitting it.

https://lists.gnu.org/archive/html/qemu … 01858.html

That's what's hitting me - I'm getting c0000221 stop's on boot on my VM, at this point, and seemingly this is due to virtio-scsi. I wonder why nobody else is hitting this?

EDIT: Of course, starting over with a fresh Win7 x64 install using ide instead of virtio, it works fine up until I try to install Windows updates, even only a few of them, at which point after reboot it dies with a different BSOD, code c000021a. -sigh-


Sorry for the OT here, but I've also been hitting this issue. I'm not able to install or boot Windows 8.1 with virtio-scsi and qemu 2.0.
Just to let others know that the issue is known:
https://lists.gnu.org/archive/html/qemu … 02280.html

Offline

#1627 2014-05-01 21:29:43

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
myweb wrote:
aw wrote:

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.

This is amazing news!
Could you please specify where are you going to upload/write info about fix (twitter, github, ... - how to follow news)?

I posted the patches today, here's a link to the header with a description:

http://www.spinics.net/lists/linux-pci/msg30941.html

For your convenience, I've also pushed a branch out to github that is 3.15-rc3 + these changes, you'll need to add whatever other patches (i915/acs) on top:

git://github.com/awilliam/linux-vfio.git

branch: dma-alias

Let me know how it goes.

Alex, thank you!
I have specified:

source=("git://github.com/awilliam/linux-vfio.git#branch=dma-alias"

in PKGBUILD, I am not sure that is right. Could you please specify how to check that sourced are exactly from dma-alias branch?

Offline

#1628 2014-05-01 21:33:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:
myweb wrote:

This is amazing news!
Could you please specify where are you going to upload/write info about fix (twitter, github, ... - how to follow news)?

I posted the patches today, here's a link to the header with a description:

http://www.spinics.net/lists/linux-pci/msg30941.html

For your convenience, I've also pushed a branch out to github that is 3.15-rc3 + these changes, you'll need to add whatever other patches (i915/acs) on top:

git://github.com/awilliam/linux-vfio.git

branch: dma-alias

Let me know how it goes.

Alex, thank you!
I have specified:

source=("git://github.com/awilliam/linux-vfio.git#branch=dma-alias"

in PKGBUILD, I am not sure that is right. Could you please specify how to check that sourced are exactly from dma-alias branch?

I don't use archlinux, so I have no idea what that does.  If you have the build tree available to examine, the last patch removes the function named pci_get_dma_source from drivers/pci/quirks.c.  If that matches what you find in the tree, you probably have the correct code.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1629 2014-05-01 22:04:29

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can't apply ACS patch:

src/linux-vfio/drivers/pci/quirks.c.rej --- drivers/pci/quirks.c 2013-10-19 19:28:15.000000000 +0000 +++ drivers/pci/quirks.c 2013-10-23 18:02:18.000000000 +0000 @@ -3517,6 +3619,7 @@ { PCI_VENDOR_ID_ATI, 0x439d, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs }, + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } };

Offline

#1630 2014-05-01 22:12:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Can't apply ACS patch:

src/linux-vfio/drivers/pci/quirks.c.rej --- drivers/pci/quirks.c 2013-10-19 19:28:15.000000000 +0000 +++ drivers/pci/quirks.c 2013-10-23 18:02:18.000000000 +0000 @@ -3517,6 +3619,7 @@ { PCI_VENDOR_ID_ATI, 0x439d, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs }, + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } };

Sooo... add it


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1631 2014-05-01 22:14:18

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, modified patch to:

 /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_6.html topic_7.html topic_8.html topic_9.html AMD has indicated that the devices below do not support peer-to-peer 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_6.html topic_7.html topic_8.html topic_9.html in any system where they are found in the southbridge with an AMD @@ -3404,6 +3505,7 @@ static int pci_quirk_amd_sb_acs(struct p #endif } + static const struct pci_dev_acs_enabled { u16 vendor; u16 device; @@ -3415,6 +3517,7 @@ static const struct pci_dev_acs_enabled { PCI_VENDOR_ID_ATI, 0x439d, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_intel_pch_acs }, + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } };

added line

 { PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_intel_pch_acs },

Offline

#1632 2014-05-01 23:54:00

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strange, but no any changes: still Dmar error.

Offline

#1633 2014-05-02 00:40:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Strange, but no any changes: still Dmar error.

Is it the same DMAR error, on device 04:00.0?  I've got a new processor coming tomorrow that will let me do VT-d with that bridge, until then I don't really know whether to handle it as a PCIe-to-PCI bridge or a PCI-to-PCI bridge.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1634 2014-05-02 07:52:48

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
myweb wrote:

Strange, but no any changes: still Dmar error.

Is it the same DMAR error, on device 04:00.0?  I've got a new processor coming tomorrow that will let me do VT-d with that bridge, until then I don't really know whether to handle it as a PCIe-to-PCI bridge or a PCI-to-PCI bridge.

Yes, it is the same error:

DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear dmar: DRHD: handling fault status reg 2 dmar: DRHD: handling fault status reg 3

Last edited by myweb (2014-05-02 07:54:04)

Offline

#1635 2014-05-02 14:48:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:
myweb wrote:

Strange, but no any changes: still Dmar error.

Is it the same DMAR error, on device 04:00.0?  I've got a new processor coming tomorrow that will let me do VT-d with that bridge, until then I don't really know whether to handle it as a PCIe-to-PCI bridge or a PCI-to-PCI bridge.

Yes, it is the same error:

DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear dmar: DRHD: handling fault status reg 2 dmar: DRHD: handling fault status reg 3

Does this help?

https://github.com/awilliam/linux-vfio/ … b2dc4db04c

I've pushed it to the dma-alias-test branch for you to try.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1636 2014-05-02 15:16:29

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
myweb wrote:
aw wrote:

Is it the same DMAR error, on device 04:00.0?  I've got a new processor coming tomorrow that will let me do VT-d with that bridge, until then I don't really know whether to handle it as a PCIe-to-PCI bridge or a PCI-to-PCI bridge.

Yes, it is the same error:

DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear dmar: DRHD: handling fault status reg 2 dmar: DRHD: handling fault status reg 3

Does this help?

https://github.com/awilliam/linux-vfio/ … b2dc4db04c

I've pushed it to the dma-alias-test branch for you to try.

Thank you!
I will recompile kernel and try.
I will let you know results in hour or two.

Offline

#1637 2014-05-02 16:03:09

ProFfeSsoRr
Member
Registered: 2009-11-30
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anybody used vfio with libvirt and virt-manager? I create VM in virt-manager, it starts but in commandline of qemu proccess a can't find "magick" option "-M q35", virt-manager or libvirt not support this or I do something wrong? I need to passthrough my second GPU GTX750Ti to Win8.1 VM, and i've got 43 error now.
P.S. I need virt-manager or something GUI tool because this is for my parents smile

Last edited by ProFfeSsoRr (2014-05-02 16:16:08)

Offline

#1638 2014-05-02 17:02:49

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does this help?

https://github.com/awilliam/linux-vfio/ … b2dc4db04c

I've pushed it to the dma-alias-test branch for you to try.

May 02 23:00:46 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear May 02 23:00:46 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3

No, it does not help.

Offline

#1639 2014-05-02 20:27:42

brainkiller
Member
Registered: 2014-05-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

All,

Let me first take the time to thank you all for the detailed information and patches.

My setup looks like this:
Asus Z87-A
Intel I7 4770
Asus GTX 760

OS:
Vanilla Kernel 3.14.1 + mainline patches (from main post)
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-4)

Guest:
windows 8.1 + latest patches
Nvidia driver 9.18.13.2619

All KVM modules are build into the kernel. No modules are used for KVM.
Boot options:
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.14.1-kvm root=/dev/mapper/vg00-debian--root ro pci-stub.ids=10de:1187,10de:0e0a intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1

Qemu (Still WIP as I'm quite new to KVM, I used to be a XEN user):
qemu-system-x86_64 \
        -name Windows-8 \
        -cpu host \
        -smp 4,sockets=1,cores=2,threads=2 \
        -enable-kvm \
        -m 4G \
        -boot menu=on \
        -balloon none \
        -serial null \
        -parallel null \
        -soundhw hda \
        -k fr-be \
        -machine type=q35,accel=kvm \
        -boot order=cd \
        -device ahci,id=ahci \
        -device virtio-scsi-pci,id=scsi \
        -drive id=win8,format=raw,file=/dev/vg00/win8 -device ide-hd,bus=ide.0,drive=win8 \
        -drive id=win7,format=raw,file=/dev/vg00/win7-x64 -device ide-hd,bus=ide.1,drive=win7 \
        -drive file=/home/brainkiller/Downloads/virtio-win-0.1-74.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd \
        -net nic,macaddr=00:16:3E:51:20:4C,model=virtio -net tap \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
        -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
        -vga none \
        -monitor none \
        -display none \


Initially I was blocked but this turned out to solved by upgrading qemu to 2.0.0
Thanks again for all advice, I just wanted to share another success story and to indicate another working setup

Regards,
Brainkiller

Offline

#1640 2014-05-02 22:00:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:

Does this help?

https://github.com/awilliam/linux-vfio/ … b2dc4db04c

I've pushed it to the dma-alias-test branch for you to try.

May 02 23:00:46 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear May 02 23:00:46 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3

No, it does not help.

Hmm, I got my VT-d CPU installed in the system with same bridge and can confirm that the patch in the dma-alias-test branch is definitely needed for the ASM1083 bridge.  Without it, a conventional PCI NIC installed in the system streams IOMMU faults.  So, I'm not sure what's happening.  The fault address you're getting does look suspicious, being exactly the last page before the 32bit, 4G barrier.  Is that the only fault address you get?  There are devices out there that are incompatible with IOMMUs, either the driver doesn't use the DMA API correctly or the hardware makes stray accesses.  Sometimes the only solution for those devices is to run the system with iommu=pt to put them into passthrough mode.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1641 2014-05-02 23:34:01

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
myweb wrote:
aw wrote:

Does this help?

https://github.com/awilliam/linux-vfio/ … b2dc4db04c

I've pushed it to the dma-alias-test branch for you to try.

May 02 23:00:46 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear May 02 23:00:46 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3

No, it does not help.

Hmm, I got my VT-d CPU installed in the system with same bridge and can confirm that the patch in the dma-alias-test branch is definitely needed for the ASM1083 bridge.  Without it, a conventional PCI NIC installed in the system streams IOMMU faults.  So, I'm not sure what's happening.  The fault address you're getting does look suspicious, being exactly the last page before the 32bit, 4G barrier.  Is that the only fault address you get?  There are devices out there that are incompatible with IOMMUs, either the driver doesn't use the DMA API correctly or the hardware makes stray accesses.  Sometimes the only solution for those devices is to run the system with iommu=pt to put them into passthrough mode.

I am terribly sorry, I used old branch dma-alias instead of dma-alias-test.
I compiled a dma-alias-test and my DVB card works in host system (I can't check in VM since I have trouble to boot it)

But I experience some issue with USB controller (00:1a.0)

dmar: DRHD: handling fault status reg 3 DMAR:[DMA Read] Request device [00:1a.0] fault addr ee000 DMAR:[fault reason 06] PTE Read access is not set dmar: DRHD: handling fault status reg 3 dmar: DMAR:[DMA Read] Request device [00:1a.0] fault addr ee000 DMAR:[fault reason 06] PTE Read access is not set

Offline

#1642 2014-05-03 01:52:30

Sphaerophoria
Member
Registered: 2011-07-14
Posts: 41

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ProFfeSsoRr wrote:

Anybody used vfio with libvirt and virt-manager? I create VM in virt-manager, it starts but in commandline of qemu proccess a can't find "magick" option "-M q35", virt-manager or libvirt not support this or I do something wrong? I need to passthrough my second GPU GTX750Ti to Win8.1 VM, and i've got 43 error now.
P.S. I need virt-manager or something GUI tool because this is for my parents smile


I just got it working the other day, it's actually quite simple.

Set up libvirt as you normally would and remove the display server/display driver.

Then do a virsh edit, alter the first line to look like this

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

and then add a pci device like you would with regular qemu

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis =1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on ,x-vga=on> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

Last edited by Sphaerophoria (2014-05-03 01:53:07)

Offline

#1643 2014-05-03 05:41:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:
myweb wrote:
May 02 23:00:46 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr fffff000 DMAR:[fault reason 02] Present bit in context entry is clear May 02 23:00:46 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3

No, it does not help.

Hmm, I got my VT-d CPU installed in the system with same bridge and can confirm that the patch in the dma-alias-test branch is definitely needed for the ASM1083 bridge.  Without it, a conventional PCI NIC installed in the system streams IOMMU faults.  So, I'm not sure what's happening.  The fault address you're getting does look suspicious, being exactly the last page before the 32bit, 4G barrier.  Is that the only fault address you get?  There are devices out there that are incompatible with IOMMUs, either the driver doesn't use the DMA API correctly or the hardware makes stray accesses.  Sometimes the only solution for those devices is to run the system with iommu=pt to put them into passthrough mode.

I am terribly sorry, I used old branch dma-alias instead of dma-alias-test.
I compiled a dma-alias-test and my DVB card works in host system (I can't check in VM since I have trouble to boot it)

But I experience some issue with USB controller (00:1a.0)

dmar: DRHD: handling fault status reg 3 DMAR:[DMA Read] Request device [00:1a.0] fault addr ee000 DMAR:[fault reason 06] PTE Read access is not set dmar: DRHD: handling fault status reg 3 dmar: DMAR:[DMA Read] Request device [00:1a.0] fault addr ee000 DMAR:[fault reason 06] PTE Read access is not set

These patches should not have changed anything with respect to this device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1644 2014-05-03 09:45:03

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could you please help to fix:

 sudo virsh -c qemu:///system start X error: Failed to start domain X error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe2de4e6f70, 0x0, 0x80000000, 0x7fe1c4000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe2de4e6f70, 0x100000000, 0x80000000, 0x7fe244000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.

Dmesg

[ 3684.367018] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 3684.367036] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 3684.367362] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 3684.367377] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

VM XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>X</name> <uuid>9efa8a71-b193-c50b-c770-9ce6dad02390</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/md1'/> <target dev='sda' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='52:54:00:4b:be:6d'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1b.0,bus=root.1'/> </qemu:commandline> </domain>

It would be nice to use <hostdev type='pci' managed='yes'>      <driver name='vfio'/> since I use libvirt git, but it is unclear how to set "x-vga=on".

Last edited by myweb (2014-05-03 09:50:34)

Offline

#1645 2014-05-03 10:00:07

easd
Member
Registered: 2013-08-15
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm totally new about KVM, I read in the first post in the requirements: "At least 2 GPU's, one primary boot device and the card you wish to pass-through."
If I have an intel cpu (i7 3770) with an iGPU and a nvidia card (GTX 780) , I have to physically connect the monitor to the iGPU?
In that way I can't use the nvidia card in linux (for example for native gaming), am I missing something?

Last edited by easd (2014-05-03 10:00:31)

Offline

#1646 2014-05-04 13:31:36

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have managed to pass through my GeForce GTX 770M into KVM.
The GPU showed Error 10 for the very beginning and after I have installed latest ForceWare drivers, it showed error 43.
However if i disable and re-enable the card in Device Manager it showed that the device is working correctly.
Unfortunately as a second device and I was unable to boot Windows with nvidia card set as primary due to error 43, which i come across every time i reboot the VM.
All the time i need to disable and re-enable the dGPU in order to make Windows marking it as working.


I am not sure if this is connected with nVidia Optimus, or just GTX 770M. I have found that some of you managed to get GF 7xx working, while others not.
Also there is no such problem with ATI cards. Due to that i think that more people having notebooks should give it a try, especially with older GeForce cards and with mobile ATI cards.

As the prove I want to show you a screenshot:

snapshot2_1_png_750x750_q85.jpg

Last edited by belliash (2014-05-04 13:47:55)

Offline

#1647 2014-05-04 13:42:49

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:

I have managed to pass through my GeForce GTX 770M into KVM.
The GPU showed Error 10 for the very beginning and after I have installed latest ForceWare drivers, it showed error 43.
However if i disable and re-enable the card in Device Manager it showed that the device is working correctly.
Unfortunately as a second device and I was unable to boot Windows with nvidia card set as primary due to error 43, which i come across every time i reboot the VM.
All the time i need to disable and re-enable the dGPU in order to make Windows marking it as working.


I am not sure if this is connected with nVidia Optimus, or just GTX 770M. I have found that some of you managed to get GF 7xx working, while others not.
Also there is no such problem with ATI cards. Due to that i think that more people having notebooks should give it a try, especially with older GeForce cards and with mobile ATI cards.

As the prove I want to show you a screenshot:

Thumbnail and stuff… https://wiki.archlinux.org/index.php/Fo … s_and_Code


i'm sorry for my poor english wirting skills…

Offline

#1648 2014-05-05 00:59:05

svirpridon
Member
Registered: 2014-05-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to get this working on a Ubuntu 14.04 system, host on onboard 4770 intel graphics, guest on a 7870. I grabbed kernel 3.14.1 mainline from the kernel PPA but I'm still getting the host "taste the rainbow" color change when I try to boot the test guest. I take it the intel vga fix isn't yet upstreamed? Or is it in 3.15rcX and I should try one of the RC kernels?

Offline

#1649 2014-05-05 01:18:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

svirpridon wrote:

I'm trying to get this working on a Ubuntu 14.04 system, host on onboard 4770 intel graphics, guest on a 7870. I grabbed kernel 3.14.1 mainline from the kernel PPA but I'm still getting the host "taste the rainbow" color change when I try to boot the test guest. I take it the intel vga fix isn't yet upstreamed? Or is it in 3.15rcX and I should try one of the RC kernels?

https://bbs.archlinux.org/viewtopic.php … 2#p1400212


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1650 2014-05-05 02:41:20

svirpridon
Member
Registered: 2014-05-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That was it, thanks aw. Can you link that off your first post? (or is it and I just missed it? I read the first post a couple times)

Maybe I'll go back to the secondary vga as you referenced there. That's how I'm operating now, my only problem with 14.04 (and 13.10) was that my current config (non-vfio style) didn't work anymore.

Last edited by svirpridon (2014-05-05 02:41:41)

Offline

#1651 2014-05-05 11:02:01

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Val532 wrote:
aw wrote:

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.

Ok so for the moment i have to use the old patch not the new because of probably not a safe patch.

Ok ^^.

Are your planning to make an adition to the new patch ? That add the processor-based root ports a not safe patch but just for people like me with multiple VGA passthrough ^^.

No, I proposed the ACS override patch upstream and it was rejected for fear that it once again places a dangerous tool in the arsenal that users are quick to jump on for functionality without fully understanding the implications (as evidenced by this forum wink).  The kernel will only accept quirks that the vendors identify as safe.  I expect that as a result of this, we'll see a resurgence of ACS support in Intel chipsets, but getting confirmation and code in place for existing devices is a tedious process.


So then, ACS support in intel chipsets is in? (GVT-d)

Last edited by rabcor (2014-05-05 11:05:10)

Offline

#1652 2014-05-05 12:20:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
aw wrote:
Val532 wrote:

Ok so for the moment i have to use the old patch not the new because of probably not a safe patch.

Ok ^^.

Are your planning to make an adition to the new patch ? That add the processor-based root ports a not safe patch but just for people like me with multiple VGA passthrough ^^.

No, I proposed the ACS override patch upstream and it was rejected for fear that it once again places a dangerous tool in the arsenal that users are quick to jump on for functionality without fully understanding the implications (as evidenced by this forum wink).  The kernel will only accept quirks that the vendors identify as safe.  I expect that as a result of this, we'll see a resurgence of ACS support in Intel chipsets, but getting confirmation and code in place for existing devices is a tedious process.


So then, ACS support in intel chipsets is in? (GVT-d)

Unrelated


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1653 2014-05-05 16:40:52

maduraa
Member
Registered: 2014-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I saw on an online forum that some motherboards have the PCI slots behind a switch and they do not support passthrough, is this true?

-[0000:00]-+-00.0 +-01.0-[01]--+-00.0 | \-00.1 +-02.0 +-03.0 +-14.0 +-16.0 +-1a.0 +-1b.0 +-1c.0-[02]-- +-1c.3-[03]----00.0 +-1c.7-[04]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3

Is there any thing wrong in the about lspci -t output that'd disqualify my board? 01:00.0, 01:00.1 are for VGA and VGA audio

Last edited by maduraa (2014-05-05 16:43:30)

Offline

#1654 2014-05-05 17:01:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

maduraa wrote:

I saw on an online forum that some motherboards have the PCI slots behind a switch and they do not support passthrough, is this true?

-[0000:00]-+-00.0 +-01.0-[01]--+-00.0 | \-00.1 +-02.0 +-03.0 +-14.0 +-16.0 +-1a.0 +-1b.0 +-1c.0-[02]-- +-1c.3-[03]----00.0 +-1c.7-[04]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3

Is there any thing wrong in the about lspci -t output that'd disqualify my board? 01:00.0, 01:00.1 are for VGA and VGA audio

a) PCIe switches don't prevent device assigned, b) you don't have one.  PCIe-to-PCI bridges mean that all the devices behind the bridge may need to be assigned together, which may include multiple slots... you don't have one of those either.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1655 2014-05-06 00:48:14

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi

I am not archlinux user but this post is so great, many thanks.

Unfortunately my virtual machine is not working as I expected.

My setup:
Slackware 14.1
Kernel 3.14.2 (with vfio as module and vfio-vga enabled)
qemu 1.6.2
seabios 0.6.1
Motherboard Gygabyte z77x-ud5h
Micro Intel i5 3330
1st Video: Builtin Intel
2nd Video: MSI Nvidia 7600

Using two patch for the kernel and one for the qemu.

Kernel:
1-Patch_with_quirk_for_incorrect_PCIe_requester_IDs-SATA_Marvell

@@ -, +, @@ --- as 'ghost' functions. --- a/drivers/iommu/amd_iommu_init.c +++ a/drivers/iommu/amd_iommu_init.c @@ -1023,6 +1023,62 @@ static void __init free_iommu_all(void) } } +static void quirk_map_multi_requester_ids(struct pci_dev *pdev) +{ + int fn; + u16 devid = PCI_DEVID(pdev->bus->number, + pdev->devfn); + u8 fn_map = pci_multi_requesters(pdev); + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is the common case; a non-quirky device. */ + if (!fn_map) + return; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Skip the known, already mapped func */ + fn_map &= ~(1<<PCI_FUNC(pdev->devfn)); + + for (fn = 0; fn_map >> fn; fn++) { + if (fn_map & (1<<fn)) { + u8 quirk_devfn = PCI_DEVFN(PCI_SLOT(devid), fn); + u16 quirk_devid = PCI_DEVID(pdev->bus->number, + quirk_devfn); + amd_iommu_alias_table[devid] = quirk_devid; + set_dev_entry_from_acpi(amd_iommu_rlookup_table[devid], + quirk_devid, 0, 0); + dev_dbg(&pdev->dev, + "requester id quirk; ghost func %d mapped", fn); + } + } +} + +static void quirk_map_requester_id(struct pci_dev *pdev) +{ + u8 quirk_devfn = pci_requester(pdev); + u16 devid = PCI_DEVID(pdev->bus->number, pdev->devfn); + u16 quirk_devid = PCI_DEVID(pdev->bus->number, quirk_devfn); + + dev_dbg(&pdev->dev, "checking for incorrect pci requester id quirk.."); + + if (pdev->devfn == quirk_devfn) + return; + + amd_iommu_alias_table[devid] = quirk_devid; + set_dev_entry_from_acpi(amd_iommu_rlookup_table[devid], + quirk_devid, 0, 0); +} + +static void amd_iommu_pci_quirks(void) +{ + struct pci_dev *pdev = NULL; + + for_each_pci_dev(pdev) { + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Check for devices that use multiple requester IDs */ + quirk_map_multi_requester_ids(pdev); + + quirk_map_requester_id(pdev); + } +} + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Family15h Model 10h-1fh erratum 746 (IOMMU Logging May Stall Translations) 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Workaround: @@ -1336,6 +1392,8 @@ static int __init amd_iommu_init_pci(void) break; } + amd_iommu_pci_quirks(); + ret = amd_iommu_init_devices(); print_iommu_info(); --- a/drivers/iommu/intel-iommu.c +++ a/drivers/iommu/intel-iommu.c @@ -1649,6 +1649,108 @@ static int domain_context_mapping_one(struct dmar_domain *domain, int segment, return 0; } +static void iommu_detach_dev(struct intel_iommu *iommu, u8 bus, u8 devfn); + +static void quirk_unmap_multi_requesters(struct pci_dev *pdev, u8 fn_map) +{ + int fn; + struct intel_iommu *iommu = device_to_iommu(pci_domain_nr(pdev->bus), + pdev->bus->number, pdev->devfn); + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Skip the known, already mapped func */ + fn_map &= ~(1<<PCI_FUNC(pdev->devfn)); + + for (fn = 0; fn_map >> fn; fn++) { + if (fn_map & (1<<fn)) { + iommu_detach_dev(iommu, + pdev->bus->number, + PCI_DEVFN(PCI_SLOT(pdev->devfn), fn)); + dev_dbg(&pdev->dev, + "requester id quirk; ghost func %d unmapped", + fn); + } + } +} + +/* For quirky devices that use multiple requester ids. */ +static int quirk_map_multi_requester_ids(struct dmar_domain *domain, + struct pci_dev *pdev, + int translation) +{ + int fn, err = 0; + u8 fn_map = pci_multi_requesters(pdev); + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is the common case; a non-quirky device. */ + if (!fn_map) + return 0; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Skip the known, already mapped func */ + fn_map &= ~(1<<PCI_FUNC(pdev->devfn)); + + for (fn = 0; fn_map >> fn; fn++) { + if (fn_map & (1<<fn)) { + err = domain_context_mapping_one(domain, + pci_domain_nr(pdev->bus), + pdev->bus->number, + PCI_DEVFN(PCI_SLOT(pdev->devfn), fn), + translation); + if (err) { + dev_err(&pdev->dev, + "mapping ghost func %d failed", fn); + quirk_unmap_multi_requesters(pdev, + fn_map & ((1<<fn)-1)); + return err; + } + dev_dbg(&pdev->dev, + "requester id quirk; ghost func %d mapped", fn); + } + } + return 0; +} + +static void quirk_unmap_requester_id(struct pci_dev *pdev) +{ + u8 devfn = pci_requester(pdev); + struct intel_iommu *iommu = device_to_iommu(pci_domain_nr(pdev->bus), + pdev->bus->number, pdev->devfn); + + if (pdev->devfn == devfn) + return; + + iommu_detach_dev(iommu, pdev->bus->number, devfn); + dev_dbg(&pdev->dev, "requester id quirk; bugged device unmapped"); +} + +static int quirk_map_requester_id(struct dmar_domain *domain, + struct pci_dev *pdev, + int translation) +{ + u8 devfn = pci_requester(pdev); + int err; + + dev_dbg(&pdev->dev, + "checking for incorrect pci requester id quirk..."); + + if (pdev->devfn == devfn) + return 0; + + err = domain_context_mapping_one(domain, + pci_domain_nr(pdev->bus), + pdev->bus->number, + devfn, + translation); + if (err) { + dev_err(&pdev->dev, + "requester id quirk: mapping dev %02x:%02x.%d failed", + pdev->bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn)); + return err; + } + dev_dbg(&pdev->dev, + "requester id quirk; dmar context entry added: %02x:%02x.%d", + pdev->bus->number, PCI_SLOT(devfn), PCI_FUNC(devfn)); + return 0; +} + static int domain_context_mapping(struct dmar_domain *domain, struct pci_dev *pdev, int translation) @@ -1662,6 +1764,16 @@ domain_context_mapping(struct dmar_domain *domain, struct pci_dev *pdev, if (ret) return ret; + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup quirk for devices using multiple pci requester ids */ + ret = quirk_map_multi_requester_ids(domain, pdev, translation); + if (ret) + return ret; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup quirk for devices using single incorrect pci requester id */ + ret = quirk_map_requester_id(domain, pdev, translation); + if (ret) + return ret; + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup dependent device mapping */ tmp = pci_find_upstream_pcie_bridge(pdev); if (!tmp) @@ -3766,6 +3878,9 @@ static void domain_remove_one_dev_info(struct dmar_domain *domain, iommu_disable_dev_iotlb(info); iommu_detach_dev(iommu, info->bus, info->devfn); iommu_detach_dependent_devices(iommu, pdev); + quirk_unmap_multi_requesters(pdev, + pci_multi_requesters(pdev)); + quirk_unmap_requester_id(pdev); free_devinfo_mem(info); spin_lock_irqsave(&device_domain_lock, flags); --- a/drivers/pci/quirks.c +++ a/drivers/pci/quirks.c @@ -3336,6 +3336,10 @@ static struct pci_dev *pci_func_0_dma_source(struct pci_dev *dev) return pci_get_slot(dev->bus, PCI_DEVFN(PCI_SLOT(dev->devfn), 0)); } +/* Table of source functions for real devices. The DMA requests for the + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html device are tagged with a different real function as source. This is + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html relevant to multifunction devices. + */ static const struct pci_dev_dma_source { u16 vendor; u16 device; @@ -3362,7 +3366,8 @@ static const struct pci_dev_dma_source { 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html the device doing the DMA, but sometimes hardware is broken and will 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html tag the DMA as being sourced from a different device. This function 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html allows that translation. Note that the reference count of the - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html returned device is incremented on all paths. + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html returned device is incremented on all paths. Translation is done when + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html the device is added to an IOMMU group. */ struct pci_dev *pci_get_dma_source(struct pci_dev *dev) { @@ -3423,6 +3428,148 @@ static int pci_quirk_amd_sb_acs(struct pci_dev *dev, u16 acs_flags) #endif } +/* Table of multiple requester IDs, aka ghost DMA source functions. + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Devices that may need this quirk show the following behaviour: + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html 1. the device may use multiple PCI requester IDs during operation, + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html (eg. one pci transaction uses xx:yy.0, the next uses xx:yy.1) + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html 2. the requester ID may point to an absent/ghost device. + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html (eg. lspci does not show xx:yy.1 to be present) + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html The bitmap contains ALL of the functions used as requester IDs by the + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html device, including the one known to 'lspci'. If the func is known to 'lspci', + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html but not used for DMA, don't include it. + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html See https://bugzilla.redhat.com/show_bug.cgi?id=757166, + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=42679 + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1089768 + */ +static const struct pci_dev_dma_multi_source_map { + u16 vendor; + u16 device; + u8 func_map; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup bit map. lsb is fn 0. */ +} pci_dev_dma_multi_source_map[] = { + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Patrick Bregman + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.redhat.com/show_bug.cgi?id=863653 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9120, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Paweł Żak, Korneliusz Jarzębski, Daniel Mayer + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=42679 and by + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Justin Piszcz https://lkml.org/lkml/2012/11/24/94 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9123, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Used in a patch by Ying Chu + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.redhat.com/show_bug.cgi?id=757166 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9125, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Robert Cicconetti + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=42679 and by + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Fernando https://bugzilla.redhat.com/show_bug.cgi?id=757166 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9128, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Stijn Tintel + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=42679 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9130, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Gaudenz Steinlin + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://lkml.org/lkml/2013/3/5/288 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9143, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Andrew Cooks + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=42679 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9172, (1<<0)|(1<<1)}, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Reported by Martin Öhrling + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=42679#c28 */ + { PCI_VENDOR_ID_MARVELL_EXT, 0x9230, (1<<0)|(1<<1)}, + + { 0 } +}; + +/* + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html The mapping of quirky requester ids is used when the device driver sets up + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html dma, if iommu is enabled. + */ +u8 pci_multi_requesters(struct pci_dev *dev) +{ + const struct pci_dev_dma_multi_source_map *i; + + for (i = pci_dev_dma_multi_source_map; i->func_map; i++) { + if ((i->vendor == dev->vendor || + i->vendor == (u16)PCI_ANY_ID) && + (i->device == dev->device || + i->device == (u16)PCI_ANY_ID)) { + return i->func_map; + } + } + return 0; +} + +/* These are one-to-one translations for devices that use a single incorrect + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html requester ID. The requester id may not be the BDF of a real device. + */ +static const struct pci_dev_dma_source_map { + u16 vendor; + u16 device; + u8 devfn; + u8 dma_devfn; +} pci_dev_dma_source_map[] = { + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Ricoh IEEE 1394 Controller */ + { + PCI_VENDOR_ID_RICOH, + 0xe832, + PCI_DEVFN(0x00, 3), + PCI_DEVFN(0x00, 0) + }, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Nils Caspar - Adaptec 3405 + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html http://www.mail-archive.com/centos@centos.org/msg90986.html + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Jonathan McCune + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html http://old-list-archives.xen.org/archives/html/xen-users/2010-04/msg00535.html */ + { + PCI_VENDOR_ID_ADAPTEC2, + 0x028b, + PCI_DEVFN(0x0e, 0), + PCI_DEVFN(0x01, 0) + }, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Mateusz Murawski - LSI SAS based MegaRAID + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://lkml.org/lkml/2011/9/12/104 + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html M. Nunberg - Dell PERC 5/i Integrated RAID Controller + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html http://lists.xen.org/archives/html/xen-devel/2010-05/msg01563.html */ + { + PCI_VENDOR_ID_LSI_LOGIC, + 0x0411, + PCI_DEVFN(0x0e, 0), + PCI_DEVFN(0x08, 0) + }, + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Steven Dake, Markus Stockhausen - Mellanox 26428 + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.redhat.com/show_bug.cgi?id=713774 + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Note: mellanox uses decimal product numbers, convert to hex for PCI + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html device ID. ie. 26428 == 0x673c */ + { + PCI_VENDOR_ID_MELLANOX, + 0x673c, + PCI_DEVFN(0x00, 0), + PCI_DEVFN(0x00, 6) + }, + + { 0 } +}; + +u8 pci_requester(struct pci_dev *dev) +{ + const struct pci_dev_dma_source_map *i; + + for (i = pci_dev_dma_source_map; i->vendor; i++) { + if ((i->vendor == dev->vendor) && + (i->device == dev->device) && + (i->devfn == dev->devfn)) { + return i->dma_devfn; + } + } + return dev->devfn; +} + static const struct pci_dev_acs_enabled { u16 vendor; u16 device; --- a/include/linux/pci.h +++ a/include/linux/pci.h @@ -1509,6 +1509,8 @@ enum pci_fixup_pass { #ifdef CONFIG_PCI_QUIRKS void pci_fixup_device(enum pci_fixup_pass pass, struct pci_dev *dev); struct pci_dev *pci_get_dma_source(struct pci_dev *dev); +u8 pci_multi_requesters(struct pci_dev *dev); +u8 pci_requester(struct pci_dev *dev); int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags); #else static inline void pci_fixup_device(enum pci_fixup_pass pass, @@ -1517,6 +1519,14 @@ static inline struct pci_dev *pci_get_dma_source(struct pci_dev *dev) { return pci_dev_get(dev); } +u8 pci_multi_requesters(struct pci_dev *dev) +{ + return 0; +} +u8 pci_requester(struct pci_dev *dev) +{ + return dev->devfn; +} static inline int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags) {

2-1- Add_PCI_walk_function_and_PCIe_bridge_test

diff --git a/drivers/pci/search.c b/drivers/pci/search.c index d0627fa..0357f74 100644 --- a/drivers/pci/search.c +++ b/drivers/pci/search.c @@ -17,6 +17,63 @@ DECLARE_RWSEM(pci_bus_sem); EXPORT_SYMBOL_GPL(pci_bus_sem); +/* Test for PCIe bridges. */ +bool pci_is_pcie_bridge(struct pci_dev *pdev) +{ + if (!pci_is_bridge(pdev)) + return false; + + if (pci_is_pcie(pdev)) + return true; + +#ifdef CONFIG_PCI_QUIRKS + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html If we're not on the root bus, look one device upstream of the + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html current device. If that device is PCIe and is not a PCIe-to-PCI + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html bridge, then the current device is effectively PCIe as it must + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html be the PCIe-to-PCI bridge. This handles several bridges that + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html violate the PCIe spec by not exposing a PCIe capability: + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html https://bugzilla.kernel.org/show_bug.cgi?id=44881 + */ + if (!pci_is_root_bus(pdev->bus)) { + struct pci_dev *parent = pdev->bus->self; + + if (pci_is_pcie(parent) && + pci_pcie_type(parent) != PCI_EXP_TYPE_PCI_BRIDGE) + + return true; + } +#endif + return false; +} + +/* + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Walk upstream from the given pdev for the first device returning + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html true for the provided match function. If no match is found, return + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html NULL. *last records the previous step in the walk. + */ +struct pci_dev *pci_walk_up_to_first_match(struct pci_dev *pdev, + bool (*match)(struct pci_dev *), + struct pci_dev **last) +{ + *last = NULL; + + if (match(pdev)) + return pdev; + + *last = pdev; + + while (!pci_is_root_bus(pdev->bus)) { + *last = pdev; + pdev = pdev->bus->self; + + if (match(pdev)) + return pdev; + } + + return NULL; +} + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html find the upstream PCIe-to-PCI bridge of a PCI device 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html if the device is PCIE, return NULL diff --git a/include/linux/pci.h b/include/linux/pci.h index bd8ec30..e87423a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1855,6 +1855,29 @@ static inline struct eeh_dev *pci_dev_to_eeh_dev(struct pci_dev *pdev) #endif /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html pci_walk_up_to_first_match - Generic upstream search function + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html @pdev: starting PCI device to search + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html @match: match function to call on each device (true = match) + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html @last: last device examined prior to returned device + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Walk upstream from the given device, calling match() at each device. + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Returns the first device matching match(). If the root bus is reached + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html without finding a match, return NULL. last returns the N-1 step in + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html the search path. + */ +struct pci_dev *pci_walk_up_to_first_match(struct pci_dev *pdev, + bool (*match)(struct pci_dev *), + struct pci_dev **last); + +/** + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html pci_is_pcie_bridge - Match a PCIe bridge device + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html @pdev: device to test + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Return true if the given device is a PCIe bridge, false otherwise. + */ +bool pci_is_pcie_bridge(struct pci_dev *pdev); + +/** 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html pci_find_upstream_pcie_bridge - find upstream PCIe-to-PCI bridge of a device 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html @pdev: the PCI device *

2-2-Add_PCI_walk_function_and_PCIe_bridge_test

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 0099667..277c18f 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -1688,9 +1688,13 @@ domain_context_mapping(struct dmar_domain *domain, struct pci_dev *pdev, return ret; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup dependent device mapping */ - tmp = pci_find_upstream_pcie_bridge(pdev); - if (!tmp) + if (pci_is_pcie(pdev) || pci_is_root_bus(pdev->bus)) return 0; + + tmp = pci_walk_up_to_first_match(pdev, pci_is_pcie_bridge, &parent); + if (!tmp) + tmp = parent; + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Secondary interface's bus number and devfn 0 */ parent = pdev->bus->self; while (parent != tmp) { @@ -1702,7 +1706,7 @@ domain_context_mapping(struct dmar_domain *domain, struct pci_dev *pdev, return ret; parent = parent->bus->self; } - if (pci_is_pcie(tmp)) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a PCIe-to-PCI bridge */ + if (pci_is_pcie_bridge(tmp)) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a PCIe-to-PCI bridge */ return domain_context_mapping_one(domain, pci_domain_nr(tmp->subordinate), tmp->subordinate->number, 0, @@ -1729,10 +1733,15 @@ static int domain_context_mapped(struct pci_dev *pdev) ret = device_context_mapped(iommu, pdev->bus->number, pdev->devfn); if (!ret) return ret; + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup dependent device mapping */ - tmp = pci_find_upstream_pcie_bridge(pdev); + if (pci_is_pcie(pdev) || pci_is_root_bus(pdev->bus)) + return 0; + + tmp = pci_walk_up_to_first_match(pdev, pci_is_pcie_bridge, &parent); if (!tmp) - return ret; + tmp = parent; + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Secondary interface's bus number and devfn 0 */ parent = pdev->bus->self; while (parent != tmp) { @@ -1742,7 +1751,7 @@ static int domain_context_mapped(struct pci_dev *pdev) return ret; parent = parent->bus->self; } - if (pci_is_pcie(tmp)) + if (pci_is_pcie_bridge(tmp)) return device_context_mapped(iommu, tmp->subordinate->number, 0); else @@ -1973,7 +1982,7 @@ static struct dmar_domain *get_domain_for_dev(struct pci_dev *pdev, int gaw) struct intel_iommu *iommu; struct dmar_drhd_unit *drhd; struct device_domain_info *info, *tmp; - struct pci_dev *dev_tmp; + struct pci_dev *dev_tmp = NULL; unsigned long flags; int bus = 0, devfn = 0; int segment; @@ -1985,9 +1994,15 @@ static struct dmar_domain *get_domain_for_dev(struct pci_dev *pdev, int gaw) segment = pci_domain_nr(pdev->bus); - dev_tmp = pci_find_upstream_pcie_bridge(pdev); - if (dev_tmp) { - if (pci_is_pcie(dev_tmp)) { + if (!pci_is_pcie(pdev) && !pci_is_root_bus(pdev->bus)) { + struct pci_dev *last; + + dev_tmp = pci_walk_up_to_first_match(pdev, pci_is_pcie_bridge, + &last); + if (!dev_tmp) + dev_tmp = last; + + if (pci_is_pcie_bridge(dev_tmp)) { bus = dev_tmp->subordinate->number; devfn = 0; } else { @@ -3738,26 +3753,24 @@ static void iommu_detach_dependent_devices(struct intel_iommu *iommu, { struct pci_dev *tmp, *parent; - if (!iommu || !pdev) + if (!iommu || !pdev || pci_is_pcie(pdev) || pci_is_root_bus(pdev->bus)) return; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup dependent device detach */ - tmp = pci_find_upstream_pcie_bridge(pdev); + tmp = pci_walk_up_to_first_match(pdev, pci_is_pcie_bridge, &parent); + if (!tmp) + tmp = parent; + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Secondary interface's bus number and devfn 0 */ - if (tmp) { - parent = pdev->bus->self; - while (parent != tmp) { - iommu_detach_dev(iommu, parent->bus->number, - parent->devfn); - parent = parent->bus->self; - } - if (pci_is_pcie(tmp)) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a PCIe-to-PCI bridge */ - iommu_detach_dev(iommu, - tmp->subordinate->number, 0); - else /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a legacy PCI bridge */ - iommu_detach_dev(iommu, tmp->bus->number, - tmp->devfn); + parent = pdev->bus->self; + while (parent != tmp) { + iommu_detach_dev(iommu, parent->bus->number, parent->devfn); + parent = parent->bus->self; } + if (pci_is_pcie_bridge(tmp)) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a PCIe-to-PCI bridge */ + iommu_detach_dev(iommu, tmp->subordinate->number, 0); + else /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a legacy PCI bridge */ + iommu_detach_dev(iommu, tmp->bus->number, tmp->devfn); } static void domain_remove_one_dev_info(struct dmar_domain *domain, @@ -4148,7 +4161,7 @@ static void swap_pci_ref(struct pci_dev **from, struct pci_dev *to) static int intel_iommu_add_device(struct device *dev) { struct pci_dev *pdev = to_pci_dev(dev); - struct pci_dev *bridge, *dma_pdev = NULL; + struct pci_dev *dma_pdev = NULL; struct iommu_group *group; int ret; @@ -4156,9 +4169,15 @@ static int intel_iommu_add_device(struct device *dev) pdev->bus->number, pdev->devfn)) return -ENODEV; - bridge = pci_find_upstream_pcie_bridge(pdev); - if (bridge) { - if (pci_is_pcie(bridge)) + if (!pci_is_pcie(pdev) && !pci_is_root_bus(pdev->bus)) { + struct pci_dev *bridge, *last; + + bridge = pci_walk_up_to_first_match(pdev, pci_is_pcie_bridge, + &last); + if (!bridge) + bridge = last; + + if (pci_is_pcie_bridge(bridge)) dma_pdev = pci_get_domain_bus_and_slot( pci_domain_nr(pdev->bus), bridge->subordinate->number, 0); diff --git a/drivers/iommu/intel_irq_remapping.c b/drivers/iommu/intel_irq_remapping.c index f3b8f23..873da76 100644 --- a/drivers/iommu/intel_irq_remapping.c +++ b/drivers/iommu/intel_irq_remapping.c @@ -373,7 +373,6 @@ static int set_hpet_sid(struct irte *irte, u8 id) static int set_msi_sid(struct irte *irte, struct pci_dev *dev) { - struct pci_dev *bridge; if (!irte || !dev) return -1; @@ -385,9 +384,15 @@ static int set_msi_sid(struct irte *irte, struct pci_dev *dev) return 0; } - bridge = pci_find_upstream_pcie_bridge(dev); - if (bridge) { - if (pci_is_pcie(bridge))/* this is a PCIe-to-PCI/PCIX bridge */ + if (!pci_is_pcie(dev) && !pci_is_root_bus(dev->bus)) { + struct pci_dev *bridge, *last; + + bridge = pci_walk_up_to_first_match(dev, + pci_is_pcie_bridge, &last); + if (!bridge) + bridge = last; + + if (pci_is_pcie_bridge(bridge)) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup PCIe-to-PCI/PCIX bridge */ set_irte_sid(irte, SVT_VERIFY_BUS, SQ_ALL_16, (bridge->bus->number << 8) | dev->bus->number); else /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup this is a legacy PCI bridge */

2-3-Add_PCI_walk_function_and_PCIe_bridge_test

diff --git a/drivers/pci/search.c b/drivers/pci/search.c index 0357f74..bdbff05 100644 --- a/drivers/pci/search.c +++ b/drivers/pci/search.c @@ -74,41 +74,6 @@ struct pci_dev *pci_walk_up_to_first_match(struct pci_dev *pdev, return NULL; } -/* - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html find the upstream PCIe-to-PCI bridge of a PCI device - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html if the device is PCIE, return NULL - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html if the device isn't connected to a PCIe bridge (that is its parent is a - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html legacy PCI bridge and the bridge is directly connected to bus 0), return its - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html parent - */ -struct pci_dev 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html -pci_find_upstream_pcie_bridge(struct pci_dev *pdev) -{ - struct pci_dev *tmp = NULL; - - if (pci_is_pcie(pdev)) - return NULL; - while (1) { - if (pci_is_root_bus(pdev->bus)) - break; - pdev = pdev->bus->self; - /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup a p2p bridge */ - if (!pci_is_pcie(pdev)) { - tmp = pdev; - continue; - } - /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup PCI device should connect to a PCIe bridge */ - if (pci_pcie_type(pdev) != PCI_EXP_TYPE_PCI_BRIDGE) { - /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Busted hardware? */ - WARN_ON_ONCE(1); - return NULL; - } - return pdev; - } - - return tmp; -} - static struct pci_bus *pci_do_find_bus(struct pci_bus *bus, unsigned char busnr) { struct pci_bus* child; diff --git a/include/linux/pci.h b/include/linux/pci.h index e87423a..9d9dcf5 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1877,15 +1877,4 @@ struct pci_dev *pci_walk_up_to_first_match(struct pci_dev *pdev, */ bool pci_is_pcie_bridge(struct pci_dev *pdev); -/** - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html pci_find_upstream_pcie_bridge - find upstream PCIe-to-PCI bridge of a device - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html @pdev: the PCI device - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html if the device is PCIE, return NULL - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html if the device isn't connected to a PCIe bridge (that is its parent is a - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html legacy PCI bridge and the bridge is directly connected to bus 0), return its - 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html parent - */ -struct pci_dev *pci_find_upstream_pcie_bridge(struct pci_dev *pdev); - #endif /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup LINUX_PCI_H */

A patch for qemu:
1-vfio-pci_Lazy_PCI_option_ROM_loading

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index ede026d..730dec5 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -166,6 +166,7 @@ typedef struct VFIODevice { off_t config_offset; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Offset of config space region within device fd */ unsigned int rom_size; off_t rom_offset; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Offset of ROM region within device fd */ + void *rom; int msi_cap_size; VFIOMSIVector *msi_vectors; VFIOMSIXInfo *msix; @@ -1058,6 +1059,125 @@ static const MemoryRegionOps vfio_bar_ops = { .endianness = DEVICE_LITTLE_ENDIAN, }; +static void vfio_pci_load_rom(VFIODevice *vdev) +{ + struct vfio_region_info reg_info = { + .argsz = sizeof(reg_info), + .index = VFIO_PCI_ROM_REGION_INDEX + }; + uint64_t size; + off_t off = 0; + size_t bytes; + + if (ioctl(vdev->fd, VFIO_DEVICE_GET_REGION_INFO, &reg_info)) { + error_report("vfio: Error getting ROM info: %m"); + return; + } + + DPRINTF("Device %04x:%02x:%02x.%x ROM:\n", vdev->host.domain, + vdev->host.bus, vdev->host.slot, vdev->host.function); + DPRINTF(" size: 0x%lx, offset: 0x%lx, flags: 0x%lx\n", + (unsigned long)reg_info.size, (unsigned long)reg_info.offset, + (unsigned long)reg_info.flags); + + vdev->rom_size = size = reg_info.size; + vdev->rom_offset = reg_info.offset; + + if (!vdev->rom_size) { + return; + } + + vdev->rom = g_malloc(size); + memset(vdev->rom, 0xff, size); + + while (size) { + bytes = pread(vdev->fd, vdev->rom + off, size, vdev->rom_offset + off); + if (bytes == 0) { + break; + } else if (bytes > 0) { + off += bytes; + size -= bytes; + } else { + if (errno == EINTR || errno == EAGAIN) { + continue; + } + error_report("vfio: Error reading device ROM: %m"); + break; + } + } +} + +static uint64_t vfio_rom_read(void *opaque, hwaddr addr, unsigned size) +{ + VFIODevice *vdev = opaque; + uint64_t val = ((uint64_t)1 << (size 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html 8)) - 1; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Load the ROM lazily when the guest tries to read it */ + if (unlikely(!vdev->rom)) { + vfio_pci_load_rom(vdev); + } + + memcpy(&val, vdev->rom + addr, + (addr < vdev->rom_size) ? MIN(size, vdev->rom_size - addr) : 0); + + DPRINTF("%s(%04x:%02x:%02x.%x, 0x%"HWADDR_PRIx", 0x%x) = 0x%"PRIx64"\n", + __func__, vdev->host.domain, vdev->host.bus, vdev->host.slot, + vdev->host.function, addr, size, val); + + return val; +} + +static const MemoryRegionOps vfio_rom_ops = { + .read = vfio_rom_read, + .endianness = DEVICE_LITTLE_ENDIAN, +}; + +static void vfio_pci_size_rom(VFIODevice *vdev) +{ + uint32_t orig, size = (uint32_t)PCI_ROM_ADDRESS_MASK; + off_t offset = vdev->config_offset + PCI_ROM_ADDRESS; + char name[32]; + + if (vdev->pdev.romfile || !vdev->pdev.rom_bar) { + return; + } + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html Use the same size ROM BAR as the physical device. The contents + 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_6.html topic_7.html topic_8.html topic_9.html will get filled in later when the guest tries to read it. + */ + if (pread(vdev->fd, &orig, 4, offset) != 4 || + pwrite(vdev->fd, &size, 4, offset) != 4 || + pread(vdev->fd, &size, 4, offset) != 4 || + pwrite(vdev->fd, &orig, 4, offset) != 4) { + error_report("%s(%04x:%02x:%02x.%x) failed: %m", + __func__, vdev->host.domain, vdev->host.bus, + vdev->host.slot, vdev->host.function); + return; + } + + size = ~(size & PCI_ROM_ADDRESS_MASK) + 1; + + if (!size) { + return; + } + + DPRINTF("%04x:%02x:%02x.%x ROM size 0x%x\n", vdev->host.domain, + vdev->host.bus, vdev->host.slot, vdev->host.function, size); + + snprintf(name, sizeof(name), "vfio[%04x:%02x:%02x.%x].rom", + vdev->host.domain, vdev->host.bus, vdev->host.slot, + vdev->host.function); + + memory_region_init_io(&vdev->pdev.rom, OBJECT(vdev), + &vfio_rom_ops, vdev, name, size); + + pci_register_bar(&vdev->pdev, PCI_ROM_SLOT, + PCI_BASE_ADDRESS_SPACE_MEMORY, &vdev->pdev.rom); + + vdev->pdev.has_rom = true; +} + static void vfio_vga_write(void *opaque, hwaddr addr, uint64_t data, unsigned size) { @@ -2638,51 +2758,6 @@ static int vfio_add_capabilities(VFIODevice *vdev) return vfio_add_std_cap(vdev, pdev->config[PCI_CAPABILITY_LIST]); } -static int vfio_load_rom(VFIODevice *vdev) -{ - uint64_t size = vdev->rom_size; - char name[32]; - off_t off = 0, voff = vdev->rom_offset; - ssize_t bytes; - void *ptr; - - /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup If loading ROM from file, pci handles it */ - if (vdev->pdev.romfile || !vdev->pdev.rom_bar || !size) { - return 0; - } - - DPRINTF("%s(%04x:%02x:%02x.%x)\n", __func__, vdev->host.domain, - vdev->host.bus, vdev->host.slot, vdev->host.function); - - snprintf(name, sizeof(name), "vfio[%04x:%02x:%02x.%x].rom", - vdev->host.domain, vdev->host.bus, vdev->host.slot, - vdev->host.function); - memory_region_init_ram(&vdev->pdev.rom, OBJECT(vdev), name, size); - ptr = memory_region_get_ram_ptr(&vdev->pdev.rom); - memset(ptr, 0xff, size); - - while (size) { - bytes = pread(vdev->fd, ptr + off, size, voff + off); - if (bytes == 0) { - break; /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup expect that we could get back less than the ROM BAR */ - } else if (bytes > 0) { - off += bytes; - size -= bytes; - } else { - if (errno == EINTR || errno == EAGAIN) { - continue; - } - error_report("vfio: Error reading device ROM: %m"); - memory_region_destroy(&vdev->pdev.rom); - return -errno; - } - } - - pci_register_bar(&vdev->pdev, PCI_ROM_SLOT, 0, &vdev->pdev.rom); - vdev->pdev.has_rom = true; - return 0; -} - static int vfio_connect_container(VFIOGroup *group) { VFIOContainer *container; @@ -2916,22 +2991,6 @@ static int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vdev) QLIST_INIT(&vdev->bars[i].quirks); } - reg_info.index = VFIO_PCI_ROM_REGION_INDEX; - - ret = ioctl(vdev->fd, VFIO_DEVICE_GET_REGION_INFO, &reg_info); - if (ret) { - error_report("vfio: Error getting ROM info: %m"); - goto error; - } - - DPRINTF("Device %s ROM:\n", name); - DPRINTF(" size: 0x%lx, offset: 0x%lx, flags: 0x%lx\n", - (unsigned long)reg_info.size, (unsigned long)reg_info.offset, - (unsigned long)reg_info.flags); - - vdev->rom_size = reg_info.size; - vdev->rom_offset = reg_info.offset; - reg_info.index = VFIO_PCI_CONFIG_REGION_INDEX; ret = ioctl(vdev->fd, VFIO_DEVICE_GET_REGION_INFO, &reg_info); @@ -3229,7 +3288,7 @@ static int vfio_initfn(PCIDevice *pdev) memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24); memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4); - vfio_load_rom(vdev); + vfio_pci_size_rom(vdev); ret = vfio_early_setup_msix(vdev); if (ret) { @@ -3294,6 +3353,7 @@ static void vfio_exitfn(PCIDevice *pdev) vfio_teardown_msi(vdev); vfio_unmap_bars(vdev); g_free(vdev->emulated_config_bits); + g_free(vdev->rom); vfio_put_device(vdev); vfio_put_group(group); } 

lspci -t

-[0000:00]-+-00.0 +-01.0-[01]-- +-01.1-[02]----00.0 +-02.0 +-14.0 +-16.0 +-19.0 +-1a.0 +-1b.0 +-1c.0-[03]-- +-1c.1-[04]----00.0 +-1c.5-[05-06]----00.0-[06]----01.0 +-1c.6-[07]----00.0 +-1c.7-[08]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3

lspci -nnk

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5000] 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09) Kernel driver in use: pcieport Kernel modules: shpchp 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09) Kernel driver in use: pcieport Kernel modules: shpchp 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0152] (rev 09) Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000] Kernel driver in use: i915 Kernel modules: i915 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007] Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 [8086:1e3a] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:1c3a] Kernel driver in use: mei_me Kernel modules: mei_me 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:e000] Kernel driver in use: e1000e Kernel modules: e1000e 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5006] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:a072] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.1 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 [8086:1e12] (rev c4) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.5 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev c4) 00:1c.6 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 [8086:1e1c] (rev c4) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4) Kernel driver in use: pcieport Kernel modules: shpchp 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5006] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001] Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:b005] Kernel driver in use: ahci 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller [8086:1e22] (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5001] Kernel modules: i2c_i801 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation G73 [GeForce 7600 GT] [10de:0391] (rev a1) Subsystem: Micro-Star International Co., Ltd. Device [1462:0452] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau 04:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) Subsystem: Gigabyte Technology Co., Ltd Device [1458:b000] Kernel driver in use: ahci 05:00.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 30) 06:01.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller [1106:3044] (rev c0) Subsystem: Gigabyte Technology Co., Ltd GA-7VT600-1394 Motherboard [1458:1000] Kernel driver in use: firewire_ohci Kernel modules: firewire_ohci 07:00.0 Ethernet controller [0200]: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet [1969:1083] (rev c0) Subsystem: Gigabyte Technology Co., Ltd Device [1458:e000] Kernel driver in use: atl1c Kernel modules: atl1c 08:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) Subsystem: Gigabyte Technology Co., Ltd Device [1458:b000] Kernel driver in use: ahci

Adding parameter to kernel

intel_iommu=on,igfx_off

Without igfx_off the system frezze.

dmesg after boot

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.14.2 (root@kbrown) (gcc version 4.8.2 (GCC) ) #1 SMP Fri May 2 00:04:02 CDT 2014 [ 0.000000] Command line: auto BOOT_IMAGE=Generic3.14.2 ro root=803 intel_iommu=on,igfx_off [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001fffffff] usable [ 0.000000] BIOS-e820: [mem 0x0000000020000000-0x00000000201fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000020200000-0x0000000040003fff] usable [ 0.000000] BIOS-e820: [mem 0x0000000040004000-0x0000000040004fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000040005000-0x000000008d455fff] usable [ 0.000000] BIOS-e820: [mem 0x000000008d456000-0x000000008d905fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000008d906000-0x000000008d906fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000008d907000-0x000000008da2ffff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000008da30000-0x000000008e265fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000008e266000-0x000000008e266fff] usable [ 0.000000] BIOS-e820: [mem 0x000000008e267000-0x000000008e2a9fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000008e2aa000-0x000000008ec6bfff] usable [ 0.000000] BIOS-e820: [mem 0x000000008ec6c000-0x000000008efd6fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000008efd7000-0x000000008effffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000cf800000-0x00000000cf9fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000042f5fffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77X-UD5H, BIOS F14 08/22/2012 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] No AGP bridge found [ 0.000000] e820: last_pfn = 0x42f600 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-CFFFF write-protect [ 0.000000] D0000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000 mask C00000000 write-back [ 0.000000] 1 base 400000000 mask FE0000000 write-back [ 0.000000] 2 base 420000000 mask FF0000000 write-back [ 0.000000] 3 base 0C0000000 mask FC0000000 uncachable [ 0.000000] 4 base 0A0000000 mask FE0000000 uncachable [ 0.000000] 5 base 090000000 mask FF0000000 uncachable [ 0.000000] 6 base 08F800000 mask FFF800000 uncachable [ 0.000000] 7 base 42F800000 mask FFF800000 uncachable [ 0.000000] 8 base 42F600000 mask FFFE00000 uncachable [ 0.000000] 9 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: update [mem 0x8f800000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0x8f000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd830-0x000fd83f] mapped at [ffff8800000fd830] [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01cac000, 0x01cacfff] PGTABLE [ 0.000000] BRK [0x01cad000, 0x01cadfff] PGTABLE [ 0.000000] BRK [0x01cae000, 0x01caefff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x42f400000-0x42f5fffff] [ 0.000000] [mem 0x42f400000-0x42f5fffff] page 2M [ 0.000000] BRK [0x01caf000, 0x01caffff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x42c000000-0x42f3fffff] [ 0.000000] [mem 0x42c000000-0x42f3fffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x400000000-0x42bffffff] [ 0.000000] [mem 0x400000000-0x42bffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0x1fffffff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x1fffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x20200000-0x40003fff] [ 0.000000] [mem 0x20200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x40003fff] page 4k [ 0.000000] BRK [0x01cb0000, 0x01cb0fff] PGTABLE [ 0.000000] BRK [0x01cb1000, 0x01cb1fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x40005000-0x8d455fff] [ 0.000000] [mem 0x40005000-0x401fffff] page 4k [ 0.000000] [mem 0x40200000-0x8d3fffff] page 2M [ 0.000000] [mem 0x8d400000-0x8d455fff] page 4k [ 0.000000] init_memory_mapping: [mem 0x8e266000-0x8e266fff] [ 0.000000] [mem 0x8e266000-0x8e266fff] page 4k [ 0.000000] init_memory_mapping: [mem 0x8e2aa000-0x8ec6bfff] [ 0.000000] [mem 0x8e2aa000-0x8e3fffff] page 4k [ 0.000000] [mem 0x8e400000-0x8ebfffff] page 2M [ 0.000000] [mem 0x8ec00000-0x8ec6bfff] page 4k [ 0.000000] init_memory_mapping: [mem 0x8efd7000-0x8effffff] [ 0.000000] [mem 0x8efd7000-0x8effffff] page 4k [ 0.000000] init_memory_mapping: [mem 0x100000000-0x3ffffffff] [ 0.000000] [mem 0x100000000-0x3ffffffff] page 2M [ 0.000000] RAMDISK: [mem 0x7fbb0000-0x7fffffff] [ 0.000000] ACPI: RSDP 00000000000f0490 000024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 000000008da0e078 00006C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 000000008da195f8 0000F4 (v04 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: DSDT 000000008da0e178 00B47F (v02 ALASKA A M I 00000012 INTL 20051117) [ 0.000000] ACPI: FACS 000000008da2ef80 000040 [ 0.000000] ACPI: APIC 000000008da196f0 000072 (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: MCFG 000000008da19768 00003C (v01 01072009 MSFT 00000097) [ 0.000000] ACPI: HPET 000000008da197a8 000038 (v01 ALASKA A M I 01072009 AMI. 00000005) [ 0.000000] ACPI: SSDT 000000008da197e0 00036D (v01 SataRe SataTabl 00001000 INTL 20091112) [ 0.000000] ACPI: SSDT 000000008da19b50 0009AA (v01 PmRef Cpu0Ist 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 000000008da1a500 000A92 (v01 PmRef CpuPm 00003000 INTL 20051117) [ 0.000000] ACPI: DMAR 000000008da1af98 0000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.000000] ACPI: MATS 000000008da1b050 000034 (v02 ALASKA A M I 00000002 w?x2 00000000) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] [ffffea0000000000-ffffea0010bfffff] PMD -> [ffff88041fc00000-ffff88042ebfffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x42f5fffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0x1fffffff] [ 0.000000] node 0: [mem 0x20200000-0x40003fff] [ 0.000000] node 0: [mem 0x40005000-0x8d455fff] [ 0.000000] node 0: [mem 0x8e266000-0x8e266fff] [ 0.000000] node 0: [mem 0x8e2aa000-0x8ec6bfff] [ 0.000000] node 0: [mem 0x8efd7000-0x8effffff] [ 0.000000] node 0: [mem 0x100000000-0x42f5fffff] [ 0.000000] On node 0 totalpages: 3920349 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 9010 pages used for memmap [ 0.000000] DMA32 zone: 576577 pages, LIFO batch:31 [ 0.000000] Normal zone: 52184 pages used for memmap [ 0.000000] Normal zone: 3339776 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ2 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] nr_irqs_gsi: 40 [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x20000000-0x201fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x40004000-0x40004fff] [ 0.000000] PM: Registered nosave memory: [mem 0x8d456000-0x8d905fff] [ 0.000000] PM: Registered nosave memory: [mem 0x8d906000-0x8d906fff] [ 0.000000] PM: Registered nosave memory: [mem 0x8d907000-0x8da2ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x8da30000-0x8e265fff] [ 0.000000] PM: Registered nosave memory: [mem 0x8e267000-0x8e2a9fff] [ 0.000000] PM: Registered nosave memory: [mem 0x8ec6c000-0x8efd6fff] [ 0.000000] PM: Registered nosave memory: [mem 0x8f000000-0x8f9fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x8fa00000-0xb19fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xb1a00000-0xcf7fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xcf800000-0xcf9fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xcfa00000-0xf7ffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfecfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed00000-0xfed03fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed04000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff] [ 0.000000] e820: [mem 0xcfa00000-0xf7ffffff] available for PCI devices [ 0.000000] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 27 pages/cpu @ffff88042f200000 s78848 r8192 d23552 u524288 [ 0.000000] pcpu-alloc: s78848 r8192 d23552 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 3859070 [ 0.000000] Kernel command line: auto BOOT_IMAGE=Generic3.14.2 ro root=803 intel_iommu=on,igfx_off [ 0.000000] Intel-IOMMU: enabled [ 0.000000] Intel-IOMMU: disable GFX device mapping [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.000000] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 [ 0.000000] Checking aperture... [ 0.000000] No AGP bridge found [ 0.000000] Memory: 15326756K/15681396K available (5963K kernel code, 687K rwdata, 2080K rodata, 1088K init, 924K bss, 354640K reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] CONFIG_RCU_FANOUT set to non-default value of 32 [ 0.000000] RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS:8448 nr_irqs:712 16 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 3002.957 MHz processor [ 0.000002] Calibrating delay loop (skipped), value calculated using timer frequency.. 6005.91 BogoMIPS (lpj=3002957) [ 0.000005] pid_max: default: 32768 minimum: 301 [ 0.000010] ACPI: Core revision 20131218 [ 0.004301] ACPI: All ACPI Tables successfully acquired [ 0.006502] Security Framework initialized [ 0.006512] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.006514] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.006689] Initializing cgroup subsys devices [ 0.006691] Initializing cgroup subsys freezer [ 0.006693] Initializing cgroup subsys net_cls [ 0.006695] Initializing cgroup subsys blkio [ 0.006697] Initializing cgroup subsys perf_event [ 0.006716] CPU: Physical Processor ID: 0 [ 0.006717] CPU: Processor Core ID: 0 [ 0.006722] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.006722] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.007019] mce: CPU supports 9 MCE banks [ 0.007030] CPU0: Thermal monitoring enabled (TM1) [ 0.007037] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8 [ 0.007037] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0 [ 0.007037] tlb_flushall_shift: 2 [ 0.007115] Freeing SMP alternatives memory: 24K (ffffffff81bbd000 - ffffffff81bc3000) [ 0.007118] ftrace: allocating 23798 entries in 93 pages [ 0.015849] dmar: Host address width 36 [ 0.015852] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.015858] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.015860] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.015864] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.015865] dmar: RMRR base: 0x0000008d86f000 end: 0x0000008d899fff [ 0.015867] dmar: RMRR base: 0x000000cf800000 end: 0x000000cf9fffff [ 0.015937] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.015938] HPET id 0 under DRHD base 0xfed91000 [ 0.015940] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.016058] Enabled IRQ remapping in x2apic mode [ 0.016059] Enabling x2apic [ 0.016060] Enabled x2apic [ 0.016064] Switched APIC routing to cluster x2apic. [ 0.016474] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.026477] smpboot: CPU0: Intel(R) Core(TM) i5-3330 CPU @ 3.00GHz (fam: 06, model: 3a, stepping: 09) [ 0.026483] TSC deadline timer enabled [ 0.026489] Performance Events: PEBS fmt1+, 16-deep LBR, IvyBridge events, full-width counters, Intel PMU driver. [ 0.026496] ... version: 3 [ 0.026497] ... bit width: 48 [ 0.026498] ... generic registers: 8 [ 0.026499] ... value mask: 0000ffffffffffff [ 0.026500] ... max period: 0000ffffffffffff [ 0.026501] ... fixed-purpose events: 3 [ 0.026503] ... event mask: 00000007000000ff [ 0.026647] x86: Booting SMP configuration: [ 0.026649] .... node #0, CPUs: #1 #2 #3 [ 0.067044] x86: Booted up 1 node, 4 CPUs [ 0.067049] smpboot: Total of 4 processors activated (24023.65 BogoMIPS) [ 0.069376] devtmpfs: initialized [ 0.071890] PM: Registering ACPI NVS region [mem 0x8d907000-0x8da2ffff] (1216512 bytes) [ 0.071903] PM: Registering ACPI NVS region [mem 0x8e267000-0x8e2a9fff] (274432 bytes) [ 0.071993] xor: automatically using best checksumming function: [ 0.081222] avx : 21368.000 MB/sec [ 0.081226] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.081228] pinctrl core: initialized pinctrl subsystem [ 0.081266] regulator-dummy: no parameters [ 0.081296] NET: Registered protocol family 16 [ 0.081343] cpuidle: using governor ladder [ 0.081344] cpuidle: using governor menu [ 0.081359] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.081361] ACPI: bus type PCI registered [ 0.081363] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.081405] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000) [ 0.081408] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820 [ 0.087132] PCI: Using configuration type 1 for base access [ 0.087919] bio: create slab <bio-0> at 0 [ 0.104238] raid6: sse2x1 8238 MB/s [ 0.121252] raid6: sse2x2 10230 MB/s [ 0.138260] raid6: sse2x4 11851 MB/s [ 0.138262] raid6: using algorithm sse2x4 (11851 MB/s) [ 0.138263] raid6: using ssse3x2 recovery algorithm [ 0.138287] ACPI: Added _OSI(Module Device) [ 0.138289] ACPI: Added _OSI(Processor Device) [ 0.138290] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.138291] ACPI: Added _OSI(Processor Aggregator Device) [ 0.140554] ACPI: Executed 1 blocks of module-level executable AML code [ 0.147583] ACPI: SSDT 000000008d8a8018 00083B (v01 PmRef Cpu0Cst 00003001 INTL 20051117) [ 0.147861] ACPI: Dynamic OEM Table Load: [ 0.147863] ACPI: SSDT (null) 00083B (v01 PmRef Cpu0Cst 00003001 INTL 20051117) [ 0.152472] ACPI: SSDT 000000008d8a9a98 000303 (v01 PmRef ApIst 00003000 INTL 20051117) [ 0.152766] ACPI: Dynamic OEM Table Load: [ 0.152768] ACPI: SSDT (null) 000303 (v01 PmRef ApIst 00003000 INTL 20051117) [ 0.157394] ACPI: SSDT 000000008d8b5c18 000119 (v01 PmRef ApCst 00003000 INTL 20051117) [ 0.157665] ACPI: Dynamic OEM Table Load: [ 0.157667] ACPI: SSDT (null) 000119 (v01 PmRef ApCst 00003000 INTL 20051117) [ 0.162731] ACPI: Interpreter enabled [ 0.162737] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20131218/hwxface-580) [ 0.162741] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20131218/hwxface-580) [ 0.162752] ACPI: (supports S0 S3 S4 S5) [ 0.162753] ACPI: Using IOAPIC for interrupt routing [ 0.162771] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.162908] ACPI: No dock devices found. [ 0.168303] ACPI: Power Resource [FN00] (off) [ 0.168363] ACPI: Power Resource [FN01] (off) [ 0.168417] ACPI: Power Resource [FN02] (off) [ 0.168472] ACPI: Power Resource [FN03] (off) [ 0.168526] ACPI: Power Resource [FN04] (off) [ 0.168897] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e]) [ 0.168901] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.169051] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug PME] [ 0.169140] acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability] [ 0.169578] PCI host bridge to bus 0000:00 [ 0.169581] pci_bus 0000:00: root bus resource [bus 00-3e] [ 0.169583] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] [ 0.169585] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.169587] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.169589] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff] [ 0.169590] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] [ 0.169592] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] [ 0.169594] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff] [ 0.169596] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff] [ 0.169598] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff] [ 0.169599] pci_bus 0000:00: root bus resource [mem 0xcfa00000-0xfeafffff] [ 0.169606] pci 0000:00:00.0: [8086:0150] type 00 class 0x060000 [ 0.169669] pci 0000:00:01.0: [8086:0151] type 01 class 0x060400 [ 0.169695] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 0.169724] pci 0000:00:01.0: System wakeup disabled by ACPI [ 0.169753] pci 0000:00:02.0: [8086:0152] type 00 class 0x030000 [ 0.169762] pci 0000:00:02.0: reg 0x10: [mem 0xf7400000-0xf77fffff 64bit] [ 0.169767] pci 0000:00:02.0: reg 0x18: [mem 0xd0000000-0xdfffffff 64bit pref] [ 0.169771] pci 0000:00:02.0: reg 0x20: [io 0xf000-0xf03f] [ 0.169849] pci 0000:00:14.0: [8086:1e31] type 00 class 0x0c0330 [ 0.169870] pci 0000:00:14.0: reg 0x10: [mem 0xf7d20000-0xf7d2ffff 64bit] [ 0.169937] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 0.169970] pci 0000:00:14.0: System wakeup disabled by ACPI [ 0.169999] pci 0000:00:16.0: [8086:1e3a] type 00 class 0x078000 [ 0.170020] pci 0000:00:16.0: reg 0x10: [mem 0xf7d3b000-0xf7d3b00f 64bit] [ 0.170090] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 0.170155] pci 0000:00:19.0: [8086:1503] type 00 class 0x020000 [ 0.170171] pci 0000:00:19.0: reg 0x10: [mem 0xf7d00000-0xf7d1ffff] [ 0.170179] pci 0000:00:19.0: reg 0x14: [mem 0xf7d39000-0xf7d39fff] [ 0.170186] pci 0000:00:19.0: reg 0x18: [io 0xf080-0xf09f] [ 0.170245] pci 0000:00:19.0: PME# supported from D0 D3hot D3cold [ 0.170278] pci 0000:00:19.0: System wakeup disabled by ACPI [ 0.170308] pci 0000:00:1a.0: [8086:1e2d] type 00 class 0x0c0320 [ 0.170329] pci 0000:00:1a.0: reg 0x10: [mem 0xf7d38000-0xf7d383ff] [ 0.170415] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 0.170460] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 0.170489] pci 0000:00:1b.0: [8086:1e20] type 00 class 0x040300 [ 0.170502] pci 0000:00:1b.0: reg 0x10: [mem 0xf7d30000-0xf7d33fff 64bit] [ 0.170572] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold [ 0.170606] pci 0000:00:1b.0: System wakeup disabled by ACPI [ 0.170632] pci 0000:00:1c.0: [8086:1e10] type 01 class 0x060400 [ 0.170708] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 0.170745] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 0.170770] pci 0000:00:1c.1: [8086:1e12] type 01 class 0x060400 [ 0.170846] pci 0000:00:1c.1: PME# supported from D0 D3hot D3cold [ 0.170883] pci 0000:00:1c.1: System wakeup disabled by ACPI [ 0.170910] pci 0000:00:1c.3: [8086:1e16] type 01 class 0x060400 [ 0.170985] pci 0000:00:1c.3: PME# supported from D0 D3hot D3cold [ 0.171023] pci 0000:00:1c.3: System wakeup disabled by ACPI [ 0.171049] pci 0000:00:1c.5: [8086:244e] type 01 class 0x060401 [ 0.171125] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold [ 0.171161] pci 0000:00:1c.5: System wakeup disabled by ACPI [ 0.171187] pci 0000:00:1c.6: [8086:1e1c] type 01 class 0x060400 [ 0.171262] pci 0000:00:1c.6: PME# supported from D0 D3hot D3cold [ 0.171299] pci 0000:00:1c.6: System wakeup disabled by ACPI [ 0.171327] pci 0000:00:1c.7: [8086:1e1e] type 01 class 0x060400 [ 0.171402] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 0.171439] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 0.171469] pci 0000:00:1d.0: [8086:1e26] type 00 class 0x0c0320 [ 0.171488] pci 0000:00:1d.0: reg 0x10: [mem 0xf7d37000-0xf7d373ff] [ 0.171573] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 0.171616] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 0.171645] pci 0000:00:1f.0: [8086:1e44] type 00 class 0x060100 [ 0.171796] pci 0000:00:1f.2: [8086:1e02] type 00 class 0x010601 [ 0.171812] pci 0000:00:1f.2: reg 0x10: [io 0xf0d0-0xf0d7] [ 0.171819] pci 0000:00:1f.2: reg 0x14: [io 0xf0c0-0xf0c3] [ 0.171826] pci 0000:00:1f.2: reg 0x18: [io 0xf0b0-0xf0b7] [ 0.171833] pci 0000:00:1f.2: reg 0x1c: [io 0xf0a0-0xf0a3] [ 0.171839] pci 0000:00:1f.2: reg 0x20: [io 0xf060-0xf07f] [ 0.171846] pci 0000:00:1f.2: reg 0x24: [mem 0xf7d36000-0xf7d367ff] [ 0.171887] pci 0000:00:1f.2: PME# supported from D3hot [ 0.171939] pci 0000:00:1f.3: [8086:1e22] type 00 class 0x0c0500 [ 0.171953] pci 0000:00:1f.3: reg 0x10: [mem 0xf7d35000-0xf7d350ff 64bit] [ 0.171971] pci 0000:00:1f.3: reg 0x20: [io 0xf040-0xf05f] [ 0.172065] pci 0000:01:00.0: [10de:0391] type 00 class 0x030000 [ 0.172071] pci 0000:01:00.0: reg 0x10: [mem 0xf6000000-0xf6ffffff] [ 0.172077] pci 0000:01:00.0: reg 0x14: [mem 0xe0000000-0xefffffff 64bit pref] [ 0.172084] pci 0000:01:00.0: reg 0x1c: [mem 0xf5000000-0xf5ffffff 64bit] [ 0.172088] pci 0000:01:00.0: reg 0x24: [io 0xe000-0xe07f] [ 0.172092] pci 0000:01:00.0: reg 0x30: [mem 0xf7000000-0xf701ffff pref] [ 0.173332] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.173338] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.173341] pci 0000:00:01.0: bridge window [mem 0xf5000000-0xf70fffff] [ 0.173346] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xefffffff 64bit pref] [ 0.173430] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.173509] pci 0000:03:00.0: [1b4b:9172] type 00 class 0x010601 [ 0.173526] pci 0000:03:00.0: reg 0x10: [io 0xd040-0xd047] [ 0.173538] pci 0000:03:00.0: reg 0x14: [io 0xd030-0xd033] [ 0.173549] pci 0000:03:00.0: reg 0x18: [io 0xd020-0xd027] [ 0.173561] pci 0000:03:00.0: reg 0x1c: [io 0xd010-0xd013] [ 0.173573] pci 0000:03:00.0: reg 0x20: [io 0xd000-0xd00f] [ 0.173585] pci 0000:03:00.0: reg 0x24: [mem 0xf7c10000-0xf7c101ff] [ 0.173596] pci 0000:03:00.0: reg 0x30: [mem 0xf7c00000-0xf7c0ffff pref] [ 0.173657] pci 0000:03:00.0: PME# supported from D3hot [ 0.173680] pci 0000:03:00.0: System wakeup disabled by ACPI [ 0.175340] pci 0000:00:1c.1: PCI bridge to [bus 03] [ 0.175346] pci 0000:00:1c.1: bridge window [io 0xd000-0xdfff] [ 0.175350] pci 0000:00:1c.1: bridge window [mem 0xf7c00000-0xf7cfffff] [ 0.175457] pci 0000:04:00.0: [168c:0034] type 00 class 0x028000 [ 0.175482] pci 0000:04:00.0: reg 0x10: [mem 0xf7b00000-0xf7b7ffff 64bit] [ 0.175544] pci 0000:04:00.0: reg 0x30: [mem 0xf7b80000-0xf7b8ffff pref] [ 0.175625] pci 0000:04:00.0: supports D1 D2 [ 0.175627] pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.175655] pci 0000:04:00.0: System wakeup disabled by ACPI [ 0.177344] pci 0000:00:1c.3: PCI bridge to [bus 04] [ 0.177353] pci 0000:00:1c.3: bridge window [mem 0xf7b00000-0xf7bfffff] [ 0.177461] pci 0000:05:00.0: [8086:244e] type 01 class 0x060401 [ 0.177616] pci 0000:05:00.0: supports D1 D2 [ 0.177617] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.177642] pci 0000:05:00.0: System wakeup disabled by ACPI [ 0.177662] pci 0000:00:1c.5: PCI bridge to [bus 05-06] (subtractive decode) [ 0.177666] pci 0000:00:1c.5: bridge window [io 0xc000-0xcfff] [ 0.177669] pci 0000:00:1c.5: bridge window [mem 0xf7a00000-0xf7afffff] [ 0.177674] pci 0000:00:1c.5: bridge window [io 0x0000-0x0cf7] (subtractive decode) [ 0.177675] pci 0000:00:1c.5: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.177677] pci 0000:00:1c.5: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.177678] pci 0000:00:1c.5: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) [ 0.177679] pci 0000:00:1c.5: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 0.177680] pci 0000:00:1c.5: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 0.177681] pci 0000:00:1c.5: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 0.177682] pci 0000:00:1c.5: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 0.177684] pci 0000:00:1c.5: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 0.177685] pci 0000:00:1c.5: bridge window [mem 0xcfa00000-0xfeafffff] (subtractive decode) [ 0.177767] pci 0000:06:01.0: [1106:3044] type 00 class 0x0c0010 [ 0.177803] pci 0000:06:01.0: reg 0x10: [mem 0xf7a00000-0xf7a007ff] [ 0.177823] pci 0000:06:01.0: reg 0x14: [io 0xc000-0xc07f] [ 0.177975] pci 0000:06:01.0: supports D2 [ 0.177976] pci 0000:06:01.0: PME# supported from D2 D3hot D3cold [ 0.178088] pci 0000:05:00.0: PCI bridge to [bus 06] (subtractive decode) [ 0.178099] pci 0000:05:00.0: bridge window [io 0xc000-0xcfff] [ 0.178105] pci 0000:05:00.0: bridge window [mem 0xf7a00000-0xf7afffff] [ 0.178116] pci 0000:05:00.0: bridge window [io 0xc000-0xcfff] (subtractive decode) [ 0.178117] pci 0000:05:00.0: bridge window [mem 0xf7a00000-0xf7afffff] (subtractive decode) [ 0.178118] pci 0000:05:00.0: bridge window [??? 0x00000000 flags 0x0] (subtractive decode) [ 0.178120] pci 0000:05:00.0: bridge window [??? 0x00000000 flags 0x0] (subtractive decode) [ 0.178121] pci 0000:05:00.0: bridge window [io 0x0000-0x0cf7] (subtractive decode) [ 0.178122] pci 0000:05:00.0: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.178123] pci 0000:05:00.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.178124] pci 0000:05:00.0: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) [ 0.178125] pci 0000:05:00.0: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 0.178127] pci 0000:05:00.0: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 0.178128] pci 0000:05:00.0: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 0.178129] pci 0000:05:00.0: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 0.178130] pci 0000:05:00.0: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 0.178131] pci 0000:05:00.0: bridge window [mem 0xcfa00000-0xfeafffff] (subtractive decode) [ 0.178218] pci 0000:07:00.0: [1969:1083] type 00 class 0x020000 [ 0.178244] pci 0000:07:00.0: reg 0x10: [mem 0xf7900000-0xf793ffff 64bit] [ 0.178258] pci 0000:07:00.0: reg 0x18: [io 0xb000-0xb07f] [ 0.178392] pci 0000:07:00.0: PME# supported from D0 D1 D2 D3hot D3cold [ 0.178421] pci 0000:07:00.0: System wakeup disabled by ACPI [ 0.180351] pci 0000:00:1c.6: PCI bridge to [bus 07] [ 0.180358] pci 0000:00:1c.6: bridge window [io 0xb000-0xbfff] [ 0.180362] pci 0000:00:1c.6: bridge window [mem 0xf7900000-0xf79fffff] [ 0.180464] pci 0000:08:00.0: [1b4b:9172] type 00 class 0x010601 [ 0.180480] pci 0000:08:00.0: reg 0x10: [io 0xa040-0xa047] [ 0.180492] pci 0000:08:00.0: reg 0x14: [io 0xa030-0xa033] [ 0.180504] pci 0000:08:00.0: reg 0x18: [io 0xa020-0xa027] [ 0.180515] pci 0000:08:00.0: reg 0x1c: [io 0xa010-0xa013] [ 0.180527] pci 0000:08:00.0: reg 0x20: [io 0xa000-0xa00f] [ 0.180539] pci 0000:08:00.0: reg 0x24: [mem 0xf7810000-0xf78101ff] [ 0.180551] pci 0000:08:00.0: reg 0x30: [mem 0xf7800000-0xf780ffff pref] [ 0.180611] pci 0000:08:00.0: PME# supported from D3hot [ 0.180634] pci 0000:08:00.0: System wakeup disabled by ACPI [ 0.182352] pci 0000:00:1c.7: PCI bridge to [bus 08] [ 0.182358] pci 0000:00:1c.7: bridge window [io 0xa000-0xafff] [ 0.182362] pci 0000:00:1c.7: bridge window [mem 0xf7800000-0xf78fffff] [ 0.182416] pci_bus 0000:00: on NUMA node 0 [ 0.182419] acpi PNP0A08:00: Disabling ASPM (FADT indicates it is unsupported) [ 0.182582] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.182619] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 *4 5 6 10 11 12 14 15) [ 0.182653] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.182687] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.182721] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15) [ 0.182755] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 0.182789] ACPI: PCI Interrupt Link [LNKG] (IRQs *3 4 5 6 10 11 12 14 15) [ 0.182823] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.182893] ACPI: Enabled 4 GPEs in block 00 to 3F [ 0.182943] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.182948] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.182950] vgaarb: loaded [ 0.182951] vgaarb: bridge control possible 0000:01:00.0 [ 0.182953] vgaarb: no bridge control possible 0000:00:02.0 [ 0.182990] SCSI subsystem initialized [ 0.183014] libata version 3.00 loaded. [ 0.183017] ACPI: bus type USB registered [ 0.183030] usbcore: registered new interface driver usbfs [ 0.183038] usbcore: registered new interface driver hub [ 0.183050] usbcore: registered new device driver usb [ 0.183100] PCI: Using ACPI for IRQ routing [ 0.184475] PCI: pci_cache_line_size set to 64 bytes [ 0.184541] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff] [ 0.184542] e820: reserve RAM buffer [mem 0x40004000-0x43ffffff] [ 0.184543] e820: reserve RAM buffer [mem 0x8d456000-0x8fffffff] [ 0.184545] e820: reserve RAM buffer [mem 0x8e267000-0x8fffffff] [ 0.184546] e820: reserve RAM buffer [mem 0x8ec6c000-0x8fffffff] [ 0.184547] e820: reserve RAM buffer [mem 0x8f000000-0x8fffffff] [ 0.184548] e820: reserve RAM buffer [mem 0x42f600000-0x42fffffff] [ 0.184624] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 0.184629] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 0.188094] Switched to clocksource hpet [ 0.189439] pnp: PnP ACPI init [ 0.189448] ACPI: bus type PNP registered [ 0.189498] system 00:00: [mem 0xfed40000-0xfed44fff] has been reserved [ 0.189501] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.189509] pnp 00:01: [dma 4] [ 0.189518] pnp 00:01: Plug and Play ACPI device, IDs PNP0200 (active) [ 0.189531] pnp 00:02: Plug and Play ACPI device, IDs INT0800 (active) [ 0.189591] pnp 00:03: Plug and Play ACPI device, IDs PNP0103 (active) [ 0.189623] system 00:04: [io 0x0680-0x069f] has been reserved [ 0.189625] system 00:04: [io 0x0200-0x020f] has been reserved [ 0.189627] system 00:04: [io 0xffff] has been reserved [ 0.189629] system 00:04: [io 0xffff] has been reserved [ 0.189631] system 00:04: [io 0x0400-0x0453] could not be reserved [ 0.189633] system 00:04: [io 0x0458-0x047f] has been reserved [ 0.189635] system 00:04: [io 0x0500-0x057f] has been reserved [ 0.189637] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.189658] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.189689] system 00:06: [io 0x0454-0x0457] has been reserved [ 0.189692] system 00:06: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 0.189776] system 00:07: [io 0x0a00-0x0a0f] has been reserved [ 0.189778] system 00:07: [io 0x0a30-0x0a3f] has been reserved [ 0.189780] system 00:07: [io 0x0a20-0x0a2f] has been reserved [ 0.189782] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.189832] system 00:08: [io 0x04d0-0x04d1] has been reserved [ 0.189835] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.189852] pnp 00:09: Plug and Play ACPI device, IDs PNP0c04 (active) [ 0.190037] system 00:0a: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 0.190040] system 00:0a: [mem 0xfed10000-0xfed17fff] has been reserved [ 0.190044] system 00:0a: [mem 0xfed18000-0xfed18fff] has been reserved [ 0.190046] system 00:0a: [mem 0xfed19000-0xfed19fff] has been reserved [ 0.190048] system 00:0a: [mem 0xf8000000-0xfbffffff] has been reserved [ 0.190050] system 00:0a: [mem 0xfed20000-0xfed3ffff] has been reserved [ 0.190053] system 00:0a: [mem 0xfed90000-0xfed93fff] could not be reserved [ 0.190055] system 00:0a: [mem 0xfed45000-0xfed8ffff] has been reserved [ 0.190057] system 00:0a: [mem 0xff000000-0xffffffff] has been reserved [ 0.190059] system 00:0a: [mem 0xfee00000-0xfeefffff] could not be reserved [ 0.190061] system 00:0a: [mem 0xcfa00000-0xcfa00fff] has been reserved [ 0.190063] system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.190180] system 00:0b: [mem 0x20000000-0x201fffff] has been reserved [ 0.190182] system 00:0b: [mem 0x40004000-0x40004fff] has been reserved [ 0.190184] system 00:0b: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.190198] pnp: PnP ACPI: found 12 devices [ 0.190199] ACPI: bus type PNP unregistered [ 0.196070] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.196073] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.196076] pci 0000:00:01.0: bridge window [mem 0xf5000000-0xf70fffff] [ 0.196078] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xefffffff 64bit pref] [ 0.196082] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.196093] pci 0000:00:1c.1: PCI bridge to [bus 03] [ 0.196099] pci 0000:00:1c.1: bridge window [io 0xd000-0xdfff] [ 0.196104] pci 0000:00:1c.1: bridge window [mem 0xf7c00000-0xf7cfffff] [ 0.196111] pci 0000:00:1c.3: PCI bridge to [bus 04] [ 0.196116] pci 0000:00:1c.3: bridge window [mem 0xf7b00000-0xf7bfffff] [ 0.196124] pci 0000:05:00.0: PCI bridge to [bus 06] [ 0.196128] pci 0000:05:00.0: bridge window [io 0xc000-0xcfff] [ 0.196137] pci 0000:05:00.0: bridge window [mem 0xf7a00000-0xf7afffff] [ 0.196153] pci 0000:00:1c.5: PCI bridge to [bus 05-06] [ 0.196155] pci 0000:00:1c.5: bridge window [io 0xc000-0xcfff] [ 0.196160] pci 0000:00:1c.5: bridge window [mem 0xf7a00000-0xf7afffff] [ 0.196168] pci 0000:00:1c.6: PCI bridge to [bus 07] [ 0.196170] pci 0000:00:1c.6: bridge window [io 0xb000-0xbfff] [ 0.196175] pci 0000:00:1c.6: bridge window [mem 0xf7900000-0xf79fffff] [ 0.196182] pci 0000:00:1c.7: PCI bridge to [bus 08] [ 0.196185] pci 0000:00:1c.7: bridge window [io 0xa000-0xafff] [ 0.196189] pci 0000:00:1c.7: bridge window [mem 0xf7800000-0xf78fffff] [ 0.196197] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7] [ 0.196199] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff] [ 0.196200] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff] [ 0.196201] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.196202] pci_bus 0000:00: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.196203] pci_bus 0000:00: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.196205] pci_bus 0000:00: resource 10 [mem 0x000dc000-0x000dffff] [ 0.196206] pci_bus 0000:00: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.196207] pci_bus 0000:00: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.196208] pci_bus 0000:00: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.196209] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.196211] pci_bus 0000:01: resource 1 [mem 0xf5000000-0xf70fffff] [ 0.196212] pci_bus 0000:01: resource 2 [mem 0xe0000000-0xefffffff 64bit pref] [ 0.196213] pci_bus 0000:03: resource 0 [io 0xd000-0xdfff] [ 0.196214] pci_bus 0000:03: resource 1 [mem 0xf7c00000-0xf7cfffff] [ 0.196216] pci_bus 0000:04: resource 1 [mem 0xf7b00000-0xf7bfffff] [ 0.196217] pci_bus 0000:05: resource 0 [io 0xc000-0xcfff] [ 0.196218] pci_bus 0000:05: resource 1 [mem 0xf7a00000-0xf7afffff] [ 0.196219] pci_bus 0000:05: resource 4 [io 0x0000-0x0cf7] [ 0.196220] pci_bus 0000:05: resource 5 [io 0x0d00-0xffff] [ 0.196221] pci_bus 0000:05: resource 6 [mem 0x000a0000-0x000bffff] [ 0.196222] pci_bus 0000:05: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.196224] pci_bus 0000:05: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.196225] pci_bus 0000:05: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.196226] pci_bus 0000:05: resource 10 [mem 0x000dc000-0x000dffff] [ 0.196227] pci_bus 0000:05: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.196228] pci_bus 0000:05: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.196229] pci_bus 0000:05: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.196231] pci_bus 0000:06: resource 0 [io 0xc000-0xcfff] [ 0.196232] pci_bus 0000:06: resource 1 [mem 0xf7a00000-0xf7afffff] [ 0.196233] pci_bus 0000:06: resource 4 [io 0xc000-0xcfff] [ 0.196234] pci_bus 0000:06: resource 5 [mem 0xf7a00000-0xf7afffff] [ 0.196235] pci_bus 0000:06: resource 8 [io 0x0000-0x0cf7] [ 0.196236] pci_bus 0000:06: resource 9 [io 0x0d00-0xffff] [ 0.196238] pci_bus 0000:06: resource 10 [mem 0x000a0000-0x000bffff] [ 0.196239] pci_bus 0000:06: resource 11 [mem 0x000d0000-0x000d3fff] [ 0.196240] pci_bus 0000:06: resource 12 [mem 0x000d4000-0x000d7fff] [ 0.196241] pci_bus 0000:06: resource 13 [mem 0x000d8000-0x000dbfff] [ 0.196242] pci_bus 0000:06: resource 14 [mem 0x000dc000-0x000dffff] [ 0.196243] pci_bus 0000:06: resource 15 [mem 0x000e0000-0x000e3fff] [ 0.196245] pci_bus 0000:06: resource 16 [mem 0x000e4000-0x000e7fff] [ 0.196246] pci_bus 0000:06: resource 17 [mem 0xcfa00000-0xfeafffff] [ 0.196247] pci_bus 0000:07: resource 0 [io 0xb000-0xbfff] [ 0.196248] pci_bus 0000:07: resource 1 [mem 0xf7900000-0xf79fffff] [ 0.196249] pci_bus 0000:08: resource 0 [io 0xa000-0xafff] [ 0.196250] pci_bus 0000:08: resource 1 [mem 0xf7800000-0xf78fffff] [ 0.196267] NET: Registered protocol family 2 [ 0.196351] TCP established hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.196492] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.196598] TCP: Hash tables configured (established 131072 bind 65536) [ 0.196611] TCP: reno registered [ 0.196614] UDP hash table entries: 8192 (order: 6, 262144 bytes) [ 0.196646] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes) [ 0.196708] NET: Registered protocol family 1 [ 0.196757] RPC: Registered named UNIX socket transport module. [ 0.196759] RPC: Registered udp transport module. [ 0.196760] RPC: Registered tcp transport module. [ 0.196761] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.196768] pci 0000:00:02.0: Boot video device [ 0.229229] PCI: CLS 64 bytes, default 64 [ 0.229264] Trying to unpack rootfs image as initramfs... [ 0.284002] Freeing initrd memory: 4416K (ffff88007fbb0000 - ffff880080000000) [ 0.284018] DMAR: No ATSR found [ 0.284039] IOMMU 1 0xfed91000: using Queued invalidation [ 0.284041] IOMMU: Setting RMRR: [ 0.284051] IOMMU: Setting identity map for device 0000:00:1d.0 [0x8d86f000 - 0x8d899fff] [ 0.284073] IOMMU: Setting identity map for device 0000:00:1a.0 [0x8d86f000 - 0x8d899fff] [ 0.284089] IOMMU: Setting identity map for device 0000:00:14.0 [0x8d86f000 - 0x8d899fff] [ 0.284110] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.284117] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.284200] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.286013] RAPL PMU detected, hw unit 2^-16 Joules, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer [ 0.286395] AVX version of gcm_enc/dec engaged. [ 0.286784] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 0.286800] audit: initializing netlink subsys (disabled) [ 0.286812] audit: type=2000 audit(1399140815.274:1): initialized [ 0.286984] bounce pool size: 64 pages [ 0.287917] VFS: Disk quotas dquot_6.5.2 [ 0.287946] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.288061] NFS: Registering the id_resolver key type [ 0.288068] Key type id_resolver registered [ 0.288069] Key type id_legacy registered [ 0.288074] ROMFS MTD (C) 2007 Red Hat, Inc. [ 0.288093] msgmni has been set to 30072 [ 0.288892] async_tx: api initialized (async) [ 0.288914] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.288935] io scheduler noop registered [ 0.288936] io scheduler deadline registered [ 0.288954] io scheduler cfq registered (default) [ 0.289090] pcieport 0000:00:01.0: irq 42 for MSI/MSI-X [ 0.289578] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.289596] vesafb: mode is 800x600x8, linelength=832, pages=202 [ 0.289598] vesafb: scrolling: redraw [ 0.289599] vesafb: Pseudocolor: size=8:8:8:8, shift=0:0:0:0 [ 0.289701] vesafb: framebuffer at 0xd0000000, mapped to 0xffffc90004100000, using 975k, total 104384k [ 0.297086] Console: switching to colour frame buffer device 100x37 [ 0.305277] fb0: VESA VGA frame buffer device [ 0.305380] intel_idle: MWAIT substates: 0x1120 [ 0.305380] intel_idle: v0.4 model 0x3A [ 0.305381] intel_idle: lapic_timer_reliable_states 0xffffffff [ 0.305485] GHES: HEST is not enabled! [ 0.306647] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.308085] brd: module loaded [ 0.308238] ahci 0000:00:1f.2: version 3.0 [ 0.308319] ahci 0000:00:1f.2: irq 43 for MSI/MSI-X [ 0.319117] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0xd impl SATA mode [ 0.319296] ahci 0000:00:1f.2: flags: 64bit ncq led clo pio slum part ems apst [ 0.323564] scsi0 : ahci [ 0.326680] scsi1 : ahci [ 0.329688] scsi2 : ahci [ 0.332600] scsi3 : ahci [ 0.335430] scsi4 : ahci [ 0.338174] scsi5 : ahci [ 0.340731] ata1: SATA max UDMA/133 abar m2048@0xf7d36000 port 0xf7d36100 irq 43 [ 0.343395] ata2: DUMMY [ 0.345961] ata3: SATA max UDMA/133 abar m2048@0xf7d36000 port 0xf7d36200 irq 43 [ 0.348676] ata4: SATA max UDMA/133 abar m2048@0xf7d36000 port 0xf7d36280 irq 43 [ 0.351370] ata5: DUMMY [ 0.354009] ata6: DUMMY [ 0.356681] ahci 0000:03:00.0: irq 44 for MSI/MSI-X [ 0.356730] ahci 0000:03:00.0: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode [ 0.359509] ahci 0000:03:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs [ 0.362537] scsi6 : ahci [ 0.365330] scsi7 : ahci [ 0.367926] ata7: SATA max UDMA/133 abar m512@0xf7c10000 port 0xf7c10100 irq 44 [ 0.370685] ata8: SATA max UDMA/133 abar m512@0xf7c10000 port 0xf7c10180 irq 44 [ 0.373486] ahci 0000:08:00.0: irq 45 for MSI/MSI-X [ 0.373533] ahci 0000:08:00.0: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode [ 0.376323] ahci 0000:08:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs [ 0.379379] scsi8 : ahci [ 0.382250] scsi9 : ahci [ 0.384920] ata9: SATA max UDMA/133 abar m512@0xf7810000 port 0xf7810100 irq 45 [ 0.387686] ata10: SATA max UDMA/133 abar m512@0xf7810000 port 0xf7810180 irq 45 [ 0.391105] scsi10 : pata_legacy [ 0.393829] ata11: PATA max PIO4 cmd 0x1f0 ctl 0x3f6 irq 14 [ 0.662132] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 0.664922] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 0.667591] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 0.670237] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359) [ 0.672885] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT2._GTF] (Node ffff88041e04f2d0), AE_NOT_FOUND (20131218/psparse-536) [ 0.678218] ata3.00: ATAPI: HL-DT-ST DVDRAM GH22NS50, TN01, max UDMA/100 [ 0.681062] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359) [ 0.684126] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT0._GTF] (Node ffff88041e04f1e0), AE_NOT_FOUND (20131218/psparse-536) [ 0.690623] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359) [ 0.694115] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT3._GTF] (Node ffff88041e04f348) [ 0.696131] ata9: SATA link down (SStatus 0 SControl 300) [ 0.701136] , AE_NOT_FOUND<6>[ 0.701179] ata10: SATA link down (SStatus 0 SControl 300) [ 0.708170] (20131218/psparse-536) [ 0.711757] ata4.00: ATAPI: HL-DT-ST BD-RE BH14NS40, 1.01, max UDMA/100 [ 0.715538] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359) [ 0.719420] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT2._GTF] (Node ffff88041e04f2d0), AE_NOT_FOUND (20131218/psparse-536) [ 0.727197] ata3.00: configured for UDMA/100 [ 0.731264] ata1.00: ATA-7: SAMSUNG HD080HJ/P, ZH100-34, max UDMA7 [ 0.735257] ata1.00: 156250000 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 0.739341] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359) [ 0.743433] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT3._GTF] (Node ffff88041e04f348), AE_NOT_FOUND (20131218/psparse-536) [ 0.751643] ata4.00: configured for UDMA/100 [ 0.760719] ACPI Error: [DSSP] Namespace lookup failure, AE_NOT_FOUND (20131218/psargs-359) [ 0.765068] ACPI Error: Method parse/execution failed [\_SB_.PCI0.SAT0.SPT0._GTF] (Node ffff88041e04f1e0), AE_NOT_FOUND (20131218/psparse-536) [ 0.773876] ata1.00: configured for UDMA/133 [ 0.833136] ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.837532] ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 0.842064] ata8.00: ATA-8: ST2000DL003-9VT166, CC32, max UDMA/133 [ 0.846537] ata8.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 0.851070] ata7.00: ATA-8: ST3500418AS, CC38, max UDMA/133 [ 0.855392] ata7.00: 976773168 sectors, multi 16: LBA48 NCQ (depth 31/32) [ 0.860254] ata8.00: configured for UDMA/133 [ 0.864716] ata7.00: configured for UDMA/133 [ 1.286130] tsc: Refined TSC clocksource calibration: 3002.953 MHz [ 2.290320] Switched to clocksource tsc [ 3.322312] floppy0: no floppy controllers found [ 3.326349] scsi 0:0:0:0: Direct-Access ATA SAMSUNG HD080HJ/ ZH10 PQ: 0 ANSI: 5 [ 3.330315] sd 0:0:0:0: [sda] 156250000 512-byte logical blocks: (80.0 GB/74.5 GiB) [ 3.334167] sd 0:0:0:0: [sda] Write Protect is off [ 3.337946] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 3.337957] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.337979] scsi 2:0:0:0: CD-ROM HL-DT-ST DVDRAM GH22NS50 TN01 PQ: 0 ANSI: 5 [ 3.353791] sda: sda1 sda3 sda4 [ 3.357854] sd 0:0:0:0: [sda] Attached SCSI disk [ 3.365291] sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray [ 3.368997] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 3.372768] sr 2:0:0:0: Attached scsi CD-ROM sr0 [ 3.373177] scsi 3:0:0:0: CD-ROM HL-DT-ST BD-RE BH14NS40 1.01 PQ: 0 ANSI: 5 [ 3.377682] sr1: scsi3-mmc drive: 153x/306x writer dvd-ram cd/rw xa/form2 cdda tray [ 3.381622] sr 3:0:0:0: Attached scsi CD-ROM sr1 [ 3.381877] scsi 6:0:0:0: Direct-Access ATA ST3500418AS CC38 PQ: 0 ANSI: 5 [ 3.385838] sd 6:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 3.389624] sd 6:0:0:0: [sdb] Write Protect is off [ 3.389635] scsi 7:0:0:0: Direct-Access ATA ST2000DL003-9VT1 CC32 PQ: 0 ANSI: 5 [ 3.389735] sd 7:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) [ 3.389754] sd 7:0:0:0: [sdc] Write Protect is off [ 3.389755] sd 7:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 3.389763] sd 7:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.411312] sd 6:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 3.411323] sd 6:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 3.411335] sdc: sdc1 sdc2 sdc3 [ 3.411642] sd 7:0:0:0: [sdc] Attached SCSI disk [ 3.435094] sdb: sdb1 sdb2 sdb3 [ 3.438999] sd 6:0:0:0: [sdb] Attached SCSI disk [ 3.443001] scsi11 : pata_legacy [ 3.446568] ata12: PATA max PIO4 cmd 0x170 ctl 0x376 irq 15 [ 3.601504] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 3.605604] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.609295] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.613049] mousedev: PS/2 mouse device common for all mice [ 3.616696] rtc_cmos 00:05: RTC can wake from S4 [ 3.620425] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.624007] rtc_cmos 00:05: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 3.627596] md: linear personality registered for level -1 [ 3.631105] md: raid0 personality registered for level 0 [ 3.634573] md: raid1 personality registered for level 1 [ 3.637938] md: raid10 personality registered for level 10 [ 3.641443] md: raid6 personality registered for level 6 [ 3.644779] md: raid5 personality registered for level 5 [ 3.647992] md: raid4 personality registered for level 4 [ 3.651082] md: multipath personality registered for level -4 [ 3.654177] device-mapper: uevent: version 1.0.3 [ 3.657269] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com [ 3.660357] Intel P-state driver initializing. [ 3.663393] Intel pstate controlling: cpu 0 [ 3.666361] Intel pstate controlling: cpu 1 [ 3.669207] Intel pstate controlling: cpu 2 [ 3.671961] Intel pstate controlling: cpu 3 [ 3.674703] TCP: cubic registered [ 3.677170] Initializing XFRM netlink socket [ 3.679568] NET: Registered protocol family 17 [ 3.681896] Key type dns_resolver registered [ 3.684630] registered taskstats version 1 [ 3.687252] regulator-dummy: disabling [ 3.741594] Freeing unused kernel memory: 1088K (ffffffff81aad000 - ffffffff81bbd000) [ 3.746033] Write protecting the kernel read-only data: 10240k [ 3.751286] Freeing unused kernel memory: 168K (ffff8800015d6000 - ffff880001600000) [ 3.761439] Freeing unused kernel memory: 2016K (ffff880001808000 - ffff880001a00000) [ 3.778921] udevd[177]: starting version 182 [ 3.909538] random: nonblocking pool is initialized [ 5.366297] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null) [ 6.535776] loop: module loaded [ 6.741465] udevd[302]: starting version 182 [ 7.436152] microcode: CPU0 sig=0x306a9, pf=0x2, revision=0x12 [ 7.451666] microcode: CPU1 sig=0x306a9, pf=0x2, revision=0x12 [ 7.456544] microcode: CPU2 sig=0x306a9, pf=0x2, revision=0x12 [ 7.461232] microcode: CPU3 sig=0x306a9, pf=0x2, revision=0x12 [ 7.465689] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 7.580269] input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input2 [ 7.585231] ACPI: Power Button [PWRB] [ 7.590320] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3 [ 7.592892] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20131218/utaddress-258) [ 7.592893] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 7.592898] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20131218/utaddress-258) [ 7.592899] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 7.592902] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x000000000000051f (\LED_) (20131218/utaddress-258) [ 7.592904] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20131218/utaddress-258) [ 7.592905] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 7.592905] lpc_ich: Resource conflict(s) found affecting gpio_ich [ 7.694470] ACPI: Power Button [PWRF] [ 7.706431] atl1c 0000:07:00.0: version 1.0.1.1-NAPI [ 7.758058] ACPI Warning: SystemIO range 0x000000000000f040-0x000000000000f05f conflicts with OpRegion 0x000000000000f040-0x000000000000f04f (\_SB_.PCI0.SBUS.SMBI) (20131218/utaddress-258) [ 7.773406] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 7.966567] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 7.974710] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1 [ 7.982932] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported [ 7.982959] xhci_hcd 0000:00:14.0: irq 46 for MSI/MSI-X [ 7.983029] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 7.990685] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 7.995799] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 8.002720] ehci-pci: EHCI PCI platform driver [ 8.014456] usb usb1: Product: xHCI Host Controller [ 8.021993] usb usb1: Manufacturer: Linux 3.14.2 xhci_hcd [ 8.029408] usb usb1: SerialNumber: 0000:00:14.0 [ 8.036423] hub 1-0:1.0: USB hub found [ 8.039177] Linux agpgart interface v0.103 [ 8.049809] hub 1-0:1.0: 4 ports detected [ 8.056273] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 8.059294] wmi: Mapper loaded [ 8.059334] pps_core: LinuxPPS API ver. 1 registered [ 8.059334] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 8.072047] PTP clock support registered [ 8.091050] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ 8.096048] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k [ 8.096048] e1000e: Copyright(c) 1999 - 2013 Intel Corporation. [ 8.106748] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 [ 8.111574] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 8.116295] usb usb2: Product: xHCI Host Controller [ 8.120786] usb usb2: Manufacturer: Linux 3.14.2 xhci_hcd [ 8.125158] usb usb2: SerialNumber: 0000:00:14.0 [ 8.129602] hub 2-0:1.0: USB hub found [ 8.133700] hub 2-0:1.0: 4 ports detected [ 8.137998] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 8.143612] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 3 [ 8.149229] ACPI: Fan [FAN0] (off) [ 8.149248] ehci-pci 0000:00:1a.0: debug port 2 [ 8.153167] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 8.153187] ehci-pci 0000:00:1a.0: irq 16, io mem 0xf7d38000 [ 8.159159] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 8.159194] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 [ 8.159196] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 8.159197] usb usb3: Product: EHCI Host Controller [ 8.159199] usb usb3: Manufacturer: Linux 3.14.2 ehci_hcd [ 8.159200] usb usb3: SerialNumber: 0000:00:1a.0 [ 8.159347] hub 3-0:1.0: USB hub found [ 8.159353] hub 3-0:1.0: 2 ports detected [ 8.159619] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 8.159674] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 4 [ 8.159687] ehci-pci 0000:00:1d.0: debug port 2 [ 8.163604] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 8.163619] ehci-pci 0000:00:1d.0: irq 23, io mem 0xf7d37000 [ 8.169159] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 8.169188] usb usb4: New USB device found, idVendor=1d6b, idProduct=0002 [ 8.169189] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 8.169190] usb usb4: Product: EHCI Host Controller [ 8.169192] usb usb4: Manufacturer: Linux 3.14.2 ehci_hcd [ 8.169193] usb usb4: SerialNumber: 0000:00:1d.0 [ 8.169327] hub 4-0:1.0: USB hub found [ 8.169334] hub 4-0:1.0: 2 ports detected [ 8.169617] mei_me 0000:00:16.0: irq 47 for MSI/MSI-X [ 8.179432] [drm] Initialized drm 1.1.0 20060810 [ 8.268729] ACPI: Fan [FAN1] (off) [ 8.269408] e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 8.269429] e1000e 0000:00:19.0: irq 48 for MSI/MSI-X [ 8.281495] ACPI: Fan [FAN2] (off) [ 8.285682] ACPI: Fan [FAN3] (off) [ 8.289827] ACPI: Fan [FAN4] (off) [ 8.300765] thermal LNXTHERM:00: registered as thermal_zone0 [ 8.305258] ACPI: Thermal Zone [TZ00] (28 C) [ 8.309988] thermal LNXTHERM:01: registered as thermal_zone1 [ 8.314526] ACPI: Thermal Zone [TZ01] (30 C) [ 8.351233] firewire_ohci 0000:06:01.0: added OHCI v1.10 device as card 0, 4 IR + 8 IT contexts, quirks 0x11 [ 8.411175] usb 1-4: new low-speed USB device number 2 using xhci_hcd [ 8.529359] e1000e 0000:00:19.0 eth1: registered PHC clock [ 8.534400] e1000e 0000:00:19.0 eth1: (PCI Express:2.5GT/s:Width x1) 90:2b:34:39:9e:ad [ 8.539538] e1000e 0000:00:19.0 eth1: Intel(R) PRO/1000 Network Connection [ 8.544508] e1000e 0000:00:19.0 eth1: MAC: 10, PHY: 11, PBA No: FFFFFF-0FF [ 8.550596] [drm] Memory usable by graphics device = 2048M [ 8.555463] checking generic (d0000000 65f0000) vs hw (d0000000 10000000) [ 8.555465] fb: conflicting fb hw usage inteldrmfb vs VESA VGA - removing generic driver [ 8.560905] Console: switching to colour dummy device 80x25 [ 8.583427] usb 1-4: New USB device found, idVendor=04f3, idProduct=0801 [ 8.583433] usb 1-4: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 8.583517] usb 1-4: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 8.583524] usb 1-4: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [ 8.606219] i915 0000:00:02.0: irq 49 for MSI/MSI-X [ 8.606227] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 8.606231] [drm] Driver supports precise vblank timestamp query. [ 8.606294] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none [ 8.626354] [drm] Wrong MCH_SSKPD value: 0x20100406 [ 8.626358] [drm] This can cause pipe underruns and display issues. [ 8.626360] [drm] Please upgrade your BIOS to fix this. [ 8.651677] fbcon: inteldrmfb (fb0) is primary device [ 8.687695] usb 2-3: new SuperSpeed USB device number 2 using xhci_hcd [ 8.719763] cfg80211: Calling CRDA to update world regulatory domain [ 8.751917] usb 2-3: New USB device found, idVendor=2109, idProduct=0810 [ 8.751919] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 8.751921] usb 2-3: Product: 4-Port USB 3.0 Hub [ 8.751922] usb 2-3: Manufacturer: VIA Labs, Inc. [ 8.772738] hub 2-3:1.0: USB hub found [ 8.776085] hub 2-3:1.0: 4 ports detected [ 8.814223] Console: switching to colour frame buffer device 180x56 [ 8.818959] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device [ 8.818995] i915 0000:00:02.0: registered panic notifier [ 8.824499] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 8.824853] acpi device:66: registered as cooling_device10 [ 8.824978] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/LNXVIDEO:00/input/input5 [ 8.825146] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0 [ 8.825612] snd_hda_intel 0000:00:1b.0: irq 50 for MSI/MSI-X [ 8.861258] firewire_core 0000:06:01.0: created device fw0: GUID 0049e550cda64b00, S400 [ 8.898050] ath: EEPROM regdomain: 0x6a [ 8.898053] ath: EEPROM indicates we should expect a direct regpair map [ 8.898056] ath: Country alpha2 being used: 00 [ 8.898057] ath: Regpair used: 0x6a [ 8.898837] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' [ 8.899024] ieee80211 phy0: Atheros AR9462 Rev:2 mem=0xffffc90004480000, irq=19 [ 8.915833] usb 2-4: new SuperSpeed USB device number 3 using xhci_hcd [ 8.980242] usb 2-4: New USB device found, idVendor=2109, idProduct=0810 [ 8.980287] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 8.980329] usb 2-4: Product: 4-Port USB 3.0 Hub [ 8.980358] usb 2-4: Manufacturer: VIA Labs, Inc. [ 8.988807] i2c /dev entries driver [ 9.003347] hub 2-4:1.0: USB hub found [ 9.006782] hub 2-4:1.0: 4 ports detected [ 9.146202] usb 3-1: new high-speed USB device number 2 using ehci-pci [ 9.185839] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input15 [ 9.185976] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input14 [ 9.186084] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13 [ 9.186196] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12 [ 9.187282] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11 [ 9.188872] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10 [ 9.190445] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9 [ 9.192000] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8 [ 9.193553] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7 [ 9.195110] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6 [ 9.260650] usb 3-1: New USB device found, idVendor=8087, idProduct=0024 [ 9.262356] usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 9.264405] hub 3-1:1.0: USB hub found [ 9.266230] hub 3-1:1.0: 6 ports detected [ 9.309531] hidraw: raw HID events driver (C) Jiri Kosina [ 9.371185] usb 4-1: new high-speed USB device number 2 using ehci-pci [ 9.486646] usb 4-1: New USB device found, idVendor=8087, idProduct=0024 [ 9.488338] usb 4-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 9.490310] hub 4-1:1.0: USB hub found [ 9.492127] hub 4-1:1.0: 8 ports detected [ 9.516451] cfg80211: World regulatory domain updated: [ 9.518097] cfg80211: DFS Master region: unset [ 9.518125] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp) [ 9.521303] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 9.522913] cfg80211: (2457000 KHz - 2482000 KHz @ 20000 KHz), (300 mBi, 2000 mBm) [ 9.524501] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm) [ 9.526067] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 9.527625] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm) [ 9.568256] usb 3-1.3: new high-speed USB device number 3 using ehci-pci [ 9.628152] usbcore: registered new interface driver usbhid [ 9.629701] usbhid: USB HID core driver [ 9.657128] usb 3-1.3: New USB device found, idVendor=05e3, idProduct=0605 [ 9.658674] usb 3-1.3: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 9.660213] usb 3-1.3: Product: USB2.0 Hub [ 9.662193] hub 3-1.3:1.0: USB hub found [ 9.664002] hub 3-1.3:1.0: 4 ports detected [ 9.714546] [drm] Enabling RC6 states: RC6 on, RC6p on, RC6pp off [ 9.740246] usb 3-1.4: new high-speed USB device number 4 using ehci-pci [ 9.846628] usb 3-1.4: New USB device found, idVendor=0ac8, idProduct=3450 [ 9.848192] usb 3-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 9.849744] usb 3-1.4: Product: Vimicro USB Camera (Altair) [ 9.851285] usb 3-1.4: Manufacturer: Vimicro [ 9.852413] input: HID 04f3:0801 as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.0/0003:04F3:0801.0001/input/input16 [ 9.852695] hid-generic 0003:04F3:0801.0001: input,hidraw0: USB HID v1.10 Keyboard [HID 04f3:0801] on usb-0000:00:14.0-4/input0 [ 9.856922] input: HID 04f3:0801 as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.1/0003:04F3:0801.0002/input/input17 [ 9.860061] hid-generic 0003:04F3:0801.0002: input,hidraw1: USB HID v1.10 Mouse [HID 04f3:0801] on usb-0000:00:14.0-4/input1 [ 9.931236] usb 3-1.5: new high-speed USB device number 5 using ehci-pci [ 10.019355] usb 3-1.5: New USB device found, idVendor=2109, idProduct=0810 [ 10.020976] usb 3-1.5: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 10.022588] usb 3-1.5: Product: USB2.0 Hub [ 10.024515] hub 3-1.5:1.0: USB hub found [ 10.026355] hub 3-1.5:1.0: 4 ports detected [ 10.039206] Linux video capture interface: v2.00 [ 10.107235] usb 3-1.6: new high-speed USB device number 6 using ehci-pci [ 10.143896] uvcvideo: Found UVC 1.00 device Vimicro USB Camera (Altair) (0ac8:3450) [ 10.147333] input: Vimicro USB Camera (Altair) as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1.4/3-1.4:1.0/input/input18 [ 10.149091] usbcore: registered new interface driver uvcvideo [ 10.150767] USB Video Class driver (1.1.1) [ 10.195785] usb 3-1.6: New USB device found, idVendor=2109, idProduct=0810 [ 10.197458] usb 3-1.6: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 10.199128] usb 3-1.6: Product: USB2.0 Hub [ 10.201202] hub 3-1.6:1.0: USB hub found [ 10.202996] hub 3-1.6:1.0: 4 ports detected [ 10.228181] [drm] GMBUS [i915 gmbus panel] timed out, falling back to bit banging on pin 3 [ 10.284289] usb 4-1.6: new high-speed USB device number 3 using ehci-pci [ 10.289182] [drm] GMBUS [i915 gmbus ssc] timed out, falling back to bit banging on pin 1 [ 10.375394] usb 4-1.6: New USB device found, idVendor=05e3, idProduct=0716 [ 10.377077] usb 4-1.6: New USB device strings: Mfr=0, Product=1, SerialNumber=2 [ 10.378739] usb 4-1.6: Product: USB Storage [ 10.380398] usb 4-1.6: SerialNumber: 000000009727 [ 10.402102] usb-storage 4-1.6:1.0: USB Mass Storage device detected [ 10.404082] scsi12 : usb-storage 4-1.6:1.0 [ 10.405833] usbcore: registered new interface driver usb-storage [ 10.456235] usb 4-1.7: new full-speed USB device number 4 using ehci-pci [ 10.533852] usb 4-1.7: string descriptor 0 read error: -22 [ 10.535546] usb 4-1.7: New USB device found, idVendor=04ca, idProduct=3006 [ 10.537224] usb 4-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 10.606803] Bluetooth: Core ver 2.18 [ 10.608501] NET: Registered protocol family 31 [ 10.610164] Bluetooth: HCI device and connection manager initialized [ 10.611865] Bluetooth: HCI socket layer initialized [ 10.613585] Bluetooth: L2CAP socket layer initialized [ 10.615315] Bluetooth: SCO socket layer initialized [ 10.615340] usb 3-1.3.4: new low-speed USB device number 7 using ehci-pci [ 10.651159] usbcore: registered new interface driver btusb [ 10.656054] usbcore: registered new interface driver ath3k [ 10.708515] usb 3-1.3.4: New USB device found, idVendor=0bc7, idProduct=0004 [ 10.710244] usb 3-1.3.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 10.711983] usb 3-1.3.4: Product: USB Receiver [ 10.713692] usb 3-1.3.4: Manufacturer: X10 Wireless Technology Inc [ 10.814199] Registered IR keymap rc-ati-x10 [ 10.816029] input: X10 Wireless Technology Inc USB Receiver as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1.3/3-1.3.4/3-1.3.4:1.0/rc/rc0/input19 [ 10.817931] rc0: X10 Wireless Technology Inc USB Receiver as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1.3/3-1.3.4/3-1.3.4:1.0/rc/rc0 [ 10.819812] input: X10 Wireless Technology Inc USB Receiver mouse as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1.3/3-1.3.4/3-1.3.4:1.0/input/input20 [ 10.821919] usbcore: registered new interface driver ati_remote [ 10.902866] fuse init (API version 7.22) [ 11.407682] scsi 12:0:0:0: Direct-Access Generic STORAGE DEVICE-A 9727 PQ: 0 ANSI: 0 [ 11.412102] scsi 12:0:0:1: Direct-Access Generic STORAGE DEVICE-A 9727 PQ: 0 ANSI: 0 [ 11.414125] sd 12:0:0:0: [sdd] Attached SCSI removable disk [ 11.418118] scsi 12:0:0:2: Direct-Access Generic STORAGE DEVICE-A 9727 PQ: 0 ANSI: 0 [ 11.423617] scsi 12:0:0:3: Direct-Access Generic STORAGE DEVICE-A 9727 PQ: 0 ANSI: 0 CCS [ 11.425883] sd 12:0:0:1: [sde] Attached SCSI removable disk [ 11.429368] scsi 12:0:0:4: Direct-Access Generic STORAGE DEVICE-A 9727 PQ: 0 ANSI: 0 [ 11.434365] sd 12:0:0:2: [sdf] Attached SCSI removable disk [ 11.442237] sd 12:0:0:3: [sdg] Attached SCSI removable disk [ 11.450737] sd 12:0:0:4: [sdh] Attached SCSI removable disk [ 11.916324] EXT4-fs (sda3): re-mounted. Opts: (null) [ 12.374753] ppdev: user-space parallel port driver [ 12.382409] lp: driver loaded but no devices found [ 15.609650] EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: (null) [ 20.465461] NET: Registered protocol family 10 [ 31.151029] e1000e 0000:00:19.0: irq 48 for MSI/MSI-X [ 31.251373] e1000e 0000:00:19.0: irq 48 for MSI/MSI-X [ 31.251481] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready [ 31.261402] atl1c 0000:07:00.0: irq 51 for MSI/MSI-X [ 31.261949] atl1c 0000:07:00.0: atl1c: eth0 NIC Link is Up<100 Mbps Full Duplex> [ 32.787211] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx [ 32.787219] e1000e 0000:00:19.0 eth1: 10/100 speed: disabling TSO [ 32.787255] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready [ 51.481723] Guest personality initialized and is inactive [ 51.481982] VMCI host device registered (name=vmci, major=10, minor=56) [ 51.481985] Initialized host personality [ 51.520320] NET: Registered protocol family 40 [ 53.298288] tun: Universal TUN/TAP device driver, 1.6 [ 53.298292] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 56.579323] Ebtables v2.0 registered [ 57.557764] ip_tables: (C) 2000-2006 Netfilter Core Team [ 58.163779] ip6_tables: (C) 2000-2006 Netfilter Core Team [ 79.261291] Bridge firewalling registered [ 79.289732] device virbr4-nic entered promiscuous mode [ 79.429923] virbr4: port 1(virbr4-nic) entered listening state [ 79.429939] virbr4: port 1(virbr4-nic) entered listening state [ 79.429986] IPv6: ADDRCONF(NETDEV_UP): virbr4: link is not ready [ 79.442113] virbr4: port 1(virbr4-nic) entered disabled state [ 79.544700] device virbr2-nic entered promiscuous mode [ 79.624098] virbr2: port 1(virbr2-nic) entered listening state [ 79.624113] virbr2: port 1(virbr2-nic) entered listening state [ 79.624179] IPv6: ADDRCONF(NETDEV_UP): virbr2: link is not ready [ 79.639076] virbr2: port 1(virbr2-nic) entered disabled state [ 79.712701] device virbr3-nic entered promiscuous mode [ 79.758094] virbr3: port 1(virbr3-nic) entered listening state [ 79.758109] virbr3: port 1(virbr3-nic) entered listening state [ 79.758129] IPv6: ADDRCONF(NETDEV_UP): virbr3: link is not ready [ 79.764835] virbr3: port 1(virbr3-nic) entered disabled state [ 79.819701] device virbr1-nic entered promiscuous mode [ 79.871879] virbr1: port 1(virbr1-nic) entered listening state [ 79.871893] virbr1: port 1(virbr1-nic) entered listening state [ 79.871965] IPv6: ADDRCONF(NETDEV_UP): virbr1: link is not ready [ 79.883150] virbr1: port 1(virbr1-nic) entered disabled state [ 80.001807] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 80.061613] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready

Then

#vfio-pci 0000:02:00.0 # qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/media/800/destroy_varios/donwload2/virtualizacion/MSI.7600GT.256.060310.rom -bios /var/lib/libvirt/bios/seabios-hp2.bin -vga none -drive file=/media/Utils/Sistemas\ Operativos/linux/Express-3.1-rc5-x86_64.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -vnc 127.0.0.1:0 -vga none

and the result is in my "intel video" that color changed and in my "virtual Nvidia video" has not signal, the second monitor is in stand by mode . If a go to Console 1 (CONTROL+ALT+F1) and back to Grafic consle (ALT+F7) all seems fine.

All I see in dmesg is

l# dmesg [ 1873.878446] vfio-pci 0000:02:00.0: enabling device (0400 -> 0403)

Any clue why the nvdia is not sending signal?

Then when I open the VNC session all is in blank.

Last edited by Destroy (2014-05-06 00:51:03)

Offline

#1656 2014-05-06 00:53:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Destroy wrote:

Any clue why the nvdia is not sending signal?

You need the i915 vga arbiter patches


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1657 2014-05-07 05:23:54

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You need the i915 vga arbiter patches

Thanks!!! It is sending signal!!!


Now I just need a mouse and keyboard usb to attach to VM

Offline

#1658 2014-05-07 12:48:48

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks nbhs for this excellent tutorial! I've wanted to try kvm for a long time, but only recently found time to do it. I have successfully passed through a Sapphire Radeon HD 7770 using primary passthrough to a Windows 7 Pro 64 bit guest. Below some more info:
OS: Ubuntu 14.04
Guest OS: Windows 7 Pro 64 bit
M/B: Asus Sabertooth X79
CPU: Intel i7 3930K (LGA 2011)
Host OS graphics card: Nvidia Quadro 2000 with Nvidia proprietary driver
bridged network configuration
vfio driver for network and disk
Disk is a LVM drive using raw format

Here some remarks and questions:
Passmark PerformanceTest benchmark only works in SAFEMODE, else Windows BSODs. Under Xen there is no such issue. Furthermore Passmark is unable to detect the SSE extensions of the CPU (again under Xen it works fine). CPU-Z, however, does recognize the SSE extensions.
PCmark has similar issues.
Is there anything I do wrong with the configuration?

EDIT: Here the configuration:
sudo qemu-system-x86_64 \
-bios /usr/share/qemu/bios.bin -vga none \
-name win7 \
-cpu host \
-smp 10,sockets=1,cores=5,threads=2 \
-enable-kvm \
-m 24G \
-rtc clock=host \
-vga none \
-serial null \
-parallel null \
-monitor none \
-display none \
-k en-us \
-machine type=q35,accel=kvm \
-boot order=cd \
-device ahci,id=ahci \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/external/win7amd,cache=none,if=virtio \
-drive file=/home/user/Win7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/user/virtio-win-0.1-74.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-net nic,model=virtio,macaddr=00:16:3e:00:03:03 -net tap

Last edited by powerhouse (2014-05-07 18:54:16)

Offline

#1659 2014-05-07 13:19:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

Thanks nbhs for this excellent tutorial! I've wanted to try kvm for a long time, but only recently found time to do it. I have successfully passed through a Sapphire Radeon HD 7770 using primary passthrough to a Windows 7 Pro 64 bit guest. Below some more info:
OS: Ubuntu 14.04
Guest OS: Windows 7 Pro 64 bit
M/B: Asus Sabertooth X79
CPU: Intel i7 3930K (LGA 2011)
Host OS graphics card: Nvidia Quadro 2000 with Nvidia proprietary driver
bridged network configuration
vfio driver for network and disk
Disk is a LVM drive using raw format

Here some remarks and questions:
Passmark PerformanceTest benchmark only works in SAFEMODE, else Windows BSODs. Under Xen there is no such issue. Furthermore Passmark is unable to detect the SSE extensions of the CPU (again under Xen it works fine). CPU-Z, however, does recognize the SSE extensions.
PCmark has similar issues.
Is there anything I do wrong with the configuration?

See http://www.passmark.com/forum/showthrea … r-Number-4 comment #6 and beyond.  My suspicion is that Passmark makes some platform assumptions based on the CPU family.  It would be a great data point if you could confirm or deny that with Xen.  Thanks.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1660 2014-05-07 14:46:17

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

adejour wrote:
gneville wrote:

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

adejour wrote:

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

I'm also having this problem.
Do you have any updates regarding it ?
This is my lspci diff before and after restart.

--- /ok 2014-05-06 21:17:34.720433511 +0300 +++ /nok 2014-05-06 21:13:16.259217401 +0300 @@ -1,9 +1,8 @@ 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 7790 Dual-X OC - Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ + Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- - Latency: 0, Cache Line Size: 64 bytes - Interrupt: pin A routed to IRQ 71 + Interrupt: pin A routed to IRQ 16 Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at f0000000 (64-bit, prefetchable) [size=8M] Region 4: I/O ports at e000 [size=256] @@ -16,8 +15,8 @@ Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- - DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- - RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ + DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- + RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us @@ -32,14 +31,14 @@ Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- - Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ - Address: 00000000fee004b8 Data: 0000 + Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ + Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- + CESta: RxErr- BadTLP+ BadDLLP- Rollover+ Timeout+ NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19

Last edited by anickname (2014-05-07 14:47:10)

Offline

#1661 2014-05-07 14:59:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:
adejour wrote:
gneville wrote:

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

I'm also having this problem.
Do you have any updates regarding it ?

It's complicated.  The only way we can reset these GPUs is via a bus reset, but there are multiple devices on the bus.  We therefore have an interface for QEMU to say "I own all of these devices and I want a reset", that's what happens on startup and on each VM reset.  When QEMU is shutdown, there is no final reset (maybe there should be).  VFIO sees the devices getting released, but it can't tell the difference between a single device being hot-unplugged from the guest and all of the devices getting released.  So, we probably need to add another reset when the group is being closed, but the device interface doesn't really have any visibility to that event.  Code contributions welcome...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1662 2014-05-07 15:16:20

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
anickname wrote:
adejour wrote:

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

I'm also having this problem.
Do you have any updates regarding it ?

It's complicated.  The only way we can reset these GPUs is via a bus reset, but there are multiple devices on the bus.  We therefore have an interface for QEMU to say "I own all of these devices and I want a reset", that's what happens on startup and on each VM reset.  When QEMU is shutdown, there is no final reset (maybe there should be).  VFIO sees the devices getting released, but it can't tell the difference between a single device being hot-unplugged from the guest and all of the devices getting released.  So, we probably need to add another reset when the group is being closed, but the device interface doesn't really have any visibility to that event.  Code contributions welcome...

Thanks for the answer.
Still, I can't understand what is the difference between my setup and a setup which doesn't have this problem.
Looking at lspci's output the video card is the only device on that bus and in that specific group :

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:02.0 Display controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002 02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 05:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01) 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 0a:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 0b:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller 0c:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)
### Group 0 ### 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) ### Group 2 ### 00:02.0 Display controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) ### Group 6 ### 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) ### Group 7 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) ### Group 8 ### 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) ### Group 9 ### 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) ### Group 12 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002 ### Group 13 ### 02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) ### Group 14 ### 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) ### Group 15 ### 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 16 ### 05:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 17 ### 05:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 18 ### 05:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 19 ### 05:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 20 ### 05:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 21 ### 05:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 22 ### 05:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 23 ### 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 24 ### 07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01) ### Group 25 ### 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 26 ### 0a:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) ### Group 27 ### 0b:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller ### Group 28 ### 0c:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)

Offline

#1663 2014-05-07 15:20:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:
aw wrote:
anickname wrote:

I'm also having this problem.
Do you have any updates regarding it ?

It's complicated.  The only way we can reset these GPUs is via a bus reset, but there are multiple devices on the bus.  We therefore have an interface for QEMU to say "I own all of these devices and I want a reset", that's what happens on startup and on each VM reset.  When QEMU is shutdown, there is no final reset (maybe there should be).  VFIO sees the devices getting released, but it can't tell the difference between a single device being hot-unplugged from the guest and all of the devices getting released.  So, we probably need to add another reset when the group is being closed, but the device interface doesn't really have any visibility to that event.  Code contributions welcome...

Thanks for the answer.
Still, I can't understand what is the difference between my setup and a setup which doesn't have this problem.
Looking at lspci's output the video card is the only device on that bus and in that specific group :

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002
### Group 12 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002

The only way you would not have this problem would be if your VGA device was a single function device (ie. it lacked an audio function).  For our purposes, function 0 and function 1 are two separate devices on the same bus.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1664 2014-05-07 15:38:17

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

Regarding "hv-time" problem do you know if will it be fixed in 3.15 ?

https://bugzilla.kernel.org/show_bug.cgi?id=73721

Thanks.

Offline

#1665 2014-05-07 15:59:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

@aw

Regarding "hv-time" problem do you know if will it be fixed in 3.15 ?

https://bugzilla.kernel.org/show_bug.cgi?id=73721

I don't know, but I just poked one of the KVM maintainers about it and he's going to investigate.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1666 2014-05-07 16:18:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
anickname wrote:

@aw

Regarding "hv-time" problem do you know if will it be fixed in 3.15 ?

https://bugzilla.kernel.org/show_bug.cgi?id=73721

I don't know, but I just poked one of the KVM maintainers about it and he's going to investigate.

Apparently it's already fixed in current 3.15-rc and Paolo plans send it for stable as well.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1667 2014-05-07 18:08:46

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just bought a SSD for my virtual Win7 because Steam keeps telling me that my HDD is busy when downloading games/updates,
but now I wonder what might be the best way to pass it to the VM? (enabling TRIM and stuff)

I can't pass the whole SATA-Controller, so I guess passing the device using Q35 sata controller for installation and VIRTIO after installation would be the way to go? (I never managed to use VIRTIO during installation).

Any experiences doing this? Will TRIM work?
If not, anyone passing through a pcie-sata controller card? Recommendations for buying one?
I tried a Transcend TS-PDC3 Combocard last year, didn't exactly work since it always lost connection to the drives.

Offline

#1668 2014-05-07 18:51:25

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
powerhouse wrote:

Thanks nbhs for this excellent tutorial! I've wanted to try kvm for a long time, but only recently found time to do it. I have successfully passed through a Sapphire Radeon HD 7770 using primary passthrough to a Windows 7 Pro 64 bit guest. Below some more info:
OS: Ubuntu 14.04
Guest OS: Windows 7 Pro 64 bit
M/B: Asus Sabertooth X79
CPU: Intel i7 3930K (LGA 2011)
Host OS graphics card: Nvidia Quadro 2000 with Nvidia proprietary driver
bridged network configuration
vfio driver for network and disk
Disk is a LVM drive using raw format

Here some remarks and questions:
Passmark PerformanceTest benchmark only works in SAFEMODE, else Windows BSODs. Under Xen there is no such issue. Furthermore Passmark is unable to detect the SSE extensions of the CPU (again under Xen it works fine). CPU-Z, however, does recognize the SSE extensions.
PCmark has similar issues.
Is there anything I do wrong with the configuration?

See http://www.passmark.com/forum/showthrea … r-Number-4 comment #6 and beyond.  My suspicion is that Passmark makes some platform assumptions based on the CPU family.  It would be a great data point if you could confirm or deny that with Xen.  Thanks.

Thanks for the link. I had read that thread and tried -cpu=SandyBridge, but it didn't make a difference - the same BSOD when running Passmark. I did not try any model= parameters though, nor other options like "-cpu Penryn,+popcnt,+sse4_2".

With Xen I did not have any issue running Passmark 8 - it just works. See my passmark results here. In that same thread I posted a CPU performance comparison between KVM and Xen, see here.

However, I'm a total newbie to KVM and haven't found any good source of information on it, except this tutorial/ thread!!!

Can you or someone point me to some good sources for information, in particular an overview of the zillion+1 qemu-system-x86_64 command line options with some useful explanations? (See also my next post - to keep things separate.)

Offline

#1669 2014-05-07 19:17:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:
aw wrote:
powerhouse wrote:

Thanks nbhs for this excellent tutorial! I've wanted to try kvm for a long time, but only recently found time to do it. I have successfully passed through a Sapphire Radeon HD 7770 using primary passthrough to a Windows 7 Pro 64 bit guest. Below some more info:
OS: Ubuntu 14.04
Guest OS: Windows 7 Pro 64 bit
M/B: Asus Sabertooth X79
CPU: Intel i7 3930K (LGA 2011)
Host OS graphics card: Nvidia Quadro 2000 with Nvidia proprietary driver
bridged network configuration
vfio driver for network and disk
Disk is a LVM drive using raw format

Here some remarks and questions:
Passmark PerformanceTest benchmark only works in SAFEMODE, else Windows BSODs. Under Xen there is no such issue. Furthermore Passmark is unable to detect the SSE extensions of the CPU (again under Xen it works fine). CPU-Z, however, does recognize the SSE extensions.
PCmark has similar issues.
Is there anything I do wrong with the configuration?

See http://www.passmark.com/forum/showthrea … r-Number-4 comment #6 and beyond.  My suspicion is that Passmark makes some platform assumptions based on the CPU family.  It would be a great data point if you could confirm or deny that with Xen.  Thanks.

Thanks for the link. I had read that thread and tried -cpu=SandyBridge, but it didn't make a difference - the same BSOD when running Passmark. I did not try any model= parameters though, nor other options like "-cpu Penryn,+popcnt,+sse4_2".

-cpu=SandyBridge isn't going to prove anything, SandyBridge is newer than Nehalem and I'd expect the same problem.  If it's the same BSOD, -cpu=Penryn should work.

With Xen I did not have any issue running Passmark 8 - it just works. See my passmark results here.

I notice in the CPU-z screen that Xen is reporting the CPU as family 6, model D.  So that's not accurate either.  It would be really interesting if you could change that model to match your physical CPU and see if Xen gets the same BSOD.  I expect KVM would also work with -cpu SandyBridge,model=13 (aka 'D' in decimal)

In that same thread I posted a CPU performance comparison between KVM and Xen, see here.

Questionable comparison with one running in safemode.  Safemode may exclude SSE, since you note it didn't run that test.  I'm confused though that your model is also listed as D under KVM, maybe my hypothesis is completely off.  You also seem to have nested VT-x support enabled.  Some here seem to think that hurts performance (I've never tried it myself).  To disable, unload and reload the kvm-intel module with nested=0 (modprobe -r kvm-intel; modprobe kvm-intel nested=0).

However, I'm a total newbie to KVM and haven't found any good source of information on it, except this tutorial/ thread!!!

Can you or someone point me to some good sources for information, in particular an overview of the zillion+1 qemu-system-x86_64 command line options with some useful explanations? (See also my next post - to keep things separate.)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1670 2014-05-07 19:20:04

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:
aw wrote:
powerhouse wrote:

Thanks nbhs for this excellent tutorial! I've wanted to try kvm for a long time, but only recently found time to do it. I have successfully passed through a Sapphire Radeon HD 7770 using primary passthrough to a Windows 7 Pro 64 bit guest. Below some more info:
OS: Ubuntu 14.04
Guest OS: Windows 7 Pro 64 bit
M/B: Asus Sabertooth X79
CPU: Intel i7 3930K (LGA 2011)
Host OS graphics card: Nvidia Quadro 2000 with Nvidia proprietary driver
bridged network configuration
vfio driver for network and disk
Disk is a LVM drive using raw format

Here some remarks and questions:
Passmark PerformanceTest benchmark only works in SAFEMODE, else Windows BSODs. Under Xen there is no such issue. Furthermore Passmark is unable to detect the SSE extensions of the CPU (again under Xen it works fine). CPU-Z, however, does recognize the SSE extensions.
PCmark has similar issues.
Is there anything I do wrong with the configuration?

See http://www.passmark.com/forum/showthrea … r-Number-4 comment #6 and beyond.  My suspicion is that Passmark makes some platform assumptions based on the CPU family.  It would be a great data point if you could confirm or deny that with Xen.  Thanks.

Thanks for the link. I had read that thread and tried -cpu=SandyBridge, but it didn't make a difference - the same BSOD when running Passmark. I did not try any model= parameters though, nor other options like "-cpu Penryn,+popcnt,+sse4_2".

With Xen I did not have any issue running Passmark 8 - it just works. See my passmark results here. In that same thread I posted a CPU performance comparison between KVM and Xen, see here.

However, I'm a total newbie to KVM and haven't found any good source of information on it, except this tutorial/ thread!!!

Can you or someone point me to some good sources for information, in particular an overview of the zillion+1 qemu-system-x86_64 command line options with some useful explanations? (See also my next post - to keep things separate.)

I had a similar problem with passmark (or was it pc mark?) on my amd system, what solved it for me was doing this

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Offline

#1671 2014-05-07 19:43:38

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In addition to my previous observations and questions above, here some more:

1. I found it difficult to find useful information on the command line options for qemu-system-x86_64, or any other kvm related documentation. Are there any good sources of information (except this excellent thread)?

2. With KVM I managed to pass through an AMD Radeon HD 7770 using primary passthrough, under Xen I never succeeded with primary passthrough, only secondary (booting up using the Cirrus virtual adapter). As a result, with KVM the Windows guest boots faster.

3. With KVM, I only succeeded to pass through the AMD 7770 card, I had no success with my Nvidia Quadro 2000. Under Xen, it's almost the opposite - the Nvidia Quadro 2000 in secondary passthrough is a piece of cake, but the AMD 7770 can be a challenge (it works with Xen 4.1.2 and an old kernel, but not with Xen 4.4 and a 3.13 kernel, i.e. Ubuntu 14.04). With Xen, kernel changes can break VGA passthrough support (I've seen that with AMD cards like the Radeon HD 6450 and the Radeon HD 7770), though with the Nvidia Quadro 2000 I only once had a problem (error 22) after a Xen security update (Xen update). Also under Xen, often xm works whereas xl toolstack doesn't.

4. When trying to pass through the Nvidia Quadro 2000 (following nbhs tutorial/post #1), I got primary passthrough to a low res screen when installing Windows. Only after installing the Nvidia driver under Windows and rebooting would the boot process hang with a black screen. I wasn't able to boot Windows again, as somehow also the PCI passed through USB keyboard/mouse would not respond anymore when rebooting Windows into a recovery screen (normal start, etc...). I tried to load the VGA BIOS (I had prepared a .rom file running Xen and using GPU-Z to save the file), but it didn't work, nor did the rombar=0 option - like here:
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/user/GF106.rom \

Is there anything special to consider when passing through a Nvidia Quadro card? I vaguely remember having read something somewhere, but cannot find the info.

5. I also gave virt-manager a (short) try. Somehow most kvm information sources refer to it (or virt-install or libvirt). I had some slim hope that it might work, but I my trials were in vain. Besides, I still cannot figure out where virt-manager stores its guest config files which is quite unnerving.

6. With Xen I got either the xl or the xm toolstack, giving me enough control over the guest(s). I'm still trying to figure out how to do things in KVM, but haven't found much useful information (except references to virt-manager - see above). Is there any simple tool - preferably command line or web, or a GUI if it must be? I don't need a cloud orchestration suite, just a simple tool to get the job done?

7. Last not least, I haven't applied any patches or compiled anything. The only thing I did was loading the VGA BIOS for the AMD 7770 card, but it wasn't necessary.

Please forgive me when asking perhaps redundant questions, but I haven't found the time to read the entire thread, yet.

Offline

#1672 2014-05-07 20:00:32

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

-cpu=SandyBridge isn't going to prove anything, SandyBridge is newer than Nehalem and I'd expect the same problem.  If it's the same BSOD, -cpu=Penryn should work.

Thanks for that hint - I will give it a try, but it might take a few days (need to reconfigure the test drive for AMD passthrough and have little time these days). Will report back.

aw wrote:
powerhouse wrote:

With Xen I did not have any issue running Passmark 8 - it just works. See my passmark results here.

I notice in the CPU-z screen that Xen is reporting the CPU as family 6, model D.  So that's not accurate either.  It would be really interesting if you could change that model to match your physical CPU and see if Xen gets the same BSOD.  I expect KVM would also work with -cpu SandyBridge,model=13 (aka 'D' in decimal)

I will try. Where can I find an overview of these command line options?

aw wrote:
powerhouse wrote:

In that same thread I posted a CPU performance comparison between KVM and Xen, see here.

Questionable comparison with one running in safemode.  Safemode may exclude SSE, since you note it didn't run that test.  I'm confused though that your model is also listed as D under KVM, maybe my hypothesis is completely off.  You also seem to have nested VT-x support enabled.  Some here seem to think that hurts performance (I've never tried it myself).  To disable, unload and reload the kvm-intel module with nested=0 (modprobe -r kvm-intel; modprobe kvm-intel nested=0).

SSE was definitely excluded, it doesn't show up in the graph. For that reason there is no CPU performance summary under the KVM column. By the way, this Passmark CPU test (in SAFEMODE) was run using the -cpu=host option. I was under the impression that that option would pass through the host CPU capabilities? As I said I have absolutely no clew as to which options there are and what they do (and --help doesn't seem to list all of them).
I'll give your suggestion a try - thanks!

Offline

#1673 2014-05-07 20:07:25

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I had a similar problem with passmark (or was it pc mark?) on my amd system, what solved it for me was doing this

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Thanks for the reply, and for your great tutorial! I wouldn't have made it without.

I remember that I've seen this option but never tried it since you said you are using an AMD machine. What does this option do?

By the way, what other configuration options are there for the kvm.conf file - any link to further documentation?

Offline

#1674 2014-05-08 07:52:25

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
anickname wrote:

@aw

Regarding "hv-time" problem do you know if will it be fixed in 3.15 ?

https://bugzilla.kernel.org/show_bug.cgi?id=73721

I don't know, but I just poked one of the KVM maintainers about it and he's going to investigate.

Apparently it's already fixed in current 3.15-rc and Paolo plans send it for stable as well.

I applied the patch to 3.14.1 and now everything works good. The cpu load dropped from ~60-65% to 40-45% for an
idle windows 8.1 vm (with and without hv-time). Also the vm (when using hv-time) seems to have a more "native" feeling.

--- a/arch/x86/kvm/x86.c 2014-05-07 21:15:22.099260195 +0300 +++ b/arch/x86/kvm/x86.c 2014-05-07 21:19:56.712586783 +0300 @@ -1111,7 +1111,6 @@ { struct timespec ts; - WARN_ON(preemptible()); ktime_get_ts(&ts); monotonic_to_bootbased(&ts); return timespec_to_ns(&ts);

Thanks.

Offline

#1675 2014-05-08 22:08:20

th3flyboy
Member
Registered: 2014-03-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:

I have managed to pass through my GeForce GTX 770M into KVM.
The GPU showed Error 10 for the very beginning and after I have installed latest ForceWare drivers, it showed error 43.
However if i disable and re-enable the card in Device Manager it showed that the device is working correctly.
Unfortunately as a second device and I was unable to boot Windows with nvidia card set as primary due to error 43, which i come across every time i reboot the VM.
All the time i need to disable and re-enable the dGPU in order to make Windows marking it as working.


I am not sure if this is connected with nVidia Optimus, or just GTX 770M. I have found that some of you managed to get GF 7xx working, while others not.
Also there is no such problem with ATI cards. Due to that i think that more people having notebooks should give it a try, especially with older GeForce cards and with mobile ATI cards.

As the prove I want to show you a screenshot:

http://wstaw.org/m/2014/05/04/snapshot2_1_png_750x750_q85.jpg


Could you please put up some detailed instructions on how you did that? I tried a while back with my 780M and wasn't even able to boot the VM when the device was attached to the VM.

Offline

#1676 2014-05-10 03:52:51

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
powerhouse wrote:

With Xen I did not have any issue running Passmark 8 - it just works. See my passmark results here.

I notice in the CPU-z screen that Xen is reporting the CPU as family 6, model D.  So that's not accurate either.  It would be really interesting if you could change that model to match your physical CPU and see if Xen gets the same BSOD.  I expect KVM would also work with -cpu SandyBridge,model=13 (aka 'D' in decimal)

powerhouse wrote:

In that same thread I posted a CPU performance comparison between KVM and Xen, see here.

Questionable comparison with one running in safemode.  Safemode may exclude SSE, since you note it didn't run that test.  I'm confused though that your model is also listed as D under KVM, maybe my hypothesis is completely off.  You also seem to have nested VT-x support enabled.  Some here seem to think that hurts performance (I've never tried it myself).  To disable, unload and reload the kvm-intel module with nested=0 (modprobe -r kvm-intel; modprobe kvm-intel nested=0).

I checked my CPU model and CPU-Z - whether running on KVM or Xen - does report it correctly. See cpu-world. My CPU is a SandyBridge-E i7 3930K, C2 stepping.

From that I take that I should try -cpu SandyBridge,model=45

Unfortunately it might take me a couple of days until I can test again.

Offline

#1677 2014-05-11 02:18:25

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again. Everything working fine for me, but I met weird problem with my SATA controller passthrough:

05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

Everything working fine on Windows 8.1 guest, but with Ubuntu 12.04 or Windows 7 guest there is problem appear. First time VM booting just fine, but next time I VM stuck on boot for long time and then report "Boot failed: could not read the boot disk". It's looks like both Ubuntu and Win7 don't reset controller properly on shutdown.

I find out that if I attach controller to ahci driver and back to vfio it's working fine again:

echo "1b21 0612" > /sys/bus/pci/drivers/ahci/new_id echo "0000:05:00.0" > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind echo "0000:05:00.0" > /sys/bus/pci/drivers/ahci/bind echo "1b21 0612" > /sys/bus/pci/drivers/ahci/remove_id echo "1b21 0612" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:05:00.0" > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind echo "1b21 0612" > /sys/bus/pci/drivers/vfio-pci/remove_id

Is anyone else have same problem? Any solution for this?

PS: Full configuration of qemu VM:

qemu-system-x86_64 \ -name "vm_vfio_pci" \ \ -bios /usr/share/qemu/bios.bin \ -enable-kvm \ \ -M q35 \ \ -vga none \ -device qxl \ \ -m 5G \ \ -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ \ -net none \ \ -soundhw hda \ \ -usb \ -device usb-kbd \ -device usb-mouse \ \ \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.0,bus=pcie.0 \ -device vfio-pci,host=05:00.0,bus=pcie.0 \ \ -device virtio-scsi-pci,id=scsi \ -drive file=/opt/vm/win.iso,id=disk,format=qcow2 \ -device scsi-hd,drive=disk

Last edited by SXX (2014-05-11 02:19:49)

Online

#1678 2014-05-11 10:08:41

maduraa
Member
Registered: 2014-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
maduraa wrote:

I saw on an online forum that some motherboards have the PCI slots behind a switch and they do not support passthrough, is this true?

-[0000:00]-+-00.0 +-01.0-[01]--+-00.0 | \-00.1 +-02.0 +-03.0 +-14.0 +-16.0 +-1a.0 +-1b.0 +-1c.0-[02]-- +-1c.3-[03]----00.0 +-1c.7-[04]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3

Is there any thing wrong in the about lspci -t output that'd disqualify my board? 01:00.0, 01:00.1 are for VGA and VGA audio

a) PCIe switches don't prevent device assigned, b) you don't have one.  PCIe-to-PCI bridges mean that all the devices behind the bridge may need to be assigned together, which may include multiple slots... you don't have one of those either.

Thanks aw, I tried the VGA arbiter patches but didn't see any output from the card, should I try 3.15?

Offline

#1679 2014-05-11 17:10:21

manjack
Member
Registered: 2014-05-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, thanks for the great guide. I'm kinda new here and arch linux.
I would like to ask if anything work fine, do i have to type those command every time i wanna start the VM?

Offline

#1680 2014-05-11 21:18:30

jb660
Member
Registered: 2014-05-11
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks nbhs for this excellent tutorial.


I previously managed to successfully pass-through my video card but after formatting my desktop I can't make it work again.
When I try to launch the vm qemu starts without errors but nothing appears on the output of my graphics card .

I use all the packages given on the first post.
I have a GTX780 - i7 4770 (host GPU) - Asrock Z87 pro3

output of dmesg

[ 1873.347383] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x900

output of lspci -v

04:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 104b Flags: fast devsel, IRQ 16 Memory at ea000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at eb000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau 04:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1) Subsystem: NVIDIA Corporation Device 104b Flags: fast devsel, IRQ 17 Memory at eb080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

Offline

#1681 2014-05-12 03:29:31

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@jb660: The nouveau driver is taking control of your video card. You should blacklist it. Also, I'm not sure which method nbhs described in his tutorial, but I have seen good results with binding the graphics card to pci_stub using the initramfs method.

You could check that your pci_stub binding method is working by checking for USB hosts in lspci and binding one of it to pci_stub (preferable not the one with keyboard/mouse connected to it). Then check with lspci -v.

Offline

#1682 2014-05-12 03:35:17

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

manjack wrote:

Hi, thanks for the great guide. I'm kinda new here and arch linux.
I would like to ask if anything work fine, do i have to type those command every time i wanna start the VM?

You can copy/paste the command into a file, put

#!/bin/sh

in the first line (or .../bash instead of .../sh), and make it executable. Voila, you got a shell script.

Offline

#1683 2014-05-12 06:07:05

jb660
Member
Registered: 2014-05-11
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

@jb660: The nouveau driver is taking control of your video card. You should blacklist it. Also, I'm not sure which method nbhs described in his tutorial, but I have seen good results with binding the graphics card to pci_stub using the initramfs method.

You could check that your pci_stub binding method is working by checking for USB hosts in lspci and binding one of it to pci_stub (preferable not the one with keyboard/mouse connected to it). Then check with lspci -v.

The nouveau driver is already blacklisted
cat /etc/modprobe.d/blacklist.conf

blacklist nouveau

I use the pci-stub method and it seems to work as I have this output.

dmesg | grep stub

[ 0.419737] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.419744] pci-stub 0000:04:00.0: claimed by stub [ 0.419747] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.419751] pci-stub 0000:04:00.1: claimed by stub

and vfio-bind works without problems afterwards

Offline

#1684 2014-05-12 08:21:30

manjack
Member
Registered: 2014-05-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:
manjack wrote:

Hi, thanks for the great guide. I'm kinda new here and arch linux.
I would like to ask if anything work fine, do i have to type those command every time i wanna start the VM?

You can copy/paste the command into a file, put

#!/bin/sh

in the first line (or .../bash instead of .../sh), and make it executable. Voila, you got a shell script.


Many Thanks!

Offline

#1685 2014-05-12 18:29:37

thehighhat
Member
Registered: 2014-05-12
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How do passthrough'd devices get returned to the host?


-----

Here is what I have:
  mobo:  Tyan S7025
  kernel:  3.13
  guest:  windows 7 64bit 
  host vga:  aspeed onboard graphics
  pcie gpu:  gtx470

Here is what I did:
  set bios to enable vt-d
  set bios to use onboard aspeed for host
  no gui installed on linux host
  ssh into linux host from another machine
  blacklist nouveau
  vfio_iommu_type1 allow_unsafe_interrupts=1
  using vfio-bind script with the gtx470 lscpi values
  started vm with this command

qemu-system-x86_64 \ -enable-kvm \ -machine type=q35,accel=kvm \ -cpu host \ -smp cpus=4 \ -m 4G \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:b1.0 \ -drive file=/path/to/win7.raw,format=raw,cache=none,id=disk \ -device ide-hd,bus=ide.0,drive=disk \ -usb \ -usbdevice host:064d:c25b \ -bios /path/to/qemu/bios.bin \ -vga none \ -nographic \ -name "vm-kvm" 

  attach monitor to gtx470

  the usb keyboard was originally attached to host
    this command assigns it to the guest
  the onboard sound device is attached


the vm runs perfectly. 

except, after it shutdowns, two issues:

  1.  how do reassign the usb keyboard and audio chipset back to the host
  2.  how can reuse the graphics card. 
       2a.  whenever I try to start the vm again i get error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

 

thoughts?


GPU passthrough of GTX470 to a Windows 7 guest

Last edited by thehighhat (2014-05-14 20:14:20)

Offline

#1686 2014-05-12 18:53:05

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jb660 wrote:
powerhouse wrote:

@jb660: The nouveau driver is taking control of your video card. You should blacklist it. Also, I'm not sure which method nbhs described in his tutorial, but I have seen good results with binding the graphics card to pci_stub using the initramfs method.

You could check that your pci_stub binding method is working by checking for USB hosts in lspci and binding one of it to pci_stub (preferable not the one with keyboard/mouse connected to it). Then check with lspci -v.

The nouveau driver is already blacklisted
cat /etc/modprobe.d/blacklist.conf

blacklist nouveau

I use the pci-stub method and it seems to work as I have this output.

dmesg | grep stub

[ 0.419737] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.419744] pci-stub 0000:04:00.0: claimed by stub [ 0.419747] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.419751] pci-stub 0000:04:00.1: claimed by stub

and vfio-bind works without problems afterwards

Strange, so how come your lspci -v lists nouveau?

Kernel modules: nouveau

Last edited by powerhouse (2014-05-12 18:54:18)

Offline

#1687 2014-05-12 23:37:25

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This begins to take form but...
I could install windows 7 but at the last part when is detecting and configuring the hardware I got a BSOD error 0x00000116 with the file nvlddmkm.sys (nvidia driver).

As I say in my post 1655, my 1st  video card is a buildin intel and the 2nd is a Nvidia 7600GT.

I was able to start windows removing the parameter "-vga none" and "x-vga=on".it said In the proprieties of the Nvidia card  that It does not have enough resources. I thought that installing the latest drivers for my video card it will work but still crash. sad

The next thing I did was start windows in SAFE MODE. Windows finally start this are the resources that use:
nvidia 7600 gt
windows 7
interval
memory     FB000000-FBFFFFFF
  E0000000-EFFFFFFF
  FC000000-FCFFFFFF
I/O    C000-C07F
IRQ    10

My command line is

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/media/800/destroy_varios/donwload2/virtualizacion/MSI.7600GT.256.060310.rom \ -bios /var/lib/libvirt/bios/seabios-hp2.bin \ -boot menu=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/media/Utils/Sistemas\ Operativos/windows/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \ -drive file=/media/800/Virtual\ Machines/disks-virt/win7-A.qcow2,id=disk,format=qcow2 -device ide-hd,bus=piix4-ide.0,drive=disk \ -usb -usbdevice host:040b:2013 

any idea? Please help

Offline

#1688 2014-05-13 05:07:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Destroy wrote:

This begins to take form but...
I could install windows 7 but at the last part when is detecting and configuring the hardware I got a BSOD error 0x00000116 with the file nvlddmkm.sys (nvidia driver).

As I say in my post 1655, my 1st  video card is a buildin intel and the 2nd is a Nvidia 7600GT.

I was able to start windows removing the parameter "-vga none" and "x-vga=on".it said In the proprieties of the Nvidia card  that It does not have enough resources. I thought that installing the latest drivers for my video card it will work but still crash. sad

The next thing I did was start windows in SAFE MODE. Windows finally start this are the resources that use:
nvidia 7600 gt
windows 7
interval
memory     FB000000-FBFFFFFF
  E0000000-EFFFFFFF
  FC000000-FCFFFFFF
I/O    C000-C07F
IRQ    10

My command line is

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/media/800/destroy_varios/donwload2/virtualizacion/MSI.7600GT.256.060310.rom \ -bios /var/lib/libvirt/bios/seabios-hp2.bin \ -boot menu=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/media/Utils/Sistemas\ Operativos/windows/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \ -drive file=/media/800/Virtual\ Machines/disks-virt/win7-A.qcow2,id=disk,format=qcow2 -device ide-hd,bus=piix4-ide.0,drive=disk \ -usb -usbdevice host:040b:2013 

any idea? Please help

I was about to write how I installed a 7300LE in my system and it seems to be running just fine... but then the VM hung refreshing the Windows experience index.  Repeatedly.  It does seem to run the Heaven benchmark though, very, very slowly, and some parts it just doesn't render correctly.  Nvidia doesn't even support 7-series cards anymore and cards with better performance can be bought for $10US.  Anyway,  my experience seems better than yours.  I'm running an upstream 3.15-rc4 kernel with the i915 patches plus stock Fedora qemu-kvm-2.0.0-4.fc20.x86_64.  I've been having quite a bit of success using the default 440FX QEMU machine:

#!/bin/sh -x virsh nodedev-detach pci_0000_01_00_0 QEMU=/bin/qemu-kvm QEMU_AUDIO_DRV=pa \ $QEMU \ -enable-kvm -rtc base=localtime \ -m 4g -smp sockets=1,cores=2 -cpu host,hv-time \ -vga none -nographic \ -monitor stdio \ -nodefconfig \ -device intel-hda -device hda-output \ -device vfio-pci,host=1:00.0,x-vga=on \ -netdev tap,id=br0,vhost=on \ -device virtio-net-pci,mac=02:12:34:56:78:91,netdev=br0 \ -drive file=/dev/vg_vm/lv_win7_nvidia,cache=none,if=none,id=win7,aio=native \ -device virtio-blk-pci,drive=win7,bootindex=0 virsh nodedev-reattach pci_0000_01_00_0

Also, if you need Marvell SATA patches, try my dma-alias-v3 branch at git://github.com/awilliam/linux-vfio.git


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1689 2014-05-13 18:41:32

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi, not really pci passthrough, but related. Bought this DisplayLink AOC monitor, and unfortunately I am victim of superficial information about availability of linux drivers. That would be still ok, but the real disaster is that I cannot use the Displaylink in a virtual machine. Funny enough under Virtualbox it complains that it cannot work with the primary card, but under kvm/qemu it tries to turn on the monitor, it seems like the backlight is on but that is all. Anybody any experience with this DisplayLink in virtual machine? I know that AOC stated that it would not work... nevertheless...

Offline

#1690 2014-05-13 18:49:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pereczes wrote:

hi, not really pci passthrough, but related. Bought this DisplayLink AOC monitor, and unfortunately I am victim of superficial information about availability of linux drivers. That would be still ok, but the real disaster is that I cannot use the Displaylink in a virtual machine. Funny enough under Virtualbox it complains that it cannot work with the primary card, but under kvm/qemu it tries to turn on the monitor, it seems like the backlight is on but that is all. Anybody any experience with this DisplayLink in virtual machine? I know that AOC stated that it would not work... nevertheless...

It's just a USB device, right?  You might need to expose a USB 2 or 3 controller to the guest for it to work with USB passthrough.  Otherwise, I don't see why it wouldn't work with an assigned PCI USB2+ controller.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1691 2014-05-13 18:56:14

pereczes
Member
Registered: 2014-03-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
pereczes wrote:

hi, not really pci passthrough, but related. Bought this DisplayLink AOC monitor, and unfortunately I am victim of superficial information about availability of linux drivers. That would be still ok, but the real disaster is that I cannot use the Displaylink in a virtual machine. Funny enough under Virtualbox it complains that it cannot work with the primary card, but under kvm/qemu it tries to turn on the monitor, it seems like the backlight is on but that is all. Anybody any experience with this DisplayLink in virtual machine? I know that AOC stated that it would not work... nevertheless...

It's just a USB device, right?  You might need to expose a USB 2 or 3 controller to the guest for it to work with USB pass-through.  Otherwise, I don't see why it wouldn't work with an assigned PCI USB2+ controller.

You see... that was my reasoning as well, easy to go in wrong direction. The problem that these b*tards look at the graphic card. They seem to attach to the graphic card like a mushroom or other parasite, extract the liquid and send it USB smile .

Offline

#1692 2014-05-13 18:57:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pereczes wrote:
aw wrote:
pereczes wrote:

hi, not really pci passthrough, but related. Bought this DisplayLink AOC monitor, and unfortunately I am victim of superficial information about availability of linux drivers. That would be still ok, but the real disaster is that I cannot use the Displaylink in a virtual machine. Funny enough under Virtualbox it complains that it cannot work with the primary card, but under kvm/qemu it tries to turn on the monitor, it seems like the backlight is on but that is all. Anybody any experience with this DisplayLink in virtual machine? I know that AOC stated that it would not work... nevertheless...

It's just a USB device, right?  You might need to expose a USB 2 or 3 controller to the guest for it to work with USB pass-through.  Otherwise, I don't see why it wouldn't work with an assigned PCI USB2+ controller.

You see... that was my reasoning as well, easy to go in wrong direction. The problem that these b*tards look at the graphic card. They seem to attach to the graphic card like a mushroom or other parasite, extract the liquid and send it USB smile .

Ugh.  Send it back


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1693 2014-05-13 19:52:36

jb660
Member
Registered: 2014-05-11
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

...

Strange, so how come your lspci -v lists nouveau?

Kernel modules: nouveau

I think that is the standard behaviour ( https://bbs.archlinux.org/viewtopic.php … 9#p1373419 )

moreover

lsmod | grep nouveau

does not return anything

Offline

#1694 2014-05-14 01:47:52

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jb660 wrote:

I previously managed to successfully pass-through my video card but after formatting my desktop I can't make it work again.
When I try to launch the vm qemu starts without errors but nothing appears on the output of my graphics card .

This happened to my too. Only the first time the VM seems fine, the next call nothing appear until I added the rom of the video card

....,x-vga=on,romfile=/media/800/destroy_varios/donwload2/virtualizacion/MSI.7600GT.256.060310.rom
jb660 wrote:

output of lspci -v

04:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 104b Flags: fast devsel, IRQ 16 Memory at ea000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at eb000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau

powerhouse is incorrect, here you have 2 lines:
Kernel driver in use: vfio-pci - is saying you current driver
Kernel modules: nouveau - is saying that this driver could use your hardware

but this is weird: Capabilities: <access denied>

Last edited by Destroy (2014-05-14 02:13:47)

Offline

#1695 2014-05-14 02:11:41

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I was about to write how I installed a 7300LE in my system and it seems to be running just fine... but then the VM hung refreshing the Windows experience index.  Repeatedly.  It does seem to run the Heaven benchmark though, very, very slowly, and some parts it just doesn't render correctly.  Nvidia doesn't even support 7-series cards anymore and cards with better performance can be bought for $10US.  Anyway,  my experience seems better than yours.  I'm running an upstream 3.15-rc4 kernel with the i915 patches plus stock Fedora qemu-kvm-2.0.0-4.fc20.x86_64.  I've been having quite a bit of success using the default 440FX QEMU machine:

I have upgraded to qemu-2 and now the VM start fine big_smile

aw wrote:
#!/bin/sh -x virsh nodedev-detach pci_0000_01_00_0 QEMU=/bin/qemu-kvm QEMU_AUDIO_DRV=pa \ $QEMU \ -enable-kvm -rtc base=localtime \ -m 4g -smp sockets=1,cores=2 -cpu host,hv-time \ -vga none -nographic \ -monitor stdio \ -nodefconfig \ -device intel-hda -device hda-output \ -device vfio-pci,host=1:00.0,x-vga=on \ -netdev tap,id=br0,vhost=on \ -device virtio-net-pci,mac=02:12:34:56:78:91,netdev=br0 \ -drive file=/dev/vg_vm/lv_win7_nvidia,cache=none,if=none,id=win7,aio=native \ -device virtio-blk-pci,drive=win7,bootindex=0 virsh nodedev-reattach pci_0000_01_00_0

What are the advantages of "rtc base=localtime", "-monitor stdio" "-nodefconfig" and "hv-time"?

aw wrote:

Also, if you need Marvell SATA patches, try my dma-alias-v3 branch at git://github.com/awilliam/linux-vfio.git

do you have a patch file? I don't know how to get it for linux-3.14.2. I have never used git before.

Nevertheless I am able to boot windows correctly without BSOD but after I tried to run the test using Passmark PerformanceTest benchmark I got another BSOD now with DirectIO64 file.
I think this is the same problem that have powerhouse.

I'll try this later

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

More news tomorrow

Offline

#1696 2014-05-14 06:22:26

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's working!!!

thanks for this

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

The Windows 7 experience runs fully and give me a 5.1 because my video card
Also running  Passmark PerformanceTest does not crash the system,
meanwhile in the host dmesg shows

[ 8805.345943] kvm [2308]: vcpu0 ignored rdmsr: 0xce [10525.815384] kvm [2308]: vcpu0 ignored rdmsr: 0xce [10525.819608] kvm [2308]: vcpu0 ignored rdmsr: 0xe8 [10525.823808] kvm [2308]: vcpu0 ignored rdmsr: 0xe8 [10525.823819] kvm [2308]: vcpu0 ignored rdmsr: 0xe7 [10525.828023] kvm [2308]: vcpu0 ignored rdmsr: 0xe7 [10525.828045] kvm [2308]: vcpu0 ignored rdmsr: 0x31 [10525.828049] kvm [2308]: vcpu0 ignored rdmsr: 0x35 [10525.828052] kvm [2308]: vcpu0 ignored rdmsr: 0x39 [10525.828068] kvm [2308]: vcpu0 ignored rdmsr: 0xce [10525.828071] kvm [2308]: vcpu0 ignored rdmsr: 0x1ad [10531.211800] kvm_get_msr_common: 56 callbacks suppressed [10531.211805] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10531.211811] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10531.211897] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10531.211972] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10531.726573] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10531.726581] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10531.726664] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10531.726739] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10532.241382] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10532.241391] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10536.359825] kvm_get_msr_common: 30 callbacks suppressed [10536.359830] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10536.359836] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10536.359921] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10536.359995] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10536.874625] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10536.874633] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10536.874718] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10536.874793] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10537.389450] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10537.389459] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10541.507914] kvm_get_msr_common: 30 callbacks suppressed [10541.507919] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10541.507926] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10541.508012] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10541.508087] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10542.032144] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10542.032152] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10542.032232] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10542.032302] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10542.553052] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10542.553060] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10546.671482] kvm_get_msr_common: 30 callbacks suppressed [10546.671487] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10546.671494] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10546.671578] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10546.671653] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10547.186743] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10547.186749] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10547.186800] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10547.186845] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10547.701166] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10547.701176] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10551.819542] kvm_get_msr_common: 30 callbacks suppressed [10551.819547] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10551.819554] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10551.819685] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10551.820311] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10552.334294] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10552.334303] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10552.334386] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10552.334472] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10552.849163] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10552.849172] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10556.967604] kvm_get_msr_common: 30 callbacks suppressed [10556.967609] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10556.967616] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10556.967701] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10556.967775] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10557.482480] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10557.482489] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10557.482588] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10557.482713] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10557.997470] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10557.997479] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10562.115632] kvm_get_msr_common: 30 callbacks suppressed [10562.115637] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10562.115643] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10562.115728] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10562.115801] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10562.630407] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10562.630416] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10562.630499] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10562.630588] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10563.145239] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10563.145247] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10567.263657] kvm_get_msr_common: 30 callbacks suppressed [10567.263661] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10567.263667] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10567.263754] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10567.263828] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10567.778420] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10567.778430] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10567.778517] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10567.778607] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10568.293303] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10568.293312] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10572.411700] kvm_get_msr_common: 30 callbacks suppressed [10572.411704] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10572.411711] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10572.411797] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10572.411871] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10572.926492] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10572.926502] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10572.926587] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10572.926783] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10573.446144] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10573.446153] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10577.575379] kvm_get_msr_common: 30 callbacks suppressed [10577.575384] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10577.575391] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10577.575475] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10577.575549] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10578.090146] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10578.090155] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10578.090241] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10578.090315] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10578.604912] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10578.604921] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10582.723344] kvm_get_msr_common: 30 callbacks suppressed [10582.723348] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10582.723355] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10582.723439] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10582.723514] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10583.238144] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10583.238153] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10583.238743] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10583.238969] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10583.757245] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10583.757253] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10587.871387] kvm_get_msr_common: 30 callbacks suppressed [10587.871391] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10587.871398] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10587.871482] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10587.871556] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10588.386205] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10588.386214] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10588.386298] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10588.386371] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10588.901250] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10588.901259] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10593.019411] kvm_get_msr_common: 30 callbacks suppressed [10593.019415] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10593.019422] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10593.019506] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10593.019630] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10593.534219] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10593.534229] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10593.534358] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10593.534432] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10594.049187] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10594.049196] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10598.183053] kvm_get_msr_common: 30 callbacks suppressed [10598.183058] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10598.183064] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10598.183149] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10598.183223] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10598.697835] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10598.697843] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10598.697928] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10598.698002] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10599.212738] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10599.212747] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10603.331055] kvm_get_msr_common: 30 callbacks suppressed [10603.331059] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10603.331066] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10603.331151] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10603.331224] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10603.845911] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10603.845920] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10603.846006] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10603.846137] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10604.360691] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10604.360701] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10608.494737] kvm_get_msr_common: 30 callbacks suppressed [10608.494742] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10608.494749] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10608.494833] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10608.494922] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10609.009558] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10609.009567] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10609.009649] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10609.009722] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10609.524341] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10609.524350] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10613.649505] kvm_get_msr_common: 30 callbacks suppressed [10613.649510] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10613.649516] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10613.649637] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10613.649707] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10614.177452] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10614.177460] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10614.177539] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10614.177609] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10614.687996] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10614.688004] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10618.806408] kvm_get_msr_common: 30 callbacks suppressed [10618.806413] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10618.806419] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10618.806503] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10618.806769] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10619.321199] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10619.321208] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10619.321293] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10619.321367] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10619.835983] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10619.835992] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10623.954401] kvm_get_msr_common: 30 callbacks suppressed [10623.954406] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10623.954412] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10623.954496] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10623.954570] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10624.469241] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10624.469250] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10624.469334] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10624.469408] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10624.984063] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10624.984072] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10629.102434] kvm_get_msr_common: 30 callbacks suppressed [10629.102438] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10629.102445] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10629.102528] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10629.102602] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10629.617282] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10629.617291] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10629.617376] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10629.617449] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10630.132252] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10630.132260] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10634.266098] kvm_get_msr_common: 30 callbacks suppressed [10634.266103] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10634.266110] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10634.266228] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10634.266862] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10634.780919] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10634.780928] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10634.781013] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10634.781103] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10635.300753] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10635.300761] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10639.429779] kvm_get_msr_common: 30 callbacks suppressed [10639.429783] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10639.429790] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10639.429875] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10639.429950] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10639.944542] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10639.944551] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10639.944636] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10639.944709] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10640.459346] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10640.459355] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10644.577795] kvm_get_msr_common: 30 callbacks suppressed [10644.577799] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10644.577806] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10644.577888] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10644.577962] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10645.092611] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10645.092620] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10645.092774] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10645.092849] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10645.611679] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10645.611687] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10649.725820] kvm_get_msr_common: 30 callbacks suppressed [10649.725824] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10649.725831] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10649.725966] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10649.726589] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10650.240613] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10650.240621] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10650.240748] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10650.240820] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10650.755402] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10650.755411] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10654.874206] kvm_get_msr_common: 30 callbacks suppressed [10654.874210] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10654.874217] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10654.874303] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10654.874376] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10655.388652] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10655.388661] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10655.388747] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10655.388821] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10655.903505] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10655.903515] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10660.021885] kvm_get_msr_common: 30 callbacks suppressed [10660.021889] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10660.021896] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10660.021981] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10660.022055] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10660.536674] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10660.536682] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10660.536767] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10660.536841] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10661.051499] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10661.051508] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10665.169926] kvm_get_msr_common: 30 callbacks suppressed [10665.169930] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10665.169937] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10665.170021] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10665.170096] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10665.684738] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10665.684748] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10665.685339] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10665.685436] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10666.199504] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10666.199513] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10670.333574] kvm_get_msr_common: 30 callbacks suppressed [10670.333579] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10670.333585] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10670.333669] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10670.333798] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10670.848386] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10670.848394] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10670.848528] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10670.848602] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10671.363159] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10671.363169] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10675.481562] kvm_get_msr_common: 30 callbacks suppressed [10675.481567] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10675.481573] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10675.481669] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10675.481743] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10675.996663] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10675.996673] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10675.996755] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10675.996829] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10676.511176] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10676.511184] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10680.629924] kvm_get_msr_common: 30 callbacks suppressed [10680.629929] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10680.629936] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10680.630020] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10680.630094] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10681.144424] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10681.144434] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10681.145031] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10681.145188] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10681.659228] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10681.659237] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10685.777657] kvm_get_msr_common: 30 callbacks suppressed [10685.777662] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10685.777668] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10685.777801] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10685.777875] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10686.292426] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10686.292435] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10686.292519] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10686.292594] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10686.807295] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10686.807305] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10690.925679] kvm_get_msr_common: 30 callbacks suppressed [10690.925683] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10690.925690] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10690.925775] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10690.925850] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10691.440495] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10691.440504] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10691.440629] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10691.440704] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10691.955308] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10691.955317] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10696.073708] kvm_get_msr_common: 30 callbacks suppressed [10696.073712] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10696.073719] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10696.073806] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10696.073880] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10696.588536] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10696.588545] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10696.588630] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10696.588754] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10697.103308] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10697.103317] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10701.246230] kvm_get_msr_common: 30 callbacks suppressed [10701.246234] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10701.246241] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10701.246357] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10701.246431] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10701.767725] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10701.767733] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10701.767812] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10701.767881] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10702.282606] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10702.282616] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10706.401172] kvm_get_msr_common: 30 callbacks suppressed [10706.401177] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10706.401183] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10706.401268] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10706.401341] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10706.915828] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10706.915837] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10706.915925] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10706.915999] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10707.430601] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10707.430610] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10711.549035] kvm_get_msr_common: 30 callbacks suppressed [10711.549040] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10711.549046] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10711.549131] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10711.549205] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10712.064015] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10712.064023] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10712.064107] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10712.064181] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10712.578612] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10712.578622] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10716.697039] kvm_get_msr_common: 30 callbacks suppressed [10716.697044] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10716.697051] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10716.697135] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10716.697208] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10717.211880] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10717.211890] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10717.212004] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10717.212078] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10717.726689] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10717.726698] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10721.845098] kvm_get_msr_common: 30 callbacks suppressed [10721.845103] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10721.845110] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10721.845193] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10721.845266] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10722.360083] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10722.360091] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10722.360207] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10722.360280] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10722.874715] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10722.874724] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10726.993455] kvm_get_msr_common: 30 callbacks suppressed [10726.993459] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10726.993466] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10726.993551] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10726.993626] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10727.507937] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10727.507945] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10727.508030] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10727.508104] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10728.022716] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10728.022724] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10732.156819] kvm_get_msr_common: 30 callbacks suppressed [10732.156824] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10732.156831] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10732.156917] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10732.157626] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10732.671554] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10732.671563] kvm [2308]: vcpu0 ignored rdmsr: 0x611 [10732.671649] kvm [2308]: vcpu0 ignored rdmsr: 0x639 [10732.671800] kvm [2308]: vcpu0 ignored rdmsr: 0x641 [10733.186383] kvm [2308]: vcpu0 ignored rdmsr: 0x1a2 [10733.186393] kvm [2308]: vcpu0 ignored rdmsr: 0x611

The bad things are the result in the Passmark PerformanceTest for my Intel i5 3330 (using 1 core and 1 thread), the video card Nvidia 7600GT seems in the range. I don't know what thing about the memory test.
CPU Mark: 1633
2D Graphics Mark: 571
3D Graphics Mark: 185.5
Memory Mark: 1693
Disk Mark: 1151

Using

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/media/800/destroy_varios/donwload2/virtualizacion/MSI.7600GT.256.060310.rom \ -bios /var/lib/libvirt/bios/seabios-hp2.bin \ -vnc 127.0.0.1:0 -vga none \ -boot menu=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/media/800/Virtual\ Machines/disks-virt/win7-A.qcow2,id=disk,format=qcow2 -device ide-hd,bus=piix4-ide.0,drive=disk \ -usb -usbdevice host:040b:2013 

Last edited by Destroy (2014-05-14 06:26:37)

Offline

#1697 2014-05-14 13:42:36

manjack
Member
Registered: 2014-05-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi guys, i got the following error when compiling qemu

/home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:315:31: note: in expansion of macro ‘__put_user_e’ # define __put_user(x, hptr) __put_user_e(x, hptr, le) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/syscall.c:6958:17: note: in expansion of macro ‘__put_user’ __put_user(value.freeswap, &target_value->freeswap); ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:301:19: error: right-hand operand of comma expression has no effect [-Werror=unused-value] ((hptr), (x)), 0) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:315:31: note: in expansion of macro ‘__put_user_e’ # define __put_user(x, hptr) __put_user_e(x, hptr, le) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/syscall.c:6959:17: note: in expansion of macro ‘__put_user’ __put_user(value.procs, &target_value->procs); ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:301:19: error: right-hand operand of comma expression has no effect [-Werror=unused-value] ((hptr), (x)), 0) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:315:31: note: in expansion of macro ‘__put_user_e’ # define __put_user(x, hptr) __put_user_e(x, hptr, le) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/syscall.c:6960:17: note: in expansion of macro ‘__put_user’ __put_user(value.totalhigh, &target_value->totalhigh); ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:301:19: error: right-hand operand of comma expression has no effect [-Werror=unused-value] ((hptr), (x)), 0) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:315:31: note: in expansion of macro ‘__put_user_e’ # define __put_user(x, hptr) __put_user_e(x, hptr, le) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/syscall.c:6961:17: note: in expansion of macro ‘__put_user’ __put_user(value.freehigh, &target_value->freehigh); ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:301:19: error: right-hand operand of comma expression has no effect [-Werror=unused-value] ((hptr), (x)), 0) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/qemu.h:315:31: note: in expansion of macro ‘__put_user_e’ # define __put_user(x, hptr) __put_user_e(x, hptr, le) ^ /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/linux-user/syscall.c:6962:17: note: in expansion of macro ‘__put_user’ __put_user(value.mem_unit, &target_value->mem_unit); ^ cc1: all warnings being treated as errors /home/manjack/Downloads/qemu-git/src/qemu-git-2.0.r32705.gf30d56e/rules.mak:33: recipe for target 'linux-user/syscall.o' failed make[1]: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_6.html topic_7.html topic_8.html topic_9.html [linux-user/syscall.o] Error 1 Makefile:163: recipe for target 'subdir-aarch64-linux-user' failed make: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_6.html topic_7.html topic_8.html topic_9.html [subdir-aarch64-linux-user] Error 2 ==> ERROR: A failure occurred in build(). Aborting...

can anyone help please?

Last edited by manjack (2014-05-15 11:47:48)

Offline

#1698 2014-05-14 15:18:30

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Every time a member posts program output without using BBCode [ code] tags God kills a kitten.
Please fix your post.  There is also a link to the BBCode help under every message post box on the forums.

Last edited by ewaller (2014-05-14 15:18:45)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#1699 2014-05-14 15:25:29

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are those the first errors?  Can you capture the entire output and post it to pastebin and provide us a link?
How are you compiling it?  Are you using makepkg, or are you using something like  config and make install ?


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#1700 2014-05-15 03:54:37

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@jb660: Destroy is right - the "Driver in use" is what counts.

@Destroy: Good to know that it works now, and that you can run Passmark Performance Test. Need to try it (perhaps on Sunday I'll have a little time).

Was it upgrading to qemu 2 that did the trick with booting the VM?

About using only one VCPU: I noticed that you don't have any

smp sockets=...,cores=...

option.

@aw: Your VM startup script uses virsh nodedev-detach pci_0000_01_00_0 and virsh nodedev-reattach pci_0000_01_00_0 to detach and later reattach the graphics card. Does the reattach command reset the graphics card, and/or make it available to the host? Are these virsh commands in lieu of binding the card to pci-stub?

Now to the qemu / kvm command options: Aside from running --help and getting a shortlist of options, is there any source of information on the many command line options that are used?

Perhaps I'm too stupid or incapable of doing a Google search, but I get a feeling I'm being kept in the dark. Can someone point me to a (concise) source of information?

Offline

#1701 2014-05-15 04:18:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

@aw: Your VM startup script uses virsh nodedev-detach pci_0000_01_00_0 and virsh nodedev-reattach pci_0000_01_00_0 to detach and later reattach the graphics card. Does the reattach command reset the graphics card

no

and/or make it available to the host?

yes

Are these virsh commands in lieu of binding the card to pci-stub?

It's in place of binding and unbinding the device to vfio-pci

man virsh:

 nodedev-detach nodedev [--driver backend_driver] Detach nodedev from the host, so that it can safely be used by guests via <hostdev> passthrough. This is reversed with nodedev- reattach, and is done automatically for managed devices. For compatibility purposes, this command can also be spelled nodedev- dettach. Different backend drivers expect the device to be bound to different dummy devices. For example, QEMU's "kvm" backend driver (the default) expects the device to be bound to pci-stub, but its "vfio" backend driver expects the device to be bound to vfio-pci. The --driver parameter can be used to specify the desired backend driver.

virsh nodedev-dumpxml will also list the iommu group dependencies

Now to the qemu / kvm command options: Aside from running --help and getting a shortlist of options, is there any source of information on the many command line options that are used?

Use the source.  It's also useful to use virt-manager then look in the libvirt log files to extract the options it used for a given feature.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1702 2014-05-15 04:28:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can also use -device ? to get a list of devices and -device foo,? to get a list of options for device foo.  '?' works to get options lists for various other commands too.  So start with --help and drill down with other options.  There's also a fairly extensive man page for qemu.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1703 2014-05-15 11:52:26

manjack
Member
Registered: 2014-05-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ewaller wrote:

Are those the first errors?  Can you capture the entire output and post it to pastebin and provide us a link?
How are you compiling it?  Are you using makepkg, or are you using something like  config and make install ?

Sorry, first time posting code here. i have already fixed the post.

Yes, I'm using makepkg to compile it. The log is here

Thanks!

Last edited by manjack (2014-05-15 11:57:05)

Offline

#1704 2014-05-15 15:27:14

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

manjack wrote:

Sorry, first time posting code here. i have already fixed the post.

No worries.  And your readers appreciate it. smile

Yes, I'm using makepkg to compile it. The log is here

There is a mismatch somewhere.  The there is a macro definition in an included file that does not match the way it is being called.
Which qemu is it that you are trying to build?  I looked in the AUR and there are several, and of course, there is the one in the ABS.
Also, is your system fully up to date?  Have you run pacman -Syu to ensure you've the latest version of all packagers?


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#1705 2014-05-16 03:54:56

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You can also use -device ? to get a list of devices and -device foo,? to get a list of options for device foo.  '?' works to get options lists for various other commands too.  So start with --help and drill down with other options.  There's also a fairly extensive man page for qemu.

Thanks! And also for explaining virsh nodedev-detach... etc. I had used virsh before, but only for USB hosts. Under Xen I didn't have luck reattaching my graphics card to the host and using it (IIRC it would crash the host). Is that possible with kvm and a little luck?

A second scenario would be to pass the graphics card to another guest. Is this possible?

Offline

#1706 2014-05-16 04:12:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:
aw wrote:

You can also use -device ? to get a list of devices and -device foo,? to get a list of options for device foo.  '?' works to get options lists for various other commands too.  So start with --help and drill down with other options.  There's also a fairly extensive man page for qemu.

Thanks! And also for explaining virsh nodedev-detach... etc. I had used virsh before, but only for USB hosts. Under Xen I didn't have luck reattaching my graphics card to the host and using it (IIRC it would crash the host). Is that possible with kvm and a little luck?

Unfortunately the Linux graphics drivers behave very poorly when you attempt to unbind devices from them.  I haven't looked at fixing any of them yet, but I did run across this video recently that seems to show a GPU being dynamically released from Xorg, booting Windows, then being re-attached to Xorg - http://163216.genaker.web.hosting-test. … sthfI.html  Sounds like some scripts to do some xrandr magic before and after using the vm.  blu3bird84, are you around to share your scripts?

A second scenario would be to pass the graphics card to another guest. Is this possible?

If you mean sequentially, then yes, of course that works (assuming we can reset the device when qemu starts up).  There's no concurrent access support.

I'll also note for this thread that I tried a GeForce 8400GS and found that there was absolutely no way to reset the card.  A secondary bus reset doesn't phase it, the mode and framebuffer remain intact.  Windows guests still manage to work well with it, but not so much Linux.  The card is too old to spend much effort on it, but fyi, folks using vintage cards may run into such things.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1707 2014-05-16 06:55:58

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

Was it upgrading to qemu 2 that did the trick with booting the VM?

Yes. Qemu2, kernel 3.14.2 (with a patch for a problem with chipset SATA Marvell)

powerhouse wrote:

About using only one VCPU: I noticed that you don't have any

smp sockets=...,cores=...

I have added

-smp sockets=1,cores=2 -cpu host,hv-time 

and I getting a better response big_smile CPU Mark from 1633 to 2950, it is a 80%

Offline

#1708 2014-05-16 09:09:14

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally, in today, I success to build 2 VMs with 2 x GTX480 on another PC.
Before that, I can't boot up 2nd VM when 1st VM is working.

And here is another PC's spec:
CPU: Intel Core i7 4820K
MB: MSI X79A-GD45 Plus
RAM: DDR3-1600 4G x 8
GPU(Host): AMD Radeon HD5450
GPU(Guest 1): NVIDIA GeForce GTX480
GPU(Guest 2): Same as Guest 1
Host OS: Ubuntu Server 12.04-3 (Kernel: Linux 3.14.0-rc4 x86_64)
Guest OS: Ubuntu Server 12.04-4 (Both)

Few days ago I found that I doing a wrong installation steps before, that's why I can't passthrough my USB keyboard into my VM.
So, I decide to delete everything and restart at begin.

Here is my new steps:

1. Download VFIO kernel and qemu from aw's github (qemu-vfio & linux-vfio)

2. Install qemu-vfio first, note that if you want to use usb device in VM, you need to set configure as

./configure --prefix=/usr --enable-libusb

3. Then install linux-vfio, before reboot, set /etc/default/grub as

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on pci-stub.ids=10de:06c0,10de:0be5 vfio_iommu_type1.allow_unsafe_interrupts=1"

and update grub, then reboot.

4. After reboot, make vfio-bind.sh file at #1 post then save to /usr/bin

5. In #1 post, nbhs wrote Systemd Service so can make vfio-bind to bind GPU into VFIO automatically when boot, but Ubuntu 12.04 still using Upstart, so I have no idea but skip it.

6. So I make sh file so it can bind into VFIO when start VM:

#!/bin/bash #1st VM settings vfio-bind 0000:03:00.0 0000:03:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 3,sockets=1,cores=3,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:03f0:0024 \ -drive file=/home/eddieak/uc1.img,id=disk,format=raw,if=virtio

7. 1st VM is bootup, now bootup 2nd VM, same code, but different USB HID, GPU PCI num, and image:

#!/bin/bash #2nd VM settings vfio-bind 0000:01:00.0 0000:01:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 3,sockets=1,cores=3,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:413c:2003 \ -drive file=/home/eddieak/uc2.img,id=disk,format=raw,if=virtio

8. Finally, this time each VM can bootup, and can boot into Ubuntu tty.
Also, type lspci can found GTX480.



Now it can do well at this time, but I got problems

1. When I reboot one of VM, it will stuck at blank screen again, need to reboot whole host so that can boot 2 VMs together again.
2. If one of the VM stuck, I can't terminate it immediately, even I tried Ctrl+C is no use. Need to reboot or hit restart button.

Does any solutions?

Last edited by AKSN74 (2014-05-16 09:16:19)

Offline

#1709 2014-05-16 16:30:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Here is my new steps:

1. Download VFIO kernel and qemu from aw's github (qemu-vfio & linux-vfio)

No, there's nothing there that you need.  In your setup qemu 2.0 and kernel 3.15-rc should be perfectly fine.  Most users will also need to use the i915 VGA arbiter patches for host IGD graphics.  The only reason for 3.15-rc is because it has the bulk of the PCH root port ACS quirks.

2. Install qemu-vfio first, note that if you want to use usb device in VM, you need to set configure as

./configure --prefix=/usr --enable-libusb

3. Then install linux-vfio, before reboot, set /etc/default/grub as

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on pci-stub.ids=10de:06c0,10de:0be5 vfio_iommu_type1.allow_unsafe_interrupts=1"

and update grub, then reboot.

You also likely don't need vfio_iommu_type1.allow_unsafe_interrupts=1.  Any one on an AMD system and most people on Intel don't need this option.

4. After reboot, make vfio-bind.sh file at #1 post then save to /usr/bin

5. In #1 post, nbhs wrote Systemd Service so can make vfio-bind to bind GPU into VFIO automatically when boot, but Ubuntu 12.04 still using Upstart, so I have no idea but skip it.

6. So I make sh file so it can bind into VFIO when start VM:

#!/bin/bash #1st VM settings vfio-bind 0000:03:00.0 0000:03:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 3,sockets=1,cores=3,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:03f0:0024 \ -drive file=/home/eddieak/uc1.img,id=disk,format=raw,if=virtio

7. 1st VM is bootup, now bootup 2nd VM, same code, but different USB HID, GPU PCI num, and image:

#!/bin/bash #2nd VM settings vfio-bind 0000:01:00.0 0000:01:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 3,sockets=1,cores=3,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:413c:2003 \ -drive file=/home/eddieak/uc2.img,id=disk,format=raw,if=virtio

8. Finally, this time each VM can bootup, and can boot into Ubuntu tty.
Also, type lspci can found GTX480.



Now it can do well at this time, but I got problems

1. When I reboot one of VM, it will stuck at blank screen again, need to reboot whole host so that can boot 2 VMs together again.
2. If one of the VM stuck, I can't terminate it immediately, even I tried Ctrl+C is no use. Need to reboot or hit restart button.

Does any solutions?

Could you remove the comment around #define DEBUG_VFIO in hw/misc/vfio.c in your qemu tree, rebuild and install and report the last few lines you see before it hangs?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1710 2014-05-16 16:57:58

manjack
Member
Registered: 2014-05-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ewaller wrote:
manjack wrote:

Sorry, first time posting code here. i have already fixed the post.

No worries.  And your readers appreciate it. smile

Yes, I'm using makepkg to compile it. The log is here

There is a mismatch somewhere.  The there is a macro definition in an included file that does not match the way it is being called.
Which qemu is it that you are trying to build?  I looked in the AUR and there are several, and of course, there is the one in the ABS.
Also, is your system fully up to date?  Have you run pacman -Syu to ensure you've the latest version of all packagers?

Im trying to build the one on this post. I have tried to build again after i run pacman -Syu but i got the same error.

Offline

#1711 2014-05-17 01:30:37

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Very thanks for your reply, I'm really appreciate. Because I'm still a beginner, so there is some steps I don't know how to do.

aw wrote:

No, there's nothing there that you need.  In your setup qemu 2.0 and kernel 3.15-rc should be perfectly fine.  Most users will also need to use the i915 VGA arbiter patches for host IGD graphics.  The only reason for 3.15-rc is because it has the bulk of the PCH root port ACS quirks.

Do I need to uninstall previous QEMU and kernel? If so, how do I uninstall it? Because I tried "make uninstall" is no use, and type "make clean" will delete whole install data.

EDIT: I successfully uninstall the kernel and installing a new kernel now.


aw wrote:

Could you remove the comment around #define DEBUG_VFIO in hw/misc/vfio.c in your qemu tree, rebuild and install and report the last few lines you see before it hangs?

I use vi to open this file and found what you said

/* #define DEBUG_VFIO */ #ifdef DEBUG_VFIO #define DPRINTF(fmt, ...) \ do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0) #else #define DPRINTF(fmt, ...) \ do { } while (0) #endif

So I just need to remove these code and compile again?

Last edited by AKSN74 (2014-05-17 01:58:52)

Offline

#1712 2014-05-17 03:46:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Very thanks for your reply, I'm really appreciate. Because I'm still a beginner, so there is some steps I don't know how to do.

aw wrote:

No, there's nothing there that you need.  In your setup qemu 2.0 and kernel 3.15-rc should be perfectly fine.  Most users will also need to use the i915 VGA arbiter patches for host IGD graphics.  The only reason for 3.15-rc is because it has the bulk of the PCH root port ACS quirks.

Do I need to uninstall previous QEMU and kernel? If so, how do I uninstall it? Because I tried "make uninstall" is no use, and type "make clean" will delete whole install data.

EDIT: I successfully uninstall the kernel and installing a new kernel now.


aw wrote:

Could you remove the comment around #define DEBUG_VFIO in hw/misc/vfio.c in your qemu tree, rebuild and install and report the last few lines you see before it hangs?

I use vi to open this file and found what you said

/* #define DEBUG_VFIO */ #ifdef DEBUG_VFIO #define DPRINTF(fmt, ...) \ do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0) #else #define DPRINTF(fmt, ...) \ do { } while (0) #endif

So I just need to remove these code and compile again?

Just change:

/* #define DEBUG_VFIO */

to

#define DEBUG_VFIO

Recompile, install, and report the last few lines after it hangs, hopefully it gives us something to investigate next.  You can change it back and recompile/install to remove the debug output.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1713 2014-05-17 05:03:55

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Just change:

/* #define DEBUG_VFIO */

to

#define DEBUG_VFIO

Recompile, install, and report the last few lines after it hangs, hopefully it gives us something to investigate next.  You can change it back and recompile/install to remove the debug output.

OK, I'll try in these days and report immediately after the result.

Oh, I forgot to ask, this will show on dmesg or tty?

Last edited by AKSN74 (2014-05-17 08:11:31)

Offline

#1714 2014-05-17 19:28:51

manjack
Member
Registered: 2014-05-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does the qemu on this post have problem?

Offline

#1715 2014-05-17 22:14:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

manjack wrote:

does the qemu on this post have problem?

Do you really think this is a productive question?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1716 2014-05-19 03:16:29

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Just change:

/* #define DEBUG_VFIO */

to

#define DEBUG_VFIO

Recompile, install, and report the last few lines after it hangs, hopefully it gives us something to investigate next.  You can change it back and recompile/install to remove the debug output.

Hello, I'm testing right now.

when I reboot one of VM, it hangs, and vfio debug message unstoppable and repeatedly printing this

vfio: vfio_vga_read (0x3cc, 1) = 0x1 vfio: vfio_vga_read (0x3d4, 1) = 0x0 vfio: vfio_vga_write (0x3d4, 0x0,1) vfio: vfio_vga_write (0x3d4, 0xa8, 1) vfio: vfio_vga_write (0x3d4, 0x80a8, 2) vfio: vfio_bar_write (0000:01:00.0:BAR5+0x8, 0x61000c, 4) ← This may happen in some times

Because its too fast. So I use camera with high-speed shutter.
That binary code not only 0x3d4, but also 0x3cc 0x3d5, and recursing.
So I realize that it hangs like it stuck at these binary code.

EDIT: Disable nouveau still useless. And I still observe to message, so code may update.

EDIT2: After the message runs about 18 minutes the message only shows:

vfio: vfio_vga_read (0xa8044, 2) = 0x0

But after about 3 minutes, the message back to begining code again

Last edited by AKSN74 (2014-05-19 03:59:08)

Offline

#1717 2014-05-19 06:20:58

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, guys!

Nowdays, I also get problems with vga passthrough sad
Maybe, it is simple and you can help me?

I have Gentoo Linux, GeForce 7900GS with nvidia proprietary driver on host and Radeon HD6950 passthrough to guest.
On guest I tried to install Windows 7 SP1.

So, the main problem is very common - when I installed ATI driver on Windows (so, I have installed only video driver, not all CCC) - after reboot I got BSOD with atikmpag.sys

Sad, but true sad

Some more info about my system:

Linux StaticEvil 3.12.13-gentoo #8 SMP Sun May 18 23:54:05 MSK 2014 x86_64 Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz GenuineIntel GNU/Linux
dmesg | grep IOMMU [ 0.017465] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.017888] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.018537] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.204011] IOMMU 0 0xfed90000: using Queued invalidation [ 0.204168] IOMMU 1 0xfed91000: using Queued invalidation [ 0.204325] IOMMU: Setting RMRR: [ 0.204487] IOMMU: Setting identity map for device 0000:00:02.0 [0xab800000 - 0xaf9fffff] [ 0.205117] IOMMU: Setting identity map for device 0000:00:1d.0 [0xa837a000 - 0xa8386fff] [ 0.205407] IOMMU: Setting identity map for device 0000:00:1a.0 [0xa837a000 - 0xa8386fff] [ 0.205694] IOMMU: Setting identity map for device 0000:00:14.0 [0xa837a000 - 0xa8386fff] [ 0.205988] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.206150] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.12.13-gentoo root=/dev/sda3 ro net.ifnames=0 modprobe.blacklist=fglrx \ pci-stub.ids=1002:6719,1002:aa80 vfio_iommu_type1.allow_unsafe_interrupts=1
lsmod Module Size Used by 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_6.html topic_7.html topic_8.html topic_9.html kvm_intel 126738 0 kvm 369668 1 kvm_intel nvidia 11271958 30 ***
sudo lspci -nn 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_6.html topic_7.html topic_8.html topic_9.html 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80] ***
qemu-system-x86_64 --version QEMU emulator version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard
cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 options kvm allow_unsafe_assigned_interrupts=1

What can I forget? So, the most strange is, that once it boot good, without BSOD, but after reboot I get BSOD again and again. Strange and sad.

P.S. Before starting qemu I do:

sudo vfio-bind 0000:01:00.0 0000:01:00.1

then

pkexec qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device vfio-pci,host=01:00.0,x-vga=on -drive file=/var/lib/libvirt/images/Windows_7.img,id=disk,format=raw

Offline

#1718 2014-05-19 06:59:31

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello.
Is it possible to passthrough radeon card and use another radeon in host?
How don't blacklist radeon driver and get it works.

Looks like I just have to use psi-stub for guest card and don't blacklist radeon driver.

Last edited by zzz3000 (2014-05-19 07:19:21)

Offline

#1719 2014-05-19 08:48:18

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Namelles_One wrote:

Hi, guys!

Nowdays, I also get problems with vga passthrough sad
Maybe, it is simple and you can help me?

I have Gentoo Linux, GeForce 7900GS with nvidia proprietary driver on host and Radeon HD6950 passthrough to guest.
On guest I tried to install Windows 7 SP1.

So, the main problem is very common - when I installed ATI driver on Windows (so, I have installed only video driver, not all CCC) - after reboot I got BSOD with atikmpag.sys

Sad, but true sad

Some more info about my system:

Linux StaticEvil 3.12.13-gentoo #8 SMP Sun May 18 23:54:05 MSK 2014 x86_64 Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz GenuineIntel GNU/Linux
dmesg | grep IOMMU [ 0.017465] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.017888] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.018537] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.204011] IOMMU 0 0xfed90000: using Queued invalidation [ 0.204168] IOMMU 1 0xfed91000: using Queued invalidation [ 0.204325] IOMMU: Setting RMRR: [ 0.204487] IOMMU: Setting identity map for device 0000:00:02.0 [0xab800000 - 0xaf9fffff] [ 0.205117] IOMMU: Setting identity map for device 0000:00:1d.0 [0xa837a000 - 0xa8386fff] [ 0.205407] IOMMU: Setting identity map for device 0000:00:1a.0 [0xa837a000 - 0xa8386fff] [ 0.205694] IOMMU: Setting identity map for device 0000:00:14.0 [0xa837a000 - 0xa8386fff] [ 0.205988] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.206150] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.12.13-gentoo root=/dev/sda3 ro net.ifnames=0 modprobe.blacklist=fglrx \ pci-stub.ids=1002:6719,1002:aa80 vfio_iommu_type1.allow_unsafe_interrupts=1
lsmod Module Size Used by 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_6.html topic_7.html topic_8.html topic_9.html kvm_intel 126738 0 kvm 369668 1 kvm_intel nvidia 11271958 30 ***
sudo lspci -nn 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_6.html topic_7.html topic_8.html topic_9.html 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80] ***
qemu-system-x86_64 --version QEMU emulator version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard
cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 options kvm allow_unsafe_assigned_interrupts=1

What can I forget? So, the most strange is, that once it boot good, without BSOD, but after reboot I get BSOD again and again. Strange and sad.

P.S. Before starting qemu I do:

sudo vfio-bind 0000:01:00.0 0000:01:00.1

then

pkexec qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device vfio-pci,host=01:00.0,x-vga=on -drive file=/var/lib/libvirt/images/Windows_7.img,id=disk,format=raw

Not only AMD GPU need to mount into VM, but also the AMD Audio device need to mount together.

Offline

#1720 2014-05-19 08:50:39

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Namelles_One

I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly for me now.

Namelles_One wrote:

What can I forget? So, the most strange is, that once it boot good, without BSOD, but after reboot I get BSOD again and again. Strange and sad.

Offline

#1721 2014-05-19 08:57:00

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Not only AMD GPU need to mount into VM, but also the AMD Audio device need to mount together.

So, I tried with Audio too - BSOD anyway, sorry.

gneville wrote:

I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly for me now.

So, you did not with qemu-kvm, but with xen?

Offline

#1722 2014-05-19 09:46:19

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It works with qemu as well as Xen.

Namelles_One wrote:

So, you did not with qemu-kvm, but with xen?

Offline

#1723 2014-05-19 10:18:19

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

It works with qemu as well as Xen.

Namelles_One wrote:

So, you did not with qemu-kvm, but with xen?

It's all about rebooting, I see. But I get BSOD also when I boot qemu firstly, after rebooting host.

Or, even when I reboot guest with ATI drivers at least once - I got corrupted state and then can't boot it, before driver removing?

Offline

#1724 2014-05-19 10:55:26

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Offline

#1725 2014-05-19 17:57:56

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zzz3000 wrote:

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

I've update to 3.14.4 kernel and have stable Windows with driver! smile

So, let's rock!

Rating.png

Offline

#1726 2014-05-19 18:01:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Namelles_One wrote:
zzz3000 wrote:

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

If you don't want to go bleeding edge for the whole distro, you can use the Fedora virt-preview repo: http://fedoraproject.org/wiki/Virtualiz … Repository

It effectively gives you "rawhide" virt bits while keeping the rest on the latest stable version.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1727 2014-05-19 18:30:01

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Namelles_One wrote:
zzz3000 wrote:

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

If you don't want to go bleeding edge for the whole distro, you can use the Fedora virt-preview repo: http://fedoraproject.org/wiki/Virtualiz … Repository

It effectively gives you "rawhide" virt bits while keeping the rest on the latest stable version.

I use Gentoo. "Rawhide" - gives me an idea about kernel version.

Offline

#1728 2014-05-20 01:57:37

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also, I got another problem yesterday.

Because my target is build VM with GTX480, and want to run CUDA with GTX480 in VM.

As guest OS is Ubuntu 12.04-4, first I install NVIDIA Driver without a problem.
Then I install CUDA SDK 6.0, and still no problem. And it can compile the sample application.

But when I tried to run deviceQueryDrv, it got error.

CUDA_ERROR_INVALID_DEVICE (device specified is not a valid CUDA device)

So I downgrade my SDK version to 5.5, because GTX480 is old GPU, some functions in 6.0 may not support.
When I installed 5.5, and compile ver 5.5 samples, still got a same error while I ran deviceQueryDrv.

But before that, I can run CUDA while guest OS is Windows 7. And it can get matrix multiply result.
Maybe it's driver and SDK compatible problem (if it is VFIO problem, graphic driver will got issue first. Neither NVIDIA offical driver nor Nouveau.)

Last edited by AKSN74 (2014-05-20 01:58:52)

Offline

#1729 2014-05-20 04:58:30

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This post is very funny and exciting!!! Thanks again to all, specially to nbhs and AW

Now I'm trying to pass my SATA port Marvell at 08:00.0
lspci -tv

-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller +-01.0-[01]----00.0 NVIDIA Corporation G73 [GeForce 7600 GT] +-02.0 Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller +-14.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller +-16.0 Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 +-19.0 Intel Corporation 82579V Gigabit Network Connection +-1a.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 +-1b.0 Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller +-1c.0-[02]-- +-1c.1-[03]----00.0 Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller +-1c.3-[04]----00.0 Atheros Communications Inc. AR9462 Wireless Network Adapter +-1c.5-[05-06]----00.0-[06]----01.0 VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller +-1c.6-[07]----00.0 Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet +-1c.7-[08]----00.0 Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller +-1d.0 Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 +-1f.0 Intel Corporation Z77 Express Chipset LPC Controller +-1f.2 Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

but it's in a group

lsgroup

### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4) 00:1c.3 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 4 (rev c4) 00:1c.5 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 00:1c.6 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 (rev c4) 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) 03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 04:00.0 Network controller: Atheros Communications Inc. AR9462 Wireless Network Adapter (rev 01) 05:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30) 06:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) 07:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0) 08:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

I have been reading and it says  I must apply the path to override ACS but that is from march 2013 (https://lkml.org/lkml/2013/5/30/513) , also I found the v2 from February 2014 (https://lkml.org/lkml/2014/2/3/487) but it does haven the kernel parameters.

the questions are:
are for the same?
then I must apply the v2?
it's included in kernel 3.15?

Offline

#1730 2014-05-20 05:23:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Destroy wrote:

I have been reading and it says  I must apply the path to override ACS but that is from march 2013 (https://lkml.org/lkml/2013/5/30/513) , also I found the v2 from February 2014 (https://lkml.org/lkml/2014/2/3/487) but it does haven the kernel parameters.

the questions are:
are for the same?
then I must apply the v2?
it's included in kernel 3.15?

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1731 2014-05-20 06:26:15

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.

Last edited by AKSN74 (2014-05-20 13:35:14)

Offline

#1732 2014-05-20 11:25:58

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zzz3000 wrote:

Hello.
Is it possible to passthrough radeon card and use another radeon in host?
How don't blacklist radeon driver and get it works.

Looks like I just have to use psi-stub for guest card and don't blacklist radeon driver.

If you are on Arch Linux you can make it even more simple. I use a 7870 for my Linux system and a R9 290X for my VM. In Arch Linux just do the following:

#1 Create file '/etc/modules-load.d/vfio-pci.conf' with content:
vfio-pci

#2 Create file '/etc/modprobe.d/vfio-pci.conf' with content:
install vfio-pci /usr/bin/modprobe --ignore-install vfio-pci ; echo VENDOR DEVICE > /sys/bus/pci/drivers/vfio-pci/new_id

Just replace VENDOR and DEVICE with your appropriate IDs from 'lspci -n' output. Also this can be extended with any other device, if they are bound already (most probably USB devices) you can unbind them with the modprobe configuration.

Offline

#1733 2014-05-20 11:32:48

Chrissi
Member
Registered: 2014-05-20
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys!
I followed the instruction step by step, but when i try to start the vm with

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.$
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

i get the following error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The interesting thing is I got the same error output on a ubuntu based machine, so I guess its something hardware related or something I just don't get.

dmesg | grep -e DMAR -e IOMMU

[    0.000000] ACPI: DMAR 000000007c1789f8 0000B8 (v01 INTEL      HSW  00000001 INTL 00000001)
[    0.016013] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.016017] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.016082] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1

dmesg | grep vfio

[    2.966707] vfio-pci: probe of 0000:01:00.0 failed with error -22 ... for every card

full dmesg on pastebin

my system:
i7 4770
asrock extreme 6 (vt-d in bios enabled)
2x 290x / 1x290 (my good old coin mining cards)
grub boots with  intel_iommu=on

i am using qemu 2.0.0.3 (installed via pacman) and the mainline kernel with patches by OP.
Since I want to pass all my cards, I just blacklisted "radeon" in modprobe.d.

would be great if someone could give me a little advice how i may proceed with this. Thanks in advance !

Offline

#1734 2014-05-20 12:58:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chrissi wrote:

grub boots with  intel_iommu=on

Your pastebin says otherwise

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e60e5956-3c30-4b5d-a86b-5272a29bc246 rw quiet

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1735 2014-05-20 15:48:19

Chrissi
Member
Registered: 2014-05-20
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Chrissi wrote:

grub boots with  intel_iommu=on

Your pastebin says otherwise

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e60e5956-3c30-4b5d-a86b-5272a29bc246 rw quiet

sometimes there is a very easy way but we just don't see it. thank you.

Offline

#1736 2014-05-21 07:08:36

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Also, I got another problem yesterday.

Because my target is build VM with GTX480, and want to run CUDA with GTX480 in VM.

As guest OS is Ubuntu 12.04-4, first I install NVIDIA Driver without a problem.
Then I install CUDA SDK 6.0, and still no problem. And it can compile the sample application.

But when I tried to run deviceQueryDrv, it got error.

CUDA_ERROR_INVALID_DEVICE (device specified is not a valid CUDA device)

So I downgrade my SDK version to 5.5, because GTX480 is old GPU, some functions in 6.0 may not support.
When I installed 5.5, and compile ver 5.5 samples, still got a same error while I ran deviceQueryDrv.

But before that, I can run CUDA while guest OS is Windows 7. And it can get matrix multiply result.
Maybe it's driver and SDK compatible problem (if it is VFIO problem, graphic driver will got issue first. Neither NVIDIA offical driver nor Nouveau.)

Also I found that if I using Ubuntu 12.04 as guest OS, it always came out these message:

[81.114285] kvm [1993]: vcpu0 unhandled wrmsr: 0xxxx data 0

And here is my dmesg.
dmesg download

Hope it can solve reboot hangs when I using Ubuntu as guest OS.

Offline

#1737 2014-05-21 08:32:10

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all, so I'm having some trouble. I feel like I'm close to finishing this, but the qemu test does not display video though my passthrough'd card. And in my Windows guest OS, it disables the card because it's "not functioning correctly":

https://i.imgur.com/inKbcua.png

Here's my dmesg | grep pci-stub:

[ 0.000000] Command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a [ 0.000000] Kernel command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a [ 0.438320] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.438326] pci-stub 0000:02:00.0: claimed by stub [ 0.438329] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.438333] pci-stub 0000:02:00.1: claimed by stub

Anyone got any ideas?

Last edited by Slabity (2014-05-22 06:12:08)

Offline

#1738 2014-05-21 11:26:32

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Hey all, so I'm having some trouble. I feel like I'm close to finishing this, but the qemu test does not display video though my passthrough'd card. And in my Windows guest OS, it disables the card because it's "not functioning correctly":

https://i.imgur.com/inKbcua.png

Here's my dmesg | grep pci-stub:

[ 0.000000] Command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a [ 0.000000] Kernel command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a [ 0.438320] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.438326] pci-stub 0000:02:00.0: claimed by stub [ 0.438329] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.438333] pci-stub 0000:02:00.1: claimed by stub

Anyone got any ideas?

Looks like you're using virtual display right?
You need to disable virtual display, and plug a cable on your graphic card that you mounted.
Also, can you show us your launch command?

Last edited by AKSN74 (2014-05-21 11:27:38)

Offline

#1739 2014-05-21 16:19:07

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Looks like you're using virtual display right?
You need to disable virtual display, and plug a cable on your graphic card that you mounted.
Also, can you show us your launch command?

Well, if I run the example qemu script:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

In the above case, a black window appears with nothing in it. Nothing appears on my monitor attached through the passthrough'd card.

I am running the Windows VM through Virt-manager. Here's my configuration file:

http://pastebin.com/zJ4CbqZc

I try to remove the <video> section, but it seems to add itself back each time I run virt-manager.

EDIT: It seems the qemu script creates a black window, but if I type anything or resize the window, the window displays a console with the following:

compat_monitor0 console QEMU 2.0.0 monitor - type 'help' for more information' (qemu)

How is that possible if the script disables vga?

Last edited by Slabity (2014-05-21 16:32:15)

Offline

#1740 2014-05-21 17:30:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
AKSN74 wrote:

Looks like you're using virtual display right?
You need to disable virtual display, and plug a cable on your graphic card that you mounted.
Also, can you show us your launch command?

Well, if I run the example qemu script:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

In the above case, a black window appears with nothing in it. Nothing appears on my monitor attached through the passthrough'd card.

I am running the Windows VM through Virt-manager. Here's my configuration file:

http://pastebin.com/zJ4CbqZc

I try to remove the <video> section, but it seems to add itself back each time I run virt-manager.

EDIT: It seems the qemu script creates a black window, but if I type anything or resize the window, the window displays a console with the following:

compat_monitor0 console QEMU 2.0.0 monitor - type 'help' for more information' (qemu)

How is that possible if the script disables vga?

Are you using the i915 vga arbiter patch?

Offline

#1741 2014-05-21 18:13:13

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Are you using the i915 vga arbiter patch?

I believe so. I'm using the linux-mainline from the first post. The makepkg command automatically patches it, right? Then I just used pacman -U to install the kernel and change my boot command to use that kernel.

Offline

#1742 2014-05-21 21:00:19

thehighhat
Member
Registered: 2014-05-12
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
aw wrote:

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.


Same problem here.  I cannot reboot or reissue the qemu command without this error.  Do I need to undo the vfio-bind stuffs?

Offline

#1743 2014-05-22 01:44:38

esmth
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, first post here. I'm having some trouble trying to get this to work correctly. My hardware supports IOMMU and it is enabled in the BIOS, and the IOMMU kernel driver loads. My system specs are here: AMD A10-7850k, Gigabyte F2A88X-D3H. I'm using the integrated graphics as the host, and i'm passing though a reference R9-290x. I also have older GPUs i've tested that cause the same issue.  The problem is that when I run the command to start the VM, the colors on the host's monitor become distorted, and the 2nd monitor connected to the passed though GPU turns on, and sometimes gets passed the seabios screen and onto the  OS and then freezes the host and guest, sometimes it doesn't, and the both the guest and host freeze. When the PC freezes, it seems like everything connected to the PCI bus is down, but the host is still running (the terminal cursor on the host continues to blink), while the guest also seems to be frozen.

I use this command to start the guest:

sudo su bin/vfio-bind 0000:01:00:0 0000:01:00.1 ./qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/esmth/src/seabios/out/bios.bin -vga std -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/esmth/image.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/home/esmth/tmp/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -usb -usbdevice host:046d:c52b

if anyone can help, it'd be appreciated, thanks

edit: i git clone'd both seabios and qemu and compiled them , and am running kernel 3.15rc5 with the vfio and iommu options enabled

Last edited by esmth (2014-05-22 12:59:21)

Offline

#1744 2014-05-22 06:05:02

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

1.- I've compiled 3.15-rc5 but after boot, it gave me error with the Marvell card as you say.
2.- I've tried to applied the same patch for the Marvell card but it gave me one error
cat intel-iommu.c.rej

--- drivers/iommu/intel-iommu.c +++ drivers/iommu/intel-iommu.c @@ -3878,6 +3990,9 @@ iommu_disable_dev_iotlb(info); iommu_detach_dev(iommu, info->bus, info->devfn); iommu_detach_dependent_devices(iommu, pdev); + quirk_unmap_multi_requesters(pdev, + pci_multi_requesters(pdev)); + quirk_unmap_requester_id(pdev); free_devinfo_mem(info); spin_lock_irqsave(&device_domain_lock, flags);

so I manually edit the file and added the lines closes to line 4197 and compiled it but I had an error

 CC drivers/iommu/intel-iommu.o drivers/iommu/intel-iommu.c: In function 'quirk_unmap_multi_requesters': drivers/iommu/intel-iommu.c:1850:7: warning: passing argument 1 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default] pdev->bus->number, pdev->devfn); ^ drivers/iommu/intel-iommu.c:667:28: note: expected 'struct device *' but argument is of type 'int' static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn) ^ drivers/iommu/intel-iommu.c:1850:7: warning: passing argument 2 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default] pdev->bus->number, pdev->devfn); ^ drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned char' static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn) ^ drivers/iommu/intel-iommu.c:1850:7: warning: passing argument 3 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default] pdev->bus->number, pdev->devfn); ^ drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned int' static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn) ^ drivers/iommu/intel-iommu.c: In function 'quirk_map_multi_requester_ids': drivers/iommu/intel-iommu.c:1888:6: warning: passing argument 2 of 'domain_context_mapping_one' makes pointer from integer without a cast [enabled by default] translation); ^ drivers/iommu/intel-iommu.c:1718:12: note: expected 'struct intel_iommu *' but argument is of type 'int' static int domain_context_mapping_one(struct dmar_domain *domain, ^ drivers/iommu/intel-iommu.c: In function 'quirk_unmap_requester_id': drivers/iommu/intel-iommu.c:1907:7: warning: passing argument 1 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default] pdev->bus->number, pdev->devfn); ^ drivers/iommu/intel-iommu.c:667:28: note: expected 'struct device *' but argument is of type 'int' static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn) ^ drivers/iommu/intel-iommu.c:1907:7: warning: passing argument 2 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default] pdev->bus->number, pdev->devfn); ^ drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned char' static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn) ^ drivers/iommu/intel-iommu.c:1907:7: warning: passing argument 3 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default] pdev->bus->number, pdev->devfn); ^ drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned int' static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn) ^ drivers/iommu/intel-iommu.c: In function 'quirk_map_requester_id': drivers/iommu/intel-iommu.c:1933:4: warning: passing argument 2 of 'domain_context_mapping_one' makes pointer from integer without a cast [enabled by default] translation); ^ drivers/iommu/intel-iommu.c:1718:12: note: expected 'struct intel_iommu *' but argument is of type 'int' static int domain_context_mapping_one(struct dmar_domain *domain, ^ drivers/iommu/intel-iommu.c: In function 'domain_remove_one_dev_info': drivers/iommu/intel-iommu.c:4197:33: error: 'pdev' undeclared (first use in this function) quirk_unmap_multi_requesters(pdev, ^ drivers/iommu/intel-iommu.c:4197:33: note: each undeclared identifier is reported only once for each function it appears in make[2]: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_7.html topic_8.html topic_9.html [drivers/iommu/intel-iommu.o] Error 1 make[1]: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_7.html topic_8.html topic_9.html [drivers/iommu] Error 2 make: 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_7.html topic_8.html topic_9.html [drivers] Error 2

So I'll wait for more of your magic AW big_smile
Thanks again for your harder work

Offline

#1745 2014-05-22 06:10:11

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've completely reset my system from scratch to see if I could fix my issue. I still do not get any output on my passthrough'd VGA card.

Here is the qemu command I'm running:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

The result is a blank window with nothing being displayed on the monitor.

My host is running off of a Radeon HD 6870 with open-source drivers. My passthrough card is a Geforce 780:

$ lspci -v 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 2784 Flags: fast devsel, IRQ 17 Memory at ee000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=32M] I/O ports at d000 [size=128] Expansion ROM at ef000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau 02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1) Subsystem: eVga.com. Corp. Device 2784 Flags: fast devsel, IRQ 18 Memory at ef080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel
$ lspci -n 01:00.0 0300: 1002:6738 01:00.1 0403: 1002:aa88 02:00.0 0300: 10de:1004 (rev a1) 02:00.1 0403: 10de:0e1a (rev a1)

My kernel's command line is:

root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pci-stub.ids=10de:1004,10de:0e1a intel_iommu=on pcie_acs_override=downstream

I see some other people had similar issues but didn't report a solution. Is there something fundamental that I'm missing here?

Last edited by Slabity (2014-05-22 06:18:33)

Offline

#1746 2014-05-22 06:39:44

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

So I've completely reset my system from scratch to see if I could fix my issue. I still do not get any output on my passthrough'd VGA card.

Here is the qemu command I'm running:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

The result is a blank window with nothing being displayed on the monitor.

My host is running off of a Radeon HD 6870 with open-source drivers. My passthrough card is a Geforce 780:

I had similar troubles in the past with my AMD R9 290X inside guest and what helped me to get it fairly stable (at least for the first boot of VM as AMD Radeon cards are known to not support proper reset) was to boot my host with VGA enabled on card used for passthrough. So I changed in BIOS that my R9 290X will show the host BIOS post, boot the system and after binding this card to VFIO during boot it loads radeon driver for my second card (HD 7870) which works now as my Linux primary card. After this change my VM always allows me to use x-vga=on and shows SeaBIOS screen.

If I boot my system with second HD 7870 enabled as primary device in BIOS I was not able to use x-vga=on ever for the R9 290X.

Offline

#1747 2014-05-22 06:42:11

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

esmth wrote:
sudo su bin/vfio-bind 0000:01:00:0 0000:01:00.1 ./qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/esmth/src/seabios/out/bios.bin -vga std -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/esmth/image.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/home/esmth/tmp/Windows_8.1_Pro_X64_Activated_Final/Windows_8.1_Pro_X64_Activated.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -usb -usbdevice host:046d:c52b

if anyone can help, it'd be appreciated, thanks

Could you try to use it with

-vga none

Offline

#1748 2014-05-22 06:44:52

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, now please help me another way)

With VGA - all is good, but audio  - not(

In host Linus (I use ALSA) - all is good and best quality, but in Windows guest - quality is ugly, sound is like a frog - "He-quauq-llo!" 0_o

Where is problem may be?

P.S. I use

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa

and

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

Offline

#1749 2014-05-22 09:13:04

IyaFR
Member
Registered: 2014-05-22
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's possible to do this vga-passthrough with 1 gpu. I boot in multi-user.target and  i have modify my grub for enable the serial mode.
in grub.cfg:
terminal_input serial
terminal_output serial

And in the menuentry:
linux    .... quiet  APPEND console=ttyS0

After that, i run my vm with a ssh access.

Offline

#1750 2014-05-22 10:29:19

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've had partial success so far with passing through an AMD 6450 on an Asrock H61 and i5-2400. Partial success meaning I can get it to work invoking qemu directly but I can't get it working when using libvirt. There is no other discrete graphics present, the integrated graphics is for the host.

The kernel is 3.14 recompiled with vfio flags set and the i915 VGA arbiter patch applied, qemu version 2.0.0 and seabios version 1.7.4

lsgroups.sh shows

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]

   
hwinfo --gfxcard shows the 6450 (which is not the primary card) bound to vfio-pci (done through vfio-bind.sh 0000:01:00.0 0000:01:00.1)

20: PCI 100.0: 0300 VGA compatible controller (VGA) SysFS ID: /devices/pci0000:00/0000:00:01.0/0000:01:00.0 SysFS BusID: 0000:01:00.0 Hardware Class: graphics card Model: "ATI VGA compatible controller" Vendor: pci 0x1002 "ATI Technologies Inc" Device: pci 0x6779 SubVendor: pci 0x1043 "ASUSTeK Computer Inc." SubDevice: pci 0x047b Driver: "vfio-pci" Driver Modules: "vfio_pci" Memory Range: 0xe0000000-0xefffffff (ro,non-prefetchable,disabled) Memory Range: 0xf7c20000-0xf7c3ffff (rw,non-prefetchable,disabled) I/O Ports: 0xe000-0xefff (rw,disabled) Memory Range: 0xf7c00000-0xf7c1ffff (ro,non-prefetchable,disabled) IRQ: 11 (no events) Module Alias: "pci:v00001002d00006779sv00001043sd0000047Bbc03sc00i00" Driver Info #0: Driver Status: radeon is not active Driver Activation Cmd: "modprobe radeon" Config Status: cfg=no, avail=yes, need=no, active=unknown Attached to: #9 (PCI bridge) Primary display adapter: #10

ls -l /sys/bus/pci/drivers/vfio-pci/ looks well

total 0 lrwxrwxrwx 1 root root 0 May 22 11:14 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 May 22 11:14 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 --w------- 1 root root 4096 May 22 11:14 bind lrwxrwxrwx 1 root root 0 May 22 11:14 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 May 22 11:14 new_id --w------- 1 root root 4096 May 22 11:14 remove_id --w------- 1 root root 4096 May 22 11:14 uevent --w------- 1 root root 4096 May 22 11:14 unbind

 
Invoking qemu as

qemu-system-x86_64 -enable-kvm -M q35 -m 512 \ -smp 1,sockets=1,cores=1,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 

works (showing seabios output on the display attached to the 6450) with dmesg | grep -e vfio showing

[ 526.686460] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

but! When I attempt to start a libvirt managed guest defined from the following xml (note that here I'm not including the HDMI audio device)

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>test</name> <uuid>c4658435-dfb4-4c8d-afc2-f9dfad2e10f0</uuid> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='q35'>hvm</type> <loader>/usr/share/qemu/bios.bin</loader> </os> <features> <acpi/> </features> <cpu> <topology sockets='1' cores='1' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <controller type='sata' index='0'/> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> </qemu:commandline> </domain>

the error output is

error: Failed to start domain test error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fce2c9e4b00, 0x0, 0x20000000, 0x7fcdf0000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

dmesg | grep -e vfio then shows

[302278.633004] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [302278.633015] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

VT-d is enabled as per dmesg | grep -e DMAR -e IOMMU output

[ 0.000000] ACPI: DMAR 00000000be148e78 0000B0 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.018580] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.018585] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.018657] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.485319] DMAR: No ATSR found [ 0.485340] IOMMU 0 0xfed90000: using Queued invalidation [ 0.485341] IOMMU 1 0xfed91000: using Queued invalidation [ 0.485342] IOMMU: Setting RMRR: [ 0.485352] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.486624] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdf8d000 - 0xbdfb9fff] [ 0.486642] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdf8d000 - 0xbdfb9fff] [ 0.486652] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.486659] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 1.157895] [drm] DMAR active, disabling use of stolen memory

Permissions for /dev/vfio/ are

crw-rw-rw- 1 root root 251, 0 May 18 22:48 1 crw-rw-rw- 1 root root 10, 196 May 18 22:45 vfio

qemu is running its processes as root and the acl in /etc/libvirt/qemu.conf is

cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1" ]

I'm researched this but haven't found a way forward. Any pointers are very much appreciated. Thanks to the Arch community (in particular you who have provided mountains of insight here!) for all the work done so far, this has all been highly educational! smile

Offline

#1751 2014-05-22 12:58:41

esmth
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
esmth wrote:
sudo su bin/vfio-bind 0000:01:00:0 0000:01:00.1 ./qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/esmth/src/seabios/out/bios.bin -vga std -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/esmth/image.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/home/esmth/tmp/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -usb -usbdevice host:046d:c52b

if anyone can help, it'd be appreciated, thanks

Could you try to use it with

-vga none

I actually did run it with -vga none, but i copied the wrong command. I used -vga std to install windows in the VM

Offline

#1752 2014-05-22 14:57:23

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@esmth: I had a lot of similar troubles recently (similar to yours with sporadic freeze after start, freeze of VM only, etc...) and I found out that stability of VGA passthrough highly depends on which PCIe x16 slot I'm using. If you don't mind and if it is possible I would just experiment a bit with it.

Offline

#1753 2014-05-22 15:03:37

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

dmesg | grep -e vfio then shows

[302278.633004] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [302278.633015] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

Permissions for /dev/vfio/ are

crw-rw-rw- 1 root root 251, 0 May 18 22:48 1 crw-rw-rw- 1 root root 10, 196 May 18 22:45 vfio

qemu is running its processes as root and the acl in /etc/libvirt/qemu.conf is

cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1" ]

In addition, I've increased default memlock limits in /etc/security/limits.conf and confirmed with ulimit.

user@kvmhost-2:~> ulimit -l 1048576 user@kvmhost-2:~> ulimit -Hl 1048576

So I have no idea why dmesg still gives the RLIMIT_MEMLOCK 65k...

It does look like a permissions issue somewhere but I'm all out of ideas...

Last edited by siddharta (2014-05-22 15:04:30)

Offline

#1754 2014-05-22 18:43:44

Chrissi
Member
Registered: 2014-05-20
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally I figured out how to start and boot my vm. Passthrough works fine. Performance is also good. The only problem is and therefore I originally tried to use virtualitation that I am not able to start two virtual machines with a dedicated graphic card each. I always get:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/1: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I figured out that all my 3 cards are in the same iommu_group and therefore got in the same vfio_group (in my case its group 1).
Doing some research I even found the procedures/functions in the iommu.c part of the kernel in which iommu_groups are assigned.
But i cant manage to find a working solution to change this group.

Does anybody have an idea how I could solve this issue?
Help would be really appreciated, thanks smile

Offline

#1755 2014-05-22 18:47:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chrissi wrote:

Finally I figured out how to start and boot my vm. Passthrough works fine. Performance is also good. The only problem is and therefore I originally tried to use virtualitation that I am not able to start two virtual machines with a dedicated graphic card each. I always get:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/1: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I figured out that all my 3 cards are in the same iommu_group and therefore got in the same vfio_group (in my case its group 1).
Doing some research I even found the procedures/functions in the iommu.c part of the kernel in which iommu_groups are assigned.
But i cant manage to find a working solution to change this group.

Does anybody have an idea how I could solve this issue?
Help would be really appreciated, thanks smile

Google for "ACS override", apply the patch you find and enable it via kernel options.  If one of the cards can be attached to a PCH root port (00:1c.*), the v3.15 kernel may help.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1756 2014-05-22 18:50:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
siddharta wrote:

dmesg | grep -e vfio then shows

[302278.633004] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [302278.633015] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

Permissions for /dev/vfio/ are

crw-rw-rw- 1 root root 251, 0 May 18 22:48 1 crw-rw-rw- 1 root root 10, 196 May 18 22:45 vfio

qemu is running its processes as root and the acl in /etc/libvirt/qemu.conf is

cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1" ]

In addition, I've increased default memlock limits in /etc/security/limits.conf and confirmed with ulimit.

user@kvmhost-2:~> ulimit -l 1048576 user@kvmhost-2:~> ulimit -Hl 1048576

So I have no idea why dmesg still gives the RLIMIT_MEMLOCK 65k...

It does look like a permissions issue somewhere but I'm all out of ideas...

VFIO requires that the user has permission to lock the memory used by the guest.  You can change the user libvirt uses to root to avoid the problem (/etc/libvirt/qemu.conf).  You can also assign some other device to the VM using libvirt, then it will know to set the locked memory limit for the process appropriately.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1757 2014-05-22 19:26:49

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: Okay, so it seems I was wrong.

I can successfully run the test code if I set the host to run only on the intel processor. It does not matter which PCIe slot my passthrough card is in.

However, if I set my host to run on a dedicated card, I get no output from the other card running the test code.

Has anyone here successfully run a 2 card setup with 1 for the host and 1 for a passthrough? Or does anyone else have any intuition of what might be going on?

Last edited by Slabity (2014-05-22 23:00:52)

Offline

#1758 2014-05-22 23:22:04

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

EDIT: Okay, so it seems I was wrong.

I can successfully run the test code if I set the host to run only on the intel processor. It does not matter which PCIe slot my passthrough card is in.

However, if I set my host to run on a dedicated card, I get no output from the other card running the test code.

Has anyone here successfully run a 2 card setup with 1 for the host and 1 for a passthrough? Or does anyone else have any intuition of what might be going on?

Are you using 2 nvidia cards on your host?, because if you are you'll need to patch the nvidia drivers

Last edited by nbhs (2014-05-22 23:28:55)

Offline

#1759 2014-05-22 23:27:46

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Chrissi wrote:

Finally I figured out how to start and boot my vm. Passthrough works fine. Performance is also good. The only problem is and therefore I originally tried to use virtualitation that I am not able to start two virtual machines with a dedicated graphic card each. I always get:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/1: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I figured out that all my 3 cards are in the same iommu_group and therefore got in the same vfio_group (in my case its group 1).
Doing some research I even found the procedures/functions in the iommu.c part of the kernel in which iommu_groups are assigned.
But i cant manage to find a working solution to change this group.

Does anybody have an idea how I could solve this issue?
Help would be really appreciated, thanks smile

Google for "ACS override", apply the patch you find and enable it via kernel options.  If one of the cards can be attached to a PCH root port (00:1c.*), the v3.15 kernel may help.

You can get the acs override patch from my linux-mainline build, download linux-mainline.tar.gz, unpack it and it should be there

Offline

#1760 2014-05-22 23:43:43

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Slabity wrote:

EDIT: Okay, so it seems I was wrong.

I can successfully run the test code if I set the host to run only on the intel processor. It does not matter which PCIe slot my passthrough card is in.

However, if I set my host to run on a dedicated card, I get no output from the other card running the test code.

Has anyone here successfully run a 2 card setup with 1 for the host and 1 for a passthrough? Or does anyone else have any intuition of what might be going on?

Are you using 2 nvidia cards on your host?, because if you are you'll need to patch the nvidia drivers

No. I have one AMD and one Nvidia. I'm trying to let my host use the AMD card while I pass the Nvidia one to the guest.

$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)

I can pass through the card if I am running my host on integrated graphics. I can't seem to pass the card if I run the host on the AMD card, which is my current target.

Last edited by Slabity (2014-05-22 23:46:13)

Offline

#1761 2014-05-23 00:29:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
nbhs wrote:
Slabity wrote:

EDIT: Okay, so it seems I was wrong.

I can successfully run the test code if I set the host to run only on the intel processor. It does not matter which PCIe slot my passthrough card is in.

However, if I set my host to run on a dedicated card, I get no output from the other card running the test code.

Has anyone here successfully run a 2 card setup with 1 for the host and 1 for a passthrough? Or does anyone else have any intuition of what might be going on?

Are you using 2 nvidia cards on your host?, because if you are you'll need to patch the nvidia drivers

No. I have one AMD and one Nvidia. I'm trying to let my host use the AMD card while I pass the Nvidia one to the guest.

$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)

I can pass through the card if I am running my host on integrated graphics. I can't seem to pass the card if I run the host on the AMD card, which is my current target.

Are you using the open source driver or fglrx?

Offline

#1762 2014-05-23 00:37:38

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Are you using the open source driver or fglrx?

Open source.

$ lspci -v ... 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 21fa Flags: bus master, fast devsel, latency 0, IRQ 47 Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at ef320000 (64-bit, non-prefetchable) [size=128K] I/O ports at e000 [size=256] Expansion ROM at ef300000 [disabled] [size=128K] Capabilities: <access denied> Kernel driver in use: radeon Kernel modules: radeon ... 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 2784 Flags: fast devsel, IRQ 10 Memory at ee000000 (32-bit, non-prefetchable) [disabled] [size=16M] Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M] Memory at e8000000 (64-bit, prefetchable) [disabled] [size=32M] I/O ports at d000 [disabled] [size=128] Expansion ROM at ef000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau ...

Last edited by Slabity (2014-05-23 00:38:33)

Offline

#1763 2014-05-23 02:07:26

shelladept
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I am also trying to get PCI-Passthrough to work on my system and am having issues generating the test window.  I've followed all of the tutorial steps to the best of my knowledge. When attempting to run the test window (sudo):

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get the following errors:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/1: No such file or directory qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Some notes about my system:

MOBO: ASROCK Z77 (VT-d enabled)
CPU: i7-3770 (non-K)
HOST OS: Ubuntu 14.04 LTS
Primary GPU: On-Board Graphics
Secondary GPU: Saphire Radeon 4770

and configuration:

~$ grep GRUB_CMDLINE /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:xxxx,1002:yyyy" GRUB_CMDLINE_LINUX="" ~$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.15.0-rc6 root=/dev/mapper/rocko--vg-root ro quiet splash intel_iommu=on pci-stub.ids=1002:xxxx,1002:yyyy vt.handoff=7

#NOTE: I compiled 3.15.0-rc6 after having the same issue with the stock Ubuntu 14.04-provided kernel.  So it doesn't seem related.
#NOTE: I tried booting with and without 'intel_iommu=on' in grub.cfg
#NOTE: I can't seem to get "vfio_iommu_type1.allow_unsafe_interrupts=1" to show up in /proc/cmdline even though its been added to /etc/default/grub, which is used to update /boot/grub/grub.cfg in Ubuntu. I've run 'update-grub' and 'update-initramfs -u'.  I also tried adding "options vfio_iommu_type1 allow_unsafe_interrupts=1" to "/etc/modprobe.d/vfio_iommu_type1.conf".  I'm not sure if this addition is necessary in my case regardless.

I'm using QEMU 2.0.0 and included the pci_stub, vfio, vfio_pci, vfio_iommu_type1, kvm, and kvm_intel modules.  I also blacklisted 'radeon' and 'fglrx' on the host Ubuntu system and bound the Radeon 4770 to PCI-stub.

~$ qemu -version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1), Copyright (c) 2003-2008 Fabrice Bellard ~$ sudo apt-get install seabios | grep seabios seabios is already the newest version. ~$ lsmod | grep -e pci -e vfio -e kvm| tail -n 6 pci_stub 12622 1 vfio_pci 36474 0 vfio_iommu_type1 17632 0 kvm_intel 143255 0 kvm 446871 1 kvm_intel vfio 25164 2 vfio_iommu_type1,vfio_pci ~$ cat /etc/modprobe.d/blacklist.conf | grep -e radeon -e fglrx blacklist radeon blacklist fglrx ~$ lspci | grep Advanced 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV740 PRO [Radeon HD 4770] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] ~$ lspci -n | grep "01:00" 01:00.0 0300: 1002:xxxx 01:00.1 0403: 1002:yyyy ~$ dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.15.0-rc6 root=/dev/mapper/rocko--vg-root ro quiet splash intel_iommu=on pci-stub.ids=1002:xxxx,1002:yyyy vt.handoff=7 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.15.0-rc6 root=/dev/mapper/rocko--vg-root ro quiet splash intel_iommu=on pci-stub.ids=1002:xxxx,1002:yyyy vt.handoff=7 [ 47.239570] pci-stub: add 1002:xxxx sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 47.239584] pci-stub 0000:01:00.0: claimed by stub [ 47.239590] pci-stub: add 1002:yyyy sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 47.239597] pci-stub 0000:01:00.1: claimed by stub ~$ cat /etc/vfio-pci.cfg DEVICES="0000:01:00.0 0000:01:00.1"

Note that I have created /usr/bin/vfio-bind and /etc/systemd/system/binds-vfio-pci.service with no customizations.  I am not trying to bind usb ports yet, just the 2 radeon 'devices' (gpu+gpu audio component).  From what I can tell there is supposed to be a vfio 'group' containing these devices at /dev/vfio/1, but there isn't:

~$ ls /dev/vfio/ vfio

I'm basing this on 2 of the errors and the output of readlink, per further investigation:

vfio: failed to get group 1 vfio: error opening /dev/vfio/1 ~$ readlink /sys/bus/pci/devices/0000\:01\:00.0/iommu_group ../../../../kernel/iommu_groups/1

I apologize if I've spammed too much information here; I just want to make sure that I've included all relevant troubleshooting information.  Let me know if I need to share something else.  Or if i need to be clearer.  I suspect the key to my issue is the lack of '/dev/vfio/1' creation.  If true, I'm also not sure why it isn't being created.  I trawled through these 70+ pages yesterday trying to find a solution but am not sure where I've gone wrong.  I would appreciate any feedback.

Thanks in advance,
shelladept

Offline

#1764 2014-05-23 02:19:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shelladept wrote:

I apologize if I've spammed too much information here; I just want to make sure that I've included all relevant troubleshooting information.  Let me know if I need to share something else.  Or if i need to be clearer.  I suspect the key to my issue is the lack of '/dev/vfio/1' creation.  If true, I'm also not sure why it isn't being created.  I trawled through these 70+ pages yesterday trying to find a solution but am not sure where I've gone wrong.  I would appreciate any feedback.

Thanks for the detailed report, more information is always better.  What I don't see is any evidence that 1:00.0 and 1:00.1 are actually bound to vfio-pci, you've kind of skimmed over the vfio-bind scripts you're using.  What does 'lspci -ks 1:' report for the driver in use?  Your lsmod shows the reference count of vfio-pci is 0, so I don't think you've got any devices bound to it.  The vfio group dev file is created when devices are bound to vfio-pci.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1765 2014-05-23 04:03:30

shelladept
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
shelladept wrote:

I apologize if I've spammed too much information here; I just want to make sure that I've included all relevant troubleshooting information.  Let me know if I need to share something else.  Or if i need to be clearer.  I suspect the key to my issue is the lack of '/dev/vfio/1' creation.  If true, I'm also not sure why it isn't being created.  I trawled through these 70+ pages yesterday trying to find a solution but am not sure where I've gone wrong.  I would appreciate any feedback.

Thanks for the detailed report, more information is always better.  What I don't see is any evidence that 1:00.0 and 1:00.1 are actually bound to vfio-pci, you've kind of skimmed over the vfio-bind scripts you're using.  What does 'lspci -ks 1:' report for the driver in use?  Your lsmod shows the reference count of vfio-pci is 0, so I don't think you've got any devices bound to it.  The vfio group dev file is created when devices are bound to vfio-pci.

Thank you for the quick reply!  'lspci -ks 1:' returns the following:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV740 PRO [Radeon HD 4770] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0d00 Kernel driver in use: pci-stub 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] Kernel driver in use: pci-stub

If I understand what is going on here, 'pci-stub' has the 2 card 'devices' instead of vfio_pci? But why does it show only 1 in the lsmod output?  Is this because PCI 'sees' one object while VFIO distinguishes between the 2 'devices'? Pardon my ignorance, I'm still figuring out exactly how VFIO does its magic.  I also apologize for leaving out those particular details you mentioned.  I was trying to keep my post as short as possible.  To clear up any confusion, the verbatim contents of '/usr/bin/vfio-bind' are:

~$ cat /usr/bin/vfio-bind #!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Now on further review I think the 'modprobe vfio-pci' line may be redundant since I included vfio-pci in my /etc/modules file:

~$ cat /etc/modules | grep -e vfio -e pci vfio vfio_pci pci_stub vfio_iommu_type1

'vfio-bind' is launched by systemd using '/etc/systemd/system/binds-vfio-pci.service' at boot:

~$ cat /etc/systemd/system/binds-vfio-pci.service [Unit] Description=Binds devices to vfio-pci After=syslog.target [Service] EnvironmentFile=-/etc/vfio-pci.cfg Type=oneshot RemainAfterExit=yes ExecStart=-/usr/bin/vfio-bind $DEVICES [Install] WantedBy=multi-user.target

and 'binds-vfio-pci.service' obtains the value of $DEVICES from '/etc/vfio-pci.cfg':

~$ cat /etc/vfio-pci.cfg DEVICES="0000:01:00.0 0000:01:00.1"

At least that is what I believe is happening; I very well could be misinterpreting one of these files and would appreciate any clarification.  Let me know if I can provide further details and thank you again!

shelladept

Offline

#1766 2014-05-23 04:15:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shelladept wrote:
aw wrote:
shelladept wrote:

I apologize if I've spammed too much information here; I just want to make sure that I've included all relevant troubleshooting information.  Let me know if I need to share something else.  Or if i need to be clearer.  I suspect the key to my issue is the lack of '/dev/vfio/1' creation.  If true, I'm also not sure why it isn't being created.  I trawled through these 70+ pages yesterday trying to find a solution but am not sure where I've gone wrong.  I would appreciate any feedback.

Thanks for the detailed report, more information is always better.  What I don't see is any evidence that 1:00.0 and 1:00.1 are actually bound to vfio-pci, you've kind of skimmed over the vfio-bind scripts you're using.  What does 'lspci -ks 1:' report for the driver in use?  Your lsmod shows the reference count of vfio-pci is 0, so I don't think you've got any devices bound to it.  The vfio group dev file is created when devices are bound to vfio-pci.

Thank you for the quick reply!  'lspci -ks 1:' returns the following:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV740 PRO [Radeon HD 4770] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0d00 Kernel driver in use: pci-stub 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] Kernel driver in use: pci-stub

...

At least that is what I believe is happening; I very well could be misinterpreting one of these files and would appreciate any clarification.  Let me know if I can provide further details and thank you again!

Well, the device are bound to pci-stub, so either something is wrong with the scripts or they aren't being run.  So start with running them by hand...

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

Run lspci -k again, are the device bound to vfio-pci?  If so, is the service enabled in systemd?  If not, is the script executable?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1767 2014-05-23 04:54:54

shelladept
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
shelladept wrote:
aw wrote:

Thanks for the detailed report, more information is always better.  What I don't see is any evidence that 1:00.0 and 1:00.1 are actually bound to vfio-pci, you've kind of skimmed over the vfio-bind scripts you're using.  What does 'lspci -ks 1:' report for the driver in use?  Your lsmod shows the reference count of vfio-pci is 0, so I don't think you've got any devices bound to it.  The vfio group dev file is created when devices are bound to vfio-pci.

Thank you for the quick reply!  'lspci -ks 1:' returns the following:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV740 PRO [Radeon HD 4770] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0d00 Kernel driver in use: pci-stub 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series] Kernel driver in use: pci-stub

...

At least that is what I believe is happening; I very well could be misinterpreting one of these files and would appreciate any clarification.  Let me know if I can provide further details and thank you again!

Well, the device are bound to pci-stub, so either something is wrong with the scripts or they aren't being run.  So start with running them by hand...

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

Run lspci -k again, are the device bound to vfio-pci?  If so, is the service enabled in systemd?  If not, is the script executable?

Thank you again, aw!  Running vfio-bind manually works:

~$ lsmod | grep vfio vfio_iommu_type1 17632 0 vfio_pci 36474 0 vfio 25164 2 vfio_iommu_type1,vfio_pci

so the issue is probably related to my configuration of systemd/binds-vfio-pci.service.

It looks like I am getting the test window to display now, although I get some weird colors on my host screen output afterward.  I'm guessing this is related to the Intel issue addressed by linux-mainline.tar.gz per the first page (If memory serves--more research for me!).  I may have to compile another kernel but this is plenty of progress for me to get back on track.  Thanks again!

shelladept

Offline

#1768 2014-05-23 04:57:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shelladept wrote:

It looks like I am getting the test window to display now, although I get some weird colors on my host screen output afterward.  I'm guessing this is related to the Intel issue addressed by linux-mainline.tar.gz per the first page (If memory serves--more research for me!).  I may have to compile another kernel but this is plenty of progress for me to get back on track.  Thanks again!

Yes, i915 patches should clear up the host screen.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1769 2014-05-23 05:50:28

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, sorry for posting again, but I'm at a real loss here on what's wrong with my setup.

My current system has 3 VGA outputs. Intel, AMD, and Nvidia:

$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)

When I run the example script:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

My system works perfectly when my host is using intel graphics. A blank window appears and the seabios screen is being passed to another monitor.

But when I set my host to use the AMD card, the blank window appears, but there is nothing being passed to the other monitor.

I am using the open source radeon drivers.

Sorry again for posting so much, but I'm running out of ideas.

Offline

#1770 2014-05-23 08:22:52

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

VFIO requires that the user has permission to lock the memory used by the guest.  You can change the user libvirt uses to root to avoid the problem (/etc/libvirt/qemu.conf).  You can also assign some other device to the VM using libvirt, then it will know to set the locked memory limit for the process appropriately.

Thanks for this. I dug back into qemu.conf and found that, while qemu processes were supposed to run as root

user = "root" group = "root"

I had missed the clear_emulator_capabilities flag

# If clear_emulator_capabilities is enabled, libvirt will drop all # privileged capabilities of the QEmu/KVM emulator. This is enabled by # default. # # Warning: Disabling this option means that a compromised guest can # exploit the privileges and possibly do damage to the host. # clear_emulator_capabilities = 0

which was set to its default restrictive setting.

For my understanding, could you please clarify

aw wrote:

You can also assign some other device to the VM using libvirt, then it will know to set the locked memory limit for the process appropriately.

Thanks again, I'm thrilled to have this working.

Offline

#1771 2014-05-23 11:43:57

Chrissi
Member
Registered: 2014-05-20
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Google for "ACS override", apply the patch you find and enable it via kernel options.  If one of the cards can be attached to a PCH root port (00:1c.*), the v3.15 kernel may help.

I just installed the new 3.15-rc6 kernel which in my understanding contains the acs override patch. 
Attaching one card to one vm still works fine. ( Starting more than one vm each with its own card assigned still gives the same 'failed to get group 1' error. )
I can't figure out how to attach a card to a specific PCH root port and what I would have to do afterwards to pass those root ports to my vm.
It would be great if you could give me some more advice regarding this topic.

Offline

#1772 2014-05-23 15:43:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Hey guys, sorry for posting again, but I'm at a real loss here on what's wrong with my setup.

My current system has 3 VGA outputs. Intel, AMD, and Nvidia:

$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)

When I run the example script:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

My system works perfectly when my host is using intel graphics. A blank window appears and the seabios screen is being passed to another monitor.

But when I set my host to use the AMD card, the blank window appears, but there is nothing being passed to the other monitor.

I am using the open source radeon drivers.

Sorry again for posting so much, but I'm running out of ideas.

It sounds like VGA arbitration isn't working when the AMD card is in use by the host.  What does 'dmesg | grep vgaarb' show?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1773 2014-05-23 15:46:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
aw wrote:

VFIO requires that the user has permission to lock the memory used by the guest.  You can change the user libvirt uses to root to avoid the problem (/etc/libvirt/qemu.conf).  You can also assign some other device to the VM using libvirt, then it will know to set the locked memory limit for the process appropriately.

Thanks for this. I dug back into qemu.conf and found that, while qemu processes were supposed to run as root

user = "root" group = "root"

I had missed the clear_emulator_capabilities flag

# If clear_emulator_capabilities is enabled, libvirt will drop all # privileged capabilities of the QEmu/KVM emulator. This is enabled by # default. # # Warning: Disabling this option means that a compromised guest can # exploit the privileges and possibly do damage to the host. # clear_emulator_capabilities = 0

which was set to its default restrictive setting.

For my understanding, could you please clarify

libvirt attempts to run qemu with as few privileges as possible to protect the host from the guest "breaking out" of it's VM.  Using an assigned device requires certain privileges which libvirt retains when device assignment is used.   When libvirt doesn't know about an assigned device, it can't grant those privileges.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1774 2014-05-23 16:01:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chrissi wrote:
aw wrote:

Google for "ACS override", apply the patch you find and enable it via kernel options.  If one of the cards can be attached to a PCH root port (00:1c.*), the v3.15 kernel may help.

I just installed the new 3.15-rc6 kernel which in my understanding contains the acs override patch.

Nope, 3.15 includes quirks approved by Intel to advertise the ACS-like isolation capabilities of PCH-based root ports and make sure they're enabled.  This means that the hardware is actually providing the isolation required.  The ACS override patch is a way for the user to override the hardware isolation requirement and is not going upstream.

Attaching one card to one vm still works fine. ( Starting more than one vm each with its own card assigned still gives the same 'failed to get group 1' error. )
I can't figure out how to attach a card to a specific PCH root port and what I would have to do afterwards to pass those root ports to my vm.
It would be great if you could give me some more advice regarding this topic.

The host root ports never get passed to the VM.  Which root ports are used is a physical property of your motherboard layout.  It may require that you move the card to a different slot to connect to the PCH root port, if it's even possible.  On my system I have:

$ lspci | grep "Root Port" 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5) 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)

You can tell by the description (and bus addresses) that 00:01.0 is a processor-based root port while 00:1c.* are PCH-based root ports.  My graphics cards are installed as:

$ lspci -tv | grep -e NVIDIA -e AMD +-01.0-[01]--+-00.0 NVIDIA Corporation GK208 [GeForce GT 635] | \-00.1 NVIDIA Corporation Device 0e0f +-1c.0-[02]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R7 240 OEM] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

So that only one is connected to the processor-based root port and the other is connected to the PCH-based root port.  This way I don't need the ACS override patch since the v3.15 ACS quirks work for this chipset.  Intel currently has no plans to provide information about whether hardware isolation is possible with processor-based root ports, so we shouldn't expect similar quirks for them anytime soon.  We can hope that they've learned their lesson with ACS and intend to support it on future consumer grade processors.  BTW, even though the root port description indicates a Xeon processor, this is just a regular i5, apparently the PCI device ID is re-used between parts.  Actual Xeon processors do have ACS support on the processor-based root ports.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1775 2014-05-23 18:15:59

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It sounds like VGA arbitration isn't working when the AMD card is in use by the host.  What does 'dmesg | grep vgaarb' show?

When I run with Intel as the host:

[ 0.333676] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.333680] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.333682] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.333682] vgaarb: loaded [ 0.333683] vgaarb: bridge control possible 0000:02:00.0 [ 0.333683] vgaarb: bridge control possible 0000:01:00.0 [ 0.333684] vgaarb: no bridge control possible 0000:00:02.0 [ 2.855092] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none [ 3.069294] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io

When I run with AMD as the host:

[ 0.330061] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.330064] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.330065] vgaarb: loaded [ 0.330065] vgaarb: bridge control possible 0000:02:00.0 [ 0.330066] vgaarb: bridge control possible 0000:01:00.0 [ 2.609390] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none

Doesn't seem too much different. Anything catch your eye?

In case it matters:

$ lspci -tv +-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] +-01.1-[02]--+-00.0 NVIDIA Corporation GK110 [GeForce GTX 780] | \-00.1 NVIDIA Corporation GK110 HDMI Audio

Last edited by Slabity (2014-05-23 18:19:00)

Offline

#1776 2014-05-23 18:26:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
aw wrote:

It sounds like VGA arbitration isn't working when the AMD card is in use by the host.  What does 'dmesg | grep vgaarb' show?

When I run with Intel as the host:

[ 0.333676] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.333680] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.333682] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.333682] vgaarb: loaded [ 0.333683] vgaarb: bridge control possible 0000:02:00.0 [ 0.333683] vgaarb: bridge control possible 0000:01:00.0 [ 0.333684] vgaarb: no bridge control possible 0000:00:02.0 [ 2.855092] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none [ 3.069294] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io

When I run with AMD as the host:

[ 0.330061] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.330064] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.330065] vgaarb: loaded [ 0.330065] vgaarb: bridge control possible 0000:02:00.0 [ 0.330066] vgaarb: bridge control possible 0000:01:00.0 [ 2.609390] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none

Doesn't seem too much different. Anything catch your eye?

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1777 2014-05-23 18:33:14

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

Well thank god we finally found the issue, even if I have no idea what the VGA arbiter is.

Here's my relevant lspci:

http://pastebin.com/4YHn4eH0

Thanks again for the quick response. Sorry about being pushy earlier, I'm still a bit put around from surgery yesterday.

Offline

#1778 2014-05-23 18:37:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
aw wrote:

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

Well thank god we finally found the issue, even if I have no idea what the VGA arbiter is.

Here's my relevant lspci:

http://pastebin.com/4YHn4eH0

Thanks again for the quick response. Sorry about being pushy earlier, I'm still a bit put around from surgery yesterday.

VGA+ is set on 00:01.0 which is the root port for the AMD card, so VGA accesses are going there, not to the Nvidia card.  I'll take a look.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1779 2014-05-23 19:02:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Slabity wrote:
aw wrote:

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

Well thank god we finally found the issue, even if I have no idea what the VGA arbiter is.

Here's my relevant lspci:

http://pastebin.com/4YHn4eH0

Thanks again for the quick response. Sorry about being pushy earlier, I'm still a bit put around from surgery yesterday.

VGA+ is set on 00:01.0 which is the root port for the AMD card, so VGA accesses are going there, not to the Nvidia card.  I'll take a look.

BTW, if you want to make it work in the meantime, try this:

vga-clear.sh

#!/bin/sh CTRL=$(setpci -s $1 3e.w) CTRL=$(printf %04x $(( 0x$CTRL & ~0x8 ))) setpci -s $1 3e.w=$CTRL

vga-set.sh

#!/bin/sh CTRL=$(setpci -s $1 3e.w) CTRL=$(printf %04x $(( 0x$CTRL | 0x8 ))) setpci -s $1 3e.w=$CTRL

Then before running the guest, do:

sudo vga-clear.sh 00:01.0 sudo vga-set.sh 00:01.1

This is only a workaround and assumes that radeon isn't just lying about no longer needing VGA access.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1780 2014-05-23 19:28:51

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

BTW, if you want to make it work in the meantime, try this:

vga-clear.sh

#!/bin/sh CTRL=$(setpci -s $1 3e.w) CTRL=$(printf %04x $(( 0x$CTRL & ~0x8 ))) setpci -s $1 3e.w=$CTRL

vga-set.sh

#!/bin/sh CTRL=$(setpci -s $1 3e.w) CTRL=$(printf %04x $(( 0x$CTRL | 0x8 ))) setpci -s $1 3e.w=$CTRL

Then before running the guest, do:

sudo vga-clear.sh 00:01.0 sudo vga-set.sh 00:01.1

This is only a workaround and assumes that radeon isn't just lying about no longer needing VGA access.

Wow, that works perfectly with the test script! Thank you so much!

Offline

#1781 2014-05-23 19:36:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Wow, that works perfectly with the test script! Thank you so much!

Does radeon still work ok for the host?

Anyone using radeon on the host (APU users) may want to try this too.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1782 2014-05-23 20:15:09

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Slabity wrote:

Wow, that works perfectly with the test script! Thank you so much!

Does radeon still work ok for the host?

Anyone using radeon on the host (APU users) may want to try this too.

Yep, my AMD card didn't even noticeably react through the process.

It's a good thing too, my monitor doesn't like the output that the graphics card gives for some reason...

Now I just gotta get this set up with virsh.

Offline

#1783 2014-05-23 20:34:52

HPvirt
Member
Registered: 2014-05-23
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everybody

That post is AWESOME! It fits perfectly in what I was wondering these last years: playing in windows (or other tasks without Linux counterpart, as BluRay reading) and use Linux for everyday tasks without to reboot everytime I need to go from one to the other.

I played with Virtual Box and use it for testing Linux, but gaming is a no way.
I've seen Xen, but it's lack of support for NVidia mainstream videocards slowed me.

KVM seems the way to go for, thanks to you nbhs and this topic.

Before buying some pieces, I would like to have your opinion on them.
I'm planning to buy:
- AsRock Z97 Extreme6
- Core i7-4790 (Haswell)
Both supports VT-d.

I already have 16Go of Ram and a NVidia 680.

My wish:
- Use Linux Mint as host with i7 IGP to display onto a Dell U3014 (DisplayPort) with the Intel LAN
- Use Win7 as guest with GTX 680 to display onto a Dell 3007 (DVI Dual Link) with the Realtek LAN
--> Passthrough for to Win7: GTX 680, Realtek LAN, the ASMedia USB3 controller and the ASMedia SATA Controler.

Linux and Win7 would be installed on 2 different SSDs.

I've read like 25 pages of the topic, and planned to read all. But I've an opportunity to buy these 2 pieces at a good price, so I ask now.

At first glance, does anyone see an impossibility in my project?

I think that I can launch the Win7 VM with more or less memory (playing with -m value of qemu), depending on tasks I planned to do on Win7:
- FullHD Video editing => 12Go or more
- Gaming => 4-8Go
For my Win7 it would be like adding or removing ram. I'm right?
Later, I planned to go to 24 or 32Go Ram if 16Go are not enough.

thanks

(And I beg your pardon for my poor english wording sad)

Last edited by HPvirt (2014-05-23 20:49:36)


Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

Offline

#1784 2014-05-23 20:43:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

HPvirt wrote:

Hi everybody

That post is AWESOME! It fits perfectly in what I was wondering these last years: playing in windows (or other tasks without Linux counterpart, as BluRay reading) and use Linux for everyday tasks without to reboot everytime I need to go from one to the other.

I played with Virtual Box and use it for testing Linux, but gaming is a no way.
I've seen Xen, but it's lack of support for NVidia mainstream videocards slowed me.

KVM seems the way to go for, thanks to you nbhs and this topic.

Before buying some pieces, I would like to have your opinion on them.
I'm planning to buy:
- AsRock Z97 Extreme6
- Core i7-4790 (Haswell)
Both supports VT-d.

I already have 16Go of Ram and a NVidia 680.

My wish:
- Use Linux Mint as host with i7 IGP to display onto a Dell U3014 (DisplayPort) with the Intel LAN
- Use Win7 as guest with GTX 680 to display onto a Dell 3007 (DVI Dual Link) with the Realtek LAN
--> Passthrough for to Win7: GTX 680, Realtek LAN, the ASMedia USB3 controller and the ASMedia SATA Controler.

Linux and Win7 would be installed on 2 different SSDs.

I've read like 25 pages of the topic, and planned to read all. But I've an opportunity to buy these 2 pieces at a good price, so I ask now.

At first glance, does anyone see an impossibility in my project?

I think that I can launch the Win7 VM with more or less memory (playing with -m value of qemu), depending on tasks I planned to do on Win7:
- FullHD Video editing => 12Go or more
- Gaming => 4-8Go
For my Win7 it would be like adding or removing ram. I'm right?
Later, I planned to go to 24 or 32Go Ram if 16Go are not enough.

thanks

I would not recommend doing device assignment of a Realtek LAN, I've only recently made it work and it's just scary.  IMHO there's not sufficient value in passing LAN or disk controllers to a guest in a home environment.  Virtio LAN and disk controllers will provide you with 90% of the performance with less hassle.  I don't know if you're asking about dynamically changing memory, but you can certainly adjust the memory for a guest between boots.  If you plan to use IGD for the host, you will need the i915 patches.  I don't know the state of Z97 wrt the PCH root port ACS support, so you may need the ACS override patches as well.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1785 2014-05-23 21:03:55

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To add what aw said, I recently just got the 4790 when I found out the 4770K did not support Vt-d.

The only reason I got it was because it was on sale for cheaper than the normal 4770. I've only recently compared the two, and honestly it's not worth the extra $15. Hell, it's not even worth an extra $5 in my opinion. The speed is completely unnoticeable, as the only difference is the clock speed. Unless you can find the 4790 someplace cheaper than the 4770, stick with the 4770. I will absolutely guarantee that you will not notice any.

Second, that motherboard is designed for overclocking and high end gaming. The non-k series can not be overclocked, and if you're gaming in a VM then I doubt you'll be taking advantage of the 'gamer features' that the motherboard is flaunting.

I'm not saying that you shouldn't get these things, but there are cheaper options that'll suit your needs perfectly fine.

Last edited by Slabity (2014-05-23 21:07:27)

Offline

#1786 2014-05-23 21:18:06

HPvirt
Member
Registered: 2014-05-23
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I would not recommend doing device assignment of a Realtek LAN, I've only recently made it work and it's just scary.  IMHO there's not sufficient value in passing LAN or disk controllers to a guest in a home environment.  Virtio LAN and disk controllers will provide you with 90% of the performance with less hassle.

Device management under Linux is not the domain I have the most knowledge in. But I'm eager to learn. smile

If I do device assignment of the Intel LAN, it would be easier?
Performance is a need I have.
But, I will try anyway, for the challenge and to learn.

aw wrote:

I don't know if you're asking about dynamically changing memory, but you can certainly adjust the memory for a guest between boots.

It was at boot time. It seems clear it's possible, but I prefer ask and not assume anything.

aw wrote:

If you plan to use IGD for the host, you will need the i915 patches.  I don't know the state of Z97 wrt the PCH root port ACS support, so you may need the ACS override patches as well.

Yes the IGP will be for the host. I planned to have 2 keayboards and 2 mouses: my son will be able to play while I do something else. And if the need to have a secondary GPU arises, I'll buy another, but right now, my wallet and my banker would scream if I buy a top GPU. big_smile


Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

Offline

#1787 2014-05-23 21:22:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

HPvirt wrote:
aw wrote:

I would not recommend doing device assignment of a Realtek LAN, I've only recently made it work and it's just scary.  IMHO there's not sufficient value in passing LAN or disk controllers to a guest in a home environment.  Virtio LAN and disk controllers will provide you with 90% of the performance with less hassle.

Device management under Linux is not the domain I have the most knowledge in. But I'm eager to learn. smile

If I do device assignment of the Intel LAN, it would be easier?
Performance is a need I have.
But, I will try anyway, for the challenge and to learn.

Yes, Intel NICs are more amenable to device assignment, but really virtio can easily handle gigabit networks.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1788 2014-05-23 21:35:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
aw wrote:

BTW, if you want to make it work in the meantime, try this:

vga-clear.sh

#!/bin/sh CTRL=$(setpci -s $1 3e.w) CTRL=$(printf %04x $(( 0x$CTRL & ~0x8 ))) setpci -s $1 3e.w=$CTRL

vga-set.sh

#!/bin/sh CTRL=$(setpci -s $1 3e.w) CTRL=$(printf %04x $(( 0x$CTRL | 0x8 ))) setpci -s $1 3e.w=$CTRL

Then before running the guest, do:

sudo vga-clear.sh 00:01.0 sudo vga-set.sh 00:01.1

This is only a workaround and assumes that radeon isn't just lying about no longer needing VGA access.

Wow, that works perfectly with the test script! Thank you so much!

Slabity, if you'd be willing to test it, I think this might resolve the problem (kernel patch):  http://pastebin.com/2ukyQbkG

I'll need to swap around some hardware to make my AMD card the default before I can test it (curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1789 2014-05-23 21:51:05

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

(curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

Is there even such a thing? I've yet to see more options than PCI or PEG (and AGP back in the days).

Offline

#1790 2014-05-23 21:52:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:
aw wrote:

(curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

Is there even such a thing? I've yet to see more options than PCI or PEG (and AGP back in the days).

I've seen it.  I can't remember where, but I'm pretty sure at least one of my systems lets me do it (which means it's either Gigabyte or Lenovo).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1791 2014-05-23 21:53:55

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Slabity, if you'd be willing to test it, I think this might resolve the problem (kernel patch):  http://pastebin.com/2ukyQbkG

Sure! I'll try it out immediately!

aw wrote:

I'll need to swap around some hardware to make my AMD card the default before I can test it (curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

Blargh, I know what you mean. I always loved ASUS years ago when I was just into overclocking. But now it feels like their boards have no utility. And their technical support is hell. But I'm stuck with them for the time being.

Last edited by Slabity (2014-05-23 21:54:40)

Offline

#1792 2014-05-23 21:56:17

HPvirt
Member
Registered: 2014-05-23
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

To add what aw said, I recently just got the 4790 when I found out the 4770K did not support Vt-d.

The only reason I got it was because it was on sale for cheaper than the normal 4770. I've only recently compared the two, and honestly it's not worth the extra $15. Hell, it's not even worth an extra $5 in my opinion. The speed is completely unnoticeable, as the only difference is the clock speed. Unless you can find the 4790 someplace cheaper than the 4770, stick with the 4770. I will absolutely guarantee that you will not notice any.

I'll have the 4790 cheaper than a 4770 smile

Slabity wrote:

Second, that motherboard is designed for overclocking and high end gaming. The non-k series can not be overclocked, and if you're gaming in a VM then I doubt you'll be taking advantage of the 'gamer features' that the motherboard is flaunting.
I'm not saying that you shouldn't get these things, but there are cheaper options that'll suit your needs perfectly fine.

During my search for the MB, I've seen that VT-d support is random for Asus, Gigabyte, MSI as it's not officialy supported. And when it's working, it's mostly haphazardly. Updating a bios can remove the support, which come back with the next update... or not.
So AsRock was the way to go.
As I want to passthrough Sata & LAN & USB, I need 2 controllers of each type if I've understand how the passthrough works (my flaw is perhaps here), and I don't want to buy LAN and SATA card. So choices are limited. "Gamer features" are not the reason I choose this MB.

But, I take into account all advices, so if you know a MB compliant with my needs, I'll check it and see if I can buy it in France.

Go back to the topic: I'll repost here during the configuration of my system, for help (most probably), or, if it works flawlessly, to record my conf as working smile

Last edited by HPvirt (2014-05-23 22:02:06)


Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

Offline

#1793 2014-05-23 22:14:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
aw wrote:

Slabity, if you'd be willing to test it, I think this might resolve the problem (kernel patch):  http://pastebin.com/2ukyQbkG

Sure! I'll try it out immediately!

Hmm, it's not working.  I can reproduce it now, so I'll keep poking at it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1794 2014-05-24 01:54:06

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried in these days that I want to make CUDA works in guest OS.
I've tested many possible reasons cause CUDA can't work if guest OS is Ubuntu.

If I use Windows 7 as guest, it can work perfectly.
But if I use Ubuntu as guest, it can install graphic driver, can install CUDA SDK, can compile sample codes, but when execute CUDA application, it still gets error.

At beginnig, I thought it's driver compatible problem, so I changed to older version, and still didn't work.
Now I'm try to upgrade guest's kernel and see it work or not.

But I think out there is another problem may cause it happen.
Because when we tried to launch QEMU, KVM is required. But my KVM is installed from Ubuntu ( sudo apt-get install kvm ) , and install KVM first, then QEMU 2.0 .
From Ubuntu official version, this modules may not compatible with newer QEMU or our settings.
Is that possible?

Offline

#1795 2014-05-24 03:10:15

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hmm, it's not working.  I can reproduce it now, so I'll keep poking at it.

Nope, it didn't work for me either. I added those scripts to the vfio-bind service though, so it seems to work perfectly fine. My Windows VM allows me to to run the passthrough and view the results on another monitor.

However, I'd like to know if it's possible to connect to the VM using virt-manager. I have my configuration file as such:

http://pastebin.com/L2Exj8bR

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Last edited by Slabity (2014-05-24 03:31:22)

Offline

#1796 2014-05-24 05:14:52

Destroy
Member
Registered: 2014-05-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Could you point us where did you looked that?

Offline

#1797 2014-05-24 05:26:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

However, I'd like to know if it's possible to connect to the VM using virt-manager. I have my configuration file as such:

http://pastebin.com/L2Exj8bR

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

No, it's not possible with current code.  When you use GPU assignment, the guest own the display, the host has no visibility to it.  Things like Spice are out-of-band, host driven protocols.  Once you assign a GPU you're left with in-band, guest driven protocols.  You have the same ability to get to the display remotely as you do a separate physical system, which does not currently involved libvirt/virt-manager.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1798 2014-05-24 05:36:44

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Destroy wrote:
Slabity wrote:

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Could you point us where did you looked that?

Hmm, well it seems like I may have been sorta wrong. Here's the original video I was looking at:

https://www.youtube.com/watch?v=Qi1LdFkRzIs&html5=1

It seems he used a combination of a spice display and a VNC server to run this.

This one:

https://www.youtube.com/watch?v=37D2bRsthfI&html5=1

Used some xrandr magic and synergy to synchronize his desktops.

There were 2 more videos that I can't seem to find right now. But I don't know what sort of setup would be useful for this.

So I guess that sort of rephrases my question (since I don't think Linux can grab the video output of a passthrough card). What would you do if you were trying to accomplish this sort of thing? I'm gonna try and see if VNC is fast enough for 60 FPS output, but I suppose I'll see.

Offline

#1799 2014-05-24 05:42:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
Destroy wrote:
Slabity wrote:

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Could you point us where did you looked that?

Hmm, well it seems like I may have been sorta wrong. Here's the original video I was looking at:

https://www.youtube.com/watch?v=Qi1LdFkRzIs&html5=1

It seems he used a combination of a spice display and a VNC server to run this.

No spice, clearly running tigervnc in the host as the view and a vnc server in the guest.  You can see the icon in the toolbar.

This one:

https://www.youtube.com/watch?v=37D2bRsthfI&html5=1

Used some xrandr magic and synergy to synchronize his desktops.

Yep, no remote display at all, xrandr releases the display, which is then assigned to a guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1800 2014-05-24 09:19:59

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i know that it is kinda wrong to bring my problems to arch linux boards, but i am pretty desperate at that point )

MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20

host vga: IGD (details below)

kernel (with i915 module being recompiled with i915_314.patch from linux-mainline.tar.gz in first post):

[s@localhost ~]$ uname -r 3.14.4-200.fc20.x86_64 [s@localhost ~]$ modinfo i915 | head -n1 filename: /lib/modules/3.14.4-200.fc20.x86_64/updates/i915.ko [s@localhost ~]$ 

couldnt have any success at all for 5 days while trying different variations, completely out of ideas for now on what may be wrong - too much outdated info out there

target guest os is win7

when trying to passthrough HD 6450:
- [test command from first post] host screen usually gets screwed (looking like pallete was overwritten randomly, i doubt X is running in indexed mode though) - fixable by switching to tty console and then back to X
- [virt-manager/kvm] Code 10 (device cannot be started) in device manager, latest catalyst drivers setup terminates while trying to detect hardware

when trying to passthrough GT 430:
- [test command from first post] seen this:

KVM internal error. Suberror: 1 emulation failure EAX=0000b422 EBX=0000b4b4 ECX=00005dd7 EDX=000003d4 ESI=00009e48 EDI=000018d2 EBP=0000d494 ESP=000018ca EIP=0000b4b4 EFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =b4b4 000b4b40 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =0000 00000000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6c48 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=20 07 20 07 20 07 20 07 20 07 20 07 20 07 20 07 20 07 20 07 <20> 07 20 07 20 07 20 07 20 07 20 07 55 aa 6d eb 4b 37 34 30 30 e9 4c 19 77 cc 56 49 44 45

- [virt-manager/kvm] Code 43 (windows stopped the device) in device manager

if someone could point me to the direction to look at, that would be great
thanks in advance
some environment info:

[s@localhost ~]$ lspci -nn | grep VGA 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] [1002:6779] 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108 [GeForce GT 430] [10de:0de1] (rev a1) [s@localhost ~]$ [s@localhost ~]$ sudo lspci -s0:2.0 -k 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) Subsystem: ASRock Incorporation Device 0412 Kernel driver in use: i915 Kernel modules: i915 [s@localhost ~]$ sudo lspci -s1:0.0 -k 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] Subsystem: Gigabyte Technology Co., Ltd Device 2549 Kernel driver in use: vfio-pci Kernel modules: radeon [s@localhost ~]$ sudo lspci -s5:0.0 -k 05:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 430] (rev a1) Kernel driver in use: vfio-pci Kernel modules: nouveau [s@localhost ~]$ [s@localhost ~]$ readlink /sys/bus/pci/devices/0000\:01\:00.0/iommu_group ../../../../kernel/iommu_groups/1 [s@localhost ~]$ readlink /sys/bus/pci/devices/0000\:05\:00.0/iommu_group ../../../../kernel/iommu_groups/9 [s@localhost ~]$ ls /dev/vfio/ total 0 drwxr-xr-x 2 root root 100 May 24 11:22 . drwxr-xr-x 22 root root 3780 May 24 12:31 .. crw------- 1 root root 247, 0 May 24 11:22 1 crw------- 1 root root 247, 1 May 24 11:22 9 crw-rw-rw- 1 root root 10, 196 May 24 11:22 vfio [s@localhost ~]$ [s@localhost ~]$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.14.4-200.fc20.x86_64 root=UUID=588911d3-23f7-4544-9d46-0b06b85f287c ro nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off pci-stub.ids=1002:6779,1002:aa98,10de:0de1,10de:0bea intel_iommu=on [s@localhost ~]$ [s@localhost ~]$ tail -n100 /etc/modprobe.d/alsa-base.conf /etc/modprobe.d/blacklist-ath_pci.conf /etc/modprobe.d/blacklist.conf /etc/modprobe.d/blacklist-firewire.conf /etc/modprobe.d/blacklist-framebuffer.conf /etc/modprobe.d/blacklist-gpu.conf /etc/modprobe.d/blacklist-modem.conf /etc/modprobe.d/blacklist-oss.conf /etc/modprobe.d/blacklist-rare-network.conf /etc/modprobe.d/blacklist-watchdog.conf /etc/modprobe.d/dkms.conf /etc/modprobe.d/fbdev-blacklist.conf /etc/modprobe.d/iwlwifi.conf /etc/modprobe.d/mlx4.conf /etc/modprobe.d/modprobe.conf /etc/modprobe.d/pci-stub.conf /etc/modprobe.d/qemu-system-x86.conf /etc/modprobe.d/vfio_iommu_type1.conf /etc/modprobe.d/vmwgfx-fbdev.conf ==> /etc/modprobe.d/kvm.conf <== options kvm ignore_msrs=1 ==> /etc/modprobe.d/pci-stub.conf <== blacklist nouveau blacklist radeon ==> /etc/modprobe.d/vfio_iommu_type1.conf <== options vfio_iommu_type1 allow_unsafe_interrupts=1 [s@localhost ~]$ [s@localhost ~]$ sudo grep -v "^#" /etc/libvirt/qemu.conf | grep -v "^[ \t]*$" user = "root" group = "root" clear_emulator_capabilities = 0 [s@localhost ~]$ [s@localhost ~]$ rpm -qa | grep "qemu\|seabios\|virt" | sort ipxe-roms-qemu-20140303-1.gitff1e7fc7.fc20.noarch libgovirt-0.3.0-3.fc20.x86_64 libvirt-client-1.2.4-1.fc20.x86_64 libvirt-daemon-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-interface-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-network-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-nodedev-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-nwfilter-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-qemu-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-secret-1.2.4-1.fc20.x86_64 libvirt-daemon-driver-storage-1.2.4-1.fc20.x86_64 libvirt-daemon-kvm-1.2.4-1.fc20.x86_64 libvirt-glib-0.1.7-2.fc20.x86_64 libvirt-python-1.2.4-1.fc20.x86_64 qemu-common-2.0.0-4.fc20.x86_64 qemu-img-2.0.0-4.fc20.x86_64 qemu-kvm-2.0.0-4.fc20.x86_64 qemu-system-x86-2.0.0-4.fc20.x86_64 seabios-bin-1.7.4-5.fc20.noarch virt-install-1.0.1-3.fc20.noarch virt-manager-1.0.1-3.fc20.noarch virt-manager-common-1.0.1-3.fc20.noarch virt-top-1.0.8-7.fc20.x86_64 virt-viewer-0.6.0-1.fc20.x86_64 [s@localhost ~]$ 

Offline

#1801 2014-05-24 12:49:15

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Slabity wrote:

Wow, that works perfectly with the test script! Thank you so much!

Does radeon still work ok for the host?

Anyone using radeon on the host (APU users) may want to try this too.

Many thanks Alex. This also works for me with 2 discrete Radeon cards (no APU). With the shell scripts I'm able to switch my boot VGA in BIOS to any card and VGA passthrough is always working. The configuration is the following:

 +-02.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] +-0b.0-[04]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device aac8

I will give the patch mentioned here http://pastebin.com/2ukyQbkG a try in the next hours. I'll hope it will solve the issue permantly for me. smile Many thanks again!

Offline

#1802 2014-05-24 15:26:12

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey,

I switched my hardware to accomplish this, kernel compiled etc. everything went good.
Problem now is I can't get Qemu to start..

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I can't anything on the net by searching for "vfio: error no iommu_group for device".. I have no idea what I should do..
BTW I get the same error with both GPUs!

System:
- CPU: Intel Xeon E3-1245v3 (Host uses iGPU)
- MB: MSI Z97 Gaming 7 (Supports VT-d)
- GPU1: Nvidia GeForce GTX 770 (for Gaming)
- GPU2: AMD Radeon HD5450 (for virtual HTPC)
- OS: Arch Linux (Kernel 3.14.4-3-mainline with all the patches in the package from the first post, qemu-git (2.1)  and seabios-git (1.7.5))

I know that the GTX 770 works, since I saw some guys that got it to work.
I'm not sure but I don't think that it's because of two GPUs..

/sys/bus/pci/devices/0000\:01\:00.0/ does NOT contain any iommu_group devices. FYI

Last edited by shawly (2014-05-24 15:29:37)

Offline

#1803 2014-05-24 15:35:26

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

Hey,

I switched my hardware to accomplish this, kernel compiled etc. everything went good.
Problem now is I can't get Qemu to start..

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I can't anything on the net by searching for "vfio: error no iommu_group for device".. I have no idea what I should do..
BTW I get the same error with both GPUs!

System:
- CPU: Intel Xeon E3-1245v3 (Host uses iGPU)
- MB: MSI Z97 Gaming 7 (Supports VT-d)
- GPU1: Nvidia GeForce GTX 770 (for Gaming)
- GPU2: AMD Radeon HD5450 (for virtual HTPC)
- OS: Arch Linux (Kernel 3.14.4-3-mainline with all the patches in the package from the first post, qemu-git (2.1)  and seabios-git (1.7.5))

I know that the GTX 770 works, since I saw some guys that got it to work.
I'm not sure but I don't think that it's because of two GPUs..

/sys/bus/pci/devices/0000\:01\:00.0/ does NOT contain any iommu_group devices. FYI

check kernel parameters or bios (seems to me like iommu might be disabled in your case) - see first post for details

Offline

#1804 2014-05-24 15:43:14

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

check kernel parameters or bios (seems to me like iommu might be disabled in your case) - see first post for details

Oh my god, I just needed to set intel_iommu=on to the kernel parameters.. I feel really stupid now..
Thanks <3

Offline

#1805 2014-05-24 16:14:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:
Slabity wrote:

Wow, that works perfectly with the test script! Thank you so much!

Does radeon still work ok for the host?

Anyone using radeon on the host (APU users) may want to try this too.

Many thanks Alex. This also works for me with 2 discrete Radeon cards (no APU). With the shell scripts I'm able to switch my boot VGA in BIOS to any card and VGA passthrough is always working. The configuration is the following:

 +-02.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] +-0b.0-[04]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device aac8

I will give the patch mentioned here http://pastebin.com/2ukyQbkG a try in the next hours. I'll hope it will solve the issue permantly for me. smile Many thanks again!

That patch isn't effective, don't bother with it.  This one seems to work for me, but it's more heavyweight than I'd like.  The problem sequence is that Xorg tries to lock VGA resources which triggers some first access callbacks.  The callback in the radeon code tells the arbiter that the device no longer decodes VGA resources.  Now, the arbiter still let's a process lock resources that the device doesn't decode, but it loses track of them when they're released.  So vfio comes along and tries to lock VGA resources on the other card and the arbiter skips disabling VGA on the radeon bridge because the device doesn't decode VGA.  The patch below gratuitously disables VGA on any potentially conflicting bridge on every vga_get, which makes VGA access even slower (not that we care so much since VGA stops being used shortly after the guest OS starts running).  I'll probably rework the code more for upstream to allow devices to own resources they don't decode rather than just lock them.  Enjoy.

--- a/drivers/gpu/vga/vgaarb.c +++ b/drivers/gpu/vga/vgaarb.c @@ -244,8 +244,12 @@ static struct vga_device *__vga_tryget(struct vga_device *v */ WARN_ON(conflict->owns & ~conflict->decodes); match = lwants & conflict->owns; - if (!match) + if (!match) { + if (change_bridge) + pci_set_vga_state(conflict->pdev, false, 0, + PCI_VGA_STATE_CHANGE_BRIDGE); continue; + } /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup looks like he doesn't have a lock, we can steal 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_7.html topic_8.html topic_9.html them from him

dropbox link since the forum will break formatting - https://dl.dropboxusercontent.com/u/198 … able.patch


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1806 2014-05-24 16:27:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

i know that it is kinda wrong to bring my problems to arch linux boards, but i am pretty desperate at that point )

I'm not running arch either, don't feel bad

MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20

host vga: IGD (details below)

kernel (with i915 module being recompiled with i915_314.patch from linux-mainline.tar.gz in first post):

[s@localhost ~]$ uname -r 3.14.4-200.fc20.x86_64 [s@localhost ~]$ modinfo i915 | head -n1 filename: /lib/modules/3.14.4-200.fc20.x86_64/updates/i915.ko [s@localhost ~]$ 

This doesn't prove anything, in fact it just looks like you're running the f20 distro kernel and not one recompiled to include the i915 patch.  Are you trying to rebuild the kernel rpm?

couldnt have any success at all for 5 days while trying different variations, completely out of ideas for now on what may be wrong - too much outdated info out there

target guest os is win7

when trying to passthrough HD 6450:
- [test command from first post] host screen usually gets screwed (looking like pallete was overwritten randomly, i doubt X is running in indexed mode though) - fixable by switching to tty console and then back to X
- [virt-manager/kvm] Code 10 (device cannot be started) in device manager, latest catalyst drivers setup terminates while trying to detect hardware

This also sounds like the host kernel doesn't actually include the i915 patch.  I'd start out with making sure you're really running a kernel patched the way you think it is.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1807 2014-05-24 18:18:18

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This doesn't prove anything, in fact it just looks like you're running the f20 distro kernel and not one recompiled to include the i915 patch.  Are you trying to rebuild the kernel rpm?

well, all i wanted to "prove" - is that module currently loaded is the one rebuild by me and manually placed in module updates directory
correct me if i'm wrong, but vga arbiter patch only involves i915 module, not the kernel itself.

aw wrote:

This also sounds like the host kernel doesn't actually include the i915 patch.  I'd start out with making sure you're really running a kernel patched the way you think it is.

well, i taken the i915_314.patch from linux-mainline.tar.gz in first post - it differs from vga arbiter patch v3 that may be found in internet. but i tried the v3 one as well with no success

Offline

#1808 2014-05-24 18:18:55

blacky
Member
Registered: 2012-10-26
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alphahere wrote:

I had a similar problem. Change this line below
        -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
TO THIS:
        -device vfio-pci,host=01:00.1,bus=pcie.0 \

Actually this solved one of my more bizarre issues where my new R290X would stop working whenever I physically removed my old, not passed through 4850. I always got a BSoD  from atikmdag.sys “PAGE_FAULT_IN_NONPAGED_AREA”. Changing this solved this issue.

Now the only issue which remains is that my card is not being reset on VM reboots which causes the same BSoD. Sending the host to ACPI S3 (suspend to RAM, results in a D3Cold for the card thus reseting it — if I got this right) does the trick but of course I have to go to standby and back every time I want to (re-)start the VM.

Powering off the slot is unfortunately not an option:

root@myhost ~ # ls /sys/bus/pci/slots total 0

Let’s see what my card supports:

root@myhost ~ # lspci -vvv -s 01:00 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Device e285 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 11 Region 0: Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 2: Memory at f0000000 (64-bit, prefetchable) [disabled] [size=8M] Region 4: I/O ports at e000 [disabled] [size=256] Region 5: Memory at f0800000 (32-bit, non-prefetchable) [disabled] [size=256K] Expansion ROM at f0840000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 Subsystem: PC Partner Limited / Sapphire Technology Device aac8 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin B routed to IRQ 10 Region 0: Memory at f0860000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

As you can see, Function Level Reset is not supported (FLReset-). I didn’t check this on my old card (which worked perfectly fine including reboots of the virtual machine).

D3Hot should be supported though but I haven’t found a way yet to convince the kernel to put it in this state but I guess it wouldn’t help anyway since NoSoftRst+.

Setting power control to auto, unbinding and rebinding didn’t help either. Even if I removed the device and rescanned … Maybe I wasn’t patient enough

root@myhost ~ # echo auto > /sys/bus/pci/devices/0000:00:01.0/power/control root@myhost ~ # echo '0000:01:00.0' > /sys/bus/pci/devices/0000:01:00.0/driver/unbind root@myhost ~ # echo '0000:01:00.1' > /sys/bus/pci/devices/0000:01:00.1/driver/unbind root@myhost ~ # echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove root@myhost ~ # echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove root@myhost ~ # echo 1 > /sys/bus/pci/rescan root@myhost ~ # echo '0000:01:00.0' > /sys/bus/pci/devices/0000:01:00.0/driver/unbind root@myhost ~ # echo '0000:01:00.1' > /sys/bus/pci/devices/0000:01:00.1/driver/unbind root@myhost ~ # echo '0000:01:00.1' > /sys/bus/pci/drivers/vfio-pci/bind root@myhost ~ # echo '0000:01:00.0' > /sys/bus/pci/drivers/vfio-pci/bind

See the documentation on sysfs-bus-pci for explanation.

I also explored other options like setpci and AMD ZeroCore but did not find anything useful I could leverage so far. 

For today I’m giving up; at least I was able to solve one of my issues and, quite frankly, it was the more important one. The fans of the old card were either dead or close to it resulting in nice “atmospheric sound” and the card itself did not really improve the thermal landscape. Keeping you posted.

For further reference:
Motherboard: ASRock Z87 Extreme6
Processor: Intel Core i7-4771
Host graphics card: Intel HD Graphics 4600
Passthrough device: Sapphire Radeon R9 290X Tri-X OC
OS: Arch Linux x86_64
Kernel: linux-mainline (3.14.1 includes acs override patch, i935 vga arbiter fixes, debug registers patch — custom config)
qemu: default (2.0.0-3)
seabios: default (1.7.3.1-2)
Status: Working
Constraints: Bus reset and CCC not working, loading VBIOS.

Last edited by blacky (2014-05-24 18:19:42)

Offline

#1809 2014-05-24 18:43:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
aw wrote:

This doesn't prove anything, in fact it just looks like you're running the f20 distro kernel and not one recompiled to include the i915 patch.  Are you trying to rebuild the kernel rpm?

well, all i wanted to "prove" - is that module currently loaded is the one rebuild by me and manually placed in module updates directory
correct me if i'm wrong, but vga arbiter patch only involves i915 module, not the kernel itself.

But it doesn't do that either.  I think all it shows is that modinfo thinks that's the module that would load, but that's after you've already booted, there may be a different i915.ko in your initramfs.

aw wrote:

This also sounds like the host kernel doesn't actually include the i915 patch.  I'd start out with making sure you're really running a kernel patched the way you think it is.

well, i taken the i915_314.patch from linux-mainline.tar.gz in first post - it differs from vga arbiter patch v3 that may be found in internet. but i tried the v3 one as well with no success

I've tried to get something upstream again recently, so this is the latest patch - https://lkml.org/lkml/2014/5/9/517

The maintainer won't have it because he thinks that adding a module option is admitting defeat and will prevent anyone from solving the problem (while at the same time not coming up with any workable solution himself).  The nice thing about this patch in your situation is that we can tell if you're running the right module by whether /sys/modules/i915/parameters/enable_hd_vgaarb is present (the one in the currently running module, not the one on disk somewhere).  You will need to set this with a module option in your /etc/modprobe.d/ (and rebuild the initramfs) to make it work.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1810 2014-05-24 19:45:31

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
sinny wrote:
aw wrote:

This doesn't prove anything, in fact it just looks like you're running the f20 distro kernel and not one recompiled to include the i915 patch.  Are you trying to rebuild the kernel rpm?

well, all i wanted to "prove" - is that module currently loaded is the one rebuild by me and manually placed in module updates directory
correct me if i'm wrong, but vga arbiter patch only involves i915 module, not the kernel itself.

But it doesn't do that either.  I think all it shows is that modinfo thinks that's the module that would load, but that's after you've already booted, there may be a different i915.ko in your initramfs.

aw wrote:

This also sounds like the host kernel doesn't actually include the i915 patch.  I'd start out with making sure you're really running a kernel patched the way you think it is.

well, i taken the i915_314.patch from linux-mainline.tar.gz in first post - it differs from vga arbiter patch v3 that may be found in internet. but i tried the v3 one as well with no success

I've tried to get something upstream again recently, so this is the latest patch - https://lkml.org/lkml/2014/5/9/517

The maintainer won't have it because he thinks that adding a module option is admitting defeat and will prevent anyone from solving the problem (while at the same time not coming up with any workable solution himself).  The nice thing about this patch in your situation is that we can tell if you're running the right module by whether /sys/modules/i915/parameters/enable_hd_vgaarb is present (the one in the currently running module, not the one on disk somewhere).  You will need to set this with a module option in your /etc/modprobe.d/ (and rebuild the initramfs) to make it work.

ok, you were right on the initramfs point - i completely forgot about it

tried applying your latest patch with no luck to my stock kernel sources - seems like entire drm code branch received quite a bit of work.
after that tried rebuilding i915 module with i915_314.patch from linux-mainline.tar.gz in first post, but now rebuilding initramfs as well - got corrupted graphics on boot as a result. could not even see tty consoles. the good point is that now i can be sure that rebuilt module was loaded )

will try to figure out next course of actions to take

for now thank you very much for pointing out my lack of understanding of what am i doing at all

Last edited by sinny (2014-05-24 19:46:58)

Offline

#1811 2014-05-24 20:42:14

Chrissi
Member
Registered: 2014-05-20
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Google for "ACS override", apply the patch you find and enable it via kernel options.  If one of the cards can be attached to a PCH root port (00:1c.*), the v3.15 kernel may help.

Chrissi wrote:

I just installed the new 3.15-rc6 kernel which in my understanding contains the acs override patch.

aw wrote:

Nope, 3.15 includes quirks approved by Intel to advertise the ACS-like isolation capabilities of PCH-based root ports and make sure they're enabled.  This means that the hardware is actually providing the isolation required.  The ACS override patch is a way for the user to override the hardware isolation requirement and is not going upstream.

With 3.14 and the ACS patch I got 3 vfio groups, one for every card and I am able to pass them to any vm in any combination I want while running them parallel.(I didnt know I had to enable acs in the grub cmdline)
Just for others, my setup is an i7 4770 with an asrock z87 extreme 6 and 3x 290x running arch with 3.14-1 mainline provided by OP. Looks like I got 4 PCH root ports. Everything is now running fine and smooth.
Thank you very much !

Offline

#1812 2014-05-25 02:53:37

shelladept
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

Thanks again for the help so far.  I patched in 'i915_314.patch' and recompiled my kernel.  I used 3.15-rc6 since the page 1 linux-mainline link appears to direct to 3.15-rc6 anyway (though other posts seem to refute what I'm seeing).  This cleared up the graphical issues on my host.  Now everything appears to be mostly working.  I haven't encountered any issues with my Debian test guest (so far). 

There are Windows issues though.  I am able to successfully install Window 7 SP1 but hit a snag as soon as I attempt to update said guest.  I've tried a few different update strategies and noted different behaviors.  When I tried updating all 150 or so updates at once everything appears fine until Windows attempts to reboot.  Post-reboot I am greeted with the Windows repair utilities which are unable to fix the now-broken install.  This problem seems to stem from cumulative security updates and other major system updates.  Some updates are okay alone (for example, Windows Explorer 11). 

When I try applying just the Radeon driver for my 4770 (via Windows Update), my host system freezes.  After restarting my host I am able to reboot the Windows guest *once* with the device software having installed "successfully".  Subsequent Windows guest restarts lead to BSODs, however.

Does this sound like a previously discussed issue that I've missed? Maybe an issue related to another .patch from the page 1 download?  Or might I have introduced odd behavior by applying the i915_3.14 patch to a kernel higher than 3.14.1?

I found an issue that was possibly related on page 62/66:

Nex7 wrote:

"EDIT: Of course, starting over with a fresh Win7 x64 install using ide instead of virtio, it works fine up until I try to install Windows updates, even only a few of them, at which point after reboot it dies with a different BSOD, code c000021a. -sigh-"

Except I do not get a blue screen in all my tests.  Fruitless restarts with the repair utilities are more common.  I am using ide-hd for my disk though.

I'm not sure what logs might provide useful information related to this issue.  I noted some logs in /var/log/libvirt/qemu which seem to be restricted to VMs created with virt-manager.  /var/log/libvirt/libvirtd.log doesn't appear to have related information either.   Please let me know if anybody has any suggestions.  I can't thank the contributors to this thread enough!

shelladept

Offline

#1813 2014-05-25 03:18:52

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
aw wrote:
sinny wrote:

well, all i wanted to "prove" - is that module currently loaded is the one rebuild by me and manually placed in module updates directory
correct me if i'm wrong, but vga arbiter patch only involves i915 module, not the kernel itself.

But it doesn't do that either.  I think all it shows is that modinfo thinks that's the module that would load, but that's after you've already booted, there may be a different i915.ko in your initramfs.

well, i taken the i915_314.patch from linux-mainline.tar.gz in first post - it differs from vga arbiter patch v3 that may be found in internet. but i tried the v3 one as well with no success

I've tried to get something upstream again recently, so this is the latest patch - https://lkml.org/lkml/2014/5/9/517

The maintainer won't have it because he thinks that adding a module option is admitting defeat and will prevent anyone from solving the problem (while at the same time not coming up with any workable solution himself).  The nice thing about this patch in your situation is that we can tell if you're running the right module by whether /sys/modules/i915/parameters/enable_hd_vgaarb is present (the one in the currently running module, not the one on disk somewhere).  You will need to set this with a module option in your /etc/modprobe.d/ (and rebuild the initramfs) to make it work.

ok, you were right on the initramfs point - i completely forgot about it

tried applying your latest patch with no luck to my stock kernel sources - seems like entire drm code branch received quite a bit of work.
after that tried rebuilding i915 module with i915_314.patch from linux-mainline.tar.gz in first post, but now rebuilding initramfs as well - got corrupted graphics on boot as a result. could not even see tty consoles. the good point is that now i can be sure that rebuilt module was loaded )

will try to figure out next course of actions to take

for now thank you very much for pointing out my lack of understanding of what am i doing at all

switched to fedora mainline kernel (3.15-rc5), applied your latest vga arbiter patch to i915 module, rebuilt i915 module - resulted in the same behavior i got in my previous attempt (3.14 with i915_314.patch). screen gets corrupted when X login manager starts ( i use slim, but dont think it actually matters). switching to ttys resets screen to black and stays this way whatever i try to do.

what can i do to further investigate this? maybe some X settings are needed as well?

Last edited by sinny (2014-05-25 03:19:52)

Offline

#1814 2014-05-25 03:24:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you set the enable_hd_vgaarb i915 option?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1815 2014-05-25 04:28:23

thehighhat
Member
Registered: 2014-05-12
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thehighhat wrote:
AKSN74 wrote:
aw wrote:

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.


Same problem here.  I cannot reboot or reissue the qemu command without this error.  Do I need to undo the vfio-bind stuffs?


Fixed.  I needed to download the rom file from http://www.techpowerup.com/vgabios/ .   Then add romfile=/path/to/romfile.rom to the -device line for the GPU.

Also changed the -cpu from type "host" to "qemu64"

Offline

#1816 2014-05-25 04:31:04

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Did you set the enable_hd_vgaarb i915 option?

previously - no
just tried setting in with no change in behavior - corrupted display on X start. on the other hand, just to see whether its just gpu die or the whole system - successfully ssh'ed to system.
journal is full of lines like that:

... May 25 08:19:35 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 2 May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 43 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 3 May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 44 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 3 May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 44 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 2 May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 44 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DMAR:[DMA Read] Request device [00:02.0] fault addr 7fcd0000 DMAR:[fault reason 06] PTE Read access is not set May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 59 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DMAR:[DMA Read] Request device [00:02.0] fault addr 7fce4000 DMAR:[fault reason 06] PTE Read access is not set May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 44 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DMAR:[DMA Read] Request device [00:02.0] fault addr 7fcf4000 DMAR:[fault reason 06] PTE Read access is not set May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 45 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DMAR:[DMA Read] Request device [00:02.0] fault addr 7fd03000 DMAR:[fault reason 06] PTE Read access is not set May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 43 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 2 May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 45 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DRHD: handling fault status reg 3 May 25 08:19:35 localhost.localdomain systemd-journal[477]: Missed 46 kernel messages May 25 08:19:35 localhost.localdomain kernel: dmar: DMAR:[DMA Read] Request device [00:02.0] fault addr 7fd32000 DMAR:[fault reason 06] PTE Read access is not set ...

seems like on screen corruption it just keeps spamming these. i doubt they are meaningful though, useful errors were probably at the very beginning

Offline

#1817 2014-05-25 13:12:08

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So vfio comes along and tries to lock VGA resources on the other card and the arbiter skips disabling VGA on the radeon bridge because the device doesn't decode VGA.  The patch below gratuitously disables VGA on any potentially conflicting bridge on every vga_get, which makes VGA access even slower (not that we care so much since VGA stops being used shortly after the guest OS starts running).  I'll probably rework the code more for upstream to allow devices to own resources they don't decode rather than just lock them.  Enjoy.

This seems like something a headless host (like mine) would benefit from, do you plan to push this/similar behaviour to official kernel with kernel option to switch it on? Ideally I would like to be able to use two GPUs each assigned to one VMs (I have enough resources to run 2 VMs concurrently, just not enough slots for more GPUs) while the host runs headless.

Thanks for your work on passthrough and please let us known when this gets into official RH release (I will probably use Fedora until then) smile


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#1818 2014-05-25 17:09:20

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

As you can see, Function Level Reset is not supported (FLReset-). I didn’t check this on my old card (which worked perfectly fine including reboots of the virtual machine).

D3Hot should be supported though but I haven’t found a way yet to convince the kernel to put it in this state but I guess it wouldn’t help anyway since NoSoftRst+.

It looks like no AMD Radeon R9 290X supports any kind of reset anymore. sad You are not alone as I have the same problem.

Offline

#1819 2014-05-25 18:12:29

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

That patch isn't effective, don't bother with it.  This one seems to work for me, but it's more heavyweight than I'd like.  The problem sequence is that Xorg tries to lock VGA resources which triggers some first access callbacks.  The callback in the radeon code tells the arbiter that the device no longer decodes VGA resources.  Now, the arbiter still let's a process lock resources that the device doesn't decode, but it loses track of them when they're released.  So vfio comes along and tries to lock VGA resources on the other card and the arbiter skips disabling VGA on the radeon bridge because the device doesn't decode VGA.  The patch below gratuitously disables VGA on any potentially conflicting bridge on every vga_get, which makes VGA access even slower (not that we care so much since VGA stops being used shortly after the guest OS starts running).  I'll probably rework the code more for upstream to allow devices to own resources they don't decode rather than just lock them.  Enjoy.

--- a/drivers/gpu/vga/vgaarb.c +++ b/drivers/gpu/vga/vgaarb.c @@ -244,8 +244,12 @@ static struct vga_device *__vga_tryget(struct vga_device *v */ WARN_ON(conflict->owns & ~conflict->decodes); match = lwants & conflict->owns; - if (!match) + if (!match) { + if (change_bridge) + pci_set_vga_state(conflict->pdev, false, 0, + PCI_VGA_STATE_CHANGE_BRIDGE); continue; + } /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup looks like he doesn't have a lock, we can steal 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_7.html topic_8.html topic_9.html them from him

dropbox link since the forum will break formatting - https://dl.dropboxusercontent.com/u/198 … able.patch

Alex, this patch works fine. I will apply until a better solution is upstream. smile So only problem left is reset - well known for not reliable working on AMD GPUs.

Offline

#1820 2014-05-25 18:17:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:

That patch isn't effective, don't bother with it.  This one seems to work for me, but it's more heavyweight than I'd like.  The problem sequence is that Xorg tries to lock VGA resources which triggers some first access callbacks.  The callback in the radeon code tells the arbiter that the device no longer decodes VGA resources.  Now, the arbiter still let's a process lock resources that the device doesn't decode, but it loses track of them when they're released.  So vfio comes along and tries to lock VGA resources on the other card and the arbiter skips disabling VGA on the radeon bridge because the device doesn't decode VGA.  The patch below gratuitously disables VGA on any potentially conflicting bridge on every vga_get, which makes VGA access even slower (not that we care so much since VGA stops being used shortly after the guest OS starts running).  I'll probably rework the code more for upstream to allow devices to own resources they don't decode rather than just lock them.  Enjoy.

--- a/drivers/gpu/vga/vgaarb.c +++ b/drivers/gpu/vga/vgaarb.c @@ -244,8 +244,12 @@ static struct vga_device *__vga_tryget(struct vga_device *v */ WARN_ON(conflict->owns & ~conflict->decodes); match = lwants & conflict->owns; - if (!match) + if (!match) { + if (change_bridge) + pci_set_vga_state(conflict->pdev, false, 0, + PCI_VGA_STATE_CHANGE_BRIDGE); continue; + } /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup looks like he doesn't have a lock, we can steal 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_7.html topic_8.html topic_9.html them from him

dropbox link since the forum will break formatting - https://dl.dropboxusercontent.com/u/198 … able.patch

Alex, this patch works fine. I will apply until a better solution is upstream. smile So only problem left is reset - well known for not reliable working on AMD GPUs.

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index 76715cc..bdb6126 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -3350,7 +3350,7 @@ static void vfio_pci_reset_handler(void *opaque) QLIST_FOREACH(group, &group_list, next) { QLIST_FOREACH(vdev, &group->device_list, next) { - if (!vdev->reset_works || (!vdev->has_flr && vdev->has_pm_reset)) { + if (!vdev->reset_works || !vdev->has_flr) { vdev->needs_reset = true; } }

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1821 2014-05-25 18:40:13

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index 76715cc..bdb6126 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -3350,7 +3350,7 @@ static void vfio_pci_reset_handler(void *opaque) QLIST_FOREACH(group, &group_list, next) { QLIST_FOREACH(vdev, &group->device_list, next) { - if (!vdev->reset_works || (!vdev->has_flr && vdev->has_pm_reset)) { + if (!vdev->reset_works || !vdev->has_flr) { vdev->needs_reset = true; } }

Alex I tried it already as you posted it a couple of weeks ago on qemu-devel smile but this doesn't help on my R9 290X. Windows VM still got BSOD on reboot. Linux VM with fglrx got kernel oops once X is started. Only thing which works is Linux VM with radeon driver but this is still more or less useless test as HAWAII XT is unaccelerated in X at all for now due to stability concerns.

Last edited by mbroemme (2014-05-25 18:41:18)

Offline

#1822 2014-05-25 18:57:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:
diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index 76715cc..bdb6126 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -3350,7 +3350,7 @@ static void vfio_pci_reset_handler(void *opaque) QLIST_FOREACH(group, &group_list, next) { QLIST_FOREACH(vdev, &group->device_list, next) { - if (!vdev->reset_works || (!vdev->has_flr && vdev->has_pm_reset)) { + if (!vdev->reset_works || !vdev->has_flr) { vdev->needs_reset = true; } }

Alex I tried it already as you posted it a couple of weeks ago on qemu-devel smile but this doesn't help on my R9 290X. Windows VM still got BSOD on reboot. Linux VM with fglrx got kernel oops once X is started. Only thing which works is Linux VM with radeon driver but this is still more or less useless test as HAWAII XT is unaccelerated in X at all for now due to stability concerns.

I guess I keep posting it because I don't know why it doesn't work.  I added an HD8570 to my collection, since it seems to be based on the same generation of chips as the latest, but it works well.  Can you manually test whether the card is affected by a bus reset?  Try getting the device with something on the screen, ex. quit QEMU while the device is at seabios.  Find the bridge upstream from the device, ex

$ lspci -tv | grep AMD +-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R7 240 OEM] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

00:01.0 is the upstream bridge.

Read the bridge control register

$ sudo setpci -s 00:01.0 3e.w 0018

OR 0x40 into the result

$ printf %04x $(( 0x0018 | 0x40 )) 0058

Write this to the same register:

$ sudo sudo setpci -s 00:01.0 3e.w=0058

Then restore the original value:

$ sudo sudo setpci -s 00:01.0 3e.w=0018

If the bus reset works, then the screen should have cleared and the monitor gone out of sync.  If nothing happened, then I don't know how we can reset the device.  I think one of my old GeForce cards is also impervious to bus resets.

Last edited by aw (2014-05-25 18:58:50)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1823 2014-05-25 20:15:08

thelamer
Member
Registered: 2014-05-25
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Did you set the enable_hd_vgaarb i915 option?

How do you achieve this ? I have applied the i915 patch to the 3.15r6 branch here: (needed it for my marvell controller)

https://github.com/awilliam/linux-vfio release dma-alias-v4

And on my system I have:

for i in /sys/module/i915/parameters/*; do echo $i=$(cat $i); done /sys/module/i915/parameters/disable_display=N /sys/module/i915/parameters/disable_power_well=1 /sys/module/i915/parameters/enable_cmd_parser=0 /sys/module/i915/parameters/enable_fbc=-1 /sys/module/i915/parameters/enable_hangcheck=Y /sys/module/i915/parameters/enable_hd_vgaarb=N /sys/module/i915/parameters/enable_ips=1 /sys/module/i915/parameters/enable_ppgtt=1 /sys/module/i915/parameters/enable_psr=0 /sys/module/i915/parameters/enable_rc6=-1 /sys/module/i915/parameters/fastboot=N /sys/module/i915/parameters/invert_brightness=0 /sys/module/i915/parameters/lvds_channel_mode=0 /sys/module/i915/parameters/lvds_downclock=0 /sys/module/i915/parameters/lvds_use_ssc=-1 /sys/module/i915/parameters/modeset=-1 /sys/module/i915/parameters/panel_ignore_lid=1 /sys/module/i915/parameters/powersave=1 /sys/module/i915/parameters/prefault_disable=N /sys/module/i915/parameters/preliminary_hw_support=0 /sys/module/i915/parameters/reset=Y /sys/module/i915/parameters/semaphores=-1 /sys/module/i915/parameters/vbt_sdvo_panel_type=-1

Also I am reading about FLR which I am sure my R9 270x will not support:

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

Is this a dead end for me? Want to know If I should swap cards to one of my nvidias before trying anymore troubleshooting.

Right now when running :

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -vnc :1

I get an output on my monitor but it is just a black screen.

Thanks in advance!

Offline

#1824 2014-05-25 20:21:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thelamer wrote:
aw wrote:

Did you set the enable_hd_vgaarb i915 option?

How do you achieve this ? I have applied the i915 patch to the 3.15r6 branch here: (needed it for my marvell controller)

https://github.com/awilliam/linux-vfio release dma-alias-v4

And on my system I have:

for i in /sys/module/i915/parameters/*; do echo $i=$(cat $i); done ... /sys/module/i915/parameters/enable_hd_vgaarb=N ...

Boot with i915.enable_hd_vgaarb=1 or add "options i915 enable_hd_vgaarb=1" to modprobe.d and rebuild your initramfs

Also I am reading about FLR which I am sure my R9 270x will not support:

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

Is this a dead end for me? Want to know If I should swap cards to one of my nvidias before trying anymore troubleshooting.

No graphics cards support FLR.  The question is whether the newer AMD cards respond to bus reset or not.  See previous post to test.

Right now when running :

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu qemu64 -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -vnc :1

I get an output on my monitor but it is just a black screen.

You won't get any output until i915 VGA arbiter support is enabled.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1825 2014-05-25 20:36:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
--- a/drivers/gpu/vga/vgaarb.c +++ b/drivers/gpu/vga/vgaarb.c @@ -244,8 +244,12 @@ static struct vga_device *__vga_tryget(struct vga_device *v */ WARN_ON(conflict->owns & ~conflict->decodes); match = lwants & conflict->owns; - if (!match) + if (!match) { + if (change_bridge) + pci_set_vga_state(conflict->pdev, false, 0, + PCI_VGA_STATE_CHANGE_BRIDGE); continue; + } /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup looks like he doesn't have a lock, we can steal 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_7.html topic_8.html topic_9.html them from him

dropbox link since the forum will break formatting - https://dl.dropboxusercontent.com/u/198 … able.patch


Here's the version proposed upstream https://lkml.org/lkml/2014/5/25/94


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1826 2014-05-26 00:05:17

unixninja92
Member
Registered: 2014-05-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Offline

#1827 2014-05-26 00:06:02

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
$ lspci -tv | grep AMD +-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R7 240 OEM] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

00:01.0 is the upstream bridge.

Read the bridge control register

$ sudo setpci -s 00:01.0 3e.w 0018

OR 0x40 into the result

$ printf %04x $(( 0x0018 | 0x40 )) 0058

Write this to the same register:

$ sudo sudo setpci -s 00:01.0 3e.w=0058

Then restore the original value:

$ sudo sudo setpci -s 00:01.0 3e.w=0018

If the bus reset works, then the screen should have cleared and the monitor gone out of sync.  If nothing happened, then I don't know how we can reset the device.  I think one of my old GeForce cards is also impervious to bus resets.

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003) [21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270 [21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0 [21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

Offline

#1828 2014-05-26 00:07:44

thelamer
Member
Registered: 2014-05-25
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: NM I was able to build the latest qemu from source and it can use this config file, 2.0.0 from the debian repos must have a bug in it. I still get a atikmpag.sys bsod but at least I know I have tried everything. Looks like this R9 270x was not meant to be. Time to try some other cards.

Norcoen wrote:

Sorry, I was looking in /etc/qemu/, and it wasn't there, I forgot the connection to libvirtd.
I already have "clear_emulator_capabilities=0" in said qemu.conf, because before finding this board, I followed another tutorial and applied the last paragraph of this site:
http://fedoraproject.org/wiki/How_to_de … assignment

So, I have the same bluescreen error, but your solution somehow does not apply for me :/

//edit:
I found out, the qemu-git I built isn't looking in /etc/libvirt/ but in /etc/qemu, so I copied my qemu.conf to that location, now getting this error:

qemu-system-x86_64:/etc/qemu/qemu.conf:198: no group defined

Line 198 is

user = "root"

So I'm stuck again...

//edit2:
I played around a bit with the config, and the seems to be around the group for the QEMU processes:

# The group for QEMU processes run by the system instance. It can be # specified in a similar way to user. group = "root"

If I delete that line, I get a parser error, but if I set it to any group, root, my user, qemu, kvm, I just get the error above, that no group is defined.

I am currently running into the same thing, can you define this options from the command line ? I did not see anything in the man page.

Current config: (/etc/qemu/qemu.conf)

user = "root" group = "root" clear_emulator_capabilities = 0

Error :

qemu-system-x86_64:/etc/qemu/qemu.conf:1: no group defined

Version:

QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-4+b1), Copyright (c) 2003-2008 Fabrice Bellard

Last edited by thelamer (2014-05-26 02:41:00)

Offline

#1829 2014-05-26 00:08:14

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

unixninja92 wrote:

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

Last edited by mbroemme (2014-05-26 00:08:27)

Offline

#1830 2014-05-26 00:10:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003) [21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270 [21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0 [21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

You need to reboot after doing this, I should have mentioned that.  If a bus reset works, then I don't see why they change to invoke a bus reset regardless of pm-reset support doesn't work.  Can you uncomment the DEBUG_VFIO in hw/misc/vfio.c, rebuild and pastebin the log? /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup #define DEBUG_VFIO */ -> #define DEBUG_VFIO  Do that with the reset patch applied.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1831 2014-05-26 00:13:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
unixninja92 wrote:

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1832 2014-05-26 00:17:07

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mbroemme wrote:
unixninja92 wrote:

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

With safely I meant I've tried it already several times on another machine and it worked very smooth. smile

Offline

#1833 2014-05-26 02:32:57

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thehighhat wrote:
thehighhat wrote:
AKSN74 wrote:

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.


Same problem here.  I cannot reboot or reissue the qemu command without this error.  Do I need to undo the vfio-bind stuffs?


Fixed.  I needed to download the rom file from http://www.techpowerup.com/vgabios/ .   Then add romfile=/path/to/romfile.rom to the -device line for the GPU.

Also changed the -cpu from type "host" to "qemu64"

Thanks for your solution, looks like some graphic cards need to do this as well.

Offline

#1834 2014-05-26 04:09:12

unixninja92
Member
Registered: 2014-05-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:
mbroemme wrote:

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

With safely I meant I've tried it already several times on another machine and it worked very smooth. smile

My GPU is an Intel 4600. I haven't looked into detaching the GPU from vgacon yet, but I believe I have everything else set up correctly. But I keep getting this error:
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
when I try running the test qemu stuff from the first post. Dmesg says that Intel IOMMU is enabled, and vfio-pci is loaded. When I boot my pci-stub is set as the driver for my graphics card, and running the vfio-bind script seems to have no effect.

Offline

#1835 2014-05-26 04:12:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

unixninja92 wrote:
mbroemme wrote:
aw wrote:

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

With safely I meant I've tried it already several times on another machine and it worked very smooth. smile

My GPU is an Intel 4600. I haven't looked into detaching the GPU from vgacon yet, but I believe I have everything else set up correctly. But I keep getting this error:
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
when I try running the test qemu stuff from the first post. Dmesg says that Intel IOMMU is enabled, and vfio-pci is loaded. When I boot my pci-stub is set as the driver for my graphics card, and running the vfio-bind script seems to have no effect.

You don't have VT-d enabled, but it doesn't matter, assignment of an IGD device does not currently work.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1836 2014-05-26 05:19:55

unixninja92
Member
Registered: 2014-05-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
unixninja92 wrote:

My GPU is an Intel 4600. I haven't looked into detaching the GPU from vgacon yet, but I believe I have everything else set up correctly. But I keep getting this error:
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
when I try running the test qemu stuff from the first post. Dmesg says that Intel IOMMU is enabled, and vfio-pci is loaded. When I boot my pci-stub is set as the driver for my graphics card, and running the vfio-bind script seems to have no effect.

You don't have VT-d enabled, but it doesn't matter, assignment of an IGD device does not currently work.

Ok, good to know. IGD stuff aside, my VT-d is enabled.

Offline

#1837 2014-05-26 07:24:14

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just fyi, because I have seen the discussion come up a while back:
In my setup I am running an X server on my nvidia gpu, can stop it, boot the vm, stop the vm and start the X server again. I mainly use it to save power, because the X server can put the GPU in a low power state with no VM running that wastes CPU cycles.

The only "special" thing I had to do was to issue a bus reset (?) after shutting down the VM, which I do by:

qemu-system-x86_64 -name noop -enable-kvm -boot order=c,menu=off,reboot-timeout=0 -machine type=q35,accel=kvm -cpu host -smp 1 -m 32 -k de -vga none -display none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -net none -no-reboot

Everything else should be relatively standard.

Last edited by Flyser (2014-05-26 07:43:13)

Offline

#1838 2014-05-26 10:07:18

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, i managed to overcome my problems (more like stupidity, in fact) - the i915 module i rebuilt had different vermagic, so it failed loading at boot time, which lead to X server not having any devices to operate on. though i never would expect that X server would just freeze like that on no devices instead of just quitting.

now comes another portion of problems - successfully managed to install windows 7 64bit using cirrus vga. after that using only HD 6450 passed through downloaded and installed latest catalyst drivers (excluding CCC). now getting solid BSOD on windows login - will try with restarted host system when i get a chance for this restart.

do catalyst drivers only work on limited set of versions or is it just some random problem with my setup/configuration?

Offline

#1839 2014-05-26 15:00:47

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mbroemme wrote:

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003) [21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270 [21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0 [21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

You need to reboot after doing this, I should have mentioned that.  If a bus reset works, then I don't see why they change to invoke a bus reset regardless of pm-reset support doesn't work.  Can you uncomment the DEBUG_VFIO in hw/misc/vfio.c, rebuild and pastebin the log? /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup #define DEBUG_VFIO */ -> #define DEBUG_VFIO  Do that with the reset patch applied.

Okay I tested it now with the following sequence for my system:

#1 Started QEMU to boot Linux VM, load fglrx driver and start X.
#2 Graceful shutdown of VM
#3 Started QEMU and pressed STRG+C during seabios boot.
#4 Executed:

setpci -s 00:0b.0 3e.w printf %04x $(( 0x0008 | 0x40 )) setpci -s 00:0b.0 3e.w=0048 setpci -s 00:0b.0 3e.w=0008

After that screen goes blank
#5 Started QEMU again and now it hangs and logfile is here: http://paste2.org/wEay3vFV

Offline

#1840 2014-05-26 15:22:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:
mbroemme wrote:

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003) [21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270 [21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0 [21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

You need to reboot after doing this, I should have mentioned that.  If a bus reset works, then I don't see why they change to invoke a bus reset regardless of pm-reset support doesn't work.  Can you uncomment the DEBUG_VFIO in hw/misc/vfio.c, rebuild and pastebin the log? /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup #define DEBUG_VFIO */ -> #define DEBUG_VFIO  Do that with the reset patch applied.

Okay I tested it now with the following sequence for my system:

#1 Started QEMU to boot Linux VM, load fglrx driver and start X.
#2 Graceful shutdown of VM
#3 Started QEMU and pressed STRG+C during seabios boot.
#4 Executed:

setpci -s 00:0b.0 3e.w printf %04x $(( 0x0008 | 0x40 )) setpci -s 00:0b.0 3e.w=0048 setpci -s 00:0b.0 3e.w=0008

After that screen goes blank
#5 Started QEMU again and now it hangs and logfile is here: http://paste2.org/wEay3vFV

Nothing is expected to work after the setpci secondary bus reset, it's only a test.  Do not attempt to do anything with devices downstream of the bridge until after a reboot.  That said, I do see QEMU reporting success for doing a hot reset on the bus, so I don't see why we're seeing so many reports that reset doesn't work on these cards.

Last edited by aw (2014-05-26 15:22:46)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1841 2014-05-26 16:49:54

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Nothing is expected to work after the setpci secondary bus reset, it's only a test.  Do not attempt to do anything with devices downstream of the bridge until after a reboot.  That said, I do see QEMU reporting success for doing a hot reset on the bus, so I don't see why we're seeing so many reports that reset doesn't work on these cards.

Looks like only folks from AMD/ATI knows the magic for a proper reset.

Offline

#1842 2014-05-26 17:18:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:

Nothing is expected to work after the setpci secondary bus reset, it's only a test.  Do not attempt to do anything with devices downstream of the bridge until after a reboot.  That said, I do see QEMU reporting success for doing a hot reset on the bus, so I don't see why we're seeing so many reports that reset doesn't work on these cards.

Looks like only folks from AMD/ATI knows the magic for a proper reset.

I hope not.  You've shown that a secondary bus reset seems to reset the card and that QEMU is doing a secondary bus reset, we need to figure out why that's not sufficient.  Does it help to provide the ROM via romfile instead of reading it from the card?  Does playing with the parameters for the reset help?  Will multiple resets help?  Does PCIe link down produce a different result?  We need someone that can debug and experiment with the card in front of them.  It might help to document what guests and what drivers result in the problem.  Does Windows with the Catalyst driver cause it?  Does Linux with the radeon driver cause it?  fglrx?  What exactly goes wrong in each case?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1843 2014-05-26 18:07:13

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I hope not.  You've shown that a secondary bus reset seems to reset the card and that QEMU is doing a secondary bus reset, we need to figure out why that's not sufficient.  Does it help to provide the ROM via romfile instead of reading it from the card?  Does playing with the parameters for the reset help?  Will multiple resets help?  Does PCIe link down produce a different result?  We need someone that can debug and experiment with the card in front of them.  It might help to document what guests and what drivers result in the problem.  Does Windows with the Catalyst driver cause it?  Does Linux with the radeon driver cause it?  fglrx?  What exactly goes wrong in each case?

I've tried it already with romfile half an hour ago but it doesn't help. The result is the same. About which reset parameters you're talking? Basically I can debug it and I've started already to read instructions on how to use mmio trace as I believe fglrlx did some magic on unloading the module because once I unload the fglrx module before rebooting the VM, everything works fine. So right now I (mabye other as well with R9 series) have the following situation:

#1 Using Linux VM with radeon driver:

- This works fine on several reboots, but I guess the test case is not very meaningful because R9 series has only KMS working and everything on-top (Glamor, X, DRI) is unaccelerated.

#2 Using Linux VM with fglrx driver:

- Works on first boot only if you don't unload fglrx module before VM shutdown/reboot.
- Works on multiple reboots (I've tried around 20 times) if you stop X inside VM, unload fglrx and make VM shutdown/reboot.

#3 Using Windows VM with catalyst:

- Works on first boot only if you don't eject the card before VM shutdown/reboot.
- Works on multiple reboots (also tried several times) if you eject the card inside VM from device manager. This is only workaround for Win8/8.1 as Win7 and earlier require reboot once you remove PCI/PCIe device.

This is what I have for now, maybe others have different experience.

Last edited by mbroemme (2014-05-26 18:07:49)

Offline

#1844 2014-05-26 19:07:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:

I hope not.  You've shown that a secondary bus reset seems to reset the card and that QEMU is doing a secondary bus reset, we need to figure out why that's not sufficient.  Does it help to provide the ROM via romfile instead of reading it from the card?  Does playing with the parameters for the reset help?  Will multiple resets help?  Does PCIe link down produce a different result?  We need someone that can debug and experiment with the card in front of them.  It might help to document what guests and what drivers result in the problem.  Does Windows with the Catalyst driver cause it?  Does Linux with the radeon driver cause it?  fglrx?  What exactly goes wrong in each case?

I've tried it already with romfile half an hour ago but it doesn't help. The result is the same. About which reset parameters you're talking? Basically I can debug it and I've started already to read instructions on how to use mmio trace as I believe fglrlx did some magic on unloading the module because once I unload the fglrx module before rebooting the VM, everything works fine. So right now I (mabye other as well with R9 series) have the following situation:

#1 Using Linux VM with radeon driver:

- This works fine on several reboots, but I guess the test case is not very meaningful because R9 series has only KMS working and everything on-top (Glamor, X, DRI) is unaccelerated.

#2 Using Linux VM with fglrx driver:

- Works on first boot only if you don't unload fglrx module before VM shutdown/reboot.
- Works on multiple reboots (I've tried around 20 times) if you stop X inside VM, unload fglrx and make VM shutdown/reboot.

#3 Using Windows VM with catalyst:

- Works on first boot only if you don't eject the card before VM shutdown/reboot.
- Works on multiple reboots (also tried several times) if you eject the card inside VM from device manager. This is only workaround for Win8/8.1 as Win7 and earlier require reboot once you remove PCI/PCIe device.

This is what I have for now, maybe others have different experience.

So both the Linux and Windows drivers from AMD are doing something to the card on shutdown/reboot that they don't do if unbinding the device prior to reboot.  My first guess would be that they put it into D3 power state in the former case, but not the latter.  We do try to wake the device back into D0 before we do a bus reset, but maybe that's not working (it works for most devices).  You could try to hide the PM capability from the guest.  Some drivers will ignore whether we expose the capability and have it hard coded, but it's worth a shot.  IIRC, to hide the capability, find vfio_add_std_cap() in qemu hw/misc/vfio.c, after the "case PCI_CAP_ID_PM:" line add a new line that does "return 0;".  You can also compare the state of the device in the host with lspci -vvv after a shutdown with the driver attached and a shutdown with the device ejected/unbound.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1845 2014-05-26 19:56:45

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So both the Linux and Windows drivers from AMD are doing something to the card on shutdown/reboot that they don't do if unbinding the device prior to reboot.  My first guess would be that they put it into D3 power state in the former case, but not the latter.  We do try to wake the device back into D0 before we do a bus reset, but maybe that's not working (it works for most devices).  You could try to hide the PM capability from the guest.  Some drivers will ignore whether we expose the capability and have it hard coded, but it's worth a shot.  IIRC, to hide the capability, find vfio_add_std_cap() in qemu hw/misc/vfio.c, after the "case PCI_CAP_ID_PM:" line add a new line that does "return 0;".

Okay I tried this but it doesn't change the behavior. On reboot of Linux VM fglrx still oops and Windows VM get BSOD.

aw wrote:

You can also compare the state of the device in the host with lspci -vvv after a shutdown with the driver attached and a shutdown with the device ejected/unbound.

This is the difference between "#1 boot of Linux VM, start X, stop X, poweroff" and "#2 boot of Linux VM, start X, stop X, rmmod fglrx, poweroff". Between both tests I've rebooted the host.

--- lspci.1 2014-05-26 21:48:06.460661219 +0200 +++ lspci.2 2014-05-26 21:50:23.875239015 +0200 @@ -23,7 +23,7 @@ ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- - LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- + LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

lspci.1 is after shutdown of Linux VM in #1
lspci.2 is after shutdown of Linux VM in #2

Offline

#1846 2014-05-26 20:36:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:

So both the Linux and Windows drivers from AMD are doing something to the card on shutdown/reboot that they don't do if unbinding the device prior to reboot.  My first guess would be that they put it into D3 power state in the former case, but not the latter.  We do try to wake the device back into D0 before we do a bus reset, but maybe that's not working (it works for most devices).  You could try to hide the PM capability from the guest.  Some drivers will ignore whether we expose the capability and have it hard coded, but it's worth a shot.  IIRC, to hide the capability, find vfio_add_std_cap() in qemu hw/misc/vfio.c, after the "case PCI_CAP_ID_PM:" line add a new line that does "return 0;".

Okay I tried this but it doesn't change the behavior. On reboot of Linux VM fglrx still oops and Windows VM get BSOD.

aw wrote:

You can also compare the state of the device in the host with lspci -vvv after a shutdown with the driver attached and a shutdown with the device ejected/unbound.

This is the difference between "#1 boot of Linux VM, start X, stop X, poweroff" and "#2 boot of Linux VM, start X, stop X, rmmod fglrx, poweroff". Between both tests I've rebooted the host.

--- lspci.1 2014-05-26 21:48:06.460661219 +0200 +++ lspci.2 2014-05-26 21:50:23.875239015 +0200 @@ -23,7 +23,7 @@ ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- - LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- + LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

lspci.1 is after shutdown of Linux VM in #1
lspci.2 is after shutdown of Linux VM in #2

#1 is the case that fails and #2 is the case that works, right?  The link status looks a lot more correct in #2.  The next thing I would try would be testing whether we can poke at the PCIe capability registers on the device to get it to state #2.  So I would look at the LnkSta just after boot, repeat the steps for #1, do the setpci bus reset, then look at the LnkSta for the device again.  Is it any closer to the state just after boot or state #2 (several things should go to reset values after bus reset, concentrate on PCIe status registers)?  If not, we can try to retrain the link.  Here's another setpci sequence, do this after doing the setpci bus reset:

(assuming GPU is 2:00.0, replace with your device address)

$ sudo lspci -v -s 2:00.0 | grep Express Capabilities: [58] Express Legacy Endpoint, MSI 00

The PCIe capability is therefore at 0x58, the link control register is offset 0x10 into the PCIe capability, therefore

printf %x $(( 0x58 + 0x10 )) 68

Read the original value of this register

sudo setpci -s 2:00.0 68.w 0040

The retrain bit is bit 5 (0x20)

printf %x $(( 0x40 | 0x20 )) 60

Write this value back

sudo setpci -s 2:00.0 68.w=60

This bit doesn't need to be cleared, re-read LnkSta with lspci and see if anything has changed.  Regardless of whether it has, a reboot is required before trying to use the device, so don't bother trying to start QEMU again.

If it doesn't do anything, then we might need to look at a VFIO debug trace of case #1 to see if we can get any clues what the driver is doing to the device to get it into that state.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1847 2014-05-26 22:00:59

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

#1 is the case that fails and #2 is the case that works, right?

Yes.

aw wrote:

The link status looks a lot more correct in #2.

Yes.

aw wrote:

The next thing I would try would be testing whether we can poke at the PCIe capability registers on the device to get it to state #2.  So I would look at the LnkSta just after boot, repeat the steps for #1, do the setpci bus reset, then look at the LnkSta for the device again.  Is it any closer to the state just after boot or state #2 (several things should go to reset values after bus reset, concentrate on PCIe status registers)?

Okay I did the setpci bus reset using the following command sequence, valid for my system:

setpci -s 00:0b.0 3e.w printf %04x $(( 0x0008 | 0x40 )) setpci -s 00:0b.0 3e.w=0048 setpci -s 00:0b.0 3e.w=0008

I'll hope it was right as you mentioned in previous post "secondary bus reset", after that I get original LnkSta back:

LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

And of course I cannot start the VM via QEMU as mentioned by you.

aw wrote:

If not, we can try to retrain the link.  Here's another setpci sequence, do this after doing the setpci bus reset:

(assuming GPU is 2:00.0, replace with your device address)

$ sudo lspci -v -s 2:00.0 | grep Express Capabilities: [58] Express Legacy Endpoint, MSI 00

The PCIe capability is therefore at 0x58, the link control register is offset 0x10 into the PCIe capability, therefore

printf %x $(( 0x58 + 0x10 )) 68

Read the original value of this register

sudo setpci -s 2:00.0 68.w 0040

The retrain bit is bit 5 (0x20)

printf %x $(( 0x40 | 0x20 )) 60

Write this value back

sudo setpci -s 2:00.0 68.w=60

This bit doesn't need to be cleared, re-read LnkSta with lspci and see if anything has changed.  Regardless of whether it has, a reboot is required before trying to use the device, so don't bother trying to start QEMU again.

If it doesn't do anything, then we might need to look at a VFIO debug trace of case #1 to see if we can get any clues what the driver is doing to the device to get it into that state.

This was not needed as bus reset above already restored LinkSta.

Offline

#1848 2014-05-26 22:06:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:

#1 is the case that fails and #2 is the case that works, right?

Yes.

aw wrote:

The link status looks a lot more correct in #2.

Yes.

aw wrote:

The next thing I would try would be testing whether we can poke at the PCIe capability registers on the device to get it to state #2.  So I would look at the LnkSta just after boot, repeat the steps for #1, do the setpci bus reset, then look at the LnkSta for the device again.  Is it any closer to the state just after boot or state #2 (several things should go to reset values after bus reset, concentrate on PCIe status registers)?

Okay I did the setpci bus reset using the following command sequence, valid for my system:

setpci -s 00:0b.0 3e.w printf %04x $(( 0x0008 | 0x40 )) setpci -s 00:0b.0 3e.w=0048 setpci -s 00:0b.0 3e.w=0008

I'll hope it was right as you mentioned in previous post "secondary bus reset", after that I get original LnkSta back:

LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

And of course I cannot start the VM via QEMU as mentioned by you.

Ok, so do we get the same result with QEMU?  Setup the #1 case again, then start QEMU again with no disk so the VM only boots up to seabios.  Quit QEMU.  What's the lspci state of the device at that point compared to clean boot?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1849 2014-05-26 22:19:55

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Ok, so do we get the same result with QEMU?  Setup the #1 case again, then start QEMU again with no disk so the VM only boots up to seabios.  Quit QEMU.  What's the lspci state of the device at that point compared to clean boot?

EDIT: Oops I was wrong with diff. I've edited the lspci and updated it!

I created new lspci outputs. This is the diff after this test case:

--- lspci.3 2014-05-27 00:23:49.016539086 +0200 +++ lspci.4 2014-05-27 00:27:07.796718133 +0200 @@ -1,12 +1,12 @@ 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b00 - Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- + Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- - Interrupt: pin A routed to IRQ 10 - Region 0: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] - Region 2: Memory at df800000 (64-bit, prefetchable) [disabled] [size=8M] - Region 4: I/O ports at ce00 [disabled] [size=256] - Region 5: Memory at fdc80000 (32-bit, non-prefetchable) [disabled] [size=256K] + Interrupt: pin A routed to IRQ 19 + Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] + Region 2: Memory at df800000 (64-bit, prefetchable) [size=8M] + Region 4: I/O ports at ce00 [size=256] + Region 5: Memory at fdc80000 (32-bit, non-prefetchable) [size=256K] [virtual] Expansion ROM at d0000000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 @@ -44,7 +44,7 @@ Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 - ATSCtl: Enable+, Smallest Translation Unit: 00 + ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: vfio-pci

Last edited by mbroemme (2014-05-26 22:26:26)

Offline

#1850 2014-05-26 22:38:55

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm... If I extend test case #1 by the following scenario I get another lspci diff:

#1 Test case from original #1
#2 Suspend/resume host
#3 lspci -vvv -s 04:00.0 > lspci.5
#4 Repeat test case from original #1
#5 Start QEMU without any drive, wait until seabios shows "No bootable device." and press STRG+C
#6 lspci -vvv -s 04:00.0 > lspci.6

The diff is the following:

--- lspci.5 2014-05-27 00:34:37.703926234 +0200 +++ lspci.6 2014-05-27 00:35:53.712763395 +0200 @@ -18,7 +18,7 @@ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes - DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- + DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+ @@ -38,7 +38,7 @@ UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- + CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19

Offline

#1851 2014-05-27 07:27:55

thelamer
Member
Registered: 2014-05-25
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I thought I would write down what I needed to do to get my gaming VM up and running. I am not really in a position to answer any hardcore technical questions as this mostly came from googling and this thread.

I am happy to report I got to get some Wolfenstein in on my linux desktop even if it was only for 2 hours out of the entire weekend sad .

GameVM

I am running relatively bleeding edge hardware. It is a i7-4790 on a Z87X-UD4H and a XFX R9 270x I bought at Best Buy ( it was the weekend and I was desperate to test)

First problem I had is with IOMMU enabled is my marvell 88SE9172 SATA controller was broken on boot(running the current stable kernel 3.14) which dropped 2 disks out of my 16 driver raid array. Two things I can say is, praise science for raid 6 and bitmap array rebuilds, otherwise I would have said nope right there and just made sure my data was safe.

Second problem was that my motherboard has a i915 VGA adapter and is effected by an outstanding arbitration bug.

To resolve this I compiled Alex Williamson's fork of the kernel mainline and applied a i915 patch he put together ( big shout out to Alex he is the man and extrememly helpful on all the forums he communicates on. As far as I know is the current king of VFIO help it seems like).

I am running Debian Jessie on my server to checkout and compile Alex's kernel I needed to:

git clone git://github.com/awilliam/linux-vfio.git linux-vfio cd linux-vfio git checkout dma-alias-v4

created a file called i915patch:

http://pastebin.com/dZqD3k5a

Now patched it in:

patch -i i915patch

Now to compile:

cp /boot/config-`uname -r` ./.config make menuconfig

At this point all the VFIO stuff will be built as modules given my current kernel config, but I ended up setting them anyway.

edit the .config file and change/add the following lines:

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y

Now compiled kernel and output debs:

fakeroot make-kpkg -j8 --initrd --append-to-version=-awilliam kernel_image kernel_headers

Before I install them updated grub options:

in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1"

Now install them:

dpkg -i linux-image-3.15.0-rc6-awilliam-i915-2+_3.15.0-rc6-awilliam-10.00.Custom_amd64.deb dpkg -i linux-headers-3.15.0-rc6-awilliam-i915-2+_3.15.0-rc6-awilliam-10.00.Custom_amd64.deb

Finally blacklist radeon:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

Reboot!

Ok so now I am booted and have a kernel that finally uses lube. Through troubleshooting I found out I needed the latest qemu and seabios from the current git tree. The ones bundled in jessie even in experimental did not cut it.

So lets build qemu:

git clone http://git.qemu.org/git/qemu.git cd qemu ./configure --enable-libusb make make install

Resulting binaries will be in /usr/local/bin/ so I just use full path to ensure I am using the latest.

Now build seabios:

git clone git://git.seabios.org/seabios.git seabios cd seabios/ make menuconfig (save and exit) make 

Resulting bios.bin in the out/ subfolder.

Ok so I got all the tools I need to build a gaming VM, not gonna lie at this point I was 3 days in and was getting pretty excited to play the new Wolfenstein on my linux Desktop from my server using steam's new streaming service.

Get networking setup:

/etc/network/interfaces:

 # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo br0 iface lo inet loopback # Set up interfaces manually, avoiding conflicts with, e.g., network manager iface eth0 inet manual # Bridge setup iface br0 inet static pre-up ip tuntap add dev tap0 mode tap user root pre-up ip link set tap0 up bridge_ports all tap0 address 192.168.1.150 netmask 255.255.255.0 gateway 192.168.1.1 bridge_stp off bridge_maxwait 0 bridge_fd 0 post-down ip link set tap0 down post-down ip tuntap del dev tap0 mode tap 

I only have one NIC and I plan on bridging it.

Setup qemu.conf: (needed for the card to work properly)

/etc/qemu/qemu.conf

group = "root" user = "root" clear_emulator_capabilities = 0

First installed windows 7 (windows 8 and 8.1 did not work for me something is broken in the catalyst driver for now)


/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \ -bios /storage/OSs/seabios/out/bios.bin \ -boot menu=on \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -device ide-hd,bus=piix4-ide.0,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \ -device ide-cd,bus=piix4-ide.1,drive=isocd -drive file=/storage/OSs/vm/windows7.iso,id=isocd \ -usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \ -net nic -net tap,ifname=tap0,script=no,downscript=no \ -vnc :1

Explanation, 01:00.0 is my 270x, 00:1b.0 is my MB audio the hdmi audio on the card did not work for me, initially to get windows to install I use the piix4 controller as it does not need drivers at install time virtio or q35 at install did not work for me, the usb devices are keyboard and mouse.

Got monitor output Hooray!

Install windows like normal and shutdown.

Now I installed the virtio drivers found here: http://alt.fedoraproject.org/pub/alt/vi … st/images/

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \ -bios /storage/OSs/seabios/out/bios.bin \ -boot menu=on \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -device virtio-scsi-pci,id=storage -drive file=/storage/OSs/vm/storage.img,id=storage,format=raw \ -device ide-hd,bus=piix4-ide.0,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \ -device ide-cd,bus=piix4-ide.1,drive=isocd -drive file=/storage/OSs/vm/virtio-win-0.1-74.iso,id=isocd \ -usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \ -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \ -vnc :1

Here I added a disk image and changed my nic to virtio so I could easily install the drivers in windows 7 and poweroff.

Now that you have virtio installed boot the machine make sure it works work:

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \ -bios /storage/OSs/seabios/out/bios.bin \ -boot menu=on \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -device scsi-hd,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \ -device scsi-hd,drive=data -drive file=/storage/OSs/vm/storage.img,id=data,format=raw \ -usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \ -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \ -vnc :1

Ok driver time. For me I spent a whole day installing and re-installing windows and fighting BSODs only to find out I needed to reboot the host after a driver upgrade. Maybe catalyst writes a new rom to the card or something during the install not sure.

So with your machine still open installed the latest beta drivers on amd's website for my card. Then powered off (not reboot only got BSOD with reboot)

Rebooted the host.

Fire back up:

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \ -bios /storage/OSs/seabios/out/bios.bin \ -boot menu=on \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -device scsi-hd,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \ -device scsi-hd,drive=data -drive file=/storage/OSs/vm/storage.img,id=data,format=raw \ -usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \ -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \ -vnc :1

From here I installed tightvnc and left my monitor and usb devices in the dust. Planning on getting one of these:

http://www.ebay.com/itm/Headless-server … 1087766664

With steam running I just enabled in game streaming: https://support.steampowered.com/kb_art … -RIAV-1617

Now I need to get steamOS as a network boot option for all my HTPCs so I can play games on the couch.

Last edited by thelamer (2014-05-27 07:28:45)

Offline

#1852 2014-05-27 11:18:21

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Namelles_One wrote:
zzz3000 wrote:

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

If you don't want to go bleeding edge for the whole distro, you can use the Fedora virt-preview repo: http://fedoraproject.org/wiki/Virtualiz … Repository

It effectively gives you "rawhide" virt bits while keeping the rest on the latest stable version.

Thanks. This is realy help me.

Offline

#1853 2014-05-27 15:52:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the report and details!

thelamer wrote:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1"

You shouldn not need vfio_iommu_type1.allow_unsafe_interrupts=1 on your hardware

Finally blacklist radeon:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

Some distros require rebuilding the initramfs after this or else the driver is loaded before we get to the rootfs

Now build seabios:

git clone git://git.seabios.org/seabios.git seabios cd seabios/ make menuconfig (save and exit) make 

Resulting bios.bin in the out/ subfolder.

Is this really needed?  I don't know of anything wrong with the BIOS supplied by upstream QEMU.

Setup qemu.conf: (needed for the card to work properly)

/etc/qemu/qemu.conf

group = "root" user = "root" clear_emulator_capabilities = 0

This is only needed for libvirt support, which you don't seem to be using, at least not yet.

Now I installed the virtio drivers found here: http://alt.fedoraproject.org/pub/alt/vi … st/images/

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \ -bios /storage/OSs/seabios/out/bios.bin \ -boot menu=on \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -device virtio-scsi-pci,id=storage -drive file=/storage/OSs/vm/storage.img,id=storage,format=raw \ -device ide-hd,bus=piix4-ide.0,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \ -device ide-cd,bus=piix4-ide.1,drive=isocd -drive file=/storage/OSs/vm/virtio-win-0.1-74.iso,id=isocd \ -usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \ -net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \ -vnc :1

Here I added a disk image and changed my nic to virtio so I could easily install the drivers in windows 7 and poweroff.

You can also pass the virtio ISO as a 2nd CD during install, load both the network and disk drivers during install, then you don't need to do this 2-step process.

From here I installed tightvnc and left my monitor and usb devices in the dust. Planning on getting one of these:

http://www.ebay.com/itm/Headless-server … 1087766664

Heh, neat.  Thanks


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1854 2014-05-27 21:57:01

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex, so anymore ideas to my lcpsi output at post #1849 and #1850?

Offline

#1855 2014-05-27 22:36:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

Alex, so anymore ideas to my lcpsi output at post #1849 and #1850?

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1856 2014-05-27 22:47:35

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.

In #1850 it is another working case. However I'm not sure if radeon driver maintainer might help as they may know some magic of these devices.

Offline

#1857 2014-05-28 06:53:53

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I want to passthrought my ATI radeon 38xx card to a KVM guest (windows), host is using i7's onboard video card.

Blacklisted radeon driver, and successfully bind these two driver to vfio-pci. (via the script in first post)

root@robin:~# lspci | grep ATI 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series]

Getting this error:

Error starting domain: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: vfio: error, group 6 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: vfio: failed to get group 6 qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized

All of devices are listed in same iommu_group, including my secondary network and onboard network card and graphics.

root@robin:/dev# ls /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices/ 0000:00:1c.0 0000:00:1c.5 0000:01:00.0 0000:02:00.0 0000:04:00.0 0000:00:1c.4 0000:00:1c.6 0000:01:00.1 0000:03:00.0
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series] 02:00.0 Ethernet controller: Qualcomm Atheros AR8161 Gigabit Ethernet (rev 10) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41)

My kernel cmdline

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img
Linux robin 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux

How i can only pass the ATI card to the guest? Not all the devices in iommu group.

Thanks

Offline

#1858 2014-05-28 10:32:15

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

How i can only pass the ATI card to the guest? Not all the devices in iommu group.

You need the ACS override patch for your hardware. It is not in mainline Linux kernel (neither in 3.14 nor in upcoming 3.15). You can get it from https://lkml.org/lkml/2013/5/30/513 or in the linux-mainline archive from #1 post in this thread. You need to patch and rebuild your kernel for it.

Offline

#1859 2014-05-28 12:15:20

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
buraktamturk wrote:

How i can only pass the ATI card to the guest? Not all the devices in iommu group.

You need the ACS override patch for your hardware. It is not in mainline Linux kernel (neither in 3.14 nor in upcoming 3.15). You can get it from https://lkml.org/lkml/2013/5/30/513 or in the linux-mainline archive from #1 post in this thread. You need to patch and rebuild your kernel for it.

Thank you, applied patches and rebuilded the kernel, it worked. My new cmdline is

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 pci-stub.ids=1002:9505,1002:aa18 pcie_acs_override=downstream

Guest see the graphics card but just used the emulated one, but i see there is two Standard VGA Adapter, i think guest excepting install the drivers to show up the monitor (or excepting to me remove regular one).

Now when i tried to stop the virtual machine i get following error (dmesg output of error) and kvm, libvirt is complately hanged until i reboot the computer. Is there anything i miss?

[ 639.297949] BUG: unable to handle kernel paging request at 000000fd000000fc [ 639.298021] IP: [<000000fd000000fc>] 0xfd000000fc [ 639.298066] PGD 0 [ 639.298083] Oops: 0010 [#1] SMP [ 639.298111] Modules linked in: tun nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc bridge 8021q garp stp mrp llc snd_hda_codec_via snd_hda_codec_generic snd_hda_codec_hdmi nls_utf8 nls_cp437 vfat fat x86_pkg_temp_thermal intel_powerclamp iTCO_wdt iTCO_vendor_support ppdev intel_rapl coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel joydev snd_hda_intel aes_x86_64 lrw snd_hda_codec gf128mul snd_hwdep glue_helper ablk_helper cryptd snd_pcm efi_pstore evdev psmouse pcspkr serio_raw snd_timer snd soundcore i2c_i801 efivars lpc_ich i915 mfd_core drm_kms_helper drm i2c_algo_bit mei_me i2c_core mei parport_pc tpm_infineon battery parport tpm_tis tpm video button processor ip6table_filter ip6_tables ipt_REJECT xt_tcpudp xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables fuse autofs4 xfs crc32c libcrc32c dm_mod sd_mod crc_t10dif crct10dif_generic hid_generic ata_generic usbhid hid crct10dif_pclmul crct10dif_common ata_piix libata scsi_mod ehci_pci xhci_hcd ehci_hcd r8169 alx mii mdio usbcore thermal fan usb_common thermal_sys [ 639.298909] CPU: 0 PID: 2361 Comm: libvirtd Tainted: G W 3.14.4-rt5 #3 [ 639.298951] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77M-D3H, BIOS F15a 12/31/2013 [ 639.299005] task: ffff88041dcaac80 ti: ffff88041ce52000 task.ti: ffff88041ce52000 [ 639.299046] RIP: 0010:[<000000fd000000fc>] [<000000fd000000fc>] 0xfd000000fc [ 639.299091] RSP: 0018:ffff88041ce53de0 EFLAGS: 00010202 [ 639.299121] RAX: 0000000000000008 RBX: ffff88041e520098 RCX: 0000000000000000 [ 639.299160] RDX: 000000fd000000fc RSI: ffff88041e520098 RDI: ffff88041e520098 [ 639.299200] RBP: ffff88041e520140 R08: 0000000000000000 R09: 0000000000000008 [ 639.299239] R10: 0000000000000074 R11: ffff88041ce53ba6 R12: 0000000000000004 [ 639.299278] R13: 0000000000000000 R14: 000000000000000c R15: ffff88041c01e810 [ 639.299318] FS: 00007f3da4c98700(0000) GS:ffff88042f200000(0000) knlGS:0000000000000000 [ 639.299362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 639.299394] CR2: 000000fd000000fc CR3: 000000041b183000 CR4: 00000000001407e0 [ 639.299433] Stack: [ 639.299448] ffffffff813ab9d7 0000000000000004 ffff88041e520098 0000000000000004 [ 639.299497] ffffffff813ac3b8 0000000000000004 ffff88041e520098 ffff88041e520140 [ 639.299546] 0000000000000246 ffffffff813ac504 ffff88041e520098 ffffffff81885b60 [ 639.299594] Call Trace: [ 639.299617] [<ffffffff813ab9d7>] ? __rpm_callback+0x27/0x70 [ 639.299653] [<ffffffff813ac3b8>] ? rpm_idle+0x198/0x290 [ 639.299686] [<ffffffff813ac504>] ? __pm_runtime_idle+0x54/0x80 [ 639.299721] [<ffffffff812ee2e7>] ? pci_device_remove+0x67/0xa0 [ 639.299757] [<ffffffff813a1e45>] ? __device_release_driver+0x75/0xf0 [ 639.299794] [<ffffffff813a1ed9>] ? device_release_driver+0x19/0x30 [ 639.299831] [<ffffffff813a0e35>] ? unbind_store+0xb5/0xe0 [ 639.299865] [<ffffffff81233c2e>] ? kernfs_fop_write+0xbe/0x120 [ 639.299901] [<ffffffff811c5080>] ? vfs_write+0xb0/0x1e0 [ 639.299932] [<ffffffff811c5a2d>] ? SyS_write+0x3d/0xa0 [ 639.299965] [<ffffffff81508e39>] ? system_call_fastpath+0x16/0x1b [ 639.299999] Code: Bad RIP value. [ 639.300024] RIP [<000000fd000000fc>] 0xfd000000fc [ 639.300058] RSP <ffff88041ce53de0> [ 639.300079] CR2: 000000fd000000fc [ 639.313543] ---[ end trace 1613cc056b6b661d ]---

Offline

#1860 2014-05-28 12:35:51

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone knows how to add ",multifunction=on,x-vga=on" to pci config on virsh xml configuration file?

My configuration file is follows

<domain type='kvm'> <name>Windows7</name> <uuid>42716343-35d0-e786-5a86-d6d66dc6ced2</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <loader>/usr/share/ovmf/OVMF.fd</loader> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/storage/share/programlar/ISO/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/buraktamturk/Downloads/virtio-win-0.1-74.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/storage/vm/disk/Windows7_C.img'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:76:17:e2'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x0752'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x00cb'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

Thanks

Offline

#1861 2014-05-28 12:58:26

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

Anyone knows how to add ",multifunction=on,x-vga=on" to pci config on virsh xml configuration file?

My configuration file is follows

<domain type='kvm'> <name>Windows7</name> <uuid>42716343-35d0-e786-5a86-d6d66dc6ced2</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <loader>/usr/share/ovmf/OVMF.fd</loader> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/storage/share/programlar/ISO/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/buraktamturk/Downloads/virtio-win-0.1-74.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/storage/vm/disk/Windows7_C.img'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:76:17:e2'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x0752'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x00cb'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

Thanks

Devices you want to pass have to be defined using qemu:commandlie. Here is my config: http://pastebin.com/ZVwvv9Hc

Because of this, you have to bind them to vfio by hand. You can use my libvirt's hook script to bind these devices to vfio (or bind HD Audio to vfio and rebind it to host's driver after vm is down; should work with every pci device): http://pastebin.com/fznjWcZ4 (put it as /etc/libvirt/hooks/qemu and change device id's).

Offline

#1862 2014-05-28 14:25:52

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Devices you want to pass have to be defined using qemu:commandlie. Here is my config: http://pastebin.com/ZVwvv9Hc

Because of this, you have to bind them to vfio by hand. You can use my libvirt's hook script to bind these devices to vfio (or bind HD Audio to vfio and rebind it to host's driver after vm is down; should work with every pci device): http://pastebin.com/fznjWcZ4 (put it as /etc/libvirt/hooks/qemu and change device id's).

Very thanks.

I feel very stupid. I can't find the bus that graphics card attached.

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> </qemu:commandline>

I get "Bus 'root.1' not found", i can't find the bus with lspci. (tried several ones including root.0 to 5 and not giving bus number but didn't do the trick)

How i can find it? The device i want to attach is Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850]

Thank you again,

lspci output

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) Subsystem: Gigabyte Technology Co., Ltd Device 5000 Flags: bus master, fast devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device d000 Flags: bus master, fast devsel, latency 0, IRQ 45 Memory at f7800000 (64-bit, non-prefetchable) [size=4M] Memory at d0000000 (64-bit, prefetchable) [size=256M] I/O ports at f000 [size=64] Expansion ROM at <unassigned> [disabled] Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [d0] Power Management version 2 Capabilities: [a4] PCI Advanced Features Kernel driver in use: i915 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI]) Subsystem: Gigabyte Technology Co., Ltd Device 5007 Flags: bus master, medium devsel, latency 0, IRQ 42 Memory at f7f00000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Kernel driver in use: xhci_hcd 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device 1c3a Flags: bus master, fast devsel, latency 0, IRQ 44 Memory at f7f19000 (64-bit, non-prefetchable) [size=16] Capabilities: [50] Power Management version 3 Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+ Kernel driver in use: mei_me 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI]) Subsystem: Gigabyte Technology Co., Ltd Device 5006 Flags: bus master, medium devsel, latency 0, IRQ 16 Memory at f7f17000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci-pci 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device a014 Flags: bus master, fast devsel, latency 0, IRQ 46 Memory at f7f10000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [130] Root Complex Link Kernel driver in use: snd_hda_intel 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: f7e00000-f7efffff Prefetchable memory behind bridge: 00000000e0000000-00000000efffffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001 Capabilities: [a0] Power Management version 2 Kernel driver in use: pcieport 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f7d00000-f7dfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001 Capabilities: [a0] Power Management version 2 Kernel driver in use: pcieport 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: f7c00000-f7cfffff Prefetchable memory behind bridge: 00000000f0000000-00000000f00fffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001 Capabilities: [a0] Power Management version 2 Kernel driver in use: pcieport 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) (prog-if 01 [Subtractive decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=04, subordinate=05, sec-latency=0 Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001 Capabilities: [a0] Power Management version 2 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI]) Subsystem: Gigabyte Technology Co., Ltd Device 5006 Flags: bus master, medium devsel, latency 0, IRQ 23 Memory at f7f16000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci-pci 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device 5001 Flags: bus master, medium devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: lpc_ich 00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04) (prog-if 8f [Master SecP SecO PriP PriO]) Subsystem: Gigabyte Technology Co., Ltd Device b005 Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19 I/O ports at f110 [size=8] I/O ports at f100 [size=4] I/O ports at f0f0 [size=8] I/O ports at f0e0 [size=4] I/O ports at f0d0 [size=16] I/O ports at f0c0 [size=16] Capabilities: [70] Power Management version 3 Capabilities: [b0] PCI Advanced Features Kernel driver in use: ata_piix 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) Subsystem: Gigabyte Technology Co., Ltd Device 5001 Flags: medium devsel, IRQ 18 Memory at f7f15000 (64-bit, non-prefetchable) [size=256] I/O ports at f040 [size=32] 00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04) (prog-if 85 [Master SecO PriO]) Subsystem: Gigabyte Technology Co., Ltd Device b002 Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19 I/O ports at f0b0 [size=8] I/O ports at f0a0 [size=4] I/O ports at f090 [size=8] I/O ports at f080 [size=4] I/O ports at f070 [size=16] I/O ports at f060 [size=16] Capabilities: [70] Power Management version 3 Capabilities: [b0] PCI Advanced Features Kernel driver in use: ata_piix 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850] (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 2542 Flags: fast devsel, IRQ 11 Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at f7e20000 (64-bit, non-prefetchable) [disabled] [size=64K] I/O ports at e000 [disabled] [size=256] Expansion ROM at f7e00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: vfio-pci 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series] Flags: fast devsel, IRQ 17 Memory at f7e30000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: vfio-pci 02:00.0 Ethernet controller: Qualcomm Atheros AR8161 Gigabit Ethernet (rev 10) Subsystem: Gigabyte Technology Co., Ltd Device e000 Flags: bus master, fast devsel, latency 0, IRQ 48 Memory at f7d00000 (64-bit, non-prefetchable) [size=256K] I/O ports at d000 [size=128] Capabilities: [40] Power Management version 3 Capabilities: [58] Express Endpoint, MSI 00 Capabilities: [c0] MSI: Enable+ Count=1/16 Maskable+ 64bit+ Capabilities: [d8] MSI-X: Enable- Count=16 Masked- Capabilities: [100] Advanced Error Reporting Capabilities: [180] Device Serial Number ff-5e-25-1c-90-2b-34-ff Kernel driver in use: alx 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) Subsystem: Realtek Semiconductor Co., Ltd. Device 0123 Flags: bus master, fast devsel, latency 0, IRQ 43 I/O ports at c000 [size=256] Memory at f7c00000 (64-bit, non-prefetchable) [size=4K] Memory at f0000000 (64-bit, prefetchable) [size=16K] Expansion ROM at f0020000 [disabled] [size=128K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [70] Express Endpoint, MSI 01 Capabilities: [b0] MSI-X: Enable- Count=4 Masked- Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Virtual Channel Capabilities: [160] Device Serial Number 71-02-00-00-68-4c-e0-00 Kernel driver in use: r8169 04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41) (prog-if 01 [Subtractive decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=04, secondary=05, subordinate=05, sec-latency=32 Capabilities: [90] Power Management version 2 Capabilities: [a0] Subsystem: Gigabyte Technology Co., Ltd Device 8892 lspci -t -[0000:00]-+-00.0 +-02.0 +-14.0 +-16.0 +-1a.0 +-1b.0 +-1c.0-[01]--+-00.0 | \-00.1 +-1c.4-[02]----00.0 +-1c.5-[03]----00.0 +-1c.6-[04-05]----00.0-[05]-- +-1d.0 +-1f.0 +-1f.2 +-1f.3 \-1f.5

Offline

#1863 2014-05-28 17:56:13

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

<qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/var/lib/libvirt/firmware/vbios_GF106.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

First device is PCH bridge emulated by qemu to which you want connect gpu you're passing, second and third are gpu itself and gpu's hda. TBH you can just copy-paste it wink

EDIT:
Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

Last edited by dwe11er (2014-05-28 18:12:54)

Offline

#1864 2014-05-28 18:30:23

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
<qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/var/lib/libvirt/firmware/vbios_GF106.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

First device is PCH bridge emulated by qemu to which you want connect gpu you're passing, second and third are gpu itself and gpu's hda. TBH you can just copy-paste it wink

EDIT:
Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

Thank You! I just see the windows installer on the monitor that connected to the ati card. I was giving up!

Now i understant what "bus" means here. Just had to change "ioh3420,bus=pcie.0" to "ioh3420,bus=pci.0". Everything seems working! except OVMF UEFI bios firmware (Black screen at all, UEFI doesn't important for me at this point).

What does "romfile=/var/lib/libvirt/firmware/vbios_GF106.rom" means? I do not have such file.

Thank you again!!

Offline

#1865 2014-05-28 19:24:02

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now another problem at windows installation's final step :'(. Is there really something that i miss?

svrhqhD.png

Offline

#1866 2014-05-28 19:37:17

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

Now another problem at windows installation's final step :'(. Is there really something that i miss?

Did you rebooted your Windows during driver installation? Could you try to make a clean host reboot and then start VM with installed driver.

Offline

#1867 2014-05-28 19:53:00

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

It's not only you. Same here for my gtx660

EDIT: Downgrading worked

Last edited by Flyser (2014-05-28 19:59:28)

Offline

#1868 2014-05-28 20:08:55

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
buraktamturk wrote:

Now another problem at windows installation's final step :'(. Is there really something that i miss?

Did you rebooted your Windows during driver installation? Could you try to make a clean host reboot and then start VM with installed driver.

Setup automatically rebooted guest after completing step 1. installing windows.

When installation is in 2. Step ("Completing Installation"), screen flickered (i think it was installing the graphics driver), then BSOD show up. When i start the guest again, (still standard vga driver), installer said something like "the system was unexceptedly restarted and setup cannot contiune"

Now tried with Windows 8.1 Update 1 iso, the os is installed without any graphics driver, and after setup, i manually installed latest graphics of ati 3800. voilla, worked!, resolution is fine and ATI control center is working.
So i have to block the default bundled ati driver installation of unmodified Windows 7 (i belive radeon5400 and older cards have microsoft driver by default), i will figure out somehow. I will edit this post when i can successfully boot windows 7 os.

Thank you again.

Offline

#1869 2014-05-28 22:12:33

thelamer
Member
Registered: 2014-05-25
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:
mbroemme wrote:
buraktamturk wrote:

Now another problem at windows installation's final step :'(. Is there really something that i miss?

Did you rebooted your Windows during driver installation? Could you try to make a clean host reboot and then start VM with installed driver.

Setup automatically rebooted guest after completing step 1. installing windows.

When installation is in 2. Step ("Completing Installation"), screen flickered (i think it was installing the graphics driver), then BSOD show up. When i start the guest again, (still standard vga driver), installer said something like "the system was unexceptedly restarted and setup cannot contiune"

Now tried with Windows 8.1 Update 1 iso, the os is installed without any graphics driver, and after setup, i manually installed latest graphics of ati 3800. voilla, worked!, resolution is fine and ATI control center is working.
So i have to block the default bundled ati driver installation of unmodified Windows 7 (i belive radeon5400 and older cards have microsoft driver by default), i will figure out somehow. I will edit this post when i can successfully boot windows 7 os.

Thank you again.

For me to get windows 7 working I needed to reboot the host (not guest) after the ati driver installation. (just selected poweroff after the driver install)

I was getting the same atikpmag BSOD you were.

It is worth a try.

Offline

#1870 2014-05-28 23:08:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
dwe11er wrote:

Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

It's not only you. Same here for my gtx660

EDIT: Downgrading worked

Same here *sigh* time to start comparing traces :-\


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1871 2014-05-29 03:23:38

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!
I am working on passthrough my gpu to windows VM by this guide, but I can't resolve one bug:

me@host:~$sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom -device vfio-pci,host=06:00.1,bus=root.1,addr=01.1 \ -vga cirrus -cdrom /home/me/Win_8.1.iso 

gives

qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: vfio: Device does not support requested feature x-vga qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: vfio: failed to get device 0000:05:00.0 qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: Device 'vfio-pci' could not be initialized

System is Ubuntu 14.04,

me@host:~# cat /boot/config-3.13.0-27-generic| grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y me@host:~# lsmod |grep vfio vfio_pci 36474 0 vfio_iommu_type1 17636 0 vfio 24873 2 vfio_iommu_type1,vfio_pci me@host:~# lspci -s 05:00.0 -v 05:00.0 3D controller: NVIDIA Corporation GF110 [GeForce GTX 590] (rev a1) Subsystem: NVIDIA Corporation Device 0868 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at ec000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=32M] Expansion ROM at ed080000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci

Ummm... Does anyone have an idea? At least how to debug?

Last edited by gutleib (2014-05-29 03:24:41)

Offline

#1872 2014-05-29 03:29:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gutleib wrote:
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: vfio: Device does not support requested feature x-vga
05:00.0 3D controller: NVIDIA Corporation GF110 [GeForce GTX 590] (rev a1)

Ummm... Does anyone have an idea? At least how to debug?

The class code for you device is "3D controller", not "VGA compatible controller", therefore x-vga doesn't work for it because it doesn't support VGA.

EDIT: Are you using SLI and picked the slave card as your VGA device by mistake?

Last edited by aw (2014-05-29 03:35:00)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1873 2014-05-29 03:39:40

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ah! thanks
search doesn't show much on how to fix this, do you have a suggestion?

edit: i see 06.00.0 is vga. going to try use it

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=01.1

hmmm.
No errors.
Nothing in vnc.
Nothing on screen, though.

Last edited by gutleib (2014-05-29 03:54:46)

Offline

#1874 2014-05-29 04:08:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gutleib wrote:

Ah! thanks
search doesn't show much on how to fix this, do you have a suggestion?

edit: i see 06.00.0 is vga. going to try use it

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=01.1

hmmm.
No errors.
Nothing in vnc.
Nothing on screen, though.

Maybe you should start with one card and build from there.  You haven't identified what patches are in your kernel, your host graphics device, the version of qemu used, etc.  Besides, you've made a non-standard PCIe configuration in the guest by using PCI slots 0 and 1 behind your root port.  Typically only slot 0 is scanned by the guest.  Once you get your VGA card working, add another ioh3420 and put the other card behind it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1875 2014-05-29 04:47:05

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

So i have to block the default bundled ati driver installation of unmodified Windows 7 (i belive radeon5400 and older cards have microsoft driver by default), i will figure out somehow. I will edit this post when i can successfully boot windows 7 os.

in fact, you dont have to block or figure anything.
just dont use passthrough during installation/inital setup (==use only emulated qemu vga, not host cards)
after you are done, you can add passthrough and install needed graphics drivers

in my case (HD 6450) it served one more purpose - i was getting BSOD (stop 0x0000007b i think?) every time i tried to poweroff guest (and at least once after some idle time - i think win7 tried to power down monitor) with passthrough. the following (before installing amd drivers) resolved the issue: i disabled any power saving related stuff i could find - HDD/monitor sleep-on-idle, etc. plus, after installing drivers, just in case, i disabled anything power saving in driver's settings.

gutleib wrote:

hmmm.
No errors.
Nothing in vnc.
Nothing on screen, though.

you sure you don't use intel's IGD for host and miss vga arbiter patch?

Offline

#1876 2014-05-29 04:52:51

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The thing is, gpu is gtx590, which is basically two 580 on single board. Not sure how it will behave with only one "half" passthru'ed.
The kernel is Ubuntu 14.04 from updates repo, no patches. Only some modules.
Host gpu is radeon HD2600, afair, but generaly it's a smallest radeon hd with passive heatsink I could find.
Qemu is also ubuntu 14.04 stock, which is 2.0.
I am trying to make gpu passthrough with minimal tinkering, since I have quite a bunch of 3d workstations. Windows and custom linux builds equally suck.

aw wrote:

...Besides, you've made a non-standard PCIe configuration in the guest by using PCI slots 0 and 1 behind your root port.  Typically only slot 0 is scanned by the guest.  Once you get your VGA card working, add another ioh3420 and put the other card behind it.

Ummm... Didn't know that.
Will check.

But would be really great if you could explain the whole ioh3420 argument. Or point to where I could read the short version smile
So far, I don't understand which variables should be changed for second ioh.

Offline

#1877 2014-05-29 04:57:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need the VGA arbiter patch from a page or two back


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1878 2014-05-29 05:26:17

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Today test again with Kernel 3.15-rc7, still without any luck.

The VM still hangs during reboot when Guest OS is Ubuntu Server.
And CUDA still get "InvalidDevice" error to me, even I tried use dumped VGA BIOS and -cpu using qemu64.

I still can't understand that with same settings, same functions, why Windows 7 can run CUDA but Ubuntu Server can't.
Now update the kernel of guest Ubuntu is my only hope for now.

But I'm not get rid of the possibility that maybe I'm doing a wrong installation.
If update the kernel still can't fix the problem. I'll really have no idea.

Does anyone tried run CUDA in guest Ubuntu and works?

Last edited by AKSN74 (2014-05-29 05:27:27)

Offline

#1879 2014-05-29 08:44:32

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For a while I have been a lurker on this thread and been very impressed by the efforts and time of nhbs, aw and others to get peoples set-up working. I would like a quick feasibility check of my hardware from the other veterans of this thread.

I have the i7-4771 with HD 4600 IGP, AMD 7950 GPU and AsRock Extreme 6 Z87 Motherboard with 16GB RAM. I did my research (hopefully it's right) and this hardware does support the necessary technologies.

I want the host to use the IGP, and passthrough the GPU. I have read mixed reports in this thread about the success for this type of setup, what is the current thinking? In the future I will add an Nvidia GPU for the host.

I have read the Wiki pages on the ABS, Creating Packages and Kernels. I haven't seen an easy-to-follow guide how I would compile the linux-mainline kernel from the AUR. And I've been directed to the Gentoo handbook for more info or Kernel configurations; interesting stuff, but a little over my head and, I think, my needs too.

I haven't touched my Arch partition yet but I have created a seperate Arch install and installed qemu, seabios and abs. I will be creating the virtual machine there and writing down all the steps I take to get it working before transferring that work and attempting it in my main install. Hopefully this method will give me the fewest headaches in the future.

But in relative-newbie terms, how does one install the linux-mainline?

BTW if anyone is interested there is an Arch Linux Class this Saturday on the Linux Kernel and will cover things like compiling your own.

Last edited by cdrjameson (2014-05-29 08:54:16)

Offline

#1880 2014-05-29 09:38:16

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cdrjameson wrote:

For a while I have been a lurker on this thread and been very impressed by the efforts and time of nhbs, aw and others to get peoples set-up working. I would like a quick feasibility check of my hardware from the other veterans of this thread.

I have the i7-4771 with HD 4600 IGP, AMD 7950 GPU and AsRock Extreme 6 Z87 Motherboard with 16GB RAM. I did my research (hopefully it's right) and this hardware does support the necessary technologies.

I want the host to use the IGP, and passthrough the GPU. I have read mixed reports in this thread about the success for this type of setup, what is the current thinking? In the future I will add an Nvidia GPU for the host.

I have read the Wiki pages on the ABS, Creating Packages and Kernels. I haven't seen an easy-to-follow guide how I would compile the linux-mainline kernel from the AUR. And I've been directed to the Gentoo handbook for more info or Kernel configurations; interesting stuff, but a little over my head and, I think, my needs too.

I haven't touched my Arch partition yet but I have created a seperate Arch install and installed qemu, seabios and abs. I will be creating the virtual machine there and writing down all the steps I take to get it working before transferring that work and attempting it in my main install. Hopefully this method will give me the fewest headaches in the future.

But in relative-newbie terms, how does one install the linux-mainline?

BTW if anyone is interested there is an Arch Linux Class this Saturday on the Linux Kernel and will cover things like compiling your own.

i can confirm the following setupd working:
MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20 x86_64 (with additional repositories: kernel-vanilla-mainline and virt-preview)
kernel: kernel-3.15.0-0.rc5.git2.8.vanilla.mainline.knurd.1.fc20.x86_64.rpm
host GPU: integrated (intel i7 4770)
guest GPU0: radeon hd 6450 (Gigabyte GV-R645-1GI)
guest GPU1: nvidia gt 430 // not used at the moment - pending

kernel patchwork performed (basic info for fedora: here), $DST_VER refers to destination kernel version:

  1. grab kernel source (.srpm, as described above) for mainline kernel being used

  2. modify EXTRAVERSION variable in Makefile at root level of mainline kernel source (NOTE: needed for vermagic, see step 6 below)

  3. "make prepare scripts" at root level of mainline kernel source

  4. apply vga arbiter patch (NOTE: this modifies only i915 kernel module)

  5. "make modules SUBDIRS=drivers/gpu/drm/i915" at root level of mainline kernel source

  6. if "modinfo /lib/modules/$DST_VER/kernel/drivers/gpu/drm/i915/i915.ko drivers/gpu/drm/i915/i915.ko | grep vermagic" does not match - restart from step 2

  7. copy built i915.ko module to "/lib/modules/$DST_VER/updates"

  8. perform "depmod" as root

  9. rebuild initramfs (something like "dracut /boot/initramfs-$DST_VER.img $DST_VER") - you might want to save previous initramfs or build initramfs to different filename to enable it by modifying grub2 initrd parameter

this is "easy" but pretty inconvenient way to do things: easy because only one kernel module is rebuilt, unconvenient because on every kernel update the rebuild procedure should be remade from scratch
NOTE: vermagic of built module not matching with running kernel may result in module not being loaded, which in turn (in my case) resulted in X server start rendering host system completely unusable (hang with screen showing garbage-colored lines)

more info bits may be found in my previous posts here

Offline

#1881 2014-05-29 12:40:13

derbroti
Member
Registered: 2014-05-09
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have a working setup with win7 as guest:

intel e3-1230v3
asus p9d ws
sapphire radeon 9200 SE Lite (PCI - host)
sapphire radeon R9 280 (PCIe - guest)
silverstone card Marvell 88SE9130 (PCIe)


I am using the PCI graphics card to be able to set that one as primary in the bios (there is no possibility to manually select the primary when two pcie cards are installed)
The Marvell sata card is primarily used because of its hyperduo (combining ssd + hdd) feature.

One sad thing is, that I can not boot from the Marvell card, if i use passthrough on it. The cards bios shows up but does not detect a single drive... The only working card i found was a LSI card with a sas2008 chip but the ssd caching feature of the marvell was more convenient through... so now I am using virtio-scsi as I gave up on the idea to be able to boot from the passed through marvell controller.

I am using a mainline 3.13 kernel with acs-override patch, because I was unable to build ZFS with >= 3.14

One "problem" i have left:

I am using an old dma-alias patch, that i found on the "Bug 42679 - DMA Read on Marvell 88SE9128 fails" kernel bugzilla post.

Is there any chance to get a patchfile for the new dma-alias-v4 without using aw's whole kernel?

Offline

#1882 2014-05-29 14:16:12

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Okay, I can't seem to get this to work anymore. I can pass the card to the guest, and I get video output from it, but Windows keeps reporting error code 43 in the device manager whenever I install the device drivers.

My setup consists of an AMD 6780 host card (open source drivers) and an Nvidia 780 guest card. I've also installed linux-mainline with all the patches on the front page.

My kernel options are as such:

root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pci-stub.ids=10de:1004,10de:0e1a intel_iommu=on pcie_acs_override=downstream

My qemu command:

qemu-system-x86_64 \ -enable-kvm -M q35 -m 8192 -cpu host \ -smp 6,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

Before I run the VM, I run:

sudo vfio-bind 0000:02:00.0 0000:02:00.1 sudo vga-clear 00:01.0 sudo vga-set 00:01.1

I feel like I'm missing one simple thing, but I've scanned the thread and can't figure out what that is.

Offline

#1883 2014-05-29 14:18:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Okay, I can't seem to get this to work anymore. I can pass the card to the guest, and I get video output from it, but Windows keeps reporting error code 43 in the device manager whenever I install the device drivers.

My setup consists of an AMD 6780 host card (open source drivers) and an Nvidia 780 guest card. I've also installed linux-mainline with all the patches on the front page.

My kernel options are as such:

root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pci-stub.ids=10de:1004,10de:0e1a intel_iommu=on pcie_acs_override=downstream

My qemu command:

qemu-system-x86_64 \ -enable-kvm -M q35 -m 8192 -cpu host \ -smp 6,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

Before I run the VM, I run:

sudo vfio-bind 0000:02:00.0 0000:02:00.1 sudo vga-clear 00:01.0 sudo vga-set 00:01.1

I feel like I'm missing one simple thing, but I've scanned the thread and can't figure out what that is.

Did you upgrade the guest to the 337.88 Nvidia driver?  We're all having trouble with the new driver.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1884 2014-05-29 14:38:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Before I run the VM, I run:

sudo vfio-bind 0000:02:00.0 0000:02:00.1 sudo vga-clear 00:01.0 sudo vga-set 00:01.1

There's a kernel patch for this, use it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1885 2014-05-29 15:04:47

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Did you upgrade the guest to the 337.88 Nvidia driver?  We're all having trouble with the new driver.

Damnit, that explains it. Downgrading to the previous driver fixes it...

aw wrote:

There's a kernel patch for this, use it.

Hey thanks!

Offline

#1886 2014-05-29 20:27:15

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey together,

I achieved to run my qemu script without root priviledges. I needed to set priviledges to three devices, I pass through to my VM. My Script looks as follows:

#!/bin/bash #QEMU settings for user access chown :kvm /dev/vfio/13 chmod g+rw /dev/vfio/13 chown :kvm /dev/vfio/14 chmod g+rw /dev/vfio/14 chown :kvm /dev/vfio/17 chmod g+rw /dev/vfio/17

To automatically run this script, I made a systemd service file:

[Unit] Description=grants members of group kvm access to passthroughed devices After=syslog.target [Service] #EnvironmentFile=-/etc/vfio-pci.cfg Type=oneshot RemainAfterExit=yes ExecStart=-/usr/bin/vfiouseraccess [Install] WantedBy=multi-user.target

My problems is, that this doesn't work:

systemctl status vfiouseraccess.service ⏎ ● vfiouseraccess.service - grants members of group kvm access to passthroughed devices Loaded: loaded (/usr/lib/systemd/system/vfiouseraccess.service; enabled) Active: active (exited) since Do 2014-05-29 22:15:35 CEST; 8min ago Process: 337 ExecStart=/usr/bin/vfiouseraccess (code=exited, status=1/FAILURE) Main PID: 337 (code=exited, status=1/FAILURE) CGroup: /system.slice/vfiouseraccess.service chown: cannot access ‘/dev/vfio/13’: No such file or directory chmod: cannot access ‘/dev/vfio/13’: No such file or directory chown: cannot access ‘/dev/vfio/14’: No such file or directory chmod: cannot access ‘/dev/vfio/14’: No such file or directory chown: cannot access ‘/dev/vfio/17’: No such file or directory chmod: cannot access ‘/dev/vfio/17’: No such file or directory

It seems, that a the point where my script is called, the devices aren't present. Can someone tell my, how I could fix this?

Thanks in advance
apex8

EDIT: I forgot to mention: Once the system has started and I logged in, I can easily run my script manually and its setting the priviledges as intended.

Last edited by apex8 (2014-05-29 20:28:21)

Offline

#1887 2014-05-29 22:01:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Flyser wrote:
dwe11er wrote:

Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

It's not only you. Same here for my gtx660

EDIT: Downgrading worked

Same here *sigh* time to start comparing traces :-\

The driver doesn't even seem to be trying to work, I've reported it to the GeForce Forums, not sure what more to do about it other than hope it gets fixed  https://forums.geforce.com/default/topi … 9/#4229739


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1888 2014-05-30 00:44:25

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You need the VGA arbiter patch from a page or two back

Could you explain? I don't have any intel graphics in my pc at all...

Also: I tried passthrugh'ing only "vga controller" half of gtx590, and also tried different pcie addresses. Results are:
apparently, if I use the line
"sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1
-bios /usr/share/qemu/bios.bin
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom 
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1
-device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on
-device vfio-pci,host=06:00.1,bus=root.1,addr=01.1
-vga cirrus",
the bios boot process doesn't complete. It goes first "seabios..." line in vnc, then the second, "ipxe..." then (in second line) "PnP PNN..." and lots of numbers. then it just sits there.
If I use only one pci device (the vga half, not the 3d controller),
"sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1
-bios /usr/share/qemu/bios.bin
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 ",
it goes through full seabios routine, yet on vnc (note no -vga cirrus, but...).

Offline

#1889 2014-05-30 05:14:30

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gutleib wrote:
aw wrote:

You need the VGA arbiter patch from a page or two back

Could you explain? I don't have any intel graphics in my pc at all...

Also: I tried passthrugh'ing only "vga controller" half of gtx590, and also tried different pcie addresses. Results are:
apparently, if I use the line
"sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1
-bios /usr/share/qemu/bios.bin
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom 
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1
-device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on
-device vfio-pci,host=06:00.1,bus=root.1,addr=01.1
-vga cirrus",
the bios boot process doesn't complete. It goes first "seabios..." line in vnc, then the second, "ipxe..." then (in second line) "PnP PNN..." and lots of numbers. then it just sits there.
If I use only one pci device (the vga half, not the 3d controller),
"sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1
-bios /usr/share/qemu/bios.bin
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 ",
it goes through full seabios routine, yet on vnc (note no -vga cirrus, but...).

did you mean to use "-vga none"?

[s@localhost ~]$ man qemu-system-x86_64 | grep "cirrus" -4

       -vga type
           Select type of VGA card to emulate. Valid values for type are

           cirrus
               Cirrus Logic GD5446 Video card. All Windows versions starting from Windows 95 should recognize and use this graphic card. For optimal performances, use 16 bit color depth in the guest and the host OS.  (This one is the default)

           std Standard VGA card with Bochs VBE extensions.  If your guest OS supports the VESA 2.0 VBE extensions (e.g. Windows XP) and if you want to use high resolution modes (>= 1280x1024x16) then you should use this option.

[s@localhost ~]$

Offline

#1890 2014-05-30 10:26:35

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

> sinny

Dang! forgot it's default!

With vga none, vnc and display are off.

Offline

#1891 2014-05-30 12:16:26

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm having a problem applying the ACS Override patch in Gentoo, can anyone help me figure out what's up?

I copied it from the page starting at:

----
Documentation/kernel-parameters.txt |   10 +++
...

This is the output I get from portage when I emerge the gentoo-sources (that's gentoo's default kernel):

***** ACS_Override.patch 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_7.html topic_8.html topic_9.html PWD: /var/tmp/portage/sys-kernel/gentoo-sources-3.14.4/work/linux-3.14.4-gentoo ============================== PATCH COMMAND: patch -p0 -g0 -E --no-backup-if-mismatch < '/etc/portage/patches/sys-kernel/gentoo-sources/ACS_Override.patch' ============================== can't find file to patch at input line 9 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |--- | Documentation/kernel-parameters.txt | 10 +++ | drivers/pci/quirks.c | 102 +++++++++++++++++++++++++++++++++++ | 2 files changed, 112 insertions(+) |diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt |index 47bb23c..a60e6ad 100644 |--- a/Documentation/kernel-parameters.txt |+++ b/Documentation/kernel-parameters.txt -------------------------- No file to patch. Skipping patch. 1 out of 1 hunk ignored can't find file to patch at input line 30 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c |index 0369fb6..c7609f6 100644 |--- a/drivers/pci/quirks.c |+++ b/drivers/pci/quirks.c -------------------------- No file to patch. Skipping patch. patch unexpectedly ends in middle of line 1 out of 1 hunk ignored patch program exited with status 1

I have no idea what to do here, I'm a total noob when it comes to "patch" and "diff". I know it fails at line 9 and 30 (i.e. the "@@" lines in the patch) but how to fix it I have no clue. The Nvidia Proprietary driver patch applied just fine though.

The lines it's failing at are these:

9: @@ -2349,6 +2349,16 @@ bytes respectively. Such letter suffixes can also be entirely omitted. 30: @@ -3292,11 +3292,113 @@ struct pci_dev *pci_get_dma_source(struct pci_dev *dev)

Offline

#1892 2014-05-30 14:24:29

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
PATCH COMMAND: patch -p0 -g0 -E --no-backup-if-mismatch < '/etc/portage/patches/sys-kernel/gentoo-sources/ACS_Override.patch'

I have no idea what to do here, I'm a total noob when it comes to "patch" and "diff". I know it fails at line 9 and 30 (i.e. the "@@" lines in the patch) but how to fix it I have no clue. The Nvidia Proprietary driver patch applied just fine though.

Try it with -p1

Offline

#1893 2014-05-30 15:17:42

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

i can confirm the following setupd working:
MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20 x86_64 (with additional repositories: ...)
kernel: kernel-3.15.0-0.rc5.git2.8.vanilla.mainline.knurd.1.fc20.x86_64.rpm
host GPU: integrated (intel i7 4770)
guest GPU0: radeon hd 6450 (Gigabyte GV-R645-1GI)

Well that is good to know, thanks for the info. Apart from the OS the only thing massively different is my knowledge!

I am attempting to follow #1 and have a few (very basic) questions, suffice to say it is not working yet.

Question 1 All I did to get the linux-mainline was:

yaourt linux-mainline

followed by

mkinitcpio -p linux-mainline

should I have done anything else?

Question 2 I copied the vfio-bind script and used it to bind my gpu which was

vfio-bind 0000:01:00.0 0000:01:00.1

but after I start the systemd service I cannot see any output from the next command:

[cdrjameson@experiment ~]$ cat /etc/vfio-pci.cfg cat: /etc/vfio-pci.cfg: No such file or directory

I called my systemd service file vfiobind.service and there didn't look to be anything in the file that was specific or needed changing per user. Did I miss something?

Question 3 The next step: to test if working fails. I changed the qemu test invocation nbhs uses for my processor (4 cores instead of 6), gave more memory and changed the address of the vfio-pci.

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Are these sane defaults...?
The errors read:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

After I create the blank windows.img file I am able to start qemu by a very simple

 qemu-system-x86_64 windows.img

which then opens the VM, proving qemu is working...
I dont want to/cant go any further without fixing these fundamental issues.

Last edited by cdrjameson (2014-05-30 15:24:09)

Offline

#1894 2014-05-30 15:23:30

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey,
in #1 there are mentioned two links for linux-mainline. Use "linux-mainline.tar.gz (3.14.1 includes acs override patch, i935 vga arbiter fixes, debug registers patch)" right unter "Ingredients".
You need to build this kernel for yourself, but there are all required patches already included. If you have all required packages for building an arch package yourself installed, you just need to unzip this file, cd into the new directory and run "makepkg".

If you don't have the necessary packages installed, read this

Once the kernel is build you will find a package file in your build directory. You can install the kernel via

yaourt -U <package filename> 

Last edited by apex8 (2014-05-30 15:25:58)

Offline

#1895 2014-05-30 15:31:09

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hey,
in #1 there are mentioned two links for linux-mainline. Use "linux-mainline.tar.gz

That link is broken for me, I get to the download page but cant download it... so I used the second link which is the AUR package. Is there another location for the first link?

Offline

#1896 2014-05-30 15:33:27

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested it. The links isn't broken. Click on it and then on slow download.

Offline

#1897 2014-05-30 15:38:11

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've tried on two computers and after selecting slow download I just get 'invalid download link'. Using firefox and chromium. Did the link actually take you to the download page?

Offline

#1898 2014-05-30 15:41:37

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, it takes you to a download page. Could you try this link?

Offline

#1899 2014-05-30 15:47:09

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey Guys,

I finally managed to get my VM started and installed Windows 8.1, Passthrough works since I get the output from my GTX 770 and the device-manager also says that it uses a NVIDIA GTX 770.
But I got two problems, first, I can't install the Nvidia drivers, because it says the Windows version is incompatible and no compatible graphics hardware was found.. The driver IS for Win8.1 x64 which IS installed and like I said, the device-manager in Windows says, that there is a GTX 770 and it has NO yellow exclamation mark or something else, so I guess it's working but yeah, I have no idea how to fix this problem..

Second problem, I can't get the EFI-BIOS to work (OVMF).. I installed the version from the repository with pacman but I only get a black screen and nothing happens when I start the VM, then I tried to compile the AUR version of ovmf-svn but it doesn't want to compile so I gave up.. Tbh I don't really need EFI, but I want it so the Windows Bootlogo is in the full resolution, which is way sexier and I can use the UEFI features like secure boot. I tried all parameters like -boot -pflash and -L, it finds and uses the OVMF-Bios, but as I said, when I start the VM I only get a black screen and it doesn't boot anything.

I hope someone atleast can give me an answer to the problem with the nvidia gpu drivers.. everything works fine, except I can't install them.


And I got another little offtopic question, is there anything BESIDES Synergy to seamlessly integrate the VM with my desktop?
I don't want to pass my keyboard to my VM directly, because when there are problems and the VM is stuck for example, I'm pretty screwed. big_smile

Last edited by shawly (2014-05-30 15:49:59)

Offline

#1900 2014-05-30 15:53:43

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@shawly regarding synergy: It should be the normal case, that your VM works flawlessly. Whenever there is a serious problem, e.g. an bsod, you can go to the qemu console and enter system_reset; the vm will reboot. If this doesn't solve the problem you could prepare a second qemu script, which runs the vm in a window where you are able to use your keyboad and mouse.
This strategy works good enough for me, because like I said: the normal case is that the VM runs without any problems.

Offline

#1901 2014-05-30 15:54:35

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Apex8: Oh wow, I thought this might be a DNS issue so I've just installed a VPN and I get through first time. Stupid ISP. Sorry for the off topic posts.

Last edited by cdrjameson (2014-05-30 15:55:37)

Offline

#1902 2014-05-30 16:04:11

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
rabcor wrote:
PATCH COMMAND: patch -p0 -g0 -E --no-backup-if-mismatch < '/etc/portage/patches/sys-kernel/gentoo-sources/ACS_Override.patch'

I have no idea what to do here, I'm a total noob when it comes to "patch" and "diff". I know it fails at line 9 and 30 (i.e. the "@@" lines in the patch) but how to fix it I have no clue. The Nvidia Proprietary driver patch applied just fine though.

Try it with -p1

I highly doubt doing that would change anything (then again I am noob) but my understanding is that doing -p1 would make

a/drivers/pci/quirks.c

into

/drivers/pci/quirks.c

(but maybe -p2 might make a difference)

Either way, I decided to try and copy the file to my home folder and manually patch it.

I created 'a/drivers/pci/quirks.c' and then:

$ patch -p0 -g0 -E < 'ACS_Override.patch' patching file a/drivers/pci/quirks.c patch unexpectedly ends in middle of line Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file a/drivers/pci/quirks.c.rej

What do I do? I suspect this means that the patch is incompatible with my pci quirks.c file (made for a different version of it) does this mean that I have to somehow make a new patch file?

Edit: That was indeed it, posted my file below.

Last edited by rabcor (2014-05-30 16:28:32)

Offline

#1903 2014-05-30 16:16:04

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

@shawly regarding synergy: It should be the normal case, that your VM works flawlessly. Whenever there is a serious problem, e.g. an bsod, you can go to the qemu console and enter system_reset; the vm will reboot. If this doesn't solve the problem you could prepare a second qemu script, which runs the vm in a window where you are able to use your keyboad and mouse.
This strategy works good enough for me, because like I said: the normal case is that the VM runs without any problems.

Thanks for the answer, but this doesn't really answer my question, I just asked for options BESIDES synergy, because maybe there are better ways.

Anyone got an answer for my problem with efi or the Nvidia drivers under Windows? hmm

Offline

#1904 2014-05-30 16:18:25

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well you could get an 2:1 KVM-Switch and pass through one of your usb-controllers to the vm. Then you can switch between host-/vm access wich your keyboard and mouse.

Offline

#1905 2014-05-30 16:30:04

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is my Gentoo 3.14.4 (Not 100% sure if this is a Gentoo exclusive issue though, for all I know kernel 3.14.4 might have come with a different version of quirks.c than previous releases) compatible ACS Override patch (note: I did not bother adding in the documentation changes with it like in the original patch)

(command used: diff -Naur a/drivers/pci/quirks.c b/drivers/pci/quirks.c > ACS.patch)

--- a/drivers/pci/quirks.c 2014-05-30 16:08:19.561232936 +0000 +++ b/drivers/pci/quirks.c 2014-05-30 16:18:44.745241038 +0000 @@ -3423,6 +3423,107 @@ #endif } +static bool acs_on_downstream; +static bool acs_on_multifunction; + +#define NUM_ACS_IDS 16 +struct acs_on_id { + unsigned short vendor; + unsigned short device; +}; +static struct acs_on_id acs_on_ids[NUM_ACS_IDS]; +static u8 max_acs_id; + +static __init int pcie_acs_override_setup(char *p) +{ + if (!p) + return -EINVAL; + + while (*p) { + if (!strncmp(p, "downstream", 10)) + acs_on_downstream = true; + if (!strncmp(p, "multifunction", 13)) + acs_on_multifunction = true; + if (!strncmp(p, "id:", 3)) { + char opt[5]; + int ret; + long val; + + if (max_acs_id >= NUM_ACS_IDS - 1) { + pr_warn("Out of PCIe ACS override slots (%d)\n", + NUM_ACS_IDS); + goto next; + } + + p += 3; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].vendor = val; + + p += strcspn(p, ":"); + if (*p != ':') { + pr_warn("PCIe ACS invalid ID\n"); + goto next; + } + + p++; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].device = val; + max_acs_id++; + } +next: + p += strcspn(p, ","); + if (*p == ',') + p++; + } + + if (acs_on_downstream || acs_on_multifunction || max_acs_id) + pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n"); + + return 0; +} +early_param("pcie_acs_override", pcie_acs_override_setup); + +static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags) +{ + int i; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Never override ACS for legacy devices or devices with ACS caps */ + if (!pci_is_pcie(dev) || + pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS)) + return -ENOTTY; + + for (i = 0; i < max_acs_id; i++) + if (acs_on_ids[i].vendor == dev->vendor && + acs_on_ids[i].device == dev->device) + return 1; + + switch (pci_pcie_type(dev)) { + case PCI_EXP_TYPE_DOWNSTREAM: + case PCI_EXP_TYPE_ROOT_PORT: + if (acs_on_downstream) + return 1; + break; + case PCI_EXP_TYPE_ENDPOINT: + case PCI_EXP_TYPE_UPSTREAM: + case PCI_EXP_TYPE_LEG_END: + case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) + return 1; + } + + return -ENOTTY; +} + static const struct pci_dev_acs_enabled { u16 vendor; u16 device; @@ -3434,6 +3535,7 @@ { PCI_VENDOR_ID_ATI, 0x439d, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs }, + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } }; 

Compatible with gentoo kernel 3.14.4, not sure if it was gentoo only or if it was just a recent kernel update that changed the file, either way if anyone fails to run the original ACS patch try this instead.

Last edited by rabcor (2014-05-30 16:41:50)

Offline

#1906 2014-05-30 18:14:48

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Well you could get an 2:1 KVM-Switch and pass through one of your usb-controllers to the vm. Then you can switch between host-/vm access wich your keyboard and mouse.

Would be an option yeah, but I guess I'll pass the mouse and keyboard through and use Synergy to access my Gnome desktop, then I shouldn't get any delay while playing, since I can't imagine that it's possible to play games as Synergy-Client without any delay or other problems.

Anyway, I looked through this thread and it seems, that nobody has tried to get OVMF working since february.. Guess it isn't working at all sad

And I'll try to use an older version of the Nvidia driver, maybe this will work.

Last edited by shawly (2014-05-30 18:15:13)

Offline

#1907 2014-05-30 18:23:39

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@shawly: If it helps, I can tell you that I was able to run bioshock infinite without any input delay. I just configured a hotkey for locking the mouse/keyboard in the VM and activated relative mouse moves.

Offline

#1908 2014-05-30 18:30:07

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

@shawly: If it helps, I can tell you that I was able to run bioshock infinite without any input delay. I just configured a hotkey for locking the mouse/keyboard in the VM and activated relative mouse moves.

Sounds promising! So you didn't pass the mouse and keyboard to the guest? How did you do that?

Offline

#1909 2014-05-30 18:36:31

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using my host as synergy server and the vm is just a client. Synergy can be configured via script or via its GUI. The latter is pretty intuitive; maybe have a look at YouTube for this. Furthermore a passed through the usb3 controller and where an gamepad is plugged in. So the gamepad is only seen by the vm and works entirely without delay. But the delay isn't the problem, as I said.

Offline

#1910 2014-05-30 20:19:36

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

[…]Anyway, I looked through this thread and it seems, that nobody has tried to get OVMF working since february.. Guess it isn't working at all sad[…]

I don't know about your setup, but OVMF works for me (see here for my HW). I just tested a VM with arch as guest and host and the open-source radeon drivers in the guest. This is the command I used

/usr/bin/qemu-system-x86_64 -enable-kvm -vga none -nographic -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 4096 -mem-path /dev/hugepages -device vfio-pci,host=06:00.0,addr=10.0,multifunction=on,x-vga=on -device vfio-pci,host=06:00.1,addr=10.1 -device vfio-pci,host=0000:00:13.0 -device vfio-pci,host=0000:00:13.2 -usb -device usb-host,vendorid=0x03f0,productid=0x0024 -netdev bridge,br=br0,id=hostnet0 -net nic,model=virtio,netdev=hostnet0 -drive file=arch_uefi.img,id=drive0,if=virtio,media=disk,aio=native -monitor stdio -boot menu=on -pflash /usr/share/ovmf/ovmf_x64.bin

My software versions are extra/ovmf 15280-1, local/qemu-git 2.1.r32448.gfdaad47-1 and linux 3.15.0-rc6-mainline+


i'm sorry for my poor english wirting skills…

Offline

#1911 2014-05-30 20:28:48

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

I don't know about your setup, but OVMF works for me (see here for my HW). I just tested a VM with arch as guest and host and the open-source radeon drivers in the guest.

My software versions are extra/ovmf 15280-1, local/qemu-git 2.1.r32448.gfdaad47-1 and linux 3.15.0-rc6-mainline+

I'm using the same version of ovmf, qemu-git 2.1 but the linux 3.14.4 kernel. Do you think it'll make any difference if I switch to 3.15? Just tried it again, blackscreen..
My setup is at the bottom of this document, sombody added it already and I added some things.

Edit: Uhm, well, the problem solved itself... I don't know how, and I don't know why but suddenly it just boots when I use the -L parameter...
AND! I'm an idiot, I downloaded the 32bit Nvidia drivers, since it's freaking hard to navigate through a website without a mouse in the VM it must have chosen 32 bit instead of 64 and I didn't saw that.. damn I feel really stupid xD

Edit2: Ok it didn't work with ovmf, it just booted from SeaBios with the -L parameter.. god knows why
Just tried with the -pflash parameter again, but had no luck.. I have no idea what causes this. When I use SeaBIOS my monitor turns on and it boots, but with ovmf it doesn't turn on and if I use qemu to test if it's working it's just black.

Edit3: I gave up on ovmf, now I installed the driver for the nvidia card but sadly, I get the error code 43 in the device manager...
I used the package in the first post, therefore I got all the patches that come with this package.. Was this a mistake?

Last edited by shawly (2014-05-30 22:01:37)

Offline

#1912 2014-05-30 23:00:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

Edit3: I gave up on ovmf, now I installed the driver for the nvidia card but sadly, I get the error code 43 in the device manager...
I used the package in the first post, therefore I got all the patches that come with this package.. Was this a mistake?

Nvidia drivers newer than 335.23 are broken


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1913 2014-05-30 23:24:39

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Nvidia drivers newer than 335.23 are broken

Good to know, that means 335.23 should work, will try that.

Offline

#1914 2014-05-31 03:30:45

esmth
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, I got it (mostly) working  with this command:

/home/esmth/bin/vfio-bind 0000:01:00.1 0000:01:00.0 /home/esmth/src/qemu/x86_64-softmmu/qemu-system-x86_64 \ -M q35 -drive file=/home/esmth/image.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -enable-kvm -m 4096 -cpu host \ -bios /home/esmth/src/seabios/out/bios.bin -vga none \ -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -usb -usbdevice host:046d:c52b -usbdevice host:0d62:001d -usbdevice host:0781:5530 \ -nographic -boot menu=on

I'm running kernel 3.15-rc6.

the only problem  I'm having right now is that I get tons of  this:

May 30 23:13:20 plebeian kernel: [ 2525.114596] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb080 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.114831] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb2c0 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.115066] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb100 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.115306] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb5c0 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.115543] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb1c0 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.115778] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb540 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.116012] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb280 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.116248] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb180 flags=0x0010] May 30 23:13:20 plebeian kernel: [ 2525.116485] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 address=0x0000000020fdb340 flags=0x0010]

when trying to install the gpu drivers in the guest. The guest restarts and the drivers can't be Installed in windows. If its  any use, I have an r9-290x as the dGPU and i'm using  my  7850k's intergrated graphics  for the host.



Edit: I installed the gpu drivers using the -vga std option, but now the guest crashes after the windows bootup logo. I removed it in safe mode and i'm back where i started at the beginning of this post.

Last edited by esmth (2014-05-31 03:45:28)

Offline

#1915 2014-05-31 07:01:32

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I'm pretty close to getting this working now. I have the ACS Override patch in my kernel for my intel i7 2600 CPU and I have the VGA_ARB patch for my nvidia proprietary drivers set up as well

(Host: GTX 550-Ti
Guest: GTX 670)

When I try to test my machine with (should be the same as in the OP but with 02 instead of 07 as device host)

I get the following errors (running as root, if I run as user I get the same last two lines, but the first two will be "vfio: error opening /dev/vfio/6: Permission denied" and "vfio: failed to get group 6")

#qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=8,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 6 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 6 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

More info:

#lspci 01:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) #lspci -n 01:00.0 0300: 10de:1244 (rev a1) 01:00.1 0403: 10de:0bee (rev a1) 02:00.0 0300: 10de:1189 (rev a1) 02:00.1 0403: 10de:0e0a (rev a1) #dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/boot/kernel-3.14.4-Gentoo root=/dev/sda1 ro quiet pci-stub.ids=10de:1189,10de:0e0a [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/kernel-3.14.4-Gentoo root=/dev/sda1 ro quiet pci-stub.ids=10de:1189,10de:0e0a [ 0.273833] pci-stub: add 10DE:1189 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.273839] pci-stub 0000:02:00.0: claimed by stub [ 0.273841] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.273846] pci-stub 0000:02:00.1: claimed by stub [ 1.446046] pci-stub 0000:02:00.0: enabling device (0000 -> 0003) # cat /etc/modprobe.d/kvm.conf options vfio_iommu_type1 allow_unsafe_interrupts=1 options kvm ignore_msrs=1 #zgrep VFIO /proc/config.gz CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y

I think with this everything should be working, I do not run vfio-bind on bootup though (haven't figured out how to do it with OpenRC which is what gentoo uses) but I run it as root before I try to launch the VM.

#vfio-bind 0000:02:00.0 0000:02:00.1

(It shows no output, I assume this is normal)

How do I fix this problem?

Edit: Solved!

Solution was to add

pcie_acs_override=downstream

To the command line (basically activate the ACS override patch, silly me...)

Oh man that feels so good, it finally works smile

Now I am wondering if this can be applied to Xen HVMs? Has anyone tried? Also I still get this when I run as user, how do I fix that?

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Permission denied qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13

Last edited by rabcor (2014-05-31 08:25:10)

Offline

#1916 2014-05-31 11:07:47

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

esmth wrote:

when trying to install the gpu drivers in the guest. The guest restarts and the drivers can't be Installed in windows. If its  any use, I have an r9-290x as the dGPU and i'm using  my  7850k's intergrated graphics  for the host.

Edit: I installed the gpu drivers using the -vga std option, but now the guest crashes after the windows bootup logo. I removed it in safe mode and i'm back where i started at the beginning of this post.

It looks like you suffer from the "no proper reset method is known for R9 290X series" like others including me. smile What I did to get it working is the following:

#1 Install Windows 7 using -vga std AND -no-reboot
#2 Install Synergy on host and inside VM and configure it
#3 Assign VGA device via vfio and use -vga none AND keep -no-reboot. After that I get already VGA post bios screen and Windows 7 boots with VGA resolution
#4 Login to Windows 7 and install Catalyst driver (latest stable)
#5 If it wants to reboot let it do and QEMU will stop and shutdown due to '-no-reboot' switch
#6 Reboot the host (or do the suspend/resume cycle as workaround)
#7 Start VM

With R9 290X series you will not get VM reboot working (for now) and that's why I keep the '-no-reboot' option always and if VM is rebooted I have to make short suspend/resume cycle.

Offline

#1917 2014-06-01 03:45:41

esmth
Member
Registered: 2014-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

It looks like you suffer from the "no proper reset method is known for R9 290X series" like others including me. smile What I did to get it working is the following:

#1 Install Windows 7 using -vga std AND -no-reboot
#2 Install Synergy on host and inside VM and configure it
#3 Assign VGA device via vfio and use -vga none AND keep -no-reboot. After that I get already VGA post bios screen and Windows 7 boots with VGA resolution
#4 Login to Windows 7 and install Catalyst driver (latest stable)
#5 If it wants to reboot let it do and QEMU will stop and shutdown due to '-no-reboot' switch
#6 Reboot the host (or do the suspend/resume cycle as workaround)
#7 Start VM

Thanks, I'll try  this tomorrow morning. I also  have a 550ti and a HD6870 that i can test with. Do those older GPUs not have these issues?

edit: is Windows 7 required? I've been trying to use windows 8.

Last edited by esmth (2014-06-01 03:47:11)

Offline

#1918 2014-06-01 09:20:58

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wonder this as well, is windows 7 the only supported version? or is it possible to use windows 8 instead? I'm thinking this since despite hating Windows 8 so much that I fled to Linux land when I saw it coming, it is hard to deny that it consumes less resources than the 7 which makes it more ideal for VMs (I mean the more left over resources for the host, the better right?)

Anyhow I have a slight problem. When I run qemu as a user I get this message

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Permission denied qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

And I absolutely do not want to be running qemu as root even if I "can"

How do I give my user the permissions to do this? I'm running Gentoo right now but it can't be that much different from Arch smile (Biggest difference I noticed as an end user is OpenRC rather than Systemd, something I chose to live with rather than fight.)

PS: I have a physical 3TB GPT partition that i want to mount on the host and share with the VM, what is the safest way for me to share it?

Offline

#1919 2014-06-01 11:19:36

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

esmth wrote:

Thanks, I'll try  this tomorrow morning. I also  have a 550ti and a HD6870 that i can test with. Do those older GPUs not have these issues?

edit: is Windows 7 required? I've been trying to use windows 8.

I've also tried Windows 8.1 but installed Catalyst driver manual because that one from Windows update freezes my VM always during installation. So I downloaded it in advance and installed it after assigning the R9 290X. AFAIR the 550 TI should work fine without having reboot issues of VM. I would expect the same for the HD6870 as my 7870 GHz edition also works fine without reset issues.

Offline

#1920 2014-06-01 12:09:02

syndtr
Member
Registered: 2014-05-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

diff --git a/target-i386/kvm.c b/target-i386/kvm.c index 4389959..e6ee444 100644 --- a/target-i386/kvm.c +++ b/target-i386/kvm.c @@ -527,17 +527,17 @@ int kvm_arch_init_vcpu(CPUState *cs) has_msr_hv_hypercall = true; } - memcpy(signature, "KVMKVMKVM\0\0\0", 12); - c = &cpuid_data.entries[cpuid_i++]; - c->function = KVM_CPUID_SIGNATURE | kvm_base; - c->eax = 0; - c->ebx = signature[0]; - c->ecx = signature[1]; - c->edx = signature[2]; - - c = &cpuid_data.entries[cpuid_i++]; - c->function = KVM_CPUID_FEATURES | kvm_base; - c->eax = env->features[FEAT_KVM]; + // memcpy(signature, "KVMKVMKVM\0\0\0", 12); + // c = &cpuid_data.entries[cpuid_i++]; + // c->function = KVM_CPUID_SIGNATURE | kvm_base; + // c->eax = 0; + // c->ebx = signature[0]; + // c->ecx = signature[1]; + // c->edx = signature[2]; + + // c = &cpuid_data.entries[cpuid_i++]; + // c->function = KVM_CPUID_FEATURES | kvm_base; + // c->eax = env->features[FEAT_KVM]; has_msr_async_pf_en = c->eax & (1 << KVM_FEATURE_ASYNC_PF); 

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Last edited by syndtr (2014-06-01 12:16:27)

Offline

#1921 2014-06-01 12:56:29

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fucking Nvidia trying to force GRID upon people... That'll teach em smile should be added to the OP for nvidia workarounds. But you could have mentioned that it's supposed to be applied to qemu.

Anyways, apart from my error 3 posts up, I have another error when trying to mount a physical partition to qemu. Using:

-drive file=/dev/sdc,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk1

I get:

qemu-system-x86_64: -device ide-hd,bus=ahci.0,drive=disk: Bus 'ahci.0' not found

Last edited by rabcor (2014-06-01 13:19:26)

Offline

#1922 2014-06-01 14:01:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

diff --git a/target-i386/kvm.c b/target-i386/kvm.c index 4389959..e6ee444 100644 --- a/target-i386/kvm.c +++ b/target-i386/kvm.c @@ -527,17 +527,17 @@ int kvm_arch_init_vcpu(CPUState *cs) has_msr_hv_hypercall = true; } - memcpy(signature, "KVMKVMKVM\0\0\0", 12); - c = &cpuid_data.entries[cpuid_i++]; - c->function = KVM_CPUID_SIGNATURE | kvm_base; - c->eax = 0; - c->ebx = signature[0]; - c->ecx = signature[1]; - c->edx = signature[2]; - - c = &cpuid_data.entries[cpuid_i++]; - c->function = KVM_CPUID_FEATURES | kvm_base; - c->eax = env->features[FEAT_KVM]; + // memcpy(signature, "KVMKVMKVM\0\0\0", 12); + // c = &cpuid_data.entries[cpuid_i++]; + // c->function = KVM_CPUID_SIGNATURE | kvm_base; + // c->eax = 0; + // c->ebx = signature[0]; + // c->ecx = signature[1]; + // c->edx = signature[2]; + + // c = &cpuid_data.entries[cpuid_i++]; + // c->function = KVM_CPUID_FEATURES | kvm_base; + // c->eax = env->features[FEAT_KVM]; has_msr_async_pf_en = c->eax & (1 << KVM_FEATURE_ASYNC_PF); 

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1923 2014-06-01 14:37:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

--- a/target-i386/kvm.c +++ b/target-i386/kvm.c @@ -528,7 +528,7 @@ int kvm_arch_init_vcpu(CPUState *cs) has_msr_hv_hypercall = true; } - memcpy(signature, "KVMKVMKVM\0\0\0", 12); + memcpy(signature, "kvmkvmkvm\0\0\0", 12); c = &cpuid_data.entries[cpuid_i++]; c->function = KVM_CPUID_SIGNATURE | kvm_base; c->eax = 0;

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1924 2014-06-01 14:37:53

syndtr
Member
Registered: 2014-05-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

But you could have mentioned that it's supposed to be applied to qemu.

Ahh, sorry.. I forgot that.

aw wrote:

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.

I can't believe it either.

aw wrote:

I suppose there's still a chance that it was an oversight and not an attempt to be malicious.

I hope that's true.

aw wrote:

Maybe we can make a runtime option for qemu to hide the hypervisor.

Great idea.

Anyway, on linux guest patching nvidia kmod by making os_is_vgx_hyper to return true also did the trick.

diff --git a/os-interface.c b/os-interface.c index 13cd3b6..79629df 100644 --- a/os-interface.c +++ b/os-interface.c @@ -1247,11 +1247,7 @@ NvBool NV_API_CALL os_is_xen_dom0(void) NvBool NV_API_CALL os_is_vgx_hyper(void) { -#if defined(NV_VGX_HYPER) return TRUE; -#else - return FALSE; -#endif } void NV_API_CALL os_bug_check(NvU32 bugCode, const char *bugCodeStr)

Offline

#1925 2014-06-01 15:14:21

syndtr
Member
Registered: 2014-05-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

--- a/target-i386/kvm.c +++ b/target-i386/kvm.c @@ -528,7 +528,7 @@ int kvm_arch_init_vcpu(CPUState *cs) has_msr_hv_hypercall = true; } - memcpy(signature, "KVMKVMKVM\0\0\0", 12); + memcpy(signature, "kvmkvmkvm\0\0\0", 12); c = &cpuid_data.entries[cpuid_i++]; c->function = KVM_CPUID_SIGNATURE | kvm_base; c->eax = 0;

You're right, it's work just by changing the signature.

Offline

#1926 2014-06-01 15:32:09

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally figured out why OVMF isn't booting, it's because of q35!
Question is, how do I get it working with q35? Do I even need q35, do I have any advantages with q35 over i440fx? (sorry I'm quite new to qemu and I'm like Jon Snow.. I know nothing.)

Last edited by shawly (2014-06-01 15:32:54)

Offline

#1927 2014-06-01 16:10:26

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Quoting OP:

NOTE: -m q35 | Enable the new q35 emulated chipset, qemu defaults to the old i440fx chipset.
You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,

Also, not sure how this is better but I posted the vfio-bind script elsewhere and someone told me I should improve it by making it like so:

#!/bin/sh modprobe vfio-pci for dev in "$@"; do read -r vendor </sys/bus/pci/devices/$dev/vendor read -r device </sys/bus/pci/devices/$dev/device if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Last edited by rabcor (2014-06-01 16:11:19)

Offline

#1928 2014-06-01 20:53:02

djbdayong
Member
Registered: 2014-05-01
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am yet another lurker that has found this thread invaluable in getting my setup working, a big thank you to nbhs, aw and indeed everyone!

Some details should they be useful to someone:

Motherboard: ASRock Fatal1ty z87 Killer
CPU: Intel Xeon E3-1245v3
Passthrough GPU: XFX Radeon R9 270X
Kernel: 3.14.4
Qemu: 2.0.0
Guest Firmware: Seabios, 1.7.5
Guest OS: Windows 8

Main motivations, aside from curiosity, were running my preferred OpenCL accelerated RAW converter and photo processing software without sacrificing the Linux desktop with dual boot. As well as some light gaming. For these purposes the setup runs wonderfully.

I installed more or less as per OP instructions and it went smoothly thanks to all of you trail blazers on this thread (Excuse any ignorance, I am still far from grasping the finer details of the moving parts):

1. Blacklisted passed GPU drivers on the host since it is only used when I run the guest.
2. Reproduced the colourful VGA arbitration issue and applying the OP i915 patch immediately solved it.
3. Was able to install virtio guest drivers during Windows 8 setup. Had to omit the OP qemu virtio controller for Windows to match the Red Hat drivers to the unknown virtio disk device. See the qemu command below for what I ended up with.
4. Windows 8 activated without any problems (wondering how sensitive it will be to future qemu/seabios upgrades ???)
5. Since I have all of my audio equipment connected to my host sound device + remote PA server and due to accounts of BSOD due to HDMI audio funny business, I decided to just install the AMD video driver and avoid as much extra Catalyst fluff as possible. So I did a custom install with HDMI audio device not passed to the guest. This went smoothly and no BSOD on reboot. For guest sound, I use an emulated sound device and qemu PA driver as per OP.
6. Installed synergy for keyboard/mouse sharing.

Qemu command line:

qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -cpu host \ -m 12288 \ -smp 6,sockets=1,cores=3,threads=2 \ -bios ${PATH_BIOS} \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive if=virtio,id=disk,file=${PATH_DISK_IMAGE},format=raw \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I'm happy to test new patches and otherwise provide more info.

Offline

#1929 2014-06-01 22:29:24

thehighhat
Member
Registered: 2014-05-12
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I had a working vga passthrough of nvidia gtx470 with a windows 8 guest.  I let the windows store upgrade to windows 8.1 update.  now, the guest will not start correct.  it always goes into automatic system repair and fails.  the refresh pc, reset pc options ask for windwos cd, but cannot detect it.  if i just let windows try to start, it shows nohing but a black screen.  then i have reboot the host becaues i cannot shutdown the guest cleanly.

any ideas why the 8.1 update breaks everything?

Offline

#1930 2014-06-02 05:12:22

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I do

sudo chown rabcor /dev/vfio/13

In a desperate attempt to run the VM as user, I get a new error

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f4b3a2cced0, 0x0, 0x80000000, 0x7f4a08000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f4b3a2cced0, 0x100000000, 0x80000000, 0x7f4a88000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I tried running

#ulimit -l 10485760

As root to allow the user to allocate that amount of memory to the VM as suggested somewhere on the net, but it had no effect on the error sad

Do I really just have to run it as root?

PS: Still can't solve that qemu is not finding the ahci.0 bus

#lspci | grep SATA 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]

Last edited by rabcor (2014-06-02 05:15:36)

Offline

#1931 2014-06-02 05:34:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1932 2014-06-02 06:38:19

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, but still not working

chown rabcor /dev/vfio/13 chmod 660 /dev/vfio/vfio chown rabcor /dev/vfio/vfio ulimit -l 8409088

Note: I had to do chown user on vfio because it denied me permission after running chmod 660 on it.

Ran ulimit -l 8409088 for 8gigs ((8192 + 20) 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_7.html topic_8.html topic_9.html 1024 as the pdf you linked suggested)

Still get

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x0, 0x80000000, 0x7fe750000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x100000000, 0x80000000, 0x7fe7d0000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Last edited by rabcor (2014-06-02 06:38:52)

Offline

#1933 2014-06-02 08:32:57

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nvidia has filed a bug now regarding the hypervisor check.
https://forums.geforce.com/default/topi … 26-14-/36/

Hopefully they will fix it, I have some doubts though!

Offline

#1934 2014-06-02 10:01:26

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

At least they know now that they're going to have to pick a side, with us or against us, and they know siding with their users is only going to make them look more cool.

From their perspective they should try to see that no workarounds (like the VGA_ARB patch) are needed to make this just work on AMD hardware (hell xen has been capable of it for a long time but when it comes to nvidia cards it was always a pain to make them work there) meaning this is a feature AMD is already offering and not fighting against like they are.

I guess we'll see in one of the next few nvidia driver updates what their reply is (i.e. they will either fix it or they will do something else to make our lives miserable)

Offline

#1935 2014-06-02 10:40:20

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hello there

amazin and long topic smile

hope someone can help me...

i got some problems... yesterday i managed to get passthrough running but performance was low... (DOTA2 native 120fps, in kvm 40-50fps, drops to 20fps sometimes jumps to 90fps

my config
i72600
Z77X-UD3H
host igpu
guest 560ti
jessie 13.4.4 + i915patch, acs and debug register


i am also interested in getting SLI running since i have 2 same cards, and is it possible to pass IGPU? so i can have hw video (quicksync) encode since 5xx series lack that and i can't stream 60fps



P.S.
what are proper patches,grub commands and kernel needed for my system?
do i need pci-stub, and vfio-bind, both?
can i pass it with pci-assign?

i can only reboot vm, after shutdown i must reboot host? is that normal? 
Thank you

Last edited by slis (2014-06-02 13:20:34)

Offline

#1936 2014-06-02 12:57:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Thanks, but still not working

chown rabcor /dev/vfio/13 chmod 660 /dev/vfio/vfio chown rabcor /dev/vfio/vfio ulimit -l 8409088

Note: I had to do chown user on vfio because it denied me permission after running chmod 660 on it.

Ran ulimit -l 8409088 for 8gigs ((8192 + 20) 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_7.html topic_8.html topic_9.html 1024 as the pdf you linked suggested)

Still get

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x0, 0x80000000, 0x7fe750000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x100000000, 0x80000000, 0x7fe7d0000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

As noted in the slides, an additional 20MB was sufficient for that example config, with an assigned NIC that has much, much less MMIO space.  If you want to make sure things are covered, add an extra 1G.  Also make sure you're setting ulimit in a way that transfers to the user process, 'sudo ulimit ...' is not sufficient.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1937 2014-06-02 15:58:33

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

I finally figured out why OVMF isn't booting, it's because of q35!
Question is, how do I get it working with q35? Do I even need q35, do I have any advantages with q35 over i440fx? (sorry I'm quite new to qemu and I'm like Jon Snow.. I know nothing.)

I changed my virtual machine configuration to q35 (had to change everything including pci, usb bus configs). Still OVMF does not come up to the monitor, thanks to you, i am now using seabios and q35 :-)

Offline

#1938 2014-06-02 18:09:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding Nvidia 337.88:

https://forums.geforce.com/default/topi … 3/#4232923

We fixed some hypervisor detection code that was breaking Hyper-V. It's possible that fix may be preventing GeForce cards from working in passthrough, but because it is not officially supported for GeForce cards, this will not be fixed.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1939 2014-06-02 18:28:22

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Urks... hmm

Offline

#1940 2014-06-02 19:49:03

daxcore
Member
Registered: 2014-05-19
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

--- a/target-i386/kvm.c +++ b/target-i386/kvm.c @@ -528,7 +528,7 @@ int kvm_arch_init_vcpu(CPUState *cs) has_msr_hv_hypercall = true; } - memcpy(signature, "KVMKVMKVM\0\0\0", 12); + memcpy(signature, "kvmkvmkvm\0\0\0", 12); c = &cpuid_data.entries[cpuid_i++]; c->function = KVM_CPUID_SIGNATURE | kvm_base; c->eax = 0;

Thanks, working great!

Offline

#1941 2014-06-03 02:25:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mbroemme wrote:

Alex, so anymore ideas to my lcpsi output at post #1849 and #1850?

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.

mbroemme, ok, here's an idea.  I think your non-working R9 290X reports NoSoftRst-, right?  IOW, it does claim to support a PM reset.  Maybe we shouldn't be trusting that PM reset is not horribly broken on this device.  We can make a device specific reset that will supersede and invalidate the standard reset mechanisms, leaving us with only bus reset.  Let's try something like this:

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index e729206..946f8d8 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3295,6 +3295,12 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, return 0; } +static int reset_no_reset(struct pci_dev *dev, int probe) +{ + printk("%s(%s, %d)\n", __func__, dev_name(&dev->dev), probe); + return -EINVAL; +} + #define PCI_DEVICE_ID_INTEL_82599_SFP_VF 0x10ed #define PCI_DEVICE_ID_INTEL_IVB_M_VGA 0x0156 #define PCI_DEVICE_ID_INTEL_IVB_M2_VGA 0x0166 @@ -3310,6 +3316,8 @@ static const struct pci_dev_reset_methods pci_dev_reset_me reset_intel_generic_dev }, { PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID, reset_chelsio_generic_dev }, + { PCI_VENDOR_ID_AMD, $GPU_DEVICE_ID, reset_no_reset }, + { PCI_VENDOR_ID_AMD, $AUDIO_DEVICE_ID, reset_no_reset }, { 0 } }; 

Replace the $GPU_DEVICE_ID and $AUDIO_DEVICE_ID with the ones for your R9 290X in standard hex notation.  Let us know what happens.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1942 2014-06-03 09:57:59

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
mbroemme wrote:

Alex, so anymore ideas to my lcpsi output at post #1849 and #1850?

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.

mbroemme, ok, here's an idea.  I think your non-working R9 290X reports NoSoftRst-, right?  IOW, it does claim to support a PM reset.  Maybe we shouldn't be trusting that PM reset is not horribly broken on this device.  We can make a device specific reset that will supersede and invalidate the standard reset mechanisms, leaving us with only bus reset.  Let's try something like this:

Unfortunately no.

 Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Offline

#1943 2014-06-03 10:14:02

sonnet
Member
Registered: 2014-01-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have one monitor with only a single dvi-d connection (no vga, no hdmi, no second dvi-d).
I have 2 vga though (integrated hd 4600 which i intended to use as main vga, and a radeon r9 270 to use for the vm)
Also my cpu and motherboard both support vt-d extension.

Can I take advantage of vga-passthrough for my vm, without attaching the second vga (R9 270) to the monitor? (since having only one connection I have no other choice).

I'm asking because I saw this video:
[YOUTUBE]Qi1LdFkRzIs[/YOUTUBE]

You can see the guy is running the vm in windowed mode, with vga passthrough enabled. Being the vm in windowed mode (rather than controlling the whole monitor)
makes me think like only is main vga (the vga that is used by the host) is controlling and attached to the monitor.

Can please anyone clarify?

Offline

#1944 2014-06-03 14:41:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sonnet wrote:

I have one monitor with only a single dvi-d connection (no vga, no hdmi, no second dvi-d).
I have 2 vga though (integrated hd 4600 which i intended to use as main vga, and a radeon r9 270 to use for the vm)
Also my cpu and motherboard both support vt-d extension.

Can I take advantage of vga-passthrough for my vm, without attaching the second vga (R9 270) to the monitor? (since having only one connection I have no other choice).

I'm asking because I saw this video:
[YOUTUBE]Qi1LdFkRzIs[/YOUTUBE]

You can see the guy is running the vm in windowed mode, with vga passthrough enabled. Being the vm in windowed mode (rather than controlling the whole monitor)
makes me think like only is main vga (the vga that is used by the host) is controlling and attached to the monitor.

Can please anyone clarify?

The video is using a VNC server running inside the guest and a VNC client on the desktop.  You have all the same remote graphics tricks at your disposal with VGA assignment as you do with physically separate systems.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1945 2014-06-03 14:41:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
aw wrote:
aw wrote:

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.

mbroemme, ok, here's an idea.  I think your non-working R9 290X reports NoSoftRst-, right?  IOW, it does claim to support a PM reset.  Maybe we shouldn't be trusting that PM reset is not horribly broken on this device.  We can make a device specific reset that will supersede and invalidate the standard reset mechanisms, leaving us with only bus reset.  Let's try something like this:

Unfortunately no.

 Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

/me goes back to beating head against desk


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1946 2014-06-03 15:28:14

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

/me goes back to beating head against desk

aw, this card is a "real" beast. smile All my other AMD cards are working fine (HD 7870 GHz, HD 5450 and HD 6670), but as time comes we might find a solution which will be helpful for other cards too I believe.

Last edited by mbroemme (2014-06-03 22:01:51)

Offline

#1947 2014-06-03 16:21:42

sonnet
Member
Registered: 2014-01-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The video is using a VNC server running inside the guest and a VNC client on the desktop.  You have all the same remote graphics tricks at your disposal with VGA assignment as you do with physically separate systems.

Thanks for your reply.
So do I need to have both the vga connected to the monitors or this escamotage with VNC would allow me to not connect the guest os card to the monitor?
Because my monitor allows only one connection (so there's no way I can attach 2 vga to the monitor)

Last edited by sonnet (2014-06-03 17:38:58)

Offline

#1948 2014-06-03 18:22:10

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@sonnet
You only need the physical monitor connection to the vm for the vnc server setup in the vm. Once this is done you can connect to the vnc desktop from your host using a vnc client.

Offline

#1949 2014-06-03 18:24:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Or you can setup everything with emulated VGA and switch over to the assigned VGA only after you have VNC access


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1950 2014-06-03 20:34:16

sonnet
Member
Registered: 2014-01-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks guys for your advices!
Does anybody have an idea about the overhead (in partiular % of cpu resources, considering I have a quadcore haswell) caused by this method with vnc?

Last edited by sonnet (2014-06-03 20:36:09)

Offline

#1951 2014-06-04 01:30:24

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

hello there

amazin and long topic smile

hope someone can help me...

i got some problems... yesterday i managed to get passthrough running but performance was low... (DOTA2 native 120fps, in kvm 40-50fps, drops to 20fps sometimes jumps to 90fps

my config
i72600
Z77X-UD3H
host igpu
guest 560ti
jessie 13.4.4 + i915patch, acs and debug register


i am also interested in getting SLI running since i have 2 same cards, and is it possible to pass IGPU? so i can have hw video (quicksync) encode since 5xx series lack that and i can't stream 60fps



P.S.
what are proper patches,grub commands and kernel needed for my system?
do i need pci-stub, and vfio-bind, both?
can i pass it with pci-assign?

i can only reboot vm, after shutdown i must reboot host? is that normal? 
Thank you

Strange no one replied sooner tongue

Your setup is nearly identical to mine, same processor, different motherboard, but I have 550-Ti running on host (670 on guest). I don't know about SLI, but in theory if you have two GTX 560s and pass them both through then in theory you should be able to make them work in SLI but you need to leave one card for your host... Also you are most definitely going to need this patch for SLI to have a chance to work (makes nvidia drivers think you're not running a virtual machine) there's a different version of it if you scroll down further but I think yuo should use this one as it's a more surefire way than the other. (You need this if you're using the latest nvidia drivers anyways)

40-50 FPS in dota 2 is bad.  Sounds like you need the debug register patch but you said you have it. Why would you passthrough the igpu? I never managed to get good recordings in windows with any software, but on linux ffmpeg recording and encoding simultaneously was perfect and ffmpeg works in windows too. Learn to use that and you should be good without the igpu (use x264 or x265 format, alternatively vp9) I don't know if iGPU can be passthroughed, but I think even if it is possible it is a bad idea. If you don't see it as "VGA" in lspci don't even bother trying. But iGPUs work very differently from dedicated ones as they rely on your processor (i.e. they need direct access to your physical processor, this not only creates a lot of vulnerabilities but also might not be possible depending on how the system is structured)

Just download the linux-mainline.tar.gz from the OP and apply all the damned patches. You will also need tha VGA_ARB patch for the nvidia proprietary drivers if you have an nvidia card on your host. and don't forget the qemu patch I linked you earlier. Unless you have a card from the same manufacturer as your guest on your host you do not need pci-stub.

aw wrote:
rabcor wrote:

Thanks, but still not working

chown rabcor /dev/vfio/13 chmod 660 /dev/vfio/vfio chown rabcor /dev/vfio/vfio ulimit -l 8409088

Note: I had to do chown user on vfio because it denied me permission after running chmod 660 on it.

Ran ulimit -l 8409088 for 8gigs ((8192 + 20) 1.sh topic_10.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_7.html topic_8.html topic_9.html 1024 as the pdf you linked suggested)

Still get

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x0, 0x80000000, 0x7fe750000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x100000000, 0x80000000, 0x7fe7d0000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

As noted in the slides, an additional 20MB was sufficient for that example config, with an assigned NIC that has much, much less MMIO space.  If you want to make sure things are covered, add an extra 1G.  Also make sure you're setting ulimit in a way that transfers to the user process, 'sudo ulimit ...' is not sufficient.

Thanks! This worked, I think the problem may have been that I needed to use su, run the ulimit command then run su again to go back into my own terminal rather than just type exit. The problem now would be that I will want to use this with the -runas function in qemu, how can I combine them?

Edit: Oh this was fairly easy, just use the /etc/security/limits.conf and do

@kvm soft memlock 8650752 @kvm hard memlock 9437184

soft lock on (8192+256)*1024, hard lock(roof) on (8192+1024)*1024 smile now it's set upon boot and I can use the asroot function (I hope)

Last edited by rabcor (2014-06-12 01:12:38)

Offline

#1952 2014-06-04 04:47:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sonnet wrote:

Thanks guys for your advices!
Does anybody have an idea about the overhead (in partiular % of cpu resources, considering I have a quadcore haswell) caused by this method with vnc?

Lots


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1953 2014-06-04 05:37:49

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

--- a/target-i386/kvm.c +++ b/target-i386/kvm.c @@ -528,7 +528,7 @@ int kvm_arch_init_vcpu(CPUState *cs) has_msr_hv_hypercall = true; } - memcpy(signature, "KVMKVMKVM\0\0\0", 12); + memcpy(signature, "kvmkvmkvm\0\0\0", 12); c = &cpuid_data.entries[cpuid_i++]; c->function = KVM_CPUID_SIGNATURE | kvm_base; c->eax = 0;

Wow! I can't believe that NVIDIA do this force limitation.

Maybe it can solve the problem that I can't use CUDA in Ubuntu guest OS.
(My guess is, because GRID is earlier product, so Linux driver may earlier get this limitation.)

But... I'm trying to edit kvm.c, I don't know which change is useful? syndtr's or Alex's?
And does any other codes need to fix?


EDIT: After this changes, FINALLY!!!!!!!
Now Linux Guest OS can RUN CUDA WITH GTX480!!!!!!!
My guess is correct! NVIDIA also do this limitation on Linux Driver!


EDIT2: With code changes, now some CUDA samples can run very well (Like deviceQuery)
But some samples can't run (Like matrixMul)

And... VM still gets panic when reboot Ubuntu.
So, still have some problems need to solve.


EDIT3: When runs sample that get error, dmesg got this message

[784.918638] nvidia 0000:01:00.0: irq 45 for MSI/MSI-X [785.394236] NVRM: os_pci_init_handle: invalid context! [785.394362] NVRM: GPU at 0000:01:00: GPU-3222d28c-c7c6-ad1a-67ce-f0e84bed1fe0 [785.394364] NVRM: Xid (0000:01:00): 13, 0001 00000000 000090c0 00002390 00000000 00000000

And CUDA application got this error message (Example for matrixMul):

Failed to create start event (error code unknown error)!

Looks like Linux driver not only have this problem.
Here is some guest OS information

OS: Ubuntu 12.04-4 LTS (Kernel 3.11.0-15-generic x86_64)
NVIDIA Driver: 331.67
CUDA SDK version: 5.5.22

Last edited by AKSN74 (2014-06-04 07:16:38)

Offline

#1954 2014-06-04 14:57:51

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, vga-passthrough on my machine is almost fully working (with OP patches and 3.14.4), BUT I got the same problem as slis, my fps drop from time to time.
Rabcor already described how to fix this, but which patches are now really needed and which patches are old?

The OP pkg has many patches, but for example, I can't apply all the patches (e.g. the i915 patch) from the OP pkg on the actual mainline (3.15).

Like I said, it's working but my question is which patches do I (really) need for 3.14.5-stable and which do I need for 3.15.*-mainline?
I would prefer the stable.
An up to date list with links to the patches with the corresponding kernel versions would be great.

My hardware:

CPU: Xeon E3-1245v3
MB: MSI Z97 Gaming 7
iGPU used by Host
NVidia GeForce GTX 770 passed to Windows Guest
AMD Radeon HD5450 passed to another Linux Guest

Edit: Can I use the code 43 fix to get the latest nvidia driver to work?

Last edited by shawly (2014-06-04 15:03:25)

Offline

#1955 2014-06-04 18:08:46

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

Ok, vga-passthrough on my machine is almost fully working (with OP patches and 3.14.4), BUT I got the same problem as slis, my fps drop from time to time.
Rabcor already described how to fix this, but which patches are now really needed and which patches are old?

The OP pkg has many patches, but for example, I can't apply all the patches (e.g. the i915 patch) from the OP pkg on the actual mainline (3.15).

Like I said, it's working but my question is which patches do I (really) need for 3.14.5-stable and which do I need for 3.15.*-mainline?
I would prefer the stable.
An up to date list with links to the patches with the corresponding kernel versions would be great.

My hardware:

CPU: Xeon E3-1245v3
MB: MSI Z97 Gaming 7
iGPU used by Host
NVidia GeForce GTX 770 passed to Windows Guest
AMD Radeon HD5450 passed to another Linux Guest

Edit: Can I use the code 43 fix to get the latest nvidia driver to work?

have you tried disabling sound in application you experience fps drops in?

i was having varied fps rates in win 7 (league of legends) between reboots. and one time, when fps was hard to bear i tried disabling sounds - it helped. after playing around with audio parameters, i tried the following:

export QEMU_PA_SAMPLES=8192 export QEMU_AUDIO_DRV=pa

so far problems with performance/sound quality are pretty much gone for the last 3 days after i switched to those settings.

p.s. i have only first core (both hyper threads) enabled for my host (isolcpus kernel parameter), win 7 guest is given 3 "dedicated" hyper threads (one from each core unused-by-host)

Offline

#1956 2014-06-04 18:45:56

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

have you tried disabling sound in application you experience fps drops in?

i was having varied fps rates in win 7 (league of legends) between reboots. and one time, when fps was hard to bear i tried disabling sounds - it helped. after playing around with audio parameters, i tried the following:

export QEMU_PA_SAMPLES=8192 export QEMU_AUDIO_DRV=pa

so far problems with performance/sound quality are pretty much gone for the last 3 days after i switched to those settings.

p.s. i have only first core (both hyper threads) enabled for my host (isolcpus kernel parameter), win 7 guest is given 3 "dedicated" hyper threads (one from each core unused-by-host)

I don't even have a soundcard in my vm at the moment, so it's not the sound.
But like I said the fps drops are not the problem, I wanna know which patches I need and which I don't.

Offline

#1957 2014-06-04 18:52:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

CPU: Xeon E3-1245v3
MB: MSI Z97 Gaming 7
iGPU used by Host
NVidia GeForce GTX 770 passed to Windows Guest
AMD Radeon HD5450 passed to another Linux Guest

For this hardware running the latest 3.15-rc, you need the i915 patch, possibly the ACS override patch and matching cmdline option (depending on where the cards are attached), and the qemu patch to hide the hypervisor if you want to run the nvidia 337.88 driver.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1958 2014-06-04 19:05:15

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

For this hardware running the latest 3.15-rc, you need the i915 patch, possibly the ACS override patch and matching cmdline option (depending on where the cards are attached), and the qemu patch to hide the hypervisor if you want to run the nvidia 337.88 driver.

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

Offline

#1959 2014-06-04 19:08:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:
aw wrote:

For this hardware running the latest 3.15-rc, you need the i915 patch, possibly the ACS override patch and matching cmdline option (depending on where the cards are attached), and the qemu patch to hide the hypervisor if you want to run the nvidia 337.88 driver.

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

https://lkml.org/lkml/2014/5/9/517

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1960 2014-06-04 19:16:14

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
shawly wrote:

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

https://lkml.org/lkml/2014/5/9/517

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.

Many thanks <3 will try that tomorrow, does the acs_override patch in the OP pkg apply to 3.15?

Offline

#1961 2014-06-04 19:17:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:
aw wrote:
shawly wrote:

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

https://lkml.org/lkml/2014/5/9/517

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.

Many thanks <3 will try that tomorrow, does the acs_override patch in the OP pkg apply to 3.15?

Dunno, I don't use it


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1962 2014-06-04 22:33:02

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strange no one replied sooner tongue

haha thank you!

anyway i got it almost all working.... except SLI...
my specs and setup are in that google doc

Problem with performace was in CPU config...
i did= -cpu host 4 1 4 2  for i7 2600 = DOTA 2 (40-50fps instead 110-120 native)

after 2 days i managed (haha) to change it to
-cpu host 8 1 8 1 = DOTA 2 (90-110fps)
and its pretty much same with qemu cpu...

with 770gtx it reaches 220fps, but drops to 30fps for no reason, i guess its driver 335 or cache in dota smile

Any suggestions on cpu config maybe to get that 5-10% more? smile it seems like cpu is lowering speed with throttling or something like that , will hugepages help or another model? and what about hv-time (what is that for)?

and about SLI... i didnt try that qemu patch but wiill do...
now: with downstream option, one card won't start either with error code 10 or 12, however after changing countless configs, audio device was reported working on second gpu but gpu itself still had error 12... not enough resources
i did try to define 2nd port root pci bridge in config but with no luck...


also did try 2 VM-s with 560ti each, dota 2 about 80fps each at same time... impressive... and i think that performace hit was due pcie x8 when there are 2 cards on my mbo. cpu usage 40%.

and about encoding... i dont think that u can do 1080p 60fps with software encode... i mean i can... 1 dota2 1080p 60fps stream ... i7 cpu about 70% usage and it cant hold 60 it falls to 40fps... not smooth...
but on windows with igpu everything is smooth constant  60fps... and i bet twice lower cpu usage but i didnt check.... steam doesnt support nvidia encode right now... only intel quick sync....


PS.
after further dota testing it seems like when it reaches 100% gpu usage to take it back to 90% for a half a second then take it back to 100%, and fps varies from 80-110fps for no reason, (it may be happening on windows at faster rate so its not that noticable).... but overall its working great, not a single BUG/CRASH in vm when i got nvidia driver up.... restart and shutdown is working with dumped bios.

Last edited by slis (2014-06-05 08:00:08)

Offline

#1963 2014-06-05 03:47:49

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using linux-mainline from OP and qemu-git from AUR... getting segfaults when I try to start the VM.

modprobe vfio-pci echo 0000:03:00.0 > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind echo 1002 6738 > /sys/bus/pci/drivers/vfio-pci/new_id echo 0000:03:00.1 > /sys/bus/pci/devices/0000\:03\:00.1/driver/unbind echo 1002 aa88 > /sys/bus/pci/drivers/vfio-pci/new_id echo 2660 > /proc/sys/vm/nr_hugepages /usr/bin/qemu-system-x86_64 -bios /usr/share/qemu/bios.bin -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -smp 8,sockets=1,cores=8,threads=1 -m 4G -mem-prealloc -mem-path /dev/hugepages -M q35 -enable-kvm \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,x-vga=on,addr=00.0,multifunction=on,bus=root.1 \ -device vfio-pci,host=03:00.1,addr=00.1,bus=root.1 \ -usb -usbdevice host:04d9:0167 -usbdevice host:0461:4d0f -usbdevice host:045e:028e \ -net nic,model=virtio -net bridge,br=br0 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -drive file=/dev/sdc,id=disk,format=raw -device virtio-blk-pci,drive=disk -drive file=/home/alexis/Downloads/virtio-win-0.1-74.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on -vga none
2021 Segmentation fault (core dumped) [root@snuggles alexis]# dmesg [...] [ 878.248063] device tap0 entered promiscuous mode [ 878.248420] br0: port 2(tap0) entered forwarding state [ 878.248446] br0: port 2(tap0) entered forwarding state [ 879.850979] vfio-pci 0000:03:00.1: enabling device (0400 -> 0402) [ 879.969089] qemu-system-x86[2021]: segfault at 28 ip 00007fe527d78a10 sp 00007fffc6564c88 error 4 in qemu-system-x86_64[7fe527b88000+42e000] [ 881.162161] br0: port 2(tap0) entered disabled state [ 881.162429] device tap0 left promiscuous mode [ 881.162460] br0: port 2(tap0) entered disabled state

Hadn't updated qemu or linux-mainline in a couple months since everything was working. When I installed qemu-git, pacman made me uninstall seabios-git and libacard-git.

[2014-06-04 21:32] [PACMAN] Running 'pacman -U qemu-git-2.1.r33116.ge00fcfe-1-x86_64.pkg.tar.xz' [2014-06-04 21:32] [PACMAN] removed libcacard-git (2.0.r31260.g4c288ac-1) [2014-06-04 21:32] [PACMAN] removed seabios-git (1.7.3.r1736.g5f2d17d-1) [2014-06-04 21:32] [PACMAN] upgraded qemu-git (2.0.r31260.g4c288ac-1 -> 2.1.r33116.ge00fcfe-1)

I can't downgrade qemu as I no longer have libcacard/seabios packages...

Last edited by alexis_evo (2014-06-05 03:49:36)

Offline

#1964 2014-06-05 05:20:17

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit: solved

Last edited by slis (2014-06-10 03:58:42)

Offline

#1965 2014-06-05 05:51:27

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in #1953, I found that CUDA can't run some application with GPU Memory using.
Seems like the memory is full and can't be clean up automatically.
And maybe memory size has been limited with unknown forces.

Only can running well is some information applications (Like deviceQuery in NVIDIA CUDA Samples).
And I think driver still a big problem in Ubuntu. Even hide the hypervisor information.

Can someone test CUDA in VM that OS is Linux? I want to know it is OS issue or driver.

Offline

#1966 2014-06-05 09:49:27

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Any suggestions on cpu config maybe to get that 5-10% more? smile it seems like cpu is lowering speed with throttling or something like that , will hugepages help or another model? and what about hv-time (what is that for)?

To reduce throttling under load I do the following:

echo 100 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor echo 75 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold

This helped to stabilize framerates in almost every game and CPU is still throttled when idle. I've automated this on boot using tmpfiles.d from systemd and added in /etc/tmpfiles.d/99-custom.conf the following:

w /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor - - - - 100 w /sys/devices/system/cpu/cpufreq/ondemand/up_threshold - - - - 75

Last edited by mbroemme (2014-06-05 17:56:37)

Offline

#1967 2014-06-05 15:06:53

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Using linux-mainline from OP and qemu-git from AUR... getting segfaults when I try to start the VM.


same here, clean install jessie, linux-vifio kernel and qemu-git, with config that worked with arch/debian qemu and aw qemu

[ 2902.040594] qemu-system-x86[2588]: segfault at 28 ip 00007fc0b7321fe4 sp 00007fff707ed3f0 error 4 in qemu-system-x86_64[7fc0b7132000+48a000]

it works with debian default qemu-kvm ....

jessie + linux vfio + default qemu-kvm

Last edited by slis (2014-06-05 15:33:50)

Offline

#1968 2014-06-05 15:42:46

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

- memcpy(signature, "KVMKVMKVM\0\0\0", 12); + memcpy(signature, "kvmkvmkvm\0\0\0", 12);

You sir, are a genius. Thanks to both of you!

Offline

#1969 2014-06-05 18:49:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aldum wrote:
aw wrote:
aw wrote:

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

- memcpy(signature, "KVMKVMKVM\0\0\0", 12); + memcpy(signature, "kvmkvmkvm\0\0\0", 12);

You sir, are a genius. Thanks to both of you!

Current qemu.git now has an option to hide the hypervisor.  You can add "kvm=off" to your list of -cpu option, ex:

-cpu host,hv-time,kvm=off

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1970 2014-06-05 19:51:50

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Current qemu.git now has an option to hide the hypervisor.  You can add "kvm=off" to your list of -cpu option, ex:

-cpu host,hv-time,kvm=off

I already compiled a custom QEMU with ABS, but that would have made it a lot easier :3

Offline

#1971 2014-06-06 00:58:07

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I managed to get it booting again by downgrading qemu-git AUR package. I grabbed the last PKGBUILD from some mirror website that hosts old PKGBUILDs, and added a git reset --hard into it to downgrade to some commit in April. Here's the PKGBUILD:

# Maintainer: Devin Cofer <ranguvar{AT]archlinux[DOT}us> # Contributor: Tobias Powalowski <tpowa@archlinux.org> # Maintainer: Frederic Bezies <fredbezies@gmail.com> pkgname=qemu-git _gitname=qemu pkgver=2.1.r32209.g2d03b49 pkgrel=1 pkgdesc="A generic and open source processor emulator which achieves a good emulation speed by using dynamic translation. Git version." arch=('i686' 'x86_64') license=('GPL2' 'LGPL2') url="http://wiki.qemu.org/Index.html" depends=('pixman' 'libjpeg' 'libpng' 'sdl' 'alsa-lib' 'nss' 'glib2' 'gnutls>=2.4.1' 'bluez-libs' 'vde2' 'util-linux' 'curl' 'libsasl' 'libgl' 'libpulse' 'libcap-ng' 'libaio' 'libseccomp' 'libiscsi' 'spice' 'seabios-git' 'usbredir') makedepends=('git' 'texi2html' 'perl' 'python2' 'dtc' 'spice-protocol') optdepends=('ovmf-bin: UEFI Firmware (OVMF) with Secure Boot Support - for Virtual Machines (QEMU)') conflicts=('qemu' 'kvm' 'kvm-git' 'qemu-spice' 'libcacard') provides=('qemu' 'qemu-kvm' 'qemu-spice' 'libcacard') backup=('etc/qemu/target-x86_64.conf') source=('git://git.qemu.org/qemu.git' '65-kvm.rules' 'qemu.install') sha256sums=('SKIP' '9c8a15c34461a9481a34ca9e0ab4ae3825eabe8fd863227f2445325413cd755c' '9970c3050e8dc6153c5955d018d114f9fcbc091843b85f9e7b247eb28f09ba10') install='qemu.install' options=(!strip) pkgver() { cd "${srcdir}/$_gitname" git reset --hard 2d03b49c3f225994c4b0b46146437d8c887d6774 echo "2.1.r$(git rev-list --count master).g$(git log -1 --format="%h")" } build() { cd $_gitname git reset --hard 2d03b49c3f225994c4b0b46146437d8c887d6774 # qemu vs. make 4 == bad export ARFLAGS="rv" # http://permalink.gmane.org/gmane.comp.emulators.qemu/238740 # gtk gui breaks keymappings at the moment ./configure --prefix=/usr --sysconfdir=/etc --audio-drv-list='pa alsa sdl' \ --python=/usr/bin/python2 --smbd=/usr/bin/smbd \ --enable-docs --libexecdir=/usr/lib/qemu \ --disable-gtk --enable-linux-aio --enable-seccomp \ --enable-spice --localstatedir=/var --disable-werror \ --enable-tpm make V=99 } package() { cd $_gitname git reset --hard 2d03b49c3f225994c4b0b46146437d8c887d6774 make DESTDIR="${pkgdir}" libexecdir="/usr/lib/qemu" install # provided by seabios package rm "${pkgdir}/usr/share/qemu/bios.bin" rm "${pkgdir}/usr/share/qemu/acpi-dsdt.aml" rm "${pkgdir}/usr/share/qemu/q35-acpi-dsdt.aml" # remove conflicting /var/run directory rm -r "${pkgdir}/var" install -D -m644 "${srcdir}/65-kvm.rules" \ "${pkgdir}/usr/lib/udev/rules.d/65-kvm.rules" # bridge_helper needs suid # https://bugs.archlinux.org/task/32565 chmod u+s "${pkgdir}/usr/lib/qemu/qemu-bridge-helper" # add sample config echo "allow br0" > ${pkgdir}/etc/qemu/bridge.conf.sample # strip scripts directory find "${pkgdir}/usr/src/linux-${_kernver}/scripts" -type f -perm -u+w 2>/dev/null | while read binary ; do case "$(file -bi "$binary")" in *application/x-executable*) # Binaries /usr/bin/strip $STRIP_BINARIES "$binary";; esac done # remove libcacard files # rm -rf ${pkgdir}/usr/include/cacard # rm -rf ${pkgdir}/usr/lib/libcacard* # rm -rf ${pkgdir}/usr/lib/pkgconfig/libcacard.pc # rm -rf ${pkgdir}/usr/bin/vscclient }

This is still using a rather old qemu build though, so I would greatly appreciate any help getting the latest qemu-git to work.

Offline

#1972 2014-06-06 01:14:06

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

So I managed to get it booting again by downgrading qemu-git AUR package.
[…]
This is still using a rather old qemu build though, so I would greatly appreciate any help getting the latest qemu-git to work.

Interesting, I actually maintain the qemu-git aur package now and the upgrade removed seabios-git and libcacard-git, because its using qemu's internal versions now.

Since it works for me, I'm going to try to blame upstream here, but this isn't really helpful, so maybe try to run it in strace or gdb or something and see if you can get some usage output/error messages.


i'm sorry for my poor english wirting skills…

Offline

#1973 2014-06-06 01:39:39

Rusky
Member
Registered: 2014-06-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thehighhat wrote:

So I had a working vga passthrough of nvidia gtx470 with a windows 8 guest.  I let the windows store upgrade to windows 8.1 update.  now, the guest will not start correct.  it always goes into automatic system repair and fails.  the refresh pc, reset pc options ask for windwos cd, but cannot detect it.  if i just let windows try to start, it shows nohing but a black screen.  then i have reboot the host becaues i cannot shutdown the guest cleanly.

any ideas why the 8.1 update breaks everything?

I have this exact same problem with a Radeon R9 270X, starting with a working Windows 8.1 install and just doing a normal update. Automatic system repair always fails, all the bootrec tricks fail, building a new BCD from scratch fails, refreshing the install fails. The host is Arch with the linux-mainline kernel from the first post, using Qemu 2.0.0 from the repositories.

Offline

#1974 2014-06-06 05:06:28

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

here is strace form segmentation fault with qemu-git, it crashes with vga none option....
i can give more logs and info if u instruct me how to smile
btw this is debian ... linux-vfio 3.15 with acs patch only


root@TEST:~# strace sh q execve("/bin/sh", ["sh", "q"], [/* 15 vars */]) = 0 brk(0) = 0x7fac63b8c000 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bc3000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=35337, ...}) = 0 mmap(NULL, 35337, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fac61bba000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1734080, ...}) = 0 mmap(NULL, 3840576, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fac615fc000 mprotect(0x7fac6179c000, 2097152, PROT_NONE) = 0 mmap(0x7fac6199c000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a0000) = 0x7fac6199c000 mmap(0x7fac619a2000, 14912, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fac619a2000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bb9000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bb8000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bb7000 arch_prctl(ARCH_SET_FS, 0x7fac61bb8700) = 0 mprotect(0x7fac6199c000, 16384, PROT_READ) = 0 mprotect(0x7fac61de2000, 8192, PROT_READ) = 0 mprotect(0x7fac61bc5000, 4096, PROT_READ) = 0 munmap(0x7fac61bba000, 35337) = 0 getpid() = 2982 rt_sigaction(SIGCHLD, {0x7fac61bd9620, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0 geteuid() = 0 brk(0) = 0x7fac63b8c000 brk(0x7fac63bad000) = 0x7fac63bad000 getppid() = 2981 stat("/root", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 open("q", O_RDONLY) = 3 fcntl(3, F_DUPFD, 10) = 10 close(3) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGINT, {0x7fac61bd9620, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0 rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGQUIT, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0 rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGTERM, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0 read(10, "#!/bin/bash\nqemu-system-x86_64 -"..., 8192) = 613 stat("/usr/local/sbin/qemu-system-x86_64", 0x7fff18e9d8d0) = -1 ENOENT (No such file or directory) stat("/usr/local/bin/qemu-system-x86_64", {st_mode=S_IFREG|0755, st_size=5998584, ...}) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fac61bb89d0) = 2983 wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV}], 0, NULL) = 2983 --- SIGCHLD (Child exited) @ 0 (0) --- rt_sigreturn(0x11) = 2983 write(2, "Segmentation fault\n", 19Segmentation fault ) = 19 read(10, "", 8192) = 0 exit_group(139) = ? root@TEST:~#

config

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 3072 -cpu Haswell,hv-time -smp 4,sockets=1,cores=4,threads=1 \ -boot menu=on \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/root/GF114.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/root/win.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -usb -usbdevice host:413c:2003 -usbdevice host:046d:c042 \ -netdev tap,id=guest0,vhost=on,script=no \ -net nic,model=e1000,netdev=guest0 \ -vnc :1

Last edited by slis (2014-06-06 06:29:29)

Offline

#1975 2014-06-06 06:31:39

rman
Member
From: United States
Registered: 2013-08-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to chime in with another success story. It took me an entire weekend plus the following Monday, but I got my VGA passthrough setup fully functional! big_smile

Specs:
CPU: Core i5-4690
MB: ASRock H97m Pro 4
GPU 1: HD Graphics 4600 (used by Arch host)
GPU 2: AMD Radeon R7 265 (passed through to Windows 7)

The only patch I needed was the i915 arbiter patch. At first I used the 3.14 kernel from the OP, but I had some suspend to RAM issues, so I applied the patch against 3.15rc8 and it built and ran fine.

I cannot passthrough the Radeon's HDMI audio controller, or I get the dreaded atikmpag.sys BSOD. The VM is otherwise flawless though - I can even reboot it with absolutely no problems.

I had some issues with Qemu's emulated audio - Intel HDA was basically unusable with pops and static. I tried ac97, which worked much better, but I still had stuttering, noise, and choppiness. At that point I gave up and bought a USB sound card, which of course is working perfectly.

Anyway, thanks to everyone who contributed to this thread! It's good to know that this is still working with the latest GPU's. smile

Last edited by rman (2014-06-06 06:34:28)


Laptop: Lenovo L440, Intel Core i3-4000M, HD Graphics 4600
Desktop: Intel Core i5-4690, HD Graphics 4600, AMD Radeon R7 265 (KVM VGA passthrough)

Offline

#1976 2014-06-06 13:39:29

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rman wrote:

Just wanted to chime in with another success story. It took me an entire weekend plus the following Monday, but I got my VGA passthrough setup fully functional! big_smile

Specs:
CPU: Core i5-4690
MB: ASRock H97m Pro 4
GPU 1: HD Graphics 4600 (used by Arch host)
GPU 2: AMD Radeon R7 265 (passed through to Windows 7)

The only patch I needed was the i915 arbiter patch. At first I used the 3.14 kernel from the OP, but I had some suspend to RAM issues, so I applied the patch against 3.15rc8 and it built and ran fine.

I cannot passthrough the Radeon's HDMI audio controller, or I get the dreaded atikmpag.sys BSOD. The VM is otherwise flawless though - I can even reboot it with absolutely no problems.

I had some issues with Qemu's emulated audio - Intel HDA was basically unusable with pops and static. I tried ac97, which worked much better, but I still had stuttering, noise, and choppiness. At that point I gave up and bought a USB sound card, which of course is working perfectly.

Anyway, thanks to everyone who contributed to this thread! It's good to know that this is still working with the latest GPU's. smile

Passing system's HD Audio works great too ;p

Offline

#1977 2014-06-06 15:56:09

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright I finally got it working with kernel 3.15rc8 and just the i915 patch. <3
Here is a little benchmark I did, to test if it's working: http://www.3dmark.com/3dm11/8402985

Compared to other native benchmarks are the ~10% difference in performance not that bad.
My question, can I optimize/tweak my performance somehow?

Edit: Compared to other benchmarks, my graphics score is pretty decent! But my physicsscore is pretty low, so this means my cpu slows this down as it seems.. Can I optimize this?

Last edited by shawly (2014-06-06 16:02:16)

Offline

#1978 2014-06-06 16:41:11

rman
Member
From: United States
Registered: 2013-08-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Passing system's HD Audio works great too ;p

I need audio for Linux too. wink


Laptop: Lenovo L440, Intel Core i3-4000M, HD Graphics 4600
Desktop: Intel Core i5-4690, HD Graphics 4600, AMD Radeon R7 265 (KVM VGA passthrough)

Offline

#1979 2014-06-06 17:43:15

bpye
Member
Registered: 2014-06-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think from what I've read my planned configuration should work, but before I go switching out my 4670k for something else I want to check.

CPU: Intel 4670/Intel 4690k (both support vt-d but I have yet to choose)
MB: MSI GD45 Gaming
GPU1: Intel iGPU (used by Arch host)
GPU2: Nvidia GTX 560 Ti (used by Windows guest)

I believe this should work if I A. use the i915 vga arbitration patch and B. -cpu host,hv-time,kvm=off . Is there anything else I should need to do, or should that setup work okay? Additionally, is there any way to switch the keyboard/mouse from the host to guest with say a keyboard shortcut or will I require a KVM, I can already hook up my monitor to both cards and just use it's source button. Lastly, does QEMU's emulated audio work sufficiently?

Offline

#1980 2014-06-06 17:51:35

rman
Member
From: United States
Registered: 2013-08-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpye wrote:

Additionally, is there any way to switch the keyboard/mouse from the host to guest with say a keyboard shortcut or will I require a KVM,

You can use Synergy, which will allow you to share the keyboard, mouse, and clipboard between the host and the VM.

bpye wrote:

Lastly, does QEMU's emulated audio work sufficiently?

Not really. Buy a $5 USB sound card and pass that through.


Laptop: Lenovo L440, Intel Core i3-4000M, HD Graphics 4600
Desktop: Intel Core i5-4690, HD Graphics 4600, AMD Radeon R7 265 (KVM VGA passthrough)

Offline

#1981 2014-06-06 20:10:17

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I decided to format my windows 8.1 guest after it somehow got its bootloader corrupted. However, one thing that has always annoyed me is that I couldn't boot from my virtio-blk-pci disk, because seabios sees the HDDs attached to the SATA controller I passed through first.

So, I decided that instead of passing the controller, I could use virtio-scsi. It works great, seabios now orders the disks the way I want it; however I can't boot windows or even finish the setup this way. If I finish the setup with virtio-blk and later switch it to scsi-hd it BSODs with error code 0xc000021a. Linux installs fine (I tried it with Ubuntu, and everything works correctly including the video card).

Can anyone help me figure out what is wrong? Or at least tell me a way to have seabios boot from a disk other than the 1st, so I can stick with what works. I'm running Arch with all of nbhs's packages and patches. Here is my config file:

qemu-system-x86_64 \ -enable-kvm \ -machine type=q35,accel=kvm \ -m 24G \ -boot menu=on \ -cpu host,hv-time \ -smp 6,sockets=1,cores=3,threads=2 \ -mem-path /dev/hugepages \ -bios /usr/share/qemu/bios.bin \ -vga none \ -nographic \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/user/tahiti.rom \ -device vfio-pci,host=02:00.1,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -drive id=ssd,format=raw,discard=on,cache=none,file=/dev/virtual-machines/windows \ -drive id=hssd,format=raw,cache=none,file=/dev/sdc \ -device scsi-hd,drive=ssd \ -device scsi-block,drive=hssd \ -netdev bridge,br=br0,id=hostnet0 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:00:01:01 \ -device vfio-pci,host=08:00.0,bus=pcie.0 \ -qmp tcp:127.0.0.1:5555,server,nowait \

Last edited by Chetyre (2014-06-06 20:12:27)

Offline

#1982 2014-06-06 20:13:32

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rman wrote:
bpye wrote:

Lastly, does QEMU's emulated audio work sufficiently?

Not really. Buy a $5 USB sound card and pass that through.

disagree. it depends on your requirements. for me it does its job almost flawlessly after some playing around with parameters (described details it in one of my posts earlier)

passing through distinct usb audio card is not viable if outputs from both host and guest systems are desired on the same audio out port

Offline

#1983 2014-06-06 20:20:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
rman wrote:
bpye wrote:

Lastly, does QEMU's emulated audio work sufficiently?

Not really. Buy a $5 USB sound card and pass that through.

disagree. it depends on your requirements. for me it does its job almost flawlessly after some playing around with parameters (described details it in one of my posts earlier)

passing through distinct usb audio card is not viable if outputs from both host and guest systems are desired on the same audio out port

Emulated audio works surprisingly well for me too.  It's a little crackly when the VM starts up, but it's good after that and I've done zero tuning for it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1984 2014-06-06 20:25:18

bpye
Member
Registered: 2014-06-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
sinny wrote:
rman wrote:

Not really. Buy a $5 USB sound card and pass that through.

disagree. it depends on your requirements. for me it does its job almost flawlessly after some playing around with parameters (described details it in one of my posts earlier)

passing through distinct usb audio card is not viable if outputs from both host and guest systems are desired on the same audio out port

Emulated audio works surprisingly well for me too.  It's a little crackly when the VM starts up, but it's good after that and I've done zero tuning for it.

Thanks, it sounds like this is the solution I've been looking for which is great. Probably going to go with the 4690 since I don't care for overclocking, my workload has changed since I've built this PC.

Offline

#1985 2014-06-06 21:43:17

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

Alright I finally got it working with kernel 3.15rc8 and just the i915 patch. <3
Here is a little benchmark I did, to test if it's working: http://www.3dmark.com/3dm11/8402985

Compared to other native benchmarks are the ~10% difference in performance not that bad.
My question, can I optimize/tweak my performance somehow?

Edit: Compared to other benchmarks, my graphics score is pretty decent! But my physicsscore is pretty low, so this means my cpu slows this down as it seems.. Can I optimize this?

what is your current cpu config?
i run my i7 2600 with
-cpu Haswell,hv-time -smp 8,sockets=1,cores=8,threads=1

i got about 98% of native.

Offline

#1986 2014-06-07 08:12:51

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

what is your current cpu config?
i run my i7 2600 with
-cpu Haswell,hv-time -smp 8,sockets=1,cores=8,threads=1

i got about 98% of native.

Wow, thats impressive.
I got an Intel Xeon E3-1245v3 with -cpu host -smp 8,sockets=1,cores=4,threads=2 (I use libvirt to set up my vms and start them).

Btw, how do you guys all use an emulated soundcard? I wanna use that too but I don't know the parameter for qemu..

Edit: Got another little question, can I change the resolution of SeaBios? My windows Logo is so streched and I want it to show in 1920x1080px

Last edited by shawly (2014-06-07 08:20:41)

Offline

#1987 2014-06-07 09:13:41

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got an Intel Xeon E3-1245v3 with -cpu host -smp 8,sockets=1,cores=4,threads=2 (I use libvirt to set up my vms and start them).

try changing cores=8 threads=1

and cpu model

Last edited by slis (2014-06-07 09:16:35)

Offline

#1988 2014-06-07 09:54:11

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

try changing cores=8 threads=1

and cpu model

When I do this my Graphicsscore drops in 3DMark and my Physicsscore is slightly higher..
Bench with cores=4 threads=2: http://www.3dmark.com/3dm11/8402985
Bench with cores=8 threads=1: http://www.3dmark.com/3dm11/8405475

Other problem is, I played Bioshock Infinite but it's barely playable, my FPS drop every 5 secs and this is very frustrating... dunno what to do anymore..
My motivation sinks with every new problem I get and I almost believe it's not possible.. but I don't wanna switch back to windows (or dualboot) so I'm stuck in a loop...

Offline

#1989 2014-06-07 10:01:27

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

strange... but same happend to me when i got my friends gtx 770 ... dota 2 had drops to 30fps.... but i didn't have much time to test it...
native average is 150fps with that gpu...

my 560ti works like native....

change drivers maybe? are u using 335?

Offline

#1990 2014-06-07 10:09:13

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

strange... but same happend to me when i got my friends gtx 770 ... dota 2 had drops to 30fps.... but i didn't have much time to test it...
native average is 150fps with that gpu...

my 560ti works like native....

change drivers maybe? are u using 335?

Yep I'm using 335, but updating to 338 scares me, because the last time, my vm didn't boot anymore and I had to reinstall it.
But well, F it, I'll try installing 338 since I'm slowly getting tired of this... it's now three weeks I've been trying to get this work and now this..

Edit: Ok atleast updating to 338 didn't gave any problems, will now test again.
Edit2: Benchmark looks like this: http://www.3dmark.com/3dm11/8405620
But Bioshock is still lagging... Same happens in Magicka WIzard Wars, every 5-10 seconds it hangs for just a half sec, which is pretty annoying..

Last edited by shawly (2014-06-07 10:45:01)

Offline

#1991 2014-06-07 14:17:22

chopsta
Member
Registered: 2014-06-06
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hallo,

when i boot the VM, i get a bsod and the machine restarts.
While the machine starts, the terminal output is: qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.0, no available reset mechanism.

i have installed the newest Kernel 3.11.10-03111011-generic (Mainline).
I also installed newest version of qemu git (from git)

The VM-Config is:
qemu-system-x86_64 -enable-kvm -M q35 -drive file=/home/florian/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -m 1024 -cpu host,hv-time,kvm=off \
-smp 1,sockets=1,cores=1,threads=1 \
-bios /usr/share/qemu/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

System:
CPU: AMD A8 6600K (IGP for host)
MB: Gigabyte (GA-F2A88XN-WIFI) Mini ITX AMD A88X (Iommu is supported an enabled)
GPU: NVidia 750ti (Guest)             

dmesg |grep pci-stub shows:
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.11.10-03111011-generic root=UUID=b301bc17-2fa2-43b9-968a-868a8aaa1954 ro vfio_iommu_type1.allow_unsafe_interrupts=1 quiet splash pci-stub.ids=10de:1380,10de:0fbc vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.11.10-03111011-generic root=UUID=b301bc17-2fa2-43b9-968a-868a8aaa1954 ro vfio_iommu_type1.allow_unsafe_interrupts=1 quiet splash pci-stub.ids=10de:1380,10de:0fbc vt.handoff=7
[    4.786007] pci-stub: add 10DE:1380 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.786019] pci-stub 0000:01:00.0: claimed by stub
[    4.786028] pci-stub: add 10DE:0FBC sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.786036] pci-stub 0000:01:00.1: claimed by stub
[   70.981336] pci-stub 0000:01:00.0: claimed by stub
[   70.981467] pci-stub 0000:01:00.1: claimed by stub

I hope, that someone of you can give me a hint.

Many thanks in advance.

Last edited by chopsta (2014-06-07 15:12:04)

Offline

#1992 2014-06-07 16:23:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chopsta wrote:

i have installed the newest Kernel 3.11.10-03111011-generic (Mainline).

Not remotely the newest kernel.  This may even be pre-bus reset support.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1993 2014-06-07 18:58:30

chopsta
Member
Registered: 2014-06-06
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, have installed 3.15.8-RC8.

Terminal doesn't output anymore "qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.0, no available reset mechanism."
The screen, which is connected to the graphics card, has no video input.
When i activate emulated vga, i get a BSOD.

Last edited by chopsta (2014-06-07 20:32:36)

Offline

#1994 2014-06-07 20:37:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chopsta wrote:

ok, have installed 3.15.8-RC8.

Terminal doesn't output anymore "qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.0, no available reset mechanism."
The screen, which is connected to the graphics card, has no video input.
When i activate emulated vga, i get a BSOD.

You probably need this patch https://lkml.org/lkml/2014/5/25/94


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1995 2014-06-08 15:23:18

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But Bioshock is still lagging... Same happens in Magicka WIzard Wars, every 5-10 seconds it hangs for just a half sec, which is pretty annoying..


can u try installing cpufreqd and force 100% in config on some profile and set it to that?

Offline

#1996 2014-06-08 16:19:14

setedivento
Member
Registered: 2014-06-08
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everybody, really great topic, thanks!

If only I could make things work. I seemed so close... may anyone help?
I am trying to vga passthrough an Nvidia Quadro 4000 on a Kubuntu 14.04 host machine. Reports indicate more than one here has coped, on Ubuntu based distros, so I've decided to ask.

My problem is, to put it short, that when I launch the vm, I land in the qemu monitor console.
This is how I launch the vm:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 -drive file=/home/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/home/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -boot d

And this is the pci infos I get when I redirect the monitor console output to the standard output (by adding the '-monitor stdio' argument to the command above) and type 'info pci' inside the console:

QEMU 2.0.0 monitor - type 'help' for more information (qemu) info pci Bus 0, device 0, function 0: Host bridge: PCI device 8086:29c0 id "" Bus 0, device 1, function 0: Ethernet controller: PCI device 8086:100e IRQ 10. BAR0: 32 bit memory at 0xfe040000 [0xfe05ffff]. BAR1: I/O at 0xd000 [0xd03f]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe]. id "" Bus 0, device 28, function 0: PCI bridge: PCI device 8086:3420 BUS 0. secondary bus 1. subordinate bus 1. IO range [0xc000, 0xcfff] memory range [0xfa000000, 0xfdffffff] prefetchable memory range [0xe8000000, 0xf7ffffff] id "root.1" Bus 1, device 0, function 0: VGA controller: PCI device 10de:06dd IRQ 10. BAR0: 32 bit memory at 0xfa000000 [0xfbffffff]. BAR1: 64 bit prefetchable memory at 0xe8000000 [0xefffffff]. BAR3: 64 bit prefetchable memory at 0xf0000000 [0xf3ffffff]. BAR5: I/O at 0xc000 [0xc07f]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0007fffe]. id "" Bus 1, device 0, function 1: Audio controller: PCI device 10de:0be5 IRQ 10. BAR0: 32 bit memory at 0xfc080000 [0xfc083fff]. id "" Bus 0, device 31, function 0: ISA bridge: PCI device 8086:2918 id "" Bus 0, device 31, function 2: SATA controller: PCI device 8086:2922 IRQ 10. BAR4: I/O at 0xd080 [0xd09f]. BAR5: 32 bit memory at 0xfe060000 [0xfe060fff]. id "" Bus 0, device 31, function 3: SMBus: PCI device 8086:2930 IRQ 10. BAR4: I/O at 0xb100 [0xb13f]. id ""

In particular,  the entries:
'VGA controller: PCI device 10de:06dd'
&
'Audio controller: PCI device 10de:0be5'
correctly represent the video card and its audio addresses, which makes me think they are bound to vfio and used.

There is no error, I just land in the monitor console!

By the way, if I launch the vm without the '-vga none' argument, the install iso (windows.iso) boots perfectly.

Provided that I am quite a newbie on the matter, this is what I've done so far (a raw summary of the complete setup):
-- selected under system settings the amd fglrx video drivers for the primary amd video card (radeon hd 5770)
-- blacklisted i915,nouveau,nvidia for the nvidia quadro to passthrough
-- installed (through apt-get) the libvirt-bin, libvirt-doc, qemu and qemu-kvm packages
-- used the procedure available here http://www.redhat.com/archives/libvir-l … 00514.html to bind the gpu (had to do because when I launched the vm there was an error about missing /dev/vfio/15)
-- added '/dev/vfio/15' to cgroup_device_acl  on /etc/libvirt/qemu.conf (as suggested by nbhs at post nr. #652, p. 27)
-- followed the guidelines for Ubuntu distros by GizmoChicken at post #430, p. 18, but just added the vga controller and its audio, to the devices list in /etc/vfio-pci.cfg (no other address)

What I am I doing wrong?

Thanks so much!!!!

Offline

#1997 2014-06-08 18:51:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

setedivento wrote:

Hello everybody, really great topic, thanks!

If only I could make things work. I seemed so close... may anyone help?
I am trying to vga passthrough an Nvidia Quadro 4000 on a Kubuntu 14.04 host machine. Reports indicate more than one here has coped, on Ubuntu based distros, so I've decided to ask.

Full stop, you have a Quadro card and while it's not the version that NVIDIA supports for device assignment, it's hopefully close enough.  Forget about Q35, -vga none, and x-vga=on, Quadro doesn't need any of that.  Just add it to the VM as you would any other assigned device, you can even use virt-manager for it.  Install using the emulated VGA.  When you install the NVIDIA driver stack, the emulated VGA will be disabled and you'll automatically switch over to the Quadro card.  We intend to support K-series Quadros on RHEL7 using this mode, but those cards are also known not to work with x-vga=on.  K-series cards are Kepler based and yours I believe is Fermi based.  Those are only one generation apart, so it will hopefully work.  For others out there hoping their cheapy Quadro NVS cards can work like this, don't bet on it.

EDIT: The audio device on K-series Quadro is also known not to work and will not be supported for assignment on RHEL7.  Try it, but expect it not to work.  Just bind it to pci-stub and forget it's there.

Last edited by aw (2014-06-08 19:02:15)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1998 2014-06-08 19:38:46

hell_n_back
Member
Registered: 2014-06-08
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Been banging my head against a brick wall with this one.
I am beginning to think something just isn't capable with this,

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

All I Ever get is:
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

and the iommu_group folders are no where to be found.

[root@nether ~]# dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled

[root@nether ~]# uname -r
3.15.0-1-mainline

linux   /vmlinuz-linux-mainline root=UUID=5995bbb5-119b-40fc-8206-4c4b75f76c19 rw intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0,13f6:8788 quiet

[    0.486310] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.486319] pci-stub 0000:03:00.0: claimed by stub
[    0.486324] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.486328] pci-stub 0000:03:00.1: claimed by stub
[    0.486329] pci-stub: add 13F6:8788 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.486333] pci-stub 0000:09:01.0: claimed by stub


or even :
[root@nether ~]# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none --device pci-assign,host=09:01.0                                                             qemu-system-x86_64: --device pci-assign,host=09:01.0: No IOMMU found.  Unable to assign device "(null)"
qemu-system-x86_64: --device pci-assign,host=09:01.0: Device initialization failed.
qemu-system-x86_64: --device pci-assign,host=09:01.0: Device 'kvm-pci-assign' could not be initialized

Intel Core i7-2600K
Gigabyte : GA-Z68X-UD7-B3
ASUS R9280X-DC2T-3GD5 Radeon R9 280X (Passthrough device)
GeForce 9500 GT - Host GPU
HT Omega Claro - want to pass this through as well

lspci
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 2 (rev b5)
00:1c.2 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch (rev a3)
02:00.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch (rev a3)
02:02.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch (rev a3)
02:03.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch (rev a3)
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
04:00.0 VGA compatible controller: NVIDIA Corporation G96 [GeForce 9500 GT] (rev a1)
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
08:00.0 PCI bridge: Integrated Technology Express, Inc. Device 8892 (rev 10)
09:01.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
09:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)
0a:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
0b:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
0c:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
0d:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11)
0e:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11)


Not sure if this is even worth continuing, I feel like something is just not compatible with it all
Anyone have any suggestions?

Offline

#1999 2014-06-08 19:43:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hell_n_back wrote:

Intel Core i7-2600K

http://ark.intel.com/products/52214/Int … o-3_80-GHz

Intel® Virtualization Technology for Directed I/O (VT-d):    No


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2000 2014-06-08 21:07:31

hell_n_back
Member
Registered: 2014-06-08
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
hell_n_back wrote:

Intel Core i7-2600K

http://ark.intel.com/products/52214/Int … o-3_80-GHz

Intel® Virtualization Technology for Directed I/O (VT-d):    No


Well that is helpful.
Wish I had seen that page a few weeks ago.  hmm

Thank you.

Offline

#2001 2014-06-09 07:34:29

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now i could make it all work with stock (from repos, no recompiling) software available:
Kernel: 3.15 (no patches)
Qemu 2.0.0
libvirt 1.2.2
virt-manager 1.0.1

CPU AMD FX(tm)-8350 Eight-Core Processor
GPU radeon R9 270X
Motherboard SABERTOOTH 990FX R2.0

This is on linux mint 17 (based on ubuntu 14.04 LTS). For libvirt i used default pc chipset (no manual xml editing!). Using emulated GPU for booting, OS switches to physical GPU later on during boot. Good to finally see this work without much hackery. :)

Last edited by novist (2014-06-09 11:47:43)

Offline

#2002 2014-06-09 12:50:51

apporc
Member
Registered: 2014-06-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey, novist. I am using the same software with you, and there is still one problem for me.
Kernel: 3.15 (no patches)
Qemu: 2.0.0
libvirt: 1.2.2

CPU: Intel(R) Core(TM) i7-4770
GPU: radeon 7870
Motherboard: Asus Z87K

My OS is Linux Mint 17 too.  I used the kvm + vfio_pci method. I configured all according to this thread except kernel/qemu version and patch.
I have succeeded to boot my guest os, as i think the VGA card is at least partially passthroughed.
When i start my guest os win7, the boot screen of this win7 will show through radeon 7870 to my secondary monitor.
Problem is i just can't find the "radeon 7870" device in win7's device manager. There is only one graphics card called "Normal VGA Monitor".

Indeed, i have tried to install ATI driver and run warcraft III, everything is ok. I am just a little confused about that.

Offline

#2003 2014-06-09 15:22:59

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sounds like it works alright. if you get expected performance dont worry about it. could be some kvm hickup or something, who knows. you did not add emulated vga right? i can suggest you trying to set up vm on pc chipset with physical vga as secondary, maybe that would change something.

Offline

#2004 2014-06-09 20:03:47

bpye
Member
Registered: 2014-06-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Got it working, however as expected audio sucks. What sort of latency would I been looking it if I connected a usb sound card to my line in?

Offline

#2005 2014-06-09 20:49:00

iwasaperson
Member
Registered: 2014-06-09
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I keep getting this error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Specs:
3570K
AMD Radeon 6970 (Card I'm trying to pass through)
Mobo: ASUS P8Z77-V LK

This is the third time I've tried this after several formats (not related to this).
Any idea on how to solve it? I've tried looking around, but I can't find anything.

Offline

#2006 2014-06-09 20:51:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iwasaperson wrote:

I keep getting this error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Specs:
3570K
AMD Radeon 6970 (Card I'm trying to pass through)
Mobo: ASUS P8Z77-V LK

This is the third time I've tried this after several formats (not related to this).
Any idea on how to solve it? I've tried looking around, but I can't find anything.

Get a CPU that supports VT-d, yours does not:

http://ark.intel.com/products/65520/Int … o-3_80-GHz


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2007 2014-06-09 20:56:28

iwasaperson
Member
Registered: 2014-06-09
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
iwasaperson wrote:

I keep getting this error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Specs:
3570K
AMD Radeon 6970 (Card I'm trying to pass through)
Mobo: ASUS P8Z77-V LK

This is the third time I've tried this after several formats (not related to this).
Any idea on how to solve it? I've tried looking around, but I can't find anything.

Get a CPU that supports VT-d, yours does not:

http://ark.intel.com/products/65520/Int … o-3_80-GHz

I was not aware of that. Thanks for letting me know. Does the 4790K support it?
EDIT: Just looked it up. It does.

Last edited by iwasaperson (2014-06-09 20:57:08)

Offline

#2008 2014-06-09 22:43:38

spicy
Member
Registered: 2014-06-09
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

First time poster here, I hope you'll forgive my inexperience. I should also point out that I am trying this with Debian testing / "jessie", hopefully that isn't a problem. This seems to be the best resource for KVM VGA passthrough right now.

I'm trying to get primary VGA passthrough working but running into an issue when trying to enable x-vga=on. Secondary vga passthrough is working fine with "-vga std" set but I get the following when using x-vga and vga none:

    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga
    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get device 0000:02:00.0
    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I should probably note here that I am using a git compiled kvm and qemu, but I am using a packaged kernel from Debian's testing repo: 3.14-1-amd64. This might be the source of my problems – it wasn't clear to me if many of the patches from the OP were still needed on newer releases of the kernel. I've read through a fair bit of the thread, but I'll confess that I haven't read all 81 pages yet. I also haven't tried a compiled kernel, but if this looks to be the source of the problem I'm happy to try it.

Otherwise, I'm not sure what could be causing this issue. Both cards are working in secondary passthrough and are ATI Radeon cards – not Nvidia (though I am hoping to test a GTX card soon as well)

Thanks for your time, any suggestions would be greatly appreciated!

Offline

#2009 2014-06-09 22:50:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

spicy wrote:

    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga

This means that either a) your kernel does not support CONFIG_VFIO_PCI_VGA or b) the device is not a VGA device.  To test a):

$ grep CONFIG_VFIO_PCI_VGA /boot/config-`uname -r`

To test b):

$ lspci -s 2:00.0 | grep VGA

If you have Intel host graphics, you still need the i915 patch for your kernel.  If you Radeon host graphics, you need the other VGA arbiter patch.  Both of these have been referenced in the last few pages, IIRC.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2010 2014-06-10 01:06:20

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Passthrough of an AMD 6450 works well with kernel 3.14 with i915 VGA arbiter patch applied. I need to passthrough the onboard SATA controller as well and that goes less smoothly. Said controller is found in iommu group 10

### Group 10 ### 00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)

When starting a guest in libvirt with the SATA controller marked for passthrough results in

Error starting domain: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=00:1f.2,bus=root.1: vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,bus=root.1: vfio: failed to get group 10 qemu-system-x86_64: -device vfio-pci,host=00:1f.2,bus=root.1: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,bus=root.1: Device 'vfio-pci' could not be initialized

The acs override patch was applied against the 3.14 sources from which the running kernel was built (doublechecked that by applying it to the same source tree again).

kvmhost-2:/home/user/Desktop # uname -r 3.14.4-vfio-acs-1.gbebeb6f-desktop kvmhost-2:/usr/src/linux-3.14.4-vfio-acs-1.gbebeb6f # patch -Np1 -i override_for_missing_acs_capabilities.patch patching file Documentation/kernel-parameters.txt Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file Documentation/kernel-parameters.txt.rej patching file drivers/pci/quirks.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/pci/quirks.c.rej

The boot loader has a flag set as per an earlier post:

intel_iommu=on pcie_acs_override=downstream

I'm completely out of ideas. Any and all feedback are much appreciated. Thanks.

Offline

#2011 2014-06-10 01:17:31

apporc
Member
Registered: 2014-06-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Error starting domain: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=00:1f.2,bus=root.1: vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

I have met this error several times.  I patched the acs patch on 3.14.5. I think it just won't work. Because that patch may not be applicable to the kernel version you used.
For now, kernel 3.15 is ok. It seems like that acs commit is got applied.

Last edited by apporc (2014-06-10 01:18:28)

Offline

#2012 2014-06-10 01:20:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
### Group 10 ### 00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)

When starting a guest in libvirt with the SATA controller marked for passthrough results in

qemu-system-x86_64: -device vfio-pci,host=00:1f.2,bus=root.1: vfio: error, group 10 is not viable...

The acs override patch was applied against the 3.14 sources from which the running kernel was built (doublechecked that by applying it to the same source tree again).
...
The boot loader has a flag set as per an earlier post:

intel_iommu=on pcie_acs_override=downstream

I'm completely out of ideas. Any and all feedback are much appreciated. Thanks.

I won't advise it, but there are more options to pcie_acs_override than just downstream.  See https://lkml.org/lkml/2013/5/30/513  IMHO, unless you have a specific need, assigning a SATA controller is more trouble than it's worth, especially when you can't guarantee device isolation.  virtio-blk provides plenty of performance when backed by a sufficiently fast disk or ssd.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2013 2014-06-10 01:31:53

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I won't advise it, but there are more options to pcie_acs_override than just downstream.  See https://lkml.org/lkml/2013/5/30/513  IMHO, unless you have a specific need, assigning a SATA controller is more trouble than it's worth, especially when you can't guarantee device isolation.  virtio-blk provides plenty of performance when backed by a sufficiently fast disk or ssd.

Thanks Alex for your fast reply. I'll have a look at those options. Agree with virtio-blk offering sufficient performance, however in this particular case I need to passthrough the SATA controller to a guest running UnRAID (a networked storage server product). I have this running on another box that is on kernel 3.11 and QEMU 1.6.2 using PCI assign.

Offline

#2014 2014-06-10 01:33:59

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apporc wrote:

I have met this error several times.  I patched the acs patch on 3.14.5. I think it just won't work. Because that patch may not be applicable to the kernel version you used.
For now, kernel 3.15 is ok. It seems like that acs commit is got applied.

Thanks! That would be good. I'll give kernel 3.15 a try tomorrow and report back.

Offline

#2015 2014-06-10 08:21:29

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just found out something unpleasant sniffing around my windows nvidia drivers and I notice it says I'm using "pci-express x4" turns out the secondary PCI-Express x16 slot on most motherboards is actually just x4 (read the small letters right?) the performance implication as far as I understand is that x16 has more bandwidth capabilites than x4 and as such this probably really borks the performance of high end graphics cards. As even old studies would suggest. I also hear that using the extra PCI-E slot will reduce the bandwidth of the primary slot (i.e. drop it from x16 to x12) but I haven't confirmed any of this yet. Chances are you have this same problem too unless you have like a really expensive high end motherboard. Mine is like medium end.

This whole thing is kinda dubious and I'm not sure what to think about it, so I asked. Only workaround for this problem I can think of is passing through the primary card which xen should be capable of but it's a lot harder to get working than this. Another workaround might be to use an intergrated gpu on the host (unless the primary gpu can actually be passed through instead of the secondary one)

Last edited by rabcor (2014-06-10 08:27:35)

Offline

#2016 2014-06-10 08:39:03

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor, I think it is possible to pass through primary card. Do need your host to use any GPU at all, or are you happy to let it run headless (and perhaps present its GUI over the network using tools such as xvnc or nomachine)?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#2017 2014-06-10 08:45:26

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lol of course I need my host to use a GPU at all, otherwise what would be the point in all of this? I'd just boot Windows and skip linux, but I want to be doing everything except gaming on Linux.

Last edited by rabcor (2014-06-10 08:46:34)

Offline

#2018 2014-06-10 09:26:36

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor maybe it is possible to use second GPU for host. after all if windows can switch to second GPU on boot then linux should be able to do this even more so. sideeffect of that would be boot messages showing on first gpu early on boot including grub menus. if i were you i would try passing through first gpu same way like you do with second and see what happens. it might work.

Offline

#2019 2014-06-10 10:04:27

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Lol of course I need my host to use a GPU at all, otherwise what would be the point in all of this? I'd just boot Windows and skip linux, but I want to be doing everything except gaming on Linux.

Dunno, I am so used to Linux command line over ssh, that I do not care to see its GUI at all. I guess I am not the only one.

Still, Linux should work with second GPU, I think the challenge is to:
1. get vgacon to give away primary GPU before you start the guest
2. get X to ignore primary GPU

Both should be doable, I think 915 patch might help with 1.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#2020 2014-06-10 10:08:09

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

I have met this error several times.  I patched the acs patch on 3.14.5. I think it just won't work. Because that patch may not be applicable to the kernel version you used.
For now, kernel 3.15 is ok. It seems like that acs commit is got applied.

Built a kernel 3.15 with only the i915 VGA arbiter patch applied and have the same result-

vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

. There is a 3.15 version of the ACS override patch in linux-mainline.tar.gz linked on the first post and I'll try that next. Thanks.

Offline

#2021 2014-06-10 12:19:05

bpye
Member
Registered: 2014-06-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is libvirt/virsh viable for running a VM with VGA passthrough or does it lack some of the configuration settings? I ask because it'd be nice to have my VM in a deamon, and hotplugging say USB devices without having to have qemu stay open would be nice.

Offline

#2022 2014-06-10 12:27:10

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpye wrote:

Is libvirt/virsh viable for running a VM with VGA passthrough or does it lack some of the configuration settings? I ask because it'd be nice to have my VM in a deamon, and hotplugging say USB devices without having to have qemu stay open would be nice.

Yes it is. You can convert a QEMU command line invocation to libvirt xml using

virsh domxml-from-native qemu-argv <qemu command line invocation>

It does not always work straight out of the box though, I've had difficulties with defining disks for example. More information can be found on http://libvirt.org/drvqemu.html#imex. I believe there are some examples in this thread of libvirt xml that you can use to define a guest. The configuration settings that are not directly supported by libvirt can be passed to QEMU as in the following example

<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> <qemu:commandline>

edit: added url

Last edited by siddharta (2014-06-10 12:30:12)

Offline

#2023 2014-06-10 13:07:22

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
bpye wrote:

Is libvirt/virsh viable for running a VM with VGA passthrough or does it lack some of the configuration settings? I ask because it'd be nice to have my VM in a deamon, and hotplugging say USB devices without having to have qemu stay open would be nice.

Yes it is. You can convert a QEMU command line invocation to libvirt xml using

virsh domxml-from-native qemu-argv <qemu command line invocation>

It does not always work straight out of the box though, I've had difficulties with defining disks for example. More information can be found on http://libvirt.org/drvqemu.html#imex. I believe there are some examples in this thread of libvirt xml that you can use to define a guest. The configuration settings that are not directly supported by libvirt can be passed to QEMU as in the following example

<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> <qemu:commandline>

edit: added url

or like this:

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit gaming or other application using the libvirt API. --> <domain type='kvm'> <name>gaming</name> <uuid>4541f648-2ada-4268-9882-0f3d4f5100b1</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <loader>/home/novist/vm/bios.bin-1.7.2</loader> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Opteron_G5</model> <vendor>AMD</vendor> <topology sockets='1' cores='4' threads='2'/> <feature policy='require' name='perfctr_core'/> <feature policy='require' name='skinit'/> <feature policy='require' name='perfctr_nb'/> <feature policy='require' name='mmxext'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='vme'/> <feature policy='require' name='topoext'/> <feature policy='require' name='fxsr_opt'/> <feature policy='require' name='bmi1'/> <feature policy='require' name='ht'/> <feature policy='require' name='cr8legacy'/> <feature policy='require' name='ibs'/> <feature policy='require' name='wdt'/> <feature policy='require' name='extapic'/> <feature policy='require' name='osvw'/> <feature policy='require' name='nodeid_msr'/> <feature policy='require' name='tce'/> <feature policy='require' name='cmp_legacy'/> <feature policy='require' name='lwp'/> <feature policy='require' name='monitor'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdc'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda'/> <target dev='vdb' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw' cache='none'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='pci' index='1' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </controller> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='scsi' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='scsi' index='1' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:29:ae:89'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/> <video> <model type='vga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x13' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x13' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/random</backend> <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/> </rng> </devices> </domain>

Last edited by novist (2014-06-10 13:07:40)

Offline

#2024 2014-06-10 13:19:47

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

or like this:

Exactly, that's a very complete xml. Thanks. Didn't have a full xml handy.

Offline

#2025 2014-06-10 14:39:50

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
siddharta wrote:

I have met this error several times.  I patched the acs patch on 3.14.5. I think it just won't work. Because that patch may not be applicable to the kernel version you used.
For now, kernel 3.15 is ok. It seems like that acs commit is got applied.

Built a kernel 3.15 with only the i915 VGA arbiter patch applied and have the same result-

vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

There is a 3.15 version of the ACS override patch in linux-mainline.tar.gz linked on the first post and I'll try that next. Thanks.

I built two versions of the 3.15 kernel, one with and one without the override_acs patch from the archive linked in the first post. Both kernel sources have the i915_315 patch applied. I have two issues, broken VGA passthrough (working on 3.14) and not being able to passthrough onboard SATA.

Using kernel 3.15 I have host display corruption and non-working VGA passthrough (host with igfx, AMD HD6450 for passthrough) despite having applied the i915_315 VGA arbiter patch. I re-ran the patch against the source to be certain:

kvmhost-2:/usr/src/linux-3.15.0-desktop # patch -Np1 -i i915_315.patch patching file drivers/gpu/drm/i915/i915_dma.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/gpu/drm/i915/i915_dma.c.rej patching file drivers/gpu/drm/i915/i915_drv.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/gpu/drm/i915/i915_drv.h.rej patching file drivers/gpu/drm/i915/i915_params.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- saving rejects to file drivers/gpu/drm/i915/i915_params.c.rej patching file drivers/gpu/drm/i915/intel_display.c Reversed (or previously applied) patch detected! Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file drivers/gpu/drm/i915/intel_display.c.rej patching file drivers/gpu/drm/i915/intel_drv.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file drivers/gpu/drm/i915/intel_drv.h.rej patching file include/linux/vgaarb.h Reversed (or previously applied) patch detected! Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file include/linux/vgaarb.h.rej

As for passing through the onboard SATA controller I have no luck

- with either 3.14 and 3.15 kernels
- with or without the respective override_acs patch
- with kernel boot parameter pcie_acs_override= set to downstream/multifunction/id:8086:1e02 or any combination thereof.

I've also attempted to unbind the SATA controller from vfio-pci and use PCI assign as I've done on QEMU 1.6.2 and kernel 3.11, defined with libvirt using Virtual Machine Manager. The device is defined as

<hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev>

Surprisingly (to me) this fails with

Error starting domain: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 8 qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized

or the same error I have when attempting passthrough of this device using vfio. It would appear the PCI assign mechanism was abandoned in favor of vfio, is that correct? If so, is there a way to force use of PCI assignment?

I would really need to passthrough SATA and USB controllers, as USB per-device passthrough doesn't work for me due to a non-libusb enabled QEMU.

Please let me know if I can provide further information or take diagnostic steps. I'm not at all opposed to testing out new patches or using debug versions, consider me a guinea pig if so needed. Thanks all.

Last edited by siddharta (2014-06-10 14:40:30)

Offline

#2026 2014-06-10 14:41:51

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I get "vfio: error no iommu_group for device"
Tell me what logs you need to help me please smile

Offline

#2027 2014-06-10 14:54:08

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Hello,

I get "vfio: error no iommu_group for device"
Tell me what logs you need to help me please smile

1) Check whether IOMMU is properly enabled

dmesg | grep -e DMAR -e IOMMU

for Intel or

dmesg | grep AMD-Vi

for AMD. If there is no output it may not be enabled in BIOS (check if it is), may not be enabled for your kernel (ensure you have

intel_iommu=on

for Intel or

iommu=pt iommu=1

as kernel parameter. All this assuming your CPU and motherboard do support VT-d. Feel free to provide more detail and the output of dmesg.

2) Check whether your device is bound to vfio-pci.

ls /sys/bus/pci/drivers/vfio-pci | grep 0000

will list the devices currently bound to vfio-pci, confirm that your device is there. If it is not, have a look at the steps listed in the first post. Feel free to provide more detail on how you are binding your device.

That's all I can think of but maybe there's other more knowledgeable persons here that can offer more advice.

Offline

#2028 2014-06-10 15:08:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Hello,

I get "vfio: error no iommu_group for device"
Tell me what logs you need to help me please smile

Most importantly, check that your CPU supports an IOMMU.  If it's Intel, check the features on ark, http://ark.intel.com

Last edited by aw (2014-06-10 15:08:28)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2029 2014-06-10 15:50:30

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
ttouch wrote:

Hello,

I get "vfio: error no iommu_group for device"
Tell me what logs you need to help me please smile

1) Check whether IOMMU is properly enabled

dmesg | grep -e DMAR -e IOMMU

for Intel or

dmesg | grep AMD-Vi

for AMD. If there is no output it may not be enabled in BIOS (check if it is), may not be enabled for your kernel (ensure you have

intel_iommu=on

for Intel or

iommu=pt iommu=1

as kernel parameter. All this assuming your CPU and motherboard do support VT-d. Feel free to provide more detail and the output of dmesg.

2) Check whether your device is bound to vfio-pci.

ls /sys/bus/pci/drivers/vfio-pci | grep 0000

will list the devices currently bound to vfio-pci, confirm that your device is there. If it is not, have a look at the steps listed in the first post. Feel free to provide more detail on how you are binding your device.

That's all I can think of but maybe there's other more knowledgeable persons here that can offer more advice.

Yea, you're right. I got on dmesg after trying to vfio-bind: "vfio-pci: probe of 0000:02:00.0 failed with error -22"

I can't find if my CPU supports IOMMU (I am from mobile). It's i7 3820. I think it supports it tongue

EDIT: I booted with "intel_iommu=on" and the device binds. Now, I just get a blank screen in qemu. is that normal? (I use the test qemu command you provided)

EDIT2: I added a cdrom device (entry you provided) and still blank screen. I booted with this "kvm_intel.emulate_invalid_guest_state=0" bit nothing changed (kvm_intel is built in).

Last edited by ttouch (2014-06-10 16:19:02)

Offline

#2030 2014-06-10 16:43:56

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

I can't find if my CPU supports IOMMU (I am from mobile). It's i7 3820. I think it supports it tongue

EDIT: I booted with "intel_iommu=on" and the device binds. Now, I just get a blank screen in qemu. is that normal? (I use the test qemu command you provided)

Yes http://ark.intel.com/products/63698/Int … o-3_80-GHz i7 3820 supports VT-d.

Blank screen is kind of normal, extra magic dust is needed for SeaBIOS to output to passed-through card (I do not remember what it was). If you booted valid OS boot image, the screen should show it. I recommend some Linux live CD, these take care to initialise the right set of drivers.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#2031 2014-06-10 16:47:57

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

YES! IT WORKS! big_smile
thank you very much big_smile
I'll report if more errors appear

Offline

#2032 2014-06-10 17:22:37

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any way to pivot the keyboard and mouse between host and guest?
Ultimately with one of the function keys of my Razer blackwidow ultimate.

Offline

#2033 2014-06-10 17:45:06

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Is there any way to pivot the keyboard and mouse between host and guest?
Ultimately with one of the function keys of my Razer blackwidow ultimate.

Try Synergy http://synergy-project.org/ I'm using the server on my Linux host and client on Windows VM.

Offline

#2034 2014-06-10 17:53:25

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
ttouch wrote:

Is there any way to pivot the keyboard and mouse between host and guest?
Ultimately with one of the function keys of my Razer blackwidow ultimate.

Try Synergy http://synergy-project.org/ I'm using the server on my Linux host and client on Windows VM.

Thank you. I'll try it.

Also, can I boot my UEFI windows 8 installation (separate partition).

Offline

#2035 2014-06-10 18:19:54

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I'm trying to Passthrough 2 GPUS to 2 diffrent Vm's, here is my specification "this the aim of this PC":

Motherboard : Asus z97-A
CPU: i7-4790
VGA at PCI-E x8 1: GeForce 8500 GT
VGA at PCI-E x8 2: Radeon HD 4870

Ram: 16 Gb Kingston 1600 Mhz
OS: Linux Mint 17 Qiana
Kernel : 3.15 RC2.
The GPU's, are a bit old (Yet the radeon still support every new game out there!), looking to upgrade according to the success.

I saw a post mentioning the drop of speed for PCI-E 3, I need to note, yes that's true, if the first PCI-Ex3 was populated you will get x16 speed, if 2 PCI-Ex3 are populated you will get x8 the speed, I don't know about using the 3 slots, but the speed will either drop or it won't work, that's for my motherboard at least.

Both my CPU and motherboard support VT-d,
I have enabled the support for 1) "intel virtualization technology", 2) "VT-d" and forced 3) "intel graphic" so, I do not need to block the driver's in the host >> right ?
I plan to use RMD "Remote Desktop" for connecting to the Vm's through the network, so I do not need any USB passthrough.

now... I need some help to start passing the GPU's...

novist wrote:

Now i could make it all work with stock (from repos, no recompiling) software available:
Kernel: 3.15 (no patches)
Qemu 2.0.0
libvirt 1.2.2
virt-manager 1.0.1

CPU AMD FX(tm)-8350 Eight-Core Processor
GPU radeon R9 270X
Motherboard SABERTOOTH 990FX R2.0

This is on linux mint 17 (based on ubuntu 14.04 LTS). For libvirt i used default pc chipset (no manual xml editing!). Using emulated GPU for booting, OS switches to physical GPU later on during boot. Good to finally see this work without much hackery. smile

Could you please detail the booting step ?

What I'v done so far installing the independents ( Qemu 2.0.0, libvirt, virt-manager ), and asking above.

Offline

#2036 2014-06-10 18:33:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:
novist wrote:

Now i could make it all work with stock (from repos, no recompiling) software available:
Kernel: 3.15 (no patches)
Qemu 2.0.0
libvirt 1.2.2
virt-manager 1.0.1

CPU AMD FX(tm)-8350 Eight-Core Processor
GPU radeon R9 270X
Motherboard SABERTOOTH 990FX R2.0

This is on linux mint 17 (based on ubuntu 14.04 LTS). For libvirt i used default pc chipset (no manual xml editing!). Using emulated GPU for booting, OS switches to physical GPU later on during boot. Good to finally see this work without much hackery. smile

Could you please detail the booting step ?

What I'v done so far installing the independents ( Qemu 2.0.0, libvirt, virt-manager ), and asking above.

Please do not expect novist's setup to work for you.  Some people have had luck using secondary device passthrough with AMD cards, others have not.  It will certainly not work for your NVIDIA card unless you plan to upgrade to a Quadro card.  The most successful method is VGA assignment, for which you'll at a minimum need to patch your kernel with the i915 patch.  You may also need the ACS override patch unless you can configure your cards to be in separate IOMMU groups by switching to different slots.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2037 2014-06-10 18:40:03

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
abdullah wrote:
novist wrote:

Now i could make it all work with stock (from repos, no recompiling) software available:
Kernel: 3.15 (no patches)
Qemu 2.0.0
libvirt 1.2.2
virt-manager 1.0.1

CPU AMD FX(tm)-8350 Eight-Core Processor
GPU radeon R9 270X
Motherboard SABERTOOTH 990FX R2.0

This is on linux mint 17 (based on ubuntu 14.04 LTS). For libvirt i used default pc chipset (no manual xml editing!). Using emulated GPU for booting, OS switches to physical GPU later on during boot. Good to finally see this work without much hackery. smile

Could you please detail the booting step ?

What I'v done so far installing the independents ( Qemu 2.0.0, libvirt, virt-manager ), and asking above.

Please do not expect novist's setup to work for you.  Some people have had luck using secondary device passthrough with AMD cards, others have not.  It will certainly not work for your NVIDIA card unless you plan to upgrade to a Quadro card.  The most successful method is VGA assignment, for which you'll at a minimum need to patch your kernel with the i915 patch.  You may also need the ACS override patch unless you can configure your cards to be in separate IOMMU groups by switching to different slots.

From the first post "3.15 includes acs override patch, i935 vga arbiter fixes", the patch is already applied, isn't it ?
it only needs to work,

Offline

#2038 2014-06-10 18:43:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

From the first post "3.15 includes acs override patch, i935 vga arbiter fixes", the patch is already applied, isn't it ?
it only needs to work,

No, that comment is really poorly worded.  It's describing the tgz file, not what's upstream.  Linux 3.15 proper does not include the ACS override patch or the i915 VGA arbiter patch.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2039 2014-06-10 19:08:37

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
abdullah wrote:

From the first post "3.15 includes acs override patch, i935 vga arbiter fixes", the patch is already applied, isn't it ?
it only needs to work,

No, that comment is really poorly worded.  It's describing the tgz file, not what's upstream.  Linux 3.15 proper does not include the ACS override patch or the i915 VGA arbiter patch.

Oh...

how about my other configuration ? Since I forced the using integrated intel HD4600, Do I still need to block the drivers for the other cards ?

Offline

#2040 2014-06-10 19:14:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:
aw wrote:
abdullah wrote:

From the first post "3.15 includes acs override patch, i935 vga arbiter fixes", the patch is already applied, isn't it ?
it only needs to work,

No, that comment is really poorly worded.  It's describing the tgz file, not what's upstream.  Linux 3.15 proper does not include the ACS override patch or the i915 VGA arbiter patch.

Oh...

how about my other configuration ? Since I forced the using integrated intel HD4600, Do I still need to block the drivers for the other cards ?

I don't really know what you mean by "forced intel graphic", but I assume that just means setting Intel as the primary display for the host.  In that case, yes you still need to prevent nouveau/radeon from binding to the devices you intend to assign.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2041 2014-06-10 19:21:01

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

...
Using kernel 3.15 I have host display corruption and non-working VGA passthrough (host with igfx, AMD HD6450 for passthrough) despite having applied the i915_315 VGA arbiter patch.
...

this brief description reminds me of my situation:
https://bbs.archlinux.org/viewtopic.php … 5#p1419025
https://bbs.archlinux.org/viewtopic.php … 2#p1419322
https://bbs.archlinux.org/viewtopic.php … 8#p1420298

Offline

#2042 2014-06-10 19:41:13

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

I built two versions of the 3.15 kernel, one with and one without the override_acs patch from the archive linked in the first post. Both kernel sources have the i915_315 patch applied. I have two issues, broken VGA passthrough (working on 3.14) and not being able to passthrough onboard SATA.

I had missed

aw wrote:

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.

so the i915 issue is taken care of. Still in the dark regarding the other device passthrough as per post 2025. Any suggestions to move on with my ACS issues are greatly appreciated.

Offline

#2043 2014-06-10 19:46:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
siddharta wrote:

I built two versions of the 3.15 kernel, one with and one without the override_acs patch from the archive linked in the first post. Both kernel sources have the i915_315 patch applied. I have two issues, broken VGA passthrough (working on 3.14) and not being able to passthrough onboard SATA.

I had missed

aw wrote:

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.

so the i915 issue is taken care of. Still in the dark regarding the other device passthrough as per post 2025. Any suggestions to move on with my ACS issues are greatly appreciated.

You haven't said why using multifunction or ID options to the override failed.  What ID did you use?  What was the resulting change in /sys/kernel/iommu_groups?

Last edited by aw (2014-06-10 20:38:45)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2044 2014-06-10 20:51:31

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You haven't said why using multifunction of id options to the override failed.  What ID did you use?  What was the resulting change in /sys/kernel/iommu_groups?

The device I'm trying to passthrough is

00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04) Subsystem: ASRock Incorporation Motherboard [1849:1e02] Kernel driver in use: vfio-pci

1) with kernel 3.15.0 (patched with i915 and override_acs patches) and kernel parameters intel_iommu=on i915.enable_hd_vgaarb=1 (and therefore the override_acs disabled) the iommu groups I'm looking at for passthrough are

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] ### Group 10 ### 00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)

2) with pcie_acs_override=downstream the graphics card devices break out of iommu group 1 and move to group 11. All other groups remain as they were.

3) with pcie_acs_override=multifunction the result is the same as in 1)

4) with pcie_acs_override=id:1849:1e02 the result is again the same as in 1)

Hope this helps. Thanks again.

Last edited by siddharta (2014-06-10 20:51:50)

Offline

#2045 2014-06-10 21:02:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
aw wrote:

You haven't said why using multifunction of id options to the override failed.  What ID did you use?  What was the resulting change in /sys/kernel/iommu_groups?

The device I'm trying to passthrough is

00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04) Subsystem: ASRock Incorporation Motherboard [1849:1e02] Kernel driver in use: vfio-pci

1) with kernel 3.15.0 (patched with i915 and override_acs patches) and kernel parameters intel_iommu=on i915.enable_hd_vgaarb=1 (and therefore the override_acs disabled) the iommu groups I'm looking at for passthrough are

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] ### Group 10 ### 00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)

2) with pcie_acs_override=downstream the graphics card devices break out of iommu group 1 and move to group 11. All other groups remain as they were.

3) with pcie_acs_override=multifunction the result is the same as in 1)

4) with pcie_acs_override=id:1849:1e02 the result is again the same as in 1)

Hope this helps. Thanks again.

Lemme guess, "lspci -vvv -s 00:1f." doesn't show a PCIe capability for any of those devices, right?  So as it is, the acs override patch isn't going to split those devices, sorry.  You can hack up the patch some more or use pci-assign, but really the only supportable option is to get a separate SATA card to install.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2046 2014-06-10 21:13:07

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Lemme guess, "lspci -vvv -s 00:1f." doesn't show a PCIe capability for any of those devices, right?  So as it is, the acs override patch isn't going to split those devices, sorry.  You can hack up the patch some more or use pci-assign, but really the only supportable option is to get a separate SATA card to install.

It does for the SATA controller (unless I'm reading this wrong) but not for the other devices. All devices in the iommu group would need to have PCIe capability? My understanding here is obviously limited. smile

00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller (rev 04) Subsystem: ASRock Incorporation Device 1e4a Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 19 Region 0: I/O ports at f0b0 [size=8] Region 1: I/O ports at f0a0 [size=4] Region 2: I/O ports at f090 [size=8] Region 3: I/O ports at f080 [size=4] Region 4: I/O ports at f060 [size=32] Region 5: Memory at f7f12000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: fee002d8 Data: 0000 Capabilities: [70] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004 Capabilities: [b0] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: vfio-pci 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) Subsystem: ASRock Incorporation Motherboard Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 18 Region 0: Memory at f7f11000 (64-bit, non-prefetchable) [size=256] Region 4: I/O ports at f040 [size=32] Kernel modules: i2c_i801

edit: yes it does look like I may be reading this wrong, when comparing to the output of lspci -vvv -s 01:00 which are the graphics card devices -

Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00

edit 2: PCI assign doesn't work for me as it did in QEMU 1.6.2 and kernel 3.11. I get the same complaint about the iommu group not being viable. Has PCI assign been removed as a passthrough mechanism in favor of vfio?

Last edited by siddharta (2014-06-10 21:19:24)

Offline

#2047 2014-06-10 21:45:22

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

edit 2: PCI assign doesn't work for me as it did in QEMU 1.6.2 and kernel 3.11. I get the same complaint about the iommu group not being viable. Has PCI assign been removed as a passthrough mechanism in favor of vfio?

When attempting PCI assign of the SATA controller with the following guest definition

<domain type='kvm'> <name>vm-storage-test</name> <uuid>850b3562-bb54-d1df-2071-1c0ebb30e9ee</uuid> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/vm-storage-2.vhd'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <interface type='bridge'> <mac address='52:54:00:40:4f:8f'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> </domain>

the error is the same as when attempting to passthrough the device with vfio.

Error starting domain: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 10 qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized

Very weird (to me). The SATA controller is no longer bound to vfio-pci at this time.

Last edited by siddharta (2014-06-10 21:46:20)

Offline

#2048 2014-06-10 21:50:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

the error is the same as when attempting to passthrough the device with vfio.

Error starting domain: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 10 qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=00:1f.2,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized

Very weird (to me). The SATA controller is no longer bound to vfio-pci at this time.

You're still using vfio, see the driver option of the hostdev device http://libvirt.org/formatdomain.html#el … tDevSubsys


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2049 2014-06-10 22:15:01

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You're still using vfio, see the driver option of the hostdev device http://libvirt.org/formatdomain.html#el … tDevSubsys

See, not weird at all. smile. Thanks. With

<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev>

the SATA controller is peachy. Now to see about USB. Thanks Alex, if you're ever in Las Palmas I'd gladly stand you a beer.

Offline

#2050 2014-06-10 22:28:58

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

rabcor maybe it is possible to use second GPU for host. after all if windows can switch to second GPU on boot then linux should be able to do this even more so. sideeffect of that would be boot messages showing on first gpu early on boot including grub menus. if i were you i would try passing through first gpu same way like you do with second and see what happens. it might work.

Of course it's possible, I'm doing it.

Bronek wrote:

Dunno, I am so used to Linux command line over ssh, that I do not care to see its GUI at all. I guess I am not the only one.

Still, Linux should work with second GPU, I think the challenge is to:
1. get vgacon to give away primary GPU before you start the guest
2. get X to ignore primary GPU

PCI-stub works, X won't even touch it (as in it doesn't show up in RandR providers and whatnot).
As for the bootup, I can set in the BIOS which PCI slot to use as primary.

I'm passing a 660Ti and using a 640 for the host. I'll write a detailed post later.

siddharta wrote:

Thanks Alex, if you're ever in Las Palmas I'd gladly stand you a beer.

I'll second that. Whenever you're in Hungary :--D

Offline

#2051 2014-06-11 01:33:55

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Apart from that benchmark I linked to before, just for fun (I've always wanted to know the below) I added in an OpenGL windows vs OpenGL linux benchmark against DirectX11 on Unigine, I will blame Unigine for having poorly optimized OpenGL code though ad the max FPS difference on windows was roughly 20 FPS between DX11 and OpenGL. (Writing OpenGL is no joke, I'm not surprised) I wanted to test the mettle of Windows vs Linux nvidia proprietary drivers in OpenGL performance matching the settings as closely as I could, as it turns out the performance was significantly better on Linux than in Windows (unsurprising as OpenGL should be the main focus of the linux drivers whereas DX should be the main focus of the windows drivers) anyways, here we go (All running on the x16 bus)

Linux (3.14.5) driver version: 337.25
Windows (7 x64) driver version: 337.88
Unigine Heaven Settings: Extreme @1920x1080 Fullscreen

Note that while launching Unigine from a terminal in linux spat out for me the OpenGL version (3.2) I have no such output in windows, so I am only assuming windows uses 3.2 as well. I might be wrong however and it might be using 2.0 which would explain the (dramatic) performance differences between windows and linux when using OpenGL. Also note that they claim on the official website to be using 4.0

Nvidia GTX-670 Linux (OpenGL 3.2)

Min FPS:12.1 Max FPS:65.5 FPS:24.9 Score:627

Nvidia GTX-670 Windows (OpenGL 3.2)

Min FPS:6.2 Max FPS:51.6 FPS:20.6 Score:518

Nvidia GTX-670 Windows (DirectX11)

Min FPS:17.1 Max FPS:69.2 FPS:32.0 Score:807

It is worth noting that for the above scores on Linux I disabled flipping in the nvidia drivers. I ran it first with flipping enabled but this resulted in some ugly (what appeared to be) delayed frames which was literally driving me nuts, there was very minimal performance difference with flipping disabled/enabled however as the score and avg FPS were the exact same. (Min and Max FPS differed only slightly as max was 66.4 and min was 6.9 with flipping enabled) however take this as a tip from me to keep flipping disabled on Linux wink

It is also worth noting that the above benchmarks are probably not good DirectX vs OpenGL benchmarks since there is no telling which code was more optimized in Unigine Heaven (but by the looks of it DX11 was better optimized than OpenGL) Unigine also only seems to use OpenGL 3.2 rather than 4.4 or similar (I assume this was done so it could work on Mac as well but naturally the performance of GL will take a hit from this)

Hopefully I can later add to that the performance of the 670 as it's been passed through to windows rather than from a windows boot.


Update:
GTX 550-Ti results, these were done on the x4 bus but the performance difference between x4 and x16 on this card was so trivial it doesn't really matter (might be like 1FPS difference tops compared to x16)

Unigine Settings: Basic @1920x1080 Fullscreen

GTX 550-Ti Linux (OpenGL 3.2)

FPS:21.0 Score:530 Min FPS:8.7 Max FPS:34.6

GTX 550-Ti Windows (OpenGL 3.2)

FPS:13.9 Score:351 Min FPS:6.4 Max FPS:29.1

GTX 550-Ti Windows (DirectX 11)

FPS:21.2 Score:534 Min FPS:9.8 Max FPS:37.8

What shocked me with these results is that the Linux OpenGL benchmark was actually on par with the DX11 benchmark, I was doing these tests (a few comments below) because I had a suspicion everything wasn't all right in linux on the x4 bus (i.e. I was getting strangely mismatching results between x16 and x4 on there) but that seems to have been little more than a fluke because I was after all running this benchmark with settings way beyond the capabilities of this card (which is how I ended up trying settings the card could actually handle).

The fact that it matched up to DX11 was more than enough to convince me that everything is in fact performing outstandingly well in Linux and I was just worrying over nothing (like I said, a fluke).

It is also interesting to note that on Windows (in both benchmarks) the card would run at 72°C but on Linux the heat would never exceed 60°C. (Meaning it was performing on par with DX11 at lower temperatures)

There are only two reasons I can think of for why OpenGL performed on par with DX11 in this test and why it didn't in the 670 test.

1: The Fermi architecture works better for the Nvidia proprietary drivers on Linux (I find this highly unlikely though as when Nvidia started working at it's hardest ever on the linux drivers, the Kepler architecture which the 600 and 700 series use was already out)
2: OpenGL 3.2 is on par with DX11 in most areas excluding Tessellation which is disabled when the Basic preset is used (This is much likelier)

Last edited by rabcor (2014-06-11 14:46:27)

Offline

#2052 2014-06-11 02:08:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Follow up on that, I have not yet tried to pass through the primary card to a virtual machine (it wil be fun to see what happens) but I did decide that no one was giving me a proper answer for my case (outdated benchmarks and whatnot) so I simply tried running a benchmark myself. These are the results. I imagine that the less bandwidth your card is capable of using (for example the 670 has 192.2GB/s bandwidth) the less noticable the difference will be, where the difference starts to be noticable remains to be seen though.

I'm not sure I agree with your hypothesis that installing the card on a x4 electrical slot gives you more overclocking headroom.  I think it shows that that particular test was bus limited and the GPU was data starved and scaling back frequency.  Overclocking the GPU isn't likely to equalize the difference and a GPU-bound workload is still likely to generate the same heat load.  Also, a GTX670 does have a PCIe 3.0 interface (http://www.geforce.com/hardware/desktop … ifications), so if you're limited to PCIe 2.0, it's because of your motherboard/processor combination.

EDIT: ark confirms your i7-2600 only supports PCIe 2.0, I didn't see your motherboard specs in the post history to lookup those specs though

Last edited by aw (2014-06-11 02:15:15)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2053 2014-06-11 02:33:13

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an i7-2600 so I guess that's why. I slightly modified that answer to clarify that the overclocking thing was just me guessing. (Thanks for looking into this for me btw)

I have this board really though I just saw it said "PCI Express 3.0" on the board which was how I found out.

It's a shame that my CPU doesn't support pci-express 3.0 >_< god damn that pisses me off considering I would get a lot better performance out of that I think (which would further increase my need to use the primary slot)

As an update on my progress with using the primary card.

Running pci-stub on it works BUT after booting the console will use the stubbed card as it is in the primary slot and what GRUB used. To force X to ignore the card I had to add this to a file in xorg.conf.d/

Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GTX 550 Ti" BusID "PCI:2:0:0" EndSection

I am pretty sure only the "Identifier" "Driver" and "BusID" values are important but I kept the others just for show. This will make the nvidia drivers in X think that the 550-Ti on bus 2:0:0 is my primary card, however when I try to start X with the 670 stubbed currently everything will freeze up. The things I'll try to do to fix this is to install that VGA_ARB patch for the nvidia drivers first, if that's a no go I will enable ACS Override and if that's a no go I'm running short on ideas.

Last edited by rabcor (2014-06-11 02:38:22)

Offline

#2054 2014-06-11 02:43:05

apporc
Member
Registered: 2014-06-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, siddharta.
To be clear, below is my information:
os: Linux Mint 17
Kernel: 3.15 (with no patches, no i915, no acs_override.) Link: http://kernel.ubuntu.com/~kernel-ppa/ma … c8-utopic/
Qemu: 2.0.0
libvirt: 1.2.2

CPU: Intel(R) Core(TM) i7-4770
GPU: radeon 7870
Motherboard: Asus Z87K

If your hardware is different from mine, it may be the reason why it don't work.
Use the software and hardware above, i almost did nothing, and it just worked.
I used to use pci_stub or radeon blacklist, vfio_bind and config .xml manually like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <boot dev='hd'/> </os> <qemu:commandline> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> </qemu:commandline>

But now all of above turned out to be not necessary.
The kernel/qemu i used now is all from linux mint' repository, with no patch.
I cleared the pci_stub in my kernel command line, just left "intel_iommu=on".
I unblacklisted radeon and fglrx in my /etc/modprobe.d/blacklist.conf.
And i didn't vfio_bind the radeon card on boot, i just removed /etc/init/vfio-bind.conf which i used at first:

# vfio-bind - Binds devices to vfio-pci # description "vfio binding devices" start on runlevel [2345] stop on runlevel [!2345] exec /usr/local/bin/vfio-bind 0000:04:00.0 0000:04:00.1

The configuration i am still using is :

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf #/etc/libvirt/qemu.conf, add: user = "root" group = "root" clear_emulator_capabilities = 0 ls -1 /dev/vfio #For ever number that appears, ensure its full path appears in the cgroup_device_acl configuration parameter. For example, mine looks like so: cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/8" ]

After above, I just used virt-manager to create my win7 vm. I totally use virt-manager to add vga card and usb device to my vm.
Here is my xml:

<domain type='kvm' id='2'> <name>win7</name> <uuid>6cb6904a-ff53-6038-8718-c82a39eeab19</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <vendor>Intel</vendor> <topology sockets='1' cores='2' threads='2'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='abm'/> <feature policy='require' name='ht'/> <feature policy='require' name='acpi'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='f16c'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='monitor'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/win7.img'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/apporc/Software/ISO/virtio-win-0.1-65.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='pci' index='1' model='pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='52:54:00:5a:a6:ef'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x17ef'/> <product id='0x6047'/> <address bus='3' device='4'/> </source> <alias name='hostdev3'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </memballoon> </devices> <seclabel type='none'/> </domain>

The method virt-manager uses is now vfio_pci too, not pci_assign.

Here is another link i used to refer to: http://www.firewing1.com/howtos/fedora- … hrough-kvm.

Good luck!

Offline

#2055 2014-06-11 02:46:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I have an i7-2600 so I guess that's why. I slightly modified that answer to clarify that the overclocking thing was just me guessing. (Thanks for looking into this for me btw)

I have this board really though I just saw it said "PCI Express 3.0" on the board which was how I found out.

It's a shame that my CPU doesn't support pci-express 3.0 >_< god damn that pisses me off considering I would get a lot better performance out of that I think (which would further increase my need to use the primary slot)

Z77 only supports PCIe 2.0 as well - http://ark.intel.com/products/64024/Intel-BD82Z77-PCH

I think the PCIe 3.0/2.0 reference in the motherboard spec is referring the the PEG express slot, ie. the one hosted from the CPU.  So if you were to install a 3rd gen Core processor, you'd have PCIe 3.0 for that one slot.

As an update on my progress with using the primary card.

Running pci-stub on it works BUT after booting the console will use the stubbed card as it is in the primary slot and what GRUB used. To force X to ignore the card I had to add this to a file in xorg.conf.d/

Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GTX 550 Ti" BusID "PCI:2:0:0" EndSection

I think requiring BusID is pretty standard for NVIDIA anytime you stray from a standard config.

I am pretty sure only the "Identifier" "Driver" and "BusID" values are important but I kept the others just for show. This will make the nvidia drivers in X think that the 550-Ti on bus 2:0:0 is my primary card, however when I try to start X with the 670 stubbed currently everything will freeze up. The things I'll try to do to fix this is to install that VGA_ARB patch for the nvidia drivers first, if that's a no go I will enable ACS Override and if that's a no go too I will start looking into third party solutions (like vgacon)

The host doesn't care about ACS stuff, so that's surely not going to do anything.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2056 2014-06-11 03:06:14

apporc
Member
Registered: 2014-06-09
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About the noise of my Radeon 7870 card. It is really very loud.
Before i tried the guide of this thread, i used fglrx driver, and Radeon 7870 card didn't make noise.
I know if i used radeon driver, the noise will be there. Maybe just as they said, radeon driver doesn't handle the frequency well.

But after pci-stub, why Radeon 7870 is still noisy. I thought when Radeon is got stubed, nobody is using it, and nobody using it should lead to "no noise".
Surpringly it's not. No mater i blacklist radeon/fglrx module or pci-stub Radeon, the noise is all there. Until i boot my guest win7 vm, then win7's ATI driver make it quiet.
My question is, after host os(linux) booted, before guest win7 booted, Is there a way to make Radeon 7870 quiet?
I hope i don't need to keep my the guest win7 always booted, it's only one game machine.

Any suggestion is apreciated.
Thanks in advance.

Last edited by apporc (2014-06-11 03:08:32)

Offline

#2057 2014-06-11 04:10:20

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rabcor, is your "4x PCIE" electrical slot directly linked to the CPU or does it go through the the Z77 PCH? If it goes through the controller hub, it would explain why you're noticing a huge performance because you're passing through what used to be the south bridge. DMI 2.0 is limited to 20gbit/s. If it was directly linked to the CPU, I doubt you'd see a substantial difference in benchmarks. If your motherboard supports SLI or Crossfire, use the slots indicated for this by your manufacturer because they are the ones with direct access to the northbridge that is part of the CPU now.

I've seen the effect this has on graphic cards and it's quite noticeable with simple desktop compositing (Aero, Compiz, Mutter).

For reference, see : http://www.anandtech.com/show/5728/inte … nd-biostar

Last edited by mv0lnicky (2014-06-11 06:11:18)

Offline

#2058 2014-06-11 04:53:23

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA_ARB patch did not seem to help (I suspect it is needed either way) is it normal that xorg.log will say this when the VGA arb patch is enabled?

[ 10.381] (WW) NVIDIA(0): [DRI2] Direct rendering is not supported when VGA arb is necessary for the device [ 10.381] (II) NVIDIA(0): The X server will not be able to send the VDPAU driver name to [ 10.381] (II) NVIDIA(0): libvdpau. [ 10.381] (--) RandR disabled [ 10.381] (II) Found 2 VGA devices: arbiter wrapping enabled

I thought it disabled arbiter wrapping entirely.

What I think I need is a method for the kernel to load my secondary GPU as if it were my primary one. Think it's possible?

Edit: I found a way, my motherboard supports selecting a main display, it offers

iGPU PCIe PCI

Choosing PCI resulted in the motherboard reading my GTX 550 rather than my 670 as the primary card even if it is in the x4 bus. It would be neat to have an alternative way to do this in case someone has to deal with a motherboard that doesn't support doing this. (And I still need an answer to that VGA Arb question please)

Last edited by rabcor (2014-06-11 05:01:43)

Offline

#2059 2014-06-11 05:06:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apporc wrote:

About the noise of my Radeon 7870 card. It is really very loud.
Before i tried the guide of this thread, i used fglrx driver, and Radeon 7870 card didn't make noise.
I know if i used radeon driver, the noise will be there. Maybe just as they said, radeon driver doesn't handle the frequency well.

But after pci-stub, why Radeon 7870 is still noisy. I thought when Radeon is got stubed, nobody is using it, and nobody using it should lead to "no noise".
Surpringly it's not. No mater i blacklist radeon/fglrx module or pci-stub Radeon, the noise is all there. Until i boot my guest win7 vm, then win7's ATI driver make it quiet.
My question is, after host os(linux) booted, before guest win7 booted, Is there a way to make Radeon 7870 quiet?
I hope i don't need to keep my the guest win7 always booted, it's only one game machine.

Sadly, that's probably the solution.  Without a driver that knows how to do power management, the cooling device on the card just runs at a default speed.  Ideally devices would turn the fan off (like AMD ZeroCore) or at least to a minimum level when put into the D3 power state, but they don't (I've tried).  It's probably still worthwhile to add some basic power management to vfio-pci to put devices in D3 while not in use, it saves 2-3 Watts on my system, but it's still quieter to run a guest with a driver that can silence the fan.

Last edited by aw (2014-06-11 05:09:23)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2060 2014-06-11 06:05:35

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Edit: I found a way, my motherboard supports selecting a main display, it offers

iGPU PCIe PCI

Choosing PCI resulted in the motherboard reading my GTX 550 rather than my 670 as the primary card even if it is in the x4 bus. It would be neat to have an alternative way to do this in case someone has to deal with a motherboard that doesn't support doing this. (And I still need an answer to that VGA Arb question please)

PCI BIOS setting pretty much confirms what I said in my previous post. Your 4x slot is most likely routed through DMI 2.0. I don't think that the results of your research you linked in one of your previous post are accurate, seeing as you're hitting a major bottleneck in the chipset configuration. And if you had PCIE3.0 slots, or even 4.0 slots, you'd have the same problem.

How many physical 16x PCIE ports are present on your motherboard?

Last edited by mv0lnicky (2014-06-11 06:14:24)

Offline

#2061 2014-06-11 06:40:18

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice? smile

Offline

#2062 2014-06-11 06:53:45

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice? smile

You mean like underscan?

Offline

#2063 2014-06-11 08:21:04

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:
rabcor wrote:

Edit: I found a way, my motherboard supports selecting a main display, it offers

iGPU PCIe PCI

Choosing PCI resulted in the motherboard reading my GTX 550 rather than my 670 as the primary card even if it is in the x4 bus. It would be neat to have an alternative way to do this in case someone has to deal with a motherboard that doesn't support doing this. (And I still need an answer to that VGA Arb question please)

PCI BIOS setting pretty much confirms what I said in my previous post. Your 4x slot is most likely routed through DMI 2.0. I don't think that the results of your research you linked in one of your previous post are accurate, seeing as you're hitting a major bottleneck in the chipset configuration. And if you had PCIE3.0 slots, or even 4.0 slots, you'd have the same problem.

How many physical 16x PCIE ports are present on your motherboard?

These are my motherboard specs, manual is here too. I have very little idea what you're talking about.

But I did try running a benchmark on the 550-Ti too (I added this to superuser too) and in that case the performance difference was trivial (less than 1FPS) but my results seem like they are accurate from my perspective, matches what I expected before benchmarking (except that I was expecting the 670 to do significantly worse than it did on the x4 bus)

My understanding is that the x4 bus is connected to the CPU by first going through the x16 bus (meaning the x16 bus actually becomes an x12 bus in the process, I might want to add that to the superuser benchmark info as well, but I just don't know anything about this)

Edit: According to the manual the x16 bus card will remain at x16 mode even when a card is in use by the x4 bus. It does not say anything about DMI 2.0 or the PCI Express connecting to the CPU via it, but according to the reference you linked to earlier the pci express x16 is connected directly to the CPU and the pci express x4 is connected to a "pci express switch" which forwards it to the processor (not through the DMI)

I think you're confusing the pci express x16 operating at x4 bandiwdth with the actual pci express x4 bus (which is another thing entirely and much smaller, you wouldn't fit a GeForce card in there)

Edit2: I am however having a quite dubious performance difference for the GTX 550-Ti on linux's OpenGL between the x4 and x16, no really it is fucking amazingly bad.

GTX 550-Ti on x16 (Linux)

FPS:3.5 Score:89 Min FPS:2.6 Max FPS:6.6

GTX 550-Ti on x16(x4) (Linux)

FPS:1.3 Score:33 Min FPS:1.1 Max FPS:2.4

I'm having a hard time believing how huge that difference was (although when I think of it it really is only a 2.2 overall FPS difference, but this I find quite odd I need to investigate this, although it should be considered I am running it on settings the card clearly can't possibly handle so my first step should of course be to try running something it can handle and comparing between windows and linux)

Edit 3: The performance difference on the same settings between Windows and Linux on OpenGL 3.2 with the 550-Ti were so huge I am inclined to believe that that 89 score I got was some kind of fluke (maybe I forgot to enable the FXAA and force x16 anisotropic filtering, I'll look into this as well)

Either way here are my results.

GTX 550-Ti on x16 (Windows)

FPS:13.9 Score:351 Min FPS:6.4 Max FPS:29.1

GTX 550-Ti on x16(x4) (Linux)

FPS:21.0 Score:530 Min FPS:8.7 Max FPS:34.6

Settings were Unigines "basic" preset with 1920x1080 full screen resolution on OpenGL renderer. What I was doing here (as I am too lazy to put the 550-Ti back in the x16 slot it's just too much trouble to switch the cards up) was testing to confirm if the performance in Windows on OpenGL on this card wasn't still lower than on Linux. This benchmark was enough to convince me that the card is functioning at full efficiency (or like 99% since I think the x4 bus chips just a tiny little bit of performance away from this card, nothing as much as on the 670) on linux and I must have made a mistake in my settings somewhere to get so different results earlier. The performance difference was much bigger than between the 670 on linux and windows (meaning it was even higher than I expected and hoped it would be)

I am doing this because I know the card is working just as well on windows on this x4 bus as it was on the x16 bus (I triple checked with DX11 and it is indeed just getting roughly the same results as it did on the x16 pass)

One thing I did notice however is that 550-Ti on OpenGL on windows ran 10°C (72°C) hotter than it ever did on linux(60°C)  so it actually may be that the performance on linux is Slightly borked somehow (I mean this is the same behavior as I got when I switched 670 between x16 and x4, except that this isn't between buses but between operating systems, and possibly drivers, might be VGA arb related)

Edit 4: Just to check I tried installing the Nvidia drivers again (without the VGA arb patch) to see if they would perform any differently and well they did to a point. It seems like everything runs the same way as it did when I scored 89 earlier. except that after an X amount of time (which actually varies) it slows down and everything drops to around 1 FPS (temperature also refuses to exceed 60, perhaps that is the problem? perhaps nvidia's linux drivers don't want this card to exceed 60°C whereas the windows ones cap it at 70°C instead?)

Ah well I will give up on it, but I am pretty sure this is temperature and power management related, if not it must be a problem with the program's code somewhere (or drivers) at least on settings the card actually can handle it performs just as well as I would have expected it to.

Edit 5: Interesting thing to add is that the 550-Ti on OpenGL 3.2 in Unigine Heaven is actually on par with DX11

GTX 550-Ti on x16(x4) (Windows, DX11)

FPS:21.2 Score:534 Min FPS:9.8 Max FPS:37.8

Last edited by rabcor (2014-06-11 10:38:30)

Offline

#2064 2014-06-11 12:39:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor: refer back to your finding that DRI is disabled when multiple graphics cards are installed.  The difference you're seeing is a software limitation, not hardware.

mv0lnicky: DMI is more than sufficient to handle a x4 PCIe link.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2065 2014-06-11 13:00:07

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But I am technically only using one card. And DRI3 gave me no error (meaning DRI3 is working) but if you're referring to the benchmarks i was doing above that had me baffled, the one that scored twice as high as the other GL benchmarks on the same settings was taken with the same setup (with my 670 stubbed)

Last edited by rabcor (2014-06-11 14:51:47)

Offline

#2066 2014-06-11 15:37:35

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

P8Z77-V LX link -  I have very little idea what you're talking about.

My understanding is that the x4 bus is connected to the CPU by first going through the x16 bus (meaning the x16 bus actually becomes an x12 bus in the process, I might want to add that to the superuser benchmark info as well, but I just don't know anything about this)

Edit: According to the manual the x16 bus card will remain at x16 mode even when a card is in use by the x4 bus. It does not say anything about DMI 2.0 or the PCI Express connecting to the CPU via it, but according to the reference you linked to earlier the pci express x16 is connected directly to the CPU and the pci express x4 is connected to a "pci express switch" which forwards it to the processor (not through the DMI)

I think you're confusing the pci express x16 operating at x4 bandiwdth with the actual pci express x4 bus (which is another thing entirely and much smaller, you wouldn't fit a GeForce card in there)

Let me get this straight, when you're referring to 4x, you're talking about the black colored slot. No I am not talking about the little PCIe slots (they are actually 1x slots on your board).

Asus does not support multi-gpu (SLI/Xfire) on the LX motherboard as a result of product differentiation. From looking at the specs, this board seems artificially limited to running only one full 16x link. (directly attached to the CPU thus, no bottleneck on this one)

The confusion comes from the fact that in the ideal scenario, a board with two 16x PCIe slots - like the P8Z77-V PRO - will run on mode 1 (only one GPU) at full 16x, and on mode 2 (two GPU) at a reduced rate 8x + 8x. This limitation has been existing with Intel motherboards for a long time (think P35/X38 days) but it is not relevant in your case. If you took the same P8Z77-V PRO, stuck a GPU into slot 3, you'd have the same scenario as you currently have n your P8Z77-V LX.

I made a little picture since I guess it's hard to explain in words. http://oi59.tinypic.com/33xkndc.jpg EDIT: I made a bad circling on the the PRO board, the writing is on the PCI slot when it should be on the PCIE slot. Doesn't change anything anyway.


aw wrote:

mv0lnicky: DMI is more than sufficient to handle a x4 PCIe link.

It is but DMI shares it's bandwidth with every other I/O (USB, SATA) and is max 20Gbit for everything with a 4x bus. I have not benchmarked it but going from two different PCIe slots, one on the PCH to one of the PCIe switch from the CPU is like night and day when using desktop compositing. Even switching workspace is noticably slower. Almost like a GPU switching power management modes even though they work at 100% clock rate. I suspect that access to system memory is significantly impacted by passing through DMI. For the gazillion SATA and USB ports, DMI is fine, but for serious 3D work, it's terrible.

A "normal" 4x PCIe slot should not be sharing it's entire bandwidth with everything under the sun.

Last edited by mv0lnicky (2014-06-11 15:41:09)

Offline

#2067 2014-06-11 17:01:42

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apporc wrote:

Hi, siddharta.
But now all of above turned out to be not necessary.
After above, I just used virt-manager to create my win7 vm. I totally use virt-manager to add vga card and usb device to my vm.

The method virt-manager uses is now vfio_pci too, not pci_assign.

Here is another link i used to refer to: http://www.firewing1.com/howtos/fedora- … hrough-kvm.

Good luck!

Thanks apporc, that's very interesting. I also like to use virsh and virt-manager to provision and manage guests. I now have VGA passthrough working (kernel 3.15 with i915 vga arbiter patch), can PCI assign those devices that can't be passed through with vfio (thanks to aw) and have USB device passthrough working again (I'm on OpenSUSE 13.1 and it seems qemu 2.0.0 in the virtualization repository is built with libusb disabled, so I had to rebuild that).

Now I'm struggling with converting everything to libvirt xml which doesn't go smootly... for example, if you provision a new guest through virt-manager, then change the chipset to q35 rather than i440 it complains that the USB controller needs a PCI bus. It's a good idea to try and stick with the virt-manager defined guest and work from there, I'll give that a go.

edit: I spoke in haste. While I don't get an error anymore when starting a guest that has a passthrough USB device in its definition, as was the case with the standard qemu 2.0.0 in OpenSUSE's virtualization repo, even a passthrough USB keyboard doesn't work in a guest. Not certain if the problem is with the updated libusb I had to install to build qemu with --enable-libusb or the qemu I built. This throws another stick in the gearbox (is that the expression?) so I might just see about building the whole thing on Arch as that seems the least painful-ish.

Last edited by siddharta (2014-06-11 17:33:17)

Offline

#2068 2014-06-11 17:03:24

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks mv0lnicky.

I have a problem, I'm setting up my system again (this time actually with arch as opposed to gentoo last time) I'm using the official arch kernel (but with ACS Override patch) here's what happens when I try to use vfio-bind (and I'm sure this is some really noobish mistake i'm making)

$ dmesg | grep vfio [ 2.613944] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 2.616525] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 2.616536] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 2.619017] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 2.619028] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 2.619037] vfio-pci: probe of 0000:06:00.0 failed with error -22

Here is my cmdline

$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=0773b55f-24c9-494a-a43b-db6b45a36a80 rw quiet pci-stub.ids=10de:1189,10de:0e0a pcie_acs_override=downstream
$ cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 options vfio_iommu_type1 allow_unsafe_interrupts=1 blacklist snd_ctxfi

(where ctxfi is the mysterious 06:00.0 soundblaster card) I know that pci-stub worked, I know that something fails between pci-stub and the vfio-bind. What am I doing wrong? "lspci -v" should show "kernel driver in use:vfio-pci" should it not? (instead I see no kernel driver line at all for the devices in question,  i get a ">?" sign of some sort instead)

/dev/vfio/vfio exists.

Edit: I think I know what the issue may be.

$ zgrep IOMMU_DEFAULT /proc/config.gz # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set

How do I work aorund this? (Is there any other way than to recompile the kernel with the setting enabled?)

Last edited by rabcor (2014-06-11 17:43:19)

Offline

#2069 2014-06-11 17:50:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Edit: I think I know what the issue may be.

$ zgrep IOMMU_DEFAULT /proc/config.gz # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set

How do I work aorund this? (Is there any other way than to recompile the kernel with the setting enabled?)

Add intel_iommu=on to your kernel boot options


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2070 2014-06-11 19:16:46

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:
ttouch wrote:

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice? smile

You mean like underscan?

Yes.

Offline

#2071 2014-06-11 20:43:06

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I have the same problem like rabcor.

vfio-bind results in

[root@ARCH iommu_groups]# dmesg | grep vfio [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0140 quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0140 quiet [ 209.145692] vfio-pci: probe of 0000:02:00.0 failed with error -22 [ 473.422577] vfio-pci: probe of 0000:02:00.0 failed with error -22 [ 832.559887] vfio-pci: probe of 0000:02:00.0 failed with error -22

I think that intel_iommu is enabled ... see kernel options above.

[root@ARCH tmp]# zgrep IOMMU /proc/config.gz CONFIG_GART_IOMMU=y CONFIG_CALGARY_IOMMU=y CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y CONFIG_IOMMU_HELPER=y CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_IOMMU_API=y CONFIG_IOMMU_SUPPORT=y CONFIG_AMD_IOMMU=y # CONFIG_AMD_IOMMU_STATS is not set CONFIG_AMD_IOMMU_V2=m CONFIG_INTEL_IOMMU=y # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set CONFIG_INTEL_IOMMU_FLOPPY_WA=y # CONFIG_IOMMU_DEBUG is not set # CONFIG_IOMMU_STRESS is not set

CPU: i5 2500k
MB: Asus P8P67 EVO

Offline

#2072 2014-06-11 20:47:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

CPU: i5 2500k
MB: Asus P8P67 EVO

Your CPU doesn't support VT-d - http://ark.intel.com/products/52210/Int … o-3_70-GHz

Thanks for playing

nbhs, maybe you could add a link to ark on the first post, we're getting this a lot.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2073 2014-06-11 21:17:35

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm surprised people can't verify for themselves since the first post mentions :

"NOTE: to get this working you'll need:

    AMD-VI/VT-D enabled and working."

Last edited by mv0lnicky (2014-06-11 21:17:49)

Offline

#2074 2014-06-11 21:20:21

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:
mv0lnicky wrote:
ttouch wrote:

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice? smile

You mean like underscan?

Yes.

I doubt it's related to QEMU since it is just passing complete control of the hardware to the guest.

What operating system are you running as guest VM? Did you install the drivers for your graphic card?

Offline

#2075 2014-06-11 21:22:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:

I'm surprised people can't verify for themselves since the first post mentions :

"NOTE: to get this working you'll need:

    AMD-VI/VT-D enabled and working."

To be fair, the kernel does like to lie and say things like "Intel-IOMMU: enabled" when actually all it's done is register that you've passed intel_iommu=on.  The actual telling line that nobody seems to get right when trying to prove that it's enabled is "PCI-DMA: Intel(R) Virtualization Technology for Directed I/O".


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2076 2014-06-11 21:29:13

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mv0lnicky wrote:

I'm surprised people can't verify for themselves since the first post mentions :

"NOTE: to get this working you'll need:

    AMD-VI/VT-D enabled and working."

To be fair, the kernel does like to lie and say things like "Intel-IOMMU: enabled" when actually all it's done is register that you've passed intel_iommu=on.  The actual telling line that nobody seems to get right when trying to prove that it's enabled is "PCI-DMA: Intel(R) Virtualization Technology for Directed I/O".

Do Intel K CPUs report the 'vmx' flag? [EDITED OUT: Don't want to confuse people]

Last edited by mv0lnicky (2014-06-11 21:33:51)

Offline

#2077 2014-06-11 21:29:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:
aw wrote:
mv0lnicky wrote:

I'm surprised people can't verify for themselves since the first post mentions :

"NOTE: to get this working you'll need:

    AMD-VI/VT-D enabled and working."

To be fair, the kernel does like to lie and say things like "Intel-IOMMU: enabled" when actually all it's done is register that you've passed intel_iommu=on.  The actual telling line that nobody seems to get right when trying to prove that it's enabled is "PCI-DMA: Intel(R) Virtualization Technology for Directed I/O".

Do Intel K CPUs report the 'vmx' flag? If not, then it'd just be a matter of adding in the first post something like :

cat /proc/cpuinfo | grep 'vmx'

vmx != VT-d


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2078 2014-06-11 21:33:11

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mv0lnicky wrote:
aw wrote:

To be fair, the kernel does like to lie and say things like "Intel-IOMMU: enabled" when actually all it's done is register that you've passed intel_iommu=on.  The actual telling line that nobody seems to get right when trying to prove that it's enabled is "PCI-DMA: Intel(R) Virtualization Technology for Directed I/O".

Do Intel K CPUs report the 'vmx' flag? If not, then it'd just be a matter of adding in the first post something like :

cat /proc/cpuinfo | grep 'vmx'

vmx != VT-d

Oh you're right, it's for VT-x. So there's no cpu flag for VT-d.

Then I guess the only way really is:

dmesg | grep -i pci-dma

Offline

#2079 2014-06-12 00:45:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
_pheinrich_ wrote:

CPU: i5 2500k
MB: Asus P8P67 EVO

Your CPU doesn't support VT-d - http://ark.intel.com/products/52210/Int … o-3_70-GHz

Thanks for playing

nbhs, maybe you could add a link to ark on the first post, we're getting this a lot.

Sure

Offline

#2080 2014-06-12 06:10:00

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:
mv0lnicky wrote:
ttouch wrote:

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice? :)

You mean like underscan?

Yes.

i'm getting this as well.
my setup is one monitor (benq g2420hd) with connected:
- DVI : host (fedora 20, i7-4770 IGP)
- HDMI : guest (win 7, radeon HD 6450)

i just switch displayed input on monitor when needed.
host video (DVI) works just fine - native 1920x1080, no problems at all
but when i switch to guest video (HDMI) using 1920x1080 - i get pretty large black frame visible on the screen. so far, i "solved" it by decreasing screen resolution in guest

but if there is some sort of solution (e.g. it is not a problem with the monitor not being able to handle dual input correctly) - that would be great

Last edited by sinny (2014-06-12 06:10:24)

Offline

#2081 2014-06-12 06:40:33

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes you are right ... I should have seen that my cpu supports VT-x only.

So if I take a http://ark.intel.com/products/52209/Int … o-3_70-GHz without that stupid --->K<--- I might have success?

Does anybody have experience with my MB (Asus P8P67 EVO) or do I get trouble with that too?

Offline

#2082 2014-06-12 09:00:22

blitzschlag
Member
Registered: 2013-10-30
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

Does anybody have experience with my MB (Asus P8P67 EVO) or do I get trouble with that too?

hey,
have a look at

https://docs.google.com/spreadsheet/ccc … _web#gid=0

maybe its in there (haven't checked myself).

that would actually be a good mention in the opening Post aswell *hinthint*

big_smile

keep up the good work guys

Offline

#2083 2014-06-12 13:38:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

Yes you are right ... I should have seen that my cpu supports VT-x only.

So if I take a http://ark.intel.com/products/52209/Int … o-3_70-GHz without that stupid --->K<--- I might have success?

Does anybody have experience with my MB (Asus P8P67 EVO) or do I get trouble with that too?

It's probably fine, I have a similar generation P8H67-M PRO/CSM that works well with i5-3470T.  Host uses the Intel HD graphics with GT635 and HD8570 for guests.  If you have a rev 01 ASMedia PCI bridge you can pretty much write off the conventional PCI slots as useless, but that's true regardless of device assignment.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2084 2014-06-12 14:36:14

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you don't ask you won't know,
How do apply the " i915: Add module option to support VGA arbiter on HD devices " patch ?

Never mind,

https://bbs.archlinux.org/search.php?ac … r_id=82136

Last edited by abdullah (2014-06-12 14:42:05)

Offline

#2085 2014-06-12 16:23:49

Torello
Member
Registered: 2014-04-01
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, somehow I'm unable to pass any usb device to guest.

My start script is:

qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu Haswell \
-smp 8,sockets=1,cores=4,threads=2 \
-vga none \
-rtc clock=host \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-scsi-pci,id=scsi \
-drive if=virtio,file=/home/user/virt_images/win7_meta.qcow2,format=qcow2,cache=none \
-drive file=/home/user/distr/gparted.iso,id=isocd -device ide-cd,bus=ide.0,drive=isocd \
-boot menu=on \
-device qxl

I have ich9 "Usb controllers" in device manager, and seems they are working fine( no any marks about errors in device manager ).

If I add, for example,
"-usb -device usb-host,hostbus=1,hostaddr=10 \"
I have got additonally 4 unknown devices( ICH9 2934, ICH9 2935 and so on ), each of them is ich9 usb host controller too.
This devices has a windows error 29 - "No drivers installed for device".
And Windows can't install a drivers for this controllers.
Do you have any suggestions how to fix that issue?

That's how device manager looks like:

Device Manager

Offline

#2086 2014-06-12 18:41:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just to give kudos where due, I finally got around to booting my Gigabyte GA-990FXA-UD3 system today and it allows full configuration of the initial display, letting me select PCI/PEG/PEG1/PEG2/PEG3 for the init display.  The only thing it won't let me select are the x1 slots.  I peeked at their latest, greatest Intel board manual (GA-Z97X-SOC) and it appears to offer the same, IGD, any one of the x16 slots, or PCI (BTW, the manual also has a great slot layout diagram for those confused about x16 vs x8 vs x4, PEG vs PCH, and DMI).  Asus hasn't really done anything to make me a loyal customer lately, so I may be looking to Gigabyte for my next board.  Please pardon this commercial interruption wink

Last edited by aw (2014-06-12 18:52:09)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2087 2014-06-12 19:06:21

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Just to give kudos where due,
...
Asus hasn't really done anything to make me a loyal customer lately, so I may be looking to Gigabyte for my next board.  Please pardon this commercial interruption wink

Interesting. I've been using Asrock boards mostly as their BIOS seems to do well for VT-x/VT-d unlike some Asus boards I have here. Is Gigabyte consistent in that respect? The Asrock boards are overall not too bad but a bit on the flimsy side, they bend if you look at them the wrong way.

Offline

#2088 2014-06-12 19:16:27

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Torello wrote:

Hello, somehow I'm unable to pass any usb device to guest.

My start script is:

qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu Haswell \
-smp 8,sockets=1,cores=4,threads=2 \
-vga none \
-rtc clock=host \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-scsi-pci,id=scsi \
-drive if=virtio,file=/home/user/virt_images/win7_meta.qcow2,format=qcow2,cache=none \
-drive file=/home/user/distr/gparted.iso,id=isocd -device ide-cd,bus=ide.0,drive=isocd \
-boot menu=on \
-device qxl

...

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209 Bus 005 Device 002: ID 2109:0811 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

Last edited by flack (2014-06-12 19:32:04)

Offline

#2089 2014-06-12 19:20:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
aw wrote:

Just to give kudos where due,
...
Asus hasn't really done anything to make me a loyal customer lately, so I may be looking to Gigabyte for my next board.  Please pardon this commercial interruption wink

Interesting. I've been using Asrock boards mostly as their BIOS seems to do well for VT-x/VT-d unlike some Asus boards I have here. Is Gigabyte consistent in that respect? The Asrock boards are overall not too bad but a bit on the flimsy side, they bend if you look at them the wrong way.

Asrock got on my personal sh*t list when I was searching for an AMD-Vi system several years ago.  When I sent a support request trying to figure out if their IOMMU option in the BIOS was actually enabling AMD-Vi or simply added a hole in memory for using a GART as an IOMMU.  They gave me a non-answer referring me back to the manual and the exact BIOS option I was asking about, then volunteered that Asrock does not support Linux (I guess based on my OS selection or browser agent).  I ended up with this Gigabyte board instead.  There seem to be plenty of success stories with Asrock, so I guess they work.  I can't speak to consistency on Gigabyte boards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2090 2014-06-12 19:29:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:
aw wrote:
mv0lnicky wrote:

Do Intel K CPUs report the 'vmx' flag? If not, then it'd just be a matter of adding in the first post something like :

cat /proc/cpuinfo | grep 'vmx'

vmx != VT-d

Oh you're right, it's for VT-x. So there's no cpu flag for VT-d.

Then I guess the only way really is:

dmesg | grep -i pci-dma

Then you may run into this:

PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2091 2014-06-13 07:56:11

Torello
Member
Registered: 2014-04-01
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:
Torello wrote:

Hello, somehow I'm unable to pass any usb device to guest.
My start script is:
...

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209 Bus 005 Device 002: ID 2109:0811 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

Thanks for help, but it doesn't work, I have same issue with your suggestion ( 4 unknown-device usb host controllers in device-manager ).

Last edited by Torello (2014-06-13 07:57:10)

Offline

#2092 2014-06-13 09:05:45

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209 Bus 005 Device 002: ID 2109:0811 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

I pass both my mouse and keyboard to VM in the same way as you, it's very robust. But if you want to use host keyboard at the same time, you need a second one with a different device ID (this probably means different model or vendor). For me it does not matter, I only ssh to host anyway (when running a VM).

I wonder if a USB hub can be passed through this way (taking with it all the attached devices), didn't have time to check yet. Even better if this also worked for USB3.0 hub ...


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#2093 2014-06-13 11:47:32

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Just to give kudos where due, I finally got around to booting my Gigabyte GA-990FXA-UD3 system today and it allows full configuration of the initial display, letting me select PCI/PEG/PEG1/PEG2/PEG3 for the init display.  The only thing it won't let me select are the x1 slots.

It is the same board I use and works really great with IOMMU and device passthrough. It is very stable solution from my point of view, except for the reset problem of R9 290X but all other devices I used for passthrough works like a charme.

Offline

#2094 2014-06-13 12:21:21

Midov
Member
Registered: 2013-10-22
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
ttouch wrote:
mv0lnicky wrote:

You mean like underscan?

Yes.

i'm getting this as well.
my setup is one monitor (benq g2420hd) with connected:
- DVI : host (fedora 20, i7-4770 IGP)
- HDMI : guest (win 7, radeon HD 6450)

i just switch displayed input on monitor when needed.
host video (DVI) works just fine - native 1920x1080, no problems at all
but when i switch to guest video (HDMI) using 1920x1080 - i get pretty large black frame visible on the screen. so far, i "solved" it by decreasing screen resolution in guest

but if there is some sort of solution (e.g. it is not a problem with the monitor not being able to handle dual input correctly) - that would be great

I had exactly same problem with my Geforce 660 over HDMI when i installed newest drivers (version 337.88) in client.
It went away after i deleted those drivers and installed older ones (version 335 i belive) and now its working in full screen without that anoying black frame around it.
It was also working without any problems when i used default drivers that windows (win8) found on its own.

Offline

#2095 2014-06-13 13:47:58

vais
Member
Registered: 2014-06-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone!

This is a great thread and using it and a few other sources I too was able to get VFIO VGA passthrough working on Fedora 20.
My specs:

MB: Asrock Z97 Extreme4
CPU: Intel i5-4570
Host GPU: IGP HD 4600
Passed GPU: Radeon R9 270x

Qemu: 2.0.0-6 (from fedora-virt-preview, no patches)
Kernel: 12.5 (patched)

The only way I got it working was to use an already built kernel (thanks Stewart!) with the necessary patches applied - however this kernel is 3.12.5, which is now rather old and I would like to update it.
The issue is I was unable to apply the patch myself (and build the RPM) successfully.
I am trying to build a 3.15.rc8 (3.15.0.1 came out today in the repo, so I will switch to that) and tried all kinds of combinations of patches - none worked

In the end I believe all I need is the VGA arbiter patch: https://lkml.org/lkml/2014/5/9/517 but the host graphics issue persists.

Are there any other patches for the VGA arb that need to be applied too?
If anyone with a similar config, and running Fedora, has a working 3.15 kernel could you list the patches used?
Also sharing the binaries will be appreciated  smile

EDIT: Something that doesn't seem right - all modules of my built kernel (3.15.0-0.rc8.git2.1) are compressed - i915.ko.xz for example, while on the working kernel (3.12.5) they are not.
Has anything change in the kernel structure or I am doing something very wrong while building the RPM?

P.S. I use simply

rpmbuild -ba ./kernel.spec

Last edited by vais (2014-06-13 13:59:16)

Offline

#2096 2014-06-13 14:19:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

vais wrote:

In the end I believe all I need is the VGA arbiter patch: https://lkml.org/lkml/2014/5/9/517 but the host graphics issue persists.

Using this version of the patch, make sue you boot with i915.enable_hd_vgaarb=1


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2097 2014-06-13 19:47:33

setedivento
Member
Registered: 2014-06-08
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
setedivento wrote:

Hello everybody, really great topic, thanks!

If only I could make things work. I seemed so close... may anyone help?
I am trying to vga passthrough an Nvidia Quadro 4000 on a Kubuntu 14.04 host machine. Reports indicate more than one here has coped, on Ubuntu based distros, so I've decided to ask.

Full stop, you have a Quadro card and while it's not the version that NVIDIA supports for device assignment, it's hopefully close enough.  Forget about Q35, -vga none, and x-vga=on, Quadro doesn't need any of that.  Just add it to the VM as you would any other assigned device, you can even use virt-manager for it.  Install using the emulated VGA.  When you install the NVIDIA driver stack, the emulated VGA will be disabled and you'll automatically switch over to the Quadro card.  We intend to support K-series Quadros on RHEL7 using this mode, but those cards are also known not to work with x-vga=on.  K-series cards are Kepler based and yours I believe is Fermi based.  Those are only one generation apart, so it will hopefully work.  For others out there hoping their cheapy Quadro NVS cards can work like this, don't bet on it.

EDIT: The audio device on K-series Quadro is also known not to work and will not be supported for assignment on RHEL7.  Try it, but expect it not to work.  Just bind it to pci-stub and forget it's there.


woa, that's great to hear, thanks aw! (and sorry for the delay in answering, always busy with work)

may I just have a quick hint? how do I correctly add the device using the qemu-system-x86_64 CLI?
I have bound the card to the ioh3420 PCI bus as per OP guidelines (and of course removed the 'x-vga=on','vga-none' and '-M q35' arguments);
but then when I start Windows, install the nvidia drivers and launch the nvidia control panel, the latter tells me 'you are not currently using a display attached to an NVIDIA GPU' (there's no switch from the emulated VGA to the passed through card).
I quite don't have my ideas that clear, too, about how to instruct qemu to make the guest OS think the passed through graphic card is connected to the display... Especially in terms of what happens behind the scenes: if the display in qemu is somehow a device/device driver one can interact with or not....(sorry for the confusion, as I said I am a little newbie)...
virt-manager gave me the error described here:
https://bugs.launchpad.net/ubuntu/+sour … /+activity
(both with and without the device being claimed by pci-stub)
and even when I successfully applied the relative workaround, it stayed an hour or so to launch the install process after I had configured the emulated hardware, before I decided to quit it.
I did not search for its log because in general, maybe it's more advisable for me to use a lower level tool, that forces me into better learning what exactly I am doing.

thanks again for your precious help!

Offline

#2098 2014-06-13 20:00:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

setedivento wrote:
aw wrote:
setedivento wrote:

Hello everybody, really great topic, thanks!

If only I could make things work. I seemed so close... may anyone help?
I am trying to vga passthrough an Nvidia Quadro 4000 on a Kubuntu 14.04 host machine. Reports indicate more than one here has coped, on Ubuntu based distros, so I've decided to ask.

Full stop, you have a Quadro card and while it's not the version that NVIDIA supports for device assignment, it's hopefully close enough.  Forget about Q35, -vga none, and x-vga=on, Quadro doesn't need any of that.  Just add it to the VM as you would any other assigned device, you can even use virt-manager for it.  Install using the emulated VGA.  When you install the NVIDIA driver stack, the emulated VGA will be disabled and you'll automatically switch over to the Quadro card.  We intend to support K-series Quadros on RHEL7 using this mode, but those cards are also known not to work with x-vga=on.  K-series cards are Kepler based and yours I believe is Fermi based.  Those are only one generation apart, so it will hopefully work.  For others out there hoping their cheapy Quadro NVS cards can work like this, don't bet on it.

EDIT: The audio device on K-series Quadro is also known not to work and will not be supported for assignment on RHEL7.  Try it, but expect it not to work.  Just bind it to pci-stub and forget it's there.


woa, that's great to hear, thanks aw! (and sorry for the delay in answering, always busy with work)

may I just have a quick hint? how do I correctly add the device using the qemu-system-x86_64 CLI?
I have bound the card to the ioh3420 PCI bus as per OP guidelines (and of course removed the 'x-vga=on','vga-none' and '-M q35' arguments);
but then when I start Windows, install the nvidia drivers and launch the nvidia control panel, the latter tells me 'you are not currently using a display attached to an NVIDIA GPU' (there's no switch from the emulated VGA to the passed through card).
I quite don't have my ideas that clear, too, about how to instruct qemu to make the guest OS think the passed through graphic card is connected to the display... Especially in terms of what happens behind the scenes: if the display in qemu is somehow a device/device driver one can interact with or not....(sorry for the confusion, as I said I am a little newbie)...
virt-manager gave me the error described here:
https://bugs.launchpad.net/ubuntu/+sour … /+activity
(both with and without the device being claimed by pci-stub)
and even when I successfully applied the relative workaround, it stayed an hour or so to launch the install process after I had configured the emulated hardware, before I decided to quit it.
I did not search for its log because in general, maybe it's more advisable for me to use a lower level tool, that forces me into better learning what exactly I am doing.

thanks again for your precious help!

If this card works like the K-series Quadros, then ignore the OP in this thread entirely.  ioh3420 is a component of q35, don't use it.  The qemu CLI for adding the GPU is simply:

-device vfio-pci,host=1:00.0

(replace with your device address as appropriate)  Also, be sure that you're installing the Quadro driver, not the standard GeForce driver.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2099 2014-06-13 21:32:02

setedivento
Member
Registered: 2014-06-08
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks, I will try that and report here the results, if of any use... smile

Offline

#2100 2014-06-14 05:17:32

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm having a bit of a problem with partition sharing. I'll make this short, I tried to follow this here once with virtualbox and I do not remember how exactly it ended, just that it did not end in any sort of success. So this time I would like to get some help. Here is the partition setup I need to pass through.

/dev/sda (MBR = Bootmgr on /dev/sda1)
/dev/sda1 = System Reserved partition (windows's bootloader)
/dev/sda2 = Windows 7's C:/ partition
/dev/sda3 = Arch Linux (Host)

I've got /dev/sda3's grub installed into the MBR of /dev/sdc (meaning booting /dev/sdc will boot me into my arch linux install, booting /dev/sda will boot me into windows)

I need to passthrough /dev/sda1, /dev/sda2 and preferably /dev/sda's MBR as well (but I don't mind using another method to make it boot as long as /dev/sda1 and /dev/sda2 are passed through)

How exactly should I proceed?

Additionally I have a GPT, NTFS partition on /dev/sdb2 that I want to be easily accessible from both the host and my guest at the same time (I want to share it between both systems) how do I do that too?

Last edited by rabcor (2014-06-14 05:20:49)

Offline

#2101 2014-06-14 13:57:41

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it... hmm
Any help appreciated smile

Offline

#2102 2014-06-14 15:15:54

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
flack wrote:

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209 Bus 005 Device 002: ID 2109:0811 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

I pass both my mouse and keyboard to VM in the same way as you, it's very robust. But if you want to use host keyboard at the same time, you need a second one with a different device ID (this probably means different model or vendor). For me it does not matter, I only ssh to host anyway (when running a VM).

I wonder if a USB hub can be passed through this way (taking with it all the attached devices), didn't have time to check yet. Even better if this also worked for USB3.0 hub ...

I know what you think. But i have KVM switch like this http://www.ebay.com/itm/BELKIN-PS-2-Por … 0710063511
It is PS/2 switch but for Host i use PS2 input. And for Quest i use adapter ps/2-->USB this one http://www.cablewholesale.com/products/ … c-451g.php
I can use one monitor one keyboard and one mouse together. USB adapter have own vID and pID. But this is older method with PS/2 today all people have USB mouses and keyboards.

Offline

#2103 2014-06-14 15:44:51

Jdioutkast
Member
Registered: 2014-06-14
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey aw,

I was wondering if you had any insight into this issue,

I'm using a XFX R7 260X and I can get GPU passthrough working fine, but as soon as I either attempt to install the ATI driver(Windows 7) or the driver installs(Windows 8) I get an immediate BSOD, or the screen just goes black and the CPU usage of the guest goes to 100%.  I also have to reboot the host to get the card working again.

I am running Fedora virt-preview with kernel 3.15.

I have tried doing the bus reset test you posted a few pages back and the screen did not clear, so I am guessing it is a bus reset issue.  Any ideas on what to do next?  Anything I can try to get around the issue?

Thanks!

Offline

#2104 2014-06-15 07:01:22

vais
Member
Registered: 2014-06-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
vais wrote:

In the end I believe all I need is the VGA arbiter patch: https://lkml.org/lkml/2014/5/9/517 but the host graphics issue persists.

Using this version of the patch, make sue you boot with i915.enable_hd_vgaarb=1


Yes, I pass this both as kernel parameter and in /etc/modprobe.d as an option, just in case. Still, it doesn't seem to work sad

Offline

#2105 2014-06-15 08:34:13

spstarr
Member
Registered: 2014-06-15
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello folks,

I have a Lenovo W500 laptop that has an Intel 4gen GPU (HD graphics). I'm trying to pass the discrete Radeon HD3650 GPU to a WIndows 7 guest here is my configuration:

grub options:

vconsole.keymap=us rhgb slub_debug=- cgroup_disable=memory console=tty0 console=ttyS0,115200n8 radeon.blacklist=1 intel_iommu=on pci-stub.ids=1002:9591 vfio_iommu_type1.allow_unsafe_interrupts=1

/etc/modprobe.d/kvm.conf:

options kvm allow_unsafe_assigned_interrupts=1 options kvm ignore_msrs=1

Kernel sub

[ 0.668873] pci-stub: add 1002:9591 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.669199] pci-stub 0000:01:00.0: claimed by stub [ 691.577263] pci-stub 0000:01:00.0: claimed by stub

Binded it with the script

vfio-bind 0000:01:00.0

PCI Information

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV635/M86 [Mobility Radeon HD 3650] (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Mobility Radeon HD 3650 Physical Slot: 1-1 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: Memory at c0000000 (32-bit, prefetchable) [disabled] [size=256M] Region 1: I/O ports at 2000 [disabled] [size=256] Region 2: Memory at bfff0000 (32-bit, non-prefetchable) [disabled] [size=64K] [virtual] Expansion ROM at bff00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range A, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: vfio-pci Kernel modules: radeon

libvirt guest XML config:

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit Windows7 or other application using the libvirt API. --> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows7</name> <uuid>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Penryn</model> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='XXXXXXXXXXXXXXX'/> <source bridge='Bridge'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=1,romfile=/home/spstarr/radeon-vbios.rom'/> </qemu:commandline> </domain>

I've tried with rombar=0, no romfile specified, none work.

- If I use virt-managet and do PCI passing I get code 43
- If I use my QEMU embedded flag options I get code 12.

I've tried

modprobe kvm_intel emulate_invalid_guest_state=0

But this didn't work.

Anyone have tips? I saw someone mentioning to use multiple monitors, but when I try, I get VGA conflict errors with I start with virsh start.

additional info:
* I am using kernel 3.15 final
* QEMU 2.0.0 final release

Any help would be great!

** Update: So I'm told on IRC, you *CANNOT* pass a secondary GPU to QEMU VM window or full-screen directly, there is no such way to get the GPU display to another overlay.

Last edited by spstarr (2014-06-15 21:05:07)

Offline

#2106 2014-06-15 09:43:46

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it... hmm
Any help appreciated smile

Solved with Nvidia 335 big_smile
(Advanced search in nvidia page)

Offline

#2107 2014-06-15 11:31:08

syndtr
Member
Registered: 2014-05-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ttouch wrote:
ttouch wrote:

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it... hmm
Any help appreciated smile

Solved with Nvidia 335 big_smile
(Advanced search in nvidia page)

Or you can hide KVM hypervisor signature and continue using latest nvidia driver. You can add '-cpu kvm=off' to hide the kvm signature if you use current qemu git.

http://lists.gnu.org/archive/html/qemu- … 00808.html

Last edited by syndtr (2014-06-15 11:33:03)

Offline

#2108 2014-06-15 12:02:49

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Midov wrote:
sinny wrote:
ttouch wrote:

Yes.

i'm getting this as well.
my setup is one monitor (benq g2420hd) with connected:
- DVI : host (fedora 20, i7-4770 IGP)
- HDMI : guest (win 7, radeon HD 6450)

i just switch displayed input on monitor when needed.
host video (DVI) works just fine - native 1920x1080, no problems at all
but when i switch to guest video (HDMI) using 1920x1080 - i get pretty large black frame visible on the screen. so far, i "solved" it by decreasing screen resolution in guest

but if there is some sort of solution (e.g. it is not a problem with the monitor not being able to handle dual input correctly) - that would be great

I had exactly same problem with my Geforce 660 over HDMI when i installed newest drivers (version 337.88) in client.
It went away after i deleted those drivers and installed older ones (version 335 i belive) and now its working in full screen without that anoying black frame around it.
It was also working without any problems when i used default drivers that windows (win8) found on its own.

ok, found solution to hdmi underscan for radeon cards - works for me: link

Offline

#2109 2014-06-15 13:06:45

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syndtr wrote:
ttouch wrote:
ttouch wrote:

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it... hmm
Any help appreciated smile

Solved with Nvidia 335 big_smile
(Advanced search in nvidia page)

Or you can hide KVM hypervisor signature and continue using latest nvidia driver. You can add '-cpu kvm=off' to hide the kvm signature if you use current qemu git.

http://lists.gnu.org/archive/html/qemu- … 00808.html

Is this the cause of the lowered FPS I get?
I tested battlefield 3 and I get a bit laggy gameplay. Just less FPS.
Will this solve it?
Also do I pass "-cpu host", or just "-cpu kvm=off"?

Thank you.

Offline

#2110 2014-06-15 13:46:37

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I get "Unable to find CPU definition: kvm=off"
Also which are the possible optimizations I could do? (I can not find a way to search only in this topic)
My system:
Intel i7-3820
16GB ram
Asus P9X79 Deluxe
Asus GTX 660 Ti DirectCU II TOP

I use kernel 3.15.0-1-mainline provided by nbhs

Last edited by ttouch (2014-06-15 13:53:24)

Offline

#2111 2014-06-15 13:56:10

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:
Bronek wrote:
flack wrote:

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209 Bus 005 Device 002: ID 2109:0811 Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

I pass both my mouse and keyboard to VM in the same way as you, it's very robust. But if you want to use host keyboard at the same time, you need a second one with a different device ID (this probably means different model or vendor). For me it does not matter, I only ssh to host anyway (when running a VM).

I wonder if a USB hub can be passed through this way (taking with it all the attached devices), didn't have time to check yet. Even better if this also worked for USB3.0 hub ...

I know what you think. But i have KVM switch like this http://www.ebay.com/itm/BELKIN-PS-2-Por … 0710063511
It is PS/2 switch but for Host i use PS2 input. And for Quest i use adapter ps/2-->USB this one http://www.cablewholesale.com/products/ … c-451g.php
I can use one monitor one keyboard and one mouse together. USB adapter have own vID and pID. But this is older method with PS/2 today all people have USB mouses and keyboards.

Having multiple USB devices with same vendor and device ID is possible with libvirt if you address them by the device address on the host (<address> element in <source> rather than <vendor> and <product>), have a look at http://libvirt.org/formatdomain.html#el … tDevSubsys. It's a bit more limiting as the address of the device will change if you plug it into a different USB port for example.

Offline

#2112 2014-06-15 14:28:47

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was planing to get a setup so I can get passthrough for the games that needs Windows to run.

My plan is to get:
-ASRock z97 extreme 6
-i7 4790

The reason I'm going for that motherboard is the M.2 slot that has a speed of 4x PCIe aswell as the more common 2x slot, so I can use faster M.2 cards (not many available yet but there more might come to market later)
Now that will cut the dGPU to 8x lanes as the fast M.2 slot uses the CPU lanes aswell, would this in any way create a problem for the dGPU passthrough?

Also, I am really lost on memory, I have selected:
G.Skill TridentX DDR3 PC21300/2666MHz CL12 2x8GB
if anyone wish to comment one way or another on that it would be great.

Offline

#2113 2014-06-15 18:12:27

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, since I'm sick of not getting my GTX 770 performance problem fixed, I built in my AMD HD 6950.
Driver install worked and the short freezes are finally gone. <3
BUT the Benchmark is still pretty low.. http://www.3dmark.com/3dm11/8432530
Since there is no user with this setup, I searched for users with the i7 4770 which is pretty much the same like my Xeon E3-1245v3 and these guys have an average of 9000 in physics and 5500 in graphics score.. which means that my performance sucks and isn't even above 90% of the native performance..

As you can see it doesn't even push my Xeon into 3.8Ghz turbomode and my GPU isn't very fast aswell, with my GTX 770 I had the average of 11000 graphics score which was higher than some native users benchmarks, only the cpu slowed down. But what could cause this now?

Another little question, I'm using libvirt but can't get the emulated sound working, passthrough with libvirt works fine, only sound causes me trouble... I used the parameter in the first post, sound device is shown in windows, but I can't hear anything.

Last edited by shawly (2014-06-15 18:12:47)

Offline

#2114 2014-06-15 18:41:13

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't bind HDMI audio device to vfio-pci after compiling new 3.15 kernel.

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] (prog-if 00 [VGA controller]) Subsystem: Hightech Information System Ltd. Device 2291 Flags: fast devsel, IRQ 11 Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at f7e20000 (64-bit, non-prefetchable) [disabled] [size=128K] I/O ports at e000 [disabled] [size=256] Expansion ROM at f7e00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] Subsystem: Hightech Information System Ltd. Device aa68 Flags: bus master, fast devsel, latency 0, IRQ 46 Memory at f7e40000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting

As you can see "vfio-bind 01:00.0" works good, the device claimed by vfio driver. But "vfio-bind 01:00.1" just freezes, this device was claimed by snd_hda_intel, whenever i try to bind to vfio, the vfio-bind script just hangs (Ctrl+C does not stop the script). This device is claimed by snd_hda_intel even my cmdline.

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 pci-stub.ids=1002:9505,1002:aa18 pcie_acs_override=downstream i915.enable_hd_vgaarb=1

vfio-bind code, (i get from somewhere at the forum)

#!/bin/bash modprobe vfio-pci for dev in "$@"; do read -r vendor </sys/bus/pci/devices/$dev/vendor read -r device </sys/bus/pci/devices/$dev/device if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

(despite the script, i did not compile the vfio-pci as a module, its bundled on the kernel)

Any throughts?
Thanks.

Offline

#2115 2014-06-16 04:37:42

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

...

As you can see "vfio-bind 01:00.0" works good, the device claimed by vfio driver. But "vfio-bind 01:00.1" just freezes, this device was claimed by snd_hda_intel, whenever i try to bind to vfio, the vfio-bind script just hangs (Ctrl+C does not stop the script). This device is claimed by snd_hda_intel even my cmdline.

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 pci-stub.ids=1002:9505,1002:aa18 pcie_acs_override=downstream i915.enable_hd_vgaarb=1

vfio-bind code, (i get from somewhere at the forum)
...

(despite the script, i did not compile the vfio-pci as a module, its bundled on the kernel)

Any throughts?
Thanks.

if it is claimed, i guess it's id is wrong
i'd first look at dmesg:

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.0-1.vanilla.mainline.knurd.1.fc20.x86_64 root=UUID=588911d3-23f7-4544-9d46-0b06b85f287c ro pci-stub.ids=1002:6779,1002:aa98,10de:0de1,10de:0bea intel_iommu=on isolcpus=1,2,3,5,6,7 LANG=en_US.UTF-8 ... [ 0.402432] pci-stub: add 1002:6779 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.402578] pci-stub 0000:01:00.0: claimed by stub [ 0.402669] pci-stub: add 1002:AA98 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.402813] pci-stub 0000:01:00.1: claimed by stub [ 0.402904] pci-stub: add 10DE:0DE1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.403058] pci-stub: add 10DE:0BEA sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 ...

it is clearly seen here, that radeon HD 6450 is present, while nvidia gt 430 - not (10DE:* reference missing hardware)

how about "lspci -nn" output for additional clarity?

p.s. you cut your "lspci -v" output too early for others to see which kernel driver actually is used for amd audio

Last edited by sinny (2014-06-16 04:42:34)

Offline

#2116 2014-06-16 11:48:08

sedstr
Member
Registered: 2014-06-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been banging away at this, trying to achieve what may be impossible, please someone tell me.

I have an asus g74vw, it only has a single nvidia gtx670m.

What I want to do:
Install linux as the host, configure the network, then disable host access to basically everything but the network, I can forward X11 to the guest if I really wanted to, everything else can be done on the command line.
Install one windows VM and have it directly access the video card/keyboard/mouse/USB/CD, everything but the network.

I've put all my efforts into a deb installation, I've compiled kernels, updated qemu, but after detaching the adapter, I keep getting to...
ERROR    Unable to read from monitor: Connection reset by peer.
Only a days efforts, but its tiring.

I'll have another go with archlinux, if there is actually a possibility of getting to where I want, but if someone can tell me if its achievable, and possibly some tips, I'll keep bashing away at it.

This has been one of my technical wants for like 10 years! Main reason I want things setup this way, is I want to isolate windows but still be able to play a game. else I'd just run windows with linux in vmware.

TIA for any advice.

Offline

#2117 2014-06-16 16:39:38

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sedstr wrote:

I've been banging away at this, trying to achieve what may be impossible, please someone tell me.

I have an asus g74vw, it only has a single nvidia gtx670m.

What I want to do:
Install linux as the host, configure the network, then disable host access to basically everything but the network, I can forward X11 to the guest if I really wanted to, everything else can be done on the command line.
Install one windows VM and have it directly access the video card/keyboard/mouse/USB/CD, everything but the network.

I've put all my efforts into a deb installation, I've compiled kernels, updated qemu, but after detaching the adapter, I keep getting to...
ERROR    Unable to read from monitor: Connection reset by peer.
Only a days efforts, but its tiring.

I'll have another go with archlinux, if there is actually a possibility of getting to where I want, but if someone can tell me if its achievable, and possibly some tips, I'll keep bashing away at it.

This has been one of my technical wants for like 10 years! Main reason I want things setup this way, is I want to isolate windows but still be able to play a game. else I'd just run windows with linux in vmware.

TIA for any advice.

the only "technical" details provided are - motherboard and video card models and a single log line (not even specifying when it happens - e.g. in response to what action(-s)?)
imo, you are pretty much limiting the amount of help you may receive yourself

Offline

#2118 2014-06-16 16:56:18

sedstr
Member
Registered: 2014-06-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

the only "technical" details provided are - motherboard and video card models and a single log line (not even specifying when it happens - e.g. in response to what action(-s)?)
imo, you are pretty much limiting the amount of help you may receive yourself

The issue was presenting on trying to define a new VM with virt-install.
I'd nodedev-detach the gpu and the audio successfully, the virt-install command would cause the display to reset, then I'd get that message ERROR    Unable to read from monitor: Connection reset by peer.

All the examples given have one adapter assigned to the host, and one to the guest, I'm trying to run the host without a graphics adapter at all...
I was mostly interested in the 'is it possible' question, obviously, if what I'm trying to do is impossible due to some technical restriction, then I'll stop trying till I have another system more suitable.

Having said that, a few hours ago, I upgraded qemu to a later version via backports, and it started saying the hardware wasn't compatible.
Installed Arch, and well, learning curve, it'll take me a while to get to the stage I was at with debian ....AIX is my regular *nix.

Last edited by sedstr (2014-06-16 17:05:17)

Offline

#2119 2014-06-16 20:19:04

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The problem, RmInitAdapter failed! (0x23:0x2f:*LineNumber*), was fixed with the kvm=off patch!

Last edited by lojsar (2014-06-17 20:00:51)

Offline

#2120 2014-06-17 07:15:02

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

Alright, since I'm sick of not getting my GTX 770 performance problem fixed, I built in my AMD HD 6950.
Driver install worked and the short freezes are finally gone. <3
BUT the Benchmark is still pretty low.. http://www.3dmark.com/3dm11/8432530
Since there is no user with this setup, I searched for users with the i7 4770 which is pretty much the same like my Xeon E3-1245v3 and these guys have an average of 9000 in physics and 5500 in graphics score.. which means that my performance sucks and isn't even above 90% of the native performance..

As you can see it doesn't even push my Xeon into 3.8Ghz turbomode and my GPU isn't very fast aswell, with my GTX 770 I had the average of 11000 graphics score which was higher than some native users benchmarks, only the cpu slowed down. But what could cause this now?

Another little question, I'm using libvirt but can't get the emulated sound working, passthrough with libvirt works fine, only sound causes me trouble... I used the parameter in the first post, sound device is shown in windows, but I can't hear anything.

/bump

I saw the OP has a similar score with the same GPU, but that post was also years ago, so I dunno anymore.. tried with so many different CPU options in libvirt.. I even tried cpu mode host-passthrough, the benchmark said the CPU had 4GHz in turbomode which can't be possible since the Xeon only has 3,8 GHz in turbomode.. the score was even lower.

Offline

#2121 2014-06-17 10:50:01

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

BUT the Benchmark is still pretty low.. http://www.3dmark.com/3dm11/8432530
Since there is no user with this setup, I searched for users with the i7 4770 which is pretty much the same like my Xeon E3-1245v3 and these guys have an average of 9000 in physics and 5500 in graphics score.. which means that my performance sucks and isn't even above 90% of the native performance..

Your results aren't that bad. The average for your graphics card is only 5240 (with 4900 you get 93% of that). The CPU score could be better, though (since the average is 8960).
Anyway, if you haven't done this yet, you can try:
1. Backing memory with hugepages.
2. Pinning vcpus.
3. Adding hyper-v enlightenments (hv-time, hv-relaxed, hv-vapic, hv-spinlocks=0x1000).
4. Kernel 3.15 since it has the debug registers patch.
5. Let libvirt set the CPU model without host passthrough.

While I was at it, I tried this benchmark and was quite pleased with what I got: http://www.3dmark.com/3dm11/8436872 The time measurement error is probably because of hv-time. My grapics card score is 14 points above the average.  As for my CPU (E3-1245v2), it supports HT but for this win7 vm I use only the four main cores because most games don't profit much from HT and the "emulated" cores are kind of dedicated to kvm and other vms. So for 3DMark needs I have i5-3570 with the average score of 6420 while my score is 7165.

Offline

#2122 2014-06-17 20:20:39

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a Dell XPS-15 (L502X) laptop with nVidia GT525M in an nVidia Optimus configuration.
The Intel GPU is connected to the internal display.
The nVidia GPU is connected to a HDMI port, which to I connected a monitor.
VBIOS is stored in ACPI. I extracted it from nouveau's debugfs.

After some fiddling, I now have it booting:

seabios runs the VGA BIOS, but nothing is displayed on the monitor.

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

I see that the nVidia driver thinks CRT-0 is connected. Could this be why I don't see anything from BIOS? I was thinking maybe the card is using the wrong output. (DFP-0/HDMI is working! I just have to disable CRT-0 with xrandr to get a sane resolution)

qemu command

exec qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 2048 \ -cpu host,kvm=off \ -vga none \ -chardev stdio,id=seabios \ -device isa-debugcon,iobase=0x402,chardev=seabios \ -drive if=virtio,file=debian.qcow2,id=disk,format=qcow2,cache=off \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,id=gt525m,romfile=nouveau-vbios.rom,x-vga=on \

host: linux-3.15.0 (i915.enable_hd_vgaarb=1)
qemu: 2.0.0 with kvm=off patch
seabios: 1.7.4
guest: Debian Sid /w linux-3.14.1 and nvidia-340.17

Last edited by lojsar (2014-06-17 20:53:02)

Offline

#2123 2014-06-17 20:31:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lojsar wrote:

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

Is the VBIOS you extracted actually a PCI ROM?  Run 'xxd nouveau-vbios.rom | less' and you should see these important markers:

0000000: 55aa 7eeb 4b37 3430 30e9 4c19 77cc 5649 U.~.K7400.L.w.VI 0000010: 4445 4f20 0d00 0000 8801 cf18 0000 4942 DEO ..........IB 0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65 M VGA Compatible 0000030: 0100 0000 8000 58be 3130 2f32 302f 3039 ......X.10/20/09

The first two bytes should always be 55aa

0000180: 91df aa8c 9af2 f5ff 5043 4952 de10 740a ........PCIR..t.

Somewhere not too far in, you should be able to find "PCIR" and following it will be the device it's for in little endian, ex: de10 740a = 0x10de 0x0a74, which translates to NVIDIA Vendor ID, G210M device ID (http://pci-ids.ucw.cz/read/PC/10de/0a74).  I just happen to have this ROM sitting around, the device itself only works with the basic VGA driver in a VM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2124 2014-06-17 20:43:25

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
lojsar wrote:

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

Is the VBIOS you extracted actually a PCI ROM?  Run 'xxd nouveau-vbios.rom | less' and you should see these important markers:

0000000: 55aa 7eeb 4b37 3430 30e9 4c19 77cc 5649 U.~.K7400.L.w.VI 0000010: 4445 4f20 0d00 0000 8801 cf18 0000 4942 DEO ..........IB 0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65 M VGA Compatible 0000030: 0100 0000 8000 58be 3130 2f32 302f 3039 ......X.10/20/09

The first two bytes should always be 55aa

0000180: 91df aa8c 9af2 f5ff 5043 4952 de10 740a ........PCIR..t.

Somewhere not too far in, you should be able to find "PCIR" and following it will be the device it's for in little endian, ex: de10 740a = 0x10de 0x0a74, which translates to NVIDIA Vendor ID, G210M device ID (http://pci-ids.ucw.cz/read/PC/10de/0a74).  I just happen to have this ROM sitting around, the device itself only works with the basic VGA driver in a VM.

I think it is. seabios seemed to run it at least (I enabled seabios debug output).

0000000: 55aa 7feb 4b37 3430 30e9 4c19 77cc 5649 U...K7400.L.w.VI 0000010: 4445 4f20 0d00 0000 8801 3c19 0000 4942 DEO ......<...IB 0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65 M VGA Compatible 0000030: 0100 0000 c000 3ab6 3032 2f31 362f 3131 ......:.02/16/11 0000040: 0000 0000 0000 0000 0010 4000 51fd 24fd ..........@.Q.$. 0000050: e996 2a00 2810 6704 ff3f fc7f 0040 0000 ..*.(.g..?...@.. 0000060: ffff ff7f 0000 0080 493f a5de e9fe 46e9 ........I?....F. 0000070: 0547 504d 4944 6c00 6f00 0000 00a0 00b0 .GPMIDl.o....... 0000080: 00b8 00c0 0033 4746 3130 3820 4531 3037 .....3GF108 E107 0000090: 3920 534b 5520 3030 3036 2056 4741 2042 9 SKU 0006 VGA B 00000a0: 494f 530d 0a00 0000 0000 0000 0000 0000 IOS............. 00000b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000c0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000d0: 0000 0000 0000 0056 6572 7369 6f6e 2037 .......Version 7 00000e0: 302e 3038 2e35 332e 3030 2e30 3420 0d0a 0.08.53.00.04 .. 00000f0: 0043 6f70 7972 6967 6874 2028 4329 2031 .Copyright (C) 1 0000100: 3939 362d 3230 3131 204e 5649 4449 4120 996-2011 NVIDIA 0000110: 436f 7270 2e0d 0a00 0000 ffff 0000 0000 Corp............ 0000120: ffff 4746 3130 3820 426f 6172 6420 2d20 ..GF108 Board - 0000130: 3130 3739 6466 3530 0000 0000 0000 0000 1079df50........ 0000140: 0000 0000 0043 6869 7020 5265 7620 2020 .....Chip Rev 0000150: 0000 0000 0000 0000 00ba 9198 9691 9a9a ................ 0000160: 8d96 9198 dfad 9a93 9a9e 8c9a dfd2 dfb1 ................ 0000170: 908b dfb9 908d dfaf 8d90 9b8a 9c8b 9690 ................ 0000180: 91df aa8c 9af2 f5ff 5043 4952 de10 f50d ........PCIR.... 0000190: 0000 1800 0000 0003 7f00 0100 0080 0000 ................ 00001a0: 4859 4224 ffb8 4249 5400 0001 0c06 1245 HYB$..BIT......E 00001b0: 3201 0400 2c02 4202 2100 3802 4301 0e00 2...,.B.!.8.C... 00001c0: 5902 4401 0400 6702 4101 0300 6b02 4901 Y.D...g.A...k.I. 00001d0: 1200 6e02 4c01 0200 8002 4d02 1100 8202 ..n.L.....M..... 00001e0: 4e00 0000 0000 5002 3000 9302 5302 1500 N.....P.0...S... 00001f0: c302 5401 0200 d802 5501 0300 da02 5601 ..T.....U.....V. 0000200: 0600 dd02 7801 0800 e302 6401 0200 eb02 ....x.....d..... 0000210: 7001 0e00 ed02 6902 4200 fc02 0000 3e03 p.....i.B.....>. 0000220: 81b0 2dc5 46cd b5ef 0000 0000 0000 0000 ..-.F........... 0000230: 0000 0000 0000 0000 0053 0870 0400 0000 .........S.p.... 0000240: 0000 a807 0000 0000 0000 0000 0200 5c5c ..............\\ 0000250: 1c02 0000 3002 0400 0000 0000 0000 0000 ....0........... 0000260: 0065 b1c6 8700 0059 b4d8 4100 0000 94b2 .e.....Y..A..... 0000270: a0b2 a6b2 beb2 0eb4 59b4 a0b2 8aef 59b4 ........Y.....Y. 0000280: 10b9 084e b113 b2b9 d200 00dc d200 008a ...N............ 0000290: b200 0047 cd00 0072 ce00 007a d000 00b9 ...G...r...z.... 00002a0: d100 0061 d200 0000 0000 00a5 d200 0000 ...a............ 00002b0: 0000 00e8 d100 00ac d100 003e d200 0042 ...........>...B 00002c0: d200 0086 0050 d700 19f1 0028 81b0 1495 .....P.....(.... 00002d0: b023 2201 2345 0114 24b5 b8b0 0033 b1ef .#".#E..$....3.. 00002e0: b000 0000 0001 0100 0000 00dd 4300 0000 ............C... 00002f0: 0000 0000 0000 0000 0000 0000 0053 0870 .............S.p 0000300: 0410 003f 0f85 001e ce00 0030 322f 3033 ...?.......02/03 0000310: 2f31 3100 0000 0000 0000 0000 008e ef01 /11............. 0000320: 0100 0000 0000 0000 0000 0000 3730 3000 ............700. 0000330: 3130 3739 3030 3030 0000 0000 0000 1000 10790000........

I uploaded the whole file here: http://cnu.no-ip.com/u/KQuzzYhbdKzG2WMI … -vbios.rom

#nouveau wasn't sure if the binary driver checked PCIROM at all. They suggested I try a tool called "nvafakebios". I will do that now!

Thank you!

Offline

#2125 2014-06-17 20:45:00

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lojsar wrote:

I have a Dell XPS-15 (L502X) laptop with nVidia GT525M in an nVidia Optimus configuration.
The Intel GPU is connected to the internal display.
The nVidia GPU is connected to a HDMI port, which to I connected a monitor.
VBIOS is stored in ACPI. I extracted it from nouveau's debugfs.

After some fiddling, I now have it booting:

seabios runs the VGA BIOS, but nothing is displayed on the monitor.

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

I see that the nVidia driver thinks CRT-0 is connected. Could this be why I don't see anything from BIOS? I was thinking maybe the card is using the wrong output.

qemu command

exec qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 2048 \ -cpu host,kvm=off \ -vga none \ -chardev stdio,id=seabios \ -device isa-debugcon,iobase=0x402,chardev=seabios \ -drive if=virtio,file=debian.qcow2,id=disk,format=qcow2,cache=off \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,id=gt525m,romfile=nouveau-vbios.rom,x-vga=on \

host: linux-3.15.0 (i915.enable_hd_vgaarb=1)
qemu: 2.0.0 with kvm=off patch
seabios: 1.7.4
guest: Debian Sid /w linux-3.14.1 and nvidia-340.17


Most probably you nvidia gpu is not connected to HDMI, but to igp. Thats why it thinks it has CRT-0 connected.

BTW: I'm also fighting with passing through my nVidia GPU, but I always got error 43 and 10/12 (cant remember) before installing nvidia driver in windows.
Can you tell sth more about what you done? Maybe some instruction please?

Offline

#2126 2014-06-17 20:56:47

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:
lojsar wrote:

...

Most probably you nvidia gpu is not connected to HDMI, but to igp. Thats why it thinks it has CRT-0 connected.

I see my post was not very clear. The HDMI output works. It's just that I get a CRT-0 output too, that I have to disable to get a sane resolution. I tried to clarify this in my post now.

belliash wrote:

BTW: I'm also fighting with passing through my nVidia GPU, but I always got error 43 and 10/12 (cant remember) before installing nvidia driver in windows.
Can you tell sth more about what you done? Maybe some instruction please?

I haven't figured out how to make it work in Windows yet. When I tried Windows I only got Code 10, so I changed to Linux to get more error messages (I recommend NVreg_ResmanDebugLevel=0)

Offline

#2127 2014-06-17 21:08:30

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lojsar wrote:
belliash wrote:
lojsar wrote:

...

Most probably you nvidia gpu is not connected to HDMI, but to igp. Thats why it thinks it has CRT-0 connected.

I see my post was not very clear. The HDMI output works. It's just that I get a CRT-0 output too, that I have to disable to get a sane resolution. I tried to clarify this in my post now.

belliash wrote:

BTW: I'm also fighting with passing through my nVidia GPU, but I always got error 43 and 10/12 (cant remember) before installing nvidia driver in windows.
Can you tell sth more about what you done? Maybe some instruction please?

I haven't figured out how to make it work in Windows yet. When I tried Windows I only got Code 10, so I changed to Linux to get more error messages (I recommend NVreg_ResmanDebugLevel=0)

Can you check one thing?
Please pass your nvidia gpu as a second card to VM (use also a cirrus/vmware/etc) and install Windows. Then please install binary nvidia drivers and reboot host.
Finally launch guest and if you see you nvidia card detected properly but with exclamation mark, please check error code. If it will be 43, please disable and reenable GPU from device manager.

These steps help to get dgpu working in guest windows os. However it remains secondary gpu, so its useless. I wonder if your problem is the same is mine...

Offline

#2128 2014-06-18 10:43:27

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

First of all, thanks OP for the tutorial.

I am running Ubuntu 14.04 and trying to run KVM VGA passthrough. I have successfully done several VGA passthrough with various distros and with Xen too. I always use IGP as main card for my Linux rig, and PCIe card as my Windows GPU. Everything works fine. I even got it working with 2 ATI cards, a 6450 in 1st PCie for the Linux host and a 7950 in 2nd PCIe slot for Windows guest. However, gaming performance are really poor since the 2nd PCIe slot is limited to 1x, and that I don't have enough physical space in my case to plug the cards differently.

I'd like to get a WQHD compliant setup. Now, what I want to do and can't seem to get working is to have my Linux rig use my 2nd PCIe (ATI 6450) card as GPU and my Windows rig use the first PCIe card (ATI 7950). The best I could get is a working and accelerated Linux desktop environnement on second GPU, pci-stub claimed the corrrect devices, vfio module and kvm loaded, etc. When I start my Windows guest with PCI devices attached to it, it starts but I have no display on guest and my qemu logfile is literraly spammed (until the /var parition is full) with something like:

qemu-system-x86_64: vfio_bar_write failed: Device or resourc e busy

I am not providing much log atm, because it might not be needed. Indeed, I remember reading that I can not pass the secondary video adapter to guest. So is it possible to get the 2nd PCIe card passed to guest and the 1st PCIe card for host?

Thanks in avance,

NB: if I absolutely want to pass the 2nd to guest, it is because I have no choice with my current case / motherboard.

Offline

#2129 2014-06-18 10:50:07

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Hi all,

First of all, thanks OP for the tutorial.

I am running Ubuntu 14.04 and trying to run KVM VGA passthrough. I have successfully done several VGA passthrough with various distros and with Xen too. I always use IGP as main card for my Linux rig, and PCIe card as my Windows GPU. Everything works fine. I even got it working with 2 ATI cards, a 6450 in 1st PCie for the Linux host and a 7950 in 2nd PCIe slot for Windows guest. However, gaming performance are really poor since the 2nd PCIe slot is limited to 1x, and that I don't have enough physical space in my case to plug the cards differently.

I'd like to get a WQHD compliant setup. Now, what I want to do and can't seem to get working is to have my Linux rig use my 2nd PCIe (ATI 6450) card as GPU and my Windows rig use the first PCIe card (ATI 7950). The best I could get is a working and accelerated Linux desktop environnement on second GPU, pci-stub claimed the corrrect devices, vfio module and kvm loaded, etc. When I start my Windows guest with PCI devices attached to it, it starts but I have no display on guest and my qemu logfile is literraly spammed (until the /var parition is full) with something like:

qemu-system-x86_64: vfio_bar_write failed: Device or resourc e busy

I am not providing much log atm, because it might not be needed. Indeed, I remember reading that I can not pass the secondary video adapter to guest. So is it possible to get the 2nd PCIe card passed to guest and the 1st PCIe card for host?

Thanks in avance,

NB: if I absolutely want to pass the 2nd to guest, it is because I have no choice with my current case / motherboard.

what card is used by bios? are you sure you are not getting hit by lack of bios configurability in terms of what non-integrated GPU is used as primary adapter?
using IGP for host os is not an option?

Offline

#2130 2014-06-18 11:14:30

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

what card is used by bios? are you sure you are not getting hit by lack of bios configurability in terms of what non-integrated GPU is used as primary adapter?
using IGP for host os is not an option?

Card used by bios is the 1st PCIe card (aka 7950, the one I want passed to Windows).

My bios kinda sucks (Asrock Z77 Pro 4M); since I can only select from IGP or 1st PCIe card. I wish I could use my 2nd PCIe card as "main GPU".

IGP for host is no longer an option since I plan on buying a WQHD screen and IGP doesn't support that resolution. sad

Offline

#2131 2014-06-18 11:45:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
sinny wrote:

what card is used by bios? are you sure you are not getting hit by lack of bios configurability in terms of what non-integrated GPU is used as primary adapter?
using IGP for host os is not an option?

Card used by bios is the 1st PCIe card (aka 7950, the one I want passed to Windows).

My bios kinda sucks (Asrock Z77 Pro 4M); since I can only select from IGP or 1st PCIe card. I wish I could use my 2nd PCIe card as "main GPU".

IGP for host is no longer an option since I plan on buying a WQHD screen and IGP doesn't support that resolution. sad

Using UEFI i'm able to passthrough my primary card by disabling efifb with video=efifb:off as a kernel parameter

Offline

#2132 2014-06-18 14:59:09

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Using UEFI i'm able to passthrough my primary card by disabling efifb with video=efifb:off as a kernel parameter

Great news, if it works for you then I might get something working too.

So far the kernel parameter didin't doo anything, but I believe I have to fix that libvirtd error first. Just curious, did you anything special with UEFI to make it work?

And if it helps anyone, the error I'am getting spammed with in my guest domain's log is:

qemu-system-x86_64: vfio_bar_write(,0xf7bf4e0, 0x0, 4) failed: Device or resource busy

Last edited by Nesousx (2014-06-18 15:49:41)

Offline

#2133 2014-06-18 17:02:16

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lil update on my progress...


got gigabyte g1 sniper 3...
4-way sli

i7 2600s

it works with 4 gpu-s...

2x 560ti
1x 660
1x 8800gt

all working... debian, linux-vfio 3.15, acs=downstream

ill post benchs latter with better gpu-s...
Yj9Eg0Us.jpg N1ROzj5s.jpg

Last edited by slis (2014-06-18 17:07:31)

Offline

#2134 2014-06-18 17:42:35

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
lojsar wrote:

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

Is the VBIOS you extracted actually a PCI ROM?  Run 'xxd nouveau-vbios.rom | less' and you should see these important markers:

You were of course correct! At some point I mistyped a character into the romfile= parameter, which apparently wasn't fatal for qemu, it just started without a ROM since the file didn't exist. Now I don't need my driver modification anymore. Thank you again! and sorry for being such an idiot :(

Furthermore, I had great success modifying the VBIOS. Using a program called "nvbios" from "envytools" together with nVidia's documentation of the DCB ( ftp://download.nvidia.com/open-gpu-doc/ … _dcb_flags ), I was able to replace the VGA output. After this, I see BIOS output on my monitor! I was also able to run Windows (with -vga none), and install the nVidia drivers!

Rebooting the guest VM doesn't seem to work tho. I've observed the same behavior when rebooting inside the guest, and shutting down the guest and starting it again. It stays a few second at "Running option rom at ce00:0003", and if I don't kill qemu fast enough the whole (host) computer freezes. I will look into this now, as rebooting seems to be an important part of Windows.

Can you check one thing?
Please pass your nvidia gpu as a second card to VM (use also a cirrus/vmware/etc) and install Windows. Then please install binary nvidia drivers and reboot host.
Finally launch guest and if you see you nvidia card detected properly but with exclamation mark, please check error code. If it will be 43, please disable and reenable GPU from device manager.

These steps help to get dgpu working in guest windows os. However it remains secondary gpu, so its useless. I wonder if your problem is the same is mine...

I didn't try this since it worked as primary GPU. My only modifications to qemu now is the kvm=off patch. You could perhaps check the DCB of your card. I assumed the VGA stuff only used the first output, which in my case wasn't really connected.

Last edited by lojsar (2014-06-18 19:04:47)

Offline

#2135 2014-06-18 17:56:38

jajacomp
Member
Registered: 2014-06-18
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! I have problem. I use:
- motherboard Asus z87pro
- first VGA - nvidia gtx 650 (HOST)
- second VGA - ATI 7770 (GUEST)

If i use first vga for system then i can't passthrough second card to virtual machine (Error = vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver)
if i use integrated video card (Intel hd 4600) for system and passthrough both video card (vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1) then i can use ATI card without problems.

version qemu is latest from git
kernel 3.15.0
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=1002:683d,1002:aab0,8086:0c05 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"


Please! Help! I want use nvidia for main system

Last edited by jajacomp (2014-06-18 17:58:48)

Offline

#2136 2014-06-18 18:56:01

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lojsar wrote:

I didn't try this since it worked as primary GPU. My only modifications to qemu now is the kvm=off patch. You could perhaps check the DCB of your card. I assumed the VGA stuff only used the first output, which in my case wasn't really connected.

Can you tell me exactly how and what should I check?

Offline

#2137 2014-06-18 18:59:10

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:

Hi! I have problem. I use:
- motherboard Asus z87pro
- first VGA - nvidia gtx 650 (HOST)
- second VGA - ATI 7770 (GUEST)

If i use first vga for system then i can't passthrough second card to virtual machine (Error = vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver)
if i use integrated video card (Intel hd 4600) for system and passthrough both video card (vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1) then i can use ATI card without problems.

version qemu is latest from git
kernel 3.15.0
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=1002:683d,1002:aab0,8086:0c05 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"


Please! Help! I want use nvidia for main system

is it just me seeing things or there are no nvidia devices in pci-stub.ids kernel parameter (amd and intel only; why intel btw?)?

Offline

#2138 2014-06-18 19:09:47

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:
lojsar wrote:

I didn't try this since it worked as primary GPU. My only modifications to qemu now is the kvm=off patch. You could perhaps check the DCB of your card. I assumed the VGA stuff only used the first output, which in my case wasn't really connected.

Can you tell me exactly how and what should I check?

Run nvbios (from https://github.com/envytools/envytools/ ) on your VBIOS dump. Then check the DCB section.
Do you also have a laptop with Optimus btw? If it's a normal desktop computer I don't think you will have the same problem.

Maybe you could try Linux as a guest instead. I found it easier to debug.

Offline

#2139 2014-06-18 19:11:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:

kernel 3.15.0
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=1002:683d,1002:aab0,8086:0c05 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"

The ACS override option requires a kernel patch, did you apply it?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2140 2014-06-18 19:13:14

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lojsar wrote:
belliash wrote:
lojsar wrote:

I didn't try this since it worked as primary GPU. My only modifications to qemu now is the kvm=off patch. You could perhaps check the DCB of your card. I assumed the VGA stuff only used the first output, which in my case wasn't really connected.

Can you tell me exactly how and what should I check?

Run nvbios (from https://github.com/envytools/envytools/ ) on your VBIOS dump. Then check the DCB section.
Do you also have a laptop with Optimus btw? If it's a normal desktop computer I don't think you will have the same problem.

Maybe you could try Linux as a guest instead. I found it easier to debug.

Yes, It's notebook equipped with gtx770m

Offline

#2141 2014-06-18 19:19:14

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw: Do you know if https://lkml.org/lkml/2014/5/25/94 is already in 3.16-rc1?

Offline

#2142 2014-06-18 19:22:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

aw: Do you know if https://lkml.org/lkml/2014/5/25/94 is already in 3.16-rc1?

It is not https://git.kernel.org/cgit/linux/kerne … a/vgaarb.c


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2143 2014-06-18 19:42:29

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@lojsar: All I got about DCB is:

Unknown DCB table version 7.13
Failed to parse DCB table at 0xffff version 7.13


Im not sure if i done it correctly, but I used vbios image ready to flash to card.

Last edited by belliash (2014-06-18 19:44:33)

Offline

#2144 2014-06-18 20:10:58

jajacomp
Member
Registered: 2014-06-18
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
jajacomp wrote:

Hi! I have problem. I use:
- motherboard Asus z87pro
- first VGA - nvidia gtx 650 (HOST)
- second VGA - ATI 7770 (GUEST)

If i use first vga for system then i can't passthrough second card to virtual machine (Error = vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver)
if i use integrated video card (Intel hd 4600) for system and passthrough both video card (vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1) then i can use ATI card without problems.

version qemu is latest from git
kernel 3.15.0
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=1002:683d,1002:aab0,8086:0c05 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"


Please! Help! I want use nvidia for main system

is it just me seeing things or there are no nvidia devices in pci-stub.ids kernel parameter (amd and intel only; why intel btw?)?

I add to pci-stub this:
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
and this
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)

Offline

#2145 2014-06-18 20:11:32

jajacomp
Member
Registered: 2014-06-18
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
jajacomp wrote:

kernel 3.15.0
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=1002:683d,1002:aab0,8086:0c05 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"

The ACS override option requires a kernel patch, did you apply it?

Hm, no. It can help?

Offline

#2146 2014-06-18 20:13:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:
sinny wrote:
jajacomp wrote:

Hi! I have problem. I use:
- motherboard Asus z87pro
- first VGA - nvidia gtx 650 (HOST)
- second VGA - ATI 7770 (GUEST)

If i use first vga for system then i can't passthrough second card to virtual machine (Error = vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver)
if i use integrated video card (Intel hd 4600) for system and passthrough both video card (vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1) then i can use ATI card without problems.

version qemu is latest from git
kernel 3.15.0
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=1002:683d,1002:aab0,8086:0c05 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"


Please! Help! I want use nvidia for main system

is it just me seeing things or there are no nvidia devices in pci-stub.ids kernel parameter (amd and intel only; why intel btw?)?

I add to pci-stub this:
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
and this
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)

Forcing root ports or bridges to bind to pci-stub is a really bad idea.  They should also not be bound to vfio-pci.  Leave them alone.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2147 2014-06-18 20:33:05

jajacomp
Member
Registered: 2014-06-18
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
jajacomp wrote:
sinny wrote:

is it just me seeing things or there are no nvidia devices in pci-stub.ids kernel parameter (amd and intel only; why intel btw?)?

I add to pci-stub this:
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
and this
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)

Forcing root ports or bridges to bind to pci-stub is a really bad idea.  They should also not be bound to vfio-pci.  Leave them alone.

Ok. I delete it. Where i can find ACS patch for 3.15 kernel?

Perfect!!! I apply patch and all work! Thank!!!

Last edited by jajacomp (2014-06-18 21:05:35)

Offline

#2148 2014-06-18 21:31:51

lojsar
Member
Registered: 2014-06-16
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do you know how to make rebooting work?

If I boot Windows 7 now, and
a) restart it from Windows
b) shut down from Windows, and start qemu again
the host will freeze when seabios is "Running option rom at c000:0003".

I tried to search around in this thread, and found a post on howto reset the PCI bus by oring 0x40 into a register. After doing this the host doesn't freeze anymore, but I still doesn't get past "Running option rom at c000:0003". I attached some lspci output.

Are there any more tricks? :

After booting the laptop and running vfio-bind.sh

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00003000-00003fff Memory behind bridge: f0000000-f10fffff Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Kernel driver in use: vfio-pci 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108M [GeForce GT 525M] [10de:0df5] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Dell Device [1028:04b6] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 11 Region 0: Memory at f0000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at 3000 [disabled] [size=128] Expansion ROM at f1000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia

After running Windows 7 in qemu and shutting down (all the BARs are [disabled] + many more changes)

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00003000-00003fff Memory behind bridge: f0000000-f10fffff Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Kernel driver in use: vfio-pci 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108M [GeForce GT 525M] [10de:0df5] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Dell Device [1028:04b6] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 3: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=32M] Region 5: I/O ports at 3000 [disabled] [size=128] Expansion ROM at f1000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia

After running vfio-unbind.sh (I didn't see any interesting changes)

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00003000-00003fff Memory behind bridge: f0000000-f10fffff Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108M [GeForce GT 525M] [10de:0df5] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Dell Device [1028:04b6] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 3: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=32M] Region 5: I/O ports at 3000 [disabled] [size=128] Expansion ROM at f1000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel modules: nouveau, nvidia

After running setpci on the bridge, writing 0x40 and clearing. Now the BARs are back! IO ports have also been enabled (they weren't after booting)

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00003000-00003fff Memory behind bridge: f0000000-f10fffff Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108M [GeForce GT 525M] [10de:0df5] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Dell Device [1028:04b6] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: [virtual] Memory at f0000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at 3000 [size=128] [virtual] Expansion ROM at f1000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel modules: nouveau, nvidia

After running vfio-bind.sh again (I didn't see any interesting changes)

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 00003000-00003fff Memory behind bridge: f0000000-f10fffff Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: Intel Corporation Device [8086:2010] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Kernel driver in use: vfio-pci 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108M [GeForce GT 525M] [10de:0df5] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Dell Device [1028:04b6] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: [virtual] Memory at f0000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at 3000 [size=128] [virtual] Expansion ROM at f1000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia
belliash wrote:

@lojsar: All I got about DCB is:

Unknown DCB table version 7.13
Failed to parse DCB table at 0xffff version 7.13


Im not sure if i done it correctly, but I used vbios image ready to flash to card.

That looks completely wrong since the latest version is 4.0. Maybe you can upload the VBIOS. Do you use the same file as romfile= when starting qemu?

Offline

#2149 2014-06-18 22:22:23

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I compiled the kernel 3.16-rc1 with VFIO,
Installed
Qemu 2.0.0
LibVirt 1.2.2

intel_immou=on was added to grub and updated.
changed the privileges in qemu.conf.
Now, when running virt-manager I get the error "vfio group".

Still testing...
I will be grateful for any attempt to help

Offline

#2150 2014-06-18 22:54:21

jajacomp
Member
Registered: 2014-06-18
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have problem with screen size on monitor. If i set resolution 1080 - then image don't fill all screen, just 80%. How fix if? Thank!

Offline

#2151 2014-06-18 23:03:05

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:

I have problem with screen size on monitor. If i set resolution 1080 - then image don't fill all screen, just 80%. How fix if? Thank!

This isn't really specific to this thread, but anyways… It's called overscan and there is an option in the AMD Catalyst Control Center to disable it.


i'm sorry for my poor english wirting skills…

Offline

#2152 2014-06-18 23:47:28

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:

I have problem with screen size on monitor. If i set resolution 1080 - then image don't fill all screen, just 80%. How fix if? Thank!

The answer is less than a page away...

https://bbs.archlinux.org/viewtopic.php … 4#p1426184

Offline

#2153 2014-06-19 14:42:57

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey I ran into some problems with my ACS Override patch (on the official gentoo/arch kernels) turns out the patch breaks vdpau (hardware decoding) for me and all applications that use or rely on it.

I tried to install the linux-mainline kernel from the OP, but now I cannot start X, here is my xorg.log. What did I do wrong?

Offline

#2154 2014-06-19 15:01:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Hey I ran into some problems with my ACS Override patch (on the official gentoo/arch kernels) turns out the patch breaks vdpau (hardware decoding) for me and all applications that use or rely on it.

This seems very unlikely to me, but if it were to break something it would probably be an indicator that the hardware in fact does not have isolation between downstream ports and the kernel is correct in grouping them together.  Is the issue with a VM running?  vdpau in the VM?

I tried to install the linux-mainline kernel from the OP, but now I cannot start X, here is my xorg.log. What did I do wrong?

It just seems to be telling us to go look at dmesg

[ 114.334] (EE) NVIDIA: Failed to initialize the NVIDIA kernel module. Please see the [ 114.334] (EE) NVIDIA: system's kernel log for additional error messages and [ 114.334] (EE) NVIDIA: consult the NVIDIA README for details.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2155 2014-06-19 15:03:39

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah I just found out through a bit of coincidence it was not the ACS Override patch that broke my hwdec (I updated the post)

It is the intel_iommu=on(CONFIG_INTEL_IOMMU_DEFAULT_ON=y) setting that causes my issue.

The problem is with VDPAU on the host, not the VM (vm not even running) it also doesn't matter if devices are bound to vfio-pci.

And the following commands find nothing. (They do on the official kernel though)

$dmesg | grep nvidia $dmesg | grep NVIDIA

The only thing I've tried to solve the problem with the linux-mainline kernel is to re-compile the nvidia drivers.It had no effect.As for the issue with Intel DMA Remapping by default, maybe that should be reported to somewhere?

Last edited by rabcor (2014-06-19 15:38:43)

Offline

#2156 2014-06-19 15:22:09

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Aight my mainboard died today <3
I won't ever buy MSI products again.. tzz..

But I came up with another question, has anyone got crossfire working with passthrough?
Thinking about buying another HD 6950 and run both with crossfire, would this be possible?

Offline

#2157 2014-06-19 18:27:44

Kaurin
Member
From: Ireland
Registered: 2011-12-11
Posts: 66

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I've come on board with this and really want to make it work.

I was hitting This bug, and with help of aw, was able to fix it (Still waiting confirmation from aw).

Here is my setup:

Mobo: Gigabyte Q87M-D2H, BIOS F7 01/17/2014 (latest bios)
CPU: Intel Core i7 4770
Ram: 32GB @ 1600Mhz
GPU: Gigabyte Nvidia GTX 770, 4GB slightly overclocked by Gigabyte
OS: Arch

My primary display is set to the Intel integrated graphics, and I'm running a bare Arch + qemu, so no graphical environment.

If I run Linux with a graphical environment, It's going to be a virt, with spice probably, and then use that via Windows, which is also a virt :>

Edit:
Grub:

linux /vmlinuz-3.15.0-rc6-ARCH-dirty root=/dev/mapper/sys-root ro intel_iommu=on vfio_iommu_type1

Before even trying GPU passthrough, this was the command line I used to install/test Windows7:

qemu-system-x86_64 \ -name win7 \ -enable-kvm \ -M q35 \ -net bridge,br=br0 \ -net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \ -nographic \ -spice addr=127.0.0.1,port=5900,disable-ticketing \ -vga qxl \ -cpu host \ -smp cores=2,threads=2 \ -m 8G \ -soundhw hda \ -balloon none \ -rtc base=localtime,clock=vm \ -drive file=/dev/sdb,index=0,if=virtio,discard=on

Win7 works like this, using spice (via ssh tunnel), and everything is dandy.

I've tried testing the GPU passtrhough (as per OP), like so:

Bind the GPU to vfio-pci:

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

This is from journalctl -b0 -ocat

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1 VFIO - User Level meta-driver version: 0.3 vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Try to boot:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Running this yields no errors, and I could hear the GPU fans spin down, and back up. However, the display does not turn on (no-signal state).

Then, I tried this:

qemu-system-x86_64 \ -name win7 \ -enable-kvm \ -M q35 \ -net bridge,br=br0 \ -net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \ -nographic \ -spice addr=127.0.0.1,port=5900,disable-ticketing \ -vga qxl \ -cpu host \ -smp cores=2,threads=2 \ -m 8G \ -soundhw hda \ -balloon none \ -rtc base=localtime,clock=vm \ -drive file=/dev/sdb,index=0,if=virtio,discard=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 

Basically the same command for win7 + GPU test. I logged in via spice, and noticed that I have a "Standard VGA Graphics Adapter" in Device Manager, but the device cannot start (Code 10). Hardware IDs look good: 10DE-1184

The "High Definition Audio Device" with the nvidia HWID seems to work (Can't really test it because I don't have HDMI cables / devices)

Can anyone suggest what to try next in order to get GPU-passtrhough to work?

Before anyone asks, the monitor is connected to the DVI port of the nvidia card since before I began testing GPU passthrough smile

Update: After a Windows Update/Reboot, the Nvidia GPU is shown as "NVIDIA GeForce GTX 770", and is now code 12 and still not working. (Cannot start b/c resources).

I have not mentioned this before, but I've tried doing the GPU passthrough on XEN last week, and hit the same thing there (device name OK + code 12). After seeing that gpu passthrough with nvidia is something I might not be able to do, i turned to KVM where I see people running virts + nvidia successfully.

Last edited by Kaurin (2014-06-19 19:44:15)

Offline

#2158 2014-06-19 19:43:20

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all.

I'm new here, but I've been reading the thread for a couple of months since I got the urge to build a server for streaming Steam from a workstation that recently came into my possession. Finally got all the parts together, and didn't have much luck with Arch since I'm not really familiar with it, so I'm now running Ubuntu Server 14.04.

Here's my config:
CPUs: 2x Xeon E5-2680
Memory: 64 GB ECC RAM @ 1600MHz
Mobo: Asus Z9PA-D8
GPU (Guest 1): MSI GTX770 4GB
GPU (Guest 2): Asus GTX560Ti 2GB

It's been going fairly well so far. Besides figuring out how to make an upstart equivalent of the systemd service, I haven't had any major issues setting up, to this point. I've been using virtual machine manager from my laptop, and configured a guest image I'm happy with. However, when I try to start the VM with the graphics card added, I get this error:

Error starting domain: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 9663676416: Operation not permitted Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup self._backend.create() File "/usr/lib/python2.7/dist-packages/libvirt.py", line 866, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 9663676416: Operation not permitted

Any ideas on what I'm doing wrong?

Offline

#2159 2014-06-19 19:44:32

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kaurin wrote:

Hi all,

I've come on board with this and really want to make it work.

I was hitting This bug, and with help of aw, was able to fix it (Still waiting confirmation from aw).

Here is my setup:

Mobo: Gigabyte Q87M-D2H, BIOS F7 01/17/2014 (latest bios)
CPU: Intel Core i7 4770
Ram: 32GB @ 1600Mhz
GPU: Gigabyte Nvidia GTX 770, 4GB slightly overclocked by Gigabyte
OS: Arch

My primary display is set to the Intel integrated graphics, and I'm running a bare Arch + qemu, so no graphical environment.

If I run Linux with a graphical environment, It's going to be a virt, with spice probably, and then use that via Windows, which is also a virt :>

Edit:
Grub:

linux /vmlinuz-3.15.0-rc6-ARCH-dirty root=/dev/mapper/sys-root ro intel_iommu=on vfio_iommu_type1

Before even trying GPU passthrough, this was the command line I used to install/test Windows7:

qemu-system-x86_64 \ -name win7 \ -enable-kvm \ -M q35 \ -net bridge,br=br0 \ -net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \ -nographic \ -spice addr=127.0.0.1,port=5900,disable-ticketing \ -vga qxl \ -cpu host \ -smp cores=2,threads=2 \ -m 8G \ -soundhw hda \ -balloon none \ -rtc base=localtime,clock=vm \ -drive file=/dev/sdb,index=0,if=virtio,discard=on

Win7 works like this, using spice (via ssh tunnel), and everything is dandy.

I've tried testing the GPU passtrhough (as per OP), like so:

Bind the GPU to vfio-pci:

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

This is from journalctl -b0 -ocat

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1 VFIO - User Level meta-driver version: 0.3 vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Try to boot:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Running this yields no errors, and I could hear the GPU fans spin down, and back up. However, the display does not turn on (no-signal state).

Then, I tried this:

qemu-system-x86_64 \ -name win7 \ -enable-kvm \ -M q35 \ -net bridge,br=br0 \ -net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \ -nographic \ -spice addr=127.0.0.1,port=5900,disable-ticketing \ -vga qxl \ -cpu host \ -smp cores=2,threads=2 \ -m 8G \ -soundhw hda \ -balloon none \ -rtc base=localtime,clock=vm \ -drive file=/dev/sdb,index=0,if=virtio,discard=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 

Basically the same command for win7 + GPU test. I logged in via spice, and noticed that I have a "Standard VGA Graphics Adapter" in Device Manager, but the device cannot start (Code 10). Hardware IDs look good: 10DE-1184

The "High Definition Audio Device" with the nvidia HWID seems to work (Can't really test it because I don't have HDMI cables / devices)

Can anyone suggest what to try next in order to get GPU-passtrhough to work?

Before anyone asks, the monitor is connected to the DVI port of the nvidia card since before I began testing GPU passthrough :)

do you expect anyone to guess all the tech details needed to even try helping you? this is simply rude, seriously...

anyway, my guess is missing vga arbiter patch in kernel

Offline

#2160 2014-06-19 19:54:50

Kaurin
Member
From: Ireland
Registered: 2011-12-11
Posts: 66

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I'm very sorry if it came off that way. It was definitely not my intention.
I will try to browse the thread again to see what I was missing.

I will look into the vga arbiter patch. Thank you!

Offline

#2161 2014-06-19 21:07:33

rakataka
Member
Registered: 2014-06-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kaurin wrote:

Hi all,

I've come on board with this and really want to make it work.

I was hitting This bug, and with help of aw, was able to fix it (Still waiting confirmation from aw).

Here is my setup:

Mobo: Gigabyte Q87M-D2H, BIOS F7 01/17/2014 (latest bios)
CPU: Intel Core i7 4770
Ram: 32GB @ 1600Mhz
GPU: Gigabyte Nvidia GTX 770, 4GB slightly overclocked by Gigabyte
OS: Arch

My primary display is set to the Intel integrated graphics, and I'm running a bare Arch + qemu, so no graphical environment.

If I run Linux with a graphical environment, It's going to be a virt, with spice probably, and then use that via Windows, which is also a virt :>

Edit:
Grub:

linux /vmlinuz-3.15.0-rc6-ARCH-dirty root=/dev/mapper/sys-root ro intel_iommu=on vfio_iommu_type1

Before even trying GPU passthrough, this was the command line I used to install/test Windows7:

qemu-system-x86_64 \ -name win7 \ -enable-kvm \ -M q35 \ -net bridge,br=br0 \ -net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \ -nographic \ -spice addr=127.0.0.1,port=5900,disable-ticketing \ -vga qxl \ -cpu host \ -smp cores=2,threads=2 \ -m 8G \ -soundhw hda \ -balloon none \ -rtc base=localtime,clock=vm \ -drive file=/dev/sdb,index=0,if=virtio,discard=on

Win7 works like this, using spice (via ssh tunnel), and everything is dandy.

I've tried testing the GPU passtrhough (as per OP), like so:

Bind the GPU to vfio-pci:

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

This is from journalctl -b0 -ocat

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1 VFIO - User Level meta-driver version: 0.3 vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Try to boot:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Running this yields no errors, and I could hear the GPU fans spin down, and back up. However, the display does not turn on (no-signal state).

Then, I tried this:

qemu-system-x86_64 \ -name win7 \ -enable-kvm \ -M q35 \ -net bridge,br=br0 \ -net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \ -nographic \ -spice addr=127.0.0.1,port=5900,disable-ticketing \ -vga qxl \ -cpu host \ -smp cores=2,threads=2 \ -m 8G \ -soundhw hda \ -balloon none \ -rtc base=localtime,clock=vm \ -drive file=/dev/sdb,index=0,if=virtio,discard=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 

Basically the same command for win7 + GPU test. I logged in via spice, and noticed that I have a "Standard VGA Graphics Adapter" in Device Manager, but the device cannot start (Code 10). Hardware IDs look good: 10DE-1184

The "High Definition Audio Device" with the nvidia HWID seems to work (Can't really test it because I don't have HDMI cables / devices)

Can anyone suggest what to try next in order to get GPU-passtrhough to work?

Before anyone asks, the monitor is connected to the DVI port of the nvidia card since before I began testing GPU passthrough smile

Update: After a Windows Update/Reboot, the Nvidia GPU is shown as "NVIDIA GeForce GTX 770", and is now code 12 and still not working. (Cannot start b/c resources).

I have not mentioned this before, but I've tried doing the GPU passthrough on XEN last week, and hit the same thing there (device name OK + code 12). After seeing that gpu passthrough with nvidia is something I might not be able to do, i turned to KVM where I see people running virts + nvidia successfully.

As it happens, we have an almost identical PC (I have the same motherboard, CPU, GFX card MSI GTX770 but with 2GB VRAM instead of 4, 32 GB RAM).
I may have been using kvm passthrough for a little bit longer than you. I updated today to linux-mainline 3.15.0 from 3.14.xx (can't remember exact xx). VGA passthrough worked fine on 3.14 with the following kernel line:

linux /boot/vmlinuz-linux-mainline root=/dev/sda1 rw modprobe.blacklist=nouveau,wmi,mxm_wmi,ttm nouveau.modeset=0 intel_iommu=on intel_iommu=igfx_off pcie_acs_override=downstream kvm.ignore_msrs=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off 

I should note that I have not used the patch you refer to in "This bug", so I get the related error in my boot log.
I am now (with the new kernel version 3.15.0) also getting a blank screen (I have a dedicated monitor connected to the HDMI output of my GTX770 card and I do not have the binary NVIDIA drivers install on the linux host) whereas it worked fine before with Windows 8.1 Pro 64bit and a number of games.
I have two things to note:

  1. I have noticed a new error when running quemu with passthough which I was not getting with the 3.14.xx kernel:

    VFIO - User Level meta-driver version: 0.3 Jun 19 22:59:05 laptop kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Jun 19 22:59:05 laptop kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Jun 19 22:59:09 laptop kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt Jun 19 23:00:00 laptop kernel: [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to 4510c

    I have read in previous thread posts that this is indicative of not having the i935 arbiter patch compiled but I used the package in the 1st post of this thread and just did

    makepkg; pacman -U *.pkg.tar.xz

    with no changes to the config.

  2. If you take a look at my kernel line (which I have carried over from 3.14.xx) I have used a

    intel_iommu=on intel_iommu=igfx_off

    instead of plain

    intel_iommu=on

    The extra

    intel_iommu=igfx_off

    made the difference between being able to do VGA passthough on 3.14.xx and failing with an error I cannot remember.

I wonder whether it is possible that something has been changed with respect to intel_iommu in 3.15.
If I find anything new I will update.

Offline

#2162 2014-06-19 21:34:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seriously people, if you have Intel graphics in your system, you need the i915 kernel patch.  If you have radeon host graphics, you need the other vga arbiter patch.  If you can't assign the devices you want because of iommu grouping, you need the ACS override patch.  If nothing happens on the display connected to the assigned GPU when you start the VM, you're probably missing the i915 or vga arbiter patches.  If you get screen corruption on the host, you're definitely missing them.  If you get weird intel error messages like above, you're definitely missing the i915 patch.

EDIT: If you think you have the i915 or ACS override patch, but it's not working, then you're not using the right kernel commandline options to enable them.

Last edited by aw (2014-06-19 21:36:04)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2163 2014-06-19 21:49:12

rakataka
Member
Registered: 2014-06-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Seriously people, if you have Intel graphics in your system, you need the i915 kernel patch.  If you have radeon host graphics, you need the other vga arbiter patch.  If you can't assign the devices you want because of iommu grouping, you need the ACS override patch.  If nothing happens on the display connected to the assigned GPU when you start the VM, you're probably missing the i915 or vga arbiter patches.  If you get screen corruption on the host, you're definitely missing them.  If you get weird intel error messages like above, you're definitely missing the i915 patch.

EDIT: If you think you have the i915 or ACS override patch, but it's not working, then you're not using the right kernel commandline options to enable them.

You are right, in my case (getting a blank screen but having used the correct kernel: 3.15.0 mainline from 1st post), I just need to add

i915.enable_hd_vgaarb=1

to my kernel command line an it worked fine.

As you can attest, this is neither horribly complex, but it's not simple either with multiple special cases. Also, I normally read change logs (I'm a developer myself) but I did not see one in a reasonably prominent place where I would be informed that I need to add an extra kernel parameter after upgrading to 3.15.

Please do not think that I take all your free work for granted, I am certainly very thankful for it and I try to do the extra work myself in order not to burden those doing the actual work. However, sometimes, one may miss some crucial information due to the lack of dedicated documentation, and searching on a forum is not always helpful (again, not saying that *you* should provide docs, just saying that mistakes may happen due to a lack of them).

Offline

#2164 2014-06-19 21:58:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rakataka wrote:
aw wrote:

Seriously people, if you have Intel graphics in your system, you need the i915 kernel patch.  If you have radeon host graphics, you need the other vga arbiter patch.  If you can't assign the devices you want because of iommu grouping, you need the ACS override patch.  If nothing happens on the display connected to the assigned GPU when you start the VM, you're probably missing the i915 or vga arbiter patches.  If you get screen corruption on the host, you're definitely missing them.  If you get weird intel error messages like above, you're definitely missing the i915 patch.

EDIT: If you think you have the i915 or ACS override patch, but it's not working, then you're not using the right kernel commandline options to enable them.

You are right, in my case (getting a blank screen but having used the correct kernel: 3.15.0 mainline from 1st post), I just need to add

i915.enable_hd_vgaarb=1

to my kernel command line an it worked fine.

As you can attest, this is neither horribly complex, but it's not simple either with multiple special cases. Also, I normally read change logs (I'm a developer myself) but I did not see one in a reasonably prominent place where I would be informed that I need to add an extra kernel parameter after upgrading to 3.15.

Please do not think that I take all your free work for granted, I am certainly very thankful for it and I try to do the extra work myself in order not to burden those doing the actual work. However, sometimes, one may miss some crucial information due to the lack of dedicated documentation, and searching on a forum is not always helpful (again, not saying that *you* should provide docs, just saying that mistakes may happen due to a lack of them).

Understood, for some reason the patches in the first post always get the changelog stripped out, the upstream patches do include a proper description:

ACS override
i915 VGA arbiter
VGA arbiter


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2165 2014-06-19 22:08:44

rakataka
Member
Registered: 2014-06-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeap, in my case (i915 VGA arbiter) I hope that if I had seen:

+module_param_named(enable_hd_vgaarb, i915.enable_hd_vgaarb, bool, 0444); +MODULE_PARM_DESC(enable_hd_vgaarb, + "Enable support for VGA arbitration on Intel HD IGD. (default: false)");

I would have taken the hint that a new parameter was available and I would have checked what's needed.

Thanks for the relevant links.

Offline

#2166 2014-06-20 03:38:38

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have been hit with a possible regression bug. A while ago I reported this:

System 2: AMD FX 8350 + Asus Sabretooth 990FX R2.0 (BIOS 2104) - OK - passthrough of the integrated audio device also tested to work with Win 7

But now, passing through the integrated audio device no longer works properly. When playing audio in guest I only get some horrible noise and, at the same time, lots of these messages appear on host:

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.2 domain=0x0025 address=0xffffffffffffffc0 flags=0x0010]

These are some of the most obvious changes since working setup:
ASUS BIOS 2104 -> 2501
Changed the host gpu card from HD3870 -> HD5450  (Can't test 3870 right now)
Guest card is still HD6670.
Linux 3.14.0-rc8(approx.) -> 3.14.8-gentoo
Qemu 1.7.0 -> 2.0.0-r1   (Already tried downgrading, It didn't help)

lsgroup output has also changed. I could have sworn first 4 of these were in the same group and I had to bind them all. Now they are separate (Edit: Perhaps it was some other system after all):

### Group 11 ### 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) ### Group 12 ### 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) ### Group 13 ### 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) ### Group 14 ### 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) ### Group 15 ### 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller

Any idea what could introduce such a problem?

UPDATE

After some very thorough testing, I found an explanation but no solution. It may be a BIOS problem. Changing the kernel did nothing.

BIOS SABERTOOTH-990FX-R20-ASUS-2501.CAP (8a19662ee80ff8b75122260baf3de1b760eeb1e6)
BIOS SABERTOOTH-990FX-R20-ASUS-2301.CAP (a593fe9cef0a5b9c5e6eff3ea3f7c5a49bd7743e)
- Problem as described above

BIOS SABERTOOTH-990FX-R20-ASUS-2104.CAP (66bd1583c73ae28ca003b5fd58edc3b998eaf215)
- When playing the sound samples in Win 7, sometimes they play correctly, sometimes they sound very distorted and sometimes they don't play at all
- No error messages appear on host

For my sanity, I'm just going to assume that the latest bioses are actually working correctly. If it does not work properly it better throw some errors rather than being half-broken. Perhaps it never fully worked -- I can't be absolutely sure anymore.

Last edited by ahl (2014-06-20 16:44:58)

Offline

#2167 2014-06-20 08:23:49

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Understood, for some reason the patches in the first post always get the changelog stripped out, the upstream patches do include a proper description:

ACS override
i915 VGA arbiter
VGA arbiter

Thanks for the links. I understand why ACS override was rejected and there is a discussion under i915 arbiter patch (leading nowhere, because the only opponent cannot be bothered to write alternative solution, as he openly admits), but what happened to the last one? There is no discussion following your post, but it is not present in current http://git.kernel.org/cgit/linux/kernel … a/vgaarb.c either.

Does that mean that VGA pass-through users cannot rely on upstream kernel if they want to use (otherwise well-functioning, but poorly supported by kernel) hardware such as Intel IGP or AMD Radeons ? sad Given that nVidia seem to be going out of their way to prevent use of consumer cards in such configuration ("Error 43"), this is very discouraging ...

Last edited by Bronek (2014-06-20 11:49:01)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#2168 2014-06-20 17:22:38

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

can someone pls help ...


I am hitting strange bug with HDMI sound...

It starts to lag on higher cpu usage and stays distorted only with 660 and 770 cards... 560 works fine...
did vm reinstall, driver reinstall, slots change, acs parameters... i got no clue what is wrong...


i7 2600
gigabyte g1 sniper3
debian

thx

dmesg:

 0.551184] ------------[ cut here ]------------ [ 0.551190] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x6b/0x80() [ 0.551191] Modules linked in: [ 0.551193] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.15.0-rc7-awilliam+ #1 [ 0.551194] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./G1.Sniper 3, BIOS F7 08/23/2012 [ 0.551196] 0000000000000009 ffffffff814c0182 0000000000000000 ffffffff8105ccc2 [ 0.551198] ffff88044a0dc098 ffff88044a0dc000 ffff88044a0dc098 0000000000000000 [ 0.551200] 0000000000000000 ffffffff812ac7eb ffffffff813b3496 0009880400000000 [ 0.551202] Call Trace: [ 0.551207] [<ffffffff814c0182>] ? dump_stack+0x41/0x51 [ 0.551210] [<ffffffff8105ccc2>] ? warn_slowpath_common+0x72/0x90 [ 0.551212] [<ffffffff812ac7eb>] ? pci_find_upstream_pcie_bridge+0x6b/0x80 [ 0.551215] [<ffffffff813b3496>] ? intel_iommu_add_device+0x36/0x210 [ 0.551218] [<ffffffff813a8a50>] ? bus_set_iommu+0x50/0x50 [ 0.551220] [<ffffffff813a8a70>] ? add_iommu_group+0x20/0x50 [ 0.551222] [<ffffffff8135e683>] ? bus_for_each_dev+0x53/0x90 [ 0.551224] [<ffffffff813a8a3e>] ? bus_set_iommu+0x3e/0x50 [ 0.551227] [<ffffffff81913bf9>] ? intel_iommu_init+0x53f/0x586 [ 0.551231] [<ffffffff818d743e>] ? memblock_find_dma_reserve+0x131/0x131 [ 0.551233] [<ffffffff818d744c>] ? pci_iommu_init+0xe/0x37 [ 0.551235] [<ffffffff81002152>] ? do_one_initcall+0xf2/0x140 [ 0.551238] [<ffffffff8107aa1a>] ? parse_args+0x1fa/0x330 [ 0.551240] [<ffffffff818cf060>] ? kernel_init_freeable+0x15f/0x1e1 [ 0.551242] [<ffffffff818ce8ac>] ? do_early_param+0x81/0x81 [ 0.551245] [<ffffffff814b24c0>] ? rest_init+0x70/0x70 [ 0.551247] [<ffffffff814b24c5>] ? kernel_init+0x5/0xf0 [ 0.551249] [<ffffffff814cd78c>] ? ret_from_fork+0x7c/0xb0 [ 0.551251] [<ffffffff814b24c0>] ? rest_init+0x70/0x70 [ 0.551254] ---[ end trace bf54b21f328db5e9 ]--- [ 2404.025436] kvm [3767]: vcpu0 unhandled rdmsr: 0x1a2 [ 2404.025505] kvm [3767]: vcpu1 unhandled rdmsr: 0x1a2 [ 2404.025566] kvm [3767]: vcpu2 unhandled rdmsr: 0x1a2 [ 2404.025623] kvm [3767]: vcpu3 unhandled rdmsr: 0x1a2

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.0-rc7-awilliam+ root=UUID=dee8e431-f41a-4ba9-89d0-dc053e70ab5e ro elevator=deadline quiet intel_iommu=on pcie_acs_override=id:8086:0101 transparent_hugepage=always
lspci -nn 00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0100] (rev 09) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) 00:02.0 VGA compatible controller [0300]: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller [8086:0102] (rev 09) 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04) 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 [8086:1e3a] (rev 04) 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 04) 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04) 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4) 00:1c.1 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 [8086:1e12] (rev c4) 00:1c.5 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev c4) 00:1c.6 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 [8086:1e1c] (rev c4) 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4) 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04) 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04) 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04) 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller [8086:1e22] (rev 04) 01:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba) 02:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba) 02:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba) 02:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba) 02:11.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba) 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 03:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1) 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 660] [10de:11c0] (rev a1) 05:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1) 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) 06:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1) 08:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) 09:00.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 41) 0c:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)

qemu

 <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <vendor>Intel</vendor> <topology sockets='1' cores='4' threads='1'/> <feature policy='force' name='rtm'/> <feature policy='force' name='invpcid'/> <feature policy='force' name='lahf_lm'/> <feature policy='force' name='erms'/> <feature policy='force' name='bmi2'/> <feature policy='force' name='smep'/> <feature policy='force' name='avx2'/> <feature policy='force' name='hle'/> <feature policy='force' name='bmi1'/> <feature policy='force' name='fsgsbase'/> <feature policy='force' name='movbe'/> <feature policy='force' name='fma'/> </cpu> <clock offset='utc'> <timer name='hypervclock' present='yes'/> </clock> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.1,bus=root.1,addr=00.1,multifunction=on'/> <qemu:env name='DISPLAY' value=':0'/> </qemu:commandline>

i tried to remove hvtime and other cpu options but it starts to lag sometimes after 1min sometimes after 10mins...

others cards on DVI without sound are working normal at same time...

i tried almost everything i could find... but ("alien",lagging) hdmi sounds are here allways.. sad

i think its something with that nvidia pci bridge PLX, but i don't know what to do...

Last edited by slis (2014-06-20 18:16:59)

Offline

#2169 2014-06-20 20:47:54

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 7390363648: Operation not permitted

Through virt-manager.
I don't know how to debug this, but here is my specs :
uname -r :

3.15.1-custom

lspci -k :

... 01:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8500 GT] (rev a1) Subsystem: XFX Pine Group Inc. Device 230b 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870] Subsystem: PC Partner Limited / Sapphire Technology Device 0851 Kernel driver in use: pci-stub 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink Kernel driver in use: pci-stub ... 

I'm trying to pass the AMD card "02:00.0".

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30

dmesg | grep pci-stub

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30 [ 4.172224] pci-stub: add 1002:9440 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 4.172229] pci-stub 0000:02:00.0: claimed by stub [ 4.172230] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 4.172233] pci-stub 0000:02:00.1: claimed by stub [ 25.757900] pci-stub 0000:02:00.0: claimed by stub [ 46.332286] pci-stub 0000:02:00.0: claimed by stub [ 46.332539] pci-stub 0000:02:00.1: claimed by stub [ 792.054112] pci-stub 0000:02:00.0: claimed by stub [ 792.054287] pci-stub 0000:02:00.1: claimed by stub [ 794.473400] pci-stub 0000:02:00.0: claimed by stub [ 794.473583] pci-stub 0000:02:00.1: claimed by stub

I did apply the following patches :  acs override patch + i915 vga arbiter fixes, I'm using Ubuntu 14.01 LTS.

One more Question, what it is meant by "Systemd service" ? I can't figure out what the equivalent in Debian based distros.

Last edited by abdullah (2014-06-20 20:50:37)

Offline

#2170 2014-06-20 21:26:40

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 7390363648: Operation not permitted

Through virt-manager.
I don't know how to debug this, but here is my specs :
uname -r :

3.15.1-custom

lspci -k :

... 01:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8500 GT] (rev a1) Subsystem: XFX Pine Group Inc. Device 230b 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870] Subsystem: PC Partner Limited / Sapphire Technology Device 0851 Kernel driver in use: pci-stub 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink Kernel driver in use: pci-stub ... 

I'm trying to pass the AMD card "02:00.0".

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30

dmesg | grep pci-stub

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30 [ 4.172224] pci-stub: add 1002:9440 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 4.172229] pci-stub 0000:02:00.0: claimed by stub [ 4.172230] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 4.172233] pci-stub 0000:02:00.1: claimed by stub [ 25.757900] pci-stub 0000:02:00.0: claimed by stub [ 46.332286] pci-stub 0000:02:00.0: claimed by stub [ 46.332539] pci-stub 0000:02:00.1: claimed by stub [ 792.054112] pci-stub 0000:02:00.0: claimed by stub [ 792.054287] pci-stub 0000:02:00.1: claimed by stub [ 794.473400] pci-stub 0000:02:00.0: claimed by stub [ 794.473583] pci-stub 0000:02:00.1: claimed by stub

I did apply the following patches :  acs override patch + i915 vga arbiter fixes, I'm using Ubuntu 14.01 LTS.

One more Question, what it is meant by "Systemd service" ? I can't figure out what the equivalent in Debian based distros.

I don't use an Intel CPU/GPU on my setup, but since you don't use the kernel parameter, please read this post.

The equivalent to a systemd service is an upstart service or an initscript.

And since you're using libvirt, where is you vm's xml file?

Oh, and what do you use the nvidia card for? And which driver do you use it with?

Are the HD4xxx series even supported with this setup?


i'm sorry for my poor english wirting skills…

Offline

#2171 2014-06-20 22:28:28

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
abdullah wrote:

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 7390363648: Operation not permitted

Through virt-manager.
I don't know how to debug this, but here is my specs :
uname -r :

3.15.1-custom

lspci -k :

... 01:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8500 GT] (rev a1) Subsystem: XFX Pine Group Inc. Device 230b 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870] Subsystem: PC Partner Limited / Sapphire Technology Device 0851 Kernel driver in use: pci-stub 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink Kernel driver in use: pci-stub ... 

I'm trying to pass the AMD card "02:00.0".

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30

dmesg | grep pci-stub

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30 [ 4.172224] pci-stub: add 1002:9440 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 4.172229] pci-stub 0000:02:00.0: claimed by stub [ 4.172230] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 4.172233] pci-stub 0000:02:00.1: claimed by stub [ 25.757900] pci-stub 0000:02:00.0: claimed by stub [ 46.332286] pci-stub 0000:02:00.0: claimed by stub [ 46.332539] pci-stub 0000:02:00.1: claimed by stub [ 792.054112] pci-stub 0000:02:00.0: claimed by stub [ 792.054287] pci-stub 0000:02:00.1: claimed by stub [ 794.473400] pci-stub 0000:02:00.0: claimed by stub [ 794.473583] pci-stub 0000:02:00.1: claimed by stub

I did apply the following patches :  acs override patch + i915 vga arbiter fixes, I'm using Ubuntu 14.01 LTS.

One more Question, what it is meant by "Systemd service" ? I can't figure out what the equivalent in Debian based distros.

I don't use an Intel CPU/GPU on my setup, but since you don't use the kernel parameter, please read this post.

The equivalent to a systemd service is an upstart service or an initscript.

And since you're using libvirt, where is you vm's xml file?

Oh, and what do you use the nvidia card for? And which driver do you use it with?

Are the HD4xxx series even supported with this setup?

The nvidia card is used as the host card (nouva), until I changed it right before posting this, so, no the parameter got nothing to do with this, I'm trying to pass the secondary card.
is this how you debug the VM's ? where I even can find the VM's xml?

I will pass the nvidia card right after I successfully pass the ATI card, or I may get a new one.
I don't know what kind of support you are referring to, but it is working right ?

Offline

#2172 2014-06-21 01:01:16

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I'm having the same issue, also using virt-manager. Here's the xml file from one of my VMs.

<domain type='kvm'> <name>Head1</name> <uuid>2f16058f-e2e3-a4a5-5b74-4a5e32623bf8</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>8</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='pbe'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='smx'/> <feature policy='require' name='ss'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='ht'/> <feature policy='require' name='dca'/> <feature policy='require' name='pcid'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> <feature policy='require' name='monitor'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/Win8Steam.qcow2'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='network'> <mac address='52:54:00:2e:35:17'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <video> <model type='vga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </memballoon> </devices> </domain>

You should find the xml files in /etc/libvirt/qemu/ abdullah.

Last edited by MCMjolnir (2014-06-21 01:30:00)

Offline

#2173 2014-06-21 08:38:03

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 7390363648: Operation not permitted

...

wild guess:

[root@sinny ~]# grep root /etc/libvirt/qemu.conf user = "root" group = "root" [root@sinny ~]# 

i think i've seen this somewhere already, but can't remember where

Offline

#2174 2014-06-21 08:40:05

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:

Hi all.

I'm new here, but I've been reading the thread for a couple of months since I got the urge to build a server for streaming Steam from a workstation that recently came into my possession. Finally got all the parts together, and didn't have much luck with Arch since I'm not really familiar with it, so I'm now running Ubuntu Server 14.04.

Here's my config:
CPUs: 2x Xeon E5-2680
Memory: 64 GB ECC RAM @ 1600MHz
Mobo: Asus Z9PA-D8
GPU (Guest 1): MSI GTX770 4GB
GPU (Guest 2): Asus GTX560Ti 2GB

It's been going fairly well so far. Besides figuring out how to make an upstart equivalent of the systemd service, I haven't had any major issues setting up, to this point. I've been using virtual machine manager from my laptop, and configured a guest image I'm happy with. However, when I try to start the VM with the graphics card added, I get this error:

Error starting domain: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 9663676416: Operation not permitted Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup self._backend.create() File "/usr/lib/python2.7/dist-packages/libvirt.py", line 866, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: Process exited prior to exec: libvirt: error : cannot limit locked memory to 9663676416: Operation not permitted

Any ideas on what I'm doing wrong?

This looks like an apparmor error, please read here: http://wiki.apparmor.net/index.php/Libvirt.

Basically, you can disable apparmor security for the Windows guest, by editing the the file /etc/apparmor.d/libvirt/libvirt-<uuid>.files and setting security profile to none or something like this. Probably need to restart the libvirt service too.

Now using virt-manager, and going on the General tab of the Windows machine, you can check that the security profile is no active anymore. By the way you find the "uuid" in virt-manager too.

Offline

#2175 2014-06-21 11:48:05

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, I tried setting security_driver = "none" in /etc/libvirt/qemu.conf as per the link but it didn't seem to do anything. So instead I ran

sudo /etc/init.d/apparmor stop sudo update-rc.d -f apparmor remove

rebooted the host, and now the guest boots! (although this may be a bit heavy handed).

However, I'm now facing an issue in the guest where the card is showing up in device manager but it's giving me a code 43. I see this issue has been resolved already, but I'm not sure how I implement it (I'm still very much a fledgling linux user).

Offline

#2176 2014-06-21 15:45:23

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

refering to problem on post https://bbs.archlinux.org/viewtopic.php … 2#p1428122


so i have sandy bridge i72600 = pcie2.0
560ti = pcie2.0
660ti = pcie3.0
770ti = pcie3.0
mbo= g1 sniper 3=pcie3.0

could this be reason for lagging hdmi sound and i think lower perfomance on kepler+ cards? 560ti works fine...

Offline

#2177 2014-06-21 17:22:34

spicy
Member
Registered: 2014-06-09
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
spicy wrote:

    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga

This means that either a) your kernel does not support CONFIG_VFIO_PCI_VGA or b) the device is not a VGA device.  To test a):

$ grep CONFIG_VFIO_PCI_VGA /boot/config-`uname -r`

Yep, I had to re-compile with CONFIG_VFIO_PCI_VGA. At the time I think I was using a stock kernel which needed some adjustments. Thanks to your guidance the guides provided by the OP, I was able to get everything running with a GTX 770 – working great.

I'm actually doing two builds with similar hardware (minus the video cards, anyway) and ran into a problem recently with my second box and a capture card. From what I've read (and it's even been brought up on the last couple of pages) this is the whole ACS override issue – the capture card looks like it is behind a bridge so I'm getting the error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=pcie.0: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

I haven't been able to test with the ACS patch and

pcie_acs_override=downstream

on because I can't apply the patch to the kernel for some reason – not sure how to troubleshoot why it won't apply properly. I'm using 3.15 from here and getting the following:

$ patch -p1 < acs.diff patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2574 with fuzz 2 (offset 225 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

and then the output of the rejects file:

# cat drivers/pci/quirks.c.rej --- drivers/pci/quirks.c +++ drivers/pci/quirks.c @@ -3292,11 +3292,113 @@ return pci_dev_get(dev); } +static bool acs_on_downstream; +static bool acs_on_multifunction; + +#define NUM_ACS_IDS 16 +struct acs_on_id { + unsigned short vendor; + unsigned short device; +}; +static struct acs_on_id acs_on_ids[NUM_ACS_IDS]; +static u8 max_acs_id; + +static __init int pcie_acs_override_setup(char *p) +{ + if (!p) + return -EINVAL; + + while (*p) { + if (!strncmp(p, "downstream", 10)) + acs_on_downstream = true; + if (!strncmp(p, "multifunction", 13)) + acs_on_multifunction = true; + if (!strncmp(p, "id:", 3)) { + char opt[5]; + int ret; + long val; + + if (max_acs_id >= NUM_ACS_IDS - 1) { + pr_warn("Out of PCIe ACS override slots (%d)\n", + NUM_ACS_IDS); + goto next; + } + + p += 3; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].vendor = val; + + p += strcspn(p, ":"); + if (*p != ':') { + pr_warn("PCIe ACS invalid ID\n"); + goto next; + } + + p++; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].device = val; + max_acs_id++; + } +next: + p += strcspn(p, ","); + if (*p == ',') + p++; + } + + if (acs_on_downstream || acs_on_multifunction || max_acs_id) + pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n"); + + return 0; +} +early_param("pcie_acs_override", pcie_acs_override_setup); + +static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags) +{ + int i; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Never override ACS for legacy devices or devices with ACS caps */ + if (!pci_is_pcie(dev) || + pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS)) + return -ENOTTY; + + for (i = 0; i < max_acs_id; i++) + if (acs_on_ids[i].vendor == dev->vendor && + acs_on_ids[i].device == dev->device) + return 1; + + switch (pci_pcie_type(dev)) { + case PCI_EXP_TYPE_DOWNSTREAM: + case PCI_EXP_TYPE_ROOT_PORT: + if (acs_on_downstream) + return 1; + break; + case PCI_EXP_TYPE_ENDPOINT: + case PCI_EXP_TYPE_UPSTREAM: + case PCI_EXP_TYPE_LEG_END: + case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) + return 1; + } + + return -ENOTTY; +} + static const struct pci_dev_acs_enabled { u16 vendor; u16 device; int (*acs_enabled)(struct pci_dev *dev, u16 acs_flags); } pci_dev_acs_enabled[] = { + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } };

I'm not super familiar with applying kernel patches, but was able to get the i915 VGA patch applied fine earlier on this same kernel version. Also not a C programmer so I'm kind of useless troubleshooting what didn't apply.

Any tips would be appreciated, it is probably something simple that I am missing here (hope I'm not being too frustrating). Also thanks again aw and everyone else who has contributed information, code and guides for this – awesome stuff! I'm hoping to put together a guide of all the specific steps I had to take for my hardware as a sort of specific mini-guide once everything is working to contribute something back in a small way.

Offline

#2178 2014-06-21 19:03:16

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

spicy wrote:
aw wrote:
spicy wrote:

    qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga

This means that either a) your kernel does not support CONFIG_VFIO_PCI_VGA or b) the device is not a VGA device.  To test a):

$ grep CONFIG_VFIO_PCI_VGA /boot/config-`uname -r`

Yep, I had to re-compile with CONFIG_VFIO_PCI_VGA. At the time I think I was using a stock kernel which needed some adjustments. Thanks to your guidance the guides provided by the OP, I was able to get everything running with a GTX 770 – working great.

I'm actually doing two builds with similar hardware (minus the video cards, anyway) and ran into a problem recently with my second box and a capture card. From what I've read (and it's even been brought up on the last couple of pages) this is the whole ACS override issue – the capture card looks like it is behind a bridge so I'm getting the error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=pcie.0: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

I haven't been able to test with the ACS patch and

pcie_acs_override=downstream

on because I can't apply the patch to the kernel for some reason – not sure how to troubleshoot why it won't apply properly. I'm using 3.15 from here and getting the following:

$ patch -p1 < acs.diff patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2574 with fuzz 2 (offset 225 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

and then the output of the rejects file:

# cat drivers/pci/quirks.c.rej --- drivers/pci/quirks.c +++ drivers/pci/quirks.c @@ -3292,11 +3292,113 @@ return pci_dev_get(dev); } +static bool acs_on_downstream; +static bool acs_on_multifunction; + +#define NUM_ACS_IDS 16 +struct acs_on_id { + unsigned short vendor; + unsigned short device; +}; +static struct acs_on_id acs_on_ids[NUM_ACS_IDS]; +static u8 max_acs_id; + +static __init int pcie_acs_override_setup(char *p) +{ + if (!p) + return -EINVAL; + + while (*p) { + if (!strncmp(p, "downstream", 10)) + acs_on_downstream = true; + if (!strncmp(p, "multifunction", 13)) + acs_on_multifunction = true; + if (!strncmp(p, "id:", 3)) { + char opt[5]; + int ret; + long val; + + if (max_acs_id >= NUM_ACS_IDS - 1) { + pr_warn("Out of PCIe ACS override slots (%d)\n", + NUM_ACS_IDS); + goto next; + } + + p += 3; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].vendor = val; + + p += strcspn(p, ":"); + if (*p != ':') { + pr_warn("PCIe ACS invalid ID\n"); + goto next; + } + + p++; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].device = val; + max_acs_id++; + } +next: + p += strcspn(p, ","); + if (*p == ',') + p++; + } + + if (acs_on_downstream || acs_on_multifunction || max_acs_id) + pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n"); + + return 0; +} +early_param("pcie_acs_override", pcie_acs_override_setup); + +static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags) +{ + int i; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Never override ACS for legacy devices or devices with ACS caps */ + if (!pci_is_pcie(dev) || + pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS)) + return -ENOTTY; + + for (i = 0; i < max_acs_id; i++) + if (acs_on_ids[i].vendor == dev->vendor && + acs_on_ids[i].device == dev->device) + return 1; + + switch (pci_pcie_type(dev)) { + case PCI_EXP_TYPE_DOWNSTREAM: + case PCI_EXP_TYPE_ROOT_PORT: + if (acs_on_downstream) + return 1; + break; + case PCI_EXP_TYPE_ENDPOINT: + case PCI_EXP_TYPE_UPSTREAM: + case PCI_EXP_TYPE_LEG_END: + case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) + return 1; + } + + return -ENOTTY; +} + static const struct pci_dev_acs_enabled { u16 vendor; u16 device; int (*acs_enabled)(struct pci_dev *dev, u16 acs_flags); } pci_dev_acs_enabled[] = { + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } };

I'm not super familiar with applying kernel patches, but was able to get the i915 VGA patch applied fine earlier on this same kernel version. Also not a C programmer so I'm kind of useless troubleshooting what didn't apply.

Any tips would be appreciated, it is probably something simple that I am missing here (hope I'm not being too frustrating). Also thanks again aw and everyone else who has contributed information, code and guides for this – awesome stuff! I'm hoping to put together a guide of all the specific steps I had to take for my hardware as a sort of specific mini-guide once everything is working to contribute something back in a small way.

seems like you try to apply "broken" patch:

Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

(note last line)
you probably decided to ignore some "unneded" empty lines, which are in fact important for patch command to verify that applied patch is well-formed/valid

Offline

#2179 2014-06-21 19:35:28

spicy
Member
Registered: 2014-06-09
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

seems like you try to apply "broken" patch:

Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

(note last line)
you probably decided to ignore some "unneded" empty lines, which are in fact important for patch command to verify that applied patch is well-formed/valid

Hi sinny,

Thanks for your reply. As far as I can tell, the patch is the correct one – I've downloaded it directly from Alex's LKML posting using wget to directly download the diff:

wget -O acs.diff https://lkml.org/lkml/diff/2013/5/30/513/1

I should note that the last line about "unexpectedly ending in the middle of line" also occurs in the i915 patch which I can apply successfully, as far as I can tell that is expected behaviour.

Any ideas?

Offline

#2180 2014-06-21 19:59:46

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

spicy wrote:
sinny wrote:

seems like you try to apply "broken" patch:

Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

(note last line)
you probably decided to ignore some "unneded" empty lines, which are in fact important for patch command to verify that applied patch is well-formed/valid

Hi sinny,

Thanks for your reply. As far as I can tell, the patch is the correct one – I've downloaded it directly from Alex's LKML posting using wget to directly download the diff:

wget -O acs.diff https://lkml.org/lkml/diff/2013/5/30/513/1

I should note that the last line about "unexpectedly ending in the middle of line" also occurs in the i915 patch which I can apply successfully, as far as I can tell that is expected behaviour.

Any ideas?

That patch is very old. Download the linux-mainline.tar.gz from first post.

$ tar xf linux-3.15.tar.xz $ tar xf linux-mainline.tar.gz $ cd linux-3.15 $ cat ../linux-mainline/override_for_missing_acs_capabilities.patch | patch -p1

Also since 3.15.1 is already out, you might want to upgrade to that too. Here is one example how to do that:

$ wget https://www.kernel.org/pub/linux/kernel/v3.x/patch-3.15.1.xz $ cd linux-3.15 $ xzcat ../patch-3.15.1.xz | patch -p1 $ cd .. $ mv linux-3.15 linux-3.15.1

...alternatively, you can just download linux-3.15.1.tar.xz

Offline

#2181 2014-06-21 20:42:07

stefan_vgapass
Member
Registered: 2014-03-17
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have finally had some time to try, again, a Linux guest with kvm vga passthrough.
Last time I tried with an Kubuntu 13.10 x64 guest I got the BSOD after installing the ATI drivers.
To my surprise this time with Kubuntu 14.04 x64 it just worked.

Host specs:
check my previous post, the host system is the same.
https://bbs.archlinux.org/viewtopic.php … 4#p1393614

Guest specs:

OS:      Kubuntu 14.04 x64
CPU:     2 cores x 2 threads Opteron emulated CPU
RAM:     8 GB of ram ( I kept the ram the same size as for the windows guest )
GPU:     XFX Radeon HD 7850 2GB RAM
Disk:    virtio ( running 2 HDD: 60GB for / and 128GB for /home )
Network: virtio bridge
Driver:  latest fglrx ATI driver, at the time of writing, Catalyst 14.4

Guest startup script:

/opt/qemu-git/bin/qemu-system-x86_64 -enable-kvm -M q35 \ -m 8196 -mem-prealloc -mem-path /mnt/hugepages \ -cpu Opteron_G5 \ -smp 4,sockets=1,cores=2,threads=2 \ -bios /root/kvm-vga-passthrough/vfio-post3.13/seabios/seabios/out/bios.bin -vga none \ -boot order=cd,menu=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1d.0,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=root.1,addr=00.2 \ -device ahci,bus=pcie.0,id=ahci \ -netdev bridge,br=br0,id=hostnet0 \ -device virtio-net-pci,netdev=hostnet0,id=net0 \ -drive file=/var/lib/libvirt/images/kubuntu_1404_x64-disk01.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/var/lib/libvirt/images/kubuntu_1404_x64-disk02.qcow2,id=disk02,format=qcow2 -device ide-hd,bus=ahci.1,drive=disk02 \ -drive file=/var/lib/libvirt/images/kubuntu-14.04-desktop-amd64.iso,id=isocd -device ide-cd,bus=ahci.2,drive=isocd

Tested:

I've only been able to test using Team Fortress 2 and Dota 2.

In Dota 2 I've measured the FPS running between max 60 fps and min 29 fps usually averaging 45 fps.
Compared with the windows VM were I get max 60 fps and min 30 averaging 58 fps.

In Team fortress 2 I've measured the FPS running between max 120 and min 76 averaging 65 FPS.
Compared with the windows VM were I get max 110 and min 19 averaging 44 FPS.

Looks like TF2 works better in Linux and Dota 2 works better in Windows on a VGA passthrough VM with an ATI graphics card.
The difference is not enourmeous though !
I don't know how they compare versus a native windows install as I have no windows install to compare to.


Steam client on linux related bugs:
For posterity I'm putting here the 2 bugs I've experienced whilst running Steam client on linux with an ATI graphics card.

1. I had to reinstall the ATI driver
I had one very strange occurance that I could only fix by reinstalling the ATI Catalyst driver.
I've played on Steam on this guest a few days ago, but when I tried it again today Steam couldn't detect that the Catalyst driver is running OpenGL 4.4.
It was complaining that it is running OpenGL 1.4 and it can't work with OpenGL < 2.0.
The exact error was:

PROBLEM: You appear to have OpenGL 1.4.0, but we need at least 2.0.0! Could not find required OpenGL entry point 'glGetError'! Either your video card is unsupported, or your OpenGL driver needs to be updated.

2. Steam doesn't correctly detect that I'm runing fglrx.

I had to forcibly tell the Steam client that I'm using fglrx as the video driver.

sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1.2.0 /usr/lib/i386-linux-gnu/mesa/libGL.so.1 sudo ln -s /usr/lib/fglrx/fglrx-libGL.so.1.2 /usr/lib/i386-linux-gnu/mesa/libGL.so.1.2.0

The link that led me to this fix, it is the last post on this steam thread: http://steamcommunity.com/app/221410/di … =german#p2



Next to try SteamOS with vga passthrough, although I expect it to be just as straightforward and easy to setup as this linux guest was !

Offline

#2182 2014-06-22 01:08:49

thehighhat
Member
Registered: 2014-05-12
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Status update to my setup.  After lots of trial and error and tweaking, the minimum amount of steps that works 100% with virsh is:

- motherboard VGA for host (Intel 5520 chipset)
- nvidia gpu for vga passthrough to guest
- ubuntu 14.04 x64_64
- qemu kvm

-- add "intel_iommu=on" to GRUB_CMDLINE_LINUX in /etc/grub/default 
-- create file /etc/modprobe.d/vfio_iommu_type.conf with contents "options vfio_iommu_type1 allow_unsafe_interrupts=1"
-- add "blacklist nouveau" line to /etc/modprobe.d/blacklist-framebuffer.conf
-- set user and group to "root" in /etc/libvirt/qemu.conf
-- add the correct /dev/vfio/#### entry to cgroup_device_acl in /etc/libvirt/qemu.conf
-- use the OP vfio-bind script with the id found from "lscpi -v | grep -i vga"
-- create a VM (virt-manager is quick and dirty way to do this)
-- download or extract the romfile for your GPU
-- then virsh edit YOUR_VM
-- -- set the first line <domain ...> to "<domain type='kvm' id='3' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>"
-- -- add a block similar to this to the xml
"   <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pci.0,multifunction=on,x-vga=on,romfile=/path/to/your/gpu.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.1,bus=pci.0'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/>
  </qemu:commandline>"
-- virt-manager to delete the display and video entries
-- if using apparmor, turn off aa-enforce on the /etc/apparmor.d/libvirt/libvirt-#####-YOUR_VMs_ENTRY

and that's it.  start the VM using virsh start or virt-manager

Offline

#2183 2014-06-22 01:57:01

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

refering to problem on post https://bbs.archlinux.org/viewtopic.php … 2#p1428122


so i have sandy bridge i72600 = pcie2.0
560ti = pcie2.0
660ti = pcie3.0
770ti = pcie3.0
mbo= g1 sniper 3=pcie3.0

could this be reason for lagging hdmi sound and i think lower perfomance on kepler+ cards? 560ti works fine...

Try changing the slot of the same GPU you're passing to the guest. If the problem stops, you know you're hitting a bottleneck with the PCIE->PCH->DMI bus.

Offline

#2184 2014-06-22 07:53:39

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try changing the slot of the same GPU you're passing to the guest. If the problem stops, you know you're hitting a bottleneck with the PCIE->PCH->DMI bus.

Can u pls explain what do u mean by changing slots?  bottleneck? cause of pcie2.0? i checked lspci and it says  for geforce 660 LnkSta: Speed 2.5GT/s, Width x16,

yeah i tested... pcie 3.0 gpus crashes with hdmi sound on gigbyte g1 sniper 3 in any port,, i belive pcie gen3 is problem... but there is no option in bios... i have option on my x77-ud3h so ill try later to lower pcie to 2.0.

Last edited by slis (2014-06-22 12:11:13)

Offline

#2185 2014-06-22 08:12:45

ebinmaymay
Member
Registered: 2014-06-22
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would like to post my own success story too. A video from youtube brought me here and motivated me to move away from vmware.

I have an 8-core AMD 8320 cpu with a ASUS M5A97 EVO R2.0 mobo. It has a AMD970 chipset and supports both VT-i and VT-d. I'm using (k)ubuntu with the latest lowlatency kernel from kernel-ppa. (I should just switch to Arch.)

I'm using a nVidia GTX 550 Ti as my primary card and my Radeon HD 7750 as my passed-through card. (The open source nvidia driver has gotten rather good to my surprise after uninstalling the proprietary driver. smile)
I had some success going the other way as well, but ran into the usual nvidia driver detecting hypervisor problem. (I read somewhere that you can pass a qemu option to hide kvm's presence to the guest, but I don't feel like swapping them around yet again just to test it. smile)

I'm going to give a blow-by-blow description of what I did, so feel free to skip most of it.



===BEGIN BLOW BY BLOW===
First of all I did an lspci and inspection of /sys/devices/ and discovered that my second card appears at 07:00:00 and 07:00:01, with its bridge at 00:15:00.

tux@oniichan:~$ lspci 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] tux@oniichan:~$ lspci -n 00:15.0 0604: 1002:43a0 07:00.0 0300: 1002:683f 07:00.1 0403: 1002:aab0 tux@oniichan:~$ find /sys/devices -iname '0000:07:00.0' /sys/devices/pci0000:00/0000:00:15.0/0000:07:00.

Then I added pci-stub to the modules inside the initramfs blob at /etc/initramfs-tools/modules.

sudo nano /etc/initramfs-tools/modules sudo update-initramfs -u -k all

Then added "pci-stub.ids=1002:683f,1002:aab0" to the kernel command by adding it to GRUB_CMDLINE_LINUX in /etc/default/grub.

sudo nano /etc/default/grub sudo update-grub

It's worth noting that I ended up not blacklisting the kernel's baked-in radeon driver, nor did I turn on unsafe interrupts.


At this stage I rebooted to check that it was catching the card. It was.

[ 16.083836] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 16.083866] pci-stub 0000:07:00.0: claimed by stub [ 16.083878] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 16.083893] pci-stub 0000:07:00.1: claimed by stub

I added the vfio-script as provided by the OP and made an upstart script at /etc/init/vfio-pci

# vfio-pci - bind certain pci devices to vfio-pci description "pci to vfio autobinder" start on runlevel [2345] task script vfio-bind 0000:00:15.0 0000:07:00.0 0000:07:00.1 end script

Rebooted, and then ran qemu as per everyone else ITT.
===END BLOW BY BLOW==



HOWEVER, I was not able to pass through my graphics card with a copy-paste of the OP's instructions.
So I gave up, installed windows 7 using sdl, first with the old default computer model, then with the virtio drivers, and then with the q35 drivers.
For the benefit of the newbies, I passed an extra qcow2 image through a virtio adapter. win7 picked it up and asked for the virtio driver then.
Thankfully I have been running a ftp server using vsftpd for sharing anime in my household, so it wasn't troublesome to get driver/software isos into the os and mount them.
fyi, I used SlySoft's Virtual CloneDrive instead of Daemon Tools because it doesn't inject shellcode to do so, although feel free to use whatever you like. smile

After getting everything else sorted, I finally tackled the pcie passthrough problem and only two days ago, after a week of fiddling, did I get it to work with my AMD card. Whenever I tried to pass it through, the card would turn on and do nothing.
I even extracted the romfile from my card via sysfs...

sudo bash echo '1' > /sys/devices/pci0000:00/0000:00:15.0/0000:07:00.0/romfile dd if=/sys/devices/pci0000:00/0000:00:15.0/0000:07:00.0/romfile of=radeon.rom

...but it still didn't do anything.

After a week of fiddling, I got OP's test command to work by passing a dummy ISA(!!!) cirrus graphics adapter.
When this isa card was passed through, the cirrus graphics adapter ALWAYS displays blank. The boot screen and splash appears on my second monitor.
I suspect that it boots the cirrus and it spoofs a boot of the pci, because my custom bootsplash of which I am immensely proud has distorted colors. sad

I also got the emulated audio device to work flawlessly by dropping the audio adapter on my gfx adapter, then passing it the ich9 version rather than the default ich6. Remember to reboot win7 at least once between hw changes. :-)
So my final working script as of this morning is the following:

#!/bin/bash export DISPLAY=:0 export QEMU_AUDIO_DRV=alsa xrandr --output DVI-I-1 --off sudo qemu-system-x86_64 \ -enable-kvm \ -cpu host,hv-time \ -smp 4,cores=4 \ -m 4G \ -M q35 \ -ctrl-grab \ -vga none \ -bios /usr/share/qemu/bios-256k.bin \ -net user,smb=/home/tux/share \ -net nic,model=virtio \ -usb \ -usbdevice host:8564:1000 \ -usbdevice host:045e:00d1 \ -device isa-cirrus-vga \ -device ioh3420,bus=pcie.0,addr=1c,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device ich9-intel-hda \ -device hda-micro \ -drive if=virtio,file=/home/tux/vms/kvm/win7.img,media=disk \ -drive if=ide,file=/home/ani/Packages/iso/win7-sp0-x64.iso,media=cdrom \ -drive if=ide,file=/home/tux/vms/kvm/amdgfx.iso,media=cdrom \ -boot menu=on,splash=/home/tux/vms/kvm/boot480.jpg,splash-time=5000 \ -name win7 xrandr --output HDMI-1 --auto --primary --output DVI-I-1 --auto --left-of HDMI-1

Those two usb devices are a usb thumbdrive and the mouse from my HTPC which I use whenever something goes wrong.

guest system specs

Last edited by ebinmaymay (2014-06-22 09:23:40)

Offline

#2186 2014-06-22 21:30:53

mdm
Member
Registered: 2014-06-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have been working through getting this setup on my laptop.  I am at the point now where I just get a black screen when booting up my virtual machine.  Here are my laptop specs:

Sager np8250
CPU: i7-4800MQ w/ HD 4600
MB Chipset: HM87
GPU: GTX 780M using Optimus

Before I get into more troubleshooting, is it possible to even use the GTX 780M with the pass through since it is on a laptop with Optimus?

Offline

#2187 2014-06-23 05:12:29

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mdm wrote:

I have been working through getting this setup on my laptop.  I am at the point now where I just get a black screen when booting up my virtual machine.  Here are my laptop specs:

Sager np8250
CPU: i7-4800MQ w/ HD 4600
MB Chipset: HM87
GPU: GTX 780M using Optimus

Before I get into more troubleshooting, is it possible to even use the GTX 780M with the pass through since it is on a laptop with Optimus?

I think it is impossible, because NVIDIA GPU /w Optimus doesn't have any independent layout.
All video layout from GTX780M connected to HD4600.

You can't get any output unless GTX780M got independent output.

Or maybe you can try the previous post that someone mount his Tesla GPU into VM.

Offline

#2188 2014-06-23 10:06:22

rakataka
Member
Registered: 2014-06-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have been running Windows 8 64bit with an Nvidia GTX 770 successfully for some time now. In the qemu command parameter for the VGA card (-device) I do *not* specify the romfile sub-parameter as suggested in the 1st post and it works seemingly fine. Does anyone know why the romfile parameter is suggested or what it accomplishes?

My qemu command is:

qemu-system-x86_64 -name Windows-8.1-Pro -enable-kvm -machine q35,accel=kvm -m 8192 -cpu Haswell,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,hv_time -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -balloon none -serial none -parallel none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device virtio-scsi-pci,id=scsi -drive file=/mnt/windows-8.1.img,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -boot menu=on -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -vga none -net none --monitor stdio -device qxl

Offline

#2189 2014-06-23 15:47:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rakataka wrote:

Hi, I have been running Windows 8 64bit with an Nvidia GTX 770 successfully for some time now. In the qemu command parameter for the VGA card (-device) I do *not* specify the romfile sub-parameter as suggested in the 1st post and it works seemingly fine. Does anyone know why the romfile parameter is suggested or what it accomplishes?

The romfile parameter is only necessary if you cannot reliably read the ROM from the device.  This may also be necessary in cases where the VGABIOS is not provided via a PCI ROM on the host, such as might be the case on a laptop.  The effect of using romfile is simply that we read the ROM from the file instead of the device.  Since the original instructions were written we've refactored a lot of the VFIO ROM code and added bus resets, so ROM access should be much more reliable than it once was.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2190 2014-06-23 17:14:33

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mdm wrote:

I have been working through getting this setup on my laptop.  I am at the point now where I just get a black screen when booting up my virtual machine.  Here are my laptop specs:

Sager np8250
CPU: i7-4800MQ w/ HD 4600
MB Chipset: HM87
GPU: GTX 780M using Optimus

Before I get into more troubleshooting, is it possible to even use the GTX 780M with the pass through since it is on a laptop with Optimus?

Install Windows (8 preferrably) using additional emulated VGA like currus or vmware. Then install nvidia drivers and check if it will be working, or if you get error 43.
If you got error 43, check if disabling and re-enablig nvidia card helps.

Offline

#2191 2014-06-24 10:03:28

ttouch
Member
From: /dev/null
Registered: 2012-05-27
Posts: 129
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

After kernel update (linux-mainline provided), I recompiled nvidia-beta-al and started the VM and I get this: `device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:02:00.0 BAR 0 mmap unsupported. Performance may be slow` and no output. Any help please?
BTW: I had to also attach 02:00.1 to avoid an iommu_group error.

Thank you.

Offline

#2192 2014-06-24 11:23:59

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm stuck on black screen from the passed through VGA card. So here's what I got, and where I'm stuck:

Spec:

  • ASRock z97 Extreme 6

  • Intel i7 4970 /w HD4600 (main gpu)

  • Gainwind Phantom Nvidia GTX680 (secondary gpu)

I followed the 1st post and installed the mainline kernel from the package linked there:

3.15.1-1-mainline

Packages come from pacman's repos, so standard qemu, seabios and the likes.

My Nvidia card:

% lspci |grep -i nvidia :( 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
% lspci -n |grep 01:00 01:00.0 0300: 10de:1180 (rev a1) 01:00.1 0403: 10de:0e0a (rev a1)

Current boot command:

BOOT_IMAGE=/rootvol/boot/vmlinuz-linux-mainline root=UUID=341a7e57-91cb-480c-be5f-2eb405ffd658 rw rootflags=subvol=rootvol intel_iommu=on pci-stub.ids=10de:1180,10de:0e0a quiet

Modprobes:

% cat /etc/modprobe.d/blacklist.conf /etc/modprobe.d/kvm.conf /etc/modprobe.d/vfio_iommu_type.conf blacklist nouveau options kvm ignore_msrs=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Configs for scripts from 1st post (I run vfio-bind.sh manually before I start VMs):

% cat /etc/vfio-pci.cfg DEVICES="0000:01:00.0 000:01:00.1"
% cat vfio-bind.sh #!/bin/sh vfio-bind 0000:01:00.0 0000:01:00.1

Qemu VM:

% cat StartWindowsVm1.sh #!/bin/sh qemu-system-x86_64 -enable-kvm -M q35 -m 10240 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \ -boot menu=on

Dmesg output when starting VM:

[Jun24 12:54] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ +0.000132] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ +4.085762] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000040] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000008] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000010] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +2.106266] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

In the Qemu window not much appear, some sort of qemu menu shows up if I activate the window and press a key.
On the external monitor nothing happens at all (connected through DVI -> VGA, but also tried via HDMI with same "non-result").
If I active KVM integrated gpu and boot windows the GTX680 card shows up in device manager with error 12, where it says the device needs more ressources.

I'm not sure where to tweak to get the ball rolling, any ideas?

Offline

#2193 2014-06-24 12:28:52

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

penetal wrote:

I'm stuck on black screen from the passed through VGA card. So here's what I got, and where I'm stuck:

Spec:

  • ASRock z97 Extreme 6

  • Intel i7 4970 /w HD4600 (main gpu)

  • Gainwind Phantom Nvidia GTX680 (secondary gpu)

I followed the 1st post and installed the mainline kernel from the package linked there:

3.15.1-1-mainline

Packages come from pacman's repos, so standard qemu, seabios and the likes.

My Nvidia card:

% lspci |grep -i nvidia :( 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
% lspci -n |grep 01:00 01:00.0 0300: 10de:1180 (rev a1) 01:00.1 0403: 10de:0e0a (rev a1)

Current boot command:

BOOT_IMAGE=/rootvol/boot/vmlinuz-linux-mainline root=UUID=341a7e57-91cb-480c-be5f-2eb405ffd658 rw rootflags=subvol=rootvol intel_iommu=on pci-stub.ids=10de:1180,10de:0e0a quiet

Modprobes:

% cat /etc/modprobe.d/blacklist.conf /etc/modprobe.d/kvm.conf /etc/modprobe.d/vfio_iommu_type.conf blacklist nouveau options kvm ignore_msrs=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Configs for scripts from 1st post (I run vfio-bind.sh manually before I start VMs):

% cat /etc/vfio-pci.cfg DEVICES="0000:01:00.0 000:01:00.1"
% cat vfio-bind.sh #!/bin/sh vfio-bind 0000:01:00.0 0000:01:00.1

Qemu VM:

% cat StartWindowsVm1.sh #!/bin/sh qemu-system-x86_64 -enable-kvm -M q35 -m 10240 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \ -boot menu=on

Dmesg output when starting VM:

[Jun24 12:54] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ +0.000132] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ +4.085762] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000040] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000008] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000010] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +2.106266] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

In the Qemu window not much appear, some sort of qemu menu shows up if I activate the window and press a key.
On the external monitor nothing happens at all (connected through DVI -> VGA, but also tried via HDMI with same "non-result").
If I active KVM integrated gpu and boot windows the GTX680 card shows up in device manager with error 12, where it says the device needs more ressources.

I'm not sure where to tweak to get the ball rolling, any ideas?

You need to boot with i915.enable_hd_vgaarb=1 as a kernel parameter

Offline

#2194 2014-06-24 14:01:28

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
penetal wrote:

I'm stuck on black screen from the passed through VGA card. So here's what I got, and where I'm stuck:

Spec:

  • ASRock z97 Extreme 6

  • Intel i7 4970 /w HD4600 (main gpu)


...


Dmesg output when starting VM:

[Jun24 12:54] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ +0.000132] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ +4.085762] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000040] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000008] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +0.000010] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ +2.106266] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

In the Qemu window not much appear, some sort of qemu menu shows up if I activate the window and press a key.
On the external monitor nothing happens at all (connected through DVI -> VGA, but also tried via HDMI with same "non-result").
If I active KVM integrated gpu and boot windows the GTX680 card shows up in device manager with error 12, where it says the device needs more ressources.

I'm not sure where to tweak to get the ball rolling, any ideas?

You need to boot with i915.enable_hd_vgaarb=1 as a kernel parameter

Thanks, that got me right up and running.
I assume you/someone has already said this some other place in the thread but honnestly it's getting a bit too big to find everything.
Might be usefull in the OP's 1st post if its necessary for every Intel install.

Offline

#2195 2014-06-24 14:59:37

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems I'm stuck again.
Now at least I do get output from the card to the screen, so its halfways working.
The problem is windows won't "acitvely" use the card,
it has the yellow triangle of fuckery in device manager and report error code 43.

Searching seems to give little insight, has anyone at all been able to solve this problem?

Offline

#2196 2014-06-24 15:49:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

penetal wrote:

Seems I'm stuck again.
Now at least I do get output from the card to the screen, so its halfways working.
The problem is windows won't "acitvely" use the card,
it has the yellow triangle of fuckery in device manager and report error code 43.

Searching seems to give little insight, has anyone at all been able to solve this problem?

I believe there are some problems with the lastest nvidia drivers, there's a discussion about this a few pages back

Last edited by nbhs (2014-06-24 15:50:48)

Offline

#2197 2014-06-24 19:56:34

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:

If you got error 43, check if disabling and re-enabling nvidia card helps.

I tried that (I used this guide to automate it), and while it does get rid of the error code and the drivers show up correctly in device manager, for all other intents and purposes the card is still unusable. It doesn't detect the monitor I have plugged into it and  I have to reinstall the drivers on every VM boot through GeForce Experience before I can do anything within it (even though Device Manager says the drivers are installed already), and after that it produces some weird results, with some fields    ⃠ ed out.

Screenshot of GeForce Experience + Device Manager

I'm using Xeons (so no iGPU) with the host running headless with a GTX 770 assigned to a VM. I'm not sure if that means I need to use any of the patches.

Last edited by MCMjolnir (2014-06-24 19:58:57)

Offline

#2198 2014-06-24 19:59:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:
belliash wrote:

If you got error 43, check if disabling and re-enabling nvidia card helps.

I tried that (I used this guide to automate it), and while it does get rid of the error code and the drivers show up correctly in device manager, for all other intents and purposes the card is still unusable. It doesn't detect the monitor I have plugged into it and  I have to reinstall the drivers on every VM boot through GeForce Experience before I can do anything within it (even though Device Manager says the drivers are installed already), and after that it produces some weird results, with some fields    ⃠ ed out.
I'm using Xeons (so no iGPU) with the host running headless with a GTX 770 assigned to a VM. I'm not sure if that means I need to use any of the patches.

Use qemu.git and add kvm=off to your -cpu options or downgrade the guest driver to 335.23.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2199 2014-06-24 20:06:04

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well as a small update. I got it working, but I have no idea why.

I was about to try to downgrade the nvidia driver, but before I started the VM I did two small changes that I can remember:

  • I passed through my keyboard aswell so I had both keyboard and mouse in the VM while its running.

  • And I switched input on my screen form DisplayPort (main intel gpu) to HDMI (nvidia) instead of just using PBP (Picture Beside Picture: both inputs at the same time with alot of wasted screen space going to black bars all done by the monitor), this shouldnt make any difference but I cant beleive that adding keyboard makes any difference either.

So I dont know why the error in Device Manager just disapeared, but I'm not sad it did.

Anyway I downgraded to an older driver and installed the graphics driver and physix engine (no 3D or audio drivers from nvidia pack was installed), and after that I ran 3Dmark. Got a lower score than native windows, so next step is gonna be to make it all more user friendly + optimizing for performance.

IT WORKS! WOHO!

Offline

#2200 2014-06-24 20:33:34

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:
belliash wrote:

If you got error 43, check if disabling and re-enabling nvidia card helps.

I tried that (I used this guide to automate it), and while it does get rid of the error code and the drivers show up correctly in device manager, for all other intents and purposes the card is still unusable. It doesn't detect the monitor I have plugged into it and  I have to reinstall the drivers on every VM boot through GeForce Experience before I can do anything within it (even though Device Manager says the drivers are installed already), and after that it produces some weird results, with some fields    ⃠ ed out.

Screenshot of GeForce Experience + Device Manager

I'm using Xeons (so no iGPU) with the host running headless with a GTX 770 assigned to a VM. I'm not sure if that means I need to use any of the patches.

Thats very interesting, as you have exactly the same problem, except I encounter this on notebook, while you got "normal PC".
Thats pretty interesting, why others managed to get GTX 770 working - at least from desktop family. If we find a solution here, it might be step forward in getting mobile cards fully working too.


I'm also looking forward to hearing that anyone tried to pass through mobile radeon, as AMD cards seems to be less problematic than manufactured by nvidia.
Another idea is to try XEN instead of KVM. Especially as XEN supports mediated GPU. This means that laptop owners could pass through both nvidia/amd card as well as intel igp (4th gen cpu only).
I'm not sure how this behaves, but if Windows and forceware detects Intel HD Card, nvidia software should be able to copy video buffer to igp, therefore we would get optimus working in VM.
Shame it needs IGP and nvidia doesnt seem to let its technology work with other GPUs. Shame, as we could get optimus working with emulated cirrus/vmware cards.

Offline

#2201 2014-06-24 22:01:19

mdm
Member
Registered: 2014-06-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does the nosnoop patch still need to be applied to qemu.git?  I have check and my Nvidia card does have NoSnoop+.  I applied this patch http://lists.nongnu.org/archive/html/qe … 00597.html, but now get the follow errors:

[ 8055.445808] vfio-pci 0000:01:00.0: enabling device (0006 -> 0007) [ 8056.454042] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 8056.713926] qemu-system-x86[7317]: segfault at 4 ip 00007f27870719b0 sp 00007fff40d8ca38 error 4 in libc-2.19.so[7f2786f3c000+1a4000]

Qemu:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 \ -cpu host,hv-time,kvm=off -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga vmware \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

Without the patch, I am able to get into Windows 8 but receive error 43.

Last edited by mdm (2014-06-24 22:08:23)

Offline

#2202 2014-06-24 23:14:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mdm wrote:

Does the nosnoop patch still need to be applied to qemu.git?  I have check and my Nvidia card does have NoSnoop+.  I applied this patch http://lists.nongnu.org/archive/html/qe … 00597.html, but now get the follow errors:

[ 8055.445808] vfio-pci 0000:01:00.0: enabling device (0006 -> 0007) [ 8056.454042] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 8056.713926] qemu-system-x86[7317]: segfault at 4 ip 00007f27870719b0 sp 00007fff40d8ca38 error 4 in libc-2.19.so[7f2786f3c000+1a4000]

Qemu:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 \ -cpu host,hv-time,kvm=off -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga vmware \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

Without the patch, I am able to get into Windows 8 but receive error 43.

No, the nosnoop patch is defunct.  It was never going to be a reliable solution.  Since you're already using kvm=off, I don't know what's wrong.  Is anyone running Windows 8 successfully with NVIDIA?  TBH, I've almost exclusively tested Windows 7.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2203 2014-06-24 23:44:54

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mdm wrote:

Does the nosnoop patch still need to be applied to qemu.git?  I have check and my Nvidia card does have NoSnoop+.  I applied this patch http://lists.nongnu.org/archive/html/qe … 00597.html, but now get the follow errors:

[ 8055.445808] vfio-pci 0000:01:00.0: enabling device (0006 -> 0007) [ 8056.454042] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 8056.713926] qemu-system-x86[7317]: segfault at 4 ip 00007f27870719b0 sp 00007fff40d8ca38 error 4 in libc-2.19.so[7f2786f3c000+1a4000]

Qemu:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 \ -cpu host,hv-time,kvm=off -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga vmware \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

Without the patch, I am able to get into Windows 8 but receive error 43.

No, the nosnoop patch is defunct.  It was never going to be a reliable solution.  Since you're already using kvm=off, I don't know what's wrong.  Is anyone running Windows 8 successfully with NVIDIA?  TBH, I've almost exclusively tested Windows 7.

I had my GTS450 running under Windows 8.1, with 335.xx driver tho.

Offline

#2204 2014-06-25 01:04:35

rabcor
Member
Registered: 2013-02-09
Posts: 457

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Enabling intel's DMA remapping in the kernel/cmdline is breaking my nvidia's vdpau on an Intel/Nvidia setup (as I state in this thread) I assume either vdpau or the DMA Remapping code need some patch for this.

Where can I get help with this?

Offline

#2205 2014-06-25 01:08:37

mdm
Member
Registered: 2014-06-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:
mdm wrote:

I have been working through getting this setup on my laptop.  I am at the point now where I just get a black screen when booting up my virtual machine.  Here are my laptop specs:

Sager np8250
CPU: i7-4800MQ w/ HD 4600
MB Chipset: HM87
GPU: GTX 780M using Optimus

Before I get into more troubleshooting, is it possible to even use the GTX 780M with the pass through since it is on a laptop with Optimus?

Install Windows (8 preferrably) using additional emulated VGA like currus or vmware. Then install nvidia drivers and check if it will be working, or if you get error 43.
If you got error 43, check if disabling and re-enablig nvidia card helps.

I was able to get Windows 8.1 to recognize the nvidia card with the latest drivers by disabling and then renabling, but I can't get Windows to use the Nvidia card.  I also cannot get the Nvidia control panel to open.  I may just be stuck since the Optimus Nvidia card doesn't have anything to write out to.  If I use -vga none, a window opens but it is a qemu command prompt.

Grub:

linux/vmlinuz-linux-mainline root=UUID=662044e6-2035-46dc-8300-6e205fc39365 rw pci-stub.ids=10de:119f intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 kvm.ignore_msrs=1 quiet

 
Qemu:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 \ -cpu host,hv-time,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

Offline

#2206 2014-06-25 17:52:00

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mdm wrote:
belliash wrote:
mdm wrote:

I have been working through getting this setup on my laptop.  I am at the point now where I just get a black screen when booting up my virtual machine.  Here are my laptop specs:

Sager np8250
CPU: i7-4800MQ w/ HD 4600
MB Chipset: HM87
GPU: GTX 780M using Optimus

Before I get into more troubleshooting, is it possible to even use the GTX 780M with the pass through since it is on a laptop with Optimus?

Install Windows (8 preferrably) using additional emulated VGA like currus or vmware. Then install nvidia drivers and check if it will be working, or if you get error 43.
If you got error 43, check if disabling and re-enablig nvidia card helps.

I was able to get Windows 8.1 to recognize the nvidia card with the latest drivers by disabling and then renabling, but I can't get Windows to use the Nvidia card.  I also cannot get the Nvidia control panel to open.  I may just be stuck since the Optimus Nvidia card doesn't have anything to write out to.  If I use -vga none, a window opens but it is a qemu command prompt.

Grub:

linux/vmlinuz-linux-mainline root=UUID=662044e6-2035-46dc-8300-6e205fc39365 rw pci-stub.ids=10de:119f intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 kvm.ignore_msrs=1 quiet

 
Qemu:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 \ -cpu host,hv-time,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

No, it opens console because it shows it when there is no emulated card enabled.
You can also disable that console. There is described the same, or nearly the same problem with desktop nvidia GTX 7X0, so i think its not related to notebook and/or optimus. Optimus is a software solution.
I think our cards can have EFI instead of BIOS or sth... I'm not sure about that. But this problem was albo observed on desktop version of nvidia cards. Thats why Im waiting 'till anyone will try to pass through mobile radeon...

Offline

#2207 2014-06-26 13:30:20

FruitieX
Member
Registered: 2008-07-03
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey, just wanted to post a short success story and thank anyone who is involved with making VGA passthrough possible.

I carefully picked my PC parts to support VGA passthrough, and since building the PC about 4 months ago I've yet to find a game that can't run well on maxed out settings inside a freaking virtual machine! I don't even have a bare metal Windows install to compare performance with because I never felt the need for that. This is a really amazing accomplishment and in my opinion one of the most amazing things I've been able to do on a computer! :-)

So again thank you and best of luck for anyone trying to set up VGA passthrough!

Fancy scripts for automatic USB passthrough once a Windows guest has finished booting if anyone is interested: https://github.com/FruitieX/vm-scripts/ … er/scripts

Offline

#2208 2014-06-26 14:56:03

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cool,

I hope I will get this working on my machine too sad

Some weeks ago I had an i5 2500K and was a little bit frustrated to hear that this CPU does not support IOMMU. After a little bit of searching around in ebay I got an i5 3470 which supports IOMMU but I have no luck getting it to work.

I set up the vfio-bind and pci-stub stuff but every time I want to launch I get the following:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

the last line makes me wonder a little bit ... why enabling ?

[pheinrich@ARCH tmp]$ dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0140 quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0140 quiet [ 0.691337] pci-stub: add 10DE:0140 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.691352] pci-stub 0000:02:00.0: claimed by stub [ 1.872554] pci-stub 0000:02:00.0: enabling device (0000 -> 0002)
[pheinrich@ARCH tmp]$ dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.114583] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.114654] IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0 [ 0.619324] IOMMU 0 0xfed90000: using Queued invalidation [ 0.619326] IOMMU: Setting RMRR: [ 0.619335] IOMMU: Setting identity map for device 0000:00:1a.0 [0xcf0d5000 - 0xcf0e3fff] [ 0.619354] IOMMU: Setting identity map for device 0000:00:1d.0 [0xcf0d5000 - 0xcf0e3fff] [ 0.619362] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.619368] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] 
[pheinrich@ARCH tmp]$ ls /dev/vfio/ 1 vfio 

I want to passthrough the geforce 6600GT

[pheinrich@ARCH tmp]$ lspci | grep VGA 01:00.0 VGA compatible controller: NVIDIA Corporation GT200 [GeForce GTX 260] (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation NV43 [GeForce 6600 GT] (rev a2) 

Offline

#2209 2014-06-26 15:15:25

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

I want to passthrough the geforce 6600GT

[pheinrich@ARCH tmp]$ lspci | grep VGA 01:00.0 VGA compatible controller: NVIDIA Corporation GT200 [GeForce GTX 260] (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation NV43 [GeForce 6600 GT] (rev a2) 

Okay, you can't do that. The 6600 GT (or any pre 7xxx card AFAWK) is not capable.
(I should know, I too wasted a week trying to get it work)
Try exchanging them, the 260 will work better for passthrough.

Offline

#2210 2014-06-26 15:37:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

Cool,

I hope I will get this working on my machine too sad

Some weeks ago I had an i5 2500K and was a little bit frustrated to hear that this CPU does not support IOMMU. After a little bit of searching around in ebay I got an i5 3470 which supports IOMMU but I have no luck getting it to work.

I set up the vfio-bind and pci-stub stuff but every time I want to launch I get the following:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

You don't need vfio_iommu_type1.allow_unsafe_interrupts=1, but that's not the problem here.

Look at:

ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/

All of the endpoints listed there need to be attached to either vfio-pci or pci-stub.  If you can't make that happen, then you need to apply and enable the ACS override patch.  And then as aldum wrote, a 6600GT is pretty old, if it doesn't work already I don't think there's any point in adding code to trying to support it.  If you want a cheap Nvidia card, go grab a GT 635 off ebay.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2211 2014-06-26 18:53:38

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok i found this:

[pheinrich@ARCH ~]$ ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ total 0 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0

but the third line is my GTX  0000:01:00.0 or am I wrong?

whould be an 7950GT better than my old 6600GT ?

Trying to passthrough the GTX 260 is a problem because the current nvidia driver does not support the GT 6600 anymore so I have to install the old nvidia driver to support the GT 6600 as host gpu

Offline

#2212 2014-06-26 19:03:44

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you dont need any drivers on host. only on guest

Offline

#2213 2014-06-26 19:05:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

ok i found this:

[pheinrich@ARCH ~]$ ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ total 0 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Jun 26 20:48 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0

but the third line is my GTX  0000:01:00.0 or am I wrong?

That's correct, Intel consumer processors don't support ACS on the PEG root ports and Intel has not been able to confirm that isolation is provided between root ports.  You'll need the ACS override patch if you want the kernel to accept them as isolated.

whould be an 7950GT better than my old 6600GT ?

A slightly newer ancient card?  Probably not.  I've tested a 7300LE and it doesn't work very well.  My 8400GS works ok though, so 8-series might be the oldest I'd bother trying.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2214 2014-06-26 20:06:24

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok thank you for your answer.

now I took the 6600GT as my primary host card and blocked the GTX 260 via pci-stub to use it with qemu.

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

same result as above ... so now I will try the ACS patch smile

maybe my ASUS P8P67 evo is the faulty part sad

Offline

#2215 2014-06-26 20:13:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

ok thank you for your answer.

now I took the 6600GT as my primary host card and blocked the GTX 260 via pci-stub to use it with qemu.

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

same result as above ... so now I will try the ACS patch smile

maybe my ASUS P8P67 evo is the faulty part sad

Why would you expect it to be any different?  Swapping the two cards between the same two slots won't change the group.  Your motherboard has 3 x16 slots, installing one graphics card into the bottom black slot should put it into a separate IOMMU group if you're running a v3.15 kernel.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2216 2014-06-26 20:17:59

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes but i wasn't able to get anything on my screen from my 6600GT plugged into the second slot.

Then I plugged it into the first slot and blocked the GTX 260 plugged into the second slot.
I thought it would work trying to passthrough the newer card smile

Now I am trying to patch a 3.15.1 kernel with the ACS patch provided in the first post.

Offline

#2217 2014-06-27 00:41:40

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, I managed to passthrough the 8500GT Nvidia card, yet got error 43 or 42, can't remember, I will try to fix it later ( how to pass kvm off in virt-manager?)

Now, with the AMD HD 4760 card ( 4xxx HD series ), I managed to pass it through, but after installing the drivers and rebooting windows, it just would hang at the start screen (logo).
The HD 4 series card is very old, yet it got some power, so I'm asking, is the problem from the card or the driver? does this happen often with AMD cards ?
I got some other entry series AMD card (6 series) that I'm looking to test it just to see the results ( and share it with you ), and then, I would buy a newer card, I'm looking at Asus r9 280x OC, so what's your recommendation about it?

The other thing I'm looking to do, is to create 2 PC's back2back, each one with it's own passedthrough keyboard and mouse, the problem I'm facing is I could not manage to get any VGA output from the cards to the second screen ?.
Thanks.

Offline

#2218 2014-06-27 01:37:21

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone figured out how to get audio to work with libvirt? I've got everything except audio working right now. I'm thinking about using a different program to stream the audio from the guest to my host, but I wasn't sure how others were handling this.

Offline

#2219 2014-06-27 10:35:23

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Has anyone figured out how to get audio to work with libvirt? I've got everything except audio working right now. I'm thinking about using a different program to stream the audio from the guest to my host, but I wasn't sure how others were handling this.

Try global pulseaudio, cheap USB audio card or passthru host's HD audio.

Offline

#2220 2014-06-27 14:03:47

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
Slabity wrote:

Has anyone figured out how to get audio to work with libvirt? I've got everything except audio working right now. I'm thinking about using a different program to stream the audio from the guest to my host, but I wasn't sure how others were handling this.

Try global pulseaudio, cheap USB audio card or passthru host's HD audio.

Or try pulseaudio over network (127.0.0.1). That's what I do.


i'm sorry for my poor english wirting skills…

Offline

#2221 2014-06-27 15:40:50

random_user9547
Member
Registered: 2014-06-27
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have tried VGA-Passthrough on my system but it fails.

My specifications are:
Motherboard: gigabyte ga 970 UD3
CPU: AMD FX 6300
VGA1: AMD HD 7790 on 01:00.0 01:00.01
VGA2: AMD HD 6450 on 06:00.0 06:00.01

The host OS uses HD 6450, I am trying to pass HD 7790 to a guest OS.

Qemu version:
qemu-x86_64 version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

kernel version:
3.14.1-3-mainline #1 SMP PREEMPT (the one nbhs posted)

Added this to grub:
vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:665c,1002:0002

binded the devices.

And got the Seabios output on the screen.

But after attachin something bootable to qemu, like this:
-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

I get:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow
qemu-system-x86_64: vfio_bar_write(,0x0, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x4, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x8, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0xc, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x10, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x14, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x18, 0x0, 4) failed: Device or resource busy
...

and a load of vfio_bar_write fails.

Any idea how to solve this one?

Offline

#2222 2014-06-27 16:48:16

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Or try pulseaudio over network (127.0.0.1). That's what I do.

What's your method of doing this?

Last edited by Slabity (2014-06-27 20:27:21)

Offline

#2223 2014-06-27 17:59:45

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

random_user9547 wrote:

Hello,

I have tried VGA-Passthrough on my system but it fails.

My specifications are:
Motherboard: gigabyte ga 970 UD3
CPU: AMD FX 6300
VGA1: AMD HD 7790 on 01:00.0 01:00.01
VGA2: AMD HD 6450 on 06:00.0 06:00.01

The host OS uses HD 6450, I am trying to pass HD 7790 to a guest OS.

Qemu version:
qemu-x86_64 version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

kernel version:
3.14.1-3-mainline #1 SMP PREEMPT (the one nbhs posted)

Added this to grub:
vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:665c,1002:0002

binded the devices.

And got the Seabios output on the screen.

But after attachin something bootable to qemu, like this:
-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

I get:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow
qemu-system-x86_64: vfio_bar_write(,0x0, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x4, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x8, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0xc, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x10, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x14, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x18, 0x0, 4) failed: Device or resource busy
...

and a load of vfio_bar_write fails.

Any idea how to solve this one?

how about reading the topic first?..

installations are usually performed with qemu-emulated video, passthrough should be enabled on already installed os - this way it's easier to see what is wrong

Offline

#2224 2014-06-27 18:25:55

random_user9547
Member
Registered: 2014-06-27
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

how about reading the topic first?..

installations are usually performed with qemu-emulated video, passthrough should be enabled on already installed os - this way it's easier to see what is wrong

It actually does not matter.
Even if I first install an OS(can be windows or linux) and later try to start it with a dedicated GPU i get the same error.

It seems to happen when qemu goes beyond seabios(starts anything graphical).

Offline

#2225 2014-06-27 18:57:33

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

random_user9547 wrote:

It actually does not matter.
Even if I first install an OS(can be windows or linux) and later try to start it with a dedicated GPU i get the same error.

It seems to happen when qemu goes beyond seabios(starts anything graphical).

Could you post the result of `lspci -v`? The section related to your graphics card.

Last edited by Slabity (2014-06-27 18:58:20)

Offline

#2226 2014-06-27 19:35:00

random_user9547
Member
Registered: 2014-06-27
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Could you post the result of `lspci -v`? The section related to your graphics card.

Yes. Here:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770 / R9 260 OEM] (prog-if 00 [VGA controller]) Subsystem: Hightech Information System Ltd. Radeon HD 7790 TurboDuo Flags: bus master, fast devsel, latency 0, IRQ 10 Memory at a0000000 (64-bit, prefetchable) [size=256M] Memory at b0000000 (64-bit, prefetchable) [size=8M] I/O ports at e000 [size=256] Memory at fea00000 (32-bit, non-prefetchable) [size=256K] Expansion ROM at fea40000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] #13 Capabilities: [2d0] #1b Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002 Subsystem: Hightech Information System Ltd. Device 0002 Flags: bus master, fast devsel, latency 0, IRQ 7 Memory at fea60000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

Offline

#2227 2014-06-27 20:24:42

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

random_user9547 wrote:
Slabity wrote:

Could you post the result of `lspci -v`? The section related to your graphics card.

Yes. Here:

Add these parameters to your kernel's command line:

intel_iommu=on pcie_acs_override=downstream

You may need to replace intel_iommu with the AMD equivelant, which I don't know off the top of my head.

Offline

#2228 2014-06-27 20:46:40

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got it to WORK !!! partly big_smile

I can successfully passthrough my GTX 260 and I see the seabios on the output of that gpu.
Now my problem is how to stop it ... if I cancel the qemu command the seabios does not go down.

Next problem is how to use the host keyboard in the vm? Do I have to plug a second keyboard and passthrough it?

Is it possible to redirect the output of the gpu into a qemu window ?

I configured my existing qemu img like this

-drive file=/srv/media/Data/qemu/win7/win7.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk

But seabios says "Boot failded: could not read the boot disk"

Last edited by _pheinrich_ (2014-06-27 20:47:40)

Offline

#2229 2014-06-27 20:53:49

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

I got it to WORK !!! partly :D

I can successfully passthrough my GTX 260 and I see the seabios on the output of that gpu.
Now my problem is how to stop it ... if I cancel the qemu command the seabios does not go down.

Next problem is how to use the host keyboard in the vm? Do I have to plug a second keyboard and passthrough it?

Is it possible to redirect the output of the gpu into a qemu window ?

I configured my existing qemu img like this

-drive file=/srv/media/Data/qemu/win7/win7.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk

But seabios says "Boot failded: could not read the boot disk"

some of your questions were discussed in this topic already

please bother reading up a bit or using search - value the time and effort of other people.

Offline

#2230 2014-06-27 21:10:34

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you!

every time I use the search function like "qemu passthrough keyboard" or something else I get this post ... the first page.
After reading 15 pages I haven't found some useful hints.

Offline

#2231 2014-06-27 21:35:13

random_user9547
Member
Registered: 2014-06-27
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

Add these parameters to your kernel's command line:

intel_iommu=on pcie_acs_override=downstream

You may need to replace intel_iommu with the AMD equivelant, which I don't know off the top of my head.

It turns out the amd equivelant is

iommu=pt iommu=1

Adding it did not help.

With

iommu=pt iommu=1

and/or

iommu=pt iommu=1 pcie_acs_override=downstream

dmesg shows that AMD-Vi is enabled, which would imply that iommu is activated.

with

vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt iommu=1 pcie_acs_override=downstream

dmesg does not show AMD-Vi activated.

None of the lines helped, I still get the same error.

The qemu line I am running is:

gksu "qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -drive file=/home/<user>/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1"

Am I maybe invoking the qemu line incorrectly?

Offline

#2232 2014-06-27 21:39:25

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
andy123 wrote:

Or try pulseaudio over network (127.0.0.1). That's what I do.

What's your method of doing this?

I enabled network "input" for my pulseaudio like this in /etc/pulse/default.pa

load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1 auth-anonymous=1

Setting these environment variables should to the job for qemu.

export QEMU_AUDIO_DRV=pa export PULSE_SERVER=127.0.0.1 export QEMU_PA_SERVER=127.0.0.1

I don't know if you can do input (aka microphones) this way  but I've used this for output the last few months.


i'm sorry for my poor english wirting skills…

Offline

#2233 2014-06-28 15:42:29

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

While connecting a second screen to the passed through card, there would no be output ?
I think it is strange, or I did something wrong.

Offline

#2234 2014-06-28 16:45:57

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
Slabity wrote:
andy123 wrote:

Or try pulseaudio over network (127.0.0.1). That's what I do.

What's your method of doing this?

I enabled network "input" for my pulseaudio like this in /etc/pulse/default.pa

load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1 auth-anonymous=1

Setting these environment variables should to the job for qemu.

export QEMU_AUDIO_DRV=pa export PULSE_SERVER=127.0.0.1 export QEMU_PA_SERVER=127.0.0.1

I don't know if you can do input (aka microphones) this way  but I've used this for output the last few months.

Beautiful! This even works with libvirt + virt-manager!

Offline

#2235 2014-06-28 19:14:51

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm attempting to follow the instructions from the OP and am having some trouble.

Hardware:
CPU: Intel i7-4790
Motherboard: GA-Z97N-WIFI
Host GPU: Intel HD Graphics 4600
Passthrough GPU: EVGA NVIDIA GTX 550Ti

Relevant package information:
Qemu 2.0.50
SeaBIOS 1.7.5
Linux Kernel 3.15.2

Steps taken to configure so far:

1. Enabled VT-d in BIOS
2. Aded intel_iommu=on in my kernel parameters. It also says I need VT-d working. Is there a way to check if it's working without just trying to implement VGA-passthrough? I checked through dmesg and found that IOMMU is enabled, is that enough?

$ dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.018572] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.018575] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.018640] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.464684] IOMMU 0 0xfed90000: using Queued invalidation [ 0.464685] IOMMU 1 0xfed91000: using Queued invalidation [ 0.464686] IOMMU: Setting RMRR: [ 0.464694] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb000000 - 0xcf1fffff] [ 0.464978] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9e7f000 - 0xc9e8dfff] [ 0.464996] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9e7f000 - 0xc9e8dfff] [ 0.465011] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9e7f000 - 0xc9e8dfff] [ 0.465023] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.465028] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

3. Blacklisted nouveau in /etc/modprobe.d/blacklist.conf
4. Found the 550Ti's PCI information

$ lspci 01:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1)

5. Used the provided vfio-bind bash script to bind the GPU

vfio-bind 0000:01:00.0 0000:01:00.1

6. Tried starting up the BIOS to test passthrough GPU

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 4,cores=4, \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

When I do this a compat_monitor0 console opens on my main display and then qemu crashes with:

KVM internal error. Suberror: 1 emulation failure EAX=000000d9 EBX=00000000 ECX=0000bd53 EDX=000003d4 ESI=00009e49 EDI=0000823b EBP=0000fc6a ESP=0000822e EIP=000a0000 EFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =0000 00000000 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =0000 00000000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6e98 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

I terminated it and just tried starting qemu again and got a different error:

Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

Checking dmesg:

$ dmesg | tail [ 4.873454] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready [ 32.469712] fuse init (API version 7.23) [ 71.008187] VFIO - User Level meta-driver version: 0.3 [ 72.716407] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 76.093483] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 76.093527] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 76.093538] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 76.093550] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 78.735841] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 108.173362] vfio-pci 0000:01:00.0: Invalid ROM contents

I decided to try qemu-git as well and have the same results.

I know my components are very new (at least my CPU and motherboard are) so I can't rule them out of being the problem. I'd really appreciate it if anyone could help me narrow things down.

Offline

#2236 2014-06-28 20:36:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:

I'm attempting to follow the instructions from the OP and am having some trouble.

Hardware:
CPU: Intel i7-4790
Motherboard: GA-Z97N-WIFI
Host GPU: Intel HD Graphics 4600
Passthrough GPU: EVGA NVIDIA GTX 550Ti

Relevant package information:
Qemu 2.0.50
SeaBIOS 1.7.5
Linux Kernel 3.15.2

Steps taken to configure so far:

1. Enabled VT-d in BIOS
2. Aded intel_iommu=on in my kernel parameters. It also says I need VT-d working. Is there a way to check if it's working without just trying to implement VGA-passthrough? I checked through dmesg and found that IOMMU is enabled, is that enough?

$ dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.018572] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.018575] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.018640] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.464684] IOMMU 0 0xfed90000: using Queued invalidation [ 0.464685] IOMMU 1 0xfed91000: using Queued invalidation [ 0.464686] IOMMU: Setting RMRR: [ 0.464694] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb000000 - 0xcf1fffff] [ 0.464978] IOMMU: Setting identity map for device 0000:00:14.0 [0xc9e7f000 - 0xc9e8dfff] [ 0.464996] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc9e7f000 - 0xc9e8dfff] [ 0.465011] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc9e7f000 - 0xc9e8dfff] [ 0.465023] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.465028] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

3. Blacklisted nouveau in /etc/modprobe.d/blacklist.conf
4. Found the 550Ti's PCI information

$ lspci 01:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1)

5. Used the provided vfio-bind bash script to bind the GPU

vfio-bind 0000:01:00.0 0000:01:00.1

6. Tried starting up the BIOS to test passthrough GPU

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 4,cores=4, \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

When I do this a compat_monitor0 console opens on my main display and then qemu crashes with:

KVM internal error. Suberror: 1 emulation failure EAX=000000d9 EBX=00000000 ECX=0000bd53 EDX=000003d4 ESI=00009e49 EDI=0000823b EBP=0000fc6a ESP=0000822e EIP=000a0000 EFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =0000 00000000 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =0000 00000000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6e98 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

I terminated it and just tried starting qemu again and got a different error:

Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

Checking dmesg:

$ dmesg | tail [ 4.873454] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready [ 32.469712] fuse init (API version 7.23) [ 71.008187] VFIO - User Level meta-driver version: 0.3 [ 72.716407] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 76.093483] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 76.093527] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 76.093538] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 76.093550] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 78.735841] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 108.173362] vfio-pci 0000:01:00.0: Invalid ROM contents

I decided to try qemu-git as well and have the same results.

I know my components are very new (at least my CPU and motherboard are) so I can't rule them out of being the problem. I'd really appreciate it if anyone could help me narrow things down.

You need the i915  vgaarb patch

Offline

#2237 2014-06-28 22:05:29

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You need the i915  vgaarb patch

Thanks for the reply. I compiled a custom kernel with this patch and have the same errors

# ./vfio-bind 0000:01:00.0 0000:01:00.1 # qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ > -smp 4,cores=4, \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 KVM internal error. Suberror: 1 emulation failure EAX=00000002 EBX=00000000 ECX=0000915e EDX=000003d4 ESI=00009e49 EDI=0000823b EBP=0000a50c ESP=00008232 EIP=000a0000 EFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =0000 00000000 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =0000 00000000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6ee8 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ^Cqemu: terminating on signal 2 # qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ > -smp 4,cores=4, \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= ^Cqemu: terminating on signal 2 # dmesg | tail [ 14.426903] fuse init (API version 7.23) [ 17.469829] snd_hda_intel 0000:00:1b.0: IRQ timing workaround is activated for card #0. Suggest a bigger bdl_pos_adj. [ 27.583355] VFIO - User Level meta-driver version: 0.3 [ 34.290376] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 37.692780] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 37.692822] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 37.692832] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 37.692841] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 40.177371] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 54.267555] vfio-pci 0000:01:00.0: Invalid ROM contents

Offline

#2238 2014-06-28 23:21:44

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:
nbhs wrote:

You need the i915  vgaarb patch

Thanks for the reply. I compiled a custom kernel with this patch and have the same errors

# ./vfio-bind 0000:01:00.0 0000:01:00.1 # qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ > -smp 4,cores=4, \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 KVM internal error. Suberror: 1 emulation failure EAX=00000002 EBX=00000000 ECX=0000915e EDX=000003d4 ESI=00009e49 EDI=0000823b EBP=0000a50c ESP=00008232 EIP=000a0000 EFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =0000 00000000 ffffffff 00809b00 SS =0000 00000000 ffffffff 00809300 DS =0000 00000000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 000f6ee8 00000037 IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ^Cqemu: terminating on signal 2 # qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ > -smp 4,cores=4, \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= ^Cqemu: terminating on signal 2 # dmesg | tail [ 14.426903] fuse init (API version 7.23) [ 17.469829] snd_hda_intel 0000:00:1b.0: IRQ timing workaround is activated for card #0. Suggest a bigger bdl_pos_adj. [ 27.583355] VFIO - User Level meta-driver version: 0.3 [ 34.290376] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 37.692780] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 37.692822] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 37.692832] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 37.692841] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 40.177371] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 54.267555] vfio-pci 0000:01:00.0: Invalid ROM contents

This is not the right patch (it doesn't hurt tho).

Offline

#2239 2014-06-28 23:52:02

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

This is not the right patch (it doesn't hurt tho).

Could you link me to the right patch? I tried both VGA patches from the OP with the same errors.

Offline

#2240 2014-06-29 09:44:41

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I havn't found an answer while searching if it is possible to redirect the output from a gpu passed through to a normal qemu window?

Is there an option to pass an ps/2 keyboard to the vm? I have read something about this and thought it would bind automatically but my keyboard stays in the host scope.

If I bind a usb hub with vfio-bind I can use it inside the vm, but if I shutdown the vm I would like to use that usb hub on my host again. I tried this but without luck:

/usr/bin/vfio-unbind

#!/bin/bash for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/remove_id if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/bind fi done 

Offline

#2241 2014-06-29 14:21:27

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:
dwe11er wrote:

This is not the right patch (it doesn't hurt tho).

Could you link me to the right patch? I tried both VGA patches from the OP with the same errors.

Along with patches, you need to add 'i915.enable_hd_vgaarb=1' to kernel params. Also, you prob need to aquire vbios file for your gpu and pass it with romfile arg during the vm boot.

Offline

#2242 2014-06-29 15:42:23

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:
dwe11er wrote:

This is not the right patch (it doesn't hurt tho).

Could you link me to the right patch? I tried both VGA patches from the OP with the same errors.

this is what works for me (vga arbioter patch link inlcuded)

Offline

#2243 2014-06-29 15:51:48

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

Hi,

I havn't found an answer while searching if it is possible to redirect the output from a gpu passed through to a normal qemu window?

can you "redirect" video from one physical computer to another physically separate one? there are certain ways for sure, most involving network. the same logic applies here

_pheinrich_ wrote:

Is there an option to pass an ps/2 keyboard to the vm? I have read something about this and thought it would bind automatically but my keyboard stays in the host scope.

If I bind a usb hub with vfio-bind I can use it inside the vm, but if I shutdown the vm I would like to use that usb hub on my host again. I tried this but without luck:


why not directly passing usb devices to qemu? in my case:

... -usbdevice host:04b4:0101 \ -usbdevice host:1038:1366 \ ...

specified mouse and keyboard get acquired by guest (stop interacting with host os) on its start.
when guest terminates, both keyboard and mouse get released (start interacting with host os again)

and yes, all these questions HAVE been disccussed in the thread. i know, it's pretty hard to look through all 90+ pages, but still

Offline

#2244 2014-06-29 22:47:59

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I dumped virt-manager, now using qemu through the command line, I managed to boot A windows iso, in the second screen, but! I'm facing the "Blue of Death Screen", will do more testing... \
side note: I have 2 identical keyboards, and to pass "not my host keyboard" I would need to unplug it first.

and an easy way to search the thread... use google with the parameter "site:https://bbs.archlinux.org/viewtopic.php?id=162768 "vfio" "kvm"", and add your key word at the end.

Offline

#2245 2014-06-30 04:01:34

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

I dumped virt-manager, now using qemu through the command line, I managed to boot A windows iso, in the second screen, but! I'm facing the "Blue of Death Screen", will do more testing... \
side note: I have 2 identical keyboards, and to pass "not my host keyboard" I would need to unplug it first.

and an easy way to search the thread... use google with the parameter "site:https://bbs.archlinux.org/viewtopic.php?id=162768 "vfio" "kvm"", and add your key word at the end.

<hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <address bus='1' device='8'/>
      </source>
    </hostdev>

you can pass it like this with addess instead of ID.

Offline

#2246 2014-06-30 04:22:26

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah i tested... pcie 3.0 gpus crashes with hdmi sound on gigbyte g1 sniper 3 in any port,, i belive pcie gen3 is problem... but there is no option in bios... i have option on my x77-ud3h so ill try later to lower pcie to 2.0.

Update:

after couple of days testing... i can conclude that HDMI sound on nvidia kepler+ cards (aka pcie 3.0) doesn't not work on h77 chipset and 2xxx processor.... (ill post update with 3770 when it arrives)
depending on settings, sound starts to lag in game, or if u force usage of sound like opening youtube while in game etc... eventualy windows stops responding saying Can't play sound or something like that...

it seems like mbo is changing speed of pcie (i think cause of processor pcie 2.0 vs gpu/mbo 3.0?), and when it changes sound desync and starts to lag.... turning off on output works for couple of seconds or so... (by lag i mean sound works but VERY SLOW, like 10-100 times slower, and eventualy it crashes game/windows)

tried it on 660,680,770 = same
560 = no problems, only difference is pcie2.0 vs 3.0 i guess...

on zx77-ud3h i have pcie port options in bios like, pcie mode= auto, gen1,gen2,gen3 = all starts to lag = different time to start, pcie_aspm=on/off/force

anyway usb sound card (5$) works...

About gaming performance...

I managed to get about 95%+ performace of native (even with 2 guests)... same on windows 7 or 8.1 x64, with any driver... 0 BSOD, 0 CRASHES, 0 PROBLEMS (except HDMI sound)
pcie 3.0 cards performs little slower than 2.0 in my case, i blame cpu.

i7 2600 & 2600s
G1 Sniper 3
ZX-77-UD3H
debian latest.... 3.15.1 + patches, qemu-git

Last edited by slis (2014-06-30 04:59:27)

Offline

#2247 2014-06-30 13:33:01

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

my setup still works fine, but I was just wondering whether it would be possible to somehow access the vms GPUs framebuffer from the host? Or is there some reason why this is impossible?

Offline

#2248 2014-06-30 13:56:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SpacePirate wrote:

Hi,

my setup still works fine, but I was just wondering whether it would be possible to somehow access the vms GPUs framebuffer from the host? Or is there some reason why this is impossible?

Theoretically you can, but there's no code to support it.  Contributions welcome.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2249 2014-06-30 18:52:28

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sry that I have to ask, but I do not know what I have done wrong that pci passthrough is not working anymore.

It was working as I used the vesa driver for my host card [6600 GT]. Now I have installed nvidia-304xx for that card.

If I try to start qemu now I get the following:

[pheinrich@ARCH ~]$ sudo win8 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The GTX 260 should not be covered by the nvidia-304xx and it is claimed by pci-stub too.

[pheinrich@ARCH ~]$ lspci | grep VGA 01:00.0 VGA compatible controller: NVIDIA Corporation NV43 [GeForce 6600 GT] (rev a2) 02:00.0 VGA compatible controller: NVIDIA Corporation GT200 [GeForce GTX 260] (rev a1) [pheinrich@ARCH ~]$ uname -a Linux ARCH 3.15.1-1-acs #1 SMP PREEMPT Fri Jun 27 11:25:56 CEST 2014 x86_64 GNU/Linux [pheinrich@ARCH ~]$ dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pci-stub.ids=10de:05e2 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pci-stub.ids=10de:05e2 [ 0.779345] pci-stub: add 10DE:05E2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.779361] pci-stub 0000:02:00.0: claimed by stub

Last edited by _pheinrich_ (2014-06-30 18:52:40)

Offline

#2250 2014-06-30 18:55:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The ACS patch requires a boot parameter to enable it


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2251 2014-06-30 19:27:15

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am sure that I havn't seen this parameter ever and it was working.

Nevertheless I have added it with no success.

[pheinrich@ARCH ~]$ dmesg | grep BOOT_IMAGE [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2

Offline

#2252 2014-06-30 19:31:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

I am sure that I havn't seen this parameter ever and it was working.

Nevertheless I have added it with no success.

[pheinrich@ARCH ~]$ dmesg | grep BOOT_IMAGE [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2

Perhaps you should report what's in group 1 (ll /sys/kernel/iommu_groups/1/devices/) and what those devices are bound to (lspci -k) so we have a remote shot at helping.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2253 2014-06-30 19:49:16

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes of course

[pheinrich@ARCH bin]$ ls -l -a --color=auto /sys/kernel/iommu_groups/1/devices/ total 0 drwxr-xr-x 2 root root 0 Jun 30 21:47 . drwxr-xr-x 3 root root 0 Jun 30 21:47 .. lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
[pheinrich@ARCH bin]$ lspci -k 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) Subsystem: ASUSTeK Computer Inc. Device 844d 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) Kernel driver in use: pcieport Kernel modules: shpchp 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) Kernel driver in use: pcieport Kernel modules: shpchp 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04) Subsystem: ASUSTeK Computer Inc. P8 series motherboard Kernel driver in use: mei_me Kernel modules: mei_me 00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 05) Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard Kernel driver in use: e1000e Kernel modules: e1000e 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) Subsystem: ASUSTeK Computer Inc. P8 series motherboard Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05) Subsystem: ASUSTeK Computer Inc. Device 8469 Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 2 (rev b5) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5) Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5) 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) Subsystem: ASUSTeK Computer Inc. P8 series motherboard Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge: Intel Corporation P67 Express Chipset Family LPC Controller (rev 05) Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05) Subsystem: ASUSTeK Computer Inc. P8 series motherboard Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05) Subsystem: ASUSTeK Computer Inc. P8 series motherboard Kernel modules: i2c_i801 01:00.0 VGA compatible controller: NVIDIA Corporation NV43 [GeForce 6600 GT] (rev a2) Kernel driver in use: nvidia Kernel modules: nouveau, nvidia 02:00.0 VGA compatible controller: NVIDIA Corporation GT200 [GeForce GTX 260] (rev a1) Subsystem: NVIDIA Corporation Device 0585 Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia 04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01) 07:00.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI Bridge Controller [Ver 1.0] (rev 01) Subsystem: TERRATEC Electronic GmbH Device 1178 Kernel driver in use: Mantis Kernel modules: mantis 07:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet (rev 10) Subsystem: ASUSTeK Computer Inc. Device 820d Kernel driver in use: r8169 Kernel modules: r8169 07:03.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0) Subsystem: ASUSTeK Computer Inc. Motherboard Kernel driver in use: firewire_ohci Kernel modules: firewire_ohci

Offline

#2254 2014-07-01 02:57:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_pheinrich_ wrote:

yes of course

[pheinrich@ARCH bin]$ ls -l -a --color=auto /sys/kernel/iommu_groups/1/devices/ total 0 drwxr-xr-x 2 root root 0 Jun 30 21:47 . drwxr-xr-x 3 root root 0 Jun 30 21:47 .. lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Jun 30 21:47 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0

You certainly need the ACS patch applied and enabled in this configuration, or you could move one of the cards to the other x16 slot on the motherboard as I suggested previously.  Since the ACS override doesn't seem to be working for you, I'd double check that it's actually applied and that you're booting the kernel you think you're booting.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2255 2014-07-01 03:47:02

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, everyone, nice to see you again!
Now I got a lot of time to investigate because summer vacation.

Now I updated kernel to 3.16.0-rc2, but I found that I can't execute CUDA application on VM.
I set CPU function with

-cpu host,hv-time,kvm=off

This is no use.
But if I take kvm=off out, and only edit kvm.c, it worked but still got some problem when execute other CUDA applications.

Maybe I should downgrade the NVIDIA driver version on guest.
But I guess I should use vgaarb patch because my host GPU is AMD.

I know how to patch it, but what command should I type into GRUB?

Many thanks,
AK.

Last edited by AKSN74 (2014-07-01 04:05:09)

Offline

#2256 2014-07-01 07:00:01

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, after doing some tests, maybe I found a reason why guest Ubuntu can't run CUDA very well.

Few weeks ago, after found that NVIDIA limit KVM to use GPU, I can run CUDA with KVM sign changed. But still got some problem.
Run some CUDA samples will get this kernel message.

NVRM: os_pci_init_handle: invalid context!

And the same application can run at first time when boot in. After that, it can't run again.

But when I running a CUDA sample called 'vectorAdd', I found that it get error when application try to copy data from GPU memory to RAM.
So I check both dmesg (host and guest.), and I found that when I execute. Guest OS got a same IRQ number, but host is not a same.

For example, when I execute application first time in guest, the host dmesg is:

vfio-pci 0000:03:00.0: irq 88 for MSI/MSI-X

But when I execute second time or run other application, the host dmesg is:

vfio-pci 0000:03:00.0: irq 89 for MSI/MSI-X

This is my guess, maybe the MSI IRQ address changed cause GPU can't send data back to RAM.
But it just a guess, I'm not sure.
Maybe it can run CUDA very well when guest OS is Windows or Ubuntu Desktop, because they have GUI (X-server), so they need MSI IRQ at beginning and always the same IRQ.
And both my host and guest OS are Ubuntu Server.


I have to do more tests so that can figure out what's a real problem.

Last edited by AKSN74 (2014-07-04 01:19:18)

Offline

#2257 2014-07-01 10:14:22

random_user9547
Member
Registered: 2014-06-27
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

random_user9547 wrote:

Hello,

I have tried VGA-Passthrough on my system but it fails.

My specifications are:
Motherboard: gigabyte ga 970 UD3
CPU: AMD FX 6300
VGA1: AMD HD 7790 on 01:00.0 01:00.01
VGA2: AMD HD 6450 on 06:00.0 06:00.01

The host OS uses HD 6450, I am trying to pass HD 7790 to a guest OS.

Qemu version:
qemu-x86_64 version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

kernel version:
3.14.1-3-mainline #1 SMP PREEMPT (the one nbhs posted)

Added this to grub:
vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:665c,1002:0002

binded the devices.

And got the Seabios output on the screen.

But after attachin something bootable to qemu, like this:
-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

I get:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow
qemu-system-x86_64: vfio_bar_write(,0x0, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x4, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x8, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0xc, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x10, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x14, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x18, 0x0, 4) failed: Device or resource busy
...

and a load of vfio_bar_write fails.

Any idea how to solve this one?

I solved the problem.
It seems that the graphic card on 01:00.0 is the primary on boot, meaning before the kernel loads this is the main card.
Because of this the os sees it is already used, and gives resource busy error.
I solved it by swapping the GPUs, so the other GPU is now on 01:00.0.


But now when I start the virtual machine it is very slow like 1/100 of the speed it should be or ti freezes. QEMU shows no error on the screen.
Any idea how to solve this one?

Offline

#2258 2014-07-01 14:32:06

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried to apply the acs overrice patch mentioned in the first post.

The first hunk (/Documentation/kernel-parameters.txt) is sucessful.
The second (/drivers/pci/quirks.c) not.

I copied the involved file to a tmp location and tried to fix the patched line but without success.

The mentioned header "struct pci_dev *pci_get_dma_source(struct pci_dev *dev)" is not the line 3292 ... it has moved to line 3372.

changing this ...

@@ -3292,11 +3292,113 @@ struct pci_dev *pci_get_dma_source(struct pci_dev *dev) ....
@@ -3372,11 +3372,113 @@ struct pci_dev *pci_get_dma_source(struct pci_dev *dev) ....

had no effect sad

#########
Edit 1

Also -F3 doesn't help

#########
Edit 2

succeeded with -F3 ... (typo)
hopefully it has done its job well smile

Last edited by _pheinrich_ (2014-07-01 14:48:54)

Offline

#2259 2014-07-01 14:33:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

So I check both dmesg (host and guest.), and I found that when I execute. Guest OS got a same IRQ number, but host is not a same.

For example, when I execute application first time in guest, the host dmesg is:

vfio-pci 0000:03:00.0: irq 88 for MSI/MSI-X

But when I execute second time or run other application, the host dmesg is:

vfio-pci 0000:03:00.1: irq 89 for MSI/MSI-X

These IRQs are on different functions, GPU vs audio


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2260 2014-07-01 16:29:27

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok I have compiled the new kernel with the acs patch and enabled it

[ 0.000000] Linux version 3.15.2-1-acs (pheinrich@ARCH) (gcc version 4.9.0 20140604 (prerelease) (GCC) ) #1 SMP PREEMPT Tue Jul 1 17:58:41 CEST 2014 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2

binding my 0000:02:00.0 device creates an iommu_group 14 so I think the acs patch was successful

[pheinrich@ARCH ~]$ ls -l -a --color=auto /sys/kernel/iommu_groups/1/devices/ total 0 drwxr-xr-x 2 root root 0 Jul 1 18:26 . drwxr-xr-x 3 root root 0 Jul 1 18:26 .. lrwxrwxrwx 1 root root 0 Jul 1 18:26 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 [pheinrich@ARCH ~]$ ls -l -a --color=auto /sys/kernel/iommu_groups/14/devices/ total 0 drwxr-xr-x 2 root root 0 Jul 1 18:26 . drwxr-xr-x 3 root root 0 Jul 1 18:26 .. lrwxrwxrwx 1 root root 0 Jul 1 18:26 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0

But now if I start up the vm, the qemu control window is comming up, no output on 0000:02:00.0 and after 5 seconds everything is black also the host.

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/srv/media/Data/qemu/win8/win8.qcow2,id=windisk -device ide-hd,bus=ide.0,drive=windisk \ -drive file=/srv/media/Data/OperatingSystem/en_windows_8_1_x64_dvd_2707217.iso,id=winiso -device ide-cd,bus=ide.1,drive=winiso

###########
Edit 1

ACS seems to be fine

[pheinrich@ARCH ~]$ dmesg | grep ACS [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] ACPI: FACS 0x00000000CF3BEF80 000040 [ 0.300365] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled [ 0.300515] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled [ 0.300666] pci 0000:00:1c.4: Intel PCH root port ACS workaround enabled

I do not get any error. The whole system is freezing, not only X because ssh sessions die too.

Last edited by _pheinrich_ (2014-07-01 21:18:52)

Offline

#2261 2014-07-01 22:15:06

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys, following this guide I got windows 8.1 installed and was able to pass through a nvidia 670 gpu, with a nvidia titan on the host side. The drivers that come with windows 8 work, however when I try to install the latest drivers from nvidia I either get a blank screen on reboot in the guest os, or non-functioning drivers in the guest os. Anyone else experience this?

Last edited by kristopher004 (2014-07-01 22:16:02)

Offline

#2262 2014-07-02 00:24:34

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
kameloc wrote:
dwe11er wrote:

This is not the right patch (it doesn't hurt tho).

Could you link me to the right patch? I tried both VGA patches from the OP with the same errors.

Along with patches, you need to add 'i915.enable_hd_vgaarb=1' to kernel params. Also, you prob need to aquire vbios file for your gpu and pass it with romfile arg during the vm boot.

The kernel parameter is what I was missing, thank you.

sinny wrote:
kameloc wrote:
dwe11er wrote:

This is not the right patch (it doesn't hurt tho).

Could you link me to the right patch? I tried both VGA patches from the OP with the same errors.

this is what works for me (vga arbioter patch link inlcuded)

This post was also very helpful, thank you.

Another success story. To reiterate my setup is:

CPU: Intel i7-4790
Motherboard: GA-Z97N-WIFI
Host GPU: Intel HD Graphics 4600
Passthrough GPU: EVGA NVIDIA GTX 550Ti

Tried installing Windows 8.1 but couldn't get the sound to not sound extremely distorted. Moved to Windows 7 with the ac97 soundcard which runs really nice except it's a little off when playing games or video (like it's slowed down). I'm going to be messing with the audio settings to try to clean it up a bit. Also the video card persists through reboots only with the NVIDIA Drivers installed on Windows 7 (persists being not freezing at the shutdown screen even with the VM powered off).

Also I'm a little confused by the networking section of their documentation. I was using -net user at first but wanted to setup synergy so opted for a bridge. Bridge works fine but was thinking of adding two NICs so I could have one externally facing and the other inside QEMU's virtual networking (primarily for access to other VMs I'll create). I tried just adding two -nic devices but only one would appear, even with setting different VLANs. Anyone have experience adding two NICs? So to the documentation, here they say that -net is obsolete but I can't find an alternative for bridged devices in the documentation. I see this page on bridges but it also uses -net nic.

Offline

#2263 2014-07-02 03:36:38

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kristopher004 wrote:

Hi guys, following this guide I got windows 8.1 installed and was able to pass through a nvidia 670 gpu, with a nvidia titan on the host side. The drivers that come with windows 8 work, however when I try to install the latest drivers from nvidia I either get a blank screen on reboot in the guest os, or non-functioning drivers in the guest os. Anyone else experience this?

You need latest qemu (option kvm=off) with 340 driver...

or u can patch it yourself, with help of this topic smile

Last edited by slis (2014-07-02 03:38:20)

Offline

#2264 2014-07-02 04:27:03

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

You need latest qemu (option kvm=off) with 340 driver...
or u can patch it yourself, with help of this topic

Thanks slis, I have a few questions though. Do you mean the nvidia 340 driver for the host or for the guest or for both? Also when you say patch it do you mean to patch the 340 driver on the host side with the patch mentioned under the issues section of this guide?

Thanks!

Last edited by kristopher004 (2014-07-02 04:27:21)

Offline

#2265 2014-07-02 07:52:15

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you can use qemu-git from AUR or patching qemu.

I would say using qemu-git is the easier option wink

Offline

#2266 2014-07-02 12:06:25

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok i found the issue.

nvidia-304xx module is the problem.
Removing all nvidia specific modules/packages and do not start X server, so both cards aren't really in use (then I do not need the ACS override), the vm is starting.

Maybe I try nouveau for the host now.

Offline

#2267 2014-07-02 13:31:55

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
AKSN74 wrote:

So I check both dmesg (host and guest.), and I found that when I execute. Guest OS got a same IRQ number, but host is not a same.

For example, when I execute application first time in guest, the host dmesg is:

vfio-pci 0000:03:00.0: irq 88 for MSI/MSI-X

But when I execute second time or run other application, the host dmesg is:

vfio-pci 0000:03:00.1: irq 89 for MSI/MSI-X

These IRQs are on different functions, GPU vs audio

I'm sorry that I key the wrong message sad

The second message is not 0000:03:00.1, it's same as first message. Just different IRQ number.

Last edited by AKSN74 (2014-07-02 13:35:20)

Offline

#2268 2014-07-02 20:29:29

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How do I go about adding the kvm=off flag when using virt-manager? I presumed I could add it to the XML file for the guest, but I don't know what format it should take. I don't seem to be having much luck with this:

 <cpu mode='custom' match='exact' kvm=off/>

Last edited by MCMjolnir (2014-07-03 00:17:55)

Offline

#2269 2014-07-02 20:30:35

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nouveau produces a warped window and is not usable. My N43 chip should be supported but it does not work.
Now I am using vesa for playing with qemu passthrough.

As I asked some post before I would like to redirect the gpu output.

First I tried the vnc option. with "-vnc :0" but the mouse pointer is lagging and has an big offset.

So I decided to use a rdp connection.
To connect one option would be to start up a tun tap device so I can access my guest from the host.
I have this option working on my laptop but I do not get it to work on my desktop pc.

However "-redir tcp:5555::3389" should work too.
After I see the SeaBios and windows 8 is going to start everything is freezing, also the host sad
Anyone experienced something like that before?

Third and last option whould be to passthrough my second network card big_smile

Offline

#2270 2014-07-02 21:45:09

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using two nvidia cards, latest qemu and nvidia 340.17 on host the nvidia driver on guest says: "Windows has stopped this device because it has reported problem. (code 43)"  Anyone else have this problem?

Offline

#2271 2014-07-02 21:53:33

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you using -cpu kvm=off in the launch arguments?

Last edited by MCMjolnir (2014-07-02 21:57:36)

Offline

#2272 2014-07-02 23:00:00

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

no I'll give that a try

Last edited by kristopher004 (2014-07-02 23:11:09)

Offline

#2273 2014-07-03 06:15:04

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:

How do I go about adding the kvm=off flag when using virt-manager? I presumed I could add it to the XML file for the guest, but I don't know what format it should take. I don't seem to be having much luck with this:

 <cpu mode='custom' match='exact' kvm=off/>

I belive it's not implemented yet.

Offline

#2274 2014-07-04 01:28:02

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed Windows 7 VM again and run CUDA to see if host got a same message in post #2256.
But after the test, there is no message and CUDA still run perfectly on Windows.

So maybe my theory is right, GPU can't send data back to main memory because IRQ changed when execute CUDA application on Ubuntu VM.
But now I have to test driver because maybe NVIDIA Linux driver still have a problem. (Linux Driver is earlier to limit VM use GRID only than Windows.) Even changed KVM signature.
Maybe I should try the previous driver before NVIDIA released GRID. Or different Linux OS.

Last edited by AKSN74 (2014-07-04 14:01:46)

Offline

#2275 2014-07-04 20:11:51

rafalcieslak
Member
Registered: 2014-07-04
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, thanks for this guide and other useful info in this thread!

I have been mostly successful with my passthrough setup, but I got stuck at a particular problem that I do not know how to trouleshoot.

I have managed to run qemu with what appears to be a working passthrough of my nvidia 770, but it seems the drivers fail to comunicate with the device itself.
If my guest system is a Linux, I can run lspci -v there, and apparently the dedicated graphics card is indeed exposed to the guest system, as lspci lists it. The `lspci -vmm` entry for the guest system is same as for the host.. On a windows, the device is also detected as a VGA display, with detailed information being also correct. So I assume up to this point I did everything right, and that the card got passed through.

However, whenever I try to install the drivers, something gets messed up.
In case of a linux system, the installation succeeds, but the X server fails to start, the logs say that nvidia xorg module complained that "The NVIDIA device on PCI:1:0:0 is not supported by this driver", which is wrong, because I have used the save driver with the very same piece of software with absolutelly no problems (and the driver manual says it does support that card).
In case of a windows, the driver installation suceeds, but the system fails to use the card afterwards, first two reboot are always a blue screen on the boot, and any subsequent will suceed, but windows will never use the card, there is nothing displayed. If the emulated vga is available, windows will use it instead (this is where I saw these blue screens too).

This feels like there is some complication with the drivers communicating with the hardware. I have no clue on how to troubleshoot that.
I will be very thankful for any hints!

PS. I have also just noticed, that I cannot even get seabios to display using the passthrough'd display. BUT the lspci of the guest system lists the device, so it is there! What am I missing? smile

Offline

#2276 2014-07-04 22:02:33

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rafalcieslak wrote:

nvidia xorg module complained that "The NVIDIA device on PCI:1:0:0 is not supported by this driver"

I have no experience with NVIDIA cards but if I understand correctly, it is working exactly as intended by NVIDIA. They desperately want you to buy a NVIDIA Quadro card. You might want the read the last few pages of this thread. Keywords: "qemu-git" and "kvm=off".

...

On the other hand, I'm searching for a new AMD card capable of working perfectly. Any problem that might require rebooting the host is not acceptable.

Any chance that either of these two would meet above criteria?
Asus Radeon R7 260 1GiB (90YV05I0-M0NA00) (UPDATE: NOT GOOD. See below.)
MSI Radeon R9 270 2GiB (R9-270-GAMING-2G)
(Note: Not the "X" versions)

I recently bought a Sapphire HD 5450 512MiB (11166-01-20R), but very surprisingly, it turned out to be trash! It indeed works, but If qemu is terminated without properly shutdowning the guest, the card stops working until next reboot.

So now I'm afraid of buying anything...

For the record, my old Sapphire HD 6670 1GiB (11192-06-20G) works absolutely perfectly. I'm still at 0 host AND 0 guest* crashes after months of use and any game I throw at it Just Works(TM).

(* Actually 2 guest crashes, but it was unrelated.)

Last edited by ahl (2014-07-09 21:29:55)

Offline

#2277 2014-07-05 22:57:30

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

I recently bought a Sapphire HD 5450 512MiB (11166-01-20R), but very surprisingly, it turned out to be trash! It indeed works, but If qemu is terminated without properly shutdowning the guest, the card stops working until next reboot.

I had similar problem with NVIDIA GTS450. Now I've got HD7870 (Pitcarin; same core as R9 270) and it's working flawlessly; unclean shutdowns aren't problems.

Offline

#2278 2014-07-06 15:08:00

heavymetal
Member
Registered: 2014-07-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I've been trying this with my ASUS P8Z77-V LK and Intel Core i5-3550. Both allegedly support VT-d. I have both the IGP bundled with the processor and a Gigabyte NVIDIA GeForce GT 630.

I've installed the kernel proposed in #1 and set the following options in grub.cfg:

intel_iommu=on pci-stub.ids=10de:0f00,10de:0bea i915.enable_hd_vgaarb=1 pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1

I've noticed lots of DMA-related problems when booting this kernel with IOMMU on. The first is this kerneloops shown in dmesg, which was already reported here:

[ 0.375724] IOMMU: Setting RMRR: [ 0.375732] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff] [ 0.376833] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd08e000 - 0xbd09ffff] [ 0.376849] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd08e000 - 0xbd09ffff] [ 0.376861] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd08e000 - 0xbd09ffff] [ 0.376870] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.376875] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.377267] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.377340] ------------[ cut here ]------------ [ 0.377344] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x65/0x90() [ 0.377345] Modules linked in: [ 0.377348] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.15.1-1-mainline #1 [ 0.377349] Hardware name: System manufacturer System Product Name/P8Z77-V LK, BIOS 0908 11/16/2012 [ 0.377350] 0000000000000000 0000000081968345 ffff880224453d10 ffffffff81507d91 [ 0.377351] 0000000000000000 ffff880224453d48 ffffffff81069acd ffff880224217098 [ 0.377353] ffff880224217000 ffff880224217098 0000000000000000 0000000000000000 [ 0.377354] Call Trace: [ 0.377358] [<ffffffff81507d91>] dump_stack+0x4d/0x6f [ 0.377361] [<ffffffff81069acd>] warn_slowpath_common+0x7d/0xa0 [ 0.377362] [<ffffffff81069bfa>] warn_slowpath_null+0x1a/0x20 [ 0.377364] [<ffffffff812cfde5>] pci_find_upstream_pcie_bridge+0x65/0x90 [ 0.377367] [<ffffffff813ea07d>] intel_iommu_add_device+0x4d/0x240 [ 0.377369] [<ffffffff813df370>] ? bus_set_iommu+0x50/0x50 [ 0.377370] [<ffffffff813df39a>] add_iommu_group+0x2a/0x60 [ 0.377373] [<ffffffff8138f633>] bus_for_each_dev+0x73/0xc0 [ 0.377374] [<ffffffff813df368>] bus_set_iommu+0x48/0x50 [ 0.377377] [<ffffffff8193d94b>] intel_iommu_init+0x1bf/0x5d2 [ 0.377379] [<ffffffff81900297>] ? memblock_find_dma_reserve+0x143/0x143 [ 0.377380] [<ffffffff819002a9>] pci_iommu_init+0x12/0x3c [ 0.377383] [<ffffffff81002162>] do_one_initcall+0xf2/0x1b0 [ 0.377385] [<ffffffff8108b47d>] ? parse_args+0x21d/0x490 [ 0.377387] [<ffffffff818f6067>] kernel_init_freeable+0x1a0/0x23c [ 0.377390] [<ffffffff814fccd0>] ? rest_init+0x90/0x90 [ 0.377391] [<ffffffff814fccde>] kernel_init+0xe/0xf0 [ 0.377394] [<ffffffff815159fc>] ret_from_fork+0x7c/0xb0 [ 0.377396] [<ffffffff814fccd0>] ? rest_init+0x90/0x90 [ 0.377399] ---[ end trace 9e3a2ac08a3dd373 ]--- [ 0.378939] RAPL PMU detected, hw unit 2^-16 Joules, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer

Once my system has booted, I have no wifi. My wireless card is a PCI one, sat on 05:01.0 according to lspci. I get these in dmesg every time:

[ 1855.821855] dmar: DMAR:[DMA Write] Request device [05:00.0] fault addr 0 DMAR:[fault reason 02] Present bit in context entry is clear [ 1855.822005] dmar: DRHD: handling fault status reg 3 [ 1855.822012] dmar: DMAR:[DMA Write] Request device [05:00.0] fault addr 0 DMAR:[fault reason 02] Present bit in context entry is clear [ 1855.854514] dmar: DRHD: handling fault status reg 3 [ 1855.854521] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffffa000 DMAR:[fault reason 02] Present bit in context entry is clear [ 1855.886652] ath: phy0: Failed to stop TX DMA, queues=0x001! [ 1855.897505] dmar: DRHD: handling fault status reg 3 [ 1855.897511] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr fffd0000 DMAR:[fault reason 02] Present bit in context entry is clear

The wireless card is the only PCI (not PCI-E) card in my system.

Not being able to reach the Internet through my PCI card, I use a USB one. It works for few minutes, but after a while it stops working, again with DMA problems reported by dmesg.

[ 529.166591] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush [ 529.306393] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush [ 529.725885] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush [ 529.865689] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush [ 529.879235] ieee80211 phy1: rt2x00usb_watchdog_tx_dma: Warning - TX queue 2 DMA timed out, invoke forced forced reset [ 579.875625] ieee80211 phy1: rt2x00usb_vendor_request: Error - Vendor Request 0x07 failed for offset 0x3040 with error -110

Disconnecting the wireless PCI card does not help.

I don't know if all these DMA problems are hardware or software fault.

Even with all of these, I'm able to boot seabios through the nVidia card with QEMU. I use the following commands:

sudo vfio-bind 0000:01:00.0 0000:01:00.1 sudo chgrp $USER /dev/vfio/13 sudo chmod g+rw /dev/vfio/13 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

And it works...! The first time. If I press ^C, the Seabios screen persists on my monitor, and running the above command again fails with "Invalid ROM". If I want to run QEMU again I have to reboot the host system.

Adding rombar=0 only yields a blank screen and an emulation error after a while.

I tried searching for my VGA BIOS file at http://www.techpowerup.com/vgabios/, finding 4 possible ROMs for my video card. Three of them made the screen spark characters here and there. None of them made the card work for a second time without rebooting, and usually made my computer hang indefinitely on the second run, needing to hard reset it.

What could be the cause of these problems, specially the DMA ones? Maybe a bug in the BIOS like commented in #4?

Offline

#2279 2014-07-06 20:57:28

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

I had similar problem with NVIDIA GTS450. Now I've got HD7870 (Pitcarin; same core as R9 270) and it's working flawlessly; unclean shutdowns aren't problems.

Thanks for the info. I hope you are right because the above mentioned R7 260 turned out to be yet another paperweight.

A quick search for PAGE_FAULT_IN_NONPAGED_AREA in this thread revealed that people have already reported this new problem of mine numerous times.

Are all of the new Sea Islands cards bad? (BONAIRE, KABINI, KAVERI, HAWAII, HD7790, R7 260(X), R9 290(X))

This is my new problem:

CPU: AMD Guest GPU: Radeon R7 260 Qemu: git or 2.0.0 Linux: 3.15.4, 3.15.3 or 3.14.10 (all with vga_arbitrer.patch) Steps to reproduce problem, works 100%: 1. Enter high resolution graphic mode in Win 7 (it works initially) 2. Shutdown guest and start qemu again (OR just soft-reboot the guest) 3. Upon entering high resolution mode again, these errors appear: Guest: PAGE_FAULT_IN_NONPAGED_AREA STOP: 0x50 Host: Mostly unresponsive, but still working. Qemu won't die even with kill -9. Unlimited number of these flooding very fast: AMD-Vi: Completion-Wait loop timed out ------------[ cut here ]------------ WARNING: CPU: 6 PID: 3841 at drivers/iommu/amd_iommu.c:1248 __domain_flush_pages+0x1b2/0x1c0() Modules linked in: radeon cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit fbcon bitblit softcursor font backlight drm_kms_helper ttm drm fb fbdev pci_stub bridge stp llc loop jc42 it87 hwmon_vid xt_physdev ebt_log ebtable_filter ebtables snd_usb_audio snd_usbmidi_lib snd_hwdep snd_pcm snd_rawmidi snd_seq_device snd_timer snd uas kvm_amd joydev soundcore evdev kvm wmi k10temp i2c_piix4 fam15h_power crct10dif_pclmul xhci_hcd CPU: 6 PID: 3841 Comm: qemu-system-x86 Not tainted 3.15.3-arbiter #2 Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2501 04/08/2014 0000000000000009 ffff8802f664bba8 ffffffff815b03fb 0000000000000007 0000000000000000 ffff8802f664bbe8 ffffffff810c0322 000000000000136b ffff880310605810 00000000fffffffb 8000000000000ffe 0000000000000000 Call Trace: [<ffffffff815b03fb>] dump_stack+0x46/0x58 [<ffffffff810c0322>] warn_slowpath_common+0x82/0xb0 [<ffffffff810c0365>] warn_slowpath_null+0x15/0x20 [<ffffffff81491e92>] __domain_flush_pages+0x1b2/0x1c0 [<ffffffff81491eba>] domain_flush_tlb_pde+0x1a/0x20 [<ffffffff81492036>] amd_iommu_unmap+0x176/0x190 [<ffffffff81491ec0>] ? domain_flush_tlb_pde+0x20/0x20 [<ffffffff8148f650>] iommu_unmap+0x90/0xe0 [<ffffffff813efdbb>] vfio_remove_dma+0xbb/0x1a0 [<ffffffff813f0526>] vfio_iommu_type1_ioctl+0x3f6/0xa70 [<ffffffffa0064bcc>] ? kvm_set_memory_region+0x3c/0x50 [kvm] [<ffffffffa0064ef2>] ? kvm_vm_ioctl+0x312/0x770 [kvm] [<ffffffff813ee1af>] vfio_fops_unl_ioctl+0x7f/0x2b0 [<ffffffff810cee24>] ? do_sigtimedwait+0xb4/0x1e0 [<ffffffff81192a8e>] do_vfs_ioctl+0x7e/0x500 [<ffffffff8111a463>] ? SyS_futex+0x93/0x1a0 [<ffffffff81192f57>] SyS_ioctl+0x47/0x90 [<ffffffff815b6b62>] system_call_fastpath+0x16/0x1b ---[ end trace 4f4b82e9c04d544c ]--- Also these: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=04:00.0 address=0x00000004485334f0] It seems there is no problem when running Win 7 in low resolution mode without amd drivers installed.

Last edited by ahl (2014-07-09 21:32:22)

Offline

#2280 2014-07-07 17:19:05

pascal257
Member
Registered: 2014-01-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA-Passthrough running perfectly here. Having ArchLinux host with several headless guests, a windows 7 guest and a linux mint guest. Ejecting the graphics gard on windows 7 works flawlessly and I can reboot the guest without having issues. But sometimes nouveau on the linux guest crashes and while I can still access the system via ssh, I'm unable to eject the gfx, so I have to reboot the whole system.

Is there any way to eject the graphics card in linux like with the windows guest?

btw this is the crash on the linux guest, maybe somehow knows how to fix this:

[15398.752221] nouveau E[ PFIFO][0000:01:00.0] read fault at 0x0000039000 [PAGE_NOT_PRESENT] from PCOPY0/PCOPY0 on channel 0x001fdb2000 [DRM] [15475.985061] nouveau E[ DRM] GPU lockup - switching to software fbcon [15491.028017] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0001 [Xorg[1418]] [15506.028007] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0001 [Xorg[1418]] [15508.028221] nouveau E[ PFIFO][0000:01:00.0] playlist update failed [15523.028018] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0000 [Xorg[1418]] [15538.028019] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0000 [Xorg[1418]] [15540.028130] nouveau E[ PFIFO][0000:01:00.0] channel 2 [Xorg[1418]] kick timeout [15540.028448] nouveau W[ PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d [15542.028332] nouveau E[ PFIFO][0000:01:00.0] playlist update failed [15557.104010] nouveau E[chrome[3057]] failed to idle channel 0xcccc0000 [chrome[3057]] [15572.104007] nouveau E[chrome[3057]] failed to idle channel 0xcccc0000 [chrome[3057]] [15574.104188] nouveau E[ PFIFO][0000:01:00.0] channel 5 [chrome[3057]] kick timeout [15574.104445] nouveau W[ PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d [15576.104311] nouveau E[ PFIFO][0000:01:00.0] playlist update failed [15591.104009] nouveau E[cinnamon[2055]] failed to idle channel 0xcccc0000 [cinnamon[2055]] [15606.104010] nouveau E[cinnamon[2055]] failed to idle channel 0xcccc0000 [cinnamon[2055]] [15608.104091] nouveau E[ PFIFO][0000:01:00.0] channel 4 [cinnamon[2055]] kick timeout [15608.104343] nouveau W[ PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d [15610.104224] nouveau E[ PFIFO][0000:01:00.0] playlist update failed [15610.295781] nouveau W[ PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d [15612.295660] nouveau E[ PFIFO][0000:01:00.0] playlist update failed [15612.310667] nouveau W[ PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d [15614.310542] nouveau E[ PFIFO][0000:01:00.0] playlist update failed

Offline

#2281 2014-07-07 19:13:19

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Guys,
I couldl need some help. My setup worked fine but i had to rip it appart because i needed my HD 6970 for a Triple-Head Setup.
At the time i had the HD6970 and a Nvidia GT 630.
Blacklisted the radeon module and pass it through to the VM. All worked fine.

Now i had a HD6970 and a R9 270x

When i try to start the VM i get

virsh # start GamingVM error: Failed to start domain GamingVM error: internal error: early end of file from monitor: possible problem: 2014-07-07T18:55:28.773551Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: vfio: No available IOMMU models 2014-07-07T18:55:28.773590Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: vfio: failed to setup container for group 23 2014-07-07T18:55:28.773607Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: vfio: failed to get group 23 2014-07-07T18:55:28.773627Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: Device initialization failed. 2014-07-07T18:55:28.773647Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

IOMMU is enabled in UEFI, Device is bound to pci-stub.

#dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=10714a4e-bd0b-4f6a-8aa4-e349e7cd4c13 rw pci-stub.ids=1002:6810,1002:aab0 iommu=pt iommu=1 vfio_iommu_type quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=10714a4e-bd0b-4f6a-8aa4-e349e7cd4c13 rw pci-stub.ids=1002:6810,1002:aab0 iommu=pt iommu=1 vfio_iommu_type quiet [ 0.971279] pci-stub: add 1002:6810 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.971291] pci-stub 0000:06:00.0: claimed by stub [ 0.971296] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.971303] pci-stub 0000:06:00.1: claimed by stub #dmesg | grep AMD-Vi [ 0.843061] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 0.843062] AMD-Vi: Interrupt remapping enabled [ 0.843201] AMD-Vi: Initialized for Passthrough Mode

Also qemu is set to run as root and allowed to access to /dev/vfio/{vfio,23} . So what could be the Problem here?

# grep "^[^#]" /etc/libvirt/qemu.conf user = "root" group="root" cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/23" ] clear_emulator_capabilities = 0

Could this be a Problem? As driver is listed the pci-stub but also the radeon module

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 04a3 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 10 Region 0: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 2: Memory at fe500000 (64-bit, non-prefetchable) [disabled] [size=256K] Region 4: I/O ports at b000 [disabled] [size=256] Expansion ROM at fe540000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [200 v1] #15 Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable+, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: vfio-pci Kernel modules: radeon 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: ASUSTeK Computer Inc. Device aab0 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 11 Region 0: Memory at fe560000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 06:00.0 0300: 1002:6810 06:00.1 0403: 1002:aab0

Offline

#2282 2014-07-07 19:37:09

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

switchphase wrote:

Could this be a Problem? As driver is listed the pci-stub but also the radeon module

No, that is normal. It's the "in use" that matters.

Perhaps, try this:
https://bbs.archlinux.org/viewtopic.php … 4#p1366134

Offline

#2283 2014-07-07 19:41:26

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, thats it!
The vfio_iommu_type1 module was missing

Offline

#2284 2014-07-08 03:40:34

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

switchphase wrote:

Now i had a HD6970 and a R9 270x

Can I ask you, how is your R9 270x working now? Does it work fine even after unclean shutdown of the guest?

My R7 260 is only usable with the "eject" method. Unclean shutdown is fatal. I now put the card on host, but it doesn't work that well on host either (with free drivers: glamor, mesa, latest kernel).

Even worse, when I did this on host:
echo high > /sys/class/drm/card0/device/power_profile

The screen went all funny and locked up the gpu...
Perhaps there is something wrong with the power tables?

UPDATE: power_method "profile" is broken, but the newer "dpm" method works assuming latest radeon firmware is installed.

Last edited by ahl (2014-07-09 21:36:34)

Offline

#2285 2014-07-08 07:53:16

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

Can I ask you, how is your R9 270x working now?

I can't say much about it ATM.
With the 440 Chipset it works like a charm, but the performance is bad. (About 20-30FPS)
With the q35 Chipset the host crashes in 3-4 sec.

It maybe a initialization error. I try to dump the BIOS from the card and see if it works.

Offline

#2286 2014-07-08 12:52:23

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

switchphase wrote:
ahl wrote:

Can I ask you, how is your R9 270x working now?

I can't say much about it ATM.
With the 440 Chipset it works like a charm, but the performance is bad. (About 20-30FPS)
With the q35 Chipset the host crashes in 3-4 sec.

It maybe a initialization error. I try to dump the BIOS from the card and see if it works.

Maybe its AMD's IOMMU problem? Both of you have AMD cpu and mobo. I've got Intel's and never encountered such problems.

Offline

#2287 2014-07-08 13:54:14

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Maybe its AMD's IOMMU problem?

Dont think so. Isn't it the same expect the name?
Intel calls it VT-d and AMD calls it IOMMU, but the technology should be the same.

Offline

#2288 2014-07-08 13:57:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

switchphase wrote:
dwe11er wrote:

Maybe its AMD's IOMMU problem?

Dont think so. Isn't it the same expect the name?
Intel calls it VT-d and AMD calls it IOMMU, but the technology should be the same.

No, they are completely different IOMMU implementations


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2289 2014-07-08 14:57:03

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does the VM's share the CPU cores ? or each VM get it owns core assigned ?

Offline

#2290 2014-07-08 18:38:13

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Does the VM's share the CPU cores ? or each VM get it owns core assigned ?

I think they are shared amongst the VM's as long as no VCPU pinning is configured


Saved the VBIOS to a file and tried it again. Nothing changed. After a few seconds my screen looks like this
http://imgur.com/G7yT4uZ
After a few seconds the host freezes completly. Output on the passed through card is seen.

I used the modified parameters for tetsting from the first post without disk

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/tma/VBIOS/r9_270x_vbios.rom \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \ -nographic

Is this VGA Arbiter related?
journalctl tells nothing related to all this.
Is patching Kernel / qemu / seabios anymore needed? (As i heard all related patches are already upstream, but not sure about it)


dwe11er wrote:

I've got Intel's and never encountered such problems.

All i can find is that a mainboard with the same chipset as mine are reported to work (990FX)
Buying new Hardware is not an option atm. I spent already to much money on that stuff.


aw wrote:

No, they are completely different IOMMU implementations

Didn't know that. Makes pinning down errors not easier with multiple implementations.
Anyone got it working with an ASRock extreme9?

Last edited by switchphase (2014-07-08 20:16:39)

Offline

#2291 2014-07-08 20:10:55

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

switchphase wrote:
ahl wrote:

Can I ask you, how is your R9 270x working now?

I can't say much about it ATM.
With the 440 Chipset it works like a charm, but the performance is bad. (About 20-30FPS)
With the q35 Chipset the host crashes in 3-4 sec.

Ouch.
I'll keep my R7 260 then and reserve it just for the Windows guest. It works mighty fine(UPDATE: see below) there except for that stupid reboot problem. Performance is good.

I must also mention that it runs VERY cool thanks to the properly sized heatsink. It is truly a rarity. At least Asus did one thing right.

Last edited by ahl (2014-07-10 09:13:16)

Offline

#2292 2014-07-08 20:28:08

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:

I must also mention that it runs VERY cool thanks to the properly sized heatsink. It is truly a rarity. At least Asus did one thing right.

True

Had a 9800GTX once.
This thing was unusable. With factory settings it runs hot and powered down the pc in less than 30min.
With custom colling settings (always on 100%) and underclocking I was able to play about 1-2h without running hot.

So no more Nvidia for me

Last edited by switchphase (2014-07-08 20:33:14)

Offline

#2293 2014-07-08 21:30:04

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Woohoo! I finally got it working with qemu.

I could do with a couple of tips though, since my setup has its own quirks:
I'm running it as a headless steam-streaming server, which means I won't ordinarily have a KB+M or monitor attached and won't generally interact with the server physically.

I'm launching qemu through ssh with the -nographic argument (or it won't work), and when I do so the ssh tab becomes unresponsive (ctrl+c,x,v don't do anything). I can see the VM boot up fine, but I don't currently have a way of interacting with it. I can't even figure out how to close the VM once it's launched.

Can anyone explain how I set up VNC with the qemu instance, as well as manage qemu guests from the command line? (google is surprisingly unhelpful) Also, can anyone explain how to use a network bridge? Annoyingly, I had both of these working with virt-manager hmm

Here's my qemu profile:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 8,sockets=1,cores=4,threads=8 \ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/var/lib/libvirt/images/Win8Steam-clone.qcow2,id=disk,format=qcow2 \ -device ide-hd,bus=piix4-ide.0,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

I'm really looking forward to getting everything together and am hoping to do a full write up once it's done smile

Last edited by MCMjolnir (2014-07-08 21:31:12)

Offline

#2294 2014-07-08 22:09:32

switchphase
Member
Registered: 2014-07-07
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:

Can anyone explain how I set up VNC with the qemu instance, as well as manage qemu guests from the command line?

When your vm is managed via libvirt you can use virsh. Its pretty the same like virt-manager on the shell. The machine/network settings and so on are done via xml files.
If pretty much the same but without VNC/Spice. I did a DHCP reservation and Autologon + Steam Autostart. Firewall rules are set to allow RDP Sessions.

But I'm far away from testing the RDP / Steam In-Home Streaming.



Gratz to your success.

Offline

#2295 2014-07-08 22:23:05

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was using libvirt/virt-manager before, however to use the kvm=off flag to make my Nvidia card play nice, I *have* to use qemu directly hmm

Running list in virsh while the qemu instance is running doesn't come up with anything, although I could be doing it wrong.

Offline

#2296 2014-07-08 22:59:48

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Maybe its AMD's IOMMU problem? Both of you have AMD cpu and mobo. I've got Intel's and never encountered such problems.

How likely is that?...

I will attempt to test all of my newer cards on Intel platform next week just to see if they behave differently. I have the following hardware:

4 computers: 2x AMD FX, Core i7, AMD A10 APU (HD 7660D)
4-6 gpus: HD3870, HD5450, (HD6450), HD6670, R7 260, (R9 270)

I don't know what to expect.

Offline

#2297 2014-07-09 00:28:10

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got it!
Now, please add AMD HD 4XXX series to the first post.

The guest is running Win8, I had a problem when installing the drivers (Beta ones), and I didn't test the stable driver, Yet, the system works, so.. I think I'm leaving it for now.

I will be editing the google Doc soon.

Offline

#2298 2014-07-09 02:19:00

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for the guide! It finally work after I tried many times, changed many kernel and qemu versions.

So I want to share my experience.

Hardware Info:
Gigabyte Z97-D3H
i5-4570
i915 built-in card for host
Sapphire hd7850 for vm

Software Info:
linux-3.12.4 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Kernel commandline:

linux /vmlinuz-3.12.4-VFIOPassthrough root=UUID=6b2e5409-8b73-40db-ad45-f4ec79cdc0cc quiet
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream
pci-stub.ids=1002:6819,1002:aab0

/etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1
options kvm_intel emulate_invalid_guest_state=0

Qemu commandline:

01:00.0 VGA compatible controller: ATI Technologies Inc Device 6819
01:00.1 Audio device: ATI Technologies Inc Device aab0

/root/vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 -cpu SandyBridge -smp 2,sockets=2,cores=1,threads=1 \
-m 4096 -M q35 -enable-kvm \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/var/lib/libvirt/images/fwq_win7_vfio.img,id=disk,format=raw \
-device ide-hd,bus=piix4-ide.0,drive=disk \
-drive file=/var/lib/libvirt/images/LENOVO_WIN7_64.iso,id=isocd \
-device ide-cd,bus=piix4-ide.1,drive=isocd \
-net nic,model=rtl8139,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \     /* the audio bus must be pcie.0 or you will be failed. */
-usb -usbdevice host:17ef:6018 -usbdevice host:17ef:600e \
-boot order=cd -monitor stdio

PS: the graphic driver for hd7850 that I use [amd_13_9_win7_win8_64_dd_ccc_whql.exe]

Last edited by erganzi (2014-07-09 02:24:45)

Offline

#2299 2014-07-09 21:44:50

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The story of Radeon R7 260 "The Problem" continues

Just when I had accepted the inconveniences of the eject workaround, I found out that the card can also freeze the whole system on Windows startup.

The freezes started happening when I increased -smp cores to 6. Annoyed by that I did about a hundred manual reboots in order to find out which settings work and which don't. Unfortunately, the freeze does not happen every time so the results came out inconclusive -- so don't take this too seriously:

FREEZE OCCURS WITH: -smp sockets=1,cores=6,threads=1 FREEZE DOES NOT OCCUR WITH: cores=2 OR cores=4 OR cores=8 OR without -smp. (-cpu value is irrelevant. Tested to crash with: qemu64, host, Opteron_G4, and Nehalem)

It could all be just a coincidence. Perhaps the settings aren't relevant at all?

Since the freeze turned out to be too hard to reproduce, next I did everything in order to get rid of the whole reboot problem (described here). This included: various motherboard bioses, various qemu versions, changing all bios settings, swapping cards, headless host with just one gpu, removing all non-critical devices... nothing helped.

In one instance I did not get the usual BSoD but just a black frozen screen. It happened when I did "-cpu Opteron_G4,kvm=off". Quite a coincidence, huh? But I do believe it was a simple coincidence.

I currently use:
Linux 3.15.4 with vga_arbitrer.patch and qemu git (or 2.0.0).

Host system:
AMD FX 8350, Asus Sabertooth 990FX R2.0 BIOS 2501.

THE GOOD:
The card seems to work fine on Linux host with free drivers. While the older power_method=profile is still broken, the newer power_method=dpm works fine assuming kernel can find the latest radeon firmware blobs.

Summary:
R7 260 on host: Should be okay
R7 260 on guest: Unstable. Only for testing

UPDATE:
After an upgrade to 3.15.5 (which contains kvm and drm fixes) the host freezing problem triggered several times in a row. I was able to refine the above list slightly. Yet my original clue still seems like a strong candidate.

Why would "-smp sockets=1,cores=6,threads=1" make my computer crash?

UPDATE 2:
It took a while, but I finally managed to prove that this card is unstable regardless of qemu settings and thus useless for PCI passthrough. I will use it on host.

The host freeze problem seems to require certain kind of disk and heavy cpu activity or it simply won't occur. It most often occurs right after reboot when caches are empty.

By the way above mostly applies to Qemu 2.0.0. The git version of Qemu seemed even more unstable although I did not experience host crashes.

I will give up for now. Tips and patches are, of course, very welcome.

Last edited by ahl (2014-07-10 09:25:26)

Offline

#2300 2014-07-09 23:30:49

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so... what does -smp intends to do ?

And, how I could block internet connection for one guest and let the other enjoy the internet ?

Last edited by abdullah (2014-07-09 23:32:37)

Offline

#2301 2014-07-11 08:16:02

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

so... what does -smp intends to do ?

And, how I could block internet connection for one guest and let the other enjoy the internet ?

-smp is number of processor sockets

iptables drop rule?

smth like

iptables -A FORWARD -s client ip -d 0.0.0.0/0 -j DROP
iptables -A FORWARD -s client ip -d LANsubnet -j ACCEPT

maybe... depending on your network configuration....

Last edited by slis (2014-07-11 08:16:16)

Offline

#2302 2014-07-11 08:21:12

MCMjolnir
Member
Registered: 2014-06-19
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone been able to get the HDMI audio working?

Offline

#2303 2014-07-11 08:25:36

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:

Has anyone been able to get the HDMI audio working?

what do you mean? on nvidia it works i think... but i have some latency issues maybe cause of pcie3.0 and sandy bridge... still waiting for ivy to arrive...
on 560ti pcie2.0 card it works flawless...

Last edited by slis (2014-07-11 08:26:02)

Offline

#2304 2014-07-11 16:15:07

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to create an audio infterface on the VM, but I end up with the scratchy / crackling sound problem that some others have come across in this thread.

I tried to set the sample rate and driver for QEMU, but that resulted in the problem becomming 100 times worse so I commented it out.
Startup Script:

#!/bin/sh vfio-bind 0000:01:00.0 0000:01:00.1 #export QEMU_PA_SAMPLES=128 #export QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -M q35 -m 10240 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/home/USER/vm/GamingWindows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -vga none \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:046d:c52b \ -monitor stdio

I also disabled nested tables in kvm_intel:

% systool -vm kvm_intel Module = "kvm_intel" Attributes: coresize = "135528" initsize = "0" initstate = "live" refcnt = "0" taint = "" uevent = <store method only> Parameters: emulate_invalid_guest_state= "Y" enable_apicv = "N" enable_shadow_vmcs = "Y" ept = "Y" eptad = "Y" fasteoi = "Y" flexpriority = "Y" nested = "N" ple_gap = "128" ple_window = "4096" unrestricted_guest = "Y" vmm_exclusive = "Y" vpid = "Y" Sections: .bss = "0xffffffffa00a7140" .data = "0xffffffffa00a6800" .data..read_mostly = "0xffffffffa00a6e68" .data.unlikely = "0xffffffffa00a6ea0" .exit.text = "0xffffffffa00971d2" .fixup = "0xffffffffa0095bdc" .gnu.linkonce.this_module= "0xffffffffa00a6ec0" .init.text = "0xffffffffa00ad000" .note.gnu.build-id = "0xffffffffa0097268" .parainstructions = "0xffffffffa00a5800" .rodata = "0xffffffffa00972c0" .rodata.str1.1 = "0xffffffffa00a5bf0" .rodata.str1.8 = "0xffffffffa00a5cc8" .smp_locks = "0xffffffffa00a5b8c" .strtab = "0xffffffffa00b0b30" .symtab = "0xffffffffa00add10" .text = "0xffffffffa008a000" .text.unlikely = "0xffffffffa00959d0" __bug_table = "0xffffffffa00a6089" __ex_table = "0xffffffffa00a5160" __jump_table = "0xffffffffa00a6cd0" __mcount_loc = "0xffffffffa00a6248" __param = "0xffffffffa00a60a8" __verbose = "0xffffffffa00a6e78"

Has anyone been able to solve this problem when those two things didn't help?

Edit:
I'm becomming unsure if the nested option is refering to the page tables or something else.

Last edited by penetal (2014-07-11 17:28:44)

Offline

#2305 2014-07-12 07:50:52

Midov
Member
Registered: 2013-10-22
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried to get snapshots working?
It seems that when i use -M -q35 snapshoting results in an -22 error.
I do have qcow2 disk

Offline

#2306 2014-07-12 19:58:59

rafalcieslak
Member
Registered: 2014-07-04
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I am into the fourth week of experimenting with passing through a GTX770 (well, apparently a number of others were successful with it!).
The card appars in the device manager of a windows7, where it is recognized as a "Standard VGA Graphics Adapter". Seems like it got passed through well.
However, it's status is:

This device cannot start. (Code 10)

Any nvidia driver fails to install, claiming it "failed to find supported hardware".
This is the closest I got to  a working setup. Has anyone seen that code 10 error? Google searching has not provided me with anything useful about this problem.  What could I make out of it? What should I investigate?

Last edited by rafalcieslak (2014-07-12 20:01:25)

Offline

#2307 2014-07-13 08:56:12

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rafalcieslak wrote:

So I am into the fourth week of experimenting with passing through a GTX770 (well, apparently a number of others were successful with it!).
The card appars in the device manager of a windows7, where it is recognized as a "Standard VGA Graphics Adapter". Seems like it got passed through well.
However, it's status is:

This device cannot start. (Code 10)

Any nvidia driver fails to install, claiming it "failed to find supported hardware".
This is the closest I got to  a working setup. Has anyone seen that code 10 error? Google searching has not provided me with anything useful about this problem.  What could I make out of it? What should I investigate?

post your configs and hardware specs?

error code 10 looks like its not passed correctly...

Offline

#2308 2014-07-13 10:53:31

rafalcieslak
Member
Registered: 2014-07-04
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

post your configs and hardware specs?

error code 10 looks like its not passed correctly...

Okay, there it goes.

The hardware:

Intel 4790K (with an integrated intel HD graphics 4600) NVIDIA GTX 770 (Gigabyte) ASRock Z97 Extreme6

I use stock kernel 3.15.3 configured with:

$ cat .config | grep -i vfi CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y

and patched with vga arbiter fixes (inclucing these for i915) and acs overrides.

Therefore, my /etc/modules contains:

vfio vfio_iommu_type1 vfio-pci

The setup I use is meant to get the host system to use the the intel GPU, and I am passing the nvidia gtx 770 to the guest.
BIOS is set to use intel GPU as the primary graphics adapter (to ensure it won't initialize the gtx 770).

My kenel boot commandline:

linux /vmlinuz-3.15.3 root=UUID=3b7c22f4-85dc-480c-90e7-58e743c1b5d8 ro nvidia.blacklist=yes nvidia-current.blacklist=yes nvidia_337.blacklist=yes nvidiafb.blacklist=yes i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a quiet

(I might skip some of these blacklists, these are just to force the host to leave the gtx770 untoched).
I also use

options kvm allow_unsafe_assigned_interrupts=1

Host:

$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) $ lspci | grep NVIDIA 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) $ lspci -n | grep 01:00 01:00.0 0300: 10de:1184 (rev a1) 01:00.1 0403: 10de:0e0a (rev a1)

I am working with qemu pulled from git.
Before I run qemu, I do:

vfio-bind 0000:01:00.0 0000:01:00.1

I start qemu with:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp cores=4 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -monitor stdio \ -vga cirrus \ -boot menu=on \ -drive file=/dev/sdb3,format=raw,media=disk,id=disk1 \ -drive file=/dev/mapper/main-win,format=raw,media=disk,id=disk2 \ -device ide-hd,bus=ide.0,drive=disk1 \ -device ide-hd,bus=ide.1,drive=disk2

It seems to work, so the host indeed never uses the nvidia GPU, there are no problems mentioned in dmesg, vfio binding is successful, and qemu never complains about a thing.
A linux distro installed as a guest apparently detects the GPU, running lspci I get, among other entries:

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)

which is probably cool.

I will repeat here (for reference and completeness) that a windows7 installed there will detect the passed through device as a "Standard VGA Graphics Adapter" in its Device Manager, but the status of this devices is

This device cannot start. (Code 10)

and the nvidia drivers fail to install, as it "cannot find supported hardware".

Other configurations I have tried that did not help in any combination
kvm=off does not change anything.
Using a romfile for my graphics card (as apparently many others do).
vga -none  is probably quite pointless before I get the drivers to install, there is never any output from the passed GPU.


I am looking both for ideas on what might be wron in my setup, as well as hints on how should I troubleshoot it to investigate the problem further.
Cheers!

Last edited by rafalcieslak (2014-07-13 10:56:20)

Offline

#2309 2014-07-13 12:20:08

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-monitor stdio \
-vga cirrus \
-boot menu=on \

u cant pass it with video cirrus .... u must disable that with -vga none or smth like that... that is error code 10. i think,
u must have SEABIOS screen through 770 when it works with -vga none and i915 patches<<<

hints:
(aka 770 must be PRIMARY card not secondary with cirrus, it can be secondary if u mod it to quadro/grid)

geforce 6xx+ doesn't need rom file... 5xx or lower needs...

your geforce must work regradless of driver installed.... you must have seabios screen on gtx770

try lower driver than 340 aka 331 to be sure driver works with -vga none ofc....

here is my xml with all options for best perfomance i think... i am only misssing x-data-plane option for hdd i guess libvirt still doesn't support it...

this setup works for 2 weeks 0-24, win 8.1 x64, debian 3.15, z77, dual vm, dual 680gtx, used for streaming win games...


<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>                                         
  <name>GTX680</name>                                                                                               
  <uuid>00000000-0000-0000-0000-000000000001</uuid>                                                                 
  <memory unit='KiB'>6242304</memory>                                                                               
  <currentMemory unit='KiB'>6242304</currentMemory>                                                                 
  <vcpu placement='static' cpuset='0-7'>4</vcpu>                                                                     
  <os>                                                                                                               
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>                                                             
    <boot dev='hd'/>                                                                                                 
    <bootmenu enable='no'/>                                                                                         
  </os>                                                                                                             
  <features>                                                                                                         
    <acpi/>                                                                                                         
    <apic/>                                                                                                         
    <hap/>                                                                                                           
    <hyperv>                                                                                                         
      <relaxed state='on'/>                                                                                         
      <vapic state='on'/>                                                                                           
      <spinlocks state='on' retries='4096'/>                                                                         
    </hyperv>                                                                                                       
    <pvspinlock state='on'/>                                                                                         
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='force' name='rtm'/>
    <feature policy='force' name='invpcid'/>
    <feature policy='force' name='lahf_lm'/>
    <feature policy='force' name='erms'/>
    <feature policy='force' name='bmi2'/>
    <feature policy='force' name='smep'/>
    <feature policy='force' name='avx2'/>
    <feature policy='force' name='hle'/>
    <feature policy='force' name='bmi1'/>
    <feature policy='force' name='fsgsbase'/>
    <feature policy='force' name='movbe'/>
    <feature policy='force' name='fma'/>
  </cpu>
  <clock offset='utc'>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/root/win1.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/root/storage/data1.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:12:34:01'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x010b'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x000a'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.scsi=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.config-wce=off'/>
    <qemu:env name='DISPLAY' value=':0'/>
  </qemu:commandline>
</domain>

Last edited by slis (2014-07-13 12:49:10)

Offline

#2310 2014-07-14 09:03:19

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with opengl driver. I am using intel i915 on host and ati radeon 5400 on guest, i applied "i915_315.patch" and enabled "i915.enable_hd_vgaarb=1" in kernel procline. DRI is working without vgaarb patch, not working with vaarb patch.

I am pretty sure i was able to play game (Don't Starve) on host, while guest is running with ati radeon 5400 card. But can't play now i don't know what problem is it.

From Xorg log:

(WW) intel(0): [DRI2] Direct rendering is not supported when VGA arb is necessary for the device

Is there any patch that i miss?

Thank you.

Offline

#2311 2014-07-14 10:26:54

rafalcieslak
Member
Registered: 2014-07-04
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis, huge thanks for your hints! With this help and some slight experiments, I have managed to get my setup working completely. The performance is pretty good, even without tuning.
One particular thing I would like to mention for others who might experience similar problems is that it would seem that I have missed

pcie_acs_override=downstream

in my kernel command line. I am not entirely sure why would be necessary in my setup, but it never works without it, so it is apparently crucial.

Now I am stuck with maintaining my windows installation, as it boots correctly for the first 10-20 times after installing, and then it refuses to cooperate and crashes on boot sad But that is another story... Passthrough is working well, thanks all for the aboundance of information in this topic!

Offline

#2312 2014-07-14 11:35:33

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pcie_acs_override=downstream

you need that if u have multiple devices / gpu-s that use same group behind pcie bridge... so u separate them with it....

Last edited by slis (2014-07-14 11:38:07)

Offline

#2313 2014-07-14 13:01:57

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[ 710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

 <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Offline

#2314 2014-07-14 13:47:29

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[ 710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

 <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Nvidia vga arbiter patch as mentioned in the original post?


i'm sorry for my poor english wirting skills…

Offline

#2315 2014-07-14 13:56:51

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
buraktamturk wrote:

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[ 710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

 <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Nvidia vga arbiter patch as mentioned in the original post?

Thank you, i didn't applied such thing.

How and where i can apply it? Should i apply it to the linux kernel or nvidia driver installer? How i can recompile kernel module to patch it?

Thank you.

Offline

#2316 2014-07-14 19:05:10

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
buraktamturk wrote:

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[ 710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

 <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Nvidia vga arbiter patch as mentioned in the original post?

Now i applied this patch and i am getting the same error. "Invalid ROM contents" error without romfile, no video with romfile.

Old configuration was working without "romfile", when i give "romfile" to old configuration, no video again. I dumped romfile with windows host and GPU-Z program, may be i got wrong dump or the romfile that program generated isn't compatible with seabios?

Any throughts?

EDIT: The ATI card was half-connected to the motherboard, i didn't think such a situation because lspci showed it up. Now everything is working fine. Thanks!

Last edited by buraktamturk (2014-07-15 07:54:09)

Offline

#2317 2014-07-16 08:08:16

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I got it to work, but only if I am using xf86-video-vesa for my host system.
Otherwise if I am using nouveau I get a mass of DMAR Read errors.

DMAR:[fault reason 06] PTE Read access is not set [ 81.044278] dmar: DRHD: handling fault status reg 3 [ 81.044279] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 81.044921] dmar: DRHD: handling fault status reg 3 [ 81.044923] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 81.054085] dmar: DRHD: handling fault status reg 3 [ 81.054086] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 81.055395] dmar: DRHD: handling fault status reg 3 [ 81.055398] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 81.058005] dmar: DRHD: handling fault status reg 3 [ 81.058007] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 81.058659] dmar: DRHD: handling fault status reg 3 [ 81.058661] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 81.061882] dmar: DRHD: handling fault status reg 3 [ 81.061883] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 DMAR:[fault reason 06] PTE Read access is not set [ 118.857410] dmar: DRHD: handling fault status reg 3 [ 118.857425] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr a7aba000 DMAR:[fault reason 05] PTE Write access is not set [ 118.876399] dmar: DRHD: handling fault status reg 3 [ 118.876405] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr c9cdc000 DMAR:[fault reason 05] PTE Write access is not set [ 118.876413] dmar: DRHD: handling fault status reg 3 [ 118.876415] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr b1b5b000 DMAR:[fault reason 05] PTE Write access is not set [ 118.876419] dmar: DRHD: handling fault status reg 3 [ 118.876422] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr c1c5c000 DMAR:[fault reason 05] PTE Write access is not set [ 158.195280] dmar: DRHD: handling fault status reg 3 [ 158.195288] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr 1f472000
01:00.0 VGA compatible controller: NVIDIA Corporation NV43 [GeForce 6600 GT] (rev a2) Kernel driver in use: nouveau Kernel modules: nouveau 07:00.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI Bridge Controller [Ver 1.0] (rev 01) Subsystem: TERRATEC Electronic GmbH Device 1178 Kernel driver in use: Mantis Kernel modules: mantis

I have seen in this thread on page 63 some discussion about that problem to use intel_iommu=iglx_off but I think I am not able to passthrough VGA controllers with that option.

Is there a option to exclude devices from iommu since I only want to passthrouch pci 02:00:00?

Offline

#2318 2014-07-16 13:07:50

d3lxa
Member
Registered: 2014-07-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, thanks for the great post! I would like to contribute what is working for me (everything for now, that's awesome). That's my personal notes:

Hardware:
    CPU: Intel i7-4770S
    MB : GA Z87N-WIFI
    GPU: Intel HD 4600 (linux, integrated)
    GPU: Asus HD 6850 (win7, dedicated)
        romfile (VERY IMPORTANT): Asus.HD6850.1024.110616.rom
        2011-06-16 22:24:00     013.012.000.021.000000  PCI-E   790 / 1000
        ^ http://www.techpowerup.com/vgabios/inde … =&memSize=
    require (1 for linux, 1 for windows):
        2 monitors + 2 keyboards + 2 mouses
        (^ or 1 monitor with two inputs)


Linux:
Debian Testing, linux-image-3.14-1-amd64=3.14.12-1 (compiled myself with CONFIG_VFIO_PCI_VGA=y CONFIG_HZ_1000=y)
    tested too: 3.15.5 (trunk + acs_override)
    bootline: quiet intel_iommu=on
        not necessary: vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream i915.enable_hd_vgaarb=1 (corresponding patch not applied anyway)
    qemu: qemu-kvm=2.0.0+dfsg-6+b1
       or from git ab6d3749c4915cd5692633e321f7745dce06fe77
    xserver-xorg 1:7.7+7
    xserver-xorg-video-intel 2:2.21.15-2+b1
    libegl1-mesa-drivers 10.2.2-1
    firmware-linux 0.43
    firmware-linux-free 3.3

qemu line:
    QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 ~/qemu-git/x86_64-softmmu/qemu-system-x86_64 \
        -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device piix4-ide,bus=pcie.0,id=piix4-ide \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/Asus.HD6850.1024.110616.rom \
        -device vfio-pci,host=01:00.1,bus=pcie.0 \
        -usb -usbdevice host:0603:00f2 -usbdevice host:046d:c01b \
        -soundhw ac97 \
        -drive file=$HOME/win7_rootfs.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
    ;

audio
    -soundhw ac97
        nearly perfect (some glitches)
        driver: http://www.realtek.com.tw/downloads/dow … Down=false
    -device usb-audio and QEMU_AUDIO_TIMER_PERIOD=10
        better but high CPU, may glitch under heavy load (or after closing program)
    -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
        audio glitches (even with timer period to 0)

cdrom
    for installation (OS or drivers):
    -drive file=$HOME/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    can be removed safely after reboot (install)

disks
    virtio: can be switched to *after* fedora driver install
    https://alt.fedoraproject.org/pub/alt/v … mages/bin/
    install manually in windows by "updating" the driver of the unknown disk device

software tested:
    Firefox WebGL (google map)
    Pangya
    Natural Selection 2 (Steam)
    Media Player Classic (h264 video)

bugs:
    each time the VM boots, my linux display has color glitches (partial inversion), need to switch TTY (ctrl+alt+X) to reset colors (framebuffer)
    sound may glitch a bit in video playback


Windows 7 SP1 Ultimate (64 Bit)
    torrent: 7ef0145bc19f53251df9ff7e11a795e9e088b65c
    drivers: 14.10.1006-140417a-171099C (Overdrive works)
          or amd_13_9_win7_win8_64_dd_ccc_whql.exe
            ^ http://www.driversdown.com/drivers.asp? … m=3&show=0

Before starting qemu, I've to switch to the vfio driver for my two devices (GPU+HDMI sound card, the readeon driver is *not* loaded it's blacklisted):

# $ lspci | grep AMD #01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] #01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88] sudo modprobe vfio sudo modprobe vfio_pci echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

One thing that would be more convenient is to share the keyboard+mouse between the two OSes with a software multiplexer: that would create two virtual devices, one for linux + one for windows and you could choose to what it's connected. For the monitors that's fine, one way would be to use VNC inside windows but I'm not sure of the overhead, wouldn't be nice. The network seems fine, I don't know if the default nic induce an important overhead, I didn't notice in game, seems fine.

I have tested the sound and it works (my microphone too! This is useful in NS2). The only problem of the whole is a minor bug when Windows init my card, it touches my linux buffer and changes some colors (kind of psychedelic) but that's once at boot, not that annoying. This looks a bit like what this guy had:

switchphase wrote:

After a few seconds my screen looks like this http://imgur.com/G7yT4uZ.

You can see the harmless messages in my kern.log while the VM was running here (unsorted) http://paste.debian.net/110091/ .

Hope this helps!

Last edited by d3lxa (2014-07-16 13:37:55)

Offline

#2319 2014-07-16 13:22:34

d3lxa
Member
Registered: 2014-07-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

penetal wrote:

I am trying to create an audio infterface on the VM, but I end up with the scratchy / crackling sound problem that some others have come across in this thread.

I tried to set the sample rate and driver for QEMU, but that resulted in the problem becomming 100 times worse so I commented it out.
[…]
Has anyone been able to solve this problem when those two things didn't help?

Edit:
I'm becomming unsure if the nested option is refering to the page tables or something else.

I had this with the same settings you were using, try me version with -soundhw ac97 (remove the two "sound" lines) and don't forget to export the delay, not sure it works for PulseAudio though.

Offline

#2320 2014-07-16 19:42:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey folks,

I've got some reset tweaks up for testing that might be useful to a number of you:

https://lkml.org/lkml/2014/7/16/562 (patch 1, patch 2, patch 3)

https://lkml.org/lkml/2014/7/16/567 (patch)

With these, the GPU should be reset regardless of how clean the guest was shutdown.  Note that all devices on the bus need to be bound to vfio-pci for this to work.  That means that if you're using pci-stub to sequester unused devices (perhaps the audio function), bind them to vfio-pci or there won't be any change.  The second set of links is a patch necessary for AMD GPU users only and I'd appreciate feedback on whether it solves and of the other reset issues (lack of reset) found on these GPUs.  Thanks

PS - these are kernel patches against the latest development kernel, 3.16-rc5.  They may or may not apply cleanly to older kernels.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2321 2014-07-16 20:10:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

d3lxa wrote:

One thing that would be more convenient is to share the keyboard+mouse between the two OSes with a software multiplexer: that would create two virtual devices, one for linux + one for windows and you could choose to what it's connected. For the monitors that's fine, one way would be to use VNC inside windows but I'm not sure of the overhead, wouldn't be nice. The network seems fine, I don't know if the default nic induce an important overhead, I didn't notice in game, seems fine.

Synergy?

I have tested the sound and it works (my microphone too! This is useful in NS2). The only problem of the whole is a minor bug when Windows init my card, it touches my linux buffer and changes some colors (kind of psychedelic) but that's once at boot, not that annoying.

This is what the i915 patch and kernel option fixes that you've decided is not necessary.  It means that you won't get any VGA output on the guest, such as seabios or maybe Windows prompting how to startup after a crash.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2322 2014-07-16 23:21:53

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

@aw Thanks for the new patch, i will test for AsMedia Sata controler.
But i found a new strange bug with Renesas USB 3.0 controler, when it use with vfio-pci the device is activate with a lot of new msi/msi-x like 43, 44, 45, 46 ... around 20 new irq adresse.

Offline

#2323 2014-07-16 23:29:23

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hey folks,

I've got some reset tweaks up for testing that might be useful to a number of you:

https://lkml.org/lkml/2014/7/16/562 (patch 1, patch 2, patch 3)

https://lkml.org/lkml/2014/7/16/567 (patch)

With these, the GPU should be reset regardless of how clean the guest was shutdown.  Note that all devices on the bus need to be bound to vfio-pci for this to work.  That means that if you're using pci-stub to sequester unused devices (perhaps the audio function), bind them to vfio-pci or there won't be any change.  The second set of links is a patch necessary for AMD GPU users only and I'd appreciate feedback on whether it solves and of the other reset issues (lack of reset) found on these GPUs.  Thanks

PS - these are kernel patches against the latest development kernel, 3.16-rc5.  They may or may not apply cleanly to older kernels.

I've tested them against 3.16-rc5 kernel. They apply clean, I've rebooted and re-tested my Linux + fglrx and Windows + catalyst with my R9 290X. It still suffers from the reset problem. sad On Linux + fglrx I get oops (inside VM) after reboot of VM and X start. On Windows + catalyst I still get the BSOD PAGE_FAULT_IN_NONPAGED_AREA.

Last edited by mbroemme (2014-07-16 23:30:29)

Offline

#2324 2014-07-17 00:49:12

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I ran

dmesg | grep -e DMAR -e IOMMU

and got:

[ 0.056332] IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0 rishub@debian:~$ dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 00000000de555810 000080 (v01 INTEL BDW 00000001 INTL 00000001) [ 0.056269] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.056332] IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0 [ 0.537854] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 0.537854] AMD IOMMUv2 functionality not available on this system

I am running i7 4790k with Msi Gaming 5
In my bios, there is a vt-d option and the processor supports it

Do I have it running on my system? Is this output correct?

Offline

#2325 2014-07-17 01:18:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

I've tested them against 3.16-rc5 kernel. They apply clean, I've rebooted and re-tested my Linux + fglrx and Windows + catalyst with my R9 290X. It still suffers from the reset problem. sad On Linux + fglrx I get oops (inside VM) after reboot of VM and X start. On Windows + catalyst I still get the BSOD PAGE_FAULT_IN_NONPAGED_AREA.

Bummer.  Is the oops a regression or is that something you see already?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2326 2014-07-17 01:39:52

paigeadele
Member
Registered: 2014-07-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got mine working on gentoo, a friend of mine helped me with some performance issues as well. Here all some links to the scripts I use to start the windows VM:

https://github.com/paigeadele/windowlic … windows.sh

I added the (-cpu host,hv_relaxed,hv_vapic,hv_time,hv_spinlocks=0x1000) options a friend of mine said that they help, but I haven't benchmarked it yet to see if there's any difference. I'm using an ATI Radeon 5770 which I may upgrade now that I've at least seen this work. Initially I had configured the cpu topology to use two sockets, 2 cores each and decided to reduce it to 1 and 6 cores in case using 2 sockets might have been adding any contention...was mostly just curious to see how it worked. I haven't found much good documentation on the hv_spinlocks and threads=1 stuff, I'm curious to see how the impact the responsiveness. I also figured out how to setup macvtap which I currently use for all of my libvirtd instances.

https://github.com/paigeadele/windowlic … 5.5-gentoo

someone posted in this forum about enabling preemption helping (though I thought preemption didn't work on it's own.) I also set the ticks frequency to 1000hz and figured at least that option might make some difference.

Everything seems to be working fine, theres one issue with netflix when the playback menu pops up when you hover over it that makes the video lag a bit... and I'm not sure if thats the card being slow or processor bottleneck....lots to figure out but everything is awesome, thank you for this post!

Offline

#2327 2014-07-17 02:44:20

paigeadele
Member
Registered: 2014-07-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Interesting, removed the threads=1 to see if it would let it spin up however many threads it wants, seem to be a little snappier but hard to tell.

http://imgur.com/6W8Nbnl

Was having some trouble with HD video playback on netflix in firefox. It seems a little slow / choppy,... yeah still same.. I wonder why... silverlight / plugin container seems to eat up quite a bit of cpu in the vm

I guess I could unmask the 9999 qemu build and give it a try.

Last edited by paigeadele (2014-07-17 02:54:04)

Offline

#2328 2014-07-17 02:54:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paigeadele wrote:

Interesting, removed the threads=1 to see if it would let it spin up however many threads it wants, seem to be a little snappier but hard to tell.

http://imgur.com/6W8Nbnl

Was having some trouble with HD video playback on netflix in firefox. It seems a little slow / choppy,... yeah still same.. I wonder why... silverlight / plugin container seems to eat up quite a bit of cpu in the vm

threads is just a modifier of the topology of the vCPU, it exposes processor threads, or hyper threadsthreads=1 is the default, each core has one thread.  Core i5/i7 processors often have 2 threads per core.  The total number of vCPUs exposed to the guest is "sockets x cores x threads".  The guest scheduler can make various optimizations given a topology, like optimizing for power efficiency by only running on a single socket, or optimizing for throughput by using cores within the same socket or only a single thread per core, etc.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2329 2014-07-17 03:02:21

paigeadele
Member
Registered: 2014-07-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gotcha, yeah I'm building the qemu 9999 package now I'll see if there's much of a performance change since 2.0

Offline

#2330 2014-07-17 09:44:47

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mbroemme wrote:

I've tested them against 3.16-rc5 kernel. They apply clean, I've rebooted and re-tested my Linux + fglrx and Windows + catalyst with my R9 290X. It still suffers from the reset problem. sad On Linux + fglrx I get oops (inside VM) after reboot of VM and X start. On Windows + catalyst I still get the BSOD PAGE_FAULT_IN_NONPAGED_AREA.

Bummer.  Is the oops a regression or is that something you see already?

It is the same.

Offline

#2331 2014-07-17 10:57:04

DLWood1001
Member
Registered: 2014-07-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Offline

#2332 2014-07-17 15:05:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DLWood1001 wrote:

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

Thanks for the test, both series weren't really targeted at fixing this, but I needed to add a device specific reset for my Oland-based HD8570 to get it to reset on release and I thought it worth a test to see if it fixed anything.  If you kill the qemu process with these patches, what happens on your r9 290x?  Does the monitor stay in sync and the framebuffer show the last image, or does the monitor go blank and the GPU fan reset to high speed?

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Great, these should show up in 3.17


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2333 2014-07-17 21:30:52

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone have success with the GTX 580?

Offline

#2334 2014-07-19 09:18:04

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shrubuntu wrote:

Anyone have success with the GTX 580?

Take a look at this
https://bbs.archlinux.org/viewtopic.php … 8#p1371808

And there is also this
https://docs.google.com/spreadsheet/ccc … _web#gid=0

Offline

#2335 2014-07-19 14:36:56

BitMaster
Member
Registered: 2011-01-15
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using this setup:

First I applied all the OP patches (acs, i915 and vga arbiter) and this worked more or less. I got it booting with display on the Nvidia card, but it breaks the intel driver on the host. X switches to software rendering and glxinfo indicates it no longer uses Intel/Mesa OpenGL but instead VMware (?):

$ glxinfo | grep OpenGL OpenGL vendor string: VMware, Inc. OpenGL renderer string: Gallium 0.4 on llvmpipe (LLVM 3.4, 256 bits) OpenGL version string: 3.0 Mesa 10.2.3 OpenGL shading language version string: 1.30 OpenGL context flags: (none) OpenGL extensions:

Next i tried to minimize the amount of patches (in an effort to pinpoint the intel host driver issue). These are the remaining patches i need to have a working VGA passthrough: i915-vtd.patch.
If i disable the

i915_disable_vga_mem(dev);

line in 'drivers/gpu/drm/i915/i915_dma.c' X is in normal mode again and Intel/Mesa OpenGL is used, but sadly VGA passthrough is broken. This means the display goes to sleep mode when the virtual machine is started.

I scanned the driver code, but I can't find the line that enables the vga memory again (only on 'deinit/cleanup'?). Is it impossible to use the onboard VGA memory in combination with VGA passthrough?
Any suggestions are very welcome!

----
Oh, by the way i disabled/blacklisted the nouveau module. My kernel parameters are nothing special:

$ cat /proc/cmdline initrd=\initramfs-linux-vtd2.img root=PARTUUID=1b8f88d4-3bb2-4eae-abe8-180d0300f908 rw nouveau.modeset=0 quiet intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a drm.debug=0x02

I don't need to use pci-stub, as my graphics card is not bound to any driver on boot. Before i start the VM, I bind them to vfio (see OP). I use qemu-git from the AUR.

Last edited by BitMaster (2014-07-19 14:58:29)

Offline

#2336 2014-07-19 21:54:53

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been lurking for a while, since the inability to bus-reset the R9 290 is kind of a show-stopper.  I couldn't find an official bug report to follow for it, and the thread I found on the mailing lists has stopped.

I hope OVMF will get support for the Q35 machine soon.  With a UEFI video card and a legacy-free (Windows 8.1) guest, you don't need the Intel VGA Arbiter.

I've noticed that the instructions in the OP, as well as virt-manager, put all passed-through devices directly behind "root port 0". 
Real boards have one device (which may be a switch) behind each port, and each port (numbered starting at 1, not 0) has a different device ID.
I can't see how multiple devices behind one port has ever worked, and it wouldn't surprise me if drivers don't like it.

When I tried Q35 in non-EFI mode, I used this file to give multiple root ports: https://github.com/qemu/qemu/blob/maste … hipset.cfg

(Interestingly enough, the '-m q35' default AHCI controller is named 'ide'; you attach disks to bus ide.0, ide.1, and so on.)

Now, for a question: do you think it would be reasonable to start factoring device types into the building of IOMMU groups?

For example, given multiple video cards behind the Intel CPU ports, they are highly likely to converse peer-to-peer.
Given an LSI SAS controller and an AMD video card, on the other hand, what are the chances of one trying to talk to the other?

Last edited by DanaGoyette (2014-07-19 22:01:42)

Offline

#2337 2014-07-20 01:20:46

k0olfir3
Member
Registered: 2014-07-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

this thread is amazing. However, I have to admit that I didn't read all the 94 pages.

I tried to replicate the OP's procedures. Unfortunately, I have not been very succesfull so far. Whenever I test passing through the Video card my monitor stays blank. I have an Intel i7 4790 CPU on an AsRock Z87 Pro 4 board and a NVIDIA GTX 770 card. I use the OP's mainline kernel and git 2.0 from the standard Arch repositories. Here are some further details on my setup:

uname -a Linux arch-desk 3.15.1-1-mainline #1 SMP PREEMPT Sun Jul 20 00:50:27 CEST 2014 x86_64 GNU/Linux
cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=xxx quiet iommu=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a

I noticed the following errors in dmesg when I run qemu without passing some romfile option:

[ 5716.098747] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to c400c [ 5716.098802] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5716.098814] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5716.098827] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

These errors vanish if I use a vga-bios rom file I downloaded from: http://www.techpowerup.com/vgabios/1454 … 30704.html. Both my screen is staying blank in either case. I tried the OP's suggestions:

vfio_iommu_type1.allow_unsafe_interrupts=1 and kvm_intel emulate_invalid_guest_state=0

But that did not appear to change very much. Do you guys have any suggestions what I could try next? Am I missing something?

EDIT: I forgot to mention that I want to use the internal Intel GPU on the host system.

Last edited by k0olfir3 (2014-07-20 01:25:21)

Offline

#2338 2014-07-20 01:25:37

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hey folks,

I've got some reset tweaks up for testing that might be useful to a number of you:

https://lkml.org/lkml/2014/7/16/562 (patch 1, patch 2, patch 3)

https://lkml.org/lkml/2014/7/16/567 (patch)

With these, the GPU should be reset regardless of how clean the guest was shutdown.  Note that all devices on the bus need to be bound to vfio-pci for this to work.  That means that if you're using pci-stub to sequester unused devices (perhaps the audio function), bind them to vfio-pci or there won't be any change.  The second set of links is a patch necessary for AMD GPU users only and I'd appreciate feedback on whether it solves and of the other reset issues (lack of reset) found on these GPUs.  Thanks

PS - these are kernel patches against the latest development kernel, 3.16-rc5.  They may or may not apply cleanly to older kernels.

Thanks for aw's patch, I'll try this to solve VM reboot stuck issue in these days. And I'll report.

Also, I got another goods news is, now I can run CUDA perfectly in VM right now.

I found that I need to do this with GUI-based OS, so if I tried to run CUDA when the OS is terminal only (like Ubuntu Server), I need to launch GPU acceleration in text mode (for Linux, it means like "init 3")
So I give up to try Ubuntu Server and change guest OS to Fedora 20, and when I finished the installation and entered into system. I checked host OS's dmesg.
And I found that IRQ is already set up while GUI interface launched.

vfio-pci: 0000:03:00.0: irq 89 for MSI/MSI-X

Then I install NVIDIA driver, CUDA SDK, and compile CUDA sample codes. After all, I run some sample application. It works perfectly! Now I'm not get any strange errors.
Also I checked host dmesg each time I run CUDA application, and VFIO not reported IRQ changed message anymore. Just showed while Fedora boot.

But I still need to turn off GUI to get a better performance, fortunately someone knows how to get GPU acceleration with NVIDIA official driver in text mode.
I'll try it up and report that.

Offline

#2339 2014-07-20 03:40:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

k0olfir3 wrote:

Hi all,

this thread is amazing. However, I have to admit that I didn't read all the 94 pages.

I tried to replicate the OP's procedures. Unfortunately, I have not been very succesfull so far. Whenever I test passing through the Video card my monitor stays blank. I have an Intel i7 4790 CPU on an AsRock Z87 Pro 4 board and a NVIDIA GTX 770 card. I use the OP's mainline kernel and git 2.0 from the standard Arch repositories. Here are some further details on my setup:

uname -a Linux arch-desk 3.15.1-1-mainline #1 SMP PREEMPT Sun Jul 20 00:50:27 CEST 2014 x86_64 GNU/Linux
cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=xxx quiet iommu=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a

I noticed the following errors in dmesg when I run qemu without passing some romfile option:

[ 5716.098747] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to c400c [ 5716.098802] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5716.098814] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 5716.098827] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

These errors vanish if I use a vga-bios rom file I downloaded from: http://www.techpowerup.com/vgabios/1454 … 30704.html. Both my screen is staying blank in either case. I tried the OP's suggestions:

vfio_iommu_type1.allow_unsafe_interrupts=1 and kvm_intel emulate_invalid_guest_state=0

But that did not appear to change very much. Do you guys have any suggestions what I could try next? Am I missing something?

EDIT: I forgot to mention that I want to use the internal Intel GPU on the host system.

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

Offline

#2340 2014-07-20 09:10:45

_pheinrich_
Member
Registered: 2014-05-26
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

all is working like expected now!

Config:

MB: Asus P8P67EVO
CPU: i5 3470
Host GPU: nvidia 6600GT
Guest GPU: nvidia GTX 260

Kernel 3.15.5-2 with acs patch
Nvidia Kernel module with http://bpaste.net/show/109185/

BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=3264d3aa-f452-4866-9397-644efdb107a1 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2

I want to thank you all for your support! Keep alive this thread.

Offline

#2341 2014-07-20 12:33:57

k0olfir3
Member
Registered: 2014-07-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

That did the trick.

When I just returned to your original post for the further proceedings, I saw that you you mentioned that option there. Did you just put it there or did I really miss it? That would be kind of embarassing. Anyway, thanks for your help.

Offline

#2342 2014-07-20 12:44:57

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

USING VIRT-MANAGER

This may help someone - after much investigation, trial & error and general messing around I have managed to get VGA passthroufgh working with virt-manager / libvirt  (on Ubuntu Trusty)

you will need to install the virt-manager, libvirt first.
The process assumes you already have a working qemu command and you have saved it to a file

1) Create the standard qemu command and save it into a text file
      Remove all the line feed and backslash entries first (they seem to cause virsh some grief) ie. one long line of text
      NOTE will probably have to add some backslash and linefeeds back in (needed for cd-rom defs at end in my case)
2) use -->  virsh domxml-from-native qemu-argv qemu_command_temp.txt > my_qemu_start.xml
      THis will create a new file (my_qemu_start.xml) which can be imported
      NOTE you may have to add some backslash and linefeeds back in (I needed them for cd defs at end, but then removed themmanually in final machine definition)
3) use --> virsh define my_qemu_start.xml            which will create the actual machine xml (file name will be the machine name specified, plus .xml suffix)
4) use --> export EDITOR=nano; virsh edit machine.xml   ...
     change the boot source to 'hd'   from 'network'   (why virsh creates this I don't know)
     check the emulator line, may need to specify location of qemu eg. in my case /usr/bin/qemu-system-x86_64
     remove the video lines in the definition (because we want primary passthru - no cirrus)
     remove the sdl line  (video type) cause  we don't want it either
     remove the "tablet" definition - it fails to conenct to USB device, along with keyboard and mouse
     remove the "\" from the cd lines (had to be added in previous step)
5) edit /etc/libvirt/qemu.conf and set

 user = "+0" group = "+0" 

     
     add /dev/vfio/xxxxx (in my case 1, 15, 16 17)     entries to cgroup_device_acl section

 cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1", "/dev/vfio/15", "/dev/vfio/16", "/dev/vfio/17" ] security_driver = "none" security_default_confined = 0 hugetlbfs_mount = "/dev/hugepages" clear_emulator_capabilities = 0 relaxed_acs_check = 1 

6) reboot to instantiate the new libvirt parameters
7) start new machine from virt-manager ......

NOTE if "open" the machine (initiate virt-viewer) then some new entries may be added to the machine definition which will stop it working. If this does happen use "virsh edit" and remove the gratuitous video, tablet, keyboard, mouse etc ... and check the updated qemu:arg entries, in my case backspaces were added which stopped the machine starting. It looked as though the originally generated definition was being restored ie. pre the edits to "fix" the generation errors.

I started with this

 qemu-system-x86_64 -name win7 -enable-kvm -M q35 -m 6144 -mem-path /dev/hugepages \ -cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -boot menu=on \ -smp 3,sockets=1,cores=3,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -usbdevice tablet \ -spice port=5902,disable-ticketing \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device ahci,bus=pcie.0,id=ahci \ -net nic,macaddr=00:00:00:aa:bb:cc,model=virtio \ -net user \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=02:00.0,bus=pcie.0 \ -device vfio-pci,host=04:00.0,bus=pcie.0 \ -device vfio-pci,host=06:00.0,bus=pcie.0 \ -device vfio-pci,host=05:00.0,bus=pcie.0 \ -drive file=/dev/virt-test/lvwin7_kvm,id=disk1,format=raw,if=virtio \ -drive file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2 \ -device ide-cd,bus=ahci.2,drive=isocd2 \ -drive file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3 \ -device ide-cd,bus=ahci.3,drive=isocd3 

I edited it and made it into this

qemu-system-x86_64 -name win7_t2 -enable-kvm -M q35 -m 6144 -mem-path /dev/hugepages -cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -boot menu=on -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -usbdevice tablet -spice port=5902,disable-ticketing -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device ahci,bus=pcie.0,id=ahci -net nic,macaddr=00:00:00:aa:bb:cc,model=virtio -net user -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device vfio-pci,host=02:00.0,bus=pcie.0 -device vfio-pci,host=04:00.0,bus=pcie.0 -device vfio-pci,host=06:00.0,bus=pcie.0 -device vfio-pci,host=05:00.0,bus=pcie.0 -drive file=/dev/virt-test/lvwin7_kvm,id=disk1,format=raw,if=virtio \ -drive file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2 \ -device ide-cd,bus=ahci.2,drive=isocd2 \ -drive file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3 \ -device ide-cd,bus=ahci.3,drive=isocd3

note that I had to restore some of the backslashes and linefeeds because the machine was giving some strange errors like

Error starting domain: internal error: process exited while connecting to monitor: 2014-07-20T12:09:56.863675Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=pcie.0: could not open disk image \ -drive: Could not open '\ -drive': No such file or directory

Other errors related to permissions and use of hugepages were addressed by the changes to /etc/libvirt/qemu.conf


that became this once virsh domxml-from-native qemu-argv was run 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7_t2</name> <uuid>f778660a-9da7-4bad-ba5b-477feb74f2b9</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>3</vcpu> <os> <type arch='x86_64' machine='q35'>hvm</type> <loader>/usr/share/qemu/bios.bin</loader> <boot dev='network'/> <boot dev='network'/> </os> <features> <acpi/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='4096'/> </hyperv> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <topology sockets='1' cores='3' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/virt-test/lvwin7_kvm'/> <target dev='vda' bus='virtio'/> </disk> <controller type='sata' index='0'/> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> <interface type='user'> <mac address='00:00:00:aa:bb:cc'/> <model type='virtio'/> </interface> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='sdl'/> <video> <model type='cirrus' vram='9216' heads='1'/> </video> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-mem-path'/> <qemu:arg value='/dev/hugepages'/> <qemu:arg value='-spice'/> <qemu:arg value='port=5902,disable-ticketing'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='ahci,bus=pcie.0,id=ahci'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/> <qemu:arg value='\ -drive'/> <qemu:arg value='file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2'/> <qemu:arg value='\ -device'/> <qemu:arg value='ide-cd,bus=ahci.2,drive=isocd2'/> <qemu:arg value='\ -drive'/> <qemu:arg value='file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3'/> <qemu:arg value='\ -device'/> <qemu:arg value='ide-cd,bus=ahci.3,drive=isocd3'/> </qemu:commandline> </domain>

and then became this after running virsh define my_qemu_start.xml and then using virsh edit to remove the unnecessary lines (and change the boot from network to hd)

<!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit win7_t2 or other application using the libvirt API. --> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7_t2</name> <uuid>f778660a-9da7-4bad-ba5b-477feb74f2b9</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>3</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <loader>/usr/share/qemu/bios.bin</loader> <boot dev='hd'/> </os> <features> <acpi/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='4096'/> </hyperv> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <topology sockets='1' cores='3' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/virt-test/lvwin7_kvm'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <interface type='user'> <mac address='00:00:00:aa:bb:cc'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-mem-path'/> <qemu:arg value='/dev/hugepages'/> <qemu:arg value='-spice'/> <qemu:arg value='port=5902,disable-ticketing'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='ahci,bus=pcie.0,id=ahci'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:00.0,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2'/> <qemu:arg value='-device'/> <qemu:arg value='ide-cd,bus=ahci.2,drive=isocd2'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3'/> <qemu:arg value='-device'/> <qemu:arg value='ide-cd,bus=ahci.3,drive=isocd3'/> </qemu:commandline> </domain>

which would run as machine win_t2
it was all quite painfull the first time since I tried to create a definition file from scratch - which was completely unsuccessful even though it looked a lot like the final product.

Comments / improvements welcome

Offline

#2343 2014-07-20 16:03:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

k0olfir3 wrote:
nbhs wrote:

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

That did the trick.

When I just returned to your original post for the further proceedings, I saw that you you mentioned that option there. Did you just put it there or did I really miss it? That would be kind of embarassing. Anyway, thanks for your help.

It was a ninja edit

Offline

#2344 2014-07-20 16:08:59

k0olfir3
Member
Registered: 2014-07-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I was now able to pass through my GTX 770 card and to install win 7 on the virtual machine. However, in windows I cannot use the card with the NVIDIA drivers. I get that annoying yellow triangle with Code 43.

My research in this thread indicates that this problem is related to the NoSnoop attribute. At least it appears to be enabled on my system:

lspci -vvvv 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) (prog-if 00 [VGA controller]) ... Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ ... 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ... Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ ...

In this forum aw posted a patch for qemu to solve this issue. However, if I understand this correctly that patch should already be included in qemu >=2.0.0.

On the web I found this site:
http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
which is suggesting to the following kernel patches.

http://git.kernel.org/cgit/linux/kernel … 7612b7f235
http://git.kernel.org/cgit/linux/kernel … 860dae4d56
http://git.kernel.org/cgit/linux/kernel … a429732ef4

They appear to older than 3.15.1, should I try them?

If yes, can I just edit nbhs's PKGBUILD such that these patches are apllied to linux-mainline? For example by adding something like this to the prepare section?

patch -p1 -i "${srcdir}/whatever_file_name.patch"

Assuming this works out, will I need to pass any additional options to the kernel to enable the patches (as was the case for the i915 patch)?

Or do you have any other ideas that may solve my problem?

Sorry for posing all those stupid questions. I don't know very much about programming, but I really want that gaming virtual machine.

Offline

#2345 2014-07-20 16:19:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

k0olfir3 wrote:

Hi all,

I was now able to pass through my GTX 770 card and to install win 7 on the virtual machine. However, in windows I cannot use the card with the NVIDIA drivers. I get that annoying yellow triangle with Code 43.

My research in this thread indicates that this problem is related to the NoSnoop attribute. At least it appears to be enabled on my system:

lspci -vvvv 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) (prog-if 00 [VGA controller]) ... Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ ... 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ... Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ ...

In this forum aw posted a patch for qemu to solve this issue. However, if I understand this correctly that patch should already be included in qemu >=2.0.0.

On the web I found this site:
http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
which is suggesting to the following kernel patches.

http://git.kernel.org/cgit/linux/kernel … 7612b7f235
http://git.kernel.org/cgit/linux/kernel … 860dae4d56
http://git.kernel.org/cgit/linux/kernel … a429732ef4

They appear to older than 3.15.1, should I try them?

If yes, can I just edit nbhs's PKGBUILD such that these patches are apllied to linux-mainline? For example by adding something like this to the prepare section?

patch -p1 -i "${srcdir}/whatever_file_name.patch"

Assuming this works out, will I need to pass any additional options to the kernel to enable the patches (as was the case for the i915 patch)?

Or do you have any other ideas that may solve my problem?

Sorry for posing all those stupid questions. I don't know very much about programming, but I really want that gaming virtual machine.

I believe there are some problems with the lastest nvidia drivers, if you are using the lastest qemu git, you can also try kvm=off as a cpu option, theres a discussion about this a few pages back

Last edited by nbhs (2014-07-20 16:42:45)

Offline

#2346 2014-07-20 16:53:30

k0olfir3
Member
Registered: 2014-07-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I believe there are some problems with the lastest nvidia drivers, if you are using the lastest qemu git, you can also try kvm=off as a cpu option, theres a discussion about this a few pages back

Great! You always come up with easy solutions when I think everything is terribly complicated. I installed an earlier version of the NVIDIA drivers (332.21) in the virtual machine and now, the graphics are working. Thanks!

Offline

#2347 2014-07-21 00:39:41

DLWood1001
Member
Registered: 2014-07-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DLWood1001 wrote:

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

Thanks for the test, both series weren't really targeted at fixing this, but I needed to add a device specific reset for my Oland-based HD8570 to get it to reset on release and I thought it worth a test to see if it fixed anything.  If you kill the qemu process with these patches, what happens on your r9 290x?  Does the monitor stay in sync and the framebuffer show the last image, or does the monitor go blank and the GPU fan reset to high speed?

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Great, these should show up in 3.17

Hi Aw,

Sorry about the delay in my response. I tested with the 3.16 rc5 kernel, reset patch applied and with a R9 290X. Still the same result. As an A/B test, I tried an HD7770 it seems to reset just fine with OSX as the guest, however I'm not sure if it had a problem pre-patch as I had never used the 7770 for VGA-Passthrough. I'm going to try an unpatched kernel with the HD 7770 later to see if it was truly the patch that allows this card to reset.

Lastly, in regards to the monitor output on the R9 290x, the output will go out of sync and I don't see/hear anything noticeable with the GPU fans. I've tried force closing QEMU and shutting down the guest gracefully and get the same result.

Last edited by DLWood1001 (2014-07-21 11:40:54)

Offline

#2348 2014-07-21 03:41:57

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DLWood1001 wrote:

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

Thanks for the test, both series weren't really targeted at fixing this, but I needed to add a device specific reset for my Oland-based HD8570 to get it to reset on release and I thought it worth a test to see if it fixed anything.  If you kill the qemu process with these patches, what happens on your r9 290x?  Does the monitor stay in sync and the framebuffer show the last image, or does the monitor go blank and the GPU fan reset to high speed?

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Great, these should show up in 3.17

Hi, aw
I tested the patch, but unluckly it still got hangs when I reboot guest OS (Fedora 20 /w GTX480).
And I'm sure that I patched with kernel 3.16-rc5.

Last edited by AKSN74 (2014-07-21 03:42:19)

Offline

#2349 2014-07-21 03:50:00

ctwdoecoss
Member
Registered: 2014-07-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone passthrough gpu successfully on a laptop?

Offline

#2350 2014-07-21 06:42:07

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I had a working Win7 VM using linux-mainline-3.14.1, along with stable qemu 2.0 as well as seabios.
Yesterday I compiled the mainline 3.15.6 package and since that the VM hangs somewhere before boot begins. No output at the monitor.
However using vga=on -vga std the VM boots up as usual.

Is there something that changed in the meantime and whichs needs to be adjusted?

Offline

#2351 2014-07-21 13:38:42

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ctwdoecoss wrote:

Did anyone passthrough gpu successfully on a laptop?

What GPU do you have in your notebook?

Offline

#2352 2014-07-21 14:36:53

ctwdoecoss
Member
Registered: 2014-07-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:
ctwdoecoss wrote:

Did anyone passthrough gpu successfully on a laptop?

What GPU do you have in your notebook?

I.   MSI GP70 2PE
CPU:       I5-4200H
Chipset:   HM86
GPU:       NVIDIA GT840M w/ 2G GDDR3 and  Intel HD 4600
Video Output:  D-sub and HDMI
2.  MSI GE60 2PC
CPU:       I5-4200H
Chipset:   HM87
GPU:      NVIDIA GTX850M w/ 2G GDDR5 and Intel HD 4600
Video Output:  D-sub and HDMI

I would buy one of these two laptop.
I know it may be difficult, for the gpu isn't quadro series.

Offline

#2353 2014-07-21 19:55:05

Hippuh
Member
Registered: 2005-01-30
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well almost got it working like i want but not there yet.

Nvidia GTX760 is the main GPU running arch latest mainline kernel.
With a Radeon 4870 its all working fine.

With a GTX 560ti 448 I managed to get it working once but I can't restart the VM. That gives me an error on the VGA rom (using a rom file doesn't help) or the screen just stays in standby mode.
I tried patching the nvidia drivers as mentioned earlier but that just makes the nvidia installer fail with a build error on nv.c (using patch -p0 < patchfile)

I'm quite sure its related to the nvidia driver. I have to stub the PCI id's to make sure the driver doesn't pick up the second card. I'll keep on trying to get it to work.
If all fails I'll just have to buy me a reasonable AMD card to run the guest on.

PC Spec:
Mainboard: Asus P8H67-V
CPU:         Intel Core I5 2300
Mem:         8GB Geil DDR3
GPU 1:      Asus Nvidia GTX760
GPU 2:      Gainward Nvidia GTX560ti 448 / HIS IceQ+ 4870
Vout:         HDMI on GTX760 / VGA on GTX560 or 4870
Vout2:       VGA on GTX760

Offline

#2354 2014-07-21 21:14:33

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/rishub/windows.img,id=disk,format=raw -device ide-hd,drive=disk -drive file=/home/rishub/Downloads/X17-24395.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Puts me into qemu compat monitor console,
How do I get it to run the ISO?

Offline

#2355 2014-07-21 21:39:00

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I followed the given tutorial step by step.

1) I compiled a new custom kernel
          (3.15.6 +radeon_load_vbios_from_file.patch,
           +i915_315.patch,
           +override_for_missing_acs_capabilities.patch)

2) installed the new kernel

3) added new parameter to grub commandline:

GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 intel_iommu=on vm.ignore_msrs=1 nomodeset pci-stub.ids=1002:9540,1002:aa38  vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

until here everthing looks good:

[    0.432038] pci-stub: add 1002:9540 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.432043] pci-stub 0000:01:00.0: claimed by stub
[    0.432045] pci-stub: add 1002:AA38 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.432049] pci-stub 0000:01:00.1: claimed by stub

4) used the given script for vfio-bind

vfio-bind 0000:01:00.0
vfio-bind 0000:01:00.1

5) tryied to start a test VM buy executing this command:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.0

but this gives me the following error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Anyone got an idea whats wrong with my config?

Here are all my connected PCIe-devices:

00:00.0 Host bridge: Intel Corporation Haswell DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Haswell PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Haswell PCI Express x8 Controller (rev 06)
00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 04)
00:1a.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #2 (rev 04)
00:1c.0 PCI bridge: Intel Corporation Lynx Point PCI Express Root Port #1 (rev d4)
00:1c.1 PCI bridge: Intel Corporation Lynx Point PCI Express Root Port #2 (rev d4)
00:1d.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Lynx Point LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation Lynx Point 6-port SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation Lynx Point SMBus Controller (rev 04)
00:1f.6 Signal processing controller: Intel Corporation Lynx Point Thermal Management Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI RV710 [Radeon HD 4550]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI RV710/730 HDMI Audio [Radeon HD 4000 series]
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)


Thanks in advance!

Offline

#2356 2014-07-22 06:11:17

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

According to the error message you got, you need to ensure that your devices have their own cgroup. This is done by the acs override patch, but its not enough just to apply that patch, you also need to enable it.
append

pcie_acs_override=downstream

to your boot parameters and lets see what happens.

Offline

#2357 2014-07-22 07:36:15

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thx apex8. This boot flag was missing.

now i get this:

qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.0: PCI: slot 0 function 0 not available for vfio-pci, in use by vfio-pci
qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.0: Device 'vfio-pci' could not be initialized

Offline

#2358 2014-07-22 08:09:11

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-device vfio-pci,host=01:00.1,bus=root.1,addr=00.0 <<<<<<<<<<<<<<<<<<<<<<<<<<< wrong address it should be 00.1

Offline

#2359 2014-07-22 08:22:50

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

atleast the VM starts now.
black screen on the terminal, no signal on my hdmi port of the passed through card :-(

QEMU doesnt give me any error messages. Any tips on how I can proceed the debugging? I've read that some cards need to load an "external" bios-rom. Is this worth a try?

Offline

#2360 2014-07-22 11:26:22

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can you try what happens when you adjust your script from above with

-vga std \

instead of

-vga none\

?

Offline

#2361 2014-07-22 15:07:23

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,
I tried to setup the passthrough according to the OP - unfortunally I have not been able to get a signal from my GPU yet.
I am using the patched kernel from the OP with qemu 2.0.0-4 and seabios 1.7.5-2 from the official repo.
I also enabled the kernel parameters needed for Intel graphics

root@Ununhaswellium / # [b]uname -a[/b] Linux Ununhaswellium 3.15.1-1-mainline #1 SMP PREEMPT Tue Jul 15 22:24:38 CEST 2014 x86_64 GNU/Linux root@Ununhaswellium / # [b]cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=6209b873-4a4f-4265-bee6-5b4596d818e0 rw quiet nmi_watchdog=0 intel_iommu=on i915.enable_hd_vgaarb=1 1 root@Ununhaswellium / # lsmod | grep kvm kvm_intel 135528 0 kvm 408519 1 kvm_intel

The hardware ofc supports VT-d:
Intel i7-4771 => IGP for the host
Gigabyte Z87-UD4H
AMD Radeon HD 7950 => for the guest

root@Ununhaswellium / # cat /etc/modprobe.d/blacklist.conf blacklist radeon root@Ununhaswellium / # lspci | grep AMD 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] root@Ununhaswellium / # ls /sys/kernel/iommu_groups/1/devices 0000:00:01.0@ 0000:01:00.0@ 0000:01:00.1@ root@Ununhaswellium / # cat /etc/vfio-pci.cfg DEVICES="0000:00:01.0 0000:01:00.0 0000:01:00.1"

The IOMMU group devices are bound with the systemd service script in the OP.
Using the qemu command line from the OP shows the black QEMU window, but the HD 7950 doesn't output anything.
"-vga std" however at least shows the Seabios output in the QEMU window

root@Ununhaswellium / # dmesg | tail [ 87.327240] VFIO - User Level meta-driver version: 0.3 [ 87.330781] pcie_pme 0000:00:01.0:pcie01: unloading service driver pcie_pme [ 583.320937] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 583.346033] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 583.346037] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 [ 604.697892] [drm:ivb_err_int_handler] *ERROR* Pipe B FIFO underrun

I honestly don't know what I am missing here..? This was likely already discussed somethere, but I skimmed through the thread and couldn't find anything helpful...

Last edited by CharlieBra7o (2014-07-22 15:10:02)

Offline

#2362 2014-07-22 16:23:07

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Can you try what happens when you adjust your script from above with

-vga std \

instead of

-vga none\

?

if i use -vga std ill get videooutput on the terminal window. but the HDMI connected monitor stays black. Im using Seabios 1.7.4 is this a problem? Do I have to upgrade to 1.7.5.
Is there anything I can do to further investigate this issue?

Offline

#2363 2014-07-22 17:57:09

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

At this point I have the same problem as you. I had a stable VM until I upgraded to kernel package 3.15.1 and later to 3.15.6. Using -vga std the VM still boots up. With version 3.14 it worked via DP signal monitor output. I'm still trying to figure out whats different now, but I do not really know where to start.
Does anyone have the archive for the 3.14 package or can I apply the recent patches to the 3.14 vanilla package?

Offline

#2364 2014-07-22 18:22:40

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

This worked for me. That flag seems to be "new", at least for me smile If I remember correctly, the vga arbiter was always on previously.

Last edited by apex8 (2014-07-22 18:23:41)

Offline

#2365 2014-07-22 20:37:10

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
nbhs wrote:

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

This worked for me. That flag seems to be "new", at least for me smile If I remember correctly, the vga arbiter was always on previously.

no success with my configuration...
Any got an idea what to do next?

Offline

#2366 2014-07-23 01:31:44

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

root@debian:/home/rishub# dmesg | tail [ 20.391870] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 20.392637] alx 0000:03:00.0 eth0: NIC Up: 100 Mbps Full [ 20.392845] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 20.410598] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready [ 78.229382] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 96.022961] kvm: zapping shadow pages for mmio generation wraparound [ 213.859145] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to a068 [ 215.665049] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to 2030 [ 318.484445] vfio-pci 0000:01:00.0: Invalid ROM contents [ 361.228915] kvm: zapping shadow pages for mmio generation wraparound

What could invalid ROM content mean? Using VGA -std works, but -none gives me this

Update: tried a ROM file from tech power up for my graphics card (gtx580)
Now I am getting lots of

*ERROR* Unknown unclaimed...

errors in dmesg

Last edited by shrubuntu (2014-07-23 02:12:05)

Offline

#2367 2014-07-23 06:35:36

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

monchi wrote:
apex8 wrote:
nbhs wrote:

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

This worked for me. That flag seems to be "new", at least for me smile If I remember correctly, the vga arbiter was always on previously.

no success with my configuration...
Any got an idea what to do next?

Hard to say where to start, but did you check your hardware for compatibility? Theres an linked Gdrive table document here, where you could look up your configuration.

Offline

#2368 2014-07-23 11:10:51

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

i just registered to the forum only for this topic which is, so far, the best resource on the web regarding this matter!
Thanks for the good job guys, i was able to successfully launch my VM following this topic. smile

I have only 2 problems i am not able to solve, being myself on Debian maybe i should tune better the workarounds, hope you can help me here.

1) KVM signature. I am on Nvidia and i am also facing the 337 driver issue (335 is fine). I would like to patch qemu but can't find the patch itself nor a good guide how to do it.
Is there anything similar on this post or the web? Despite searching i could find nothing. sad

2) that's the tricky one. I am on a Supermicro X10SL7-F mobo, it has an LSI SAS controller onboard which shares the same PCI group with VGA:

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

I would like to assing this controller and VGA to different VMs but it seems this is not possible unless i separate them in different groups.
I am not sure i can do it however, group is correclty reflecing motherboard's hw structure, here is the diagram from supermicro manual: mobo diagram. Hope you can give me some hints about.

My setup:

Intel Xeon 1245v3
16Gb ECC DDR
Supermicro x10sl7-f
MSI GeForce GTX 760 SFF
Host: Debian Sid with kernel 3.15.3 and all VM stuff activated
Guest1 : Openmediavault NAS distro with 2 HDD passd with "RDM", these are at the moment on the intel sata controller as i can't assign the LSI;
Guest2 : Win7 64Bit with working VGA passthrough;

Any help will be really appreciated. smile

Thanks!
Cheers.

Offline

#2369 2014-07-23 11:31:34

blimpi
Member
Registered: 2014-05-22
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@rg80: for point 1), you can compile qemu from git and replace debian's binary with it. For point 2), you need to patch the kernel with the ACS override patch and then boot with kernel parameter "pcie_acs_override=downstream"

Offline

#2370 2014-07-23 12:50:34

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blimpi wrote:

@rg80: for point 1), you can compile qemu from git and replace debian's binary with it. For point 2), you need to patch the kernel with the ACS override patch and then boot with kernel parameter "pcie_acs_override=downstream"

Thanks Blimpi.

Point 1, will check it later.

Point 2, tried to apply it some months ago but got this error:

rino@Hubble:~/kernel/linux-3.15.3$ sudo patch -l -p1 <acs.patch patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2554 with fuzz 1 (offset 205 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

that's char 3292:

+ case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) + return 1;

It's the > in dev->multifunction.

any idea?

Thanks.
Rino.

Offline

#2371 2014-07-23 13:25:12

blimpi
Member
Registered: 2014-05-22
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rg80 wrote:

...

Point 2, tried to apply it some months ago but got this error:

rino@Hubble:~/kernel/linux-3.15.3$ sudo patch -l -p1 <acs.patch patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2554 with fuzz 1 (offset 205 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej patch unexpectedly ends in middle of line

that's char 3292:

+ case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) + return 1;

It's the > in dev->multifunction.

any idea?

Thanks.
Rino.

The error "patch unexpectedly ends in middle of line" might point to a broken acs.patch file. See this link for some ideas about the cause.

Offline

#2372 2014-07-23 13:44:08

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First, great Thanks to nbhs for this Howto, it's was very simple to understand the basics with vfio. (Searched over three Years for a possibility to passthrough my CrossfireX Cards to a VM, with Xen only one Card was possible, or even none).

Now i have passthrough my Soundcard (5.1 Setup), my 2 Graphic Cards (HD7770) (A third one (HD7750) as Primary Card for Linux), my Sata Controller (Linux boots from a Raid-Controller-Card), and different USB-Devices (Passing through the whole Bus makes Problems). The only negative thing is that my third Graphics Card only use PCIe x4 (But thats a thing i can live with).

The Performance is good, i have setup cgroups for Portage (Yes, i am a Gentoo User :-) ) and the VM, because when i compile Packages i don't care how long it takes, but when i play a game in this time, i can't live with less Performance :-) .

I use Windows 8.1 x64 Pro, Catalyst Drivers 14.7 Beta (Windows Part).
Linux has the Linux-next Kernel and the Radeon Kernel driver, and that is the main Reason i post here. Since i have seen that in this Kernel the Amd Iommu v2 is Supported (Don't know if i have one really :-) but that's not important for my Question). Since i only know that the iommu is important for Passthrough, is there any advance of using the v2 Iommu driver (If you have a v2 Iommu on your Board (That is really distracting, because the one site says it's a chip on the Board, the next one talks from the Cpu)) or is there no support for this at the moment ? (I want to change the Board in near Future because of the PCIe x8 and x4 setup and the Probs with Usb3, so i wanted to know if it is important to look that there is iommu v2 Support on it, my Processor (FX8350) should support this).

Offline

#2373 2014-07-23 15:36:42

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blimpi wrote:

The error "patch unexpectedly ends in middle of line" might point to a broken acs.patch file. See this link for some ideas about the cause.

I think i got it, let me post again the solution found some pages behind.

If you have any problem applying original acs override patch from Alex Williamson (Hunk #1 FAILED at 3292 in my case) try downloading this package which contains a new version.

This one applies correctly on my Debian system with kernel 3.15.3
Just built a new kernel image, will test it later and update. smile

Edit: it works! Thanks again to this fantastic thread!

### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 4 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 5 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) ### Group 6 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 7 ### 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) ### Group 8 ### 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ### Group 9 ### 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) ### Group 10 ### 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) ### Group 11 ### 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) ### Group 12 ### 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

Last edited by rg80 (2014-07-23 22:34:00)

Offline

#2374 2014-07-23 17:54:48

Hippuh
Member
Registered: 2005-01-30
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well so far no luck on the NVIDIA card, I can't reboot the guest, when trying to install NVIDIA drivers in windows 8.1 guest it complains an instance is allready active which is not the case.
Does anyone know if there is an nvidia installer for the patched driver (its possible to create it with --apply-patch x.patch) so far on all my attempts I get errors on building the kernel module in nv.c.

Offline

#2375 2014-07-23 18:11:03

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

                                                                                                       
  <os>                                                                                                                 
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>                                                                 
    ...
  </os>                                                                                                                 
  ...
  <devices>                                                                                                             
    <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator>                                                         
    ...
  </devices>
  ...
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunctioIs there n=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/>
    ...
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/>
    ...
  </qemu:commandline>

I tried using these <qemu:*> tags in my virt-manager XML, but no matter if I define a new VM from the XML or edit an existing one, all the qemu tags vanish without an error. This means after I insert them, virsh edit/define, and then reopen the XML file all the qemu tags are gone...
I tried virt-manager and libvirt from both official repo and AUR. No difference sad
Am I missing some setting for qemu integration or why is this happening???

Offline

#2376 2014-07-23 22:08:55

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CharlieBra7o wrote:
kaeptnb wrote:

                                                                                                       
  <os>                                                                                                                 
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>                                                                 
    ...
  </os>                                                                                                                 
  ...
  <devices>                                                                                                             
    <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator>                                                         
    ...
  </devices>
  ...
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunctioIs there n=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/>
    ...
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/>
    ...
  </qemu:commandline>

I tried using these <qemu:*> tags in my virt-manager XML, but no matter if I define a new VM from the XML or edit an existing one, all the qemu tags vanish without an error. This means after I insert them, virsh edit/define, and then reopen the XML file all the qemu tags are gone...
I tried virt-manager and libvirt from both official repo and AUR. No difference sad
Am I missing some setting for qemu integration or why is this happening???

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

This is how the first line should look, otherwise it clears all the <qemu:*> stuff.

Offline

#2377 2014-07-23 22:50:55

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

This is how the first line should look, otherwise it clears all the <qemu:*> stuff.

That did the trick smile Thank you!

Last edited by CharlieBra7o (2014-07-23 22:51:27)

Offline

#2378 2014-07-24 15:23:51

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
monchi wrote:
apex8 wrote:

This worked for me. That flag seems to be "new", at least for me smile If I remember correctly, the vga arbiter was always on previously.

no success with my configuration...
Any got an idea what to do next?

Hard to say where to start, but did you check your hardware for compatibility? Theres an linked Gdrive table document here, where you could look up your configuration.

i could not find the document ...

But i gave it a try with XenServer 6.2.. GPU passthrough worked out of the box (same hardware). So it should be possible to set up passthrough with KVM either right?

Last edited by monchi (2014-07-24 15:32:25)

Offline

#2379 2014-07-24 22:44:14

Nickolai
Member
Registered: 2014-07-24
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's the problem with passing through an Intel VGA card? Is it impassable? I would really like to pass through an Intel card on my ThinkPad X series.

Last edited by Nickolai (2014-07-24 22:44:32)

Offline

#2380 2014-07-25 19:01:48

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nickolai wrote:

What's the problem with passing through an Intel VGA card? Is it impassable? I would really like to pass through an Intel card on my ThinkPad X series.

no, its possible. check xengt

Offline

#2381 2014-07-25 19:10:46

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:
Nickolai wrote:

What's the problem with passing through an Intel VGA card? Is it impassable? I would really like to pass through an Intel card on my ThinkPad X series.

no, its possible. check xengt

Or take a look at SmartOS. While I haven't used it personally, a friend of mine uses it and is very happy with it.


i'm sorry for my poor english wirting skills…

Offline

#2382 2014-07-26 00:31:37

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anyone using Centos 7 for VGA Passthrough ? Do you have a full patch-set for this (kernel 3.10) ... and if so, could you post the patches ....

I'd like to use Centos as a host (for stability) - but wondering about KVM patches for VGA passthrough (including performance enhancements)

Thanks for your help

Offline

#2383 2014-07-26 06:39:43

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
belliash wrote:
Nickolai wrote:

What's the problem with passing through an Intel VGA card? Is it impassable? I would really like to pass through an Intel card on my ThinkPad X series.

no, its possible. check xengt

Or take a look at SmartOS. While I haven't used it personally, a friend of mine uses it and is very happy with it.

Are you serious? SmartOS = ZFS + DTrace + Zones + KVM
KVM doesn't support mediated GPU passthrough. Only XenGT can do that.

Offline

#2384 2014-07-26 14:14:39

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

belliash wrote:
andy123 wrote:

Or take a look at SmartOS. While I haven't used it personally, a friend of mine uses it and is very happy with it.

Are you serious? SmartOS = ZFS + DTrace + Zones + KVM
KVM doesn't support mediated GPU passthrough. Only XenGT can do that.

It's KVM on the OpenIndia (from SunOS/OpenSolaris) kernel and as said I don't use it, because it only works on Intel CPUs. My friend claimed that it did GPU passthrough with his Intel GPU. I didn't verify this (and can't try myself, because I don't have the hardware), but I trust him enough to at least propose it as a possibility.

I'm sorry however, if this can't do passthrough of an Intel GPU.


i'm sorry for my poor english wirting skills…

Offline

#2385 2014-07-26 15:34:37

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I get

 Hunk #1 FAILED at 3292 

when I try to patch the 3.15.6 kernel with the acs patch.

I use this command

 patch -p1 -i ../acs.patch --dry-run 

I am running debian jessie. I have gotten kvm and IOMMU to work on arch linux, but I can't seem to get it on debian. What should I do?


----EDIT----

rg80 wrote:
blimpi wrote:

The error "patch unexpectedly ends in middle of line" might point to a broken acs.patch file. See this link for some ideas about the cause.

I think i got it, let me post again the solution found some pages behind.

If you have any problem applying original acs override patch from Alex Williamson (Hunk #1 FAILED at 3292 in my case) try downloading this package which contains a new version.

This one applies correctly on my Debian system with kernel 3.15.3
Just built a new kernel image, will test it later and update. smile

Edit: it works! Thanks again to this fantastic thread!

### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 4 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 5 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) ### Group 6 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 7 ### 00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) ### Group 8 ### 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ### Group 9 ### 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) ### Group 10 ### 03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) 04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) ### Group 11 ### 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) ### Group 12 ### 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

This post solved my issue!

Last edited by shrubuntu (2014-07-26 15:46:17)

Offline

#2386 2014-07-26 17:47:16

belliash
Member
Registered: 2014-04-25
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
belliash wrote:
andy123 wrote:

Or take a look at SmartOS. While I haven't used it personally, a friend of mine uses it and is very happy with it.

Are you serious? SmartOS = ZFS + DTrace + Zones + KVM
KVM doesn't support mediated GPU passthrough. Only XenGT can do that.

It's KVM on the OpenIndia (from SunOS/OpenSolaris) kernel and as said I don't use it, because it only works on Intel CPUs. My friend claimed that it did GPU passthrough with his Intel GPU. I didn't verify this (and can't try myself, because I don't have the hardware), but I trust him enough to at least propose it as a possibility.

I'm sorry however, if this can't do passthrough of an Intel GPU.

No, it can't. KVM doesnt support such things.

Offline

#2387 2014-07-26 18:02:44

DLWood1001
Member
Registered: 2014-07-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

redger wrote:

Is anyone using Centos 7 for VGA Passthrough ? Do you have a full patch-set for this (kernel 3.10) ... and if so, could you post the patches ....

I'd like to use Centos as a host (for stability) - but wondering about KVM patches for VGA passthrough (including performance enhancements)

Thanks for your help

Hi Redger,

I am using RHEL7 w/3.15.6 kernel.

Needed to download the src rpm files for kernel-ml, Seabios, Qemu, IPXE, EDK2-Tools and built from scratch. Note: The spec files were modded lightly to allow for the latest versions of these projects to compile.

Issues Included:
Kernel, enabled some options in the kernel-config, re-diffed the patches to merge properly. ACS, i915, VGA-ARB, DMA-v4 patches from AW's patch history on patchwork.
Seabios, out of date and some roms were not packaged correctly.
Qemu, Broken links to IPXE directory.
IPXE, Didn't inlcude EFI roms.
EDK2-Tools, required by IPXE to build EFI roms.

Currently, everything is working with VGA passthrough. I have RHEL7 host running both a Win8.1 guest and OS X mavericks guests running simultaneously.

Offline

#2388 2014-07-26 23:15:53

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks @DLWood,
sounds like it was a bit tricky smile was it worthwhile ?
I have refactored the patches for Ubuntu 3.13 kernel and am running that but considered Centos with "native" kernel in the hope it might be more stable / suitable for 24*7 ue (before I start moving other servers across to VMs)

did you consider using the "stock" Centos kernel (v3.10) ?

thanks again for your reply

Offline

#2389 2014-07-27 12:38:52

DLWood1001
Member
Registered: 2014-07-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi @redger,

I would say it is worth it for me smile, I've learned a lot in the process.

I did consider using the 3.10 kernel, but had to upgrade due to some unrelated hardware limitations that are resolved in latest 3.11+ mainline releases.

I believe my hardware problem will be resolved in RHEL 7.1, so I may try using the stock RHEL7 kernel after that is released. The stock RHEL7/CentOS7 kernel config doesn't have CONFIG_VFIO_PCI_VGA enabled, so you would need to maintain your own build either way.

Offline

#2390 2014-07-27 14:31:12

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After I booted up my freshly installed Windows 8.1 for the first time it worked for about 60 seconds or so, then the monitor lost its signal. From this moment on the monitor always loses its signal during the Windows loading screen. The VM's CPU load from this moment on seems to be completely steady using up 1 full thread.

My guess is that Windows tried to install some crappy GPU/chipset drivers or apply some wrong settings...

Does anyone have an idea what might be going on there?


Additional info:
- Radeon HD 7950
- 3.15.6 from the OP unchanged
- qemu-git 2.1.r34391.gf368c33-1 + libvirt 1.2.6-1 from GIT

Last edited by CharlieBra7o (2014-07-27 14:37:12)

Offline

#2391 2014-07-27 14:52:19

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright I have got it mostly working except for a few snags.

I get an "Code 43" on the nvidia driver within my windows 7 guest and

I cannot restart the VM once I have shut it down without restarting my computer

Is there any fix for these?

Offline

#2392 2014-07-27 15:14:15

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shrubuntu wrote:

Alright I have got it mostly working except for a few snags.

I get an "Code 43" on the nvidia driver within my windows 7 guest and

I cannot restart the VM once I have shut it down without restarting my computer

Is there any fix for these?

@Code 43: read https://bbs.archlinux.org/viewtopic.php … 0#p1426160 and following post(s)
@Reboot: This issue definitely has been discussed from page 1 on. I think you need the vga reset patch that aw posted a few pages before, but Im not sure...

Offline

#2393 2014-07-27 16:21:22

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hello it works for me with problem

I get messages

Jul 27 20:00:01 DeltaS systemd: Starting Session 5 of user root.
Jul 27 20:00:01 DeltaS systemd: Started Session 5 of user root.
Jul 27 20:01:01 DeltaS systemd: Starting Session 6 of user root.
Jul 27 20:01:01 DeltaS systemd: Started Session 6 of user root.
Jul 27 20:01:01 DeltaS systemd: Starting Session 7 of user sanek.
Jul 27 20:01:01 DeltaS systemd: Started Session 7 of user sanek.
Jul 27 20:10:01 DeltaS systemd: Starting Session 8 of user root.
Jul 27 20:10:01 DeltaS systemd: Started Session 8 of user root


And I get BSOD in some minutes

Offline

#2394 2014-07-27 20:43:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zzz3000 wrote:

hello it works for me with problem

I get messages

Jul 27 20:00:01 DeltaS systemd: Starting Session 5 of user root.
Jul 27 20:00:01 DeltaS systemd: Started Session 5 of user root.
Jul 27 20:01:01 DeltaS systemd: Starting Session 6 of user root.
Jul 27 20:01:01 DeltaS systemd: Started Session 6 of user root.
Jul 27 20:01:01 DeltaS systemd: Starting Session 7 of user sanek.
Jul 27 20:01:01 DeltaS systemd: Started Session 7 of user sanek.
Jul 27 20:10:01 DeltaS systemd: Starting Session 8 of user root.
Jul 27 20:10:01 DeltaS systemd: Started Session 8 of user root


And I get BSOD in some minutes

You don't really expect that someone can give you useful advice with this tiny bit of information, do you?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2395 2014-07-27 21:44:07

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CharlieBra7o wrote:
shrubuntu wrote:

Alright I have got it mostly working except for a few snags.

I get an "Code 43" on the nvidia driver within my windows 7 guest and

I cannot restart the VM once I have shut it down without restarting my computer

Is there any fix for these?

@Code 43: read https://bbs.archlinux.org/viewtopic.php … 0#p1426160 and following post(s)
@Reboot: This issue definitely has been discussed from page 1 on. I think you need the vga reset patch that aw posted a few pages before, but Im not sure...

Awesome, reverting to 335 drivers fixed the issue!
Everything running smoothly

My system :

Debian Jessie with custom 3.15.6 kernel with patches from OP
i7 4790k
MSI gaming 5 motherboard
GTX 580

@aw what is the status of VGA reset patches?

Offline

#2396 2014-07-27 22:02:57

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump gpu bios, use it in your config and u should be able to reset...

Offline

#2397 2014-07-27 22:05:20

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Dump gpu bios, use it in your config and u should be able to reset...

Interesting, can you describe this a bit more?

Offline

#2398 2014-07-28 00:07:50

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you are dual booting Windows, just use GPU-Z to dump your GPU BIOS into a file. Then you'll have VGA reset from QEMU.

Offline

#2399 2014-07-28 12:42:29

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CharlieBra7o wrote:

After I booted up my freshly installed Windows 8.1 for the first time it worked for about 60 seconds or so, then the monitor lost its signal. From this moment on the monitor always loses its signal during the Windows loading screen. The VM's CPU load from this moment on seems to be completely steady using up 1 full thread.

My guess is that Windows tried to install some crappy GPU/chipset drivers or apply some wrong settings...

Does anyone have an idea what might be going on there?


Additional info:
- Radeon HD 7950
- 3.15.6 from the OP unchanged
- qemu-git 2.1.r34391.gf368c33-1 + libvirt 1.2.6-1 from GIT

Just tried installing Windows 7 and after the installation I spent about 5 minutes clicking around, then shutdown the VM to make an image and started it up again. Now it always freezes during the "Starting Windows" screen and 'virsh list' lists the VM as 'paused'.
dmesg doesn't show anything peculiar, except maybe

kvm: zapping shadow pages for mmio generation wraparound

Honestly, I have absolutely no clue what to try next...

Offline

#2400 2014-07-28 17:44:17

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CharlieBra7o wrote:
CharlieBra7o wrote:

After I booted up my freshly installed Windows 8.1 for the first time it worked for about 60 seconds or so, then the monitor lost its signal. From this moment on the monitor always loses its signal during the Windows loading screen. The VM's CPU load from this moment on seems to be completely steady using up 1 full thread.

My guess is that Windows tried to install some crappy GPU/chipset drivers or apply some wrong settings...

Does anyone have an idea what might be going on there?


Additional info:
- Radeon HD 7950
- 3.15.6 from the OP unchanged
- qemu-git 2.1.r34391.gf368c33-1 + libvirt 1.2.6-1 from GIT

Just tried installing Windows 7 and after the installation I spent about 5 minutes clicking around, then shutdown the VM to make an image and started it up again. Now it always freezes during the "Starting Windows" screen and 'virsh list' lists the VM as 'paused'.
dmesg doesn't show anything peculiar, except maybe

kvm: zapping shadow pages for mmio generation wraparound

Honestly, I have absolutely no clue what to try next...

virsh tells you the reason, why vm was paused. I had similar issue (I/O Error), because of vm's disks being raw file images on btrfs (and having really low performance).

Last edited by dwe11er (2014-07-29 08:42:10)

Offline

#2401 2014-07-29 03:34:55

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:

If you are dual booting Windows, just use GPU-Z to dump your GPU BIOS into a file. Then you'll have VGA reset from QEMU.

you can dump it aslo from virtual machine wink (sometimes it doesn't work at first but it does after couple of tries/gpu-z versions...)

Offline

#2402 2014-07-29 07:59:54

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there still an issue with the nouveau driver on the host, i am currently runnin the patched nvidia driver but i would like to switch for the open source one on my host.

Anyone running nouveau on host?

Offline

#2403 2014-07-29 09:08:03

Chousuke
Member
Registered: 2014-07-29
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'd like to chime in here with a success report. I'm not an archlinux user, but I have my Radeon R9 270 successfully passed through to a kvm Windows 7 guest on Fedora 20.

The thing is, I have no idea what actually made it work. I tried all sorts of things, but in the end I reinstalled and created the virtual machine purely with virt-manager (from Fedora virt-preview repos) and just selected the GPU (only) as pass-through from the GUI. My version of virt-manager seems to use vfio by default. I did have to manually modprobe vfio-pci etc. though, and of course ensure that the radeon driver wasn't being used... X runs using the IGP and I see no graphics corruption or other issues even after repeated reboots of the VM. 3d acceleration also seems to work on the host.

I have an emulated Cirrus VGA chip on the VM too, but it stops working as soon as the guest boots up properly. I used it to install the Catalyst drivers (for some reason, with QXL the installer crashed)

The full virt-manager XML is here:
http://pastebin.com/ZX0Kfex5


My software versions are as follows:

- Kernel 3.15.6-200.fc20.x86_64 custom built with vga arbitration patches (81b5c7bc8de3e6f63419139c2fc91bf81dea8a7d, 6e1b4fdad5157bb9e88777d525704aba24389bee) though I am not sure if they're actually needed... I see no indication of it. Maybe because I do EFI boot?
- virt-manager 1.0.1
- qemu 2.0.93

and hardware:
Intel(R) Core(TM) i5-4670 CPU @ 3.40GHz
AMD Radeon R9 270
ASUS H87 Pro

Offline

#2404 2014-07-29 12:40:48

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chousuke wrote:

I'd like to chime in here with a success report. I'm not an archlinux user, but I have my Radeon R9 270 successfully passed through to a kvm Windows 7 guest on Fedora 20.

The thing is, I have no idea what actually made it work. I tried all sorts of things, but in the end I reinstalled and created the virtual machine purely with virt-manager (from Fedora virt-preview repos) and just selected the GPU (only) as pass-through from the GUI. My version of virt-manager seems to use vfio by default. I did have to manually modprobe vfio-pci etc. though, and of course ensure that the radeon driver wasn't being used... X runs using the IGP and I see no graphics corruption or other issues even after repeated reboots of the VM. 3d acceleration also seems to work on the host.

I have an emulated Cirrus VGA chip on the VM too, but it stops working as soon as the guest boots up properly. I used it to install the Catalyst drivers (for some reason, with QXL the installer crashed)

The full virt-manager XML is here:
http://pastebin.com/ZX0Kfex5


My software versions are as follows:

- Kernel 3.15.6-200.fc20.x86_64 custom built with vga arbitration patches (81b5c7bc8de3e6f63419139c2fc91bf81dea8a7d, 6e1b4fdad5157bb9e88777d525704aba24389bee) though I am not sure if they're actually needed... I see no indication of it. Maybe because I do EFI boot?
- virt-manager 1.0.1
- qemu 2.0.93

and hardware:
Intel(R) Core(TM) i5-4670 CPU @ 3.40GHz
AMD Radeon R9 270
ASUS H87 Pro

It's because you're not using vfio-pci, but normal pci-assign.

Offline

#2405 2014-07-29 13:30:17

Chousuke
Member
Registered: 2014-07-29
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

It's because you're not using vfio-pci, but normal pci-assign.

I am quite sure I'm using vfio, since the guest refuses to start without the vfio modules loaded:

Error starting domain: internal error: process exited while connecting to monitor: pulseaudio: pa_context_connect() failed pulseaudio: Reason: Connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver 2014-07-29T13:28:42.126919Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=0: vfio: No available IOMMU models 2014-07-29T13:28:42.126941Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=0: vfio: failed to setup container for group 1 2014-07-29T13:28:42.126951Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=0: vfio: failed to get group 1 2014-07-29T13:28:42.126963Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=0: Device initialization failed. 2014-07-29T13:28:42.126975Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x9,rombar=0: Device 'vfio-pci' could not be initialized

EDIT: Curiously enough, this setup also works just fine with a completely unmodified 3.15.4-200.fc20.x86_64 Fedora kernel.
EDIT2: It is entirely possible that the virt-manager and qemu packages I am using have some special modifications which enable this to function. I just now discovered that the qemu binary seems to lie about its version... the package I have installed comes from the fedora-virt-preview repository and yum tells me its version is:

Version : 2.1.0 Release : 0.5.rc3.fc20

Last edited by Chousuke (2014-07-29 13:46:05)

Offline

#2406 2014-07-30 17:22:48

chillum
Member
Registered: 2014-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

what do I do if I have exactly the same times 2 wireless keyboard/ mouse; ID bit is the same

...
Bus 002 Device 006: ID 1a81:1006 Holtek Semiconductor, Inc.
...
Bus 011 Device 004: ID 1a81:1006 Holtek Semiconductor, Inc.
...
how do I tell KVM/windows which one to passthrough?

Offline

#2407 2014-07-30 17:26:38

chillum
Member
Registered: 2014-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I know

-usbdevice host:1a81:1006

Offline

#2408 2014-07-30 18:14:40

vogets
Member
Registered: 2006-11-25
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbirkis wrote:

Anyone running nouveau on host?

A 9500 GT with nouveau runs smoothly for me. Unpatched.

Offline

#2409 2014-07-31 10:12:32

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
CharlieBra7o wrote:
CharlieBra7o wrote:

After I booted up my freshly installed Windows 8.1 for the first time it worked for about 60 seconds or so, then the monitor lost its signal. From this moment on the monitor always loses its signal during the Windows loading screen. The VM's CPU load from this moment on seems to be completely steady using up 1 full thread.

My guess is that Windows tried to install some crappy GPU/chipset drivers or apply some wrong settings...

Does anyone have an idea what might be going on there?


Additional info:
- Radeon HD 7950
- 3.15.6 from the OP unchanged
- qemu-git 2.1.r34391.gf368c33-1 + libvirt 1.2.6-1 from GIT

Just tried installing Windows 7 and after the installation I spent about 5 minutes clicking around, then shutdown the VM to make an image and started it up again. Now it always freezes during the "Starting Windows" screen and 'virsh list' lists the VM as 'paused'.
dmesg doesn't show anything peculiar, except maybe

kvm: zapping shadow pages for mmio generation wraparound

Honestly, I have absolutely no clue what to try next...

virsh tells you the reason, why vm was paused. I had similar issue (I/O Error), because of vm's disks being raw file images on btrfs (and having really low performance).

Could you please tell me where it states the reason and how you fixed your problem?
That's my disk...

Device Start End Size Type ... /dev/sda7 167774208 330248191 77.5G Linux filesystem

It worked perfectly fine with Win 8.1 without GPU passthrough and the first time I booted Win7 with GPU passthrough

 <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/disk/by-partuuid/771af6c5-d5db-421d-9f67-8070092961bf'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </disk>

EDIT:
I just stumbled upon this post:

aw wrote:
andy123 wrote:

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

Numerous folks have reported BSODs which were solved by moving the audio device to the root complex rather than exposing it as a sub-function.  It may work, but exposing it on pcie.0 has had more success it seems.

So what I did is just remove the Radeon HD audio device qemu command line and now Win 7 seems so be content...

Last edited by CharlieBra7o (2014-08-01 23:45:11)

Offline

#2410 2014-08-01 21:23:37

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VFIO passthrough works flawlessly on Ubuntu 14.04 with no patches just yet but I have a few questions if someone could please help.

1) How to shutdown/forceshutdown/kill the VM started with qemu over SSH?

2) What are the parameters to make VM connect to the physical network (to eth0 on host)?

3) Is it possible to see "display output" over SSH? The host is a minimal virtual machine host with only CLI and no GUI.

4) Whats the best way to autostart 3 VMs with host, ofcourse each with its own vfio/config?

Thanks!

Last edited by devianceluka (2014-08-01 21:35:23)

Offline

#2411 2014-08-01 22:22:12

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone managed to run osx? is kvm better than xen ?

Offline

#2412 2014-08-01 22:28:32

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

osx works great, virtio network drivers crash though, better off with e1000, have to put it on high pcie bus addr though or osx won't see it.

Offline

#2413 2014-08-02 14:39:56

penetal
Member
Registered: 2014-06-15
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can not seem to get audio working right. No matter what I do I end up with crackling sound.
If I set QEMU_PA_SAMPLES to 128 its way way worse.
This affects all sound, not just the VM. If I have something playing on the host once I start the VM the sound gets all fucky.

I have no idea if it could possibly help, but here is the pulseauido output for when I start the VM, and then kill it almost right away (made sure the sound playing on the host had gone bad before I stopped it.

I: [pulseaudio] client.c: Created 3 "Native client (UNIX socket client)" I: [pulseaudio] protocol-native.c: Client authenticated anonymously. I: [pulseaudio] module-stream-restore.c: Restoring volume for sink input sink-input-by-application-name:/run/user/1000/pulse/native. I: [pulseaudio] module-stream-restore.c: Restoring mute state for sink input sink-input-by-application-name:/run/user/1000/pulse/native. I: [pulseaudio] sink-input.c: Created input 3 "qemu" on alsa_output.pci-0000_00_1b.0.analog-stereo with sample spec s16le 2ch 44100Hz and channel map front-left,front-right I: [pulseaudio] sink-input.c: media.name = "qemu" I: [pulseaudio] sink-input.c: application.name = "/run/user/1000/pulse/native" I: [pulseaudio] sink-input.c: native-protocol.peer = "UNIX socket client" I: [pulseaudio] sink-input.c: native-protocol.version = "29" I: [pulseaudio] sink-input.c: application.process.id = "7409" I: [pulseaudio] sink-input.c: application.process.user = "root" I: [pulseaudio] sink-input.c: application.process.host = "ArcherBase" I: [pulseaudio] sink-input.c: application.process.binary = "qemu-system-x86_64" I: [pulseaudio] sink-input.c: application.language = "C" I: [pulseaudio] sink-input.c: window.x11.display = ":0" I: [pulseaudio] sink-input.c: application.process.machine_id = "81157a0d6cb84ce9b5fd22006cf55017" I: [pulseaudio] sink-input.c: module-stream-restore.id = "sink-input-by-application-name:/run/user/1000/pulse/native" I: [pulseaudio] protocol-native.c: Requested tlength=10.00 ms, minreq=4.99 ms I: [pulseaudio] protocol-native.c: Final latency 10.98 ms = 0.50 ms + 2*4.99 ms + 0.50 ms I: [alsa-source-ALC1150 Analog] alsa-source.c: Trying resume... I: [alsa-source-ALC1150 Analog] alsa-util.c: Trying to disable ALSA period wakeups, using timers only I: [alsa-source-ALC1150 Analog] alsa-util.c: ALSA period wakeups disabled I: [alsa-source-ALC1150 Analog] alsa-source.c: Time scheduling watermark is 20.00ms I: [alsa-source-ALC1150 Analog] alsa-source.c: Resumed successfully... I: [alsa-source-ALC1150 Analog] alsa-source.c: Starting capture. I: [pulseaudio] source-output.c: Created output 1 "qemu" on alsa_input.pci-0000_00_1b.0.analog-stereo with sample spec s16le 2ch 44100Hz and channel map front-left,front-right I: [pulseaudio] source-output.c: media.name = "qemu" I: [pulseaudio] source-output.c: application.name = "/run/user/1000/pulse/native" I: [pulseaudio] source-output.c: native-protocol.peer = "UNIX socket client" I: [pulseaudio] source-output.c: native-protocol.version = "29" I: [pulseaudio] source-output.c: application.process.id = "7409" I: [pulseaudio] source-output.c: application.process.user = "root" I: [pulseaudio] source-output.c: application.process.host = "ArcherBase" I: [pulseaudio] source-output.c: application.process.binary = "qemu-system-x86_64" I: [pulseaudio] source-output.c: application.language = "C" I: [pulseaudio] source-output.c: window.x11.display = ":0" I: [pulseaudio] source-output.c: application.process.machine_id = "81157a0d6cb84ce9b5fd22006cf55017" I: [pulseaudio] source-output.c: module-stream-restore.id = "source-output-by-application-name:/run/user/1000/pulse/native" I: [pulseaudio] protocol-native.c: Final latency 2000.00 ms = 1000.00 ms + 1000.00 ms I: [alsa-sink-ALC1150 Analog] alsa-sink.c: Underrun! I: [alsa-sink-ALC1150 Analog] alsa-sink.c: Increasing minimal latency to 1.00 ms I: [pulseaudio] source-output.c: Freeing output 1 "qemu" I: [pulseaudio] sink-input.c: Freeing input 3 "qemu" I: [pulseaudio] client.c: Freed 3 "/run/user/1000/pulse/native" I: [pulseaudio] protocol-native.c: Connection died.

When I kill the VM sound goes back to normal and the audio playing on the host is fine.

Script to start the VM

 1 #!/bin/sh 2 3 vfio-bind 0000:01:00.0 0000:01:00.1 4 5 export QEMU_PA_SAMPLES=2048 6 export QEMU_PA_SERVER=/run/user/1000/pulse/native 7 8 qemu-system-x86_64 -enable-kvm -M q35 -m 10240 -cpu host \ 9 -smp 6,sockets=1,cores=3,threads=2 \ 10 -bios /usr/share/qemu/bios.bin \ 11 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ 12 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ 13 -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ 14 -drive id=system,file=/home/thomas/vm/GamingWindows.img,format=raw \ 15 -device ide-hd,bus=ide.0,drive=system \ 16 -vga none \ 17 -monitor stdio \ 18 -drive id=storage,file=/home/thomas/vm/GamingWindowsStorage.qcow2,format=qcow2 \ 19 -device ide-hd,bus=ide.1,drive=storage \ 20 -usb -device usb-host,hostbus=1,hostaddr=7 \ 21 -usb -device usb-host,hostbus=1,hostaddr=14 \ 22 -usb -usbdevice host:046d:c52b \ 23 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ 24 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Offline

#2414 2014-08-02 17:51:48

valkaiser
Member
Registered: 2014-08-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all, great thread.
I have successfully passed through an nVidia 7900GS to a windows XP SP3 guest.
Host hardware: ASUS P9X79 WS, i7 4930K, nVidia 7900GS x2, ATI X300 x2.
Host is running Arch with stock kernel, qemu, and seabios.
Guest is using emulated IDE for HDD as I had trouble getting XP to run, but I'm still fiddling with it.
QEMU invocation script:

#!/bin/bash modprobe vfio-pci vfio-bind 0000:04:00.0 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/media/Data3/OS/windows.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \ -drive file=virtio-win-0.1-81.iso,id=isocd2 -device ide-cd,bus=piix4-ide.1,drive=isocd2 \ -usb -usbdevice host:05ac:0201 -usbdevice host:046d:c03d \ -boot once=d \ -no-acpi \ -nographic

Anyways, what has me stumped currently is the nVidia drivers on the guest. I installed the latest available for my 7900, 307.83, and the install goes smoothly. However, upon rebooting, XP does not use the driver.
The device status screen gives me this:
Your computer's system firmware does not include enough information to properly configure and use this device. To use this device, contact your computer manufacturer to obtain a firmware or BIOS update. [CODE 35]
Are there other drivers for the guest that I needed to install first?

Any ideas?

Oh, and thanks for pooling all this information here. It has been enormously helpful.

UPDATE: I get exactly the same results when passing through an ATI X300.
UPDATE: Now using linux-mainline+patches from OP. No change.

Last edited by valkaiser (2014-08-03 05:13:13)

Offline

#2415 2014-08-02 18:40:12

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again. smile

Do you have any idea how to add -cpu kvm=off option into a virt-manager xml?
I already tried with qemu arg:

 <qemu:arg value='-cpu'/> <qemu:arg value='kvm=off'/>

but it does not work.

Should i put it somewhere into CPU options?

 <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu>

Thanks. smile

Offline

#2416 2014-08-02 23:32:04

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm in the process of assembling a new rig primarily for gaming and would run Win 8.1 as VM, I've had good experiences with this on another setup as HTPC. I understand that GPU performance should be close to native, how does the CPU side of the equation look in terms of overhead? Is it worth getting the i7 4790K over the i5 4690K, the latter being one third cheaper (close to 100 euro difference here).

I would think that in case of using Steam in-home streaming the i7 would be better due to not having nVidia or Intel QuickSync available for hardware assisted h.264 encoding (using AMD and impossible to KVM passthrough the iGPU). If I disregard that scenario it looks less worth it. Is that correct? Thanks.

Offline

#2417 2014-08-03 07:55:58

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

I'm in the process of assembling a new rig primarily for gaming and would run Win 8.1 as VM, I've had good experiences with this on another setup as HTPC. I understand that GPU performance should be close to native, how does the CPU side of the equation look in terms of overhead? Is it worth getting the i7 4790K over the i5 4690K, the latter being one third cheaper (close to 100 euro difference here).

I would think that in case of using Steam in-home streaming the i7 would be better due to not having nVidia or Intel QuickSync available for hardware assisted h.264 encoding (using AMD and impossible to KVM passthrough the iGPU). If I disregard that scenario it looks less worth it. Is that correct? Thanks.

i7 is much better... 2x more vcpu-s..u won't have good streaming performance in software mode... but valve said they are working on AMD VCE and no ETA.

Last edited by slis (2014-08-03 07:56:31)

Offline

#2418 2014-08-03 16:43:27

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Having some trouble with sound. I am using the command in the op, but the sound in windows is filled with alot of static. When I switch to usb audio, it works, but I prefer to keep it analog.

Any fix?

Offline

#2419 2014-08-03 16:45:13

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shrubuntu wrote:

Having some trouble with sound. I am using the command in the op, but the sound in windows is filled with alot of static. When I switch to usb audio, it works, but I prefer to keep it analog.

Any fix?

I had this issue on Windows 8 and tried using different cards to emulate with no fix. Switched to Windows 7 and using ac97 (-sound ac97) works really well.

Offline

#2420 2014-08-03 16:49:16

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:
shrubuntu wrote:

Having some trouble with sound. I am using the command in the op, but the sound in windows is filled with alot of static. When I switch to usb audio, it works, but I prefer to keep it analog.

Any fix?

I had this issue on Windows 8 and tried using different cards to emulate with no fix. Switched to Windows 7 and using ac97 (-sound ac97) works really well.

-device ac97,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

like this?

Offline

#2421 2014-08-03 16:53:03

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shrubuntu wrote:
kameloc wrote:
shrubuntu wrote:

Having some trouble with sound. I am using the command in the op, but the sound in windows is filled with alot of static. When I switch to usb audio, it works, but I prefer to keep it analog.

Any fix?

I had this issue on Windows 8 and tried using different cards to emulate with no fix. Switched to Windows 7 and using ac97 (-sound ac97) works really well.

-device ac97,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

like this?

Oh sorry I don't passthrough my onboard audio, just share it. If you add

-soundhw ac97

It emulates Realtek AC'97 for the guest. It allowed me to use my speakers from the VM or host, and at the same time

Offline

#2422 2014-08-03 22:56:22

zerotri
Member
Registered: 2014-08-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Been having trouble getting this all setup on my system. Tried with both Arch and Fedora 20 with similar results.
Motherboard is an Asus P9X79 Pro, cpu is an i7-3820, both support VT-d.

When I start my host it seems to stay in low resolution efifb mode. It also never reaches a login screen, just stays at service startup output. I have to switch to a login using Ctrl + Alt + F2. I've tried going through quite a few of the pages in this thread and am not seeing anyone getting similar results.

dmesg | grep -e DMAR -e IOMMU:

[root@Heimdall ~]# dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x00000000BE4C32B0 0000B4 (v01 A M I OEMDMAR 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.166673] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0462 ecap f020fe [ 0.166765] IOAPIC id 0 under DRHD base 0xfbffc000 IOMMU 0 [ 0.166766] IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 0 [ 0.771339] IOMMU 0 0xfbffc000: using Queued invalidation [ 0.771341] IOMMU: Setting RMRR: [ 0.771352] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbea9a000 - 0xbeaa8fff] [ 0.771375] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbea9a000 - 0xbeaa8fff] [ 0.771388] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.771395] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 1.231728] dmar: DMAR:[DMA Read] Request device [09:00.1] fault addr fffe0000 DMAR:[fault reason 02] Present bit in context entry is clear

Command line:

BOOT_IMAGE=/vmlinuz-3.15.7-200.fc20.x86_64 root=/dev/mapper/fedora-root ro vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root intel_iommu=on pci-stub.ids=1002:68ba,1002:aa58 pcie_acs_override=downstream rhgb quiet LANG=en_US.UTF-8

lspci -v (cut out just the GPUs):

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 6770] (prog-if 00 [VGA controller]) Subsystem: XFX Pine Group Inc. Device 3156 Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at fbe20000 (64-bit, non-prefetchable) [size=128K] I/O ports at e000 [size=256] Expansion ROM at fbe00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] Subsystem: XFX Pine Group Inc. Device aa58 Flags: bus master, fast devsel, latency 0, IRQ 10 Memory at fbe40000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 6770] (prog-if 00 [VGA controller]) Subsystem: XFX Pine Group Inc. Device 3156 Flags: fast devsel, IRQ 11 Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at fbd20000 (64-bit, non-prefetchable) [disabled] [size=128K] I/O ports at d000 [disabled] [size=256] Expansion ROM at fbd00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: pci-stub Kernel modules: radeon 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] Subsystem: XFX Pine Group Inc. Device aa58 Flags: fast devsel, IRQ 10 Memory at fbd40000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: pci-stub Kernel modules: snd_hda_intel

ls -al /dev/vfio/:

[root@Heimdall vfio]# ls -al /dev/vfio total 0 drwxr-xr-x. 2 root root 80 Aug 3 15:45 . drwxr-xr-x. 21 root root 3420 Aug 3 15:45 .. crw-------. 1 root root 246, 0 Aug 3 15:45 15 crw-rw-rw-. 1 root root 10, 196 Aug 3 15:45 vfio

I'm out of ideas as to what it could be. I've tried with various command line options, tried swapping around which graphics card is bound as well as tried binding both.
Is there anything that is glaringly obvious to someone who has this working that I'm just completely missing here? Could it be due to me using two of the same graphics card?


And on a much less important matter for aw, I've been reading up on the dma-alias patch because I have a Marvell 88SE9128 sata controller that causes boot to take several minutes with intel_iommu=on. It is my understanding that this patch has been rejected from mainline so I'm likely out of luck. Would you happen to have any suggestions for how to deal with this?

Thank you for any and all help you can provide,
- Wynter Woods

Offline

#2423 2014-08-03 23:22:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zerotri wrote:

And on a much less important matter for aw, I've been reading up on the dma-alias patch because I have a Marvell 88SE9128 sata controller that causes boot to take several minutes with intel_iommu=on. It is my understanding that this patch has been rejected from mainline so I'm likely out of luck. Would you happen to have any suggestions for how to deal with this?

The latest attempt will come together in v3.17.  I assume that's what's causing your DMAR fault.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2424 2014-08-04 08:22:21

atmouse
Member
Registered: 2013-06-21
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have two video cards, HIS HD7870 x2. The hardware ID,vender just the same.

How could I use vga-passthrough one of them to qemu-kvm?  is it that setting pci-stub.ids to the same id block both of the gpu? working or not?

CPU is AMD-FX8350, no gpu integrated.

Last edited by atmouse (2014-08-04 13:56:52)

Offline

#2425 2014-08-04 15:11:35

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rg80 wrote:

Hi again. smile

Do you have any idea how to add -cpu kvm=off option into a virt-manager xml?
I already tried with qemu arg:

 <qemu:arg value='-cpu'/> <qemu:arg value='kvm=off'/>

but it does not work.

Should i put it somewhere into CPU options?

 <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu>

Thanks. smile


Bumping in a different format.
I want to disable the KVM Signture to update my nvidia windows drivers to any newer than v335.

Then got latest GIT version of qemu, applied this patch, then config, make, make install, reboot.

Applied:

 <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/>

but qemu reports .kvm is not an available option for qemu-system-x86_64.

On the contrary:

 <qemu:arg value='-cpu'/> <qemu:arg value='host,hv-time'/>

works.

Am i doing something wrong?

Note:
For some reason, the original patch posted on page 1 gives me a format error on:

diff --git a/target-i386/cpu.c b/target-i386/cpu.c

it's not a blank, i can't understand where the problem is...

Thanks! smile

Offline

#2426 2014-08-04 15:42:49

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rg80 wrote:
rg80 wrote:

Hi again. smile

Do you have any idea how to add -cpu kvm=off option into a virt-manager xml?
I already tried with qemu arg:

 <qemu:arg value='-cpu'/> <qemu:arg value='kvm=off'/>

but it does not work.

Should i put it somewhere into CPU options?

 <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu>

Thanks. smile


Bumping in a different format.
I want to disable the KVM Signture to update my nvidia windows drivers to any newer than v335.

Then got latest GIT version of qemu, applied this patch, then config, make, make install, reboot.

Applied:

 <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/>

but qemu reports .kvm is not an available option for qemu-system-x86_64.

On the contrary:

 <qemu:arg value='-cpu'/> <qemu:arg value='host,hv-time'/>

works.

Am i doing something wrong?

Note:
For some reason, the original patch posted on page 1 gives me a format error on:

diff --git a/target-i386/cpu.c b/target-i386/cpu.c

it's not a blank, i can't understand where the problem is...

Thanks! smile

I belive qemu from git has this patch already applied.

Offline

#2427 2014-08-04 15:46:41

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I belive qemu from git has this patch already applied.

Thanks, I believe the same but what is the -cpu option then?
I also tried without patch but it does not work on its own and -cpu kvm=off gives the same error with libvirt. sad

Offline

#2428 2014-08-04 19:17:49

rg80
Member
Registered: 2014-07-23
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Got it! smile

virsh XML was pointing to /user/bin/qemu while git qemu installs into /usr/local/bin/qemu  !!!

Changed the path in the XML, added:

 <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/>

Now i am on nvidia with windows driver 340! smile

And yes, qemu git already contains the "-cpu kvm" option, no need to apply any patch!

Offline

#2429 2014-08-04 22:27:39

HPvirt
Member
Registered: 2014-05-23
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello!

Previously, I've checked if some hardware would be able to do this kind of virtualization.
Now, I have my hardware (check my signature for details).

I started yesterday to prepare my linux for virt. (kernel compilation, grub command line,...). I've tried with kernel 3.15.6 with the 3 patches from #1. Without success: no /etc/vfio/x, qemu would not start.

And tonight, I've succeeded to start Seven SP1 boot CD! I used kernel 3.15.8 sources, apply patches, build, install, and it works.
First step done.

I've tried to read all posts. Sorting all these infos (obsolete one, AMD specific, etc.) to apply them to my specific case is quite time consuming, but it's very usefull. However, as I'm more linux user than administrator, I think I've missed some part, and there's some things I do not understand fully.

So, before going into the real seven installation, I've some questions:
- I do not use pci-stub: as I understand the first post, it must be used only if we cannot blacklist GPU drivers (as with 2 NVidia or 2 AMD). Right?
- I want to install Seven on a SSD, so I would like to have AHCI and trim working for Windows. Between solutions on #1 for giving access to disk drives, which one is recommanded?
- Moving windows under Linux is... smooth... too smooth, producing great latency: my cursor is already stopped and the window is still moving to overtake the cursor. Is the disabling of DRI the cause? If I boot from liveCD, I have the same behavior.
- In first post, I do not understand the part "Systemd service:". I think I must add a file containing following text to do the vfio bindings at start time. But I would like to be sure... Can someone confirm please?

Some info:

$ grep VFIO /boot/config-3.15.8 CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y $ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.15.8 root=UUID=a98c98e2-53a6-4771-a465-e5dd7128ae4e ro quiet splash nvidia.blacklist=yes nvidia-current.blacklist=yes nvidia_337.blacklist=yes nvidiafb.blacklist=yes i915.enable_hd_vgaarb=1 intel_iommu=on vt.handoff=7

For the cmdline, I followed post #2308 as rafalcieslak have the same hardware.

I do not use (cf. #2308):

options kvm allow_unsafe_assigned_interrupts=1

nor (not yet):

pcie_acs_override=downstream

About this last parameter, is it a requisite? I've check groups, and if I fully give ASMedia USB & Sata controllers and Realtek Network controller to Seven, as they share the same group, it would not be necessary. Or, more likely, I missed something...

(lsgroup is aw script in #722)

$ ./lsgroup.sh ### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation Device 8cb1 ### Group 5 ### 00:16.0 Communication controller: Intel Corporation Device 8cba ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V ### Group 7 ### 00:1a.0 USB controller: Intel Corporation Device 8cad ### Group 8 ### 00:1b.0 Audio device: Intel Corporation Device 8ca0 ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation Device 8c90 (rev d0) 00:1c.2 PCI bridge: Intel Corporation Device 8c94 (rev d0) 00:1c.3 PCI bridge: Intel Corporation Device 8c96 (rev d0) 00:1c.6 PCI bridge: Intel Corporation Device 8c9c (rev d0) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 0a:00.0 USB controller: ASMedia Technology Inc. Device 1142 ### Group 10 ### 00:1d.0 USB controller: Intel Corporation Device 8ca6 ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Device 8cc4 00:1f.2 SATA controller: Intel Corporation Device 8c82 00:1f.3 SMBus: Intel Corporation Device 8ca2

Thank you all for sharing your knowledge.
I hope I'm understandable as my english is not pretty good...

Last edited by HPvirt (2014-08-06 20:36:00)


Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

Offline

#2430 2014-08-05 05:38:52

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you must use pci-stub if u have 2 same cards, for host and guest.

maybe you need option downstream if u have 2 gpu-s (maybe even if u don't, depending on mbo)...

you need to bind device before u start vm, with systemd service or by hand (vfio-bind 0000:01:00.0 0000:01:00.1, assumin u created vfio-bind script as in first post)

not sure about window lag,  i don't use gui on host.

about ssd... i am not sure, but i think u need to pass whole sata controller to use trim etc...

virtio-disk is best option if u ask me wink

edit:
check my posts for configs and help...

Last edited by slis (2014-08-05 05:54:17)

Offline

#2431 2014-08-05 06:29:29

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

are the patches from #1 still needed for the recent stabel kernel version 3.16 ?

Offline

#2432 2014-08-05 08:01:54

zerotri
Member
Registered: 2014-08-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I seem to have made a bit of progress on a few fronts but still have a ways to go.

I ended up not using pci-stub at all. This may or may not be causing problems. lspci -v shows that my first hd6770 is bound to radeon while my second is bound to vfio-pci. Host's graphics are working 100% but when I start up a windows guest I am getting the infamous "video_tdr_failure / atikmpag.sys" BSoD. I've tried playing around with moving the HDMI audio device to a few other pci locations, as well as not passing it through at all, with no luck. Maybe I'm doing something wrong in the process of debugging that.

Also, I was able to turn off the DMAR error associated with my Marvell controller by completely disabling the device in my bios. Now my system boots up pretty quickly, just need to iron out the vga passthrough issues with the vm.

I guess this could be contributing to the issue?
Gonna have to dig around in dmesg a bit more to see if I'm missing anything else.

[ 4.894003] VFIO - User Level meta-driver version: 0.3 [ 4.897769] scsi 8:0:0:0: Direct-Access SanDisk Cruzer Fit 1.22 PQ: 0 ANSI: 5 [ 4.898516] sd 8:0:0:0: [sdc] 15633408 512-byte logical blocks: (8.00 GB/7.45 GiB) [ 4.899512] sd 8:0:0:0: [sdc] Write Protect is off [ 4.899515] sd 8:0:0:0: [sdc] Mode Sense: 43 00 00 00 [ 4.899873] sd 8:0:0:0: Attached scsi generic sg2 type 0 [ 4.900960] sd 8:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 4.908768] ------------[ cut here ]------------ [ 4.908774] WARNING: CPU: 5 PID: 956 at fs/sysfs/group.c:219 sysfs_remove_group+0x93/0xa0() [ 4.908776] sysfs group ffffffff81cb8dc0 not found for kobject 'i2c-21' [ 4.908777] Modules linked in: vfio_iommu_type1 vfio_pci vfio joydev uas usb_storage hid_logitech_dj iTCO_wdt iTCO_vendor_support eeepc_wmi asus_wmi sparse_keymap video mxm_wmi vfat fat x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_realtek snd_hda_codec_generic microcode snd_hda_codec_hdmi serio_raw btusb sb_edac edac_core bluetooth snd_hda_intel snd_hda_controller snd_hda_codec i2c_i801 rfkill snd_hwdep snd_seq snd_seq_device lpc_ich mfd_core snd_pcm snd_timer mei_me snd mei soundcore shpchp wmi nfsd auth_rpcgss nfs_acl lockd sunrpc radeon i2c_algo_bit drm_kms_helper e1000e ttm drm ptp i2c_core pps_core [ 4.908811] CPU: 5 PID: 956 Comm: vfio-bind Not tainted 3.15.7-200.fc20.x86_64 #1 [ 4.908813] Hardware name: System manufacturer System Product Name/P9X79 PRO, BIOS 1203 05/24/2012 [ 4.908814] 0000000000000000 000000002e9d5df5 ffff8808078abba0 ffffffff816ef92e [ 4.908816] ffff8808078abbe8 ffff8808078abbd8 ffffffff8108933d 0000000000000000 [ 4.908818] ffffffff81cb8dc0 ffff88080eef2480 ffff88080fedb000 ffff88080eef24c8 [ 4.908821] Call Trace: [ 4.908825] [<ffffffff816ef92e>] dump_stack+0x45/0x56 [ 4.908827] [<ffffffff8108933d>] warn_slowpath_common+0x7d/0xa0 [ 4.908829] [<ffffffff810893bc>] warn_slowpath_fmt+0x5c/0x80 [ 4.908832] [<ffffffff81260918>] ? kernfs_find_and_get_ns+0x48/0x60 [ 4.908834] [<ffffffff81263fc3>] sysfs_remove_group+0x93/0xa0 [ 4.908837] [<ffffffff81465197>] dpm_sysfs_remove+0x57/0x60 [ 4.908840] [<ffffffff8145a1f5>] device_del+0x45/0x1c0 [ 4.908841] [<ffffffff8145a38e>] device_unregister+0x1e/0x60 [ 4.908848] [<ffffffffa000cd2c>] i2c_del_adapter+0x1bc/0x2f0 [i2c_core] [ 4.908852] [<ffffffffa0093412>] drm_dp_aux_unregister_i2c_bus+0x12/0x20 [drm_kms_helper] [ 4.908867] [<ffffffffa01812fc>] radeon_i2c_destroy+0x3c/0x40 [radeon] [ 4.908878] [<ffffffffa018137a>] radeon_i2c_fini+0x2a/0x50 [radeon] [ 4.908890] [<ffffffffa017e6d9>] radeon_modeset_fini+0x89/0x90 [radeon] [ 4.908899] [<ffffffffa0157180>] radeon_driver_unload_kms+0x40/0x60 [radeon] [ 4.908908] [<ffffffffa00497a9>] drm_dev_unregister+0x29/0xb0 [drm] [ 4.908915] [<ffffffffa0049d13>] drm_put_dev+0x23/0x70 [drm] [ 4.908923] [<ffffffffa015327d>] radeon_pci_remove+0x1d/0x20 [radeon] [ 4.908927] [<ffffffff8138fb7b>] pci_device_remove+0x3b/0xb0 [ 4.908930] [<ffffffff8145e48f>] __device_release_driver+0x7f/0xf0 [ 4.908932] [<ffffffff8145e523>] device_release_driver+0x23/0x30 [ 4.908935] [<ffffffff8145d06d>] unbind_store+0xbd/0xe0 [ 4.908937] [<ffffffff8145c574>] drv_attr_store+0x24/0x40 [ 4.908939] [<ffffffff81262c1d>] sysfs_kf_write+0x3d/0x50 [ 4.908941] [<ffffffff812623e0>] kernfs_fop_write+0xe0/0x160 [ 4.908944] [<ffffffff811e67da>] vfs_write+0xba/0x1e0 [ 4.908946] [<ffffffff811e7395>] SyS_write+0x55/0xd0 [ 4.908950] [<ffffffff816ff969>] system_call_fastpath+0x16/0x1b [ 4.908951] ---[ end trace 4a930f0eb6428dc9 ]--- [ 4.908985] ------------[ cut here ]------------ [ 4.908988] WARNING: CPU: 5 PID: 956 at fs/sysfs/group.c:219 sysfs_remove_group+0x93/0xa0() [ 4.908989] sysfs group ffffffff81cb8dc0 not found for kobject 'i2c-24' [ 4.908989] Modules linked in: vfio_iommu_type1 vfio_pci vfio joydev uas usb_storage hid_logitech_dj iTCO_wdt iTCO_vendor_support eeepc_wmi asus_wmi sparse_keymap video mxm_wmi vfat fat x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_realtek snd_hda_codec_generic microcode snd_hda_codec_hdmi serio_raw btusb sb_edac edac_core bluetooth snd_hda_intel snd_hda_controller snd_hda_codec i2c_i801 rfkill snd_hwdep snd_seq snd_seq_device lpc_ich mfd_core snd_pcm snd_timer mei_me snd mei soundcore shpchp wmi nfsd auth_rpcgss nfs_acl lockd sunrpc radeon i2c_algo_bit drm_kms_helper e1000e ttm drm ptp i2c_core pps_core [ 4.909015] CPU: 5 PID: 956 Comm: vfio-bind Tainted: G W 3.15.7-200.fc20.x86_64 #1 [ 4.909016] Hardware name: System manufacturer System Product Name/P9X79 PRO, BIOS 1203 05/24/2012 [ 4.909017] 0000000000000000 000000002e9d5df5 ffff8808078abba0 ffffffff816ef92e [ 4.909019] ffff8808078abbe8 ffff8808078abbd8 ffffffff8108933d 0000000000000000 [ 4.909021] ffffffff81cb8dc0 ffff88080eef2c80 ffff88080fedf000 ffff88080eef2cc8 [ 4.909023] Call Trace: [ 4.909025] [<ffffffff816ef92e>] dump_stack+0x45/0x56 [ 4.909027] [<ffffffff8108933d>] warn_slowpath_common+0x7d/0xa0 [ 4.909029] [<ffffffff810893bc>] warn_slowpath_fmt+0x5c/0x80 [ 4.909031] [<ffffffff81260918>] ? kernfs_find_and_get_ns+0x48/0x60 [ 4.909033] [<ffffffff81263fc3>] sysfs_remove_group+0x93/0xa0 [ 4.909035] [<ffffffff81465197>] dpm_sysfs_remove+0x57/0x60 [ 4.909037] [<ffffffff8145a1f5>] device_del+0x45/0x1c0 [ 4.909039] [<ffffffff8145a38e>] device_unregister+0x1e/0x60 [ 4.909044] [<ffffffffa000cd2c>] i2c_del_adapter+0x1bc/0x2f0 [i2c_core] [ 4.909048] [<ffffffffa0093412>] drm_dp_aux_unregister_i2c_bus+0x12/0x20 [drm_kms_helper] [ 4.909060] [<ffffffffa01812fc>] radeon_i2c_destroy+0x3c/0x40 [radeon] [ 4.909071] [<ffffffffa018137a>] radeon_i2c_fini+0x2a/0x50 [radeon] [ 4.909082] [<ffffffffa017e6d9>] radeon_modeset_fini+0x89/0x90 [radeon] [ 4.909091] [<ffffffffa0157180>] radeon_driver_unload_kms+0x40/0x60 [radeon] [ 4.909098] [<ffffffffa00497a9>] drm_dev_unregister+0x29/0xb0 [drm] [ 4.909105] [<ffffffffa0049d13>] drm_put_dev+0x23/0x70 [drm] [ 4.909114] [<ffffffffa015327d>] radeon_pci_remove+0x1d/0x20 [radeon] [ 4.909116] [<ffffffff8138fb7b>] pci_device_remove+0x3b/0xb0 [ 4.909119] [<ffffffff8145e48f>] __device_release_driver+0x7f/0xf0 [ 4.909122] [<ffffffff8145e523>] device_release_driver+0x23/0x30 [ 4.909124] [<ffffffff8145d06d>] unbind_store+0xbd/0xe0 [ 4.909126] [<ffffffff8145c574>] drv_attr_store+0x24/0x40 [ 4.909128] [<ffffffff81262c1d>] sysfs_kf_write+0x3d/0x50 [ 4.909130] [<ffffffff812623e0>] kernfs_fop_write+0xe0/0x160 [ 4.909133] [<ffffffff811e67da>] vfs_write+0xba/0x1e0 [ 4.909135] [<ffffffff811e7395>] SyS_write+0x55/0xd0 [ 4.909137] [<ffffffff816ff969>] system_call_fastpath+0x16/0x1b [ 4.909139] ---[ end trace 4a930f0eb6428dca ]--- [ 4.909163] ------------[ cut here ]------------ [ 4.909166] WARNING: CPU: 5 PID: 956 at fs/sysfs/group.c:219 sysfs_remove_group+0x93/0xa0() [ 4.909167] sysfs group ffffffff81cb8dc0 not found for kobject 'i2c-22' [ 4.909168] Modules linked in: vfio_iommu_type1 vfio_pci vfio joydev uas usb_storage hid_logitech_dj iTCO_wdt iTCO_vendor_support eeepc_wmi asus_wmi sparse_keymap video mxm_wmi vfat fat x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_realtek snd_hda_codec_generic microcode snd_hda_codec_hdmi serio_raw btusb sb_edac edac_core bluetooth snd_hda_intel snd_hda_controller snd_hda_codec i2c_i801 rfkill snd_hwdep snd_seq snd_seq_device lpc_ich mfd_core snd_pcm snd_timer mei_me snd mei soundcore shpchp wmi nfsd auth_rpcgss nfs_acl lockd sunrpc radeon i2c_algo_bit drm_kms_helper e1000e ttm drm ptp i2c_core pps_core [ 4.909193] CPU: 5 PID: 956 Comm: vfio-bind Tainted: G W 3.15.7-200.fc20.x86_64 #1 [ 4.909195] Hardware name: System manufacturer System Product Name/P9X79 PRO, BIOS 1203 05/24/2012 [ 4.909195] 0000000000000000 000000002e9d5df5 ffff8808078abba0 ffffffff816ef92e [ 4.909197] ffff8808078abbe8 ffff8808078abbd8 ffffffff8108933d 0000000000000000 [ 4.909200] ffffffff81cb8dc0 ffff88080eef3480 ffff88080fedc000 ffff88080eef34c8 [ 4.909202] Call Trace: [ 4.909204] [<ffffffff816ef92e>] dump_stack+0x45/0x56 [ 4.909206] [<ffffffff8108933d>] warn_slowpath_common+0x7d/0xa0 [ 4.909208] [<ffffffff810893bc>] warn_slowpath_fmt+0x5c/0x80 [ 4.909210] [<ffffffff81260918>] ? kernfs_find_and_get_ns+0x48/0x60 [ 4.909212] [<ffffffff81263fc3>] sysfs_remove_group+0x93/0xa0 [ 4.909214] [<ffffffff81465197>] dpm_sysfs_remove+0x57/0x60 [ 4.909216] [<ffffffff8145a1f5>] device_del+0x45/0x1c0 [ 4.909218] [<ffffffff8145a38e>] device_unregister+0x1e/0x60 [ 4.909223] [<ffffffffa000cd2c>] i2c_del_adapter+0x1bc/0x2f0 [i2c_core] [ 4.909228] [<ffffffffa0093412>] drm_dp_aux_unregister_i2c_bus+0x12/0x20 [drm_kms_helper] [ 4.909241] [<ffffffffa01812fc>] radeon_i2c_destroy+0x3c/0x40 [radeon] [ 4.909253] [<ffffffffa018137a>] radeon_i2c_fini+0x2a/0x50 [radeon] [ 4.909264] [<ffffffffa017e6d9>] radeon_modeset_fini+0x89/0x90 [radeon] [ 4.909273] [<ffffffffa0157180>] radeon_driver_unload_kms+0x40/0x60 [radeon] [ 4.909281] [<ffffffffa00497a9>] drm_dev_unregister+0x29/0xb0 [drm] [ 4.909288] [<ffffffffa0049d13>] drm_put_dev+0x23/0x70 [drm] [ 4.909296] [<ffffffffa015327d>] radeon_pci_remove+0x1d/0x20 [radeon] [ 4.909299] [<ffffffff8138fb7b>] pci_device_remove+0x3b/0xb0 [ 4.909303] [<ffffffff8145e48f>] __device_release_driver+0x7f/0xf0 [ 4.909305] [<ffffffff8145e523>] device_release_driver+0x23/0x30 [ 4.909307] [<ffffffff8145d06d>] unbind_store+0xbd/0xe0 [ 4.909309] [<ffffffff8145c574>] drv_attr_store+0x24/0x40 [ 4.909312] [<ffffffff81262c1d>] sysfs_kf_write+0x3d/0x50 [ 4.909314] [<ffffffff812623e0>] kernfs_fop_write+0xe0/0x160 [ 4.909316] [<ffffffff811e67da>] vfs_write+0xba/0x1e0 [ 4.909318] [<ffffffff811e7395>] SyS_write+0x55/0xd0 [ 4.909321] [<ffffffff816ff969>] system_call_fastpath+0x16/0x1b [ 4.909323] ---[ end trace 4a930f0eb6428dcb ]--- [ 4.909353] ------------[ cut here ]------------ [ 4.909356] WARNING: CPU: 5 PID: 956 at fs/sysfs/group.c:219 sysfs_remove_group+0x93/0xa0() [ 4.909357] sysfs group ffffffff81cb8dc0 not found for kobject 'i2c-23' [ 4.909358] Modules linked in: vfio_iommu_type1 vfio_pci vfio joydev uas usb_storage hid_logitech_dj iTCO_wdt iTCO_vendor_support eeepc_wmi asus_wmi sparse_keymap video mxm_wmi vfat fat x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_realtek snd_hda_codec_generic microcode snd_hda_codec_hdmi serio_raw btusb sb_edac edac_core bluetooth snd_hda_intel snd_hda_controller snd_hda_codec i2c_i801 rfkill snd_hwdep snd_seq snd_seq_device lpc_ich mfd_core snd_pcm snd_timer mei_me snd mei soundcore shpchp wmi nfsd auth_rpcgss nfs_acl lockd sunrpc radeon i2c_algo_bit drm_kms_helper e1000e ttm drm ptp i2c_core pps_core [ 4.909385] CPU: 5 PID: 956 Comm: vfio-bind Tainted: G W 3.15.7-200.fc20.x86_64 #1 [ 4.909386] Hardware name: System manufacturer System Product Name/P9X79 PRO, BIOS 1203 05/24/2012 [ 4.909387] 0000000000000000 000000002e9d5df5 ffff8808078abba0 ffffffff816ef92e [ 4.909389] ffff8808078abbe8 ffff8808078abbd8 ffffffff8108933d 0000000000000000 [ 4.909391] ffffffff81cb8dc0 ffff88080eef3c80 ffff88080fede800 ffff88080eef3cc8 [ 4.909394] Call Trace: [ 4.909396] [<ffffffff816ef92e>] dump_stack+0x45/0x56 [ 4.909398] [<ffffffff8108933d>] warn_slowpath_common+0x7d/0xa0 [ 4.909400] [<ffffffff810893bc>] warn_slowpath_fmt+0x5c/0x80 [ 4.909403] [<ffffffff816f526b>] ? mutex_unlock+0x1b/0x20 [ 4.909406] [<ffffffff81260918>] ? kernfs_find_and_get_ns+0x48/0x60 [ 4.909408] [<ffffffff81263fc3>] sysfs_remove_group+0x93/0xa0 [ 4.909410] [<ffffffff81465197>] dpm_sysfs_remove+0x57/0x60 [ 4.909412] [<ffffffff8145a1f5>] device_del+0x45/0x1c0 [ 4.909414] [<ffffffff8145a38e>] device_unregister+0x1e/0x60 [ 4.909419] [<ffffffffa000cd2c>] i2c_del_adapter+0x1bc/0x2f0 [i2c_core] [ 4.909423] [<ffffffffa0093412>] drm_dp_aux_unregister_i2c_bus+0x12/0x20 [drm_kms_helper] [ 4.909435] [<ffffffffa01812fc>] radeon_i2c_destroy+0x3c/0x40 [radeon] [ 4.909446] [<ffffffffa018137a>] radeon_i2c_fini+0x2a/0x50 [radeon] [ 4.909457] [<ffffffffa017e6d9>] radeon_modeset_fini+0x89/0x90 [radeon] [ 4.909466] [<ffffffffa0157180>] radeon_driver_unload_kms+0x40/0x60 [radeon] [ 4.909479] [<ffffffffa00497a9>] drm_dev_unregister+0x29/0xb0 [drm] [ 4.909486] [<ffffffffa0049d13>] drm_put_dev+0x23/0x70 [drm] [ 4.909494] [<ffffffffa015327d>] radeon_pci_remove+0x1d/0x20 [radeon] [ 4.909497] [<ffffffff8138fb7b>] pci_device_remove+0x3b/0xb0 [ 4.909500] [<ffffffff8145e48f>] __device_release_driver+0x7f/0xf0 [ 4.909502] [<ffffffff8145e523>] device_release_driver+0x23/0x30 [ 4.909504] [<ffffffff8145d06d>] unbind_store+0xbd/0xe0 [ 4.909506] [<ffffffff8145c574>] drv_attr_store+0x24/0x40 [ 4.909509] [<ffffffff81262c1d>] sysfs_kf_write+0x3d/0x50 [ 4.909511] [<ffffffff812623e0>] kernfs_fop_write+0xe0/0x160 [ 4.909513] [<ffffffff811e67da>] vfs_write+0xba/0x1e0 [ 4.909515] [<ffffffff811e7395>] SyS_write+0x55/0xd0 [ 4.909517] [<ffffffff816ff969>] system_call_fastpath+0x16/0x1b [ 4.909519] ---[ end trace 4a930f0eb6428dcc ]--- [ 4.909550] ------------[ cut here ]------------ [ 4.909552] WARNING: CPU: 5 PID: 956 at fs/sysfs/group.c:219 sysfs_remove_group+0x93/0xa0() [ 4.909554] sysfs group ffffffff81cb8dc0 not found for kobject 'i2c-25' [ 4.909554] Modules linked in: vfio_iommu_type1 vfio_pci vfio joydev uas usb_storage hid_logitech_dj iTCO_wdt iTCO_vendor_support eeepc_wmi asus_wmi sparse_keymap video mxm_wmi vfat fat x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_codec_realtek snd_hda_codec_generic microcode snd_hda_codec_hdmi serio_raw btusb sb_edac edac_core bluetooth snd_hda_intel snd_hda_controller snd_hda_codec i2c_i801 rfkill snd_hwdep snd_seq snd_seq_device lpc_ich mfd_core snd_pcm snd_timer mei_me snd mei soundcore shpchp wmi nfsd auth_rpcgss nfs_acl lockd sunrpc radeon i2c_algo_bit drm_kms_helper e1000e ttm drm ptp i2c_core pps_core [ 4.909580] CPU: 5 PID: 956 Comm: vfio-bind Tainted: G W 3.15.7-200.fc20.x86_64 #1 [ 4.909581] Hardware name: System manufacturer System Product Name/P9X79 PRO, BIOS 1203 05/24/2012 [ 4.909582] 0000000000000000 000000002e9d5df5 ffff8808078abba0 ffffffff816ef92e [ 4.909584] ffff8808078abbe8 ffff8808078abbd8 ffffffff8108933d 0000000000000000 [ 4.909585] ffffffff81cb8dc0 ffff88080eef4480 ffff88080fedf400 ffff88080eef44c8 [ 4.909587] Call Trace: [ 4.909590] [<ffffffff816ef92e>] dump_stack+0x45/0x56 [ 4.909591] [<ffffffff8108933d>] warn_slowpath_common+0x7d/0xa0 [ 4.909593] [<ffffffff810893bc>] warn_slowpath_fmt+0x5c/0x80 [ 4.909595] [<ffffffff81260918>] ? kernfs_find_and_get_ns+0x48/0x60 [ 4.909597] [<ffffffff81263fc3>] sysfs_remove_group+0x93/0xa0 [ 4.909599] [<ffffffff81465197>] dpm_sysfs_remove+0x57/0x60 [ 4.909601] [<ffffffff8145a1f5>] device_del+0x45/0x1c0 [ 4.909603] [<ffffffff8145a38e>] device_unregister+0x1e/0x60 [ 4.909608] [<ffffffffa000cd2c>] i2c_del_adapter+0x1bc/0x2f0 [i2c_core] [ 4.909612] [<ffffffffa0093412>] drm_dp_aux_unregister_i2c_bus+0x12/0x20 [drm_kms_helper] [ 4.909623] [<ffffffffa01812fc>] radeon_i2c_destroy+0x3c/0x40 [radeon] [ 4.909634] [<ffffffffa018137a>] radeon_i2c_fini+0x2a/0x50 [radeon] [ 4.909645] [<ffffffffa017e6d9>] radeon_modeset_fini+0x89/0x90 [radeon] [ 4.909654] [<ffffffffa0157180>] radeon_driver_unload_kms+0x40/0x60 [radeon] [ 4.909661] [<ffffffffa00497a9>] drm_dev_unregister+0x29/0xb0 [drm] [ 4.909667] [<ffffffffa0049d13>] drm_put_dev+0x23/0x70 [drm] [ 4.909675] [<ffffffffa015327d>] radeon_pci_remove+0x1d/0x20 [radeon] [ 4.909678] [<ffffffff8138fb7b>] pci_device_remove+0x3b/0xb0 [ 4.909681] [<ffffffff8145e48f>] __device_release_driver+0x7f/0xf0 [ 4.909683] [<ffffffff8145e523>] device_release_driver+0x23/0x30 [ 4.909685] [<ffffffff8145d06d>] unbind_store+0xbd/0xe0 [ 4.909687] [<ffffffff8145c574>] drv_attr_store+0x24/0x40 [ 4.909689] [<ffffffff81262c1d>] sysfs_kf_write+0x3d/0x50 [ 4.909691] [<ffffffff812623e0>] kernfs_fop_write+0xe0/0x160 [ 4.909693] [<ffffffff811e67da>] vfs_write+0xba/0x1e0 [ 4.909696] [<ffffffff811e7395>] SyS_write+0x55/0xd0 [ 4.909698] [<ffffffff816ff969>] system_call_fastpath+0x16/0x1b [ 4.909700] ---[ end trace 4a930f0eb6428dcd ]--- [ 4.909742] [drm] radeon: finishing device.

EDIT: Well, I feel like a damn fool. Got it working by applying the arbiter patches to the kernel. When I had checked the patches out I had wrongly assumed they were both intel graphics only patches and so weren't necessary. This doesn't seem to be the case with the vga_arbiter_fix though. Windows 8 running with graphics passthrough working beautifully. Going to play with graphics drivers a bit, get it updated to 8.1 and see what kinds of issues I am running into then, if any. Thanks for all of your hard work everyone!

Last edited by zerotri (2014-08-05 10:44:58)

Offline

#2433 2014-08-05 20:06:58

HPvirt
Member
Registered: 2014-05-23
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

you need to bind device before u start vm, with systemd service or by hand (vfio-bind 0000:01:00.0 0000:01:00.1, assumin u created vfio-bind script as in first post)

I know this and I use vfio-bind script ... except the italic part. I know this question is off topic, so I've done some research.
This is what I found:
- Create the file (VFIOBindings.service for example) into /etc/systemd/system
- systemctl start VFIOBindings.service
- systemctl enable VFIOBindings.service
And that's all.
But I would be sure to avoid messing startup.


I progress... I assigned the ASMedia SATA controller with success.
The SSD is recognized by the live CD, and SMART works. Great!

Tried the ASMedia USB controller too with some succes: at VM boot time, the USB does not work, it's only when the live CD launch the keyboard and the mouse lights. Is there something wrong or is it normal?
This is my command line to launch qemu:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp sockets=1,cores=2,threads=2 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=07:00.0,bus=pcie.0 \ -device vfio-pci,host=09:00.0,bus=pcie.0 \ [b]-device vfio-pci,host=0a:00.0,bus=pcie.0[/b] \ -device ahci,id=ahci0 \ -boot menu=on,oder=d \ -drive file=~/Linux/Mint/64/linuxmint-17-cinnamon-64bit-v2.iso,id=isocd \ -device ide-cd,bus=ide.1,drive=isocd

(bold is ASMedia USB Controller).


And now, the network part... I tried to passthrough the Realtek Ethernet Controller...
lspci gives me:

... 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) ...

So, vfio-bind 0000:03:00.0 then launch qemu with: -device vfio-pci,host=03:00.0,bus=pcie.0 -net none
And it works. I have the Realtek under liveCD and this one.


So, everything seems to work.
Second step done.

For the moment,     options kvm allow_unsafe_assigned_interrupts=1 and pcie_acs_override=downstream are not used.

And now the last one: installing Windows!


Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

Offline

#2434 2014-08-06 00:09:06

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

monchi wrote:

are the patches from #1 still needed for the recent stabel kernel version 3.16 ?

I'm using 3.16 RC7 and didn't have to patch the Intel stuff... Not sure about the other patches though.

Offline

#2435 2014-08-06 23:52:03

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:
shrubuntu wrote:
kameloc wrote:

I had this issue on Windows 8 and tried using different cards to emulate with no fix. Switched to Windows 7 and using ac97 (-sound ac97) works really well.

-device ac97,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

like this?

Oh sorry I don't passthrough my onboard audio, just share it. If you add

-soundhw ac97

It emulates Realtek AC'97 for the guest. It allowed me to use my speakers from the VM or host, and at the same time

Can you show me the whole command? I can't seem to get it to work

Offline

#2436 2014-08-07 00:48:22

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shrubuntu wrote:
kameloc wrote:
shrubuntu wrote:
-device ac97,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

like this?

Oh sorry I don't passthrough my onboard audio, just share it. If you add

-soundhw ac97

It emulates Realtek AC'97 for the guest. It allowed me to use my speakers from the VM or host, and at the same time

Can you show me the whole command? I can't seem to get it to work

This is how I start my VM. Only passthrough my GPU and a USB device. Emulate ac97 at the end.

sudo -E qemu-system-x86_64 -enable-kvm \
-M q35 \
-cpu host \
-smp 4,cores=4,threads=8,sockets=1 \
-m 6122 \
-rtc base=localtime,clock=host \
-balloon virtio \
-boot menu=on \
-bios /usr/share/qemu/bios.bin \
-vga none \
-usbdevice host:0d8c:0008 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,romfile=/home/kameloc/windows7/EVGA.GTX550Ti.1024.121031.rom,multifunction=on,x-vga=on \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/home/kameloc/windows7/windows7.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
-net nic,macaddr=DE:AD:BE:EF:00:00 \
-net bridge,br=br0 \
-soundhw ac97

Offline

#2437 2014-08-07 08:21:41

HPvirt
Member
Registered: 2014-05-23
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

about ssd... i am not sure, but i think u need to pass whole sata controller to use trim etc...

With whole sata controller passed, I can start Win7 installation (copying file, uncompress), but at reboot time to finish installation, QEmu/SeaBios do not see the SSD anymore so cannot boot on it. If I try the installation from scratch once more, the installer see the SSD.

This is the qemu commande I used:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp sockets=1,cores=2,threads=2 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=03:00.0,bus=pcie.0 \ -device vfio-pci,host=07:00.0,bus=pcie.0 \ -device vfio-pci,host=09:00.0,bus=pcie.0 \ -device vfio-pci,host=0a:00.0,bus=pcie.0 \ -boot menu=on,oder=d \ -drive file=~/Win7SP1.iso,id=isocd \ -device ide-cd,bus=ide.1,drive=isocd \ -net none

I have to add:

-usb \ -usbdevice host=xxxx:xxxx \ -usbdevice host=xxxx:xxxx \

for mouse and keyboard as ASMedia USB controller does not seem initialized by seabios (or qemu) at boot time, so cannot use them during Win7 installation.

So, I have 2 problems that I need help:
- USB initialization at boot
- SSD no more visible by qemu/seabios after first phase of Win7 installation

I've googled each problem, but I've not found anything valuable sad


Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

Offline

#2438 2014-08-07 18:19:35

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

HPvirt wrote:
slis wrote:

about ssd... i am not sure, but i think u need to pass whole sata controller to use trim etc...

With whole sata controller passed, I can start Win7 installation (copying file, uncompress), but at reboot time to finish installation, QEmu/SeaBios do not see the SSD anymore so cannot boot on it. If I try the installation from scratch once more, the installer see the SSD.

This is the qemu commande I used:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp sockets=1,cores=2,threads=2 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=03:00.0,bus=pcie.0 \ -device vfio-pci,host=07:00.0,bus=pcie.0 \ -device vfio-pci,host=09:00.0,bus=pcie.0 \ -device vfio-pci,host=0a:00.0,bus=pcie.0 \ -boot menu=on,oder=d \ -drive file=~/Win7SP1.iso,id=isocd \ -device ide-cd,bus=ide.1,drive=isocd \ -net none

I have to add:

-usb \ -usbdevice host=xxxx:xxxx \ -usbdevice host=xxxx:xxxx \

for mouse and keyboard as ASMedia USB controller does not seem initialized by seabios (or qemu) at boot time, so cannot use them during Win7 installation.

So, I have 2 problems that I need help:
- USB initialization at boot
- SSD no more visible by qemu/seabios after first phase of Win7 installation

I've googled each problem, but I've not found anything valuable :(

maybe "-no-reboot" qemu parameter may serve you as a workaround?

Offline

#2439 2014-08-08 15:07:42

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, I was updated to qemu 2.1 on Gentoo and get input grabbing problems like in this thread

But I can't understand, how to fix it? Can I do this, or I will downgrade to qemu 2.0?

Thx.

Offline

#2440 2014-08-08 18:39:44

irtimmer
Member
Registered: 2013-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I never know it was possible to grab the mouse and keyboard with -vga none in Qemu 2.0 as it also don't work in Qemu < 2.0. But a simple fix is to add
"-device qxl"


I can read and write in English, but my english grammar is just horrible.

Offline

#2441 2014-08-09 11:31:34

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have a huge problem, no idea how to fix it, really need help as it makes a game I'm playing unplayable.

Final Fantasy XIV (an MMO) disconnects every hour. After spending ~8 hours trying to figure out why, I've narrowed it down to the cause but can't find a solution.

Every hour, the Windows VM's clock drifts to 3-4 minutes behind the real time. Windows detects this and fixes it. All applications that use the system clock then freak out as time just jumped forward 3-4 minutes, including FFXIV which causes you to disconnect.

The system time has changed to 2014-08-09T11:23:48.500000000Z from 2014-08-09T11:22:27.765200000Z.

This is the message shown in Windows Event Viewer, pointing out the change. I've tried adding two things to qemu to fix it.

-rtc clock=vm,base=localtime,driftfix=slew
-rtc base=utc,driftfix=slew

Neither of these have worked. The first one has worked the best, and has the drift down to 1 minute per hour, but that is still unacceptable and causes a disconnect. My computers RTC (in BIOS) is set to UTC time, and the time in Arch is changed via timezones to UTC-6 (MST).

Has anyone figured this out? hmm kinda desperate, need my MMO fix lol.

Offline

#2442 2014-08-09 11:48:32

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Has anyone figured this out? hmm kinda desperate, need my MMO fix lol.

You could either try to make it refresh more often so it wouldn't jump as much at once, or turn off the automatic time correction altogether and update time by hand after quitting the game smile
Key to modify:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\W32Time\TimeProviders\NtpClient\SpecialPollInterval (seconds)

Offline

#2443 2014-08-09 15:06:45

Namelles_One
Member
Registered: 2014-05-18
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

irtimmer wrote:

I never know it was possible to grab the mouse and keyboard with -vga none in Qemu 2.0 as it also don't work in Qemu < 2.0. But a simple fix is to add
"-device qxl"

This don't help me. On Qemu 2.0 I was clicking on black window and my VM grab my input. How to do this with 2.1?

Offline

#2444 2014-08-09 15:56:37

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Have a huge problem, no idea how to fix it, really need help as it makes a game I'm playing unplayable.

Final Fantasy XIV (an MMO) disconnects every hour. After spending ~8 hours trying to figure out why, I've narrowed it down to the cause but can't find a solution.

Every hour, the Windows VM's clock drifts to 3-4 minutes behind the real time. Windows detects this and fixes it. All applications that use the system clock then freak out as time just jumped forward 3-4 minutes, including FFXIV which causes you to disconnect.

The system time has changed to 2014-08-09T11:23:48.500000000Z from 2014-08-09T11:22:27.765200000Z.

This is the message shown in Windows Event Viewer, pointing out the change. I've tried adding two things to qemu to fix it.

-rtc clock=vm,base=localtime,driftfix=slew
-rtc base=utc,driftfix=slew

Neither of these have worked. The first one has worked the best, and has the drift down to 1 minute per hour, but that is still unacceptable and causes a disconnect. My computers RTC (in BIOS) is set to UTC time, and the time in Arch is changed via timezones to UTC-6 (MST).

Has anyone figured this out? hmm kinda desperate, need my MMO fix lol.

I've stumbled upon this issue, when all of hosts cores/threads where assigned to vm. Host was doing something in the background, while I was playing on Windows.

Offline

#2445 2014-08-09 19:24:28

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aldum wrote:
alexis_evo wrote:

Has anyone figured this out? hmm kinda desperate, need my MMO fix lol.

You could either try to make it refresh more often so it wouldn't jump as much at once, or turn off the automatic time correction altogether and update time by hand after quitting the game smile
Key to modify:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\W32Time\TimeProviders\NtpClient\SpecialPollInterval (seconds)

I appreciate the help, but I don't think the issue is NTP. I've already disabled NTP in Windows. It's my understanding that there's three clocks on a computer. The RTC (hardware level, set in BIOS, battery backed), the system time (low level, set by the RTC on boot, never changes, used for timing by games/etc), and the local clock (the time displayed, synced by NTP). The problem is the system time is drifting off of the RTC. Windows checks it every hour and sees how much it has drifted, then snaps it back into place. XIV expects this time to never ever change, so it messes seriously with the ping thread/connection.

dwe11er wrote:

I've stumbled upon this issue, when all of hosts cores/threads where assigned to vm. Host was doing something in the background, while I was playing on Windows.

I don't see why the # of cores would matter, Windows isn't eating up all available CPU time, but I'll try dropping it down to 7 cores anyway. Also tried adding "hv-time" to "-cpu host, hv-time", as it looks like that should fix it. I guess time will tell...

Last edited by alexis_evo (2014-08-09 19:24:45)

Offline

#2446 2014-08-09 19:27:21

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is the KVM=off applied in Qemu 2.1.0 ? it was released 8 days ago, do I still need the Qemu-git ?
And I did compile the Qemu-git, yet I receive BSOD in windows, and I can not pass the USB devices with -usb -usbdevice host:*:* anymore.

Offline

#2447 2014-08-09 20:07:56

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Still happening.

The system time has changed to 2014-08-09T20:07:47.500000000Z from 2014-08-09T20:05:54.641000000Z.

Offline

#2448 2014-08-09 22:17:42

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:
aldum wrote:
alexis_evo wrote:

Has anyone figured this out? hmm kinda desperate, need my MMO fix lol.

You could either try to make it refresh more often so it wouldn't jump as much at once, or turn off the automatic time correction altogether and update time by hand after quitting the game smile
Key to modify:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\W32Time\TimeProviders\NtpClient\SpecialPollInterval (seconds)

I appreciate the help, but I don't think the issue is NTP. I've already disabled NTP in Windows. It's my understanding that there's three clocks on a computer. The RTC (hardware level, set in BIOS, battery backed), the system time (low level, set by the RTC on boot, never changes, used for timing by games/etc), and the local clock (the time displayed, synced by NTP). The problem is the system time is drifting off of the RTC. Windows checks it every hour and sees how much it has drifted, then snaps it back into place. XIV expects this time to never ever change, so it messes seriously with the ping thread/connection.

dwe11er wrote:

I've stumbled upon this issue, when all of hosts cores/threads where assigned to vm. Host was doing something in the background, while I was playing on Windows.

I don't see why the # of cores would matter, Windows isn't eating up all available CPU time, but I'll try dropping it down to 7 cores anyway. Also tried adding "hv-time" to "-cpu host, hv-time", as it looks like that should fix it. I guess time will tell...

It's because of load on every core host can't keep up with clock emulation. I'm guessing here.

Offline

#2449 2014-08-09 22:28:49

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CPU has 8 cores, at most I'm ever using 300%. There's more than enough CPU time on the host to do things it needs. I lowered Windows to 7 cores and it still drifts. I've tried a couple different options now and it still drifts. It's annoying as it takes an hour to test every time, as it takes an hour for Windows to pick up on the drift and adjust system time. Should know in a couple of minutes if the latest incantation worked...

Offline

#2450 2014-08-09 22:31:39

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

I appreciate the help, but I don't think the issue is NTP. I've already disabled NTP in Windows. It's my understanding that there's three clocks on a computer. The RTC (hardware level, set in BIOS, battery backed), the system time (low level, set by the RTC on boot, never changes, used for timing by games/etc), and the local clock (the time displayed, synced by NTP). The problem is the system time is drifting off of the RTC.

Ok, so you tried turning it off, now try the other way: have it update every 5 minutes.

Offline

#2451 2014-08-09 22:56:57

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aldum wrote:
alexis_evo wrote:

I appreciate the help, but I don't think the issue is NTP. I've already disabled NTP in Windows. It's my understanding that there's three clocks on a computer. The RTC (hardware level, set in BIOS, battery backed), the system time (low level, set by the RTC on boot, never changes, used for timing by games/etc), and the local clock (the time displayed, synced by NTP). The problem is the system time is drifting off of the RTC.

Ok, so you tried turning it off, now try the other way: have it update every 5 minutes.

1) that is an ugly hack that will still cause problems (what if an application is affected by a time drift of 1-2 seconds? the time drift is as high as 1-2 seconds per minute, so 5-10 seconds of drift every 5 minutes is unacceptable).
2) public NTP servers will ban for polling that often.
3) AFAIK, NTP syncs the local/visible time, not the system time. With NTP enabled it will just fix the visible clock, leaving the system time to drift until the kernel snaps it back into place. If NTP were to sync the system time, it would break many applications whenever it is ran, so I don't believe they would do that (but who knows, maybe I'm wrong).

I will try it after a couple more attempts, but I really don't like the solution, even if it were to work.

After disabling HPET (passing -no-hpet to qemu), and running it for an hour, Windows' kernel hasn't snapped the time forward. can't tell for sure, but manually comparing the times it does seem to be keeping in sync. Will update if/when it kicks in again, or a couple of hours if it's still synced.

Offline

#2452 2014-08-10 03:40:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:
aldum wrote:
alexis_evo wrote:

I appreciate the help, but I don't think the issue is NTP. I've already disabled NTP in Windows. It's my understanding that there's three clocks on a computer. The RTC (hardware level, set in BIOS, battery backed), the system time (low level, set by the RTC on boot, never changes, used for timing by games/etc), and the local clock (the time displayed, synced by NTP). The problem is the system time is drifting off of the RTC.

Ok, so you tried turning it off, now try the other way: have it update every 5 minutes.

1) that is an ugly hack that will still cause problems (what if an application is affected by a time drift of 1-2 seconds? the time drift is as high as 1-2 seconds per minute, so 5-10 seconds of drift every 5 minutes is unacceptable).
2) public NTP servers will ban for polling that often.
3) AFAIK, NTP syncs the local/visible time, not the system time. With NTP enabled it will just fix the visible clock, leaving the system time to drift until the kernel snaps it back into place. If NTP were to sync the system time, it would break many applications whenever it is ran, so I don't believe they would do that (but who knows, maybe I'm wrong).

I will try it after a couple more attempts, but I really don't like the solution, even if it were to work.

After disabling HPET (passing -no-hpet to qemu), and running it for an hour, Windows' kernel hasn't snapped the time forward. can't tell for sure, but manually comparing the times it does seem to be keeping in sync. Will update if/when it kicks in again, or a couple of hours if it's still synced.

Maybe this helps? https://access.redhat.com/documentation … ement.html

Offline

#2453 2014-08-10 08:23:40

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah I saw and tried that. So far the only thing that's worked is disabling HPET. It does work though, so I guess this is 'solved'. I don't think I need HPET for anything...

Offline

#2454 2014-08-10 13:19:12

d3lxa
Member
Registered: 2014-07-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After a few weeks of test, I can say this is very stable. Had only a crash due to Windows (GPU overheat).

Nonetheless I have found a side problem with suspend/hibernate: After playing a few hours I shutdown Windows but I cannot suspend/hibernate (in Linux) at all. The computer seems to partially shutdown (mouse/keyboard off, monitor off) but it stays up (power on, fans on, etc). I can suspend/hibernate just fine when I freshly rebooted. Until I play using qemu VFIO I don't have such problem. If I use windows without gaming it's fine too (as far I can tell, I don't have tested too long). It seems, after a certain time of gaming, it becomes impossible to suspend, for some unknown reason. It may be related to the D3 state of the card. I tried to eject the VGA card but it's worse (Windows does not always shutdown).

Is it possible the card refuse to let it shutdown if it's still hot (fan at high speed) or until a certain time has passed after the VM is down? Does anyone experience this too? Does someone has any tip that I could try to see if it solve the issue, anything? I have 3.14.12 without patch (is the vga_arbitrer patch useful for this?)

EDIT: After a few tests, it's more simple than that: most of the time I was not able to suspend after running qemu with VFIO (without windows) in fact. What solved this problem (at great cost) is the VGAARB patch as expected. It prevents the color glitches I had until now on my linux screen but I am not able to play on Linux anymore (DRI disabled as described in the details) which is very unfortunate. I prefer DRI over being able to suspend. No other workaround out there?

My hardware+detailed report earlier: https://bbs.archlinux.org/viewtopic.php … 7#p1436677

Last edited by d3lxa (2014-08-11 00:20:09)

Offline

#2455 2014-08-10 17:59:27

das_j
Member
Registered: 2014-08-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am currently trying to realize this because I want to get away from Windows.
Arch looks like the best Distribution for me, even if I am using it the first time (I was using SLES at work so I have at least a bit of knowledge).
This looks like the perfect project to keep playing my Windows/DirectX games.
The VM starts perfectly (currently without any drives) and shows up on my second screen.
But when I kill it using Ctrl+C (Output: qemu: terminating on signal 2) and want to start it again, I get the following error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:04:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

The second screen (which kept the last image of the previous run) turned black and I had to kill it.

System information:

I have a Nvidia GTX 460 (to pass to the VM) and a Nvidia GeForce GT 240 for the Host.
My Mainboard is the Gigabyte GA-970A-UD3 and I have a AMD FX-8350.

lspci -k -nn (after binding to vfio):

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT215 [GeForce GT 240] [10de:0ca3] (rev a2) Subsystem: CardExpert Technology Device [10b0:0401] Kernel driver in use: nouveau Kernel modules: nouveau 01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be4] (rev a1) Subsystem: CardExpert Technology Device [10b0:0401] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 02:00.0 USB controller [0c03]: VIA Technologies, Inc. Device [1106:3483] (rev 01) Subsystem: Gigabyte Technology Co., Ltd Device [1458:5007] Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 03:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11) Subsystem: Gigabyte Technology Co., Ltd Device [1458:b000] Kernel driver in use: ahci Kernel modules: ahci 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 460 v2] [10de:1205] (rev a1) Subsystem: Gigabyte Technology Co., Ltd Device [1458:34fc] Kernel driver in use: vfio-pci Kernel modules: nouveau 04:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1) Subsystem: Gigabyte Technology Co., Ltd Device [1458:34fc] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

cat /proc/cmdline:

BOOT_IMAGE=/boot/vmlinuz-linux-custom root=UUID=5bccb51e-9801-4086-93d7-535ba06e5bae rw quiet pci-stub.ids=10de:1205,10de:0e0c iommu=pt

Note: I use iommu=pt because Amd-Vi was spamming the log and slowing down the boot process. It still dosen't work when I remove it.

qemu call:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

qemu-system-x86_64 --version:

QEMU emulator version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

cat /proc/version:

Linux version 3.15.8-1-custom (root@Acerola) (gcc version 4.9.1 (GCC) ) #1 SMP PREEMPT Sun Aug 10 14:12:21 CEST 2014

Can anybody help me with this?

Offline

#2456 2014-08-10 19:31:18

shrubuntu
Member
Registered: 2012-12-26
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kameloc wrote:
shrubuntu wrote:
kameloc wrote:

Oh sorry I don't passthrough my onboard audio, just share it. If you add

-device ac97,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

It emulates Realtek AC'97 for the guest. It allowed me to use my speakers from the VM or host, and at the same time

Can you show me the whole command? I can't seem to get it to work

This is how I start my VM. Only passthrough my GPU and a USB device. Emulate ac97 at the end.

sudo -E qemu-system-x86_64 -enable-kvm \
-M q35 \
-cpu host \
-smp 4,cores=4,threads=8,sockets=1 \
-m 6122 \
-rtc base=localtime,clock=host \
-balloon virtio \
-boot menu=on \
-bios /usr/share/qemu/bios.bin \
-vga none \
-usbdevice host:0d8c:0008 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,romfile=/home/kameloc/windows7/EVGA.GTX550Ti.1024.121031.rom,multifunction=on,x-vga=on \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/home/kameloc/windows7/windows7.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
-net nic,macaddr=DE:AD:BE:EF:00:00 \
-net bridge,br=br0 \
-soundhw ac97


alsa throws a bunch of errors when I call soundhw. Doesn't work sad

-soundhw ac97

Offline

#2457 2014-08-10 20:09:23

shazeal
Member
From: New Zealand
Registered: 2007-06-05
Posts: 339

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

penetal wrote:

I am trying to create an audio infterface on the VM, but I end up with the scratchy / crackling sound problem that some others have come across in this thread.

I tried to set the sample rate and driver for QEMU, but that resulted in the problem becomming 100 times worse so I commented it out.
Startup Script:

#!/bin/sh vfio-bind 0000:01:00.0 0000:01:00.1 #export QEMU_PA_SAMPLES=128 #export QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -M q35 -m 10240 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/home/USER/vm/GamingWindows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -vga none \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:046d:c52b \ -monitor stdio

Has anyone been able to solve this problem when those two things didn't help?

Browsing through and saw this, no one replied so I thought I would post my solution for crappy audio in qemu smile

First in the /etc/pulse/daemon.conf (or where ever it is on Arch) set these or change the values you have.

default-sample-format = s16le default-sample-rate = 48000 default-fragments = 5 default-fragment-size-msec = 2

This lowers the latency and sets the sample rate to the same as the guest so no resampling is needed.

Then in your QEMU startup use this...

export QEMU_AUDIO_DRV=pa export QEMU_PA_SAMPLES=1024

The samples value can be tweaked up a bit, but anything under 1024 seems to introduce the choppy audio issue.

I still get the occasional crackle when the guest cpus are loaded but for the most part sound is flawless.

Offline

#2458 2014-08-10 20:55:14

Mardok45
Member
Registered: 2014-08-10
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm running on Gentoo.

I have successfully passed through an old ATI HD 4850 and a Radeon HD 7470.  These cards have worked great so far, but the one problem is neither are particularly powerful for recent games.  So I tried passing through my NVidia GTX 260 and it pretty much refuses to work.  After installing the drivers, I might be able to boot into a working instance, but after rebooting, Windows permanently BSOD's, even when not passing through the GPU, and I have to reimage the instance to get it working again.

Has anyone else had the same experience?  Any suggestions on getting the NVidia card to work?

If it's not likely to get a Geforce card working, I'm prepared to buy a Radeon 270X.  I really want to get this working because it will solve a lot of headaches, not just with playing games, but with development work as well.  The problem with that, though, is I haven't seen anybody confirm that R9 cards work.  I saw a video claiming that it was a 280X working with KVM VGA passthrough, but I haven't been able to verify it.

Does anyone have any tips for getting a Geforce card to work with VGA passthrough?  If not, can anybody confirm that Radeon R9 cards work with VGA passthrough?

EDIT: Well, I tried a clean installation and tried installing the NVidia drivers again, this time I'm getting that infamous Code 43 error.  I'm running on kernel 3.15 with the i915 arbiter patch and vga arbiter patch, and QEMU 2.1.0.

EDIT 2: Now I'm really confused, I downgraded the NVidia drivers to 275.33 and it's working.  The "3d business and gaming graphics performance" score is 6.9.  I don't understand what's going on.  I'll install and play Dota 2 and see how the performance compares to native and whether or not the VM is stable during play.

EDIT 3: The VM is stable, but the performance could be better.  On bare metal, I can play Dota 2 on max.  On a KVM instance, the FPS drops dramatically.  But hey, at least it works.  Now I can at least have a multi-OS workstation.

EDIT 4: Managed to upgrade the drivers to 334.89, but still having performance problems with Dota 2.

Last edited by Mardok45 (2014-08-11 02:08:23)

Offline

#2459 2014-08-11 04:34:08

shazeal
Member
From: New Zealand
Registered: 2007-06-05
Posts: 339

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to thank everyone in this thread, picked up lots of bits and pieces in here. Been testing for days using -vga std. Finally chucked in a second card today and now have a fully armed and operational OSX 10.9.4 guest smile

My setup.

Gentoo / Custom Kernel 3.16.0 + BFQ / GCC Optimization patches
Host: ASUS Nvidia 760 GTX
Guest: ASUS AMD 6450
CPU: Intel i5 2500 2 cpus in cpuset for guest
Memory: 16G, 2G hugetlbfs for guest.
Network: -netdev user,hostfwd=tcp::6900-:5900,id=user.0 -device e1000-82545em,netdev=user.0,id=mac_vnet0
Sound: Currently using ich9-intel-hda, but will probably switch to a USB stick at some point.
Disks: Preallocated qcow2 on SSD.

Have ordered a KVM Switch so I can just share the host keyboard and mouse and primary monitor. Fingers crossed using -device usb-host,hostbus=x,hostaddr=x assignment should work for the switch.

Offline

#2460 2014-08-11 04:41:56

shazeal
Member
From: New Zealand
Registered: 2007-06-05
Posts: 339

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mardok45 wrote:

I'm running on Gentoo.

I have successfully passed through an old ATI HD 4850 and a Radeon HD 7470.  These cards have worked great so far, but the one problem is neither are particularly powerful for recent games.  So I tried passing through my NVidia GTX 260 and it pretty much refuses to work.  After installing the drivers, I might be able to boot into a working instance, but after rebooting, Windows permanently BSOD's, even when not passing through the GPU, and I have to reimage the instance to get it working again.

Has anyone else had the same experience?  Any suggestions on getting the NVidia card to work?

If it's not likely to get a Geforce card working, I'm prepared to buy a Radeon 270X.  I really want to get this working because it will solve a lot of headaches, not just with playing games, but with development work as well.  The problem with that, though, is I haven't seen anybody confirm that R9 cards work.  I saw a video claiming that it was a 280X working with KVM VGA passthrough, but I haven't been able to verify it.

Does anyone have any tips for getting a Geforce card to work with VGA passthrough?  If not, can anybody confirm that Radeon R9 cards work with VGA passthrough?

EDIT: Well, I tried a clean installation and tried installing the NVidia drivers again, this time I'm getting that infamous Code 43 error.  I'm running on kernel 3.15 with the i915 arbiter patch and vga arbiter patch, and QEMU 2.1.0.

EDIT 2: Now I'm really confused, I downgraded the NVidia drivers to 275.33 and it's working.  The "3d business and gaming graphics performance" score is 6.9.  I don't understand what's going on.  I'll install and play Dota 2 and see how the performance compares to native and whether or not the VM is stable during play.

EDIT 3: The VM is stable, but the performance could be better.  On bare metal, I can play Dota 2 on max.  On a KVM instance, the FPS drops dramatically.  But hey, at least it works.  Now I can at least have a multi-OS workstation.

EDIT 4: Managed to upgrade the drivers to 334.89, but still having performance problems with Dota 2.

The R9 270X is just a 7870 card rebranded and overclocked, regardless any video card can be passed through its just a PCI redirect afaik. Its upto the guest to support the video card.

More important than the video card is how are you running CPU/RAM, are you using cgroups and hugetlbfs?

Offline

#2461 2014-08-11 04:48:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mardok45 wrote:

I'm running on Gentoo.

I have successfully passed through an old ATI HD 4850 and a Radeon HD 7470.  These cards have worked great so far, but the one problem is neither are particularly powerful for recent games.  So I tried passing through my NVidia GTX 260 and it pretty much refuses to work.  After installing the drivers, I might be able to boot into a working instance, but after rebooting, Windows permanently BSOD's, even when not passing through the GPU, and I have to reimage the instance to get it working again.

Has anyone else had the same experience?  Any suggestions on getting the NVidia card to work?

If it's not likely to get a Geforce card working, I'm prepared to buy a Radeon 270X.  I really want to get this working because it will solve a lot of headaches, not just with playing games, but with development work as well.  The problem with that, though, is I haven't seen anybody confirm that R9 cards work.  I saw a video claiming that it was a 280X working with KVM VGA passthrough, but I haven't been able to verify it.

Does anyone have any tips for getting a Geforce card to work with VGA passthrough?  If not, can anybody confirm that Radeon R9 cards work with VGA passthrough?

EDIT: Well, I tried a clean installation and tried installing the NVidia drivers again, this time I'm getting that infamous Code 43 error.  I'm running on kernel 3.15 with the i915 arbiter patch and vga arbiter patch, and QEMU 2.1.0.

EDIT 2: Now I'm really confused, I downgraded the NVidia drivers to 275.33 and it's working.  The "3d business and gaming graphics performance" score is 6.9.  I don't understand what's going on.  I'll install and play Dota 2 and see how the performance compares to native and whether or not the VM is stable during play.

EDIT 3: The VM is stable, but the performance could be better.  On bare metal, I can play Dota 2 on max.  On a KVM instance, the FPS drops dramatically.  But hey, at least it works.  Now I can at least have a multi-OS workstation.

EDIT 4: Managed to upgrade the drivers to 334.89, but still having performance problems with Dota 2.

Add kvm=off to the -cpu parameter list to make current Nvidia driver not get Code 43.  Also add hv_time to enable hypver-v enlightened time keeping.  Older kernels can have poor performance if the game makes use of debug registers, run something new.  Hugepages, pinning, and vCPU model tuning also help.  All of this is documented in the very, very long thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2462 2014-08-11 05:39:39

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mardok45 check your cpu config

    <topology sockets='1' cores='4' threads='1'/>

if u set 2 cores 2 threads dota works 2x slower, also check your cpu model, hvtime option... etc... host cpufreq settings....

Last edited by slis (2014-08-11 05:40:22)

Offline

#2463 2014-08-11 10:51:00

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have 3 GPUs:

Passing through either GPU works flawlessly at any time.
Passing through two GPUs at the same time does not work.

Im using Ubuntu 14.04 with no "patches" whatsoever. What could be the problem?

The error is:
vfio: error opening /dev/vfio/1: Device or resource busy
vfio: failed to get group 1
Device initialization failed.
Device 'vfio-pci' could not be initialized

If I shutdown the other VM and start the one getting error at the moment, it works and then the one that works at the moment is getting the same error. They don't work at the same time in 2 different VMs.
I added "/dev/vfio/1" to /etc/libvirt/qemu.conf and "relaxed_acs_check = 1".

Please help!!

EDIT: When I start 2nd VM its always getting "vfio: failed to get group 1", thats why I added "/dev/vfio/1 to the qemu.conf. Should I do something else? Please help!

Last edited by devianceluka (2014-08-11 10:52:41)

Offline

#2464 2014-08-11 12:24:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

I have 3 GPUs:

Passing through either GPU works flawlessly at any time.
Passing through two GPUs at the same time does not work.

Im using Ubuntu 14.04 with no "patches" whatsoever. What could be the problem?

ACS override patch...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2465 2014-08-11 13:02:48

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:

I have 3 GPUs:

Passing through either GPU works flawlessly at any time.
Passing through two GPUs at the same time does not work.

Im using Ubuntu 14.04 with no "patches" whatsoever. What could be the problem?

ACS override patch...

Thank you.

Doing this command at the moment:
sudo patch -p1 override_for_missing_acs_capabilities.patch

Is this the right command? This machine is booting from a VERY slow USB drive (but it works..), VMs are on fast iSCSI. Is this command "compiling/building" anything? If it is, I guess this will take forever in this situation if I'm not mistaken. After this command there is an empty line after and don't know if its doing anything or not..?

Offline

#2466 2014-08-11 14:12:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
aw wrote:
devianceluka wrote:

I have 3 GPUs:

Passing through either GPU works flawlessly at any time.
Passing through two GPUs at the same time does not work.

Im using Ubuntu 14.04 with no "patches" whatsoever. What could be the problem?

ACS override patch...

Thank you.

Doing this command at the moment:
sudo patch -p1 override_for_missing_acs_capabilities.patch

Is this the right command?

No, patch takes the diff file on stdin: http://jungels.net/articles/diff-patch-ten-minutes.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2467 2014-08-11 14:40:03

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

one question pls smile

what are options for clock or what clock to use on windows guest to make it "tight" or something like that, i get this weird problem after 30 or 60mins clock on vm is re/set or smth, cause when streaming games logs show frame time in -12312435ms expected is 20-50ms, and it starts to lag, it shows estimated bandwith like 10-40mbps (expected is 400-1000) in steam stats on 1gbps lan (alt-tab, copy-paste at same time 100MB/s) etc.... maybe cause of high cpu load? clock lags?

i did try couple of options that i found here/on internet, but nothing seems to make it go away completely.

problem it is happening after 30mins so its pain to wait for it smile

does hwclock need to be in sync with os clock?


thx

Offline

#2468 2014-08-11 15:38:44

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:
aw wrote:

ACS override patch...

Thank you.

Doing this command at the moment:
sudo patch -p1 override_for_missing_acs_capabilities.patch

Is this the right command?

No, patch takes the diff file on stdin: http://jungels.net/articles/diff-patch-ten-minutes.html

You got me all clueless now.. What should I do exactly? Im running Ubuntu 14.04 upgraded everything to the latest. If you could please write those lines down I would really appreciate it.. Please

Offline

#2469 2014-08-11 15:53:17

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
aw wrote:
devianceluka wrote:

Thank you.

Doing this command at the moment:
sudo patch -p1 override_for_missing_acs_capabilities.patch

Is this the right command?

No, patch takes the diff file on stdin: http://jungels.net/articles/diff-patch-ten-minutes.html

You got me all clueless now.. What should I do exactly? Im running Ubuntu 14.04 upgraded everything to the latest. If you could please write those lines down I would really appreciate it.. Please

all the stuff discussed in the thread is still experimental
not everything is working, and what is working at the moment may require great amount of grinding to get it to work

so, you either get to catch-up on basic stuff yourself, or just wait for all the functionality to mature enough not to be considered experimental

people here are too busy fighting non-trivial stuff (==trying to make things more mature) - please consider possibility of not wasting their time to try to explain to you how to patch kernel and why it is needed at all (or anything else that is "entry-level")

p.s. you could start by reading all the thread posts (while googling how to do all the stuff you dont understand) - yes, the thread is pretty long, but there are people here whose time is much more valuable for overall progress than yours or mine is (maturing general vt-d support is much more important than getting your stuff to work)

Offline

#2470 2014-08-11 16:05:09

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
devianceluka wrote:
aw wrote:

No, patch takes the diff file on stdin: http://jungels.net/articles/diff-patch-ten-minutes.html

You got me all clueless now.. What should I do exactly? Im running Ubuntu 14.04 upgraded everything to the latest. If you could please write those lines down I would really appreciate it.. Please

all the stuff discussed in the thread is still experimental
not everything is working, and what is working at the moment may require great amount of grinding to get it to work

so, you either get to catch-up on basic stuff yourself, or just wait for all the functionality to mature enough not to be considered experimental

people here are too busy fighting non-trivial stuff (==trying to make things more mature) - please consider possibility of not wasting their time to try to explain to you how to patch kernel and why it is needed at all (or anything else that is "entry-level")

p.s. you could start by reading all the thread posts (while googling how to do all the stuff you dont understand) - yes, the thread is pretty long, but there are people here whose time is much more valuable for overall progress than yours or mine is (maturing general vt-d support is much more important than getting your stuff to work)

I understand that and I really respect all the work. But if someone got the time to explain a newbie here how to apply this patch, it would be awesome. Do I need the kernel source to do it? Do I need to compile kernel? Or it can be done to the existing kernel? If someone got the time I would really appreciate it.

EDIT: Especially that I got that far that I only need this patch do finish... It would be a real shame if noone got a minute to write down.. sad

Last edited by devianceluka (2014-08-11 16:08:59)

Offline

#2471 2014-08-11 16:30:26

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
sinny wrote:
devianceluka wrote:

You got me all clueless now.. What should I do exactly? Im running Ubuntu 14.04 upgraded everything to the latest. If you could please write those lines down I would really appreciate it.. Please

all the stuff discussed in the thread is still experimental
not everything is working, and what is working at the moment may require great amount of grinding to get it to work

so, you either get to catch-up on basic stuff yourself, or just wait for all the functionality to mature enough not to be considered experimental

people here are too busy fighting non-trivial stuff (==trying to make things more mature) - please consider possibility of not wasting their time to try to explain to you how to patch kernel and why it is needed at all (or anything else that is "entry-level")

p.s. you could start by reading all the thread posts (while googling how to do all the stuff you dont understand) - yes, the thread is pretty long, but there are people here whose time is much more valuable for overall progress than yours or mine is (maturing general vt-d support is much more important than getting your stuff to work)

I understand that and I really respect all the work. But if someone got the time to explain a newbie here how to apply this patch, it would be awesome. Do I need the kernel source to do it? Do I need to compile kernel? Or it can be done to the existing kernel? If someone got the time I would really appreciate it.

EDIT: Especially that I got that far that I only need this patch do finish... It would be a real shame if noone got a minute to write down.. :(

well, sorry to mention that, but you got abslutely nowhere (no way there is some "that far" you are implying) if you neither dont understand the very basics (things discussed here are much much more complex), nor got desire to explore things yourself

is it some kind of shame to google for something like "ubuntu patching kernel"? this way you only "waste" your own time. but no, somehow it seems like you prefer to waste your own AND someone else's time to explain to you things that are written about multiple times all over the internet

upd:
now me gracefully stops doing this holy-war-like thing. i have only one thing to add: respect other peoples' time and work more than your own

Last edited by sinny (2014-08-11 16:32:25)

Offline

#2472 2014-08-11 16:33:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

EDIT: Especially that I got that far that I only need this patch do finish... It would be a real shame if noone got a minute to write down.. sad

It's already been written down if you'd check.  Maybe start with http://kernelnewbies.org/


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2473 2014-08-11 18:52:45

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, I got Ubuntu 14.04 LTS, everything worked great, until I broke the system in order to ( I bought a new card, GTX 760..) getting rid of error 43, which I did! in debian tested! yet everything worked great, until I started copying and pasting cos I freaked out, yet I solved the error, but guss what, the system freeze after a guest starts. Now I'm looking forward for linux mint 17, hope to get everything sorted out this time!.

notes to new folks: it is easy ! just take the time and read here and there, which... will take long time, but knowledged is good.

Edit:

I tested Qemu 2.1.0, it includes the kvm=off patch, no need to re-compile!.

Last edited by abdullah (2014-08-11 18:55:11)

Offline

#2474 2014-08-11 19:10:59

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I got it how to compile the kernel, read few guides.

What I now get is this:

sudo patch -p1 < override_for_missing_acs_capabilities.patch
patching file Documentation/kernel-parameters.txt
Hunk #1 succeeded at 2490 with fuzz 1 (offset -64 lines).
patching file drivers/pci/quirks.c
Hunk #1 succeeded at 3394 (offset 10 lines).
Hunk #2 succeeded at 3539 with fuzz 1 (offset -45 lines).
Hunk #3 FAILED at 3597.
1 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej

Could anyone please tell me what I'm doing wrong? The kernel I got is: "linux-image-3.13.0-32-generic" which I got with "sudo apt-get source linux-image-`uname -r`"

Offline

#2475 2014-08-11 19:51:38

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

I think I got it how to compile the kernel, read few guides.

What I now get is this:

sudo patch -p1 < override_for_missing_acs_capabilities.patch
patching file Documentation/kernel-parameters.txt
Hunk #1 succeeded at 2490 with fuzz 1 (offset -64 lines).
patching file drivers/pci/quirks.c
Hunk #1 succeeded at 3394 (offset 10 lines).
Hunk #2 succeeded at 3539 with fuzz 1 (offset -45 lines).
Hunk #3 FAILED at 3597.
1 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej

Could anyone please tell me what I'm doing wrong? The kernel I got is: "linux-image-3.13.0-32-generic" which I got with "sudo apt-get source linux-image-`uname -r`"

patches fail to apply if source code is significantly different compared to source the patch was originally made for
there are several versions of every patch spread throughout the thread

i may be wrong, but it seems to me that currently most people here use 3.15.x-3.16.x kernels

Last edited by sinny (2014-08-11 19:52:23)

Offline

#2476 2014-08-11 20:19:25

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

I think I got it how to compile the kernel, read few guides.

What I now get is this:

sudo patch -p1 < override_for_missing_acs_capabilities.patch
patching file Documentation/kernel-parameters.txt
Hunk #1 succeeded at 2490 with fuzz 1 (offset -64 lines).
patching file drivers/pci/quirks.c
Hunk #1 succeeded at 3394 (offset 10 lines).
Hunk #2 succeeded at 3539 with fuzz 1 (offset -45 lines).
Hunk #3 FAILED at 3597.
1 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej

Could anyone please tell me what I'm doing wrong? The kernel I got is: "linux-image-3.13.0-32-generic" which I got with "sudo apt-get source linux-image-`uname -r`"

here is the kernel :
http://kernel.org/

change dir to

 cd /usr/src/ 

Download the last stable, extract with

tar -xvf "archive name"

Download the patches from the first page, extract to /usr/src
change dir to the extracted kernel,

 cd linux-3.16/ 

apply the needed patches by:

 patch -p1 < ../linux-mainline/ [NEEDED .PATCH] 

you could also read here, it helped me very well:
http://edencomputing.com/

Offline

#2477 2014-08-11 20:25:47

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:
devianceluka wrote:

I think I got it how to compile the kernel, read few guides.

What I now get is this:

sudo patch -p1 < override_for_missing_acs_capabilities.patch
patching file Documentation/kernel-parameters.txt
Hunk #1 succeeded at 2490 with fuzz 1 (offset -64 lines).
patching file drivers/pci/quirks.c
Hunk #1 succeeded at 3394 (offset 10 lines).
Hunk #2 succeeded at 3539 with fuzz 1 (offset -45 lines).
Hunk #3 FAILED at 3597.
1 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej

Could anyone please tell me what I'm doing wrong? The kernel I got is: "linux-image-3.13.0-32-generic" which I got with "sudo apt-get source linux-image-`uname -r`"

here is the kernel :
http://kernel.org/

change dir to

 cd /usr/src/ 

Download the last stable, extract with

tar -xvf "archive name"

Download the patches from the first page, extract to /usr/src
change dir to the extracted kernel,

 cd linux-3.16/ 

apply the needed patches by:

 patch -p1 < ../linux-mainline/ [NEEDED .PATCH] 

you could also read here, it helped me very well:
http://edencomputing.com/

Thank you very much to you and sinny! But what I want to do is compile a "stable" kernel that gets updated every now and then by the OS to more and more "stable" one. Does this kernel from this site need to get "manually" updated every now and then? Im very grateful for what you wrote but try to answer this...

Offline

#2478 2014-08-12 04:44:33

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

@aw

one question pls smile

what are options for clock or what clock to use on windows guest to make it "tight" or something like that, i get this weird problem after 30 or 60mins clock on vm is re/set or smth, cause when streaming games logs show frame time in -12312435ms expected is 20-50ms, and it starts to lag, it shows estimated bandwith like 10-40mbps (expected is 400-1000) in steam stats on 1gbps lan (alt-tab, copy-paste at same time 100MB/s) etc.... maybe cause of high cpu load? clock lags?

i did try couple of options that i found here/on internet, but nothing seems to make it go away completely.

problem it is happening after 30mins so its pain to wait for it smile

does hwclock need to be in sync with os clock?


thx

A solution is posted on the very same page of the thread that your post is on wink. Pass -no-hpet to qemu. HPET in qemu seems to cause time drift.

Offline

#2479 2014-08-12 04:46:59

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
abdullah wrote:
devianceluka wrote:

I think I got it how to compile the kernel, read few guides.

What I now get is this:

sudo patch -p1 < override_for_missing_acs_capabilities.patch
patching file Documentation/kernel-parameters.txt
Hunk #1 succeeded at 2490 with fuzz 1 (offset -64 lines).
patching file drivers/pci/quirks.c
Hunk #1 succeeded at 3394 (offset 10 lines).
Hunk #2 succeeded at 3539 with fuzz 1 (offset -45 lines).
Hunk #3 FAILED at 3597.
1 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej

Could anyone please tell me what I'm doing wrong? The kernel I got is: "linux-image-3.13.0-32-generic" which I got with "sudo apt-get source linux-image-`uname -r`"

here is the kernel :
http://kernel.org/

change dir to

 cd /usr/src/ 

Download the last stable, extract with

tar -xvf "archive name"

Download the patches from the first page, extract to /usr/src
change dir to the extracted kernel,

 cd linux-3.16/ 

apply the needed patches by:

 patch -p1 < ../linux-mainline/ [NEEDED .PATCH] 

you could also read here, it helped me very well:
http://edencomputing.com/

Thank you very much to you and sinny! But what I want to do is compile a "stable" kernel that gets updated every now and then by the OS to more and more "stable" one. Does this kernel from this site need to get "manually" updated every now and then? Im very grateful for what you wrote but try to answer this...

you will have to manually update it every time either way unless you stop updating kernel at all. or you may find somewhere some kind of ubuntu repo with kernels specifically built for vt-d puproses (though me doubts it)
once again: this IS experimental stuff, so it IS NOT a convenient thing and it MAY break with any change/update you perform on the system

@abdullah:
there are tons of distro-specific scenarios descriptions on the internet, so you probably should not even try to describe it here (simply because you will miss ton of distro-specific details: e.g. co-existing with distro infrastructure, managing initramfs/initrd, etc)

Last edited by sinny (2014-08-12 04:47:26)

Offline

#2480 2014-08-12 05:11:57

maxexcloo
Member
From: Australia
Registered: 2009-10-14
Posts: 177
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does this work with switcable graphics? It would be an awesome use of that capability in my laptop wink

Offline

#2481 2014-08-12 07:04:00

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:
slis wrote:

@aw

one question pls smile

what are options for clock or what clock to use on windows guest to make it "tight" or something like that, i get this weird problem after 30 or 60mins clock on vm is re/set or smth, cause when streaming games logs show frame time in -12312435ms expected is 20-50ms, and it starts to lag, it shows estimated bandwith like 10-40mbps (expected is 400-1000) in steam stats on 1gbps lan (alt-tab, copy-paste at same time 100MB/s) etc.... maybe cause of high cpu load? clock lags?

i did try couple of options that i found here/on internet, but nothing seems to make it go away completely.

problem it is happening after 30mins so its pain to wait for it smile

does hwclock need to be in sync with os clock?


thx

A solution is posted on the very same page of the thread that your post is on wink. Pass -no-hpet to qemu. HPET in qemu seems to cause time drift.

i did try that... but problem still presists... i think its not real clock problem, its ticks or something... cause in logs input time goes in negative values.... and it happens due high cpu load... like some ticks got delayed...

Offline

#2482 2014-08-12 08:31:35

brezerk
Member
From: Kiev, Ukraine
Registered: 2014-08-12
Posts: 1
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lojsar wrote:

Furthermore, I had great success modifying the VBIOS. Using a program called "nvbios" from "envytools" together with nVidia's documentation of the DCB ( ftp://download.nvidia.com/open-gpu-doc/ … _dcb_flags ), I was able to replace the VGA output. After this, I see BIOS output on my monitor! I was also able to run Windows (with -vga none), and install the nVidia drivers!

Hi. Does anyone have an idea or doc how I can modify DCB table?

I have an Notebook with Intel HD + GT650M glued with optimus.
I was able to switch host output to Intel Graphic and bypass GT650M to kvm, and install nvidia drivers on w7 guest. But... I see no output on HDMI at all. =\
Also windows refuse to start if I set -vga none.

DCB reports:

DCB table at 0x532a version 4.0 unk16 0x01 DCB 0: type 3 [LVDS] I2C 0 heads 0 CONN 0 [0x40] conntag 0 LOCAL unk02_6 2 OR 0 links 0 1 INFO EDID_I2C POWER 1 unk06 0x01 DCB 1: type 6 [DP] I2C 10 heads 1 CONN 1 [0x46] conntag 1 LOCAL unk02_6 2 OR 1 links 0 BW 270000 lanes 0 1 2 3 DCB 2: type 2 [TMDS] I2C 1 heads 1 CONN 1 [0x46] conntag 1 LOCAL OR 1 links 0 HDMI DCB 3: type 6 [DP] I2C 11 heads 2 CONN 2 [0x46] conntag 2 LOCAL unk02_6 2 OR 2 links 0 BW 270000 lanes 0 1 2 3 DCB 4: type 2 [TMDS] I2C 7 heads 2 CONN 2 [0x46] conntag 2 LOCAL OR 2 links 0 HDMI DCB 5: type 6 [DP] I2C 12 heads 3 CONN 3 [0x46] conntag 3 LOCAL unk02_6 2 OR 3 links 0 BW 270000 lanes 0 1 2 3 DCB 6: type 2 [TMDS] I2C 8 heads 3 CONN 3 [0x46] conntag 3 LOCAL OR 3 links 0 HDMI DCB 7: type 14 [EOL]

It looks like I need to replace VGA output as well. But I am not 100% sure what I am doing :) Any help will be appreciated.

Offline

#2483 2014-08-12 08:40:06

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey. Just wanted to report my success thanks to this guide, been running this kind of setup for a few months and it's working great.

Running:
- Kernel 3.14.2 with Gentoo patchset and the i915 drm patch listed at start of thread
- Qemu 2.0.0 Vanilla
- Seabios 1.7.4 Vanilla
- Windows 8.1

On hardware:
- Asrock Z87 Extreme3
- Xeon E3-1275v3 (Xeon equivalent of the i7-4771, same price)
- And a GeForce GTX 750 Ti, works with or without passing the sound sub-card
- Running with EFI on the Host side, BIOS on the Guest side
I initially planned to run this kind of setup under Xen but apparently GeForce requires hardmods, and the 750 doesn't even seem to be moddable : p

Now for the gotchas:

- Windows 8.1's bootloader
The install had to be done without VGA passthrough enabled: when it is enabled, any bootloader update just nukes it for some reason. I tried to install Windows 8 and upgrade to 8.1 with passthrough, nuked. I installed 8.1 and left Windows Update auto, nuked after two days. All my recovery attempts failed miserably and the bootloader partition is outright empty.

- Commandline gotchas
Some of the standard Qemu options didn't seem to work because of the custom chipset, for exemple i had to use -drive instead of -hda and such.
You wouldn't believe how long it took me to figure it out...

- Sound
Sound's been a big issue for me at start. Using AC97 froze guest. I could never get the ES1370 driver from realtek installed on Win8. HDA seemed to work fine until i realized it added huge echo after some time. I settled with usb-audio which caused occasional crashes but worked fine. However once I installed PulseAudio on guest and output wasn't on ALSA anymore, it would start stuttering. After a while digging in the QEMU Source, i found out that the hda driver is meant to work on ICH6 chipsets while the IOH3420 is of the ICH9 variety. Thanksfully there is an undocumented "ich9-intel-hda" variant which works perfectly !

-device ich9-intel-hda,bus=root.1 -device hda-duplex,bus=hda.0

- Virtio
Couldn't manage to get virtio working correctly under Win8. It's probably possible, but i'm too lazy to try again knowing about the bootloader problems.

- Awkward Disk Iops Patterns
The only real issue i still have today. Maybe because i'm running with 32GB RAM (16GB Host; 16GB to Guest), there seems to be abberrant amount of caching. Got a 4-disk SATA RAID5 array underneath.
When doing a disk-to-disk transfer in the guest, rate peaks at 500 Mb then drops to zero. The guest freezes for a few seconds, then does it again.
This is especially noticeable in some games, where any object entering the field can cause a little freeze - i believe due to texture and models loading.

- Synergy for mouse/keyboard sharing
Little offtopic, but If you plan to use Synergy, be aware that the AltGr key seems to be buggued...
Even applying the workarounds found around the net, it doesn't seem to help much.
I chose to just live with it and keep @s to copy/paste around : p
Otherwise perfs are very good. Just be aware you need to lock the mouse to your screen to use mouse-grabbing apps or they glitch.


The whole line:

QEMU_AUDIO_DRV=pa \ qemu-system-x86_64 \ -name winix \ -enable-kvm \ -M q35 \ -m 16192 \ -cpu host \ -smp 4,sockets=1,cores=4 \ -net nic -net user \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -usb \ -device ich9-intel-hda,bus=root.1 \ -device hda-duplex,bus=hda.0 \ -drive file=/dev/raidvg/winix-sys,id=system \ -device ide-hd,bus=ide.0,drive=system \ -drive file=/dev/raidvg/winix,id=data \ -device ide-hd,bus=ide.1,drive=data \ -vga none -nographic \ -boot order=dc

That's it. Thanks for all the help, and I hope these informations will help you in turn smile
Good luck with your setups !

Last edited by Fira (2014-08-12 09:00:36)

Offline

#2484 2014-08-12 09:23:21

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got an ubuntu 14.04 64bit server with an ATI HD7770 secondary card (HD5450 primary, no proprietary drivers loaded) and having issues, so hope someone could help.

Has anyone got this working with the virtual machine manager (GUI)?
I've got hardware that is capable (did get EXSi VM working with my HD7770 and streaming some steam games), but struggling with kvm.
I have used the cli method to get my video card & audio passed ( lspci -k |grep vfio shows the 2 entries) and installed Win7.
But after installing the ATI drivers (just video & audio custom install) the first reboot I got error code 43 on the card. Close?
After a shutdown I can not boot the VM again successfully.

So thought I'd try the VMM route.
Installed Win7 and passed the card, shows in device manager, but again install drivers and fails to boot again.
Remove card entries and will boot fine.
I'm using a default sata drivers, using /dev/sdb2 raw partition and a fresh Win7x64 install.
So I noticed VMM was using kvm as the emulator (my error on creating I assume), so I started again with qemu-system-x86_64 but it blue screens on any VM boot it seems.
Is it futile to continue with VMM?
I have qemu 2.0 installed.
kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.2), Copyright (c) 2003-2008 Fabrice Bellard
Any suggestions?
Thanks.

Offline

#2485 2014-08-12 09:44:38

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
devianceluka wrote:
abdullah wrote:

here is the kernel :
http://kernel.org/

change dir to

 cd /usr/src/ 

Download the last stable, extract with

tar -xvf "archive name"

Download the patches from the first page, extract to /usr/src
change dir to the extracted kernel,

 cd linux-3.16/ 

apply the needed patches by:

 patch -p1 < ../linux-mainline/ [NEEDED .PATCH] 

you could also read here, it helped me very well:
http://edencomputing.com/

Thank you very much to you and sinny! But what I want to do is compile a "stable" kernel that gets updated every now and then by the OS to more and more "stable" one. Does this kernel from this site need to get "manually" updated every now and then? Im very grateful for what you wrote but try to answer this...

you will have to manually update it every time either way unless you stop updating kernel at all. or you may find somewhere some kind of ubuntu repo with kernels specifically built for vt-d puproses (though me doubts it)
once again: this IS experimental stuff, so it IS NOT a convenient thing and it MAY break with any change/update you perform on the system

@abdullah:
there are tons of distro-specific scenarios descriptions on the internet, so you probably should not even try to describe it here (simply because you will miss ton of distro-specific details: e.g. co-existing with distro infrastructure, managing initramfs/initrd, etc)

I understand, I thought it was like that. Can you recommend me a distro which has a repo like that? I don't care to use any other distro if its better for this. I just want it to be "stable" so I dont get hacked... Thanks for explanation!

Offline

#2486 2014-08-12 11:37:46

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

I understand, I thought it was like that. Can you recommend me a distro which has a repo like that? I don't care to use any other distro if its better for this. I just want it to be "stable" so I dont get hacked... Thanks for explanation!

you really think you are worth "hacking"? more like you can get "hacked" only if you do something weird with your own hands. bad news - no distro will protect you from that

btw, "stable" is not about being "hack"-proof. in case you wondered

back to the topic...
i doubt there is any (once again: experimental feature we are talking about here). people occasionally write their success reports, some include distro mention - you can dig the thread and get something to make a choice from. manual work will be needed any way though.
i use fedora, but every kernel update i download i have to manually rebuild i915 module and update initramfs (posted more detailed description few pages "ago")

p.s. there was a google docs file containing peoples' specs with success/failure indication - could anyone please bring up the link (have trouble finding it - would be great to have it in head post)?

Offline

#2487 2014-08-12 12:17:15

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
devianceluka wrote:

I understand, I thought it was like that. Can you recommend me a distro which has a repo like that? I don't care to use any other distro if its better for this. I just want it to be "stable" so I dont get hacked... Thanks for explanation!

you really think you are worth "hacking"? more like you can get "hacked" only if you do something weird with your own hands. bad news - no distro will protect you from that

btw, "stable" is not about being "hack"-proof. in case you wondered

back to the topic...
i doubt there is any (once again: experimental feature we are talking about here). people occasionally write their success reports, some include distro mention - you can dig the thread and get something to make a choice from. manual work will be needed any way though.
i use fedora, but every kernel update i download i have to manually rebuild i915 module and update initramfs (posted more detailed description few pages "ago")

p.s. there was a google docs file containing peoples' specs with success/failure indication - could anyone please bring up the link (have trouble finding it - would be great to have it in head post)?

I will try fedora. As long as its updating kernel every now and then as it should, I'm happy with rebuilding it manually. Fedora needs qemu from git yes? Ubuntu has qemuv2.x and it wasnt needed. Thanks man!

Offline

#2488 2014-08-12 14:39:57

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
devianceluka wrote:

I understand, I thought it was like that. Can you recommend me a distro which has a repo like that? I don't care to use any other distro if its better for this. I just want it to be "stable" so I dont get hacked... Thanks for explanation!

you really think you are worth "hacking"? more like you can get "hacked" only if you do something weird with your own hands. bad news - no distro will protect you from that

btw, "stable" is not about being "hack"-proof. in case you wondered

back to the topic...
i doubt there is any (once again: experimental feature we are talking about here). people occasionally write their success reports, some include distro mention - you can dig the thread and get something to make a choice from. manual work will be needed any way though.
i use fedora, but every kernel update i download i have to manually rebuild i915 module and update initramfs (posted more detailed description few pages "ago")

p.s. there was a google docs file containing peoples' specs with success/failure indication - could anyone please bring up the link (have trouble finding it - would be great to have it in head post)?

This one :

https://docs.google.com/spreadsheet/ccc … _web#gid=0

If you have the right hardware, it should work, I did it on Ubuntu, debian, working on Linux Mint now. When I did it on ubuntu it was stable enough, yet I had to get A nvidia card, which came with troubles, ending in breaking the system trying to fix it, so yes, no, this is not about being stable, and if you value the work on your PC you should be packing it up right now before even starting.

And I found an intersting wiki here, on isolating the guest
http://wiki.qemu.org/Documentation/Networking

and I was able to pass through a USB keyboard & mouse by bus # and addr #, which is good for whom got identical keyboards,
the command would be

 -usb -device usb-host,hostbus=#,hostaddr=# 

this was a helpful doc from opensus:
http://doc.opensuse.org/products/draft/ … nning.html

Offline

#2489 2014-08-12 18:22:48

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally I got an video output on my passedthrough GPU with the following commandline:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Now I want to transfer this settings to an existing Win7 VM controlled via WebVirtmg. Here is my current configuration file:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7_test_vm</name> <uuid>a4040ddc-cb89-b885-5758-7cd84af1fc16</uuid> <description>None</description> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type> <boot dev='hd'/> <boot dev='cdrom'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/lvmlocal/win7_test_vm'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/www/webvirtmgr/images/Microsoft.Windows.7.Ultimate.SP1.x64.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/www/webvirtmgr/images/virtio-win-0.1-81.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='00:54:00:3d:df:01'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'/> <qemu:commandline> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-M'/> <qemu:arg value='q35'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='ahci,bus=pcie.0,id=ahci'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

If I try to power up the VM I get this error:

internal error: process exited while connecting to monitor: qemu-system-x86_64: -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2: Bus 'pci.0' not found 

Anyone got an idea how to fix this?

Offline

#2490 2014-08-12 22:50:38

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ monchi, the -smp should be 2, same as the # cores.
could you try to boot up from cmd first, just to make sure..?

Offline

#2491 2014-08-12 22:53:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

@ monchi, the -smp should be 2, same as the # cores.
could you try to boot up from cmd first, just to make sure..?

You can actually just specify "-smp sockets=x,cores=y,threads=z", QEMU can do the math to figure out the total.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2492 2014-08-12 23:14:57

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
abdullah wrote:

@ monchi, the -smp should be 2, same as the # cores.
could you try to boot up from cmd first, just to make sure..?

You can actually just specify "-smp sockets=x,cores=y,threads=z", QEMU can do the math to figure out the total.

will be testing that soon, thanks for the info, but what about the number just after the -smp tag?

I managed to set-up a windows 8 VM with Nvidia 760 GTX, updated to the latest drivers (no code errors), and I'm just working to get Opensuse to work with the 8500 GT. If I got the chance, I will be adding AMD 4790 HD ( which worked before I bought the 760 GTX ), I'm on Linux Mint 17 ( you need to add the packages yourself from debian testing and make sure to install the right tools, I did it through synaptic).

a side note, I got my windows key blocked, as a nice advise, isolate the windows guest with:

 -netdev user,id=mynet0,restrict=y 

Offline

#2493 2014-08-12 23:19:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:
aw wrote:
abdullah wrote:

@ monchi, the -smp should be 2, same as the # cores.
could you try to boot up from cmd first, just to make sure..?

You can actually just specify "-smp sockets=x,cores=y,threads=z", QEMU can do the math to figure out the total.

will be testing that soon, thanks for the info, but what about the number just after the -smp tag?

That's what I'm saying, it's unnecessary.  Once upon a time, -smp # was all QEMU understood.  Then people wanted to specify topology, so along came sockets, cores, and threads.  Obviously we can figure out the number of CPUs given just the latter, but the old way still exists for compatibility, and I suppose if you wanted to under populate your core... TBH, I'm not sure what happens when they're out of sync because I don't specify both.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2494 2014-08-13 02:58:53

shazeal
Member
From: New Zealand
Registered: 2007-06-05
Posts: 339

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fira wrote:

- Synergy for mouse/keyboard sharing
Little offtopic, but If you plan to use Synergy, be aware that the AltGr key seems to be buggued...
Even applying the workarounds found around the net, it doesn't seem to help much.
I chose to just live with it and keep @s to copy/paste around : p
Otherwise perfs are very good. Just be aware you need to lock the mouse to your screen to use mouse-grabbing apps or they glitch.

Thanks for this, never knew about Synergy before, just made my life a zillion times easier. I was planning on getting a mechanical KVM but this... so much better!

One part I did not really understand is the locking of the mouse? How can you do this without the SDL window?

Offline

#2495 2014-08-13 08:16:21

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shazeal wrote:
Fira wrote:

- Synergy for mouse/keyboard sharing
Little offtopic, but If you plan to use Synergy, be aware that the AltGr key seems to be buggued...
Even applying the workarounds found around the net, it doesn't seem to help much.
I chose to just live with it and keep @s to copy/paste around : p
Otherwise perfs are very good. Just be aware you need to lock the mouse to your screen to use mouse-grabbing apps or they glitch.

Thanks for this, never knew about Synergy before, just made my life a zillion times easier. I was planning on getting a mechanical KVM but this... so much better!

One part I did not really understand is the locking of the mouse? How can you do this without the SDL window?

Synergy can lock input on particular screen with key bind.

Offline

#2496 2014-08-13 23:19:53

deivid
Member
Registered: 2014-08-13
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got this to work with vainilla 3.16, an existing windows installation BSOD's on boot, but I'll just reinstall and try again.

My question is ( I did not read the 100 pages of the thread so don't kill me if this has been posted before):

I am running this on a headless server and will try to use steam in home streaming to use it. Is there any way to disable the screen output? Just seems like a waste

Offline

#2497 2014-08-13 23:26:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deivid wrote:

I got this to work with vainilla 3.16, an existing windows installation BSOD's on boot, but I'll just reinstall and try again.

My question is ( I did not read the 100 pages of the thread so don't kill me if this has been posted before):

I am running this on a headless server and will try to use steam in home streaming to use it. Is there any way to disable the screen output? Just seems like a waste

You just need to read the first page to know why that might not work.

Offline

#2498 2014-08-14 13:11:20

d3lxa
Member
Registered: 2014-07-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I applied the VGAARB patch of aw on 3.15 (and enabled i915.enable_hd_vgaarb=1) I had my host DRI disabled. I've read some posts saying people applying 81b5c7bc and 6e1b4fdad but it was back in 2013, what's the status right now? Is this normal to not have DRI although the patch purpose is to lie to Xorg so he does not see it?

I can say it solve the color glitch when the VM startup but the host cannot wake up reliably after suspend (tried twice, one failed). I could try a bit more to see if I was unlucky.

Offline

#2499 2014-08-14 14:51:29

irtimmer
Member
Registered: 2013-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DRI2 is still disabled when the VGA arbiter is used. But it looks like DRI3 don't suffer from that problem. Although is currently a little bit unstable and you have to recompile xf86-video-intel


I can read and write in English, but my english grammar is just horrible.

Offline

#2500 2014-08-14 14:52:23

monchi
Member
Registered: 2014-07-21
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
abdullah wrote:

@ monchi, the -smp should be 2, same as the # cores.
could you try to boot up from cmd first, just to make sure..?

You can actually just specify "-smp sockets=x,cores=y,threads=z", QEMU can do the math to figure out the total.

thanks for the info. It works. But the error when trying to start the VM from webvirtmgr remains:

internal error: process exited while connecting to monitor: qemu-system-x86_64: -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2: Bus 'pci.0' not found

I think its a configuration failure. Here is my XML file again:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7_test_vm</name> <uuid>a4040ddc-cb89-b885-5758-7cd84af1fc16</uuid> <description>None</description> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type> <boot dev='hd'/> <boot dev='cdrom'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/lvmlocal/win7_test_vm'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/www/webvirtmgr/images/Microsoft.Windows.7.Ultimate.SP1.x64.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/www/webvirtmgr/images/virtio-win-0.1-81.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='00:54:00:3d:df:01'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'/> <qemu:commandline> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-M'/> <qemu:arg value='q35'/> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='ahci,bus=pcie.0,id=ahci'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

Last edited by monchi (2014-08-14 15:05:08)

Offline

#2501 2014-08-14 21:50:31

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I dont know if this is the correct place to post this, but since aw (and perhaps others contributing to qemu etc) is here:

Fully updated qemu(not git, but qemu 2.1.0-2) and nvidia drivers in windows 340.52.
I cant use game optimization nor shadowplay since the only thing I lack is the CPU requirements. (However, I do have an i7 Haswell CPU, with kvm=off in my qemu settings)

Before I upgraded my qemu(and before using kvm=off) and before using the latest nvidia drivers, this worked fine.

In device manager, under processors - I can see "Intel Core Processor (Haswell), but I believe there should be standing something else here, but quite similar - so that the nvidia drivers "accept" me having the correct CPU?

Anyone else having the same issue? (PS: I dont use these features that often, however I'm somewhat pedantic about making things just right. Keep up the good work!).

Offline

#2502 2014-08-14 23:59:12

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

I dont know if this is the correct place to post this, but since aw (and perhaps others contributing to qemu etc) is here:

Fully updated qemu(not git, but qemu 2.1.0-2) and nvidia drivers in windows 340.52.
I cant use game optimization nor shadowplay since the only thing I lack is the CPU requirements. (However, I do have an i7 Haswell CPU, with kvm=off in my qemu settings)

Before I upgraded my qemu(and before using kvm=off) and before using the latest nvidia drivers, this worked fine.

In device manager, under processors - I can see "Intel Core Processor (Haswell), but I believe there should be standing something else here, but quite similar - so that the nvidia drivers "accept" me having the correct CPU?

Anyone else having the same issue? (PS: I dont use these features that often, however I'm somewhat pedantic about making things just right. Keep up the good work!).

How '-cpu ...' looks like? With '-cpu host,...'  (host-passthrough if you're using libvirt) it should work fine.

Last edited by dwe11er (2014-08-15 11:14:32)

Offline

#2503 2014-08-15 09:46:17

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
trimm wrote:

I dont know if this is the correct place to post this, but since aw (and perhaps others contributing to qemu etc) is here:

Fully updated qemu(not git, but qemu 2.1.0-2) and nvidia drivers in windows 340.52.
I cant use game optimization nor shadowplay since the only thing I lack is the CPU requirements. (However, I do have an i7 Haswell CPU, with kvm=off in my qemu settings)

Before I upgraded my qemu(and before using kvm=off) and before using the latest nvidia drivers, this worked fine.

In device manager, under processors - I can see "Intel Core Processor (Haswell), but I believe there should be standing something else here, but quite similar - so that the nvidia drivers "accept" me having the correct CPU?

Anyone else having the same issue? (PS: I dont use these features that often, however I'm somewhat pedantic about making things just right. Keep up the good work!).

How '-cpu ...' looks like? With'-cpu host,...'  (host-passthrough if you're using libvirt) it should work fine.

Ah, stupid me!
I had -cpu Haswell instead of -cpu host in my script. Thanks.

Offline

#2504 2014-08-15 23:18:49

Hspasta
Member
Registered: 2011-12-24
Posts: 187
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm running into that silly code 43 error for Nvidia on Windows. Nvidia driver version: 340.52.

I'm using libvirt to run everything because my server is headless.
I'm using qemu-git and a vanilla kernel.

Following are my settings for a win8 vm:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>38176c7c-d25c-4cb0-aac5-107056c345f8</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static' current='1'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/zfs/vmpool/win8.qcow2'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/zfs/share/isos/en_windows_8_1_enterprise_x64_dvd_2971902.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:90:71:8c'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ac97'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/> </qemu:commandline> </domain>

Did I miss something? Am I setting the qemu arg settings correctly? Does the video model type matter?

Last edited by Hspasta (2014-08-16 00:07:00)

Offline

#2505 2014-08-15 23:28:22

kameloc
Member
Registered: 2014-06-28
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hspasta wrote:

I'm running into that silly code 43 error for Nvidia on Windows. Nvidia driver version: 340.52.

I don't have a solution but I had the same problem on Windows 7 x64 when downloading and installing the Nvidia drivers. When I use the drivers supplied through windows update (335.23) it works fine. Something to try.

Offline

#2506 2014-08-16 02:05:10

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

<qemu:arg value='-cpu'/>
    <qemu:arg value='host,kvm=off'/>

for 340+ nvidia drivers

edit... i saw u got this...

then u can't have spice (video QXL) i think... thats reason for error 43

Last edited by slis (2014-08-16 02:15:55)

Offline

#2507 2014-08-17 15:11:05

k0olfir3
Member
Registered: 2014-07-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

with the help of this thread, I managed to create a gaming vm to play windows games while linux is running. Everything is working except for one aspect. I am not sure whether this is on topic here, but I don't know where to ask. Sorry for that.

I want to use only one keyboard and only one mouse to control both, the host and the windows guest. So far I have used synergy to share the host's mouse and keyboard with the vm. However, this is somewhat suboptimal since I cannot use all mouse buttons on the guest and because the mouse is not reacting very precisely. I guess it is okay for most use cases but not for gaming. One solution to this would be to give the mouse and keyboard to the vm and to run the synergy server in the guest to share the guest's controls with the host. However, when I try to run the synergy client on the host it is not able to connect to the synergy server on the guest (NOTE: connecting to '10.0.2.15': 10.0.2.15:24800 ... WARNING: failed to connect to server: Timed out). Is there any possibility to solve this?

My qemu command is:

 qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -net nic -net user,smb=/home/xxxx \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/vboxes/win_root.qcow2,id=disk,format=qcow2 -device scsi-hd,drive=disk \ -drive file=/mnt/vboxes/win_games.raw,id=disk2,format=raw -device scsi-hd,drive=disk2 \ -usb -usbdevice host:045e:0719 -usbdevice host:1e7d:2e22 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Apart from synergy, I only use networking to provide the guest with Internet access and to share my home folder. Should I try one of the more sophisticated networking approaches (e.g., bridges)? Do you have any other suggestions?

Best k0ol

Offline

#2508 2014-08-17 16:43:26

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

k0olfir3 wrote:

Apart from synergy, I only use networking to provide the guest with Internet access and to share my home folder. Should I try one of the more sophisticated networking approaches (e.g., bridges)? Do you have any other suggestions?

Best k0ol

imo, you really should try going for bridges

there was a link somewhere above (not more than few pages) for qemu/kvm networking wiki page

Offline

#2509 2014-08-18 03:01:36

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sadly it doesn't work for me. I've followed all the instructions given by nbhs. I would really appreciate some help.

CPU: I5 4570
Mainboard: Asrock H97M Pro 4 / vt-d & vt-x enabled / onboard as primary / dual gpu disabled / UEFI mode
GPU: Asus R9 270
RAM: 16 GB DDR3
OS: Arch Linux / 3.16.0-1-mainline by nbhs (no changes have been made, except arch => core2 or newer)
qemu 2.1.0-2 / seabios 1.7.5-2

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=9fb159a7-84b2-401b-a745-8199d3b0339c rw vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6811,1002:aab0 resume=UUID=9416fa53-b808-4b96-bd92-6839effde5c5 quiet

cat /etc/modprobe.d/blacklist.conf

blacklist radeon

dmesg | grep pci-stub

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=9fb159a7-84b2-401b-a745-8199d3b0339c rw vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6811,1002:aab0 resume=UUID=9416fa53-b808-4b96-bd92-6839effde5c5 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=9fb159a7-84b2-401b-a745-8199d3b0339c rw vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6811,1002:aab0 resume=UUID=9416fa53-b808-4b96-bd92-6839effde5c5 quiet
[    0.459556] pci-stub: add 1002:6811 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.459563] pci-stub 0000:01:00.0: claimed by stub
[    0.459565] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.459581] pci-stub 0000:01:00.1: claimed by stub

lpsci -n (shortened)

01:00.0 0300: 1002:6811
01:00.1 0403: 1002:aab0

cat /etc/vfio-pci.cfg

DEVICES="0000:01:00.0 000:01:00.1"

When I enter this into my terminal ->

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get this error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Offline

#2510 2014-08-18 04:28:24

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@somearchfan

missing intel_iommu=on in cmdline ?

Offline

#2511 2014-08-18 04:45:48

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks! It works now with "intel_iommu=on".

Offline

#2512 2014-08-18 05:13:07

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One more quick question:

How do I get a useful USB ID for qemu? I entered "usb-devices | egrep '(Manufacturer|SerialNumber)'" into my terminal and got the manufacturer (in this case Logitech) and a serial number (C0030059D31E) but not a correct PCI-ID. C0030059D31E is of no use. How can I find out what ID is to what device assigned specifically?

One example
[john@mainhub ~]$ usb-devices | egrep '(Manufacturer|SerialNumber)'
S:  Manufacturer=Linux 3.16.0-1-mainline ehci_hcd
S:  SerialNumber=0000:00:1a.0
S:  Manufacturer=Linux 3.16.0-1-mainline ehci_hcd
S:  SerialNumber=0000:00:1d.0
S:  Manufacturer=Linux 3.16.0-1-mainline xhci_hcd
S:  SerialNumber=0000:00:14.0
S:  Manufacturer=Logitech
S:  SerialNumber=C0030059D31E
S:  Manufacturer=LGE
S:  SerialNumber=030a483608e7ef63
S:  Manufacturer=Logitech
S:  Manufacturer=Linux 3.16.0-1-mainline xhci_hcd
S:  SerialNumber=0000:00:14.0
S:  Manufacturer=Western Digital
S:  SerialNumber=575843314131343535353730

Offline

#2513 2014-08-18 05:37:13

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

somearchfan wrote:

How can I find out what ID is to what device assigned specifically?

lsusb
or, if you like trying to drink from a firehose:
lsusb -v  smile


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#2514 2014-08-18 05:47:55

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow. Great. It works.

Thank you.

Edit: OK. I have successfully installed Windows 8.1. But after installing the AMD graphics driver my monitor goes into power saving mode and doesn't awake anymore. My graphics card has been detected by the driver but something doesn't work.

This is what I get with windows 7 after installing the driver: 79159015.jpeg

Last edited by somearchfan (2014-08-18 12:28:46)

Offline

#2515 2014-08-18 13:44:23

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

somearchfan wrote:

Wow. Great. It works.

Thank you.

Edit: OK. I have successfully installed Windows 8.1. But after installing the AMD graphics driver my monitor goes into power saving mode and doesn't awake anymore. My graphics card has been detected by the driver but something doesn't work.

This is what I get with windows 7 after installing the driver: http://www1.xup.to/tn/2014_08/79159015.jpeg

Are you passing ATI's HD Audio controller? Just bind it to vfio-pci and leave it alone, passing just VGA controller.

Offline

#2516 2014-08-18 14:09:55

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I did. Without passing the HD controller I get this under Windows 8.1:
42552220.jpeg

My config:
qemu-system-x86_64 -enable-kvm -M q35 -m 8196 -cpu host -boot d \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/john/Downloads/Asus.R9270.2048.131007.rom \
-device virtio-scsi-pci,id=scsi \
-drive file=/run/media/john/win//windows.img,id=disk,format=raw -device scsi-hd,drive=disk \
-drive file=/home/john/Desktop/Win7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/home/john/Downloads/virtio-win-0.1-81.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd \
-usb -usbdevice host:046d:c019 -usbdevice host:046a:0021

Offline

#2517 2014-08-18 20:37:07

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

that BSOD is from Windows itself, you need to go to device manager and disable the graphic from what window, then do the installtion for the drivers.

Offline

#2518 2014-08-18 22:45:26

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

this doesn't sound anything. I still get the BSOD while installing the graphics driver. ;o

Offline

#2519 2014-08-18 23:10:01

somebuddy
Member
Registered: 2014-08-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't get audio working right in Windows 8.1 as a guest, I get audio output but it is distorted.  I've tried both alsa (with libspeex_mediumrate resampler), and pulseaudio as an alsa pipe, as described at: https://wiki.archlinux.org/index.php/Pu … pe_to_ALSA
I've also tried installing Realtek audio drivers on the guest, it doesn't help at all.

I'm using an Asrock Z97 Extreme 3, which uses Realtek ALC892 for audio.

Qemu command:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \ sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 \ -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=4 \ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -boot menu=on \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -no-hpet \ -usb -usbdevice host:413c:3200 \ -usbdevice host:09da:0260 \ -usbdevice host:0810:0001 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -drive file=/home/someguy/VM/win8.1.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk 

Any ideas?

Offline

#2520 2014-08-19 01:48:35

darkstego
Member
Registered: 2014-08-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I haven't been able to get my passthrough video card to display anything through qemu. I have an nvidia gtx 770 to passthrough to my Windows 8.1 guest while the host is using video from my Intel i5 processor. I am running linux 3.16 with the i915, acs and vga arbiter patches. I have the guest running on qemu 2.0 and I can expose the video card to it and see it in device manager (it is disabled due to Code 43 error).

The problem is I can never seem to get any output from the nvidia card, not even the seabios. I have the card on the first pcie slot, and set the primary output to igfx in the UEFI/bios.

/proc/cmdline

intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 pcie_acs_override=downstream

I try to test the passthrough by running the following

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on  \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

but I don't get any output from the video card.

I do however get the following in dmesg

[  108.138741] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[  108.138896] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  110.371251] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[  110.371302] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[  110.371308] [drm:hsw_unclaimed_reg_check] *ERROR* Unclaimed write to 44040

Any ideas what I may be doing wrong?

Offline

#2521 2014-08-19 02:17:51

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@darkstego

you are missing i915 patch activation in cmdline....

and kvm=off option in qemu for 325+ nvidia driver

Last edited by slis (2014-08-19 02:18:07)

Offline

#2522 2014-08-19 02:22:07

darkstego
Member
Registered: 2014-08-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@slis, thanks, I just enabled it.

I was unaware that there is a kvm=off option for new drivers. I will be sure to add that.

Offline

#2523 2014-08-19 02:46:13

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My configuration works now. Special thanks to nbhs, abdullah, dwe11erm and ewaller.

Last edited by somearchfan (2014-08-19 02:47:03)

Offline

#2524 2014-08-19 04:30:58

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Im currently in the process of setting up my system and I am wondering does anyone have a way to disable your second monitor in linux when you start the vm and use your 2nd vga card to drive the second monitor? I would like my window manager/desktop environment to transfer the workspaces belonging to the second screen to my primary screen.

Offline

#2525 2014-08-19 09:42:14

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

somebuddy wrote:

I can't get audio working right in Windows 8.1 as a guest, I get audio output but it is distorted.  I've tried both alsa (with libspeex_mediumrate resampler), and pulseaudio as an alsa pipe, as described at: https://wiki.archlinux.org/index.php/Pu … pe_to_ALSA
I've also tried installing Realtek audio drivers on the guest, it doesn't help at all.

I'm using an Asrock Z97 Extreme 3, which uses Realtek ALC892 for audio.

Qemu command:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \ sudo qemu-system-x86_64 ...

Any ideas?

why not just pa?
why not try varying parameters being used (not to mention learning what they are supposed to mean)?
for gods sake...

sitonapanotis wrote:

Im currently in the process of setting up my system and I am wondering does anyone have a way to disable your second monitor in linux when you start the vm and use your 2nd vga card to drive the second monitor? I would like my window manager/desktop environment to transfer the workspaces belonging to the second screen to my primary screen.

xrandr?

Offline

#2526 2014-08-19 13:24:43

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm having issues too, and hoping someone could help please.
I've tried lots of combinations, so probably screwed something simple up in the process.
I followed the ubuntu 14.04 guide here:
http://www.pugetsystems.com/labs/articl … 4-KVM-585/
That is based on this arch thread, and adjusted for Ubuntu.
I can pass through the HD7770 and audio, and both show up in device manager.
But once I install the 14.4 ATI drivers (either catalyst custom install or pointing Windows driver update to extracted files) I cannot reboot unless I use safe mode. And my keyboard is not always recognised, which makes it even more difficult.

I have the following hardware (been proven to work in EXSi)
GA-X58-UD5 board
E5520 cpu
BIOS supports passthrough also
primary card HD5450
passthrough HD7770
Ubuntu 14.04 x64 server
Win7x64 kvm VM

simon@cavetroll:~$ qemu-system-x86_64 -version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.2), Copyright (c) 2003-2008 Fabrice Bellard

simon@cavetroll:~$ cat /etc/initramfs-tools/modules
...
pci_stub ids=1002:683d,1002:aab0

simon@cavetroll:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.16.0-031600-generic root=UUID=a03285ef-5e8c-4182-b517-34f9717cbd18 ro quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7

I tried on the vanilla 3.12 kernal, then also tried installing the 3.16 as noted on page 1, no difference

HD7770 is card captured by pci-stub

simon@cavetroll:~$ dmesg | grep pci-stub
[    2.064508] pci-stub: add 1002:683D sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.064524] pci-stub 0000:03:00.0: claimed by stub
[    2.064534] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.064543] pci-stub 0000:03:00.1: claimed by stub

simon@cavetroll:~$ lspci -nn |grep ATI
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d]
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1]
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

simon@cavetroll:~$ ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/
0000:03:00.0  0000:03:00.1

Can then capture video card and usb manually (script fails for me, but no need yet)

sudo vfio-bind 0000:03:00.0 0000:03:00.1
sudo vfio-bind 0000:00:1a.0 0000:00:1a.1 0000:00:1a.2 0000:00:1a.7

And can start a VM with display on HD7770 as expected, boots & installs windows7, but after installing ATI driver it fails to reboot.
I get the error recovery screen but can log in (sometimes, when keyboard works) with safe mode, so assume driver issue?

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 6144 -cpu host \
-smp 4,sockets=1,cores=2,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-device vfio-pci,host=00:1a.1,bus=pcie.0 \
-device vfio-pci,host=00:1a.2,bus=pcie.0 \
-device vfio-pci,host=00:1a.7,bus=pcie.0 \
-drive file=/home/simon/win7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/simon/Downloads/Win7_64.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on

Also tried IDE and virtio, on image file and partition, same result.
I assume I'm missing something simple?

Offline

#2527 2014-08-19 15:31:15

aldum
Member
Registered: 2012-03-14
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

somebuddy wrote:

I can't get audio working right in Windows 8.1 as a guest, I get audio output but it is distorted. 
Any ideas?

What sinny said, try messing around with the parameters. It's a trial-and-error thing, can't really get it right otherwise.

Here's what's been working for me (Win7 guest):

QEMU_ALSA_DAC_BUFFER_SIZE=1024 QEMU_AUDIO_DAC_FIXED_FREQ=48000 \ QEMU_AUDIO_ADC_FIXED_FREQ=48000 QEMU_ALSA_DAC_BUFFER_SIZE=16384 \ QEMU_AUDIO_DRV=pa \ qemu-system-x86_64 -enable-kvm -M q35 -m 4G -cpu host,hv-time,kvm=off \ (...) -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -soundhw hda,ac97 \

Offline

#2528 2014-08-19 15:32:07

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey there smile
I recently broke my RAID when actually testing it (tip of the day: don't use FakeRAID, it can't do online recovery/degraded startup, at least at Intel's).

Reinstalled my whole Virt/Passthrough setup and I wanted to update on some things I figured out:
- I profited of the occasion to downgrade from Windows 8.1 to Windows 7, hoping things would turn out better. Bad idea. The bootloader kept breaking for random reasons, even once Windows Update was turned off. Tip for anyone experimenting with this kind of problem: use LVM, make snapshots. You'll gain a lot of time rather than reinstalling Win$ every time (in my case, restoring GPT, Bootloader, and Boot partition wasn't enough).
- Be careful with the versions of graphics drivers you run in your guest. For exemple i had a lucky streak with my first install because my GTX 750 Ti doesn't seem to run at all under passthrough when using versions above 335. It's probably always better to undershoot.
- I added kernel boot args to capture the card to PCI-Stub before passing it to VFIO to ensure nothing would get loaded. This is important for nVidia cards it seems because otherwise they can load an audio driver for the sound sub-card.
- A little important gotcha if you run PulseAudio and get connection issues, PulseAudio uses a cookie to authenticate users (though that might not be the case when running Policykit ?)... Because VFIO access requires root, and your pulseaudio server is probably running under a different user, the easiest workaround i found to that is to simply add a pulse input over TCP for 127.0.0.1 and add in /root/.pulse/client.conf the variable default-server=127.0.0.1 (or you could use PULSE_SERVER env variable).

Also a little errata. I said you need to lock your mouse to a screen when using Synergy with Mouse grabbing apps. That's true, but this will only work if you set the mouse movement type to relative rather than absolute in the Settings. Otherwise anything grabbing mouse expecting relative movements (hint: any FPS) will behave plain poorly...

Good luck !

Last edited by Fira (2014-08-19 15:36:50)

Offline

#2529 2014-08-19 23:56:01

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all

I need some help here. I've tried to set up hugepages but no luck. My computer crashes each and every time I start qemu w/ hpages enabled.
63559622.jpeg

cat /etc/fstab

UUID=B039-B9D4 /boot vfat defaults,rw,relatime 0 0
UUID=9416fa53-b808-4b96-bd92-6839effde5c5 swap swap defaults 0 0
UUID=8038e49a-447d-4955-823f-803d6e0c3e32 /home ext4 defaults,rw,relatime,data=ordered 0 0
UUID=9fb159a7-84b2-401b-a745-8199d3b0339c / ext4 defaults,rw,relatime,data=ordered 0 1
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
hugetlbfs       /dev/hugepages  hugetlbfs       mode=1770,gid=78        0 0

mount | grep huge

hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,mode=1770,gid=78)

grep HugePages_Total /proc/meminfo

HugePages_Total:    4096

My config:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64 -enable-kvm -M q35 -m 8196 -mem-path /dev/hugepages -cpu host -boot d \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/john/Downloads/Asus.R9270.2048.131007.rom \
-device virtio-scsi-pci,id=scsi \
-drive file=/run/media/john/win/windows.img,id=disk,format=raw -device scsi-hd,drive=disk \
-drive file=/home/john/Downloads/virtio-win-0.1-81.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd \
-usb -usbdevice host:046d:c051 -usbdevice host:046a:0021 \
-net tap,vlan=0,ifname=tap0 -net nic,model=virtio,vlan=0 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0


Any ideas?

Last edited by somearchfan (2014-08-20 00:06:01)

Offline

#2530 2014-08-20 02:28:35

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wimma77
I run a similar setup (using kUbuntu, tried server and wanted to run desktop n an LXC container - without much success) .... and have just spent 2+ days resolving a similar issue. The vm was working fine till the end of last week
Anyway, what I found was that the most useful debugging information came from a Ubuntu vm which I created specially to test whetehr or not the problem was Windows specific.  Ubuntu startup information pointed to a sound problem, so I removed the AMD sound device passthrough (03:00.1 in your case) ... and then  itbooted ok Very odd since it had been working for a couple of months until then. The other consequence was that the dx9 graphics performance then improved by about 20% ... particularly strange since the card worked perfectly in another PC booting Windows directly, so unlikely to be a hardware issue

Anyway, if you have the disk space create a Linux (ubuntu ?) VM to see if it indicates where to look,  The other thing to be conscious of is the need to completely remove graphics drivers using something like http://www.guru3d.com/files-details/dis … nload.html before trying again

Offline

#2531 2014-08-20 02:43:38

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@redger
Thanks so much for the suggestion - will try that. But I thought you had to pass the whole iommu group through? Or just pass them both to vfio?
My card & audio share a group.
ie ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/
0000:03:00.0  0000:03:00.1
I will try though. Good idea on the ubuntu VM too - definitely worth checking.
Can I just ask - did you use just vanilla kernel and qemu?
I have a headless server, but did install xfce4 so I could VNC to it and use a GUI when required.
Would love to be able to start the vm from cli but have not researched enough yet on how to accomplish that.
Cheers.

Last edited by Wimma77 (2014-08-20 02:44:32)

Offline

#2532 2014-08-20 03:27:37

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wimma77
you don't need to pass the whole group BUT you cannot pass parts of a group to fdifferent VMs
I start my VMs from the CLI using a simple script .. looks like this (ignore extra lines at the top, I keep them for debugging & experimentation)

#!/bin/sh # Real command # sudo sh /etc/init.d/vfio-bind-init.sh # qemu-system-x86_64 \ # QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 \ QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64 \ -name win7 -enable-kvm -M q35 -m 6144 -mem-path /dev/hugepages \ -cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -rtc base=localtime \ -boot menu=on \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -usbdevice tablet \ -spice port=5902,disable-ticketing \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device ahci,bus=pcie.0,id=ahci \ -net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \ -net user \ -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device ich9-intel-hda,bus=root.2 \ -device hda-duplex,bus=hda.0 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.0,bus=pcie.0 \ -drive file=/dev/ssd-virt/lvwin7_kvm,id=disk1,format=raw,if=virtio \ -drive file=/mnt/programming_data/isos/virtio-win-0.1-74.iso,id=isocd2 \ -device ide-cd,bus=ahci.2,drive=isocd2 \ -drive file=/mnt/programming_data/isos/winfiles_2.iso,id=isocd3 \ -device ide-cd,bus=ahci.3,drive=isocd3

which I then added to the KDE menu - to be executed as root .... but you could easily adjust to enable direct execution. I have similar scripts fopr binding the VFIO devices and for strating spice (useful when debugging)

I used qemu etc. from the Ubuntu repos but I patched the kernel using the following script -

#!/bin/bash # patch --dry-run --verbose -p 1 -i re_xxxxxxxxxxxxx echo echo ... PATCHING ... VGA Arbiter patch -b -p 1 -i re_patch_01_i915_313rc4.patch echo echo ... PATCHING ... acs override patch -b -p 1 -i re_patch_02_override_for_missing_acs_capabilities.patch echo echo ... PATCHING ... memleak patch -b -p 1 -i re_patch_03_fix_memleak.patch echo echo ... PATCHING ... read DR6 patch -b -p 1 -i re_patch_04_fix_reading_of_DR6.patch echo echo ... PATCHING ... debug registers - has problem, needs to follow DR6 patch patch -b -p 1 -i re_patch_05_debug_registers.patch # patch -b -p 1 -i re_debug_registers_RE.patch # Corrected to add additional lines before DR6 patch runs echo echo ... PATCHING ... kernel__gcc patch -b -p 1 -i re_patch_06_kernel-38-gcc48-2.patch

Generally I copy the "mainline" file nbhs posts and extract the patches for each kernel generation, just be be sure. So I have the patches he posted for 3.13 and I have patched kernels for Ubuntu releases 24, 28, 32, 34 and I have copied patches for 3.13, 3.15, 3.15 (missed 3.14 somehow)

Here are my notes for patching the kernel, if they're of any use

To compile a new kernel https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel https://help.ubuntu.com/community/Kernel/Compile https://wiki.ubuntu.com/KernelTeam/KernelMaintenance https://www.debian.org/releases/stable/i386/ch08s06.html.en architecture is "amd64" or "x86_64" 1) download the source into the current directory using "apt-get source linux-image-xxxxx" where xxxx is the name of the version eg. apt-get source linux-image-3.13.0-32-generic This should download the tarball(s) and extract the source code into a directory (which should be renamed immediately because all versions use the same directory name !!!) 2) Open the new directory, clean up and prepare chmod a+x debian/scripts/* chmod a+x debian/scripts/misc/* fakeroot debian/rules clean and generate the required config by either - a) editing using "fakeroot debian/rules editconfigs" (for all targets, one after another) ... or b) "make menuconfig" and work through the various options. Remember to save before exit c) copy the current config from the boot directory as ".config" in the root direcotry for the new kernel and then use "make oldconfig" ... which will ask a question for the value of each new config option Required config options are CONFIG_VFIO_IOMMU_TYPE1=y in Device Drivers. 2 pages back from end CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_PCI_STUB=y in Bus Options, second page down HZ_1000=y in Processor Type & Features (last page) PREEMPT=voluntary 3) apply any patches /// remember to verify that they worked ok (look for fail) "sh re_apply_patches.sh > re_output_patch.txt" 4) "fakeroot debian/rules clean" to remove any old build information / files 5) Ignore modules not created with new parameters ... copy re_modules.ignore to ...debian.master/abi/<previous-version>/modules.ignore and ignore other ABI errors by copying re_prevrelease_arch_ignore (rename to "ignore") to debian.master/abi/<previous-version>/<arch> eg. to debian.master/abi/3.13.0-32.56/amd64/ 6) "DEB_BUILD_OPTIONS=parallel=3 skipmodule=true fakeroot debian/rules binary-headers binary-generic > re_output_compile.txt" to generate the deb files which can be installed (second thoughts don't pipe the poutput to a file - it will prevent selection of the CPU type) 7) Install all the Debs with command "sudo dpkg -i linux*<ver>*.deb eg. sudo dpkg -i linux*3.13.0-32_3.13.0-32.57*.deb sudo dpkg -i linux*3.13.0-32-generic_3.13.0-32.57*.deb 8) go into Synaptic and lock all the newly installed elements (linux-image*, linux-header*, linux-tool*, linx-cloud-tool*) - to prevent the new kernel and components being overwritten in the next upgrade

Offline

#2533 2014-08-20 03:36:11

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@somearchfan
have you tried either (a) increasing the size of your hugepages or (b) decreasing the amount of memory allocated to the VM
The hugepages mechanism needs a small amount of RAM for management .. over and above what is allocated to the VM(s). For instance mine is set to

grep HugePages_Total /proc/meminfo HugePages_Total: 3200

for a VM with memory allocation of 6144

qemu-system-x86_64 \ -name win7 -enable-kvm -M q35 -m 6144 -mem-path /dev/hugepages \

Offline

#2534 2014-08-20 04:00:02

deivid
Member
Registered: 2014-08-13
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

using
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I get way lower performance in general and very choppy audio (using alsa). Any clue?

In particular, I get higher in-game ping (from 60ms to ~120ms) and also my network latency increases, the games drop from 60+ fps to ~12-20fps.. I'll keep looking later

Last edited by deivid (2014-08-20 04:14:30)

Offline

#2535 2014-08-20 04:27:24

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@deivid
have a look at my command line a couple of posts above. Note the prefix to qemu right at the beginning and the 3 lines for the sound card ie. add another PCI slot for the card (ioh3420). You could also try Pulseaudio using the commented line at the top of what I posted - tho it performed poorly for me (lag), Alsa was much better performed and host tasks can still play sounds simultaneously via Pulse

Offline

#2536 2014-08-20 05:09:36

deivid
Member
Registered: 2014-08-13
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Basically copied the first line and last 3 lines from your config, but it's the same result (very choppy sound and poor graphics performance, didn't test network).. there must be something obvious I'm missing.
As 'recommended' above I removed my AMD sound device (01:00.1), which wasn't doing anything anyway

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \ qemu-system-x86_64 -enable-kvm -M q35 -m 3200 -cpu host \ -smp 3,sockets=1,cores=3,threads=1 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -usb -usbdevice host:1bcf:0002 \ -usb -usbdevice host:04d9:1602 \ -drive file=/storage/virt/windows.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk \ -vga none -vnc :0 \ -net nic -net tap,ifname=tap0,script=no,downscript=no \ -mem-path /dev/hugepages \ -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device ich9-intel-hda,bus=root.2 \ -device hda-duplex,bus=hda.0

Do you see what am  I doing wrong?

Offline

#2537 2014-08-20 05:26:33

blu3bird
Member
Registered: 2014-08-20
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@somearchfan

It does not crash, it just get's very very very slow because you are ouf of memory ;-)

Using 2MB hugepages adds 16MB overhead per vm => If you want to assign 8192MB to your vm, you'll need 8208MB of hugepages.

PS: You can try 1GB hugepages to avoid this overhead. But that requires extra hardware support: grep pdpe1gb /proc/cpuinfo

Last edited by blu3bird (2014-08-20 05:29:24)

Offline

#2538 2014-08-20 08:13:00

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@deivid
errrr I'm not an expert .... but

look at (a) your disk, it's on an IDE controller whereas it will perform much better using virtio (I create an LVM partition and passed that using virtio) (b) if running a 64 bit Windows it may be short of RAM - 3GB is a bit lean, can you allocate 4GB or more - paging could be killing the vm (c) you appear to have defined a VNC connection which may not help (shouldn't be necessary if you're passing a graphcis card, try using Spice instead). Presumably you're running this on a quad core Intel cpu / motherboard (guess) given that you're passing 3 cores, single thread each and no pinning. If this is an AMD you probably want to (i) allocate an even number of cores and (ii) pin them to avoid NUMA problems ie. cache build / rebuild
Have you set Hugepages appropriately ? See discussion up the page a bit

Also refer to NBHS's tuning comments on the first page. I have added hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 to help improve Windows performance (refered to in early pages - maybe search for them).

perhaps one of the more capable folk on the forum could offer better / more advice

Offline

#2539 2014-08-20 10:20:40

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

somearchfan wrote:

Hi all

I need some help here. I've tried to set up hugepages but no luck. My computer crashes each and every time I start qemu w/ hpages enabled.

yep, hugepages is completely undocumented and almost not yet publically known kernel feature. thread on pci passthrough is a perfect place to whine about you not being able to use google

deivid wrote:

Basically copied the first line and last 3 lines from your config, but it's the same result (very choppy sound and poor graphics performance, didn't test network).. there must be something obvious I'm missing.
As 'recommended' above I removed my AMD sound device (01:00.1), which wasn't doing anything anyway

...

Do you see what am  I doing wrong?

i see what you are doing wrong (more like - it seems for me that it's the only thing you are doing) - being completely ignorant.

you got problems with sound - too high and mighty to pinpoint command line arguments used that are related to sound and read some manuals on their meaning?
got poor graphics performance - too busy to try tons of performance tips that are all over this thread (not to mention google)?
got any problems - too cute to at least try resolving the problem yourself (and i mean really trying to soil your hands digging around the issue)?

p.s. thanks for making the thread less usable with question about something pretty trivial that was already discussed quite a few times

Offline

#2540 2014-08-20 17:17:07

deivid
Member
Registered: 2014-08-13
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

redger wrote:

@deivid
errrr I'm not an expert .... but

look at (a) your disk, it's on an IDE controller whereas it will perform much better using virtio (I create an LVM partition and passed that using virtio)

I'll look into it but I installed windows with the disk set to IDE, then installed drivers; trying to boot on virtio mode  gives me a BSOD and I'd have to format the virtual machine, which I might do anyway tomorrow.

redger wrote:

(b) if running a 64 bit Windows it may be short of RAM - 3GB is a bit lean, can you allocate 4GB or more - paging could be killing the vm

I'm running this with only windows and a game (dota 2/half life) for now, so it's enough; didn't notice any difference between 3 or 4gb (I have 8 but I'm running a few things on the host)

redger wrote:

(c) you appear to have defined a VNC connection which may not help (shouldn't be necessary if you're passing a graphcis card, try using Spice instead). Presumably you're running this on a quad core Intel cpu / motherboard (guess) given that you're passing 3 cores, single thread each and no pinning. If this is an AMD you probably want to (i) allocate an even number of cores and (ii) pin them to avoid NUMA problems ie. cache build / rebuild

VNC connection: I was getting a 'couldn't initialize SDL' on the machine, it's running headless, and this solved it. The vnc session only lets me manage the qemu console.

I'm running on an 8-code 8350, again, no difference with 3/4 cores, I'll look into pinning them.

redger wrote:

Have you set Hugepages appropriately ? See discussion up the page a bit

Also refer to NBHS's tuning comments on the first page. I have added hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 to help improve Windows performance (refered to in early pages - maybe search for them).

perhaps one of the more capable folk on the forum could offer better / more advice

I've set up 2MB hugepages but my mobo supports 1GB hugepages so I'll set it up later (didn't notice any difference with the hugepages on/off).
I'll look into the hv-* parameters.

Thank you

Last edited by deivid (2014-08-20 17:29:27)

Offline

#2541 2014-08-20 17:20:53

deivid
Member
Registered: 2014-08-13
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

you got problems with sound - too high and mighty to pinpoint command line arguments used that are related to sound and read some manuals on their meaning?
got poor graphics performance - too busy to try tons of performance tips that are all over this thread (not to mention google)?
got any problems - too cute to at least try resolving the problem yourself (and i mean really trying to soil your hands digging around the issue)?

p.s. thanks for making the thread less usable with question about something pretty trivial that was already discussed quite a few times

I'm guessing you didn't understand my post. All my problems start if I add the audio device. Without audio device, the VM works wonders, with it, I get poor network performance (LAN and WAN) and poor graphics/CPU performance, and the audio crackles like hell.

They are not separate issues and only happen with the audio device enabled; pci-passthrough is pretty obscure and most 'documentation' comes from guessing. I haven't found anything related to my problem so I posted here

Offline

#2542 2014-08-20 17:45:27

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could someone please help me with compiling kernel with acs override patch on Fedora? I want to compile the latest kernel from "yum update". I have those lines set up already and they are not the best:

yum install yum-utils
yumdownloader --source kernel
yum-builddep kernel-<version>.src.rpm
rpm -Uvh kernel-<version>.src.rpm
cd
cd rpmbuild/SOURCES
tar -xvf ....
cd rpmbuild/SOURCES/linux../
<copy acs.patch here>
patch -p1 < acs.patch
make oldconfig
make
make install

Please someone....

Offline

#2543 2014-08-20 18:29:33

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deivid wrote:
sinny wrote:

you got problems with sound - too high and mighty to pinpoint command line arguments used that are related to sound and read some manuals on their meaning?
got poor graphics performance - too busy to try tons of performance tips that are all over this thread (not to mention google)?
got any problems - too cute to at least try resolving the problem yourself (and i mean really trying to soil your hands digging around the issue)?

p.s. thanks for making the thread less usable with question about something pretty trivial that was already discussed quite a few times

I'm guessing you didn't understand my post. All my problems start if I add the audio device. Without audio device, the VM works wonders, with it, I get poor network performance (LAN and WAN) and poor graphics/CPU performance, and the audio crackles like hell.

They are not separate issues and only happen with the audio device enabled; pci-passthrough is pretty obscure and most 'documentation' comes from guessing. I haven't found anything related to my problem so I posted here

well, i had both choppy audio and poor 3d game fps. resolved this by using isolcpus/cpuset, hugetlbfs and playing around with audio parameters. everything was found in the thread - so you probably never searched or just stopped doing it after few minutes.

devianceluka wrote:

Could someone please help me with compiling kernel with acs override patch on Fedora? I want to compile the latest kernel from "yum update". I have those lines set up already and they are not the best:
yum install yum-utils
yumdownloader --source kernel
yum-builddep kernel-<version>.src.rpm
rpm -Uvh kernel-<version>.src.rpm
cd
cd rpmbuild/SOURCES
tar -xvf ....
cd rpmbuild/SOURCES/linux../
<copy acs.patch here>
patch -p1 < acs.patch
make oldconfig
make
make install
Please someone....

you have "lines" "set up"? i dont even want to think what kind of disorder it might be...
and the "lines" look very weird from fedora perspective - why use yum/rpmbuild at all if you are going to do everything (incorrectly, by the way) by hands later?

excuse me, but i hope noone will spoon-feed you with this info. it's god damned fedora - it even has its' own dedicated wiki with pretty good info (not to mention it being pretty popular/used distro).

why did you switch to fedora if you neither have an idea on its' infrastructure nor willing to use TONS of info on it in the net? search the damned web or stop begging for help (well, technically not for "help", but for someone to do everything for you)

Offline

#2544 2014-08-20 20:56:55

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

somearchfan wrote:

Hi all

I need some help here. I've tried to set up hugepages but no luck. My computer crashes each and every time I start qemu w/ hpages enabled.
http://www1.xup.to/tn/2014_08/63559622.jpeg

...

Any ideas?

Try to allocate bit more, like 5004 or something.

Last edited by dwe11er (2014-08-20 20:57:25)

Offline

#2545 2014-08-21 06:23:19

starlays
Member
Registered: 2011-02-02
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If I have only one GPU available can I use  KVM VGA-Passthrough?
I have Gnome running on the host and I want to create a windows VM for gaming, I'm new to virtualization and I'm curious if It is possible to share the GPU between the host and the guest. All the scenarios that I found for now are involving the presence of 2 video cards, one on CPU and one PCI. My CPU has VT-d, it is  Intel Xeon 1230v3.
Thank you in advance.

Offline

#2546 2014-08-21 08:37:47

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can't run both at the same time on one gpu. Just buy a cheap Radeon 5450 / 6450 for linux.

Offline

#2547 2014-08-21 12:44:07

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

redger wrote:

@Wimma77
you don't need to pass the whole group BUT you cannot pass parts of a group to fdifferent VMs
I start my VMs from the CLI using a simple script .. looks like this (ignore extra lines at the top, I keep them for debugging & experimentation)

which I then added to the KDE menu - to be executed as root .... but you could easily adjust to enable direct execution. I have similar scripts fopr binding the VFIO devices and for strating spice (useful when debugging)

I used qemu etc. from the Ubuntu repos but I patched the kernel using the following script -

Thanks so much for this.
Will try this weekend when I get some quiet time.
Got an nvidia card coming too, so might be a good time to start fresh.
Much appreciated.

Offline

#2548 2014-08-22 02:47:54

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

Thank you for the guide! It finally work after I tried many times, changed many kernel and qemu versions.

So I want to share my experience.

Hardware Info:
Gigabyte Z97-D3H
i5-4570
i915 built-in card for host
Sapphire hd7850 for vm

Software Info:
linux-3.12.4 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Kernel commandline:

linux /vmlinuz-3.12.4-VFIOPassthrough root=UUID=6b2e5409-8b73-40db-ad45-f4ec79cdc0cc quiet
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream
pci-stub.ids=1002:6819,1002:aab0

/etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1
options kvm_intel emulate_invalid_guest_state=0

Qemu commandline:

01:00.0 VGA compatible controller: ATI Technologies Inc Device 6819
01:00.1 Audio device: ATI Technologies Inc Device aab0

/root/vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 -cpu SandyBridge -smp 2,sockets=2,cores=1,threads=1 \
-m 4096 -M q35 -enable-kvm \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/var/lib/libvirt/images/fwq_win7_vfio.img,id=disk,format=raw \
-device ide-hd,bus=piix4-ide.0,drive=disk \
-drive file=/var/lib/libvirt/images/LENOVO_WIN7_64.iso,id=isocd \
-device ide-cd,bus=piix4-ide.1,drive=isocd \
-net nic,model=rtl8139,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \     /* the audio bus must be pcie.0 or you will be failed. */
-usb -usbdevice host:17ef:6018 -usbdevice host:17ef:600e \
-boot order=cd -monitor stdio

PS: the graphic driver for hd7850 that I use [amd_13_9_win7_win8_64_dd_ccc_whql.exe]

Upon this, AMD FirePro R5000 works too, but there is another problem, if I start vm with VFIO passthrough, my host terminal has no response like screen freezed, only ssh works. give me some help, thanks!

Offline

#2549 2014-08-22 03:11:28

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I found it seems like i915 driver. if I blacked i915 driver in blacklist.conf, the host is no respond. but canceled it, it normal.

Offline

#2550 2014-08-22 15:50:40

BahbGTR
Member
From: Kentucky
Registered: 2013-07-12
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got tired of manually patching and compiling the mainline kernel, so I used customizepkg to patch the default linux package automatically.  This may not be the proper way of doing things, but it's the lazy way!  This process would probably work with any kernel in AUR or any of the repositories.  I'm not sure if the patches will.  I would be surprised if they didn't, but you never know until you try. 

Sorry if this has been posted before, but it's super hard to search this thread.  My system specs are in my signature.  I passthrough my nvidia card to my VM, and I use the Intel P4600 graphics on my host.  I have been building the acs override patch, i915 VGA arbiter fix, and the VGA arbiter fix.  I boot using efi stubs and refind.

Please, make sure you understand what the following commands do.  Don't just copy and paste what I have here.  There is a serious risk it could break your install.  Don't drop your existing kernel until you know the patched one is working.  Don't blame me for any broken installs!

Requirements: yaourt, customizepkg, and Abs.

First of all, save the patches you need to apply to a location where the userid that will use yaourt can reach.  I have a build directory in my home folder, so I placed them there.

Second, take a look at How to use ABS.  Copy the kernel you want to patch and open up the PKGBUILD file in your text editor.  Look for the prepare() section.  Specifically where all the patch files are located.  You're going to use customizepkg to recognize where you want your patches to be inserted.  You'll need to copy one of the patch entries.  For the linux package I used the following as my entry point: 

patch -p1 -i "${srcdir}/change-default-console-loglevel.patch"

Open up your favorite text editor and create /etc/customizepkg.d/linux.  The file name should reflect the name of the package you are installing.  If it's linux-ck you should use /etc/customizepkg.d/linux-ck.

In that file, place the following on a single line (very important!)

replace#global#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/i915_316.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/override_for_missing_acs_capabilities.patch\"\\npatch -p1 -i \"\\PATH_TO_PATCHES\\/vga_arbitrer.patch\"

This code snippet will only for work for the linux package.  If you want to use a different kernel package, you're on your own to find the place to insert the patches.

A bit of an explanation of the snippet.  The below is the bit of the PKGBUILD customizepkg watches for.  When this point is reached, everything coming after this point on the same line is added to the PKGBUILD.

replace#global#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"

Below is what is being added to the PKGBUILD.  I added change-default-console-loglevel.patch back in because I still want that patch applied.  If you don't add it back in, that patch will be dropped. 

#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/i915_316.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/override_for_missing_acs_capabilities.patch\"\\npatch -p1 -i \"\\PATH_TO_PATCHES\\/vga_arbitrer.patch\"

Save the file, and now it's time test!

yaourt -S linux

Take a look at the output.  You should see customizepkg doing its thing.  Your three patches should show up a few lines above where yaourt asks if you wan to edit the PKGBUILD.  They probably wont be listed at the start where it receives the file list.  This is OK.  Also, it will tell you if there are any errors.  If there are errors from sed you'll need to tweak your file in customizepkg.d.  All slashes and double quotes need to be escaped.  That's a common error.  If there are no errors, continue the build and see if it completes.  If it compiles and installs properly, try booting from the new kernel.  Finally, try out your VM.

If all goes well, you should be able to update your whole system without having to do anything else manually using:

yaourt -Syua

It's worth watching the first page of this thread to see if any of the patches are changed every now and then.  Also, be careful if the kernel changes from one major kernel version to another (3.16 -> 3.17).  I just starting using this method, so I'm not sure if things will break at that point.

If there is a better, cleaner way of doing this, please let me know!  I recognize this is kind of hackish, but it works well for me so far.

Last edited by BahbGTR (2014-08-22 15:51:28)


Xeon E3 1245v3  |  ASRock Z87 Extreme6  |  20 GB DDR3-1666  |  EVGA Nvidia GTX 560 ti |  MSI Nvidia GTX 760 (passthrough)  |  256 GB Samsung 850 Pro | 120 GB Samsung 830  |  300 GB WD Velociraptor  |  2 TB WD Black

Offline

#2551 2014-08-22 18:47:04

groggster
Member
Registered: 2014-08-22
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!

I need some help with getting this running properly. I am currently able to start up the VM with the graphics card passed along, everyting runs fine. I do however have problems getting virt-manager to boot this machine. How do i achieve that? I have tried manually configuring the .xml config files, but to no avail. This is my configuration file:

export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 qemu-system-x86_64 \ -enable-kvm -M q35 -m 2048 -cpu host -smp 4 \ -bios /usr/share/qemu/bios.bin -vga none -display none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/mnt/vm/htpc/MSI.HD6450.1024.110407.rom \ -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1 \ -usb -usbdevice host:046d:c52b \ -drive file=/mnt/vm/htpc/htpc,id=disk,format=raw,index=0 -device ide-hd,bus=ide.0,drive=disk \ ;

By executing this script file the VM starts and runs as intended. How to I get it to work in virt-manager?
I am able to boot the VM through virt-manager, and it registers the graphics card ("lspci" finds it), but no video is forwarded to it, all I get is video on the virt-manager console. Whats next?

This vm is running ubuntu for use as a htpc if that matters...

Offline

#2552 2014-08-23 02:20:18

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Offline

#2553 2014-08-23 08:47:29

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit: the following problem exists in current qemu git version (August 22th/23th), version from August 18th does work. Possibly related to q35*
*) https://www.mail-archive.com/qemu-devel … 51880.html

Hi, my first try on getting gpu passthrough to work ended with an segfaulting qemu, using the testing snippet from the first post. I tried to debug a little, but using valgrind the gpu does come to live (can see GPU Post-Screen and seabios looking for disks in slow-motion, but hey)!
So first I thought my hardware has some bugs or faulty firmware or something, but I hope it might be a software problem. Does anyone else has problems with the latest qemu git?

My hardware: Intel E3-1245 (V1), MSI Z77A-GD65 / ASRock H77 Pro4-m (the MSI has been successfully used in a home server, passing through a SAS controller), NVidia GTX 750 Ti
Software: arch linux with patched mainline 3.16 kernel from first post, qemu-git from AUR (and built and installed in home for debugging purpose)

% uname -a Linux mustrum 3.16.0-1-mainline #1 SMP PREEMPT Fri Aug 22 01:35:09 CEST 2014 x86_64 GNU/Linux

kernel log

% dmesg | grep -i -e dmar -e iommu -e vfio -e arb -e vga -e drm [ 0.000000] Command line: \vmlinuz-linux-mainline root=PARTUUID=3520fde9-ca4e-4b5c-b48d-077964e89ef5 rootfstype=f2fs ro pci-stub.ids=10de:1380,10de:0fbc i915.enable_hd_vgaarb=1 intel_iommu=on initrd=initramfs-linux-mainline.img [ 0.000000] ACPI: DMAR 0x00000000C9435038 0000B8 (v01 INTEL SNB 00000001 INTL 00000001) [ 0.000000] Kernel command line: \vmlinuz-linux-mainline root=PARTUUID=3520fde9-ca4e-4b5c-b48d-077964e89ef5 rootfstype=f2fs ro pci-stub.ids=10de:1380,10de:0fbc i915.enable_hd_vgaarb=1 intel_iommu=on initrd=initramfs-linux-mainline.img [ 0.000000] Intel-IOMMU: enabled [ 0.031799] dmar: Host address width 36 [ 0.031801] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.031809] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.031810] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.031814] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.031815] dmar: RMRR base: 0x000000ca48e000 end: 0x000000ca49cfff [ 0.031816] dmar: RMRR base: 0x000000cb800000 end: 0x000000cf9fffff [ 0.031885] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.299425] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.299428] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.299430] vgaarb: loaded [ 0.299430] vgaarb: bridge control possible 0000:01:00.0 [ 0.299431] vgaarb: no bridge control possible 0000:00:02.0 [ 0.346403] DMAR: No ATSR found [ 0.346422] IOMMU 0 0xfed90000: using Queued invalidation [ 0.346423] IOMMU 1 0xfed91000: using Queued invalidation [ 0.346424] IOMMU: Setting RMRR: [ 0.346433] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff] [ 0.346726] IOMMU: Setting identity map for device 0000:00:14.0 [0xca48e000 - 0xca49cfff] [ 0.346741] IOMMU: Setting identity map for device 0000:00:1a.0 [0xca48e000 - 0xca49cfff] [ 0.346753] IOMMU: Setting identity map for device 0000:00:1d.0 [0xca48e000 - 0xca49cfff] [ 0.346761] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.346767] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.353553] fb0: EFI VGA frame buffer device [ 1.047639] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point. [ 1.047806] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ 1.222325] [drm] Initialized drm 1.1.0 20060810 [ 1.244809] [drm] Memory usable by graphics device = 2048M [ 1.244811] [drm] VT-d active for gfx access [ 1.244813] [drm] Disabling PPGTT because VT-d is on [ 1.244814] [drm] Replacing VGA console driver [ 1.244818] fb: switching to inteldrmfb from EFI VGA [ 1.262227] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 1.262228] [drm] Driver supports precise vblank timestamp query. [ 1.262270] [drm] DMAR active, disabling use of stolen memory [ 1.265141] [drm] Wrong MCH_SSKPD value: 0x16040307 [ 1.265143] [drm] This can cause pipe underruns and display issues. [ 1.265143] [drm] Please upgrade your BIOS to fix this. [ 1.329323] fbcon: inteldrmfb (fb0) is primary device [ 1.370543] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device [ 1.392060] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem [ 1.414195] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0 [ 2.344673] [drm] Enabling RC6 states: RC6 on, RC6p off, RC6pp off [ 15.312138] [drm] stuck on render ring [ 15.312526] [drm] GPU HANG: ecode 0:0x85fffffd, in systemd-logind [309], reason: Ring hung, action: reset [ 15.312528] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace. [ 15.312530] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel [ 15.312531] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue. [ 15.312532] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it. [ 15.312533] [drm] GPU crash dump saved to /sys/class/drm/card0/error [ 17.310369] [drm] Enabling RC6 states: RC6 on, RC6p off, RC6pp off [ 36.970235] VFIO - User Level meta-driver version: 0.3 [ 64.491380] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 64.491522] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258 [ 64.491534] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 64.494601] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

gdb core dump inspection

Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ffff6903fe0 in g_str_hash () from /usr/lib/libglib-2.0.so.0 (gdb) backtrace #0 0x00007ffff6903fe0 in g_str_hash () from /usr/lib/libglib-2.0.so.0 #1 0x00007ffff6903569 in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0 #2 0x00000000006b4e94 in type_table_lookup (name=0x8 <error: Cannot access memory at address 0x8>) at qom/object.c:92 #3 0x00000000006b5112 in type_get_by_name (name=0x8 <error: Cannot access memory at address 0x8>) at qom/object.c:158 #4 0x00000000006b514a in type_get_parent (type=0x1344ff0) at qom/object.c:164 #5 0x00000000006b5292 in type_is_ancestor (type=0x1344ff0, target_type=0x10f30c0) at qom/object.c:212 #6 0x00000000006b5ec8 in object_class_dynamic_cast (class=0x7ffff22fd128 <main_arena+1192>, typename=0x7c88c3 "cpu") at qom/object.c:530 #7 0x00000000006b5c1b in object_dynamic_cast (obj=0x11ec140, typename=0x7c88c3 "cpu") at qom/object.c:443 #8 0x00000000004b1890 in acpi_add_cpu_info (o=0x11ec140, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:133 #9 0x00000000006b6280 in object_child_foreach (obj=0x11ed260, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676 #10 0x00000000004b18fa in acpi_add_cpu_info (o=0x11ed260, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:140 #11 0x00000000006b6280 in object_child_foreach (obj=0x112ed60, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676 #12 0x00000000004b18fa in acpi_add_cpu_info (o=0x112ed60, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:140 #13 0x00000000006b6280 in object_child_foreach (obj=0x1117270, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676 #14 0x00000000004b18fa in acpi_add_cpu_info (o=0x1117270, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:140 #15 0x00000000006b6280 in object_child_foreach (obj=0x1117f70, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676 #16 0x00000000004b1941 in acpi_get_cpu_info (cpu=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:149 #17 0x00000000004b5054 in acpi_build (guest_info=0x1122470, tables=0x7fffebcbf9f0) at /home/marcel/src/qemu/hw/i386/acpi-build.c:1486 #18 0x00000000004b55e8 in acpi_build_update (build_opaque=0x13021b0, offset=0) at /home/marcel/src/qemu/hw/i386/acpi-build.c:1623 #19 0x000000000062f2f2 in fw_cfg_read (s=0x117c720) at hw/nvram/fw_cfg.c:255 #20 0x000000000062f404 in fw_cfg_comb_read (opaque=0x117c720, addr=1, size=1) at hw/nvram/fw_cfg.c:291 #21 0x000000000045e793 in memory_region_read_accessor (mr=0x117ec10, addr=1, value=0x7fffebcbfb50, size=1, shift=0, mask=255) at /home/marcel/src/qemu/memory.c:410 #22 0x000000000045e9ef in access_with_adjusted_size (addr=1, value=0x7fffebcbfb50, size=1, access_size_min=1, access_size_max=4, access=0x45e741 <memory_region_read_accessor>, mr=0x117ec10) at /home/marcel/src/qemu/memory.c:480 #23 0x0000000000460fc2 in memory_region_dispatch_read1 (mr=0x117ec10, addr=1, size=1) at /home/marcel/src/qemu/memory.c:1097 #24 0x0000000000461084 in memory_region_dispatch_read (mr=0x117ec10, addr=1, pval=0x7fffebcbfc08, size=1) at /home/marcel/src/qemu/memory.c:1119 #25 0x0000000000463c2c in io_mem_read (mr=0x117ec10, addr=1, pval=0x7fffebcbfc08, size=1) at /home/marcel/src/qemu/memory.c:1963 #26 0x0000000000419fcc in address_space_rw (as=0xc2e180 <address_space_io>, addr=1297, buf=0x7ffff7ea4000 <error: Cannot access memory at address 0x7ffff7ea4000>, len=1, is_write=false) at /home/marcel/src/qemu/exec.c:2086 #27 0x000000000045bfb0 in kvm_handle_io (port=1297, data=0x7ffff7ea4000, direction=0, size=1, count=1024) at /home/marcel/src/qemu/kvm-all.c:1597 #28 0x000000000045c3ee in kvm_cpu_exec (cpu=0x11323e0) at /home/marcel/src/qemu/kvm-all.c:1734 #29 0x0000000000445947 in qemu_kvm_cpu_thread_fn (arg=0x11323e0) at /home/marcel/src/qemu/cpus.c:939 #30 0x00007ffff3da5124 in start_thread () from /usr/lib/libpthread.so.0 #31 0x00007ffff203d4bd in clone () from /usr/lib/libc.so.6 (gdb) list 2914 2915 QDECREF(pdict); 2916 g_free(id); 2917 g_free(type); 2918 g_free(dummy); 2919 if (err) { 2920 qerror_report_err(err); 2921 error_free(err); 2922 return -1; 2923 }

Valgrind output: http://txt.do/k882
(Stopped right after the gpu comes to live. Lots of unimportant "Conditional jump or move depends on uninitialised value(s)", scroll almost to the end for "Invalid read of size 8")

Last edited by mauorrizze (2014-08-24 02:04:46)

Offline

#2554 2014-08-23 09:04:03

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Linux-mainline 3.16.1 + patches. Additional changes: tickrate set to 1000 Hz and preemption model has been set to voluntary.

http://netload.in/dateixLakD3kkGz/linux … tar.gz.htm

Offline

#2555 2014-08-24 18:12:32

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://edencomputing.com/index.php/2014 … ebian-way/

 Virtualization (CONFIG_VIRTUALIZATION=y) Virtualization > Kernel-based Virtual Machine (KVM) Support (CONFIG_KVM=y) Virtualization > KVM for <whichever processor you have> (CONFIG_KVM_INTEL=y and/or CONFIG_KVM_AMD=y) Bus Options (PCI etc) > PCI Stub Driver (CONFIG_PCI_STUB=y) Device Drivers > VFIO Non-Privileged userspace driver framework (CONFIG_VFIO=y) Device Drivers > VFIO Non-Privileged userspace driver framework > VFIO support for PCI Devices (CONFIG_VFIO_PCI=y) Device Drivers > VFIO Non-Privileged userspace driver framework > VFIO support for VGA Devices (CONFIG_VFIO_PCI_VGA=y) The rest of these are optional (but recommended) performance enhancements you may compile as modules [M]: Virtualization > Host kernel accelerator for virtio net (CONFIG_VHOST_NET=m) Device Drivers > Virtio drivers > PCI driver for virtio devices (CONFIG_VIRTIO_PCI=m) Device Drivers > Virtio drivers > Virtio balloon driver (CONFIG_VIRTIO_BALLOON=m) And finally these options will enhance performance and responsiveness of your virtual machines: Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop) (CONFIG_PREEMPT=y) Processor Type and Features > Timer Frequency > 1000 Hz (CONFIG_HZ_1000=y)

just apply the batch, then change the above.
you have your  kernel.
I'm not relly sure about downloading files from unknown sources.. ( the prv post).

My setup is now working with 2 GPUs, I got Vm1 and Vm2, each one has 6114 Gb of ram, 2 HDDs, the first one using Nvidia GTX 760, the second one uses the AMD sapphire toxic 4780 HD, everything looks great, but if you killed an VM through the host, you will destroy it, and it wont boot anymore, thats make me afride of power loss.

Offline

#2556 2014-08-24 22:52:49

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh right forgot to tell ya'll about dat secret NSA backdoor.

Offline

#2557 2014-08-24 23:14:50

bruce1337
Member
Registered: 2014-08-24
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there,

thanks to this pertinent resource (and some qualified outside assistance), the inappropriately monikered linux newb that is me has succeeded in passing through dedicated graphics to a VM. So far, I have relied on reading only -- but at 100+ pages, this thread has become so unwieldy that I decided de-lurk (...making it even more unwieldy hmm). Nice signup captcha by the way! Hope you will forgive any violations of tech etiquette I'm bound to make and let me know if I did, so that I may avoid them in the future.

For the record, I'm running

Xeon E3-1246v3
ASRock H97 Pro4, supporting both Vt-d and IOMMU (as far as I can tell...)

My issue is with virt-manager (both 0.9.5 & 1.01), and seems to be identical to Chetyre's discussed on p.26: qemu, when invoked by virt-manager, fails to access /dev/vfio/x while (seemingly?) succeeding when run from the command line. I followed all subsequent directions I could find, i.e.

/etc/libvirt/qemu.conf

user = "+0"
group = "+0"

group_device_acl = [   
+    "/dev/vfio/1", "/dev/vfio/13"
]

clear_emulator_capabilities = 0

relaxed_acs_check = 1

(/dev/vfio/13 contains 2 entries: video and audio from the GTX [edit: or rather the respective iommu_group does...])

Unfortunately, I could locate neither additional info nor a confirmation of the problem's solution further down the thread. Strangely, the problem persists when running virt-manager itself as root. To be clear: running the barebone qemu config (ioh3420 + GTX) from the command line results in Seabios output up to "no bootable device" from the GTX, so I'm assuming the principal setup to be correct -- it's just that for now, I would prefer to rely on the convenience of virt-manager offering a working setup for interfacing to the VM by spice and mouse/keyboard emulation instead of applying everything manually through qemu-argv. The last couple of days have been filled with a lot of studying, I think it's about time for some playing...

[edit: After some wrestling with qemu (or rather my ignorance of it), my odyssey has come to a a (certainly temporary) conclusion -- things are finally working! Has anyone ever figured out what exactly determines whether qemu will grab either mouse, keyboard, both, or sometimes nothing? Is it all a function of "-device qxl" (hard to believe...)?  Also, does anyone know if I can copy my concurrent .img directly to a dedicated logical volume or will the "physical partition" rules apply and demand loopback MBR? Thanks!]

[edit #2: Well, this is somewhat embarassing, but as I now know, all the troubles I've been having with qemu's command line invocation were clearly due to PEBKAC. Pro-tip: Do NOT insert comments into the multiline command of your shell script like ol' silly bruce did. Also, dd'ing an image to a dedicated LV works without a hitch, no additional steps required.]

Last edited by bruce1337 (2014-08-26 13:09:50)

Offline

#2558 2014-08-24 23:19:53

somearchfan
Member
Registered: 2014-08-18
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bruce1337 wrote:

For the record, I'm running

Xeon E3-1246v3
ASRock H97 Pro4, supporting both Vt-d and IOMMU (as far as I can tell...)

Just a quick fyi: the H97 Pro4 does support VT-D (you can disable / enable it in the bios settings) and IOMMU, so no worries there. I'm using a fairly similar board, the H97M Pro 4. The only difference is the form factor afaik.

Cheers

Offline

#2559 2014-08-25 00:30:25

bruce1337
Member
Registered: 2014-08-24
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's not the size that matters, it's how you wield it...still, mine *is* bigger! Then again, those 2 additional PCI slots will hardly see any action, ever...

Actually, it was ASRock's reputed feature support beyond Intel's chipset standards that made me buy it, so I wasn't worried too much. Still, good to have direct confirmation! (And should you ever be bored, do take the guided tour of the UEFI, it's pretty goofy, to say the least...)

Last edited by bruce1337 (2014-08-25 00:37:32)

Offline

#2560 2014-08-25 05:09:40

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ive been trying to get my kvm +qemu vm up and running on an arch host, eventually running windows 8.1 as a guest.

my setup is a vt-d and IOMMU compatible mobo and intel processor with a r9 270 for the host and a GTX 770 for the guest.
I am using the 3.16.1 kernel package from the ABS to which I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

running with these options

root=/dev/mapper/VolGroup00-lvroot rw initrd=/initramfs-linux-custom-acs.img intel_iommu=on

Right now im getting the devices to bind to vfio just fine but when I try to run my test script I run into errors.

here is the script im using to test.

 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

and here is the output I get from running it from root

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

any ideas as to where I am going wrong?

Last edited by sitonapanotis (2014-08-25 05:18:59)

Offline

#2561 2014-08-25 08:22:20

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:

Ive been trying to get my kvm +qemu vm up and running on an arch host, eventually running windows 8.1 as a guest.

my setup is a vt-d and IOMMU compatible mobo and intel processor with a r9 270 for the host and a GTX 770 for the guest.
I am using the 3.16.1 kernel package from the ABS to which I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

running with these options

root=/dev/mapper/VolGroup00-lvroot rw initrd=/initramfs-linux-custom-acs.img intel_iommu=on

Right now im getting the devices to bind to vfio just fine but when I try to run my test script I run into errors.

here is the script im using to test.

 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

and here is the output I get from running it from root

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

any ideas as to where I am going wrong?

Look at the QEMU configuration QEMU.conf, as three posts above. Set QEMU processes to run as root, do

ls /dev/vfio

and add what you find there to the group_device_acl in qemu.conf, as above (or look through this thread for other examples). Lastly, invoke qemu-system-x86_64 as root.

Offline

#2562 2014-08-25 12:48:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:

I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

any ideas as to where I am going wrong?

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2563 2014-08-25 20:48:48

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
sitonapanotis wrote:

I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

any ideas as to where I am going wrong?

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

It apppears that my processor  radeon card/radeon sound device and nvidia card/sound device are all in the same group which is what is causing problems.

How do I go about enableing the acs override?

Offline

#2564 2014-08-25 20:53:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:
aw wrote:
sitonapanotis wrote:

I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

any ideas as to where I am going wrong?

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

It apppears that my processor  radeon card/radeon sound device and nvidia card/sound device are all in the same group which is what is causing problems.

How do I go about enableing the acs override?

Really?  You're not even willing to google "acs override"?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2565 2014-08-25 20:58:37

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
sitonapanotis wrote:
aw wrote:

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

It apppears that my processor  radeon card/radeon sound device and nvidia card/sound device are all in the same group which is what is causing problems.

How do I go about enableing the acs override?

Really?  You're not even willing to google "acs override"?

Im reading mailing list posts on it as we speak, I just thought this would be a good place to ask for more specific instructions. I didn't know that was inapropriate for the disscussion here. Anyway thank you for the help so far, without this thread I wouldnt even have been able to get as far as I have already.

edit:

got it working with pcie_acs_override=downstream

edit2:

qemu starts now but I dont get output from my passed through card on my second monitor.
when testing using "vga std" in place of "vga none" the vm/seabios boots fine.
I searched through the thread for others who seemed to have similar problems and saw that some of them had solved those problems but I was unable to identify how. If anyone has experienced/fixed this problem let me know.

Last edited by sitonapanotis (2014-08-26 08:48:30)

Offline

#2566 2014-08-26 10:01:12

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you need to read op post about vga patches (not acs) and how to activate them...

Last edited by slis (2014-08-26 10:01:44)

Offline

#2567 2014-08-26 14:52:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's your favorite over-asked question here?  http://vfio.blogspot.com/2014/08/vfiovga-faq.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2568 2014-08-26 15:36:50

bruce1337
Member
Registered: 2014-08-24
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As someone who diligently tried to obtain answers by reading through this thread (yes...all of it), I believe you have already caught all prime suspects -- would be great to have this linked prominently in the OP. Also, I respectfully tip my hat to you, good sir! I certainly wouldn't be in the position I am in now were it not for your dedication, both in terms of technology as well as communication, and for that, you have my sincerest thanks. I shall try to pay it forward with the talents at my disposal.

Offline

#2569 2014-08-26 16:03:38

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3rd smile

Offline

#2570 2014-08-26 16:33:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:

http://vfio.blogspot.com/2014/08/primar … t-vga.html

I'm not the first to try this, former active user on this thread DanaGoyette attempted this, but never fully debugged the issues.  Using OVMF (UEFI) for the guest we can avoid all of the issues with the VGA aspects of GPU assignment.  This means that we don't care about how broken the i915 driver is and we don't need the x-vga option for the assigned device, making it just another assigned devices as far as libvirt is concerned.

Also:

http://vfio.blogspot.com/2014/08/upstre … -2014.html

If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2571 2014-08-26 20:29:58

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What's your favorite over-asked question here?  http://vfio.blogspot.com/2014/08/vfiovga-faq.html

This faq is SUPER useful, I was having problem 4, which I fixed now.
Thank you!

edit: Got everything set up sucessfully! Thank you so much to everyone in the thread!

Last edited by sitonapanotis (2014-08-27 03:16:22)

Offline

#2572 2014-08-26 22:58:55

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:
...

Thanks so much for all the details.
I've just successfully got my HD7770 passed through to my Win7 VM (trick was to not pass the sound with it - good tip redger) and proved capability.
And that was on 14.04 with vanilla qemu and 3.16 kernel.
Still have to work out networking and sound, but progress is always encouraging.
I will have to consider Win8 based on your UEFI comments, but will wait for my GTX750ti first and give that a crack.
Thanks again for the help.
Will try your blog and see how it goes.Getting closer to a steam streaming server!
Cheers.

Offline

#2573 2014-08-27 13:03:33

josipva
Member
Registered: 2014-07-21
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

Last edited by josipva (2014-08-27 13:03:50)

Offline

#2574 2014-08-27 13:52:25

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

josipva wrote:

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

I haven't tried this myself so I can't say for certain it would work but why not. If, for example, you would pass 3 vcpu to each guest, keeping 2 vcpu for the host and pinning accordingly I would assume you would get equal or faster performance for most tasks. I don't think there's a penalty to assigning an odd number of vcpu though I could be mistaken. Even with the performance hit compared to bare metal there is a 500 MHz difference in base clock and turbo speeds between the 4790K and 4690, that should in part or fully make compensate depending on the application. You'd likely have somewhat reduced performance in applications where having a fourth core available is a benefit. However again I haven't tried this.

I have an 4790K with Win 8.1 using 4 vcpu and a 7950, if you'd like I can run a benchmark if you tell me which one. Windows performance index is 8.1 for CPU, memory and graphics.

Offline

#2575 2014-08-27 14:12:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

josipva wrote:

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

I expect your clock scaling from 4GHz to 3.5GHz for the VM is probably sufficiently conservative to account for the overhead, but if you split an i7-4790k in half, you're probably looking at something more like an i3-4330 per VM.  Threading doesn't come anywhere close to doubling the throughput of a core, so running two quad-core VMs on a quad-core + threads host is overcommitting the CPU resources.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2576 2014-08-27 14:13:23

josipva
Member
Registered: 2014-07-21
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

I have an 4790K with Win 8.1 using 4 vcpu and a 7950, if you'd like I can run a benchmark if you tell me which one. Windows performance index is 8.1 for CPU, memory and graphics.

Could you try cinebench with 4vcpu and 3 vcpu so I can compare it to native 4690 results ?

P.S. Thanks "siddharta" and "aw" for info so far.

Last edited by josipva (2014-08-27 14:34:04)

Offline

#2577 2014-08-27 18:35:59

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

josipva wrote:
siddharta wrote:

I have an 4790K with Win 8.1 using 4 vcpu and a 7950, if you'd like I can run a benchmark if you tell me which one. Windows performance index is 8.1 for CPU, memory and graphics.

Could you try cinebench with 4vcpu and 3 vcpu so I can compare it to native 4690 results ?

P.S. Thanks "siddharta" and "aw" for info so far.

488cb for 3 vcpu, 590cb for 4 vcpu, 790 for 6 vcpu (for completeness). This could possibly be improved still, this is a clean install and not much tweaking was done.

http://imgur.com/touYvyQ,EqM4eGw#1 (forgot to screencap the 6 vcpu, sorry)

Offline

#2578 2014-08-27 18:58:59

josipva
Member
Registered: 2014-07-21
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Huge thanks. Helps a lot to estimate performance.

EDIT:

Interesting stuff 4vcpu performance puts it in line with 4690.

Last edited by josipva (2014-08-27 19:20:54)

Offline

#2579 2014-08-27 20:37:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

josipva wrote:

Huge thanks. Helps a lot to estimate performance.

EDIT:

Interesting stuff 4vcpu performance puts it in line with 4690.

Sure, if you only expect one VM to be under load at any given point, your i5-4690 is a good comparison.  If you want the worst case performance where both VMs are under load, I think you'll be closer to the i3 processor.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2580 2014-08-28 15:33:38

bonob
Member
Registered: 2014-02-02
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's not directly related with the topic at hand, but maybe some people around here have the same config as I have and will be able to confirm whether I'm understanding my issue correctly (and possibly confirm if they suffer the same fate.)

I send the audio from my (Windows) VM to my (Arch) host using NetJack2 (mostly following this guide.) It's been working pretty well for the past few months.
Since I rebooted yesterday, it's not working anymore. I noticed that jack2-dbus has just been updated to 1.9.10, so I'm wondering if that's the cause of my trouble.

From what I can see, everything is working as usual, jack can play audio on the host, the virtual cable is plugged on the guest, the qjackctls on either side are not reporting anything unusual.
It seems that the problem is that the jack slave and the master cannot find each other, whereas they used to. So I'm thinking that maybe the auto-discovery feature of NetJack2 is somehow broken?

Last edited by bonob (2014-08-28 15:35:03)

Offline

#2581 2014-08-28 23:35:41

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wimma77 wrote:
aw wrote:

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:
...

I've just successfully got my HD7770 passed through to my Win7 VM (trick was to not pass the sound with it - good tip redger) and proved capability.
Still have to work out networking and sound, but progress is always encouraging.
I will have to consider Win8 based on your UEFI comments, but will wait for my GTX750ti first and give that a crack.

Just in addition to this, happy with the performance and can stream a steam game across the network fine (HD7770 anyway, haven't worked out how to get the GTX750 working yet..), but high level question about sound if anyone could help please.
I see the sound config details on p1, and will use for some testing, but going forward I want to use alsa on the host for some distributed home audio stuff (with a couple pcie sound cards).
Does that mean I can really only use pulseaudio for the VM? Will I run into conflicts later if I use alsa on both?

Offline

#2582 2014-08-29 01:30:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wimma77 wrote:
Wimma77 wrote:
aw wrote:

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:
...

I've just successfully got my HD7770 passed through to my Win7 VM (trick was to not pass the sound with it - good tip redger) and proved capability.
Still have to work out networking and sound, but progress is always encouraging.
I will have to consider Win8 based on your UEFI comments, but will wait for my GTX750ti first and give that a crack.

Just in addition to this, happy with the performance and can stream a steam game across the network fine (HD7770 anyway, haven't worked out how to get the GTX750 working yet..), but high level question about sound if anyone could help please.
I see the sound config details on p1, and will use for some testing, but going forward I want to use alsa on the host for some distributed home audio stuff (with a couple pcie sound cards).
Does that mean I can really only use pulseaudio for the VM? Will I run into conflicts later if I use alsa on both?

Does audio only work with steam in-home-streaming when routed through the GPU audio?  This was the conclusion I came to when I attempted to do streaming on a bare metal system without a monitor connected to the GPU, but I didn't try too hard.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2583 2014-08-29 02:37:10

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does audio only work with steam in-home-streaming when routed through the GPU audio?  This was the conclusion I came to when I attempted to do streaming on a bare metal system without a monitor connected to the GPU, but I didn't try too hard.

Yeah, this is my question essentially. I can't add in the audio device to my VM, as it won't boot with it.
Just the video card device itself is fine though, and at least gives me accelerated video capability.
Will try some audio lines to test, but get the feeling I'll still need to keep a second box or dual boot sad
Be a bummer, as the main VM use for me would be steam streaming.
Not sure if I'll have more luck with my nvidia card, but a quick test yesterday just got me emulator error, like others have had in earlier posts. Still have to sort that out. More testing ahead...

Offline

#2584 2014-08-29 05:04:52

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

steaming doesn't need sound card on guest, if u don't want sound (some games require sound card), but it can use any sound card... i am using   

    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>

since i don't need any audio on host / audio-output  on guest, and hdmi audio causes problems after 30mins or so (in my case)....

Last edited by slis (2014-08-29 05:08:24)

Offline

#2585 2014-08-29 07:59:31

Cenm
Member
Registered: 2014-08-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to get everything working for the first boot of the VM but it fails for every subsequent boot with the error 'vfio-pci 0000:03:00.0: Invalid ROM contents'

If I specify romfile=/path/to/rom I don't get the error but nothing appears on the screen.

I have also noticed when I kill the VM whatever was on the screen remains there until I start the VM again, at which point the screen turns off.

Any advice would be great.

Offline

#2586 2014-08-29 11:52:52

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

steaming doesn't need sound card on guest, if u don't want sound (some games require sound card), but it can use any sound card... i am using   

    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>

since i don't need any audio on host / audio-output  on guest, and hdmi audio causes problems after 30mins or so (in my case)....

Awesome,thanks. Will try this. I do need alsa audio on the host, and I want audio for streaming games, so will see if anything can work on the guest.

Still having issues with nvidia card though.
Is the kvm=off option still required for nvidia?
ref "If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA" posted a while back.

I cannot get anything on screen booting with a guest GTX750ti (HD5450 host), though had video passthrough fine with guest HD7770.
I get the QEMU window for a minute, then
KVM internal error. Suberror: 1
emulation failure
EAX=ffff0 ... etc
I tried cpu kvm=off option, but that fails also with
qemu-system-x86_64: Property '.kvm' not found

Nothing else was changed from the HD7770 setup (except for stub IDs for new card)... ?

Offline

#2587 2014-08-29 12:20:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wimma77 wrote:
slis wrote:

steaming doesn't need sound card on guest, if u don't want sound (some games require sound card), but it can use any sound card... i am using   

    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>

since i don't need any audio on host / audio-output  on guest, and hdmi audio causes problems after 30mins or so (in my case)....

Awesome,thanks. Will try this. I do need alsa audio on the host, and I want audio for streaming games, so will see if anything can work on the guest.

Still having issues with nvidia card though.
Is the kvm=off option still required for nvidia?
ref "If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA" posted a while back.

These are two separate things.  kvm=off will always be required as long as the guest Nvidia driver decides to bail when it finds KVM.  The QEMU change allows Nvidia to work with OVMF (ie. without x-vga=on).

I cannot get anything on screen booting with a guest GTX750ti (HD5450 host), though had video passthrough fine with guest HD7770.
I get the QEMU window for a minute, then
KVM internal error. Suberror: 1
emulation failure
EAX=ffff0 ... etc
I tried cpu kvm=off option, but that fails also with
qemu-system-x86_64: Property '.kvm' not found

This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.

Nothing else was changed from the HD7770 setup (except for stub IDs for new card)... ?

What is your primary graphics for this test?  Does FAQ question 4 apply?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2588 2014-08-29 12:56:03

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

These are two separate things.  kvm=off will always be required as long as the guest Nvidia driver decides to bail when it finds KVM.  The QEMU change allows Nvidia to work with OVMF (ie. without x-vga=on).
This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.
What is your primary graphics for this test?  Does FAQ question 4 apply?

Ok, wasn't sure I'd need kvm=off. Looking at https://bbs.archlinux.org/viewtopic.php … 7#p1313007 shows ubuntu success using older version, and similar host ATI card (I have HD5450) and guest nvidia. Will try newer QEMU this weekend anyway, no harm.
No errors from memory in dmesg like FAQ#3, but as I have a headless server NFI if it had any video issues. VNC to my xfce4 desktop was fine though! lol
Also just noticed card has the hybrid BIOS switch thing, so may set to legacy and try also, as using Win7.
Thanks again.

Offline

#2589 2014-08-29 13:10:45

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:

http://vfio.blogspot.com/2014/08/primar … t-vga.html

I'm not the first to try this, former active user on this thread DanaGoyette attempted this, but never fully debugged the issues.  Using OVMF (UEFI) for the guest we can avoid all of the issues with the VGA aspects of GPU assignment.  This means that we don't care about how broken the i915 driver is and we don't need the x-vga option for the assigned device, making it just another assigned devices as far as libvirt is concerned.

Also:

http://vfio.blogspot.com/2014/08/upstre … -2014.html

If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA.

Nicely done!, I was thinking about the same thing, I may submit a post or 2, maybe do some translation as well, but let me find the time for it.

Now everything works perfect exept one thing, when ever the proccess of qemu is killed, the guest OS becomes corrupted, is that the matter with the format of the container (using raw data here) ?, what should be done to avoid this problem, what if the powerd is lost suddenly for example.

Offline

#2590 2014-08-29 13:57:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Nicely done!, I was thinking about the same thing, I may submit a post or 2, maybe do some translation as well, but let me find the time for it.

That would be great!

Now everything works perfect exept one thing, when ever the proccess of qemu is killed, the guest OS becomes corrupted, is that the matter with the format of the container (using raw data here) ?, what should be done to avoid this problem, what if the powerd is lost suddenly for example.

Are you specifying any cache option for the guest block device?  There is a cache=unsafe option which uses the host buffer cache, but as the name implies, it's not safe.  I typically use raw images or lvm volumes and I don't have much issue with killing the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2591 2014-08-29 15:03:28

josipva
Member
Registered: 2014-07-21
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Sure, if you only expect one VM to be under load at any given point, your i5-4690 is a good comparison.  If you want the worst case performance where both VMs are under load, I think you'll be closer to the i3 processor.

That makes sense. What do you think about newly released i7-5820k (it's only a bit more expensive than 4790k)? Could I clock it a bit, give 5 vcpu to each VM and leave 2vcpu for host while getting equivalent of 4690 performance for each VM (running simultaneously ofc)?

Last edited by josipva (2014-08-29 15:04:25)

Offline

#2592 2014-08-29 15:14:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

josipva wrote:
aw wrote:

Sure, if you only expect one VM to be under load at any given point, your i5-4690 is a good comparison.  If you want the worst case performance where both VMs are under load, I think you'll be closer to the i3 processor.

That makes sense. What do you think about newly released i7-5820k (it's only a bit more expensive than 4790k)? Could I clock it a bit, give 5 vcpu to each VM and leave 2vcpu for host while getting equivalent of 4690 performance for each VM (running simultaneously ofc)?

-ENOCPUINARK  But I see elsewhere that it's a hex-core+threads and it's got a lower clock speed (3.3GHz) and since it's not in ARK yet, unknown VT-d support.  With this CPU you've gained 2 more cores, but you've lost too much clock to be comparable to the i5-4690.  You could reasonable have 2 concurrent sockets=1,cores=3,threads=2 VMs but it's difficult to compare 3cores+threads @3.3GHz to 4cores @3.5GHz.  Seems like they could be pretty close aggregate, but your single thread performance is going to take a hit.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2593 2014-08-29 15:19:00

josipva
Member
Registered: 2014-07-21
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's a cpu with unlocked multiplier and i plan to overclock it to atleast 4.0Ghz (early leaks indicate it's a pretty good chip for oc). Seems to me like a good target for dual VM without going for full on expensive 1k$ chip, thats why im asking for opinions smile

Last edited by josipva (2014-08-29 15:20:32)

Offline

#2594 2014-08-29 15:24:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

josipva wrote:

It's a cpu with unlocked multiplier and i plan to overclock it to atleast 4.0Ghz (early leaks indicate it's a pretty good chip for oc). Seems to me like a good target for dual VM without going for full on expensive 1k$ chip, thats why im asking for opinions smile

If you can get there and have it stable then it could be a good option (/me makes note to ignore any spectacularly weird bugs from josipva).  Just be aware that there are many K-series processors that do not support VT-d.  For a while they seemed to be mutually exclusive.  Hopefully it will appear in ark soon.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2595 2014-08-29 15:27:46

josipva
Member
Registered: 2014-07-21
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well they added Vt-d support for both i5 and i7 in Devil's canyon release so i doubt it won't be supported in these new Haswell-E cpus. Anyway 4.0Ghz in Haswell is easy to get stable.

aw wrote:

(/me makes note to ignore any spectacularly weird bugs from josipva)

tongue

Last edited by josipva (2014-08-29 15:30:42)

Offline

#2596 2014-08-30 09:16:52

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all, first post here smile .
I'm getting some problems setting up a VM with Windows 7 and a GeForce GTX 650 passed through.

Software setup:
- Debian 7 (headless pc, managed through ssh)
- kernel 3.16.1 + ACS patch + i915 arbitrer patch + VGA arbitrer patch
- boot parameters "i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b"
- qemu 2.1
- seabios 1.7.5

Hardware setup:
- MSI Z77A-G43
- Intel i5-3470
- IGD as primary VGA (for host)
- nVidia GeForce GTX 650 as secondary VGA (for guest)

Steps I followed:
- enabled VT-d in motherboard
- installed Windows 7 through VNC using "-vga std" (keeping GTX 650 as secondary graphics card in guest)
- installed tightVNC inside Windows 7
- installed nVidia graphics driver (version 335.23)
- booted Windows 7 with "-vga none"
- black screen and cannot connect to VNC server inside Windows 7 (without any error from qemu)

Script used to start VM:

# Bind VGA DEVICES="0000:01:00.0 0000:01:00.1" for dev in $DEVICES ; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done # Start QEMU qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios-256k.bin -vga none \ -rtc base=localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/root/gtx650_bios.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/dev/sdc,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -boot menu=on \ -netdev tap,ifname=qemu0,id=qemu_tap -device e1000,netdev=qemu_tap,mac=00:16:3E:12:34:56 \ -usb -usbdevice tablet

If you need more info, just ask.
Thanks in advance.

Offline

#2597 2014-08-30 10:49:15

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

u don't need rombar and romfile...

Offline

#2598 2014-08-30 11:24:00

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need them as without these two options I was getting some errors about VGA initialization (or something like that).
Adding "rombar" and "romfile" the errors disappeared.

PS: obviously the file passed to "romfile" is the actual rom dumped from my nVidia card.

Offline

#2599 2014-08-30 13:04:24

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.

Ok, so got from git qemu 2.1.50, but when I then try to run my vm I just get a 'QEMU 2.1.50 monitor' prompt?
Even added full path /usr/local/bin/qemu... to make sure was using.
And then I still get the same KVM internal error. Suberror: 1 emulation failure after a short time :\
May have to put the card in another machine to confirm all is working ok for a start (used card I just picked up). Or just stick with ATI cards..

Offline

#2600 2014-08-30 13:29:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

I need them as without these two options I was getting some errors about VGA initialization (or something like that).
Adding "rombar" and "romfile" the errors disappeared.

Those errors are what we need to debug.  rombar=0 tells QEMU to not expose a PCI option ROM for the device.  Following that with romfile= makes QEMU put that rom in the genroms section, which is sort of like having it builtin to the BIOS.  The problem with that is that it is no longer associated with the device, so unless you know ROM has the ability to scan through all the devices to find the hardware, you've probably just broken the option ROM execution.

Remove the rombar option and report the error, if there is one.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2601 2014-08-30 13:40:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wimma77 wrote:
aw wrote:

This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.

Ok, so got from git qemu 2.1.50, but when I then try to run my vm I just get a 'QEMU 2.1.50 monitor' prompt?
Even added full path /usr/local/bin/qemu... to make sure was using.
And then I still get the same KVM internal error. Suberror: 1 emulation failure after a short time :\
May have to put the card in another machine to confirm all is working ok for a start (used card I just picked up). Or just stick with ATI cards..

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2602 2014-08-30 15:41:19

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Those errors are what we need to debug.  rombar=0 tells QEMU to not expose a PCI option ROM for the device.  Following that with romfile= makes QEMU put that rom in the genroms section, which is sort of like having it builtin to the BIOS.  The problem with that is that it is no longer associated with the device, so unless you know ROM has the ability to scan through all the devices to find the hardware, you've probably just broken the option ROM execution.

Remove the rombar option and report the error, if there is one.

Ok, don't know about that.


I removed "rombar" and "romfile", magically qemu doesn't report any error/warning.
The only messages I got are from dmesg:

vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 kvm: zapping shadow pages for mmio generation wraparound

However the problem persist: black screen and can't connect to VNC server inside Windows 7.
The strangest thing is that I can arping the VM and nmap report its ip as "up", but both nmap and vncviewer can't find open ports.

Offline

#2603 2014-08-30 22:47:46

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8

Thanks for the response.
Hmm, not sure sorry. Prob over my head sad
But I'm trying to install Win7, so not 8.1 and no nvidia driver (yet).
Can't even get Win7 to install.
So much more painful that AMD/ATI!

Offline

#2604 2014-08-30 23:06:59

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wimma77 wrote:
aw wrote:

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8

Thanks for the response.
Hmm, not sure sorry. Prob over my head sad
But I'm trying to install Win7, so not 8.1 and no nvidia driver (yet).
Can't even get Win7 to install.
So much more painful that AMD/ATI!

How did you install the qemu git version? Applying patches really isn't that complicated…


i'm sorry for my poor english wirting skills…

Offline

#2605 2014-08-30 23:13:16

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

How did you install the qemu git version? Applying patches really isn't that complicated…

git clone, config, make, install.
Didn't spit out any errors I noticed. Took a while to configure though.

Offline

#2606 2014-08-31 04:08:15

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

theres no pain with nvidia... it just works. you got config issues.... @ansa89 your last log shows it's running normally...

Offline

#2607 2014-08-31 07:02:56

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

theres no pain with nvidia... it just works. you got config issues.... @ansa89 your last log shows it's running normally...

well, it actually differs.
while hd 6450 worked almost without any problems, i totally failed getting my gt 430 to work as well - there were many "iterations"/attempts to get it to work and lots of reading to try to get what i am doing wrong. either my card is somehow broken by itself or there is something else i am missing - for now i stick with hd 6450 only, the goal being simultaneous passthrough of 2 gpus to 2 different vms.

p.s. totally failed trying to get gt 430 working via ovmf, but that's another thing to look into. hard to understand what's wrong when no video is shown anywhere and there is no logs aside from usual "enabling device ..."

Offline

#2608 2014-08-31 07:21:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well yeah... I tried 6 gpus from nvidia and 3 mbos form intel, all working, even on quad sli mbo 4gpu-s 4vm-s... but i guess u have amd mbo or (edit) NOT .. smile  i don't know but it works for me.

Last edited by slis (2014-08-31 07:26:00)

Offline

#2609 2014-08-31 08:24:06

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

@ansa89 your last log shows it's running normally...

Ok, but the problem persists, so somewhere there should be something that fails.
Any idea/test/patch/voodoo-magic?

Offline

#2610 2014-08-31 08:43:23

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

well yeah... I tried 6 gpus from nvidia and 3 mbos form intel, all working, even on quad sli mbo 4gpu-s 4vm-s... but i guess u have amd mbo or (edit) NOT .. :)  i don't know but it works for me.

MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20 x86_64
kernel: distro-based kernel-3.15.8 with i915 and vga arbiter patches
host GPU: integrated (intel i7 4770)
guest GPU0: radeon hd 6450 (Gigabyte GV-R645-1GI)  <-- worked almost flawlessly for several months already
guest GPU1: nvidia gt 430 (Palit GT 430)  <-- NOT working (most i got from it was showing video on first run (counting since host reboot/startup), than host freeze on second (again, counting since host reboot/startup) gpu passthrough attempt)

seems to me that there is something fundamentally different between nvidia and amd gpu passthrough (from configuration/setup/procedure point of view), but i haven't yet caught up on what exactly it is

Offline

#2611 2014-08-31 11:45:09

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@sinny well for REBOOT u need rom file from gpu (atleast that was case for mine 560ti, should be same for 4xx), without rombar option...

@ansa89 you could try to remove that bios-256k option and u don't need gpu romfile option... and if u edit that script when adding gpu, care for spaces / new lines... that could be reason for not booting

Last edited by slis (2014-08-31 11:55:38)

Offline

#2612 2014-08-31 12:55:38

Cenm
Member
Registered: 2014-08-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cenm wrote:

I've managed to get everything working for the first boot of the VM but it fails for every subsequent boot with the error 'vfio-pci 0000:03:00.0: Invalid ROM contents'

If I specify romfile=/path/to/rom I don't get the error but nothing appears on the screen.

I have also noticed when I kill the VM whatever was on the screen remains there until I start the VM again, at which point the screen turns off.

Any advice would be great.


Any ideas on getting this fixed? GPU is a GTX 560 Ti. As far as I can tell everything else works fine, but only once.

Offline

#2613 2014-08-31 16:01:11

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

Offline

#2614 2014-08-31 17:02:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

p.s. totally failed trying to get gt 430 working via ovmf, but that's another thing to look into. hard to understand what's wrong when no video is shown anywhere and there is no logs aside from usual "enabling device ..."

Just added a new blog post: Does my graphics card support EFI?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2615 2014-08-31 17:10:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cenm wrote:
Cenm wrote:

I've managed to get everything working for the first boot of the VM but it fails for every subsequent boot with the error 'vfio-pci 0000:03:00.0: Invalid ROM contents'

If I specify romfile=/path/to/rom I don't get the error but nothing appears on the screen.

I have also noticed when I kill the VM whatever was on the screen remains there until I start the VM again, at which point the screen turns off.

Any advice would be great.


Any ideas on getting this fixed? GPU is a GTX 560 Ti. As far as I can tell everything else works fine, but only once.

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2616 2014-08-31 17:20:14

Cenm
Member
Registered: 2014-08-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

aw wrote:

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

Dumped BIOS from GPU. Running QEMU 2.1.0-2, Kernel 3.16.1-1, vfio bound both the video and audio components of the GPU

Last edited by Cenm (2014-08-31 17:24:26)

Offline

#2617 2014-08-31 17:27:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cenm wrote:
slis wrote:

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

aw wrote:

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

Dumped BIOS from GPU. Running QEMU 2.1.0-2, Kernel 3.16.1-1, vfio bound

You're still not telling us how you're running it or what hardware you have other than a GTX650Ti.  If you expect your stock distro kernel to work, please see the OP or read-up on the blog below for the condensed version.  If you'd like to report a problem, please provide the base version and list any patches you've applied for QEMU and kernel, the kernel command line options you're booting with, the host and guest graphics devices, whether this is an AMD or Intel system, and the commands/configuration used to start the guest.  Please don't expect us to guess.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2618 2014-08-31 17:28:19

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

are there any benefits with using ovfm other than getting rid of i915 patches/vga?

Offline

#2619 2014-08-31 17:38:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

are there any benefits with using ovfm other than getting rid of i915 patches/vga?

If that isn't enough reason, you may not have a full grasp of the problems with i915 and VGA arbitration wink

Everything is currently upstream to be able to create OVMF VMs with assigned GPUs and manage them with libvirt.  Eventually your distro will include this.  AFAICT, i915 will never have working VGA arbitration and even if a plan comes about to support it, it could take years to deprecate the existing functionality to be able to implement it.  Not to mention all the other holes with VGA, like the vgacon driver not participating so you may end up with a host login prompt on the guest VGA console.  VGA is a fixed address space shared across the system and our management of it leaves much to be desired.  It's best to stick with the resources specific to your device, which is how we operate under OVMF.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2620 2014-08-31 17:56:54

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your  answer. Since i use headless host, i guess i don't care about i915 working or not smile  I thought about using 440fx instead q35 does it make any difference?

Offline

#2621 2014-08-31 18:14:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Thank you for your  answer. Since i use headless host, i guess i don't care about i915 working or not smile  I thought about using 440fx instead q35 does it make any difference?

The theory with Q35 is that it more closely matches the host hardware topology and therefore is more likely to "just work".  In a small number of cases, this is true.  The most apparent case is that of a Linux guest using the radeon driver, which attempts to negotiate link speed with the downstream port above it.  In a 440FX model there is no such downstream port and the driver code in the guest effectively segfaults.  If we want to use 440FX here we need to fix the driver rather than change the guest machine to accommodate the driver.  (Note that even in a Q35 model there is no negotiation with the parent downstream port.  The parent device registers are only emulated and do not support negotiation) 

In the majority of cases, the guest drivers work well independent of the VM topology.  This seems to be the case with nouveau on Linux as well as the Nvidia and AMD drivers on Windows (nvidia/fglrx on Linux is untested).  Therefore, using Q35 is often an unnecessary complication.

OVMF doesn't yet support Q35, so 440FX is currently your only option, but I expect that will eventually change.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2622 2014-08-31 18:32:05

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks again for answers. Now i know more than i wanted tongue i'll try omfv tommorow got some isp issues right now....

Offline

#2623 2014-08-31 19:35:54

Cenm
Member
Registered: 2014-08-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Cenm wrote:
slis wrote:

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

aw wrote:

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

Dumped BIOS from GPU. Running QEMU 2.1.0-2, Kernel 3.16.1-1, vfio bound

You're still not telling us how you're running it or what hardware you have other than a GTX650Ti.  If you expect your stock distro kernel to work, please see the OP or read-up on the blog below for the condensed version.  If you'd like to report a problem, please provide the base version and list any patches you've applied for QEMU and kernel, the kernel command line options you're booting with, the host and guest graphics devices, whether this is an AMD or Intel system, and the commands/configuration used to start the guest.  Please don't expect us to guess.

...

Hardware:
  3930K C2 Stepping
  Asus Extreme Rampage IV
  Nvidia GTX680 (Host (Nouveau Driver))
  Nvidia GTX560Ti (Guest)
 
Stock kernel unpatched, but >3.16 is the only thing mentioned as 'required'
No QEMU Patches are suggested...

Kernel param's
  intel_iommu=on
  pci-stub.ids=10de:1200,10de:0e0c (GTX 560Ti audio and video components)

Basic command on the first page to start the VM + romfile option

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/path/to/rom \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

Offline

#2624 2014-08-31 20:03:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cenm wrote:

Hardware:
  3930K C2 Stepping
  Asus Extreme Rampage IV
  Nvidia GTX680 (Host (Nouveau Driver))
  Nvidia GTX560Ti (Guest)
 
Stock kernel unpatched, but >3.16 is the only thing mentioned as 'required'
No QEMU Patches are suggested...

Kernel param's
  intel_iommu=on
  pci-stub.ids=10de:1200,10de:0e0c (GTX 560Ti audio and video components)

Basic command on the first page to start the VM + romfile option

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/path/to/rom \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ.  That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2625 2014-08-31 21:58:35

Cenm
Member
Registered: 2014-08-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ.  That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.

Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.

Offline

#2626 2014-08-31 22:02:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cenm wrote:
aw wrote:

It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ.  That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.

Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.

With the nvidia driver the problem is likely different, see FAQ question 9 and the link in the comment.  The nvidia driver takes the vga lock and never releases it, so the monitor stays blank in that case because the guest is stalled waiting for VGA access.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2627 2014-09-01 09:40:29

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

@ansa89 you could try to remove that bios-256k option and u don't need gpu romfile option... and if u edit that script when adding gpu, care for spaces / new lines... that could be reason for not booting

Now I'm using this script:

# Bind VGA DEVICES="0000:01:00.0 0000:01:00.1" for dev in $DEVICES ; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done # Start QEMU qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -rtc base=localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/dev/sdc,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -boot menu=on \ -netdev tap,ifname=qemu0,id=qemu_tap -device e1000,netdev=qemu_tap,mac=00:16:3E:12:34:56 \ -usb -usbdevice tablet

The script is ok, since if I change "-vga none" to "-vga std" everything works (so the problem is due to GTX 650 being the primary VGA device).
Moreover you said "not booting", but this isn't correct: the VM boots, but the monitor stays black and I can't connect to VNC server inside VM (I'm enough sure about this because I can arping the VM ip).

Last edited by Ansa89 (2014-09-01 09:42:04)

Offline

#2628 2014-09-01 14:05:31

Cenm
Member
Registered: 2014-08-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Cenm wrote:
aw wrote:

It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ.  That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.

Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.

With the nvidia driver the problem is likely different, see FAQ question 9 and the link in the comment.  The nvidia driver takes the vga lock and never releases it, so the monitor stays blank in that case because the guest is stalled waiting for VGA access.

Thanks again.

Which host driver would you suggest? I see some reports of instability with nouveau, is that still the case?

Offline

#2629 2014-09-01 15:01:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cenm wrote:
aw wrote:
Cenm wrote:

Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.

With the nvidia driver the problem is likely different, see FAQ question 9 and the link in the comment.  The nvidia driver takes the vga lock and never releases it, so the monitor stays blank in that case because the guest is stalled waiting for VGA access.

Thanks again.

Which host driver would you suggest? I see some reports of instability with nouveau, is that still the case?

Since I have no idea what you use the host graphics for, use whatever meets your needs.  I prefer to use open source drivers where I can, but with graphics cards you may only get full performance and functionality with proprietary drivers.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2630 2014-09-03 02:43:44

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.

I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?

Has anyone done this with an intel processor in an attempt to decrease stutter?
Any ideas where to go from here?

Last edited by sitonapanotis (2014-09-03 02:59:16)

Offline

#2631 2014-09-03 02:55:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:

So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.

I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?

Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?

Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it.  Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2632 2014-09-03 03:01:03

sitonapanotis
Member
Registered: 2014-01-08
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
sitonapanotis wrote:

So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.

I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?

Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?

Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it.  Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.


Ah I was only trying it because OP suggested it, but what I have read about it does seem to make what you are saying make more sense.
Any other ideas for troubleshooting latency/stuttering issues?

edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg

Last edited by sitonapanotis (2014-09-03 03:23:55)

Offline

#2633 2014-09-03 12:33:29

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've been using vfio-vga passthrough for quite some time now with my Radaeon R9 290 and everything has been working really great.

However after seeing aw's blog, I wanted to try to get libvirt working and then move to an OVMF setup but I'm having a few issues which I can't seem to resolve.

I have successfully setup 1 ubuntu guest using libvirt and virsh but I can't get my Windows8 guest, which has a Radeon R9 290 passed through to it to start. I'm getting the following:

[root@i7-4770s ~]# virsh start windows8
error: Failed to start domain windows8
error: internal error: early end of file from monitor: possible problem:
2014-09-03T13:07:19.859907Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f256     220c0c0, 0x0, 0x8be00000, 0x7f24b4200000) = -12 (Cannot allocate memory)
2014-09-03T13:07:19.859970Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listene     r initialization failed for container
2014-09-03T13:07:19.859982Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setu     p container for group 1
2014-09-03T13:07:19.860017Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get      group 1
2014-09-03T13:07:19.860031Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initializatio     n failed.
2014-09-03T13:07:19.860045Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,

I see some other posts on here about the same issue, but that seems to be when they wish to run start the guest as another user other than root. Regardless I have tried changing my ulimits but with no effect:

[root@i7-4770s ~]# ulimit -l
8650752

I have tried changing the RAM allocated to the Guest to 2GB from 6GB to see if there is any difference but no luck also.

Any suggestions please?

I'm running Arch Linux with the Kernel from the OP (3.15.0-1-mainline)

Thanks

Graham

Last edited by gneville (2014-09-03 13:15:00)

Offline

#2634 2014-09-03 13:35:32

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just spotted the following in dmseg, need to investigate further:

[   49.781160] qemu-system-x86: sending ioctl 5326 to a partition!
[   49.781168] qemu-system-x86: sending ioctl 80200204 to a partition!
[   49.781572] qemu-system-x86: sending ioctl 5326 to a partition!
[   49.781577] qemu-system-x86: sending ioctl 80200204 to a partition!
[   49.940979] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[   49.940993] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

gneville wrote:

Hi,

I've been using vfio-vga passthrough for quite some time now with my Radaeon R9 290 and everything has been working really great.

However after seeing aw's blog, I wanted to try to get libvirt working and then move to an OVMF setup but I'm having a few issues which I can't seem to resolve.

I have successfully setup 1 ubuntu guest using libvirt and virsh but I can't get my Windows8 guest, which has a Radeon R9 290 passed through to it to start. I'm getting the following:

[root@i7-4770s ~]# virsh start windows8
error: Failed to start domain windows8
error: internal error: early end of file from monitor: possible problem:
2014-09-03T13:07:19.859907Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f256     220c0c0, 0x0, 0x8be00000, 0x7f24b4200000) = -12 (Cannot allocate memory)
2014-09-03T13:07:19.859970Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listene     r initialization failed for container
2014-09-03T13:07:19.859982Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setu     p container for group 1
2014-09-03T13:07:19.860017Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get      group 1
2014-09-03T13:07:19.860031Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initializatio     n failed.
2014-09-03T13:07:19.860045Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,

I see some other posts on here about the same issue, but that seems to be when they wish to run start the guest as another user other than root. Regardless I have tried changing my ulimits but with no effect:

[root@i7-4770s ~]# ulimit -l
8650752

I have tried changing the RAM allocated to the Guest to 2GB from 6GB to see if there is any difference but no luck also.

Any suggestions please?

I'm running Arch Linux with the Kernel from the OP (3.15.0-1-mainline)

Thanks

Graham

Offline

#2635 2014-09-03 17:39:33

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:
aw wrote:
sitonapanotis wrote:

So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.

I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?

Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?

Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it.  Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.


Ah I was only trying it because OP suggested it, but what I have read about it does seem to make what you are saying make more sense.
Any other ideas for troubleshooting latency/stuttering issues?

edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg

main idea: explore and investigate yourself. the functionality in question is experimental after all. plus, it's the first time i hear about video stutter at all (maybe last few months i was not paying enough attention for all the reading i did, but still) so it may be not that common of a problem for you to wait for some outside help.

just try everything you can think of, including weird (==seemingly unrelated) stuff like playing around with audio settings (this, along with switch to hugetlbfs for memory, resolved my fps problems) or cpu isolation (run vm on cores NOT used by host at all)

Offline

#2636 2014-09-03 17:54:26

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:

Any other ideas for troubleshooting latency/stuttering issues?

edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg

Have a look at this post https://bbs.archlinux.org/viewtopic.php … 0#p1285910. If using Windows 8.x turning off dynticks could help, also verify whether your host kernel is preemptible (should have CONFIG_PREEMPT set).

Offline

#2637 2014-09-03 18:23:12

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using this for two guests and one guest (Linux guest with nVidia card) brings down the host on guest reboot.

Asrock Z97 Extreme6 mainboard
i4790K
i915 for host
AMD HD7950 for Win8.1 guest
nVidia GT 630 for Linux guest
on kernel 3.16 and qemu 2.1.0 with i915 and ACS override patches applied.

Boot parameters are

BOOT_IMAGE=/vmlinuz-3.16.1-vfio-i915-acs-vol-7.g90bc0f1-desktop root=/dev/mapper/vg1-root resume=/dev/vg1/swap splash=silent quiet showopts intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=1002:679a,1002:aaa0,10de:0f00,10de:0bea,8086:8ca0 modeset.nouveau=0 modeset.radeon=0

The devices I'm passing through are

00:1b.0 Audio device [0403]: Intel Corporation Device [8086:8ca0] Subsystem: ASRock Incorporation Device [1849:1151] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950] [1002:679a] Subsystem: Gigabyte Technology Co., Ltd Device [1458:254c] Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0] Subsystem: Gigabyte Technology Co., Ltd Device [1458:aaa0] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108 [GeForce GT 630] [10de:0f00] (rev a1) Subsystem: ASUSTeK Computer Inc. Device [1043:8498] Kernel driver in use: vfio-pci Kernel modules: nouveau 02:00.1 Audio device [0403]: NVIDIA Corporation GF108 High Definition Audio Controller [10de:0bea] (rev a1) Subsystem: ASUSTeK Computer Inc. Device [1043:8498] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

or the two graphics cards and the on-board sound (for the Win8.1 guest).

The Win8.1 guest with the AMD 7950 works perfectly. I was using an AMD 5450 for the second guest running Linux as well but was having the host crash on reboot with the radeon open source drivers, as well as low, stuttering desktop performance when using Cinnamon as DE (no issues with XFCE or LXDE), both with open radeon and fglrx drivers.

The AMD 5450 was swapped out for a nVidia 630 and this resolved the Cinnamon desktop performance issue, however when rebooting the guest this will again lock up the host (hard reset needed).

QEMU is invoked as

/usr/bin/qemu-system-x86_64 -name vm-linux -enable-kvm -rtc base=utc \ -M q35 -m 4096 -mem-path /dev/hugepages \ -smp sockets=1,cores=4 -cpu host \ -vga none -nographic -monitor stdio -serial none -parallel none -nodefconfig -boot menu=off \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=0000:02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=0000:02:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi \ -device ahci,bus=pcie.0,id=ahci -drive file=/dev/vg2/linux,id=disk,format=raw,cache=none,aio=threads,discard=unmap \ -device scsi-hd,drive=disk -drive file=/home/maarten/Downloads/openSUSE-13.1-DVD-x86_64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd -net nic,model=virtio,macaddr=52:54:00:6b:04:1d -net bridge,br=br0

I don't have this issue with the Windows guest at any point. I don't know if this is an issue resolved with the "other" VGA arbiter patch on https://lkml.org/lkml/2014/5/25/94, I do have the gfx card output on screen but I don't have complete understanding of what's going on here so I'm not certain. Thanks.

edit: I can shutdown the Linux guest without having the host hang but the guest cannot be started again:

(qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

Last edited by siddharta (2014-09-03 18:52:42)

Offline

#2638 2014-09-03 20:13:18

friedcpu
Member
Registered: 2014-09-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I've been following this thread for a while now, and a few months ago tried to passthrough my 2 NVIDIA Graphics cards, with the same results I am getting now, but this time I am determined to get it to work.

My specs are.
Asus X79-Deluxe
Intel 4930k
64 GB Ram
2GB NVIDIA GeForce GTX 770 (10de:1184) (Primary 16x Slot)
1GB NVIDIA GeForce GTX550Ti (10de:1244) (Secondary 16x Slot)

I am running the latest linux-mainline kernel from the #1 post with all patches, along with qemu-git, as of about 3hours ago. As I have no built in GPU with the processor, I am finding it very hard to release the primary GPU after booting. Here is my info and then I will get to what it is and isn't doing.

lspci 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1) lspci -n 01:00.0 0300: 10de:1184 (rev a1) 01:00.1 0403: 10de:0e0a (rev a1) 02:00.0 0300: 10de:1244 (rev a1) 02:00.1 0403: 10de:0bee (rev a1)
 Kernel command line: initrd=\initramfs-linux-mainline.img root=/dev/sda2 rw video=efifb:off intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a,10de:1244,10de:0bee [ 0.950144] pci-stub: add 10DE:1184 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.950164] pci-stub 0000:01:00.0: claimed by stub [ 0.950182] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.950194] pci-stub 0000:01:00.1: claimed by stub [ 0.950203] pci-stub: add 10DE:1244 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.950214] pci-stub 0000:02:00.0: claimed by stub [ 0.950226] pci-stub: add 10DE:0BEE sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.950237] pci-stub 0000:02:00.1: claimed by stub
Kernel Arguments (efifb:off to turn framebuffer off) root=/dev/sda2 rw video=efifb:off intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a,10de:1244,10de:0bee vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02.1 qemu (executed over ssh) qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/root/GK104.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -nographic qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 -nographic blacklist nouveau

Currently I can use device 02:00.0 absolutely fine, but if I try and use 01:00.0, I don't get any errors, just the monitor goes to sleep and this is in dmesg

 vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

I can still exit qemu fine, no lockups etc...

I have tried changing the cards around, and moving them to slower PCI slots, the GPU started by POST is the one that refuses to pass through. Both cards will pass through individually, providing they are not the first card. and I only get an Invalid ROM message when the GTX770 is in Slot 1, not in any other slot, I have verified my BIOS Dump with rom-parser, everything checks out.

When passing through the second card (in any PCI Slot, Either of the cards) I get the following in DMESG

vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) vfio-pci 0000:02:00.1: enabling device (0000 -> 0002)

But whenever I try and passthrough the first card, I get this

vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

The VGA part of the first card never gets enabled.

All todays testing was done on a fresh install with only the minimum for this installed, no xorg or any sort of graphics drivers.
I believe something is grabbing the primary card, but I have exhausted most of the search results related to passing through your only graphics cards without leaving one for the host.

Last edited by friedcpu (2014-09-04 08:10:42)

Offline

#2639 2014-09-03 23:58:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sitonapanotis wrote:
aw wrote:
sitonapanotis wrote:

So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.

I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?

Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?

Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it.  Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.


Ah I was only trying it because OP suggested it, but what I have read about it does seem to make what you are saying make more sense.
Any other ideas for troubleshooting latency/stuttering issues?

edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg

- huge pages
- vCPU pinning
- don't over commit CPUs
- try to get your assigned devices using MSI
- ???

Please share your findings.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2640 2014-09-04 00:01:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

Just spotted the following in dmseg, need to investigate further:

[   49.781160] qemu-system-x86: sending ioctl 5326 to a partition!
[   49.781168] qemu-system-x86: sending ioctl 80200204 to a partition!
[   49.781572] qemu-system-x86: sending ioctl 5326 to a partition!
[   49.781577] qemu-system-x86: sending ioctl 80200204 to a partition!
[   49.940979] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[   49.940993] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

What version of libvirt are you using?  Seems like it doesn't fully support vfio--based device assignment.  libvirt should be setting the locked memory limit for the vm process to a sufficient value to pin all of guest memory.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2641 2014-09-04 14:50:59

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the reply aw, this was totally my fault! I didn't add "clear_emulator_capabilities = 0" to /etc/libvirt/qemu.conf.

I am running libvirt 1.2.9.

I have successfully booted my Windows8 guest with the Radeon R9 290 passed through using libvirt now using a Q35 machine.

However when I try to define the OVMF Guest I get the following message:

error: Failed to define domain from /root/windows-ovmf-virsh-xml.xml
error: internal error: PCI bus is not compatible with the device at 0000:00:01.0. Device requires a standard PCI slot, which is not provided by bus 0000:00

My XML contains:

<type arch='i686' machine='pc'>hvm</type>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>

  <qemu:commandline>
    <qemu:arg value='-no-hpet'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='if=pflash,format=raw,readonly,file=/home/xen/vfio/OVMF.fd'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ahci,bus=pcie.0,id=ahci'/>
</qemu:commandline>

I've tried different varients of configuration but not having much luck, how does your xml look?


aw wrote:
gneville wrote:

Just spotted the following in dmseg, need to investigate further:

[   49.781160] qemu-system-x86: sending ioctl 5326 to a partition!
[   49.781168] qemu-system-x86: sending ioctl 80200204 to a partition!
[   49.781572] qemu-system-x86: sending ioctl 5326 to a partition!
[   49.781577] qemu-system-x86: sending ioctl 80200204 to a partition!
[   49.940979] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[   49.940993] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

What version of libvirt are you using?  Seems like it doesn't fully support vfio--based device assignment.  libvirt should be setting the locked memory limit for the vm process to a sufficient value to pin all of guest memory.

Thanks

Graham

Last edited by gneville (2014-09-04 15:13:46)

Offline

#2642 2014-09-04 15:22:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

Thanks for the reply aw, this was totally my fault! I didn't add "clear_emulator_capabilities = 0" to /etc/libvirt/qemu.conf.

I am running libvirt 1.2.9.

I have successfully booted my Windows8 guest with the Radeon R9 290 passed through using libvirt now using a Q35 machine.

However when I try to define the OVMF Guest I get the following message:

error: Failed to define domain from /root/windows-ovmf-virsh-xml.xml
error: internal error: PCI bus is not compatible with the device at 0000:00:01.0. Device requires a standard PCI slot, which is not provided by bus 0000:00

My XML contains:

<type arch='i686' machine='pc'>hvm</type>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>

  <qemu:commandline>
    <qemu:arg value='-no-hpet'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='if=pflash,format=raw,readonly,file=/home/xen/vfio/OVMF.fd'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ahci,bus=pcie.0,id=ahci'/>
</qemu:commandline>

I've tried different varients of configuration but not having much luck, how does your xml look?

Please re-read the blog post, especially the bit about OVMF not supporting Q35.  If you assign the GPU with a hostdev definition (which you can do when you don't need the x-vga=on option, which you don't need when using OVMF), then you don't need any special libvirt configurations (like retaining capabilities, running as root, or defining ACLs for /dev/vfio/foo).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2643 2014-09-04 15:34:06

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks again, aw. I'll re-read it.

I did change the PC to:

<type arch='i686' machine='pc'>hvm</type>

I'm guessing the config will be something like this then, I'll give it a go when I get home:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </hostdev>

aw wrote:

Please re-read the blog post, especially the bit about OVMF not supporting Q35.  If you assign the GPU with a hostdev definition (which you can do when you don't need the x-vga=on option, which you don't need when using OVMF), then you don't need any special libvirt configurations (like retaining capabilities, running as root, or defining ACLs for /dev/vfio/foo).

Offline

#2644 2014-09-04 15:39:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

Thanks again, aw. I'll re-read it.

I did change the PC to:

<type arch='i686' machine='pc'>hvm</type>

I'm guessing the config will be something like this then, I'll give it a go when I get home:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </hostdev>

Yes, exactly.  My full xml looks like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows8-OVMF</name> <uuid>98dd8584-aa6a-49c3-b19e-8cbb942d5261</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <topology sockets='1' cores='2' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/home/alwillia/local/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/store/vm/win8.ovmf'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <interface type='bridge'> <mac address='02:12:34:56:78:92'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <input type='tablet' bus='usb'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/> </qemu:commandline> </domain>

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2645 2014-09-04 17:07:47

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:

edit: Rebooting the Linux guest will lock up the host. I can shutdown the Linux guest without having the host hang but the guest cannot then be started again:

(qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

In trying to resolve this I've been going through this thread again as I had diagnonally read most posts involving nVidia cards (didn't own one until now).  I dumped the ROM of the GT 630 from within the guest with the nVidia gpu using

echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom cat /sys/bus/pci/devices/0000:01:00.0/rom > asus-gt630-dcsl-2gd3-v2-vbios.rom echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

and added this to the QEMU invocation as

-device vfio-pci,host=$GPU,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/user/asus-gt630-dcsl-2gd3-v2-vbios.rom

but no success - host still hangs on guest reboot, same as without romfile specified.

The other suggestion in the error message, adding rombar=0, doesn't work either. - there is no display output when specifying this. I didn't try both together as a previous post indicated this doesn't make sense. I don't find another way forward in this thread unless I missed something. Any help would be very much appreciated, this is really bugging me.

A second issue I'm seeing is when passing through a USB controller

### Group 25 ### 0b:00.0 USB controller: ASMedia Technology Inc. Device 1142

that is bound to pci-stub at boot

0b:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1142] Subsystem: ASRock Incorporation Device [1849:1142] Kernel driver in use: pci-stub Kernel modules: xhci_hcd

. Using

-device vfio-pci,host=$USB,bus=pcie.0

in the QEMU command results in dmesg

[2014-09-04T17:45:38.943613+01:00 kvmhost-3 kernel: [ 1553.980492] dmar: DRHD: handling fault status reg 2 2014-09-04T17:45:38.943638+01:00 kvmhost-3 kernel: [ 1553.980504] dmar: DMAR:[DMA Read] Request device [0b:00.0] fault addr ee000 2014-09-04T17:45:38.943642+01:00 kvmhost-3 kernel: [ 1553.980504] DMAR:[fault reason 06] PTE Read access is not set

and nothing beyond that. The guest doesn't see the USB controller.

I'm attempting to passthrough the whole controller because passing a single usb device (regardless of the device type and port the device is attached to) gives a stream of errors as

2014-09-04T10:09:31.794293+01:00 kvmhost-3 kernel: [ 153.883612] xhci_hcd 0000:00:14.0: Command completion event does not match command 2014-09-04T10:09:31.794311+01:00 kvmhost-3 kernel: [ 153.883633] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command 2014-09-04T10:09:37.010279+01:00 kvmhost-3 kernel: [ 159.099819] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command 2014-09-04T10:09:37.211259+01:00 kvmhost-3 kernel: [ 159.300820] usb 3-9.1: device not accepting address 4, error -62 2014-09-04T10:09:37.291256+01:00 kvmhost-3 kernel: [ 159.380909] usb 3-9.1: reset full-speed USB device number 4 using xhci_hcd 2014-09-04T10:09:37.291260+01:00 kvmhost-3 kernel: [ 159.381010] usb 3-9.1: Device not responding to setup address. 2014-09-04T10:09:37.492268+01:00 kvmhost-3 kernel: [ 159.581889] usb 3-9.1: Device not responding to setup address. 2014-09-04T10:09:37.693274+01:00 kvmhost-3 kernel: [ 159.782852] usb 3-9.1: device not accepting address 4, error -71 2014-09-04T10:09:37.693287+01:00 kvmhost-3 kernel: [ 159.783255] usb 3-9.1: USB disconnect, device number 4 2014-09-04T10:09:37.693288+01:00 kvmhost-3 kernel: [ 159.783347] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880444cff940 2014-09-04T10:09:37.774261+01:00 kvmhost-3 kernel: [ 159.863962] usb 3-9.1: new full-speed USB device number 6 using xhci_hcd 2014-09-04T10:09:37.855287+01:00 kvmhost-3 kernel: [ 159.944987] usb 3-9.1: device descriptor read/64, error -32 2014-09-04T10:09:37.956285+01:00 kvmhost-3 kernel: [ 160.045876] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 6. 2014-09-04T10:09:37.956292+01:00 kvmhost-3 kernel: [ 160.045878] usb 3-9.1: hub failed to enable device, error -22 2014-09-04T10:09:38.037284+01:00 kvmhost-3 kernel: [ 160.126992] usb 3-9.1: new full-speed USB device number 7 using xhci_hcd 2014-09-04T10:09:38.118289+01:00 kvmhost-3 kernel: [ 160.208006] usb 3-9.1: device descriptor read/64, error -32 2014-09-04T10:09:38.219271+01:00 kvmhost-3 kernel: [ 160.308883] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 7. 2014-09-04T10:09:38.219278+01:00 kvmhost-3 kernel: [ 160.308886] usb 3-9.1: hub failed to enable device, error -22 2014-09-04T10:09:38.299285+01:00 kvmhost-3 kernel: [ 160.389022] usb 3-9.1: new full-speed USB device number 8 using xhci_hcd 2014-09-04T10:09:38.299291+01:00 kvmhost-3 kernel: [ 160.389127] usb 3-9.1: Device not responding to setup address. 2014-09-04T10:09:38.500284+01:00 kvmhost-3 kernel: [ 160.589984] usb 3-9.1: Device not responding to setup address. 2014-09-04T10:09:38.701266+01:00 kvmhost-3 kernel: [ 160.790900] usb 3-9.1: device not accepting address 8, error -71 2014-09-04T10:09:38.781261+01:00 kvmhost-3 kernel: [ 160.870989] usb 3-9.1: new full-speed USB device number 9 using xhci_hcd 2014-09-04T10:09:38.781268+01:00 kvmhost-3 kernel: [ 160.871083] usb 3-9.1: Device not responding to setup address. 2014-09-04T10:09:38.982281+01:00 kvmhost-3 kernel: [ 161.071992] usb 3-9.1: Device not responding to setup address. 2014-09-04T10:09:39.183287+01:00 kvmhost-3 kernel: [ 161.272923] usb 3-9.1: device not accepting address 9, error -71 2014-09-04T10:09:39.183294+01:00 kvmhost-3 kernel: [ 161.273058] usb 3-9-port1: unable to enumerate USB device

and brings down the host eventually. Any insight into this is also appreciated. I've googled what I can and it may be a problem with libusb that I'm seeing but I'm not certain how to pinpoint the root cause definitely.

Offline

#2646 2014-09-04 18:04:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
siddharta wrote:

edit: Rebooting the Linux guest will lock up the host. I can shutdown the Linux guest without having the host hang but the guest cannot then be started again:

(qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

In trying to resolve this I've been going through this thread again as I had diagnonally read most posts involving nVidia cards (didn't own one until now).  I dumped the ROM of the GT 630 from within the guest with the nVidia gpu using

echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom cat /sys/bus/pci/devices/0000:01:00.0/rom > asus-gt630-dcsl-2gd3-v2-vbios.rom echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

and added this to the QEMU invocation as

-device vfio-pci,host=$GPU,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/user/asus-gt630-dcsl-2gd3-v2-vbios.rom

but no success - host still hangs on guest reboot, same as without romfile specified.

The other suggestion in the error message, adding rombar=0, doesn't work either. - there is no display output when specifying this. I didn't try both together as a previous post indicated this doesn't make sense. I don't find another way forward in this thread unless I missed something. Any help would be very much appreciated, this is really bugging me.

Host hangs can be pretty challenging to debug, especially without at least a serial console.  I'd probably start with the shutdown case you list and compare lspci -vvv in the host before and after a guest boot + shutdown.  Also, don't forget that you're applying and enabling ACS override in your kernel, which assumes isolation where it may not exist.  I'm not sure if that's contributing or not, but you may want to try the card in another slot rather than ignoring the advertised isolation of devices.  Things like ASPM and AER may play a role.  You can find options to change or disable these in kernel-parameters.txt (https://git.kernel.org/cgit/linux/kerne … meters.txt).

A second issue I'm seeing is when passing through a USB controller

### Group 25 ### 0b:00.0 USB controller: ASMedia Technology Inc. Device 1142

that is bound to pci-stub at boot

0b:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1142] Subsystem: ASRock Incorporation Device [1849:1142] Kernel driver in use: pci-stub Kernel modules: xhci_hcd

. Using

-device vfio-pci,host=$USB,bus=pcie.0

in the QEMU command results in dmesg

[2014-09-04T17:45:38.943613+01:00 kvmhost-3 kernel: [ 1553.980492] dmar: DRHD: handling fault status reg 2 2014-09-04T17:45:38.943638+01:00 kvmhost-3 kernel: [ 1553.980504] dmar: DMAR:[DMA Read] Request device [0b:00.0] fault addr ee000 2014-09-04T17:45:38.943642+01:00 kvmhost-3 kernel: [ 1553.980504] DMAR:[fault reason 06] PTE Read access is not set

and nothing beyond that. The guest doesn't see the USB controller.

Another quality ASMedia product...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2647 2014-09-04 18:56:22

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Host hangs can be pretty challenging to debug, especially without at least a serial console.  I'd probably start with the shutdown case you list and compare lspci -vvv in the host before and after a guest boot + shutdown.  Also, don't forget that you're applying and enabling ACS override in your kernel, which assumes isolation where it may not exist.  I'm not sure if that's contributing or not, but you may want to try the card in another slot rather than ignoring the advertised isolation of devices.  Things like ASPM and AER may play a role.  You can find options to change or disable these in kernel-parameters.txt (https://git.kernel.org/cgit/linux/kerne … meters.txt).

Thanks for your feedback. Interesting.

lspci -vvv -s 02:00.

gives

02:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 630] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 8498 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 10 Region 0: Memory at ee000000 (32-bit, non-prefetchable) [disabled] [size=16M] Region 1: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M] Region 5: I/O ports at d000 [disabled] [size=128] Expansion ROM at ef000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: pci-stub Kernel modules: nouveau

whereas after boot and shutdown of guest this becomes (with diff before after)

3,9c3,9 < Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- < Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- < Interrupt: pin A routed to IRQ 10 < Region 0: Memory at ee000000 (32-bit, non-prefetchable) [disabled] [size=16M] < Region 1: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=256M] < Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M] < Region 5: I/O ports at d000 [disabled] [size=128] --- > Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+ > Interrupt: pin A routed to IRQ 17 > Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=16M] > Region 1: Memory at b0000000 (64-bit, prefetchable) [size=256M] > Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M] > Region 5: I/O ports at d000 [size=128] 15c15 < Address: 0000000000000000 Data: 0000 --- > Address: 00000000fee004b8 Data: 0000 17c17 < DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us --- > DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us

but I'm not certain how to relate this correctly and whether this would be related to the issues I'm seeing (host hang on guest reboot and host crash + QEMU stating "invalid ROM contents" on guest shutdown+boot again).

aw wrote:

Another quality ASMedia product...

I wouldn't touch it with a ten foot pole if I would have USB device passthrough working properly, which unfortunately is not the case. edit: interestingly enough I find the same USB passthrough errors in dmesg on an Asrock Z77 mainboard but these don't seem to bring down the host there. Thanks.

Last edited by siddharta (2014-09-04 19:50:00)

Offline

#2648 2014-09-04 19:16:53

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Getting there slowly, but still having issues.

It looks like I've managed to successfully pass through my Radeon R9 290, although I can only tell his if I enable cirrus graphics and VNC and when at the UEFI Shell performing "pci" to see the Vendor and Device IDs that match the same as when performing "lsusb -n"

However, I can't get any display to appear when starting the guest without cirrus and VNC and can't get it to boot in to Windows either.

My XML now looks like so

 <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0' /> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <qemu:commandline> <qemu:arg value='-no-hpet'/> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin'/> </qemu:commandline>

I have just built the latest TianoCore OVMF image using "yaourt -S ovmf-svn" svn version was 16059.

If I VNC in to the device, I see :

Boot Failed EFI Floppy Boot Failed EFI Floppy 1 Boot Failed Misc Device Boot Failed Misc Device 1

The next screen shows me a mapping table where by I can see what looks like 3 Hard Drives with MBRs on, but I get no further.


aw wrote:

Yes, exactly.  My full xml looks like this:

Offline

#2649 2014-09-04 20:19:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

Getting there slowly, but still having issues.

It looks like I've managed to successfully pass through my Radeon R9 290, although I can only tell his if I enable cirrus graphics and VNC and when at the UEFI Shell performing "pci" to see the Vendor and Device IDs that match the same as when performing "lsusb -n"

However, I can't get any display to appear when starting the guest without cirrus and VNC and can't get it to boot in to Windows either.

My XML now looks like so

 <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0' /> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <qemu:commandline> <qemu:arg value='-no-hpet'/> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin'/> </qemu:commandline>

I have just built the latest TianoCore OVMF image using "yaourt -S ovmf-svn" svn version was 16059.

If I VNC in to the device, I see :

Boot Failed EFI Floppy Boot Failed EFI Floppy 1 Boot Failed Misc Device Boot Failed Misc Device 1

The next screen shows me a mapping table where by I can see what looks like 3 Hard Drives with MBRs on, but I get no further.

Did you test the ROM for your card to see if it includes EFI support?  When I have emulated graphics and an assigned graphics, OVMF initializes both.  You will need to reinstall your guest, you can't simply move the disk from a legacy BIOS to a UEFI system and expect it to work, on bare metal or virtual machine.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2650 2014-09-04 20:28:35

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've just ran the test from your website and I don't see the type 3, an EFI ROM. I'm very surprised as to why not as this is a brand new card, UEFI is written on the box and stated on the website.

Thanks for the heads up with the re-install.

Edit: Just found there is a BIOS selection switch physically on the card! Will dismantle and switch to the other BIOS and test again.

http://www.sapphiretech.com/presentatio … &psn&lid=1

[root@i7-4770s rom-parser]# ./rom-parser /root/r9-290.rom Valid ROM signature found @0h, PCIR offset 238h PCIR: type 0, vendor: 1002, device: 67b1, class: 030000 PCIR: revision 0, vendor revision: f2a Last image
aw wrote:

Did you test the ROM for your card to see if it includes EFI support?  When I have emulated graphics and an assigned graphics, OVMF initializes both.  You will need to reinstall your guest, you can't simply move the disk from a legacy BIOS to a UEFI system and expect it to work, on bare metal or virtual machine.

Last edited by gneville (2014-09-04 20:31:45)

Offline

#2651 2014-09-04 20:46:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

I've just ran the test from your website and I don't see the type 3, an EFI ROM. I'm very surprised as to why not as this is a brand new card, UEFI is written on the box and stated on the website.

Thanks for the heads up with the re-install.

Edit: Just found there is a BIOS selection switch physically on the card! Will dismantle and switch to the other BIOS and test again.

http://www.sapphiretech.com/presentatio … &psn&lid=1

[root@i7-4770s rom-parser]# ./rom-parser /root/r9-290.rom Valid ROM signature found @0h, PCIR offset 238h PCIR: type 0, vendor: 1002, device: 67b1, class: 030000 PCIR: revision 0, vendor revision: f2a Last image

That's disappointing.  The MSI "Gaming" version updated in February seems to have an update with EFI support, try that if you dare.

http://www.techpowerup.com/vgabios/inde … del=R9+290

BTW, I pushed a tiny fix to rom-parser that makes it scan that one correctly, otherwise it gets lost with a bogus match.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2652 2014-09-04 20:54:36

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I found a physical switch on the card that I had to move over. I've flicked it and booted back up again.

I now see the "type 3" smile

[root@i7-4770s rom-parser]# ./rom-parser /root/r9-290-uefi.rom Valid ROM signature found @0h, PCIR offset 238h PCIR: type 0, vendor: 1002, device: 67b1, class: 030000 PCIR: revision 0, vendor revision: f2a Valid ROM signature found @10000h, PCIR offset 1ch PCIR: type 3, vendor: 1002, device: 67b1, class: 030000 PCIR: revision 0, vendor revision: 0 EFI: Signature Valid Last image

The good news is, I now see a TianoCore BIOS screen via my HDMI port, so I can successfully passthrough my graphics card smile

I'll install windows from fresh and see if everything works, then once confirmed it does, I'll be looking to move on to Kernel 3.17 without the i915 patches.

Fingers Crossed & Thanks for the help!


aw wrote:

That's disappointing.  The MSI "Gaming" version updated in February seems to have an update with EFI support, try that if you dare.

http://www.techpowerup.com/vgabios/inde … del=R9+290

BTW, I pushed a tiny fix to rom-parser that makes it scan that one correctly, otherwise it gets lost with a bogus match.

Last edited by gneville (2014-09-04 21:03:37)

Offline

#2653 2014-09-04 21:24:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

I found a physical switch on the card that I had to move over. I've flicked it and booted back up again.

I now see the "type 3" smile

[root@i7-4770s rom-parser]# ./rom-parser /root/r9-290-uefi.rom Valid ROM signature found @0h, PCIR offset 238h PCIR: type 0, vendor: 1002, device: 67b1, class: 030000 PCIR: revision 0, vendor revision: f2a Valid ROM signature found @10000h, PCIR offset 1ch PCIR: type 3, vendor: 1002, device: 67b1, class: 030000 PCIR: revision 0, vendor revision: 0 EFI: Signature Valid Last image

Very strange, but I'm glad you found it.  The web seems to indicate the switch is for "quiet" or "uber" mode.  Any idea if it's just the "uber" mode that doesn't support EFI?  Which is default?

The good news is, I now see a TianoCore BIOS screen via my HDMI port, so I can successfully passthrough my graphics card smile

I'll install windows from fresh and see if everything works, then once confirmed it does, I'll be looking to move on to Kernel 3.17 without the i915 patches.

Good luck


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2654 2014-09-04 21:34:21

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello. I'm desperadly looking for help. I have a typical problem, that seem to happed to many people in this thread. Unfortunately I cannot overcome it. I don't see any seabios output on the screen attached to the card.

My hardware:
MB: Asus P8Z77-V LK, firmware 1402
CPU: Core i7 3770 (not the one with K)
GPU: Asus NVIDIA GTX 770 DirectCU II OC (inserted in the most top pci-e slot)

Software:
Debian Testing (don't kick me please)
3.16 Kernel with "acs override" & "i915 vga arbiter" & "vga arbitrer" patches
qemu: latest master from git://git.qemu.org/qemu.git
seabios: latest master from git://git.seabios.org/seabios.git

kernel config:

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_INTEL_IOMMU=y CONFIG_INTEL_IOMMU_DEFAULT_ON=y

Boot args (I tried all possible combinations):

intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1184,10de:0e0a
cat /etc/modprobe.d/blacklist.conf blacklist nouveau blacklist nvidia blacklist i915

I previously didn't have i915 blacklisted, no effect. I use nouveau driver, so I don't have nvidia binary installed, but I blacklist it anyway. I'm just really trying everything.

lspci -nn | grep NVIDIA 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

I set my intel GPU as primary in bios. I can successfully bind NVIDIA using the script in the first post:

sudo /usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

This looks ok:

ls -l /sys/bus/pci/drivers/vfio-pci/ total 0 lrwxrwxrwx 1 root root 0 Sep 5 00:09 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Sep 5 00:09 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 --w------- 1 root root 4096 Sep 5 00:09 bind lrwxrwxrwx 1 root root 0 Sep 5 00:09 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 Sep 4 23:31 new_id --w------- 1 root root 4096 Sep 5 00:09 remove_id --w------- 1 root root 4096 Sep 4 23:31 uevent --w------- 1 root root 4096 Sep 5 00:09 unbind

No errors here:

dmesg |dmesg | grep -i -e pci-stub -e vfio -e iommu -e dmar # (some lines skipped) [ 0.000000] Intel-IOMMU: enabled [ 0.079612] dmar: Host address width 36 [ 0.079614] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.079621] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.079622] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.079625] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.079626] dmar: RMRR base: 0x000000bc5b3000 end: 0x000000bc5c4fff [ 0.079627] dmar: RMRR base: 0x000000bf000000 end: 0x000000cf1fffff [ 0.079697] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.560480] DMAR: No ATSR found [ 0.560500] IOMMU 0 0xfed90000: using Queued invalidation [ 0.560500] IOMMU 1 0xfed91000: using Queued invalidation [ 0.560501] IOMMU: Setting RMRR: [ 0.560510] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff] [ 0.561572] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc5b3000 - 0xbc5c4fff] [ 0.561587] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc5b3000 - 0xbc5c4fff] [ 0.561600] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc5b3000 - 0xbc5c4fff] [ 0.561608] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.561613] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.561786] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.16.0-vfio-ally #1 [ 0.561801] [<ffffffff813e50b6>] ? intel_iommu_add_device+0x36/0x200 [ 0.561803] [<ffffffff813da240>] ? bus_set_iommu+0x50/0x50 [ 0.561805] [<ffffffff813da260>] ? add_iommu_group+0x20/0x50 [ 0.561808] [<ffffffff813da22e>] ? bus_set_iommu+0x3e/0x50 [ 0.561811] [<ffffffff8192ac05>] ? intel_iommu_init+0x4fe/0x582 [ 0.561814] [<ffffffff818ec7de>] ? pci_iommu_init+0xe/0x37 [ 0.588450] VFIO - User Level meta-driver version: 0.3 [ 8.299028] pci-stub: add 10DE:1184 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 8.299042] pci-stub 0000:01:00.0: claimed by stub [ 8.299049] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 8.299149] vboxpci: IOMMU found [ 8.392654] [drm] DMAR active, disabling use of stolen memory

Then I run qemu as

sudo ~/qemu.git/x86_64-softmmu/qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 1024 \ -cpu host \ -smp 1,sockets=1,cores=1,threads=1 \ -bios ~/seabios.git/out/bios.bin \ -L ~/seabios.git/out/ -L ~/seabios.git/out/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I see SDL qemu window but no signal on HDMI or DVI output of NVIDIA. I can still see these messages in dmesg when starting qemu:

[ 44.915488] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 71.971217] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 495.033377] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

I googled, read mail lists but nothing helpfull. This is SOS, please help.


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#2655 2014-09-04 21:48:07

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zsph wrote:

Hello. I'm desperadly looking for help. I have a typical problem, that seem to happed to many people in this thread. Unfortunately I cannot overcome it. I don't see any seabios output on the screen attached to the card.

My hardware:
MB: Asus P8Z77-V LK, firmware 1402
CPU: Core i7 3770 (not the one with K)
GPU: Asus NVIDIA GTX 770 DirectCU II OC (inserted in the most top pci-e slot)

Software:
Debian Testing (don't kick me please)
3.16 Kernel with "acs override" & "i915 vga arbiter" & "vga arbitrer" patches
qemu: latest master from git://git.qemu.org/qemu.git
seabios: latest master from git://git.seabios.org/seabios.git

kernel config:

CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_INTEL_IOMMU=y CONFIG_INTEL_IOMMU_DEFAULT_ON=y

Boot args (I tried all possible combinations):

intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1184,10de:0e0a
cat /etc/modprobe.d/blacklist.conf blacklist nouveau blacklist nvidia blacklist i915

I previously didn't have i915 blacklisted, no effect. I use nouveau driver, so I don't have nvidia binary installed, but I blacklist it anyway. I'm just really trying everything.

lspci -nn | grep NVIDIA 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

I set my intel GPU as primary in bios. I can successfully bind NVIDIA using the script in the first post:

sudo /usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

This looks ok:

ls -l /sys/bus/pci/drivers/vfio-pci/ total 0 lrwxrwxrwx 1 root root 0 Sep 5 00:09 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Sep 5 00:09 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 --w------- 1 root root 4096 Sep 5 00:09 bind lrwxrwxrwx 1 root root 0 Sep 5 00:09 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 Sep 4 23:31 new_id --w------- 1 root root 4096 Sep 5 00:09 remove_id --w------- 1 root root 4096 Sep 4 23:31 uevent --w------- 1 root root 4096 Sep 5 00:09 unbind

No errors here:

dmesg |dmesg | grep -i -e pci-stub -e vfio -e iommu -e dmar # (some lines skipped) [ 0.000000] Intel-IOMMU: enabled [ 0.079612] dmar: Host address width 36 [ 0.079614] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.079621] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.079622] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.079625] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.079626] dmar: RMRR base: 0x000000bc5b3000 end: 0x000000bc5c4fff [ 0.079627] dmar: RMRR base: 0x000000bf000000 end: 0x000000cf1fffff [ 0.079697] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.560480] DMAR: No ATSR found [ 0.560500] IOMMU 0 0xfed90000: using Queued invalidation [ 0.560500] IOMMU 1 0xfed91000: using Queued invalidation [ 0.560501] IOMMU: Setting RMRR: [ 0.560510] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff] [ 0.561572] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc5b3000 - 0xbc5c4fff] [ 0.561587] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc5b3000 - 0xbc5c4fff] [ 0.561600] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc5b3000 - 0xbc5c4fff] [ 0.561608] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.561613] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.561786] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.16.0-vfio-ally #1 [ 0.561801] [<ffffffff813e50b6>] ? intel_iommu_add_device+0x36/0x200 [ 0.561803] [<ffffffff813da240>] ? bus_set_iommu+0x50/0x50 [ 0.561805] [<ffffffff813da260>] ? add_iommu_group+0x20/0x50 [ 0.561808] [<ffffffff813da22e>] ? bus_set_iommu+0x3e/0x50 [ 0.561811] [<ffffffff8192ac05>] ? intel_iommu_init+0x4fe/0x582 [ 0.561814] [<ffffffff818ec7de>] ? pci_iommu_init+0xe/0x37 [ 0.588450] VFIO - User Level meta-driver version: 0.3 [ 8.299028] pci-stub: add 10DE:1184 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 8.299042] pci-stub 0000:01:00.0: claimed by stub [ 8.299049] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 8.299149] vboxpci: IOMMU found [ 8.392654] [drm] DMAR active, disabling use of stolen memory

Then I run qemu as

sudo ~/qemu.git/x86_64-softmmu/qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 1024 \ -cpu host \ -smp 1,sockets=1,cores=1,threads=1 \ -bios ~/seabios.git/out/bios.bin \ -L ~/seabios.git/out/ -L ~/seabios.git/out/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I see SDL qemu window but no signal on HDMI or DVI output of NVIDIA. I can still see these messages in dmesg when starting qemu:

[ 44.915488] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 71.971217] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 495.033377] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

I googled, read mail lists but nothing helpfull. This is SOS, please help.

This would probably not be the root cause of your issue, but i915 shouldn't have to be blacklisted, you probably don't need allow_unsafe_interrupts=1 nor the other VGA arbiter patch, only the i915 vgaarb patch. Have a look at aw's blog vfio.blogspot.com in the FAQ post.
edit: also, depending on your exact situation, your video card may already be in its own VFIO group. If that's the case and you're not passing other devices that are co-habiting a VFIO group with other devices that you don't want to pass to the guest then you don't need the ACS override patch either. You can search this thread for a script called lsgroups.sh that presents this nicely.

Last edited by siddharta (2014-09-04 21:51:32)

Offline

#2656 2014-09-04 22:12:39

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It just seems on the Sapphire Radeon R9 290 that the Default BIOS is legacy and the other switch is for UEFI. Don't think there is a quiet and uber mode on this card.

http://www.hardwareluxx.com/index.php/r … -x-oc.html

aw wrote:

Very strange, but I'm glad you found it.  The web seems to indicate the switch is for "quiet" or "uber" mode.  Any idea if it's just the "uber" mode that doesn't support EFI?  Which is default?

Offline

#2657 2014-09-04 23:01:44

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Interestingly enough, after installing closed source nvidia drivers in the guest instead of nouveau, the host no longer hangs on guest reboot, nor do I have the same behaviour after shutdown and restart of the guest (QEMU throwing "invalid rom contents" error followed by the host locking up unless QEMU is terminated). The guest doesn't seem to come back up properly after a reboot or shutdown, however I can't see what is going on now (connected to the host over SSH) so I'll see that tomorrow. The guest doesn't respond but doesn' crash the host either which is a step in the right direction, dmesg doesn't list any errors/warnings either. There are slight differences  between the nouveau and nvidia proprietary drivers in the

lspci -vvv -s 02:00.0

output taken after shutting down the guest, I can post them if it would be of interest.

I actually had a similar experience with an AMD 5450 card - host hanging after reboot with the radeon open source drivers installed in the guest, but working fine with the closed source fglrx drivers on the guest.

As to swapping the card around physically to see whether the ACS override is required, I have another (physically) x16 slot but it's not possible to install the nvidia card there due to the size of the heatsink on the card. Thanks.

edit: removed quote

Last edited by siddharta (2014-09-04 23:02:50)

Offline

#2658 2014-09-04 23:25:47

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for prompt reply.

siddharta wrote:

This would probably not be the root cause of your issue, but i915 shouldn't have to be blacklisted, you probably don't need allow_unsafe_interrupts=1 nor the other VGA arbiter patch, only the i915 vgaarb patch. Have a look at aw's blog vfio.blogspot.com in the FAQ post.

edit: also, depending on your exact situation, your video card may already be in its own VFIO group. If that's the case and you're not passing other devices that are co-habiting a VFIO group with other devices that you don't want to pass to the guest then you don't need the ACS override patch either. You can search this thread for a script called lsgroups.sh that presents this nicely.

I found the script, the card is the only member in its group:

### Group 13 ### 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)

Is there a chance that without unnecessary patches and boot args the outcome can be different? I will certainly try, it just takes hours to rebuild smile


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#2659 2014-09-04 23:48:17

siddharta
Member
Registered: 2014-05-03
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zsph wrote:

Is there a chance that without unnecessary patches and boot args the outcome can be different? I will certainly try, it just takes hours to rebuild smile

You can remove the allow unsafe interrupts and acs override boot parameters (this will disable the acs override patch) but to remove the non-i915 VGA arbiter patch you'll need to rebuild the kernel, unless there is a parameter to disable it (look at the patch notes, I'm not sure). The first two will likely not improve matters but removing the non-i915 VGA arbiter patch could possibly solve your issue, I'm not certain though. Do keep the i915 vgaarb patch and the corresponding boot parameter, that one is not optional if using Intel igfx on the host on kernel below 3.17.

Offline

#2660 2014-09-04 23:56:39

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw, everything is looking good for UEFI booting. The only thing I can't do is restart the Windows8 Guest, this just causes a booting loop, but this was similar to before whereby I have to apply the fixes found in this blog - http://blog.ktz.me/?p=219

I've moved up to Linux 3.17 RC3 Kernel and removed the "i915.enable_hd_vgaarb=1"

My prompt is now:

intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:67b1,1002:aac8 quiet

I successfully have my Intel chipset outputting XBMC whilst my AMD R9 290 is outputting Windows. smile


gneville wrote:

It just seems on the Sapphire Radeon R9 290 that the Default BIOS is legacy and the other switch is for UEFI. Don't think there is a quiet and uber mode on this card.

http://www.hardwareluxx.com/index.php/r … -x-oc.html

aw wrote:

Very strange, but I'm glad you found it.  The web seems to indicate the switch is for "quiet" or "uber" mode.  Any idea if it's just the "uber" mode that doesn't support EFI?  Which is default?

Offline

#2661 2014-09-05 00:05:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

siddharta wrote:
zsph wrote:

Is there a chance that without unnecessary patches and boot args the outcome can be different? I will certainly try, it just takes hours to rebuild smile

You can remove the allow unsafe interrupts and acs override boot parameters (this will disable the acs override patch) but to remove the non-i915 VGA arbiter patch you'll need to rebuild the kernel, unless there is a parameter to disable it (look at the patch notes, I'm not sure). The first two will likely not improve matters but removing the non-i915 VGA arbiter patch could possibly solve your issue, I'm not certain though. Do keep the i915 vgaarb patch and the corresponding boot parameter, that one is not optional if using Intel igfx on the host on kernel below 3.17.

I sure hope that removing the other VGA arbiter patch doesn't make a difference, that's upstream now.  IIRC, you do need to not blacklist i915 if you're using VGA-mode assignment.  We need the i915 driver to participate in arbitration and it likely does make a difference.  The ACS override patch shouldn't matter if it's not enabled.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2662 2014-09-05 00:08:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

aw, everything is looking good for UEFI booting. The only thing I can't do is restart the Windows8 Guest, this just causes a booting loop, but this was similar to before whereby I have to apply the fixes found in this blog - http://blog.ktz.me/?p=219

Is it really a boot loop or is it a ~1min delay in rebooting?  (see the Aug 28th update on the blog)

I've moved up to Linux 3.17 RC3 Kernel and removed the "i915.enable_hd_vgaarb=1"

My prompt is now:

intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:67b1,1002:aac8 quiet

I successfully have my Intel chipset outputting XBMC whilst my AMD R9 290 is outputting Windows. smile

Great, another satisfied OVMF customer wink


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2663 2014-09-05 03:58:56

mdx
Member
Registered: 2014-09-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

After finally getting my Win7 guest up and running, I'm not able to install Windows updates without breaking the guest OS.

I've done this a few times, without any anti-virus software and the result has been the same:
1. Install Win7 (latest attempt using X17-24209.iso), install latest virtio network driver from Redhat and reboot several times: OK
2. Install critical updates using Windows Update and reboot: Guest boots in "Startup Repair" mode, sometimes several times in a loop but is unable to repair --> OS bricked

I'm using qemu-git from aur.
Has anyone experienced this issue?

Offline

#2664 2014-09-05 08:07:36

friedcpu
Member
Registered: 2014-09-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is really driving me crazy, there must be a way to passthrough your primary PCI device (slot 1) without onboard graphics (X79) and use the host as headless

My details are in post #2638 on page 106.

Has anyone successfully done this? I have recently found in dmesg this

pci 0000:01:00.0: Boot video device

even after disabling the framebuffer, could this be whats causing it?

thank you
FriedCPU

Last edited by friedcpu (2014-09-05 08:10:33)

Offline

#2665 2014-09-05 09:06:09

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using qemu from git that I pulled down on 29th Aug.

qemu:  2.1.50 (git-290814)
libvirt:   1.2.9 (git-290814)
Kernel: 3.17.0-1-mainline (RC3)
bios:    OVMF TianoCore (svn 16059)
Kernel Boot params:

intel_iommu=on pci-stub.ids=1002:67b1,1002:aac8 quiet

After rebooting the machine comes up with the windows splash screen whilst booting and then reboots so I see the Tianocore BIOS screen again. Windows then goes in to it's recovery screen.

I have applied this patch to my qemu: https://lists.gnu.org/archive/html/qemu … 00767.html as it helped out before in other areas, maybe I shouldn't have?

aw wrote:

Is it really a boot loop or is it a ~1min delay in rebooting?  (see the Aug 28th update on the blog)

Last edited by gneville (2014-09-05 09:35:40)

Offline

#2666 2014-09-05 11:40:15

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OMG now I see seabios on HDMI output, what a relief! Thank you guys so much.

Now I use i915 vga arbiter patch only and reduced boot args to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1184,10de:0e0a"

and I removed i915 from blacklist.

It's time to install Windows.

Last edited by zsph (2014-09-05 11:50:44)


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#2667 2014-09-06 03:22:52

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I implemented this a couple of weeks ago on a Mint 17 install so I just wanted to chime in and say thanks for all the info contributed to this thread.  It actually inspired me to finally test out Arch which is running quite well I might add smile)

I believe most people are using pci-stub as builtin, but for those using pci-stub as a module this information may prove helpful.  Basically when I discovered that the MODULES line for mkinitcpio.conf did not allow for module options the next step I took was to include the respective module options file.

Ex:

[root@archost ~]# grep pci /etc/mkinitcpio.conf MODULES="pci_stub" FILES="/etc/modprobe.d/pci_stub.conf" [root@archost ~]# cat /etc/modprobe.d/pci_stub.conf options pci_stub ids=10de:1183,10de:0e0a,8086:1e20

Another thing is that those of you that are passing through Nvidia might find that Nouveau can be quite the ninja even though you feel sure you've done everything to get it blacklisted.  You might be better of just building the kernel without Nouveau support or a quick hack would be to rename the module and rebuild your initial ramdisk smile

Offline

#2668 2014-09-07 00:23:01

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Here are a couple of notes about dual monitor usage on the passthrough card to the windows vm.  That includes making the monitor automatically switch outputs and also automatically starting the vm on boot.
NOTE:  in this situation I do not run Xorg nor do I make use of libvirt.  This is just how I like it smile

I boot Arch to text console and that is using the vga connection on the onboard intel HD video to what I consider my secondary monitor.  I also have a DVI connection hooked up to this same monitor that is connected to the second dvi port on the passthrough card.  The first DVI port on the passthrough card connects to my primary monitor.

Normally on a monitor with multiple inputs I could issue a 'vbetool dpms off' and that would cause the monitor to toggle to the next active input (in this case the windows vm).  For whatever reason vbetool isn't working for me is this setup.

So I added the following code to my vm start script:

# set console screen blank to occur in the next 30 seconds (a delay that allows the vm to start sending a signal to its secondary display) chmod +w /sys/module/kernel/parameters/consoleblank echo 30 > /sys/module/kernel/parameters/consoleblank # wait 60 seconds (in the background) then revert the blanking interval back to the standard 10 minutes (sleep 60 && echo 600 > /sys/module/kernel/parameters/consoleblank) & ... screen -d -m -S myvm qemu-system-x86_64 ... ...

So the end result is that 30 seconds later both monitors are displaying vm output.


Now about auto booting.  My start script actually makes use of certain components that make this work:
1) GNU screen (screen -d -m -S myvm ...) to invoke qemu and detach
this makes it service friendly while still allowing you to reattach and issue Qemu monitor commands as needed

2) qemu is passed the following extra parameters: -nographic and -echr 2
echr 2 changes the qemu hot key from CTRL - A to CTRL - B and that is because GNU screen also uses CTRL - A


# cat /etc/systemd/system/myvm.service

[Unit] Description=myvm After=network.target [Service] Type=forking ExecStart=/root/myvm.sh [Install] WantedBy=multi-user.target

EDIT:  It's worth mentioning that not all monitors will automatically search for the next input with a signal.  And qemu monitor could be set up for telnet access if you don't want to mess with GNU screen.

Last edited by tuxuser (2014-09-07 00:36:20)

Offline

#2669 2014-09-07 02:06:17

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A quick note to anyone running VGA Passthrough on Ubuntu Trusty .....

I added

-nodefaults

to the start command and saw minimum frame rates in my test case go from <20 to >40 .. a Huge performance increase for a very minor change.

I only added the parameter after I noticed that the libvirt generated command ran much faster .. and eventually narrowed the difference down to this one parameter

ymmv

Offline

#2670 2014-09-07 04:35:39

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rig:

ASRock Z97 Extreme6
Intel 4790K
hostgpu:intel
passthrough card:Nvidia GTX580



i am useing the pkgbuild for the kernel on page 1. I am using qemu and seabios from the repo. i am using the vfio-bind script on page 1 and i have nouveau blacklisted.

mkinitcpio modules:

i915 kvm kvm_intel pci-stub

syslinux.cfg relevant information

LABEL arch-qemu MENU LABEL Arch Qemu LINUX ../vmlinuz-linux-mainline APPEND root=/dev/sda1 rootflags=subvol=__active/rootvol intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=10de:1080,10de:0e09 rw INITRD ../initramfs-linux-mainline.img

something to note when i have "intel_iommu=on" active on my kernel parameters it says something to the effect of "DRHD handling fault status reg 3" and then stalls for a while while booting, but then when i disable it it boots just fine.
here's a picture: http://i.imgur.com/whwRVgF.jpg


lspci

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller 01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1) 02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) 05:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 06:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 06:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 06:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 06:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11) 08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 09:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01) 0a:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 0b:00.0 USB controller: ASMedia Technology Inc. Device 1142

lspci -n

lspci -n 00:00.0 0600: 8086:0c00 (rev 06) 00:01.0 0604: 8086:0c01 (rev 06) 00:01.1 0604: 8086:0c05 (rev 06) 00:02.0 0300: 8086:0412 (rev 06) 00:03.0 0403: 8086:0c0c (rev 06) 00:14.0 0c03: 8086:8cb1 00:16.0 0780: 8086:8cba 00:19.0 0200: 8086:15a1 00:1a.0 0c03: 8086:8cad 00:1b.0 0403: 8086:8ca0 00:1c.0 0604: 8086:8c90 (rev d0) 00:1c.2 0604: 8086:8c94 (rev d0) 00:1c.3 0604: 8086:8c96 (rev d0) 00:1c.6 0604: 8086:8c9c (rev d0) 00:1d.0 0c03: 8086:8ca6 00:1f.0 0601: 8086:8cc4 00:1f.2 0106: 8086:8c82 00:1f.3 0c05: 8086:8ca2 01:00.0 0300: 10de:1080 (rev a1) 01:00.1 0403: 10de:0e09 (rev a1) 02:00.0 0c03: 1912:0014 (rev 03) 04:00.0 0200: 10ec:8168 (rev 11) 05:00.0 0604: 1b21:1184 06:01.0 0604: 1b21:1184 06:03.0 0604: 1b21:1184 06:05.0 0604: 1b21:1184 06:07.0 0604: 1b21:1184 07:00.0 0106: 1b4b:9130 (rev 11) 08:00.0 0106: 1b21:0612 (rev 02) 09:00.0 0280: 168c:0030 (rev 01) 0a:00.0 0106: 1b21:0612 (rev 02) 0b:00.0 0c03: 1b21:1142

qemu.sh(the op's script with my card put in and the cpu changed

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

and i'm getting output

No protocol specified Could not initialize SDL(No available video device) - exiting


then for some reason after a while i ran that qemu.sh and i got the output

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/16: Device or resource busy qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 16 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

any ideas on whats going on?

Offline

#2671 2014-09-07 04:43:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

rig:

ASRock Z97 Extreme6
Intel 4790K
hostgpu:intel
passthrough card:Nvidia GTX580



i am useing the pkgbuild for the kernel on page 1. I am using qemu and seabios from the repo. i am using the vfio-bind script on page 1 and i have nouveau blacklisted.

mkinitcpio modules:

i915 kvm kvm_intel pci-stub

syslinux.cfg relevant information

LABEL arch-qemu MENU LABEL Arch Qemu LINUX ../vmlinuz-linux-mainline APPEND root=/dev/sda1 rootflags=subvol=__active/rootvol intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=10de:1080,10de:0e09 rw INITRD ../initramfs-linux-mainline.img

something to note when i have "intel_iommu=on" active on my kernel parameters it says something to the effect of "DRHD handling fault status reg 3" and then stalls for a while while booting, but then when i disable it it boots just fine.
here's a picture: http://i.imgur.com/whwRVgF.jpg

You need to run 3.17-rc or disable the Marvell controller.  That device is known to use an invalid PCIe requester ID for DMA.  3.17 includes DMA alias support to enable support for these devices.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2672 2014-09-07 05:11:08

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

##what patches do i need for 3.17?##

EDIT:

I'm going to try to see if i can get it working at all first before i start messing with compiling different kernels, though i really do appreciate the info. I've decided to just manually remove the device and it's working fine now.

I'm still getting

No protocol specified Could not initialize SDL(No available video device) - exiting

edit 2:

i'm a dope. i was running it as root

not running it as root is giving me the error

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Last edited by risho (2014-09-07 06:47:48)

Offline

#2673 2014-09-07 15:26:22

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@risho Maybe take a look at your iommu groups using the lsgroup script that aw provided on page 30 or so?  Also you can use lspci -nn to combine the output instead of a seperate lspci / lspci -n run.

Offline

#2674 2014-09-07 18:12:04

Mardok45
Member
Registered: 2014-08-10
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shazeal wrote:

The R9 270X is just a 7870 card rebranded and overclocked, regardless any video card can be passed through its just a PCI redirect afaik. Its upto the guest to support the video card.

More important than the video card is how are you running CPU/RAM, are you using cgroups and hugetlbfs?

Thanks for the information, if I can't get my nVidia card to work, I'll probably get this.  More information below.

aw wrote:

Add kvm=off to the -cpu parameter list to make current Nvidia driver not get Code 43.  Also add hv_time to enable hypver-v enlightened time keeping.  Older kernels can have poor performance if the game makes use of debug registers, run something new.  Hugepages, pinning, and vCPU model tuning also help.  All of this is documented in the very, very long thread.

This worked, but I'm no longer using QEMU.  More information below.

slis wrote:

@mardok45 check your cpu config

    <topology sockets='1' cores='4' threads='1'/>

if u set 2 cores 2 threads dota works 2x slower, also check your cpu model, hvtime option... etc... host cpufreq settings....

I think I was not setting the right CPU topology and the performance was better after tweaking it, but I'm not on QEMU anymore.  More information below.


I played around with this for about a month and found out a lot of information.

First things first, the hardware.  If you plan on using this for games, not only should you make sure the CPU and motherboard support IOMMU/VT-x&VT-d, you should pick out a motherboard that either:

* You can alter the primary display in your motherboards BIOS or it can pick its primary display depending on which GPU slot is plugged in to a montior.
or
* Has at least two PCI-E slots that work at x16.
AND
Make sure you have at least one AMD GPU or a nVidia Quadro.  In my case, I have on old ATI 4850 and a Radeon HD 7470.

I forgot what kind of specs my motherboard had, and found out later that the second PCI-E slot operates at x8 speeds.  Not good for gaming.  That explains why Dota 2 was having rendering problems. 

So I thought, "Okay, just plug my nVidia card into the first slot so it'll operate at x16 speeds."  Well, I don't know for sure, but I think that because the nVidia GPU was bootstrapped when I turned the physical machine on, it couldn't re-bootstrap itself when I passed it through to the DomU.  This caused it to just display nothing.

But here's where it gets really weird; my AMD cards don't suffer from this problem.  Both of my Radeon cards have no problem being in the primary display, and then being passed through to the DomU.  They display just fine.  Does anyone else have the same experience?

My motherboard can either set the primary display to a PCI or PCI-E slot, so I'm going to try to get my hands on a PCI GPU and set the primary display to the PCI slot.  I'm hoping this will cause the nVidia GPU to not get bootstrapped and it can then be passed through to the DomU with no problems.  It had no problems being passed through when it was in the x8 slot, so I'm fairly confident it'll work.

Now moving on to the software.  I'm using Gentoo for the host, but it's just a personal preferance of mine.  You can accomplish this with any distribution.  At first, I was using QEMU.  I had no problems with a Linux guest, but Windows was a different story.

Right off the bat, the Windows installation took an unusually long time.  The Windows 7 installer would get hung on 0% for over an hour before files started getting copied over.  It's not due to performance on anything and I can't explain why it does this.

Then I had to go through the Windows updates.  This was scary because Windows would just become unbootable.  I cannot explain this to you.  For whatever reason, at some point during updates, Windows would just become completely unbootable.  It goes straight into the "Fix Windows boot problems" prompt and it cannot fix itself.  If anyone can shed some light on this, please do because it has caused so many headaches.

When I switched to Xen 4.4, I've had zero problems with Windows.  Zero, nada, zilch.  It works... almost perfectly.

Now you have the problem with the GPU, and it's not Window's fault.  You have to eject the card before you reboot it, and I cannot figure out how to automate this.  If anyone does, can you explain how?  I've tried the automated eject as destribed here, and it didn't work for me: http://blog.ktz.me/?p=219.  So if anyone has a suggestion, I would love to hear it.

Now there's one detail left: sound.  At first, I tried straight-up ALSA/dmix and gave it a sound card with a config like this:

builder='hvm'
device_model_override = "/usr/bin/qemu-system-x86_64"
memory = 3072
vcpus=6
name = "windows"
vif = ['']
disk = ['phy:/dev/system/windows,hda,w']
acpi = 1
stdvga=1
usbdevice='tablet'
boot="dc"
sdl=0
serial='pty'
vnc=1
vnclisten="0.0.0.0"
xen_platform_pci=1
viridian=1
apic=1
gfx_passthru=0
pci=['01:00.0', '01:00.1', '00:12.0', '00:12.2']
on_poweroff='destroy'
on_reboot='destroy'
on_crash='destroy'
soundhw='hda'

This worked fine for Linux, but Windows's sound was crackly.  I cannot explain why.  So I set up PulseAudio by running system-wide and adding the root user to the pulse and pulse-access groups (which you really shouldn't do, but I had no other choice), and now sound works great.

So there you have it.  The only thing that's left for me to do is to test the nVidia setup as I described above and get a KVM switch, then I'll have the perfect setup.  If anyone has any suggestions or comments, please give me a shout.

Last edited by Mardok45 (2014-09-07 18:21:40)

Offline

#2675 2014-09-07 18:35:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mardok45 wrote:

Now moving on to the software.  I'm using Gentoo for the host, but it's just a personal preferance of mine.  You can accomplish this with any distribution.  At first, I was using QEMU.  I had no problems with a Linux guest, but Windows was a different story.

Right off the bat, the Windows installation took an unusually long time.  The Windows 7 installer would get hung on 0% for over an hour before files started getting copied over.  It's not due to performance on anything and I can't explain why it does this.

Then I had to go through the Windows updates.  This was scary because Windows would just become unbootable.  I cannot explain this to you.  For whatever reason, at some point during updates, Windows would just become completely unbootable.  It goes straight into the "Fix Windows boot problems" prompt and it cannot fix itself.  If anyone can shed some light on this, please do because it has caused so many headaches.

I'd guess this likely has a lot to do with the QEMU Q35 machine and AHCI.  It's terrible.  Regardless of how many times I mention to just use 440FX and give examples of doing it, people continue to use Q35 when they don't need to.  Xen uses a 440FX model.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2676 2014-09-08 00:57:52

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

okay i've been messing with it more and i'm still getting the same problems

test.sh

#qemu-system-x86_64 -enable-kvm -m 1024 -cpu host \ #-smp 6,sockets=1,cores=2,threads=1 \ #-bios /usr/share/qemu/bios.bin -vga none \ #-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ ##-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ #-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none\ -rtc base=localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \

sh test.sh

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

sudo sh test.sh

No protocol specified Could not initialize SDL(No available video device) - exiting

sh lsgroup

.... ### Group 15 ### 01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1) .... ### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) ### Group 10 ### 00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0) ### Group 11 ### 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) ### Group 12 ### 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) ### Group 13 ### 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 ### Group 14 ### 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller ### Group 16 ### 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) ### Group 17 ### 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 ### Group 18 ### 05:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 ### Group 19 ### 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 ### Group 20 ### 05:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 ### Group 21 ### 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 ### Group 22 ### 06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) ### Group 23 ### 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) ### Group 24 ### 08:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01) ### Group 25 ### 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) ### Group 26 ### 0a:00.0 USB controller: ASMedia Technology Inc. Device 1142

lspci -nn

.... 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF110 [GeForce GTX 580] [10de:1080] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GF110 High Definition Audio Controller [10de:0e09] (rev a1) .... 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba] 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1] 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0] 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0) 00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 [8086:8c94] (rev d0) 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0) 00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0) 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6] 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97 LPC Controller [8086:8cc4] 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82] 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2] 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11) 04:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 05:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 05:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 05:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 05:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 06:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) 07:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02) 08:00.0 Network controller [0280]: Qualcomm Atheros AR93xx Wireless Network Adapter [168c:0030] (rev 01) 09:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02) 0a:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1142]

sh vfio-group 15

0000:01:00.0 0000:01:00.1 0x10de 0x1080

boot parameters

LABEL arch-qemu MENU LABEL Arch Qemu LINUX ../vmlinuz-linux-mainline APPEND root=/dev/sdb1 rootflags=subvol=__active/rootvol intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=10de:1080,10de:0e09,8086:0c01 modeset.nouveau=0 modeset.radeon=0 rw INITRD ../initramfs-linux-mainline.img

i use vfiobind

sudo vfio-bind 0000:01:00.0 0000:01:00.1

dmesg | grep -e DMAR -e IOMMU

[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] ACPI: DMAR 0x00000000BD1C3BC8 0000B8 (v01 INTEL BDW 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.026976] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.026979] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.027044] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.523584] DMAR: No ATSR found [ 0.523601] IOMMU 0 0xfed90000: using Queued invalidation [ 0.523601] IOMMU 1 0xfed91000: using Queued invalidation [ 0.523603] IOMMU: Setting RMRR: [ 0.523610] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff] [ 0.524672] IOMMU: Setting identity map for device 0000:00:14.0 [0xbde9a000 - 0xbdea8fff] [ 0.524691] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbde9a000 - 0xbdea8fff] [ 0.524707] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbde9a000 - 0xbdea8fff] [ 0.524718] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.524724] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 2.613731] [drm] DMAR active, disabling use of stolen memory

i am absolutely lost and do not know what i can do next

by the way tuxuser thank you very much for pointing out those scripts to me they are really helpful.

does anyone have any ideas?

Last edited by risho (2014-09-08 01:02:36)

Offline

#2677 2014-09-08 01:33:50

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@risho I haven't dealt with that vfio permission problem myself.   Does dmesg and lspci -v indicate that pci-stub claimed the device?  I assume your pci-stub support is built into the kernel.

EDIT:  as far as the SDL problem, i assume you are launching from Xorg, but just pass -nographic for now

for my case support is as a module and I wrote about this earlier [root@archost ~]# zgrep PCI_STUB /proc/config.gz CONFIG_PCI_STUB=m [root@archost ~]# dmesg|grep stub [ 0.456853] pci-stub: add 10DE:1183 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.456860] pci-stub 0000:01:00.0: claimed by stub [ 0.456863] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.456867] pci-stub 0000:01:00.1: claimed by stub [ 0.456868] pci-stub: add 8086:1E20 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.456871] pci-stub 0000:00:1b.0: claimed by stub

Last edited by tuxuser (2014-09-08 01:48:46)

Offline

#2678 2014-09-08 01:51:46

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lspci-v

... 01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 1580 Flags: fast devsel, IRQ 16 Memory at ee000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at ef000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau 01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1) Subsystem: eVga.com. Corp. Device 1580 Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at ef080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ... 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) Subsystem: ASRock Incorporation Device 0c00 Flags: bus master, fast devsel, latency 0 Capabilities: <access denied> Kernel driver in use: hsw_uncore 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: ee000000-ef0fffff Prefetchable memory behind bridge: 00000000e0000000-00000000e9ffffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) (prog-if 00 [VGA controller]) Subsystem: ASRock Incorporation Device 0412 Flags: bus master, fast devsel, latency 0, IRQ 72 Memory at ef400000 (64-bit, non-prefetchable) [size=4M] Memory at d0000000 (64-bit, prefetchable) [size=256M] I/O ports at f000 [size=64] Expansion ROM at <unassigned> [disabled] Capabilities: <access denied> Kernel driver in use: i915 Kernel modules: i915 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) Subsystem: ASRock Incorporation Device 0c0c Flags: bus master, fast devsel, latency 0, IRQ 73 Memory at efe34000 (64-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller (prog-if 30 [XHCI]) Subsystem: ASRock Incorporation Device 8cb1 Flags: bus master, medium devsel, latency 0, IRQ 48 Memory at efe20000 (64-bit, non-prefetchable) [size=64K] Capabilities: <access denied> Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 Subsystem: ASRock Incorporation Device 8cba Flags: bus master, fast devsel, latency 0, IRQ 68 Memory at efe3e000 (64-bit, non-prefetchable) [size=16] Capabilities: <access denied> Kernel driver in use: mei_me Kernel modules: mei_me 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V Subsystem: ASRock Incorporation Device 15a1 Flags: bus master, fast devsel, latency 0, IRQ 69 Memory at efe00000 (32-bit, non-prefetchable) [size=128K] Memory at efe3c000 (32-bit, non-prefetchable) [size=4K] I/O ports at f080 [size=32] Capabilities: <access denied> Kernel driver in use: e1000e Kernel modules: e1000e 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 (prog-if 20 [EHCI]) Subsystem: ASRock Incorporation Device 8cad Flags: bus master, medium devsel, latency 0, IRQ 16 Memory at efe3b000 (32-bit, non-prefetchable) [size=1K] Capabilities: <access denied> Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller Subsystem: ASRock Incorporation Device 1151 Flags: bus master, fast devsel, latency 0, IRQ 71 Memory at efe30000 (64-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: efd00000-efdfffff Prefetchable memory behind bridge: 00000000ea100000-00000000ea1fffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=04, subordinate=09, sec-latency=0 I/O behind bridge: 0000b000-0000cfff Memory behind bridge: ef800000-efbfffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0 Memory behind bridge: efc00000-efcfffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 (prog-if 20 [EHCI]) Subsystem: ASRock Incorporation Device 8ca6 Flags: bus master, medium devsel, latency 0, IRQ 23 Memory at efe3a000 (32-bit, non-prefetchable) [size=1K] Capabilities: <access denied> Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller Subsystem: ASRock Incorporation Device 8cc4 Flags: bus master, medium devsel, latency 0 Capabilities: <access denied> 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Device 8c82 Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 49 I/O ports at f0d0 [size=8] I/O ports at f0c0 [size=4] I/O ports at f0b0 [size=8] I/O ports at f0a0 [size=4] I/O ports at f060 [size=32] Memory at efe39000 (32-bit, non-prefetchable) [size=2K] Capabilities: <access denied> Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller Subsystem: ASRock Incorporation Device 8ca2 Flags: medium devsel, IRQ 10 Memory at efe38000 (64-bit, non-prefetchable) [size=256] I/O ports at f040 [size=32] 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) Subsystem: ASRock Incorporation Motherboard (one of many) Flags: bus master, fast devsel, latency 0, IRQ 70 I/O ports at d000 [size=256] Memory at efd00000 (64-bit, non-prefetchable) [size=4K] Memory at ea100000 (64-bit, prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: r8169 Kernel modules: r8169 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=04, secondary=05, subordinate=09, sec-latency=0 I/O behind bridge: 0000b000-0000cfff Memory behind bridge: ef800000-efbfffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 05:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=05, secondary=06, subordinate=06, sec-latency=0 Memory behind bridge: efb00000-efbfffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=05, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: efa00000-efafffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 05:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=05, secondary=08, subordinate=08, sec-latency=0 Memory behind bridge: ef900000-ef9fffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=05, secondary=09, subordinate=09, sec-latency=0 I/O behind bridge: 0000b000-0000bfff Memory behind bridge: ef800000-ef8fffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp 06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI]) Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at efb00000 (64-bit, non-prefetchable) [size=8K] Capabilities: <access denied> Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard Flags: bus master, fast devsel, latency 0, IRQ 58 I/O ports at c050 [size=8] I/O ports at c040 [size=4] I/O ports at c030 [size=8] I/O ports at c020 [size=4] I/O ports at c000 [size=32] Memory at efa00000 (32-bit, non-prefetchable) [size=512] Capabilities: <access denied> Kernel driver in use: ahci Kernel modules: ahci 08:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01) Subsystem: Qualcomm Atheros Device 3112 Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at ef900000 (64-bit, non-prefetchable) [size=128K] Expansion ROM at ef920000 [disabled] [size=64K] Capabilities: <access denied> Kernel driver in use: ath9k Kernel modules: ath9k 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard Flags: bus master, fast devsel, latency 0, IRQ 59 I/O ports at b050 [size=8] I/O ports at b040 [size=4] I/O ports at b030 [size=8] I/O ports at b020 [size=4] I/O ports at b000 [size=32] Memory at ef800000 (32-bit, non-prefetchable) [size=512] Capabilities: <access denied> Kernel driver in use: ahci Kernel modules: ahci 0a:00.0 USB controller: ASMedia Technology Inc. Device 1142 (prog-if 30 [XHCI]) Subsystem: ASRock Incorporation Device 1142 Flags: bus master, fast devsel, latency 0, IRQ 18 Memory at efc00000 (64-bit, non-prefetchable) [size=32K] Capabilities: <access denied> Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd

dmesg | grep vfio-pci

[ 304.062334] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) 

interesting the card says

01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 1580
        Flags: fast devsel, IRQ 16
        Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at ef000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
i dunno if that means it isnt being blacklisted?


EDIT:

EDIT:  as far as the SDL problem, i assume you are launching from Xorg, but just pass -nographic for now

yes i am running from inside xorg


i changed test.sh to:

#qemu-system-x86_64 -enable-kvm -m 1024 -cpu host \ #-smp 6,sockets=1,cores=2,threads=1 \ #-bios /usr/share/qemu/bios.bin -vga none \ #-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ ##-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ #-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none\ -rtc base=localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -nographic \###altered line to add -nographic -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \

it grabs the terminal and gives no output which makes me assume its working. in order to get the terminal back i have to kill qemu-x86_64(or whatever it was extactly). then when i ran it again

i got

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= 

Last edited by risho (2014-09-08 01:59:20)

Offline

#2679 2014-09-08 01:57:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho, if you don't run as root then you need to change the permissions of the files for the user to access them and set the locked memory limit for the user to match that necessary for the guest.  Just run as root and use the -nographic option so that QEMU won't try to create an SDL window.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2680 2014-09-08 02:05:17

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i got it working. you guys are amazing. thank you so much for the help.

now i have a couple more questions. i killed qemu and the monitor is still being shown as if the vm were still running. (it's not even holding the terminal window anymore) how do i correctly shutdown the vm from inside the host?

Last edited by risho (2014-09-08 02:06:28)

Offline

#2681 2014-09-08 02:08:12

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

instead of kill the term, if you hit CTRL A C you can get a qemu monitor prompt and just type quit,   in the log run this is why you want to either setup qemu monitor for telnet access or use gnu screen or some people go libvirt.

EDIT:  during testing i will do the following:
stop
system_reset
quit


This cleans up the passthough screen nicely.   And that's nice to hear aw's update about 3.17 smile

Last edited by tuxuser (2014-09-08 02:13:17)

Offline

#2682 2014-09-08 02:10:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

i got it working. you guys are amazing. thank you so much for the help.

now i have a couple more questions. i killed qemu and the monitor is still being shown as if the vm were still running. (it's not even holding the terminal window anymore) how do i correctly shutdown the vm from inside the host?

Since you're trying to assign nvidia, this should be fixed in 3.17.  A bus reset will be done when the devices are released if any of the devices require it and all of the affected devices are bound to vfio-pci.  (It will also work for radeon cards that report NoSoftRst+)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2683 2014-09-09 13:56:43

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an old idea that i think it is worth a thought. The normal bug/feedback reporting path is tedious and i think most ppls just wait and let somebody else do it. Speaking plainly, how about a site like kerneltest.org, where willing users can/could upload the output of some script or could enable a kernel cli of a module auto reporting tool (or something that is automatic and standardised enough). Sort of something non invasive and only user volontary. Like a sort of automatic reporting tool, not for developers, but for what is actually happening in the wild. I dont mind giving that kind of info as long as its entirely optional/volontary, code is transparent and whats being sent is visible and approvable. I think it would be a lot of good information offered by this process, not to mention nice statistics like how many bugs affect how many machines and of which type. Can be applied different ways, overall or in part. Like if some users want to test some "vfio" branch and have a script report back the issues of that branch exactly. Or if on their turn, are dubious about some other pull request, can ask the guy on that branch to first show say 100 unique reports that patch boots ok on 100 different machines. But something like this needs some form of "official" backing otherwise wont get traction. And has to rely on automatic reporting coz of user lazyness, and also to have a standardisation for data processing. I am generally stumped as how redhat has an automatic bug reporting tool (of sorts) and something much more global like kernel.org does not have a similar straightforward feedback shortcut. Well, not my place to comment, but really its just a thought.
Or maybe if you want to provide a script of some sorts, me and maybe a couple of other ppls on the thread would be happy to volonteer running it for you to get whatever info you may wish to have. And that would be in a fairly standardized manner. Meaning the data can be processed further, statistical and so on. In comparison i think most of the comments on the thread are just like any other thread, basic impressions and contextual ones, not quite able to be processed furtner because of not having details like configs, context, versions, etc. Hence quite limited usability and prone to the normal distribution being applied to ppls abilities - to put things in a nice way.

Offline

#2684 2014-09-09 14:16:17

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

system_reset

system_reset  -- reset the system || as if you hit the reset button while machine is running;
system_powerdown  -- send system power down event || as if it would be turned off from inside start menu; if you have sound, you will hear the windows sutdown sound, so this is normal shutdown. Also with this you should not need stop and quit, if the machine closes normally the qemu terminal will close too by itself.
in qemu terminal, type 'help' for a list of commands, or 'info' for getting detailed machine information

Last edited by noobman (2014-09-09 14:21:30)

Offline

#2685 2014-09-09 15:32:54

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

...
I am generally stumped as how redhat has an automatic bug reporting tool (of sorts) and something much more global like kernel.org does not have a similar straightforward feedback shortcut.
...

well, "redhat" is a distro, while "kernel.org" is distro-agnostic software piece (a core one for all the distros out there). "automatic bug reporting tool" assumes some degree of usage of distro's infrastructure. still feeling stumped?

p.s. your idea was too vague for me to grasp from the text you wrote (purpose? subject?), but judging by what i think i understood from your posts, this is too little gains for huge trouble to make it work (if possible at all, considering all the distro/setup variety out there)

Offline

#2686 2014-09-09 16:07:37

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

system_reset

system_reset  -- reset the system || as if you hit the reset button while machine is running;
system_powerdown  -- send system power down event || as if it would be turned off from inside start menu; if you have sound, you will hear the windows sutdown sound, so this is normal shutdown. Also with this you should not need stop and quit, if the machine closes normally the qemu terminal will close too by itself.
in qemu terminal, type 'help' for a list of commands, or 'info' for getting detailed machine information


noobman the context that system_reset was suggested was "testing" which you may not have a booted OS (ie no drives attached and just testing pt output).  system_powerdown does not nothing for me at the Seabios screen or inside a installer with no shutdown option.  Of course under normal circumstances you want want to do a graceful shutdown.  I thought this was self explanitory...

Offline

#2687 2014-09-09 17:57:33

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
noobman wrote:

...
I am generally stumped as how redhat has an automatic bug reporting tool (of sorts) and something much more global like kernel.org does not have a similar straightforward feedback shortcut.
...

well, "redhat" is a distro, while "kernel.org" is distro-agnostic software piece (a core one for all the distros out there). "automatic bug reporting tool" assumes some degree of usage of distro's infrastructure. still feeling stumped?

p.s. your idea was too vague for me to grasp from the text you wrote (purpose? subject?), but judging by what i think i understood from your posts, this is too little gains for huge trouble to make it work (if possible at all, considering all the distro/setup variety out there)

Yes well, as i said, its not for me to comment, just an ideea. Regarding its implementation, i dont think its impossible, nothing really is. This very thread goes towards doing something that once was not possible. And most likely just like anything else, could be implemented in a variety of ways, not just one. Then the usual, many choices to be made in that regard, etc. About distro related, could be something like only distro having src or pkg for each distro, or being half module half distro script, or maybe just rely on a kernel module alone, dunno. However i certainly think its not "impossible". These days is hard to find a thing that really is. Harder or easier yes, but impossible - dunno. So the distro part i can see that, but i think its far from being an insurmontable obstacle.

What i said its just an ideea, having shortest and most standardized feedback possible for what is being done. This is the abstract idea alone. Its not even IT specific, but like information systems, more used in other places/industries which rely on feedback and where quality is a direct function of feedback and how its being implemented. And if in general the quality depends on the feedback, so is the case with kernel: where would kernel be with no bug reports ever. And hence one step further, the better feedback (more accurate, more standardised) should go towards applying same attributes to the release product. Better feedback, better product. Otherwise feedback does not just improve the main process, but nowadays its integral part of many things as a process. As it being universal, for example you can not have a stepper motor without feedback - it just can function without it, feedback is integrated in functioning principle because it needs a closed loop to start with, the entire thing relies on that. Many may consider it differently but the feedback like it is now, bug reports that take 2 weeks + 20 comments before somebody posts an actual gdb trace is something i would (as a matter of personal opinion), i would qualify that lacking, no standardisation, bad or poor quality feedback with as little context info as possible. Then i dunno how something so big and complex as the kernel cand work with that feedback of little drips of info (where it should be a river of information). So yes im still stumped, from the perspective i am looking. Thumbs up for the army of ppls, including volonteers, digging in all that and sorting the issues one by one, from where i am looking that is really nothing short of a miracle.

So again, ofc it is just an abstract sort of more or less universal idea, widely applicable. The "how to do it" its hierarhically different question. No point going further into it, because its not my place. I just do think that could make a good difference in many situations. Like imagine you look at all the gathered data from 10k machines for xx-test-rc3, and you see 80% having an issue, one could figure out its a regression of some kind from previous, easier and faster to pinpoint. And you dont have to wait 3 months before somebody puts a bug notification and then bisects everything back to rc3 from rc2 to find the culprit. Or we can imagine other scenarios where one branch change is not being credible but it can become so when its backed by xyz reports of running ok. Or the other way around, discard a patch coz of bad reports. We can play with the scenarios like these as long as we want, maybe some can imagine a scenario where it would hurt, but i think in general it would be a good thing, ofc depends on how its implemented and used. Again, as an abstraction any tool cuts both ways, but when it cuts the user, its user fault. No point playing the "what-if" game.

The context, or the place of this is, i would like to help, i could give feedback, but i am a poor user, very limited in knowledge, and ofc very lazy. In other words typical. So realistically things must not rely on my iq or work ability, else will certainly fail. smile Hence it needs to be automated and standardised. I dont think there is a single company with marketing division providing a feedback that relies on its clients inteligence or their work addiction - or if there is one, i would not buy that stock. So about bad SNR signal to noise ratio - true that, but it can be changed and improved. Not at the user base level (again coz normal repartition is a bitch when applied to human qualities), but it can be changed at the process level.

Sry cant elaborate further, as i said its just an idea, and pretty abstract one for that matter, tho quite widely applied. Ppls may like or dislike the ideea, again, its just an opinion, mine, im entitled to one, so here it is/was thrown in. All my concerns rest, further idc, so ill rest. Ofc ppls may think differently, and by all means they better should. Thats the good part, and its how things get filtered.

Offline

#2688 2014-09-09 19:42:44

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuxuser wrote:

noobman the context that system_reset was suggested was "testing" which you may not have a booted OS (ie no drives attached and just testing pt output).  system_powerdown does not nothing for me at the Seabios screen or inside a installer with no shutdown option.  Of course under normal circumstances you want want to do a graceful shutdown.  I thought this was self explanitory...

Yes sry then my bad, i assumed it was an os running inside the machine. Otherwise i (personally) dont care how i shut down the vm. If i have an os running, only then i care because i have open files, could loose data, etc, and then if i just close the process then windows will have dirty bit set and next time will scandisk at boot time. Else with no software in machine i would not care how i close it (but that might be just my personal choice). Anyway i didnt thought of that case really, i just assumed the machine has an os.

Offline

#2689 2014-09-09 23:06:18

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It would be nice to see a X99 motherboard do this as a multi-headed PC.

And I want to show off my work, but I want to show a Linux Distro (host), VM1(Windows) and VM2(MAC OS), the problem is, where do I get MAC OS !? it seems they only come with apple MACs.

Offline

#2690 2014-09-10 00:58:54

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with suspend/resume my host. I searched the whole thread and this topic was brought up a few times but I couldn't get any hints. After using qemu and successfully shutting down the Windows guest I can no longer send the host to sleep. My desktop remains visible and frozen.

Afterwards I tried to run qemu without disks to only show seabieos, hoping it would release or reset the vfio devices. That way I can suspend but when resuming back there is no signal from iGPU, even REISUB doesn't help. Only hard reset.

I also tried to unbind the devices from vfio-pci and bind to drivers_probe but it didn't work out.


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#2691 2014-09-10 01:33:11

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

[...] the problem is, where do I get MAC OS !? it seems they only come with apple MACs.

Apple forbids the installation of OSX on non-Apple-hardware and there even don't exist installation discs any more. <Moderator edit.  Redacted>

It might be allowed to run a virtualized OSX with Linux KVM on Apple hardware. But if one would try something like this, it might be best to use a graphics card that is natively supported in OS X -> hardware database
KVM infos:
http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/ (page 28 in this thread)
http://www.reddit.com/r/hackintosh/comm … _with_vga/

Last edited by ewaller (2014-09-10 02:26:38)

Offline

#2692 2014-09-10 02:24:41

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As a moderator, I am obliged to step in here and insist there be no further discussion that encourages or provides information as to violate copyright restrictions. 
Our policy with regards to the facilitation of copyright violation

Thank you.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#2693 2014-09-10 05:48:43

friedcpu
Member
Registered: 2014-09-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried running any EA Origin Games in their Windows VM?

I am running Windows 7 x64 Enterprise and trying to run The Sims 4 (for the mrs, got to keep her happy) and it just refuses to open, I can see in task manager it opens the process, and then it just dies, can't find anything in any log files in Windows telling me whats going on and nothing happens on screen. If I boot the hard drive natively it works fine (installed windows through qemu and all drivers, I just swap the gfx card back to slot 1 and boot natively to test). My qemu settings are

qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sde,id=boot,format=raw -device scsi-hd,drive=boot \ -drive file=/dev/sdf,id=storage,format=raw -device scsi-hd,drive=storage \ -usb -usbdevice host:046d:c07d -usbdevice host:of39:1084 \ -nographic -boot menu=on

System specs are:
ASUS x79-deluxe Motherboard
Intel 4930k CPU
64GB RAM
3x250GB Samsung EVO SSD (1 host, 1 for each VM)
2x3TB WD RED (1 for each VM)
1x2TB Seagate (for host, other VMs with no GPU passthrough)
AMD HD5450 (Host)
NVIDIA GTX550Ti (Yet Unknown VM)
NVIDIA GTX770 (Windows VM)

I have also tried with passing through the sata controller (which works on first boot, but if I reboot the VM it causes seabios to hang, can't figure it out its an ASM1602 Which is known to be a problem in passthrough) but still no difference. There isn't really any performance gain from using native sata than the scsi controller.

Any help would be greatly appreciated, otherwise I'm going to have to keep a separate Windows install just for her to boot natively thus shutting down all my VM's sad

Last edited by friedcpu (2014-09-10 05:49:20)

Offline

#2694 2014-09-10 15:26:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

friedcpu wrote:

Has anyone tried running any EA Origin Games in their Windows VM?

Isn't BF4 an EA Origin game?  We seem to have reports of that working.  I try to avoid EA on principle.

Have you tried the ignore_msrs=1 option for the kvm module?

Did you re-install the game as well as the OS?  Perhaps some of the DRM that EA is so well known for doesn't like the change in computer platform.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2695 2014-09-10 19:09:30

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I will post this little Python script that I made which grabs device ids from linux commandline and returns the corresponding PCI bus ids, useful to pass returning list to vfio-bind script:

#!/usr/bin/env python """This script grabs pci bus from vendor:device using pci stubs from linux cmdline""" from subprocess import Popen, PIPE PCISTUB_CMD = "pci-stub.ids=" def shell(cmd, args = ""): """Runs shell command""" proc = Popen([cmd, args], stdout = PIPE, stderr = PIPE) result, error = proc.communicate() if error: raise Exception(error) return result def get_linux_pcistub(): """Returns dev ids list""" linux_cmds = shell("cat", "/proc/cmdline") for cmd in linux_cmds.split(" "): if PCISTUB_CMD in cmd: return cmd.split("=")[1] pcidevs = "" devids = get_linux_pcistub() if devids is not None: for devid in devids.split(","): if devid: #devid is valid pcdidevid = shell("lspci", "-d " + devid) if pcdidevid: #we found existing pci bus pcidevs += " 0000:" + pcdidevid[:7] print(pcidevs)

Last edited by Cubex (2014-09-10 19:22:12)

Offline

#2696 2014-09-11 10:19:10

guildem
Member
Registered: 2014-09-11
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I all, I have some trouble when I try to do a vga passthrough with my system.

First thing, sorry for my bad english, i'm french, and you know what they say about french people and foreign languages smile

So this is my config :

CM: Asrock Z97 Extreme 6
Proc: i7 4790k
Ram: 16GB
Host GPU: HD Graphics 4600
Guest GPU: Radeon HD 5850

Some informations :

- I use only repo packages, with last updates, no AUR, no patches
- kernel 3.16.2-1, qemu 2.1.0-2

My lspci dump :

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cypress PRO [Radeon HD 5850] [1002:6899] (prog-if 00 [VGA controller])
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cypress HDMI Audio [Radeon HD 5800 Series] [1002:aa50]

On my syslinux boot i added :

intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:6899,1002:aa50

I've added this script :

vfio-bind 0000:02:00.0 0000:02:00.1

And this is my qemu command :

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
    -smp 4,sockets=1,cores=4,threads=1 \
    -usbdevice tablet -usbdevice host:046d:c52e \
    -bios /usr/share/qemu/bios.bin -vga cirrus \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
    -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
    -device piix4-ide,bus=pcie.0,id=piix4-ide \
    -drive file=seven.img,id=disk,format=qcow2 -device ide-hd,bus=piix4-ide.0,drive=disk \
    -drive file=software/iso/windows-seven-x64.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    -soundhw hda

And now the weird thing : you'll notice the "-vga cirrus" option. when I put this option on cirrus, I boot on a cirrus card, and view my Seven desktop (all installed). If I try to view my graphic cards on the guest, I see 2 cards, the "cirrus" one acting as a VGA standard graphic card, and a second one "AMD Radeon HD 5800 Series" (i installed AMD's Catalyst driver), with an error : "this device can't find enough free resources to work correctly" (translated from French).

If I put a "-vga none" option on my qemu, I see only the compat_monitor console and nothing shown on my AMD VGA screen. And I have the color glitch on my iGPU screen (I go on tty mode and go back on X to remove it).

I don't know what else I can do to have something on my AMD screen...

Thanks for your help !

Guildem

Offline

#2697 2014-09-11 10:23:45

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

guildem wrote:

I all, I have some trouble when I try to do a vga passthrough with my system.

First thing, sorry for my bad english, i'm french, and you know what they say about french people and foreign languages smile

...

If I put a "-vga none" option on my qemu, I see only the compat_monitor console and nothing shown on my AMD VGA screen. And I have the color glitch on my iGPU screen (I go on tty mode and go back on X to remove it).

I don't know what else I can do to have something on my AMD screen...

Thanks for your help !

Guildem

Have you even read first post?

Anyway, you need to patch your kernel with i915 vgaarb patch.

Offline

#2698 2014-09-11 12:07:43

guildem
Member
Registered: 2014-09-11
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
guildem wrote:

I all, I have some trouble when I try to do a vga passthrough with my system.

First thing, sorry for my bad english, i'm french, and you know what they say about french people and foreign languages smile

...

If I put a "-vga none" option on my qemu, I see only the compat_monitor console and nothing shown on my AMD VGA screen. And I have the color glitch on my iGPU screen (I go on tty mode and go back on X to remove it).

I don't know what else I can do to have something on my AMD screen...

Thanks for your help !

Guildem

Have you even read first post?

Anyway, you need to patch your kernel with i915 vgaarb patch.

HI dwe11er,

Thank you for your answer. Yes, I saw the first post, and other articles too. I didn't view all the pages but I tried to find some informations.

I am not an expert and I try to learn all those things, but posts I found talk about many distribs, many kernels, many case (intel or amd cpu, amd or nvidia gpu,...). So this is difficult  to find a good answer for my configuration.

I didn't used patches because I found people saying that they didn't patch their kernel. With your answer, I think that those people have an amd cpu.
I saw this i915 vgaarb patch may be in the 3.17 RC3 kernel. Is it true? I'll try to find informations about this patch. And I'll may compile the patched kernel of the first post to try it... or wait for 3.17...

I was thinking if I see the graphic card on the guest, all was OK with my stock kernel.

Thanks again!

Guildem

Offline

#2699 2014-09-11 13:05:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

guildem wrote:

I saw this i915 vgaarb patch may be in the 3.17 RC3 kernel. Is it true?

No, the other VGA arbiter patch is in 3.17, the i915 patch is not currently scheduled for any upstream kernel and is a blocking issue for using an unmodified kernel for users with IGD.  Those reporting success with an unmodified kernel probably don't have IGD or are not following the directions and using their GPU as a secondary card for the VM.  Please see the articles at the URL below, especially the FAQ for more info.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2700 2014-09-11 13:20:53

guildem
Member
Registered: 2014-09-11
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
guildem wrote:

I saw this i915 vgaarb patch may be in the 3.17 RC3 kernel. Is it true?

No, the other VGA arbiter patch is in 3.17, the i915 patch is not currently scheduled for any upstream kernel and is a blocking issue for using an unmodified kernel for users with IGD.  Those reporting success with an unmodified kernel probably don't have IGD or are not following the directions and using their GPU as a secondary card for the VM.  Please see the articles at the URL below, especially the FAQ for more info.

Thanks aw, that is not a good news... i'll read the posts on your link too.

Offline

#2701 2014-09-11 16:57:29

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Offline

#2702 2014-09-11 18:27:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you're not getting monitor sync when the VM starts, then you have a problem with VGA arbitration or device ROM.  Also, please stop specifying -bios, you don't need it.

You said you've applied the i915 VGA arbiter patch and are using i915.enable_hd_vgaarb=1... Are you actually loading i915 in the host or is it perhaps blacklisted?

Pastebin of host dmesg might be the next step if you can confirm i915 is being used.  There's also the non-VGA option with OVMF...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2703 2014-09-12 06:42:44

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You said you've applied the i915 VGA arbiter patch and are using i915.enable_hd_vgaarb=1... Are you actually loading i915 in the host or is it perhaps blacklisted?

Pastebin of host dmesg might be the next step if you can confirm i915 is being used.

I'm sure the i915 is not blacklisted and it's loaded and used by the host.



aw wrote:

There's also the non-VGA option with OVMF...

I don't really know what is OVMF, but reading something on the web I understood it has something to do with UEFI.
Consider that I'm not using UEFI on the host, nor no the guest.

Offline

#2704 2014-09-12 13:11:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
aw wrote:

There's also the non-VGA option with OVMF...

I don't really know what is OVMF, but reading something on the web I understood it has something to do with UEFI.
Consider that I'm not using UEFI on the host, nor no the guest.

Look down, hint, hint


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2705 2014-09-12 18:59:12

Arakatak
Member
Registered: 2014-09-12
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear Forum,
I have some problems installing ATi drivers under Windows 8.1 Guest with passed through ASUS R7 260X videocard.

Basic Windows setup with default VGA driver works (the OS is able to output to the passed through videocard). Howewer, installation of drivers from the manufacturer or AMD support website results in black screen and "No Signal" error on passed through VGA port from the monitor. After host reboot I'm able to start the Guest, but can't see anything except Seabios messages and initial logo startup screen.

If I'll detach VGA-adapter with a help of Windows clipboard icon, I'll get blank screen too, but simple reboot would solve the problem.

My Software:

  • Linux 3.16.0-mainline (with patches from this guide)

  • qemu 2.1.0

  • Seabios 1.7.5

  • Weston with XWayland to handle SDL qemu windows

  • xf86-video-intel

On host - UEFI without CSM

My Hardware:

  • ASRock Z97 Extreme4 motherboard

  • Intel i7-4790K (supports vt-d)

  • HD4600 as primary host video adapter

  • ASUS R7260X-DC2OC-2GD5 passed through to Windows Guest

cat /proc/cmdline root=/dev/disk/by-uuid/*** rw initrd=/initramfs-linux-mainline.img resume=/dev/disk/by-uuid/*** quiet intel_iommu=on i915.enable_hd_vgaarb=1

Additional kernel boot parameters "vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1" didn't help.

cat /etc/mkinitcpio.conf ... MODULES="i915" ...
cat /etc/modprobe.d/blacklist.conf blacklist radeon
dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x00000000BD341FF0 0000B8 (v01 INTEL BDW 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.021929] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.021933] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.021997] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.465600] DMAR: No ATSR found [ 0.465618] IOMMU 0 0xfed90000: using Queued invalidation [ 0.465619] IOMMU 1 0xfed91000: using Queued invalidation [ 0.465620] IOMMU: Setting RMRR: [ 0.465628] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff] [ 0.466660] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff] [ 0.466677] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff] [ 0.466692] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff] [ 0.466703] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.466708] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.532566] [drm] DMAR active, disabling use of stolen memory
cat qemu-win #!/bin/bash QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_DEV="hw:1,0" QEMU_ALSA_ADC_DEV="hw:1,0" qemu-system-x86_64 -enable-kvm \ -M q35 \ -m 8G \ -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/System/Windows8Guest,id=disk,format=raw,discard=on -device scsi-hd,drive=disk \ -drive file=/mnt/data/win.iso,id=isocd -device scsi-cd,drive=isocd \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1,id=exthda \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:24ae:2000

Please, provide me any suggestions. I'll post additional info if required.
Thanks in advance

Offline

#2706 2014-09-13 01:57:22

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have dell laptop with i3 i3-4010U it comes with just hd4200 video card.  I wonder if  its possible to have have it working ?
I get this :
-device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:00:02.0 BAR 2 mmap unsupported. Performance may be slow
qemu-system-x86_64: -device vfio-pci,host=00:03.0,bus=root.1,addr=00.1: vfio: Error: Failed to setup INTx fd: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=00:03.0,bus=root.1,addr=00.1: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:03.0,bus=root.1,addr=00.1: Device 'vfio-pci' could not be initialized
[root@localhost ~]# winstart.sh
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:00:02.0 BAR 2 mmap unsupported. Performance may be slow

Offline

#2707 2014-09-13 02:13:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I have dell laptop with i3 i3-4010U it comes with just hd4200 video card.  I wonder if  its possible to have have it working ?

Not with the current code


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2708 2014-09-13 08:08:59

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just tried XenGT some other day, and it's working very well. Intel igpu can be passed/SHARED with XenGT (warning!!! they say all 4th gen core processors but mine i3-4130 hd 4400 doesn't work, while 4770k hd 4600 works).

Intel also said that they have been working on KVM and  it should be soon (couple of months) supported.

Last edited by slis (2014-09-13 08:16:54)

Offline

#2709 2014-09-13 11:01:14

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to pass through a Sat card and it mostly works but I suffer occasional break ups in channel feeds (Signal is fine and card is OK when run on bare metal). The VM is Ubuntu and the problem appears to be made worse by higher bandwidth (HD channels). A windows VM on the same system showed a DPC latency of 2-4000, is this likely to be part of the cause? Does anyone know how I might fix this?

Offline

#2710 2014-09-13 12:44:51

abdullah
Member
Registered: 2014-06-10
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Arakatak,

try to go to device manager under windows, navigate to display adapter, click on it, now disable it ( Don't un install ), after displaying the adapter, windows should blink, now install the ATI drivers.

This problem comes from regular installations, not KVM/QEMU


@EDIT:@

Q: I hate running sudo every time I want to boot a VM, is it not possible to assign the right groups to the Hardware access?

Last edited by abdullah (2014-09-13 12:57:14)

Offline

#2711 2014-09-13 14:03:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Q: I hate running sudo every time I want to boot a VM, is it not possible to assign the right groups to the Hardware access?

It is, you need to give the user access to the vfio group device file(s) and you need to set the locked memory limit sufficiently high enough to pin all of guest memory.  The latter is the tricky part.  libvirt does both of these for you, which is one of the reasons motivating many to move to libvirt managed VMs.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2712 2014-09-13 15:45:25

fabian-z
Member
Registered: 2014-09-12
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In case someone may have use for it:
I adapted the i915 vgaarb patch to the current 3.17-rc4 linux-mainline kernel (failed to apply).
The VGA arbiter patch apparently got applied upstream & the ACS override patch still applies cleanly.

You can find my 3.17-rc4 PKGBUILD tarball (adapted from AUR package) including patches here: linux-mainline-3_17-vfio.tar.gz
Just extract and build with makepkg as usual.

$ sha256sum linux-mainline-3_17-vfio.tar.gz b72350c959932516b39cb309e94abbe4a3b4f83bf238c9055859bccb3175d07f linux-mainline-3_17-vfio.tar.gz

Kernel is working great for my setup (although I don't currently use libvirt, so I couldn't test this).

Last edited by fabian-z (2014-09-13 15:48:20)

Offline

#2713 2014-09-13 22:16:24

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got a Windows7 VM with PCI Pass-through. The PCI device is detected and drivers installed, though not in use with a Windows error: "The device cannot find enough free resources that it can use (code 12)".    Any suggestions on how to resolve this?

Xw9KuNxl.png

Full size image: http://i.imgur.com/Xw9KuNx.png

Moderator edit [ewaller] Converted over size image tags to url tags  https://wiki.archlinux.org/index.php/Fo … s_and_Code

Last edited by Tyrewt (2014-09-13 22:31:08)

Offline

#2714 2014-09-14 02:48:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:

I've got a Windows7 VM with PCI Pass-through. The PCI device is detected and drivers installed, though not in use with a Windows error: "The device cannot find enough free resources that it can use (code 12)".    Any suggestions on how to resolve this?

How about following any of the directions in the OP or the link below?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2715 2014-09-14 03:09:20

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Tyrewt wrote:

I've got a Windows7 VM with PCI Pass-through. The PCI device is detected and drivers installed, though not in use with a Windows error: "The device cannot find enough free resources that it can use (code 12)".    Any suggestions on how to resolve this?

How about following any of the directions in the OP or the link below?

That's not exactly being helpful. More specifically, on which of the 109 pages are you refering to?

Offline

#2716 2014-09-14 03:12:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
aw wrote:
Tyrewt wrote:

I've got a Windows7 VM with PCI Pass-through. The PCI device is detected and drivers installed, though not in use with a Windows error: "The device cannot find enough free resources that it can use (code 12)".    Any suggestions on how to resolve this?

How about following any of the directions in the OP or the link below?

That's not exactly being helpful. More specifically, on which of the 109 pages are you refering to?

OP = Original Post.  Page 1, post 1.  There are also only about 10 posts so far in the link below, all very helpful if I do say so myself.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2717 2014-09-14 04:38:35

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Tyrewt wrote:
aw wrote:

How about following any of the directions in the OP or the link below?

That's not exactly being helpful. More specifically, on which of the 109 pages are you refering to?

OP = Original Post.  Page 1, post 1.  There are also only about 10 posts so far in the link below, all very helpful if I do say so myself.

Neither explicitly mention the issue I'm addressing. Thanks anyway though.

Offline

#2718 2014-09-14 09:30:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like the lastest ovmf git works with the q35 chipset.

Last edited by nbhs (2014-09-14 09:35:44)

Offline

#2719 2014-09-14 13:02:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
aw wrote:
Tyrewt wrote:

That's not exactly being helpful. More specifically, on which of the 109 pages are you refering to?

OP = Original Post.  Page 1, post 1.  There are also only about 10 posts so far in the link below, all very helpful if I do say so myself.

Neither explicitly mention the issue I'm addressing. Thanks anyway though.

You're right, neither explicitly mention a Code 12 error because if you've configured things properly with your GPU as the primary VM graphics (hint, hint), a Code 12 error isn't something that happens.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2720 2014-09-14 16:57:51

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You're right, neither explicitly mention a Code 12 error because if you've configured things properly with your GPU as the primary VM graphics (hint, hint), a Code 12 error isn't something that happens.

Wonder how many of your other posts waste peoples time like this. Total poison to a community forum.

Offline

#2721 2014-09-14 17:21:24

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://wiki.archlinux.org/index.php/Fo … ther_Users
https://wiki.archlinux.org/index.php/Fo … esponsible
https://wiki.archlinux.org/index.php/Fo … troduction

I think the perception here is that the information is available for the reading.  I also think that aw is trying to lead you to where you can find the answer.
I personally do not know id aw is correct, or not -- I do not have this configuration and have no experience with it.

Last edited by ewaller (2014-09-14 17:21:56)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#2722 2014-09-14 17:34:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
aw wrote:

You're right, neither explicitly mention a Code 12 error because if you've configured things properly with your GPU as the primary VM graphics (hint, hint), a Code 12 error isn't something that happens.

Wonder how many of your other posts waste peoples time like this. Total poison to a community forum.

Here's how it works; if you show some effort that you've researched how this is supposed to work, followed the guides to make it work, and still have a unique issue and provide sufficient information to attempt to start helping you, then I and others on this thread will go out of our way to help you.  If you only drop a screenshot with an obviously broken setup (*), showing no evidence of attempting to understand the configuration or debug it yourself then yeah, you're going to get the same amount of effort from me as you're putting into the problem yourself.

(*) I tried to hint at this previously, but to be clear, your device manager shows two graphics devices and you're showing a screenshot of a QEMU SDL window.  This means you're using using an emulated VGA device in addition to a assigned VGA device.  There are some reports of Radeons working in this configuration, but they're typically one-offs.  The method suggested in the original post is to configure the VM with only the assigned VGA device in the guest.  There are also potentially patches required to make this work depending on your kernel version, QEMU version, and host configuration, none of which were shared with us.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2723 2014-09-14 17:48:53

Arakatak
Member
Registered: 2014-09-12
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

@ Arakatak,

try to go to device manager under windows, navigate to display adapter, click on it, now disable it ( Don't un install ), after displaying the adapter, windows should blink, now install the ATI drivers.

This problem comes from regular installations, not KVM/QEMU

It didn't help. Why do you think it's not qemu/kvm? What means "regular installations"? Is it a Windows 8.1 Setup? If so, i also experimented with Windows 7.

At first Win7 showed some light on possible problems. I installed Catalyst drivers and on next reboot received BSOD (code 0x00000116 on first boot and 0x0000050 on sequential reboots). I followed the advice in this post and removed 01:00.1 audio device from passthrough, then reinstalled the drivers. Although tray icon device detach still worked wrong (resulting in still picture from the last moment before VGA detach), graphics device have functioned well with ATi driver.

Unfortunately, i wasn't able to make a fully functional fresh install of Win7 VM without audio 01:00.1 device passthrough. The problems were the same as i mentioned earlier (black screen after driver setup and reboot, but no "No Signal" error).

For the reference i provide my start scripts:

cat qemu-win7-setup qemu-win7 #!/bin/bash QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_SIZE_IN_USEC=0 QEMU_ALSA_DAC_PERIOD_SIZE=256 QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_DEV="hw:1,0" QEMU_ALSA_ADC_DEV="hw:1,0" qemu-system-x86_64 -enable-kvm \ -M q35 \ -m 8G \ -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/System/Windows7Guest,id=disk,format=raw,discard=on -device scsi-hd,drive=disk \ -drive file="/mnt/data/win.iso",id=isocd -device scsi-cd,drive=isocd \ -drive file=/mnt/data/virtio-win-0.1-81.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd \ -boot once=d \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,id=extvga \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:046d:c328 \ -usbdevice host:046d:c247 #!/bin/bash QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_SIZE_IN_USEC=0 QEMU_ALSA_DAC_PERIOD_SIZE=256 QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_DEV="hw:1,0" QEMU_ALSA_ADC_DEV="hw:1,0" qemu-system-x86_64 -enable-kvm \ -M q35 \ -m 8G \ -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/System/Windows7Guest,id=disk,format=raw,discard=on -device scsi-hd,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,id=extvga \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:046d:c328 \ -usbdevice host:046d:c247

I welcome discussions on these problems, as it may help solve them.

abdullah wrote:

@EDIT:@

Q: I hate running sudo every time I want to boot a VM, is it not possible to assign the right groups to the Hardware access?

Every additional layer of API complicates troubleshooting. Why should i use libvirt when i didn't even have minimalistic, but functional VM with 100% VGA passthrough support? As for complexity, i need proper documentation on VGA passthrough with libvirt (and possibly OVMF VGA passthrough, as my VGA functions well on native UEFI). So far i found neither.

Offline

#2724 2014-09-14 19:16:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Arakatak wrote:

As for complexity, i need proper documentation on VGA passthrough with libvirt (and possibly OVMF VGA passthrough, as my VGA functions well on native UEFI). So far i found neither.

libvirt with VGA passthrough is difficult and usually results in using <qemu:arg> elements to hack around lack of support for the x-vga=on option of vfio.  A better route would be to set the emulator in the VM xml to a wrapper script that adds the x-vga=on option so that libvirt can properly manage the vfio files and locked memory.  An example of such a wrapper can be found here:

https://access.redhat.com/documentation … /virt.html

In the example, the option ",tx=time" is added to every instance of a virtio-net-pci.  A similar option for our application here might look something like 's|host=01:00.0|host=01:00.0,x-vga=on|g'

The benefit of OVMF is that VGA is not required.  You can find a write-up of how to use it here:

http://vfio.blogspot.com/2014/08/primar … t-vga.html
http://vfio.blogspot.com/2014/08/does-m … t-efi.html
http://vfio.blogspot.com/2014/09/libvir … -ovmf.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2725 2014-09-15 04:10:58

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:

Wonder how many of your other posts waste peoples time like this. Total poison to a community forum.

wow... is this some kind of appreciation for all the hard work AND great effort at keeping public well-informed about some really complex/experimental stuff?
truly, ignorance is a bliss...

@aw:
is there any way to track any kernel patches related to vga passthrough to know what (for people living far from kernel contribution realms)? some kind of newsgroup or any kind of filter on some website?
i mean, you mostly post about it on  your blog, but just in case you "miss" something

Offline

#2726 2014-09-15 08:23:26

guiper
Member
Registered: 2014-09-12
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

During a rainy weekend, played with getting OSX (Maverick) up and running with VGA Passthrough under KVM. I own a MacBook so in my case it should be legal, in case any moderator doesn't not agree feel free to delete this post.

Machine
Arch with 3.17 kernel and patches linked a couple of posts above
MB: Asrock Z77 Extreme 4-M
CPU Some  Intel I5 with VT-d enabled
AMD HD6450 being passthrough
qemu-git installed

Mavericks.iso I made on my Macbook Pro. Will not help here.
bios-mac.bin and chameleon_svn2360_boot you can find here http://blog.ostanin.org/2014/02/11/play … -x-on-kvm/

Installed OSX without using any passthrough, just using VNC, then for booting normally:

sudo qemu-system-x86_64 -enable-kvm -m 4096 -cpu core2duo -machine q35 \ -smp 4,sockets=1,cores=4,threads=1 \ -vga none -boot menu=on \ -device isa-applesmc,osk="GOOGLEIT!" \ -kernel /home/guilherme/OSX/chameleon_svn2360_boot \ -smbios type=2 \ -bios /home/guilherme/OSX/bios-mac.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=00:1d.0,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=root.1,addr=00.2 \ -device vfio-pci,host=00:1a.0,bus=root.1,addr=00.3 \ -drive file=/dev/samsung128/macosx,id=MacHDD,format=raw -device ide-hd,bus=ide.2,drive=MacHDD \ -device ide-drive,bus=ide.0,drive=MacDVD \ -drive id=MacDVD,if=none,snapshot=on,file=./Mavericks.iso -netdev user,id=hub0port0 \ -device e1000-82545em,netdev=hub0port0,id=eth0,mac=52:54:00:8e:3a:fc \ -nographic

USB devices don't work in Chameleon boot selector, using VNC I created a Org.chameleon.Boot.plist file in /Extra settings the boot device timeout for 5 seconds, so it boots automatically the MacHDD device (hd0,2 in my case).

Current Status:
USB is flaky upon boot, disconnecting and reconnecting the devices get them to work fine.
Sleep / Resume works fine!!!
Couldn't get Audio to work but didn't test it enough. Tried to play with loading KEXTs but no success.
Graphics on the AMD 6450 work perfectly out of the box without any issues and acceleration is fine
Performance seems perfectly fine and on par with my MacBook pros, but didn't test too much.

Finally part of the experience using OSX is due to the nice mouse / gestures experience. Unless you have a proper mighty mouse an Apple keyboard maybe it doesn't worth it. For me, since I can fix Audio for the moment I'm back on Windows 8.1

Last edited by guiper (2014-09-15 08:23:56)

Offline

#2727 2014-09-15 11:11:28

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

@aw:
is there any way to track any kernel patches related to vga passthrough to know what (for people living far from kernel contribution realms)? some kind of newsgroup or any kind of filter on some website?
i mean, you mostly post about it on  your blog, but just in case you "miss" something

http://vfio.blogspot.de/

Offline

#2728 2014-09-15 13:03:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks sinny, much appreciated

sinny wrote:

@aw:
is there any way to track any kernel patches related to vga passthrough to know what (for people living far from kernel contribution realms)? some kind of newsgroup or any kind of filter on some website?
i mean, you mostly post about it on  your blog, but just in case you "miss" something

The short answer is no, it's unfortunately an involved process to track things going upstream.  You can follow along on mailing lists or by watching project commits, but it's an involved process, which is why distributions offering commercial support hire so many people.  I'll try to continue to note relevant updates on the blog as I become aware of them.

It seems like we've reached a point where everything we expect to go upstream is there, now we're just waiting for releases of these versions.  The only outstanding patches are the i915 vga arbiter patch and the acs override patch.  Am I forgetting anything?  Neither of these are expected to go upstream.  OVMF negates the need for the i915 patch and the acs problem needs to be solved by creating quirks for specific devices.  I think for the application here it is generally required to separate processor-based root ports.  Unfortunately Intel has declined to provide a guarantee of isolation of those ports on client processors (i5/i7) but does provide native acs on xeon.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2729 2014-09-15 15:30:00

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been able to boot my Windows8 Guest up using the native libvirt OVMF support and can ping it, just unable to check if the VGA passthrough with my AMD R9 290 is working as I'm doing it remotely.

I have also managed to boot it up using a q35 machine type with the new OVMF from SVN, Windows8 is pinging again can't check VGA passthrough but will do later.

My command line entry for qemu looks like so:

/usr/local/qemu-xen-4-4-git-290814/bin/qemu-system-x86_64 -name windows8 -S -machine pc-q35-2.2,accel=kvm,usb=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off -drive file=/usr/share/ovmf/x64/ovmf_x64.bin,if=pflash,format=raw,unit=0 -m 5168 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 7b222825-fc7d-4a66-a72c-4875063752d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x1.0x7 -drive file=/dev/mapper/vg1-ovmfwinv1data,if=none,id=drive-virtio-disk1,format=raw -device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1,bootindex=1 -drive file=/dev/mapper/vg1-winv1data,if=none,id=drive-virtio-disk2,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x3,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/root/virtio-win-0.1-74.iso,if=none,id=drive-ide0-2-0,readonly=on,format=raw -device ide-cd,bus=ide.2,unit=0,drive=drive-ide0-2-0,id=ide0-2-0 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:12:34:56,bus=pci.2,addr=0x2,rombar=0 -device usb-host,hostbus=1,hostaddr=4,id=hostdev0 -device usb-host,hostbus=1,hostaddr=5,id=hostdev1 -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.2,multifunction=on,addr=0x8 -device vfio-pci,host=01:00.1,id=hostdev3,bus=pci.2,addr=0x9 -msg timestamp=on

My XML looks like so:

<domain type='kvm'> <name>windows8</name> <uuid>7b222825-fc7d-4a66-a72c-4875063752d2</uuid> <memory unit='KiB'>5291456</memory> <currentMemory unit='KiB'>5291456</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='q35'>hvm</type> <loader type='pflash'>/usr/share/ovmf/x64/ovmf_x64.bin</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'/> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/qemu-xen-4-4-git-290814/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' /> <source dev='/dev/mapper/vg1-ovmfwinv1data'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/mapper/vg1-winv1data'/> <target dev='vdc' bus='virtio'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/root/virtio-win-0.1-74.iso'/> <target dev='hdd' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='2' unit='0'/> </disk> <interface type='network'> <mac address='52:54:00:12:34:56'/> <source network='default'/> <target dev='tap4'/> <model type='virtio'/> <alias name='windows-nic-0'/> <rom bar='off'/> </interface> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52e'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0719'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0' /> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x08' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain>

Offline

#2730 2014-09-16 14:20:14

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
guildem wrote:

I saw this i915 vgaarb patch may be in the 3.17 RC3 kernel. Is it true?

No, the other VGA arbiter patch is in 3.17, the i915 patch is not currently scheduled for any upstream kernel and is a blocking issue for using an unmodified kernel for users with IGD.  Those reporting success with an unmodified kernel probably don't have IGD or are not following the directions and using their GPU as a secondary card for the VM.  Please see the articles at the URL below, especially the FAQ for more info.

I can report that without IGD (disabled in bios) and with 2 discrete video cards instead, i was able to passthrough with an unmodified kernel starting with kernel 3.13. On machines with X79 and Z97 chipsets. Quite easy, mostly doing it like this:
-better investigation of how mobo sets up iommu groups to find out which slot(s) allows for the wanted config passthrough (this is to find the slot configs which work without patch).
-using /etc/modules-load.d/... conf file to load the kernel modules ( kvm \ kvm_intel \ vfio \ vfio_pci \ vfio_iommu_type1)
-using /etc/modprobe.d/... conf file to set modules options, like options kvm ignore_msrs=1 and options vfio_iommu_type1 allow_unsafe_interrupts=1
I think the above folders and methos can be safely used to load modules and hence avoid the need to compile entire kernel just for switches on those modules. I am not aware of any disadvantage using this method of module loading, maybe somebody else knows some disadvantage? If anyone else wants to try, both conf files are to be edited with one module or one option per each text line, e.g. separated by new lines, not just by spaces. Maybe not all modules really need to be there, but having them wont hurt.

Currently using 3.17-rc last, but in spite of all the patches going on, with this config i have not noticed any improvements since 3.13, which (very subjectively here) seems to work best for me. Only on Z97 machine i guess due to its bios, the 3.17-rc1 vanished an dmesg err like "could not find upstream pcie", although things were working either way. My personal conclusion, for desktop use, i would rather buy a 20-30 eur cheap video card for host, and disable the intel graphics entirely, and find the 2 slot combination it works without patches. The laptop users might not be so lucky, but for desktop i find this approach to be ideal atm. Just to avoid dealing with intel graphics crap.

I cant shake the feeling that igd holding its teeth to vga has smth to do with mei along with its vpro backdoor. Either this smells bad or i do (or both). This even more underlined when a logical disable mechanism no longer works and more when the "maintainer is not willing to accept a driver mode switch option". Why on earth would not want to accept a driver mode switch, srsly what could be wrong with that, can anybody elaborate. I sincerely dont understand. But i think these back each other, meaning then perhaps the disabling mechanism no longer working is probably "working as intended". Anyway me personally, the last thing i want is stuff i am not allowed to disable. Even just as a matter of principle, but i suspect this might be more than just principles.

Offline

#2731 2014-09-16 19:29:20

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Ansa89 wrote:

I don't really know what is OVMF, but reading something on the web I understood it has something to do with UEFI.
Consider that I'm not using UEFI on the host, nor no the guest.

Look down, hint, hint

Ok, I have compiled OVMF and now I have a "OVMF.fd" file. I guess I have to pass that file as "-bios", right?


Moreover I tested the GTX 650 rom with your program, here is the output:

Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 0, vendor revision: 1 Error, ran off the end

PS: just wondering how much important OVMF test is...I mean: is it a needed dependency for vga passthrough?

Offline

#2732 2014-09-16 20:03:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
aw wrote:
Ansa89 wrote:

I don't really know what is OVMF, but reading something on the web I understood it has something to do with UEFI.
Consider that I'm not using UEFI on the host, nor no the guest.

Look down, hint, hint

Ok, I have compiled OVMF and now I have a "OVMF.fd" file. I guess I have to pass that file as "-bios", right?


Moreover I tested the GTX 650 rom with your program, here is the output:

Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 0, vendor revision: 1 Error, ran off the end

PS: just wondering how much important OVMF test is...I mean: is it a needed dependency for vga passthrough?

Oh dear, you've gone to all this trouble to compile OVMF but haven't read this post to determine why you'd want it and how to use it?  Have I missed something in describing the motivation and requirements there?  OVMF isn't a "test", it's an alternative to VGA.  Since you're not getting any output from the monitor, it seems like you have an issue with VGA routing.  Avoiding VGA by using OVMF is one method of dealing with this.

Regarding the parse error, how recently did you grab that copy of rom-parser?  I fixed a similar issue in the code on the 4th.  Otherwise please send me a copy of the ROM binary so I can fix the code for it.  Since you already have the edk2 tree, there's also a parser there you can use, details in the comments here.

For the -bios option; you shouldn't ever need a -bios option.  The default bios that comes with QEMU is fine for seabios use and for OVMF you'd add an option like:

-drive if=pflash,format=raw,readonly,file=/path/to/OVMF.fd

Your previous posts indicate that you're using Windows 7, so please pay particular attention to the guest operating support for UEFI.  You'd need to upgrade your guest to Windows 8 for OVMF.

If that's not any option, we'll need to go back to debugging your VGA routing issue and 'dmesg | grep -i vga' might shed some light on what's wrong.  Please also pastebin the entire dmesg log from the host.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2733 2014-09-16 20:18:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:
aw wrote:
guildem wrote:

I saw this i915 vgaarb patch may be in the 3.17 RC3 kernel. Is it true?

No, the other VGA arbiter patch is in 3.17, the i915 patch is not currently scheduled for any upstream kernel and is a blocking issue for using an unmodified kernel for users with IGD.  Those reporting success with an unmodified kernel probably don't have IGD or are not following the directions and using their GPU as a secondary card for the VM.  Please see the articles at the URL below, especially the FAQ for more info.

I can report that without IGD (disabled in bios) and with 2 discrete video cards instead, i was able to passthrough with an unmodified kernel starting with kernel 3.13. On machines with X79 and Z97 chipsets. Quite easy, mostly doing it like this:
-better investigation of how mobo sets up iommu groups to find out which slot(s) allows for the wanted config passthrough (this is to find the slot configs which work without patch).
-using /etc/modules-load.d/... conf file to load the kernel modules ( kvm \ kvm_intel \ vfio \ vfio_pci \ vfio_iommu_type1)
-using /etc/modprobe.d/... conf file to set modules options, like options kvm ignore_msrs=1 and options vfio_iommu_type1 allow_unsafe_interrupts=1

You should not need vfio_iommu_type1 allow_unsafe_interrupts=1 on either of these systems unless interrupt remapping is disabled in the BIOS.

I think the above folders and methos can be safely used to load modules and hence avoid the need to compile entire kernel just for switches on those modules. I am not aware of any disadvantage using this method of module loading, maybe somebody else knows some disadvantage? If anyone else wants to try, both conf files are to be edited with one module or one option per each text line, e.g. separated by new lines, not just by spaces. Maybe not all modules really need to be there, but having them wont hurt.

Currently using 3.17-rc last, but in spite of all the patches going on, with this config i have not noticed any improvements since 3.13, which (very subjectively here) seems to work best for me. Only on Z97 machine i guess due to its bios, the 3.17-rc1 vanished an dmesg err like "could not find upstream pcie", although things were working either way. My personal conclusion, for desktop use, i would rather buy a 20-30 eur cheap video card for host, and disable the intel graphics entirely, and find the 2 slot combination it works without patches. The laptop users might not be so lucky, but for desktop i find this approach to be ideal atm. Just to avoid dealing with intel graphics crap.

I cant shake the feeling that igd holding its teeth to vga has smth to do with mei along with its vpro backdoor. Either this smells bad or i do (or both). This even more underlined when a logical disable mechanism no longer works and more when the "maintainer is not willing to accept a driver mode switch option". Why on earth would not want to accept a driver mode switch, srsly what could be wrong with that, can anybody elaborate. I sincerely dont understand. But i think these back each other, meaning then perhaps the disabling mechanism no longer working is probably "working as intended". Anyway me personally, the last thing i want is stuff i am not allowed to disable. Even just as a matter of principle, but i suspect this might be more than just principles.

The desire to reduce options and make things work out of the box for all users is a good one by the i915 maintainer.  Even users in this thread who patch the kernel themselves and add the necessary options are often surprised and complain at the loss of DRI support on the host.  Unfortunately it doesn't seem like we have many options that both work for everyone and maintain compatibility with the current versions of Xorg.  This is why I continue to push people towards OVMF as an alternative (but please read the posts I'm pointing Ansa89 to before you leap without understanding the motivation and restrictions).  If your intended guest OS supports UEFI and so does your hardware, you can use OVMF, forget about VGA routing, and you also don't need to go buy another VGA card just for the host.  Everything is already upstream to make this work, even if there aren't yet released package versions, there will be sooner than we'll have an i915 VGA arbiter solution.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2734 2014-09-16 23:51:22

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Same identical problem.

My hardware

Motherboard: B75Pro3
Cpu:I5-3470S(host  gpu)
Secondary gpu: radeon r9 270.

I've gota Proxmox(Debian)  with kernel 3.14 with ACS and I915 arbiter patch. Included also vfio and vfio-vga in kernel.

Iommu is enabled I can passthrough devices like dvb pci express cards to other vm(ubuntu) but not secondary gpu with windows...

Can install using standard adapter, install drivers and infinite atikmpag.sys 0x16 bsod.

Tried detaching gpu, disabling card + install driver and also not passing audio to no avail.

I've got signal on my secondary monitor and windows logo is on screen.

Then bsod.

Tried several catalyst version from 13.4 to 14.7 RC3...

GRUB:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:6811,1002:aab0  i915.enable_hd_vgaarb=1”



Any idea?

Another question:
I'm running this machine headless accessing via web or ssh.
Windows VM would be a Steam in Home streaming server...

Is it possible to disable in bios intel cpu and passing ati card to vm(pci stubbing card) as primary passthrough?
So I would be free to not use the i915 patch and, reading other posts, maybe results are better than mixing IGPU and discrete card...
P.S. I'm also curious about What means "regular installations"?

Thanks in advance...


Arakatak wrote:
abdullah wrote:

@ Arakatak,

try to go to device manager under windows, navigate to display adapter, click on it, now disable it ( Don't un install ), after displaying the adapter, windows should blink, now install the ATI drivers.

This problem comes from regular installations, not KVM/QEMU

It didn't help. Why do you think it's not qemu/kvm? What means "regular installations"? Is it a Windows 8.1 Setup? If so, i also experimented with Windows 7.

At first Win7 showed some light on possible problems. I installed Catalyst drivers and on next reboot received BSOD (code 0x00000116 on first boot and 0x0000050 on sequential reboots). I followed the advice in this post and removed 01:00.1 audio device from passthrough, then reinstalled the drivers. Although tray icon device detach still worked wrong (resulting in still picture from the last moment before VGA detach), graphics device have functioned well with ATi driver.

Unfortunately, i wasn't able to make a fully functional fresh install of Win7 VM without audio 01:00.1 device passthrough. The problems were the same as i mentioned earlier (black screen after driver setup and reboot, but no "No Signal" error).

For the reference i provide my start scripts:

cat qemu-win7-setup qemu-win7 #!/bin/bash QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_SIZE_IN_USEC=0 QEMU_ALSA_DAC_PERIOD_SIZE=256 QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_DEV="hw:1,0" QEMU_ALSA_ADC_DEV="hw:1,0" qemu-system-x86_64 -enable-kvm \ -M q35 \ -m 8G \ -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/System/Windows7Guest,id=disk,format=raw,discard=on -device scsi-hd,drive=disk \ -drive file="/mnt/data/win.iso",id=isocd -device scsi-cd,drive=isocd \ -drive file=/mnt/data/virtio-win-0.1-81.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd \ -boot once=d \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,id=extvga \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:046d:c328 \ -usbdevice host:046d:c247 #!/bin/bash QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_SIZE_IN_USEC=0 QEMU_ALSA_DAC_PERIOD_SIZE=256 QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_DEV="hw:1,0" QEMU_ALSA_ADC_DEV="hw:1,0" qemu-system-x86_64 -enable-kvm \ -M q35 \ -m 8G \ -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/System/Windows7Guest,id=disk,format=raw,discard=on -device scsi-hd,drive=disk \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,id=extvga \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -usb -usbdevice host:046d:c328 \ -usbdevice host:046d:c247

I welcome discussions on these problems, as it may help solve them.

abdullah wrote:

@EDIT:@

Q: I hate running sudo every time I want to boot a VM, is it not possible to assign the right groups to the Hardware access?

Every additional layer of API complicates troubleshooting. Why should i use libvirt when i didn't even have minimalistic, but functional VM with 100% VGA passthrough support? As for complexity, i need proper documentation on VGA passthrough with libvirt (and possibly OVMF VGA passthrough, as my VGA functions well on native UEFI). So far i found neither.

Last edited by evilsephiroth (2014-09-16 23:57:23)

Offline

#2735 2014-09-17 01:36:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

P.S. I'm also curious about What means "regular installations"?

I think it means trying to move a bare metal installation to a VM rather than doing a fresh install in the VM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2736 2014-09-17 03:16:40

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Speaking of R9 290, I haven't heard anything new about the inability to reset the device.  Has any progress been made on that?

If it would help, I can try out various kernel patches.

Offline

#2737 2014-09-17 06:16:18

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any hint to diagnose my problem?
My system seems compliant...

Offline

#2738 2014-09-17 06:33:23

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

speaking of ovmf, i wonder if there is some kind of list of uefi-compliant gpus, or some kind of way to know if some distinct gpu would work or now

gigabyte's hd 6450 does NOT support uefi, palit gt 430 is pending for test (i doubt it will support it though)

so i'll probably have to go "buy new cheap(-est) one(-s)" route - would not be that great to blind-pick anything

Last edited by sinny (2014-09-17 07:48:27)

Offline

#2739 2014-09-17 06:40:56

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

speaking of ovmf, i wonder if there is some kind of uefi-compliant gpus, or some kind of way to know if some distinct gpu would work or now

gigabyte's hd 6450 does NOT support uefi, palit gt 430 is pending for test (i doubt it will support it though)

so i'll probably have to go "buy new cheap(-est) one(-s)" route - would not be that great to blind-pick anything

Just to be clear...

OVMF is a replacement for VGA, right?

can it be a problem solver in case of ATI bsod?

My guess would be no because they're related mainly to passtrough itself and not to vga routing...

Offline

#2740 2014-09-17 07:59:29

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:
sinny wrote:

...

Just to be clear...

OVMF is a replacement for VGA, right?

can it be a problem solver in case of ATI bsod?

My guess would be no because they're related mainly to passtrough itself and not to vga routing...

that was NOT written in response to you, in case you care

and on your problem:
based on what you write, you did not even try to read all the stuff that was written in OP post and on THIS page (not to mention doing some "research" in the thread or web on your own)
r9 270/r9 270x was reported as working at least few times in the sheet, so i'd bet 95% that you are doing something wrong (probably because of lack of understanding of what you are doing at all)

please do not waste people's time on something trivial - first waste your own time instead: read, explore, play around with parameters or start from scratch, repeat until it works or you spent like TONS of time with no result (and read TONS of info)

Offline

#2741 2014-09-17 08:17:35

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
evilsephiroth wrote:
sinny wrote:

...

Just to be clear...

OVMF is a replacement for VGA, right?

can it be a problem solver in case of ATI bsod?

My guess would be no because they're related mainly to passtrough itself and not to vga routing...

that was NOT written in response to you, in case you care

and on your problem:
based on what you write, you did not even try to read all the stuff that was written in OP post and on THIS page (not to mention doing some "research" in the thread or web on your own)
r9 270/r9 270x was reported as working at least few times in the sheet, so i'd bet 95% that you are doing something wrong (probably because of lack of understanding of what you are doing at all)

please do not waste people's time on something trivial - first waste your own time instead: read, explore, play around with parameters or start from scratch, repeat until it works or you spent like TONS of time with no result (and read TONS of info)


Well, this is awkyard...

I've used Xen and kvm with various gpus with success. Only this gpu is causing me problems...

I did read all pages and search other users having my problems... It's been 4 weeks since I start experimenting with the system.

Applied all the solution in threads with no avail...

Based on first page, kernel 3.16 is a requirement but I wanted to achieve same thing with 13.4... Patching has been done also modifying patches from patchwork to adapt to my kernel...

Read throughly also your blog pages and my system conforms to your description...

I've also helped kantras at the time I was using xen 4.3  on AUR... So I think that I have a minimum understanding of what It's happening...

P.S. Links are broken on your post...

Why are you assuming I did not read all infos and done research? Is something not clear from my forum post?

Last edited by evilsephiroth (2014-09-17 08:28:27)

Offline

#2742 2014-09-17 08:44:51

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

...
Why are you assuming I did not read all infos and done research? Is something not clear from my forum post?

and that's why:
- first part of the reason:

evilsephiroth wrote:

Just to be clear...
OVMF is a replacement for VGA, right?

- second part of the reason(may be found on previous page; yes, i lied about it being on THIS page):

i might be wrong here, but from my point of view, after reading (yes, reading - not just skimming through) the blog post, there may be many questions, but not the one you asked.

you may check others' reports here (there is even r9 270x on 3.14 kernels): report sheet

Offline

#2743 2014-09-17 09:01:58

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
evilsephiroth wrote:

...
Why are you assuming I did not read all infos and done research? Is something not clear from my forum post?

and that's why:
- first part of the reason:

evilsephiroth wrote:

Just to be clear...
OVMF is a replacement for VGA, right?

- second part of the reason(may be found on previous page; yes, i lied about it being on THIS page):

i might be wrong here, but from my point of view, after reading (yes, reading - not just skimming through) the blog post, there may be many questions, but not the one you asked.

you may check others' reports here (there is even r9 270x on 3.14 kernels): report sheet


Checking right now the google docs...

relating to OVMF, I was confirming this information  from aw "  OVMF isn't a "test", it's an alternative to VGA.  Since you're not getting any output from the monitor, it seems like you have an issue with VGA routing."

Nevertheless, will check the google doc and report results... I would love to use proxmox and using a more recent kernel but I don't know if It's a feasible way to do this....

Offline

#2744 2014-09-17 11:31:52

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You should not need vfio_iommu_type1 allow_unsafe_interrupts=1 on either of these systems unless interrupt remapping is disabled in the BIOS.

Dunno why, but i need it just for X79. Without it does not work, and i get the warning as per last line:

dmesg | grep -E -i 'dmar|rmr|iommu|drh|remap' [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=6d0f0340-4faa-4691-b3b6-c26d936404f2 rw radeon.dpm=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a,8086:1d2d,8086:1d3a [ 0.000000] ACPI: DMAR 0x00000000CB7DF000 0000E8 (v01 INTEL DX79TO 0000028A MSFT 0100000D) [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=6d0f0340-4faa-4691-b3b6-c26d936404f2 rw radeon.dpm=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a,8086:1d2d,8086:1d3a [ 0.000000] Intel-IOMMU: enabled [ 0.024733] dmar: Host address width 40 [ 0.024734] dmar: DRHD base: 0x000000fe901000 flags: 0x0 [ 0.024739] dmar: IOMMU 0: reg_base_addr fe901000 ver 1:0 cap d2008c10ef0462 ecap f0207a [ 0.024740] dmar: DRHD base: 0x000000fe900000 flags: 0x1 [ 0.024743] dmar: IOMMU 1: reg_base_addr fe900000 ver 1:0 cap d2078c106f0462 ecap f020fe [ 0.024744] dmar: RMRR base: 0x000000c8dda000 end: 0x000000c8ddafff [ 0.024745] dmar: RMRR base: 0x000000c8ddc000 end: 0x000000c8ddcfff [ 0.378367] DMAR: No ATSR found [ 0.378402] IOMMU 0 0xfe901000: using Queued invalidation [ 0.378405] IOMMU 1 0xfe900000: using Queued invalidation [ 0.378407] IOMMU: Setting RMRR: [ 0.378417] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc8ddc000 - 0xc8ddcfff] [ 0.378441] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc8dda000 - 0xc8ddafff] [ 0.378455] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.378463] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 27.591182] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

I also get lots of errors but somehow i am used to them:

dmesg | grep -E -i 'fail|error|warn|panic| bug|backtrace' [ 0.000000] ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Pm1aEventBlock: 32/16 (20140724/tbfadt-618) [ 0.000000] ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/PmTimerBlock: 32/24 (20140724/tbfadt-618) [ 0.000000] ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aEventBlock: 16, using default 32 (20140724/tbfadt-699) [ 0.000000] ACPI BIOS Warning (bug): Invalid length for FADT/PmTimerBlock: 24, using default 32 (20140724/tbfadt-699) [ 0.095166] tsc: Marking TSC unstable due to check_tsc_sync_source failed [ 0.217170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.384545] ioapic: probe of 0000:00:05.4 failed with error -22 [ 2.077427] ACPI Warning: SystemIO range 0x0000000000004000-0x000000000000401f conflicts with OpRegion 0x0000000000004000-0x000000000000400f (\_SB_.PCI0.SMBS.SMBI) (20140724/utaddress-258) [ 2.988725] radeon 0000:03:00.0: registered panic notifier

The fadt ones started to appear since 3.14. The ioapic which fails to be recognized is "00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) \ Subsystem: Intel Corporation Device 4953". The path to use pci=nocrs or alternatives was not leading anywhere. The radeon panic notifier is weird coz the xorg log has really no (ee) and only few not related (ww). I think the video card (R7 240) should of been recognized as r600 but its seen as r300.
Maybe i am doing something wrong. In general i used to blame this board, coz i had lots of issues with this particular board and its badly broken bios(es). I tried other bioses but from previous one back i had that bug which was loading the cpu and made it hit 90 degrees at idle, not even booted, just inside bios sensors screen could watch temp increasing like crazy with apparently no reason - so since then i keep blaming mei and related "features" for almost anything. smile

Offline

#2745 2014-09-17 14:35:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

speaking of ovmf, i wonder if there is some kind of list of uefi-compliant gpus, or some kind of way to know if some distinct gpu would work or now

gigabyte's hd 6450 does NOT support uefi, palit gt 430 is pending for test (i doubt it will support it though)

so i'll probably have to go "buy new cheap(-est) one(-s)" route - would not be that great to blind-pick anything

I would hope that a card advertised with Windows 8 support would include a UEFI ROM.  If it claims to support secure boot, then it must support UEFI.  Otherwise there's quite a selection of ROMs that can be tested prior to purchase in the techpowerup database http://www.techpowerup.com/vgabios/.  It's also possible that another vendor may ship a UEFI BIOS compatible with your card that can be loaded into QEMU via the romfile option.  I also found that EVGA didn't ship a UEFI ROM for my GTX660, but will send one via support ticket.  Neither of your cards are very new though, so you may indeed be out of luck with them.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2746 2014-09-17 14:43:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:
aw wrote:

You should not need vfio_iommu_type1 allow_unsafe_interrupts=1 on either of these systems unless interrupt remapping is disabled in the BIOS.

Dunno why, but i need it just for X79. Without it does not work, and i get the warning as per last line:

[ 0.024739] dmar: IOMMU 0: reg_base_addr fe901000 ver 1:0 cap d2008c10ef0462 ecap f0207a [ 0.024743] dmar: IOMMU 1: reg_base_addr fe900000 ver 1:0 cap d2078c106f0462 ecap f020fe ... [ 27.591182] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

I suspect it's either disabled in the BIOS or your kernel doesn't support interrupt remapping.  The VT-d Extended Capability register (ecap) defines but 3 to indicate interrupt remapping support.  It's set on both the IOMMUs in your system according to the log, so the hardware is capable of interrupt remapping.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2747 2014-09-17 15:10:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:
sinny wrote:

speaking of ovmf, i wonder if there is some kind of uefi-compliant gpus, or some kind of way to know if some distinct gpu would work or now

gigabyte's hd 6450 does NOT support uefi, palit gt 430 is pending for test (i doubt it will support it though)

so i'll probably have to go "buy new cheap(-est) one(-s)" route - would not be that great to blind-pick anything

Just to be clear...

OVMF is a replacement for VGA, right?

can it be a problem solver in case of ATI bsod?

My guess would be no because they're related mainly to passtrough itself and not to vga routing...

OVMF is a replacement for VGA, the idea is to avoid legacy VGA access and thus all the arbitration issues that go along with it by using a legacy-free BIOS implementation.

It likely does not solve the ATI BSOD.

Have you tried switching to a 440FX machine type, with either VGA or OVMF?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2748 2014-09-17 15:24:55

Arakatak
Member
Registered: 2014-09-12
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

Same identical problem.

I don't think our problems are identical in nature. I suggest you should try to pass VGA without audio, then boot Windows VM in safe mode, uninstall ALL ATi drivers (better to revert the system using restore points, Catalyst creates one on installation). After that install only VGA driver without CCC and other AMD bloatware. Report if it works smile

My problem usually results in black screen, not in BSOD. Unfortunately, i still can't fix it.

aw wrote:
evilsephiroth wrote:

P.S. I'm also curious about What means "regular installations"?

I think it means trying to move a bare metal installation to a VM rather than doing a fresh install in the VM.

You've lead me to the idea a problem may be in VGA/system in general. The manufacturer claims the videocard is Win8 compatible. I can't test it though. My HDD is fully mapped as lvm. Still, i think doing a fresh install in VM is the best decision.

Are there any users who were able to do a successful AMD R7 260x setup with Catalyst drivers? The report sheet contains some mentions about this GPU. How can i find that people?

Offline

#2749 2014-09-17 16:40:10

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you install the i915 arbiter patch?

Because before patching kernel, I was having Windows logo ==> black screen ==> reboot...

Then after patch, I could see seabios + bsod...

I don't know if it's related but...

Currently, I'm redoing installation so to will try to pass only video without radeon audio...

Will make you know




Arakatak wrote:
evilsephiroth wrote:

Same identical problem.

I don't think our problems are identical in nature. I suggest you should try to pass VGA without audio, then boot Windows VM in safe mode, uninstall ALL ATi drivers (better to revert the system using restore points, Catalyst creates one on installation). After that install only VGA driver without CCC and other AMD bloatware. Report if it works smile

My problem usually results in black screen, not in BSOD. Unfortunately, i still can't fix it.

aw wrote:
evilsephiroth wrote:

P.S. I'm also curious about What means "regular installations"?

I think it means trying to move a bare metal installation to a VM rather than doing a fresh install in the VM.

You've lead me to the idea a problem may be in VGA/system in general. The manufacturer claims the videocard is Win8 compatible. I can't test it though. My HDD is fully mapped as lvm. Still, i think doing a fresh install in VM is the best decision.

Are there any users who were able to do a successful AMD R7 260x setup with Catalyst drivers? The report sheet contains some mentions about this GPU. How can i find that people?

Offline

#2750 2014-09-17 16:48:39

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Arakatak wrote:
evilsephiroth wrote:

Same identical problem.

I don't think our problems are identical in nature. I suggest you should try to pass VGA without audio, then boot Windows VM in safe mode, uninstall ALL ATi drivers (better to revert the system using restore points, Catalyst creates one on installation). After that install only VGA driver without CCC and other AMD bloatware. Report if it works smile

My problem usually results in black screen, not in BSOD. Unfortunately, i still can't fix it.

aw wrote:
evilsephiroth wrote:

P.S. I'm also curious about What means "regular installations"?

I think it means trying to move a bare metal installation to a VM rather than doing a fresh install in the VM.

You've lead me to the idea a problem may be in VGA/system in general. The manufacturer claims the videocard is Win8 compatible. I can't test it though. My HDD is fully mapped as lvm. Still, i think doing a fresh install in VM is the best decision.

Are there any users who were able to do a successful AMD R7 260x setup with Catalyst drivers? The report sheet contains some mentions about this GPU. How can i find that people?

Try to pass only gpu without hd audio (but assign both to vfio-pci).

Offline

#2751 2014-09-17 18:03:07

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Arakatak wrote:

My problem usually results in black screen, not in BSOD. Unfortunately, i still can't fix it.

Are there any users who were able to do a successful AMD R7 260x setup with Catalyst drivers? The report sheet contains some mentions about this GPU. How can i find that people?

Based on http://wiki.x.org/wiki/RadeonFeature/ , it looks like R7 260(x) is a Sea Islands card (Bonaire), like my R9 290 (Hawaii). 

Assuming your passthrough is set up propely in terms of VGA arbitration, perhaps your video issue is the same device-reset failure.

In my case, the OS loads, but the desktop is black.  I can tell the desktop is actually alive by enabling Narrator: press win+u, and press space when the voice says "hear text read out loud".  I seem to recall even being able to open Catalyst Control Center.

If that's what's happening to you, the workaround is to shut down the guest, suspend and resume the host, then start the guest again.  You'll have to do this every time the guest restarts for any reason.

The spreadsheet entries for R7-260(x) include notes to reboot the host for guest reboot to work, but suspend/resume should have the same effect.

Last edited by DanaGoyette (2014-09-17 18:05:38)

Offline

#2752 2014-09-17 18:30:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
Arakatak wrote:

My problem usually results in black screen, not in BSOD. Unfortunately, i still can't fix it.

Are there any users who were able to do a successful AMD R7 260x setup with Catalyst drivers? The report sheet contains some mentions about this GPU. How can i find that people?

Based on http://wiki.x.org/wiki/RadeonFeature/ , it looks like R7 260(x) is a Sea Islands card (Bonaire), like my R9 290 (Hawaii).

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2753 2014-09-17 21:06:11

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

My card is :
http://www.sapphiretech.com/presentatio … id=1&leg=0

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770 / R9 260 OEM] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 7790 Dual-X OC Flags: bus master, fast devsel, latency 0, IRQ 51 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d0000000 (64-bit, prefetchable) [size=8M] I/O ports at e000 [size=256] Memory at f6100000 (32-bit, non-prefetchable) [size=256K] Expansion ROM at f6140000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] #13 Capabilities: [2d0] #1b Kernel driver in use: vfio-pci Kernel modules: radeon

Offline

#2754 2014-09-18 02:41:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

Ok, I'll try to keep an eye out for a cheap one on ebay


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2755 2014-09-18 02:50:09

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not sure what the reset problem ppls talk about R9. I had success with Sapphire Radeon R9 270 Dual-X OC 2GB DDR5 256Bit, but i recall having some issues with the audio part complaining about a power state, and i think it worked when was assigned with a secondary port explicitly for the audio. So anybody willing to try something like this:

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,addr=00.0,multifunction=on,bus=root.1,x-vga=on \ -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=01:00.1,addr=00.1,multifunction=on,bus=root.2 \

Ofc with the host=...0X:00.0/1 part changed if necessary. To test if its the same problem i had, copy-paste the qemu command line in a terminal, and if its the same problem upon first run it gives no error message, but only upon shutdown and next subsequent vm start, only then it complains something related to the power state of the audio address. Sry cant recall the exact msg. But i think the above approach might just solve that, again, if its the same problem. Best of luck.

Last edited by noobman (2014-09-18 03:01:02)

Offline

#2756 2014-09-18 02:54:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

I am not sure what the reset problem ppls talk about R9. I had success with Sapphire Radeon R9 270 Dual-X OC 2GB DDR5 256Bit, but i recall having some issues with the audio part complaining about a power state, and i think it worked when was assigned with a secondary port explicitly for the audio. So anybody willing to try something like this:

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,addr=00.0,multifunction=on,bus=root.1,x-vga=on \ -device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \ -device vfio-pci,host=01:00.1,addr=00.1,multifunction=on,bus=root.2 \

Ofc with the host=...0x:00.x part changed if necessary. To test if its the same problem i had, copy-paste the qemu command line in a terminal, and if its the same problem upon first run it gives no error message, but only upon shutdown and next subsequent vm start, only then it complains something about something related to the power state of the audio address. Sry cant recall the exact msg. But i think the above approach might just solve that, again, if its the same problem. Best of luck.

As I understand it, the reset problem prevents users with these cards from even rebooting the VM without going through contortions with host suspend/resume or reboot.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2757 2014-09-18 03:09:34

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

As I understand it, the reset problem prevents users with these cards from even rebooting the VM without going through contortions with host suspend/resume or reboot.

Dunno, what others had tbh. I just described my own experience. The error message i had was upon follow-up attempt of starting the vm, and was something like "stuck in suspend" or "stuck in D3" or something like that, and was pointing not to the video of 00.0, but was instead pointing to the ...00.1 which was the audio. I first tried passing only the video card, that particular message was gone but i still had problems with the driver instalation in guest. When i passed both like that, with different port for the audio, then it stopped throwing errors upon subsequent start(s) and also drivers installed right away. Again just to be clear, that just my eperience and may not fix the others issues, i just thought its worth mentioning and maybe somebody can give it a try.

Edit: Oh i didnt meant the user would suspend/resume the vm, no that was just qemu complaining when the vm was re-started. It would restart the vm, it would just throw that error message right before starting. I mean just like those messages of "iommu group not viable" etc. Tho this time somebody needs to see that exact message, so has to copy-paste in terminal otherwise it would be missed because qemu just keeps going after it.
Edit2: Also i think the reason warning is not there at first vm start (after a host boot), maybe because the passed card is just being initialized first time. Only at subseqent vm starts the warning appears, as if the previous improper reset left the device in a bad power state. This is how it looks like at least. Again, just to say, this is my experience, not necesarily what others have. Anyway my issue was gone with that allocation of secondary port for audio. Maybe that way it results in guest doing two resets, one distinct for each device. Or maybe the driver just expects the audio to be in a port right near. No clue why it worked like that.

Last edited by noobman (2014-09-18 03:45:53)

Offline

#2758 2014-09-18 04:28:17

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

I am not sure what the reset problem ppls talk about R9. I had success with Sapphire Radeon R9 270 Dual-X OC 2GB DDR5 256Bit, ...

The R9 270 is not a Sea Islands card.  See http://xorg.freedesktop.org/wiki/Radeon … etingnames

Southern Islands:
CAPE VERDE, PITCAIRN, TAHITI, OLAND, HAINAN
HD7750 - HD7970, R9 270, R9-280, R7 240, R7 250

Sea Islands:
BONAIRE, KABINI, KAVERI, HAWAII
HD7790, R7 260, R9 290

Offline

#2759 2014-09-18 11:06:07

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bummer. I could report success wtih the R9 270 and with R7 240 which i'm using now, too bad its beyond the point. But another idea, if issues reflect on host, could you check $dmesg | grep -i hotplug if it reports back something like hotplug for pcie and pci. Cant guess something else about why host would be in trouble.

Offline

#2760 2014-09-18 18:50:12

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

Bummer. I could report success wtih the R9 270 and with R7 240 which i'm using now, too bad its beyond the point. But another idea, if issues reflect on host, could you check $dmesg | grep -i hotplug if it reports back something like hotplug for pcie and pci. Cant guess something else about why host would be in trouble.

qemu? kernel? mobo? parameters?

just to see if I'm missing something. I can0t passthrough my r9 270

Offline

#2761 2014-09-19 00:15:37

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

qemu? kernel? mobo? parameters? just to see if I'm missing something. I can0t passthrough my r9 270

I edited the spreadsheet posted by mr noctavian, the R9 270 config is on row no 276. Cpu i7-4790 and no intel graphics bs (disabled in bios), just 2 plain cards instead. Really no patches of any sort, stock kernel, and the module loading as i described a few posts before. The grub cmd line still needs intel_iommu=on and pci-stub.ids=.... but otherwise the modules are loaded via /etc/modules-load.d/... one module per line of conf file, and options if needed via the other folder, but in case of Z97 chipset i think should need no special options.
Kernels can work starting with 3.13 but recommending up to date 3.17-rc5 (or at least rc3). If you are using latest this might not be the case, but otherwise for older kernel and stuff, can use cli $ ll /sys/kernel/iommu_groups/1/devices/ and play with the 2 cards slot combination to make sure the iommu group with the passed gpu is clean and does not contain other devices.
About R9 270 i had that issue i also described a few posts above, with qemu complaining/warning "stuck in D3", which dissapeared when putting the hdmi audio on secondary ioh3420 port. You could also describe your configuration.

Last edited by noobman (2014-09-19 00:21:49)

Offline

#2762 2014-09-19 04:24:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So... just happened to get an update for Nvidia 344.11... Code 43 again sad

I have no words...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2763 2014-09-19 04:30:23

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

just installed new nvidia driver 344.11 and got error code 43 sad(((

Last edited by slis (2014-09-19 04:31:01)

Offline

#2764 2014-09-19 05:56:09

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So... just happened to get an update for Nvidia 344.11... Code 43 again sad

I have no words...

Lol, you guys should just be grateful you got your cards working - why try to update? yikes

Haven't read this for a bit (my nvidia failure killed my enthusiasm somewhat), but great to see that spreadsheet with results - well done whoever did it.

Offline

#2765 2014-09-19 10:12:54

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So... just happened to get an update for Nvidia 344.11... Code 43 again sad

I have no words...

Silly nVidia sad I'm glad I bought AMD R9 big_smile


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#2766 2014-09-19 11:41:00

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

Bummer. I could report success wtih the R9 270 and with R7 240 which i'm using now, too bad its beyond the point. But another idea, if issues reflect on host, could you check $dmesg | grep -i hotplug if it reports back something like hotplug for pcie and pci. Cant guess something else about why host would be in trouble.

would it be too much of a trouble to ask you some more details on r7 240?
some points of special interest (r7 240 as passed gpu, not host one):
- any problems with it so far? reboots or anything else
- vendor/model/memory?
- does it support UEFI boot (just firmware check would be enough in fact)?

Offline

#2767 2014-09-19 14:27:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

My card is :
http://www.sapphiretech.com/presentatio … id=1&leg=0

Does the Sapphire card have an EFI BIOS?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2768 2014-09-19 14:37:45

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

would it be too much of a trouble to ask you some more details on r7 240?
some points of special interest (r7 240 as passed gpu, not host one): [...]

Ups i am very sry i think that was innacurate from my part, R7 240 was host card. Anyway I tried to do a "quickie" passthrough now to see how it is, and i got lots errors like these:

[...]VFIO 0000:03:00.0 BAR 0 mmap unsupported. Performance may be slow [...]vfio_bar_read(,0x9080, 4) failed: Device or resource busy" and "vfio_bar_write(,0x9084, 0xffffff, 4) failed: Device or resource busy [...]vfio_bar_write(,0x9084, 0xffffff, 4) failed: Device or resource busy [dmesg:] [ 57.855348] systemd-journald[183]: /dev/kmsg buffer overrun, some messages lost. [ 57.855359] vfio-pci 0000:03:00.0: BAR 0: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref] [ 57.857534] vfio-pci 0000:03:00.0: BAR 0: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]

Sry have not investigated any further. Maybe i was doing something wrong. And/or this mobo i have (dx79to) has bios revisions which are total crap, all of them.
On this system i pass the nvidia and i dont do driver updates. On the other system with Z97 chipset the R9 270 is passed and i could say i am happier with that instead, costed ~50% less and benchmarks ~20% better, tho this is a subjective opinion.

Last edited by noobman (2014-09-19 14:38:40)

Offline

#2769 2014-09-19 14:44:09

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So... just happened to get an update for Nvidia 344.11... Code 43 again sad

I have no words...

Then it looks like they are doing this on purpose? Couldnt be a coincidence could it?

Offline

#2770 2014-09-19 14:57:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

would it be too much of a trouble to ask you some more details on r7 240?
some points of special interest (r7 240 as passed gpu, not host one):
- any problems with it so far? reboots or anything else
- vendor/model/memory?
- does it support UEFI boot (just firmware check would be enough in fact)?

I have an HD8570 OEM card, which I think is effectively the same as the r7 240.  It works well for me, no reset problem.  It does report NoSoftRst-, which makes the vfio code attempting to reset the device on release not activate, even though PM reset itself doesn't seem to do anything.  That just means if the VM is stopped rather than doing a clean shutdown, the last image on the screen remains (for now).  The OEM card is not the highest quality, a little noisy, but I found that the fanless CoolViva Z1 can take care of that problem if you've got the room.  Yes, the OEM HD8570 supports UEFI.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2771 2014-09-19 15:51:03

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw, noobman:
thanks for the info.

aw wrote:
sinny wrote:

would it be too much of a trouble to ask you some more details on r7 240?
some points of special interest (r7 240 as passed gpu, not host one):
- any problems with it so far? reboots or anything else
- vendor/model/memory?
- does it support UEFI boot (just firmware check would be enough in fact)?

I have an HD8570 OEM card, which I think is effectively the same as the r7 240.  It works well for me, no reset problem.  It does report NoSoftRst-, which makes the vfio code attempting to reset the device on release not activate, even though PM reset itself doesn't seem to do anything.  That just means if the VM is stopped rather than doing a clean shutdown, the last image on the screen remains (for now).  The OEM card is not the highest quality, a little noisy, but I found that the fanless CoolViva Z1 can take care of that problem if you've got the room.  Yes, the OEM HD8570 supports UEFI.

well, seems like i'll have to bet a bit and blindly try r7 240 of some vendor (no 8570 at my place at all)
was initially looking at nvidia cards, but all this driver absurd (not to mention that i never had success trying to passthrough gt 430)... "nvidia, .uck you!" (c) linus

Offline

#2772 2014-09-19 16:07:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

@aw, noobman:
thanks for the info.

aw wrote:
sinny wrote:

would it be too much of a trouble to ask you some more details on r7 240?
some points of special interest (r7 240 as passed gpu, not host one):
- any problems with it so far? reboots or anything else
- vendor/model/memory?
- does it support UEFI boot (just firmware check would be enough in fact)?

I have an HD8570 OEM card, which I think is effectively the same as the r7 240.  It works well for me, no reset problem.  It does report NoSoftRst-, which makes the vfio code attempting to reset the device on release not activate, even though PM reset itself doesn't seem to do anything.  That just means if the VM is stopped rather than doing a clean shutdown, the last image on the screen remains (for now).  The OEM card is not the highest quality, a little noisy, but I found that the fanless CoolViva Z1 can take care of that problem if you've got the room.  Yes, the OEM HD8570 supports UEFI.

well, seems like i'll have to bet a bit and blindly try r7 240 of some vendor (no 8570 at my place at all)
was initially looking at nvidia cards, but all this driver absurd (not to mention that i never had success trying to passthrough gt 430)... "nvidia, .uck you!" (c) linus

Don't forget you can download ROMs from the TechPowerUp database and test them for UEFI support.  This can at least provide some indication of which vendor to pick.

EDIT: to save you the trouble, there are only 3 ROMs listed for the R7 240 and only the VTX3D includes EFI support.  That's certainly not definitive of the other vendors though.

Last edited by aw (2014-09-19 16:09:49)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2773 2014-09-19 19:16:13

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
anickname wrote:

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

My card is :
http://www.sapphiretech.com/presentatio … id=1&leg=0

Does the Sapphire card have an EFI BIOS?

No EFI BIOS.

See :

https://www.sendspace.com/file/2mjmw8

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770 / R9 260 OEM] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 7790 Dual-X OC Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 51 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at d0000000 (64-bit, prefetchable) [size=8M] Region 4: I/O ports at e000 [size=256] Region 5: Memory at f6100000 (32-bit, non-prefetchable) [size=256K] Expansion ROM at f6140000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00718 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover+ Timeout+ NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: vfio-pci Kernel modules: radeon

Last edited by anickname (2014-09-19 19:25:43)

Offline

#2774 2014-09-20 00:56:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So... just happened to get an update for Nvidia 344.11... Code 43 again sad

I have no words...

"bluebird" reported on oftc/#qemu that 344.11 is now checking hyper-v extensions, so if you remove all those hv-foo cpu parameters and leave kvm=off, 344.11 works.  I've confirmed this on my VM.  Unfortunately some of those hypver-v extensions are fairly useful and have shown to help performance, so NVIDIA appears to be intentionally crippling GeForce, possibly to make Quadro appear to be a better choice for a VM (as if official support isn't enough).  NVIDIA already seems to disable MSI on GeForce, so it's not the first time GeForce has been intentionally crippled to boost the professional cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2775 2014-09-20 12:37:09

stokbaek
Member
Registered: 2014-09-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have been trying to get this VGA Passthrough to work now for some time but seems like I am stuck now sad

When I try to start the VM will it open in a QEMU window and the terminal will say:

qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.0, no available reset mechanism.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.9.3-030903-generic root=UUID=8c0763f7-e4c0-4b58-80e3-6db2416ea751 ro quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1

lspci -nn | grep NVIDIA

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 660] [10de:11c0] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)

dmesg | grep pci-stub

[ 0.500541] pci-stub: add 10DE:1187 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.500549] pci-stub 0000:01:00.0: claimed by stub [ 0.500556] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.500559] pci-stub 0000:01:00.1: claimed by stub

Any help would be greatly appreciated

Offline

#2776 2014-09-20 13:29:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

stokbaek wrote:

Hello,

I have been trying to get this VGA Passthrough to work now for some time but seems like I am stuck now sad

Your kernel version indicates that you're attempting to use stock Ubuntu 13.04 and we can only assume stock QEMU from that release too.  VGA assignment did not work at the time that release.  I'd suggest at least kernel 3.16 and QEMU 2.1, but you may still need patches beyond that depending on your host graphics.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2777 2014-09-20 14:59:55

stokbaek
Member
Registered: 2014-09-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
stokbaek wrote:

Hello,

I have been trying to get this VGA Passthrough to work now for some time but seems like I am stuck now sad

Your kernel version indicates that you're attempting to use stock Ubuntu 13.04 and we can only assume stock QEMU from that release too.  VGA assignment did not work at the time that release.  I'd suggest at least kernel 3.16 and QEMU 2.1, but you may still need patches beyond that depending on your host graphics.

Sorry forgot to mention kernel and QEMU version

uname -r

3.9.3-030903-generic

kvm -version

QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.3), Copyright (c) 2003-2008 Fabrice Bellard

Offline

#2778 2014-09-20 16:32:07

0011001011
Member
Registered: 2014-09-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everybody, and thank you for this amazing thread ! I was able to make my virtual gaming machine and definitively erase my dualboot. I am almost free from windows (and with steam on linux things will only get better). Information on internet is not clear so I want to confirm that GTX 570 works well with VGA passthrough on HD4000 with CGI working on arch host (don't know why btw)/

However.. my guest sometimes violently crashes : impossible to boot and I have to restore it from a copy of last working raw.img, it seems to be caused by updates but dont know which one (300+ to check) on w7 64bits ultimate. I checked to disable driver integrity check and stuff, run a chkdsk bootrec and bcdedit but nothing works. Does anybody has a solution or plan of solution ? Impossible to debug since windows repair tool thinks that a number of things are broken, not only the boot (and I think it broke it itself while trying to repair)

Offline

#2779 2014-09-20 16:35:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

0011001011 wrote:

Hi everybody, and thank you for this amazing thread ! I was able to make my virtual gaming machine and definitively erase my dualboot. I am almost free from windows (and with steam on linux things will only get better). Information on internet is not clear so I want to confirm that GTX 570 works well with VGA passthrough on HD4000 with CGI working on arch host (don't know why btw)/

However.. my guest sometimes violently crashes : impossible to boot and I have to restore it from a copy of last working raw.img, it seems to be caused by updates but dont know which one (300+ to check) on w7 64bits ultimate. I checked to disable driver integrity check and stuff, run a chkdsk bootrec and bcdedit but nothing works. Does anybody has a solution or plan of solution ? Impossible to debug since windows repair tool thinks that a number of things are broken, not only the boot (and I think it broke it itself while trying to repair)

Are you running virtio or AHCI for the guest disk?  Q35 or 440FX?  virtio + 440FX are likely to be the best performing and best supported combo.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2780 2014-09-20 16:45:47

0011001011
Member
Registered: 2014-09-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owh i didn't know about that, I use Q35 and for the disk I typed the following :

drive file=/home/sofian/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

So I used q35.. I really thought it was superior to 440fx.
It might be uninteresting to notice but I never installed virtio driver for windows because I wanted to update the system first (and thought it would crash the system because it may change the device windows detect)

Last edited by 0011001011 (2014-09-20 17:04:54)

Offline

#2781 2014-09-20 17:16:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

0011001011 wrote:

Owh i didn't know about that, I use Q35 and for the disk I typed the following :

drive file=/home/sofian/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

So I used q35.. I really thought it was superior to 440fx.
It might be uninteresting to notice but I never installed virtio driver for windows because I wanted to update the system first (and thought it would crash the system because it may change the device windows detect)

You can find an example of a 440FX+virtio config here

For installation to virtio you can configure the VM with 2 CD images, the second being the virtio ISO image you can find here.  When discovering the disk the Windows installer will prompt to load external drivers.  It's also recommended to use virtio for the NIC rather than an emulated device.

Q35 is largely overrated.  There are some configurations of Linux that need it, but Windows is fine on 440FX.

Last edited by aw (2014-09-20 17:16:44)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2782 2014-09-20 17:40:25

0011001011
Member
Registered: 2014-09-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Right I will test it right now and try my 5th win installation (love you SSDs)

I know that vrtio is better for NIC however i was not using it, I already have a dedicated wifi PCi card but it disturb iommu at boot.
So you think Q35 and drivers might be the explanation of these crashes ? In fact I absolutely don't know why win update fail at some point. Assuming that it is the root of the error.

Offline

#2783 2014-09-20 19:10:45

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:

So... just happened to get an update for Nvidia 344.11... Code 43 again sad

I have no words...

"bluebird" reported on oftc/#qemu that 344.11 is now checking hyper-v extensions, so if you remove all those hv-foo cpu parameters and leave kvm=off, 344.11 works.  I've confirmed this on my VM.  Unfortunately some of those hypver-v extensions are fairly useful and have shown to help performance, so NVIDIA appears to be intentionally crippling GeForce, possibly to make Quadro appear to be a better choice for a VM (as if official support isn't enough).  NVIDIA already seems to disable MSI on GeForce, so it's not the first time GeForce has been intentionally crippled to boost the professional cards.

After 4 months I tried vga passthrough again and succeeded this time and now this happens  :X

What were they thinking? disabling the product i own? mad mad
I will make it my mission to find a way to get back at them!

Do you think it can be solved like the kvm flag check or we can no more use hyper-v enlightments?

by the way my setup: i5-4670, Asrock Z87 Extreme4, 2x MSI Geforce 770
i hope i can make an awesome permanent setup. using onboard vga for xbmc mediacenter, one 770 for my little brother, one vm with 770 for me streamed  to my tablet, and automatic tiered storage in the virtual machines

Offline

#2784 2014-09-21 01:20:25

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I made the switch from legacy VGA to libvirt + OVMF and I report success with some small issues.

My card (Sapphire 7970 Vapor-X) didn't have UEFI, so I flashed it with the Sapphire x280 Vapor-X bios (basically the same card, but it has UEFI). I left it like that for a week to see if I'd run into any problems and I didn't.

So today I set it up with OVMF (libvirt-git, ovmf-svn) and I got it to boot and work, but only as a secondary passthrough. Even if I remove the video devices I still don't get any output before windows loads the AMD drivers

Also, I need to use the audio device on the card because it is how I take sound to the TV, but it is crackling and lagging. It didn't do that before on legacy. And when I tried to play a game the card was stuttering in places where it didn't before.

Here is my xml:

<domain type='kvm'> <name>windows</name> <uuid>298e3f5f-1546-424c-b8f8-1225241eb5d0</uuid> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='3' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/virtual-machines/windows'/> <target dev='vda' bus='virtio'/> <boot order='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/home/lucas/media/windows.img'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:5a:02:e8'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

Offline

#2785 2014-09-21 07:30:30

kevinxucs
Member
Registered: 2014-09-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For anyone who owns a Radeon HD 7850 (made by MSI), and having no output on Windows 8.1 after installing catalyst (or BSoD on Windows 7).

Try not passing through the HDMI audio controller inside the graphic card (address should be 01:00.1), it works for me.

Last edited by kevinxucs (2014-09-21 07:53:59)

Offline

#2786 2014-09-21 10:20:50

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Question on USB:
I have passed through one of my USB controllers, so all attached devices are directly recognized by the VM. This works fine for all kinds of devices, but one problem exists: It seems Seabios does not support USB keyboards, so I can not use the keyboard untill the VM has booted up. This is mostly not a problem, but I can't influence the boot process of Seabios (boot order etc.) and in case of unclean Windows guest shutdown I can't move the cursor in the "start windows in safe mode or boot normally"-screen...
I suspect this is a limitation of Seabios, but I did not find any confirmation on this. Any advice?

Offline

#2787 2014-09-21 10:58:53

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Question on USB:
I have passed through one of my USB controllers, so all attached devices are directly recognized by the VM. This works fine for all kinds of devices, but one problem exists: It seems Seabios does not support USB keyboards, so I can not use the keyboard untill the VM has booted up. This is mostly not a problem, but I can't influence the boot process of Seabios (boot order etc.) and in case of unclean Windows guest shutdown I can't move the cursor in the "start windows in safe mode or boot normally"-screen...
I suspect this is a limitation of Seabios, but I did not find any confirmation on this. Any advice?


Use : -bios bios-256k.bin

Offline

#2788 2014-09-21 11:36:24

Arakatak
Member
Registered: 2014-09-12
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems like the problem in black screen after driver setup (VGA AMD R7 260x) is actually in lack of reset capability.
I got working passthrough with the following workarounds:
Passthrough only VGA, not its audio device
Cut power to VGA on every VM restart (possible with shutdown or suspend, not with reboot)

Thank you for all suggestions!

aw wrote:

Q35 is largely overrated.  There are some configurations of Linux that need it, but Windows is fine on 440FX.

Will you explain the advantages of Q35 over 440FX, please? I know that Q35 supports pci-e, and 440FX doesn't. Are there any performance penalties for passing through pci-e card to virtual pci bus?

Offline

#2789 2014-09-21 14:07:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chetyre wrote:

I made the switch from legacy VGA to libvirt + OVMF and I report success with some small issues.

My card (Sapphire 7970 Vapor-X) didn't have UEFI, so I flashed it with the Sapphire x280 Vapor-X bios (basically the same card, but it has UEFI). I left it like that for a week to see if I'd run into any problems and I didn't.

So today I set it up with OVMF (libvirt-git, ovmf-svn) and I got it to boot and work, but only as a secondary passthrough. Even if I remove the video devices I still don't get any output before windows loads the AMD drivers

Also, I need to use the audio device on the card because it is how I take sound to the TV, but it is crackling and lagging. It didn't do that before on legacy. And when I tried to play a game the card was stuttering in places where it didn't before.

Hmm, the setup is a little suspect since we're using a ROM image from a different card, even though I agree they seem like they should be pretty much the same.  Not getting any output until the driver loads sounds like the UEFI ROM isn't working.  Maybe try this ovmf image, which is just a copy of OVMF-pure-efi.fd from Gerd Hoffmann's Fedora build.  Maybe your OVMF image includes the CSM and is trying to use the VGA BIOS, I don't know.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2790 2014-09-21 14:21:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Arakatak wrote:
aw wrote:

Q35 is largely overrated.  There are some configurations of Linux that need it, but Windows is fine on 440FX.

Will you explain the advantages of Q35 over 440FX, please? I know that Q35 supports pci-e, and 440FX doesn't. Are there any performance penalties for passing through pci-e card to virtual pci bus?

Two things that Q35 does:

1) The guest PCI topology is closer to the host's.  IOW, you can put your assigned graphics card functions behind an emulated root port.

2) It provides access to extended PCIe config space of the device.

AFAIK, 1) is the main reason it gets used, but the topology difference is largely FUD.  There are some cases where Linux drivers find a PCIe device and assume they can poke the downstream port above it without testing whether it actually exists.  If you're attempting to do SLI, it might also be useful since the docs seem to be rather particular about the topology of the hardware for that use case.

For everything else, I don't expect it makes much difference and I'm not aware of any performance difference (if there is one, please share).  Having a Q35 model does not enable any sort of link negotiation; the link registers on the root port is emulated and fixed.  If anything I'd be afraid that a negotiation would result in lowering the speed of the device to match the root port (but I assume that's not happening because then the device and physical root port would be out of sync and probably stop working altogether).  The extended config space is potentially an area where there could be some added performance, manipulating the device IOTLB for instance, but I have doubts whether it's actually used and if it is, how well it works on Q35 anyway.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2791 2014-09-21 15:16:53

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:
TripleSpeeder wrote:

Question on USB:
I have passed through one of my USB controllers, so all attached devices are directly recognized by the VM. This works fine for all kinds of devices, but one problem exists: It seems Seabios does not support USB keyboards, so I can not use the keyboard untill the VM has booted up. This is mostly not a problem, but I can't influence the boot process of Seabios (boot order etc.) and in case of unclean Windows guest shutdown I can't move the cursor in the "start windows in safe mode or boot normally"-screen...
I suspect this is a limitation of Seabios, but I did not find any confirmation on this. Any advice?


Use : -bios bios-256k.bin

Cool, works smile

If you are using libvirt to manage your VMs you need to

virsh edit <vmname>

and change the "loader" line to point to bios-256k.bin:

... <os> <type arch='x86_64' machine='pc-q35-2.0'>hvm</type> <loader>/usr/share/qemu/bios-256k.bin</loader> <bootmenu enable='yes' timeout='3000'/> </os> ...

(At least I did not find a way to change this setting with the virt-manager GUI)

Offline

#2792 2014-09-21 16:19:22

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Chetyre wrote:

I made the switch from legacy VGA to libvirt + OVMF and I report success with some small issues.

My card (Sapphire 7970 Vapor-X) didn't have UEFI, so I flashed it with the Sapphire x280 Vapor-X bios (basically the same card, but it has UEFI). I left it like that for a week to see if I'd run into any problems and I didn't.

So today I set it up with OVMF (libvirt-git, ovmf-svn) and I got it to boot and work, but only as a secondary passthrough. Even if I remove the video devices I still don't get any output before windows loads the AMD drivers

Also, I need to use the audio device on the card because it is how I take sound to the TV, but it is crackling and lagging. It didn't do that before on legacy. And when I tried to play a game the card was stuttering in places where it didn't before.

Hmm, the setup is a little suspect since we're using a ROM image from a different card, even though I agree they seem like they should be pretty much the same.  Not getting any output until the driver loads sounds like the UEFI ROM isn't working.  Maybe try this ovmf image, which is just a copy of OVMF-pure-efi.fd from Gerd Hoffmann's Fedora build.  Maybe your OVMF image includes the CSM and is trying to use the VGA BIOS, I don't know.


I have tried it just now, but it still didn't work. Still only showing an image when windows loads the driver.

But what really worries me is the performance issue, I tried it with Resident Evil 6 which shouldn't be a really demanding game for this card (and it wasn't when I was using qemu directly with legacy), but I couldn't even get it to the menu, it was rendering things at ~13 FPS.

Any idea on what may be wrong? I'm thinking about just returning my setup to how it was (might keep the new bios though, it wasn't causing any problems).

EDIT: Ok... Afterburner says my GPU clock is at 300 MHz when the game starts. This might not be a passthrough problem afterall.

Last edited by Chetyre (2014-09-21 16:27:09)

Offline

#2793 2014-09-22 12:19:50

nose
Member
Registered: 2014-09-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

i get this error:

qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: Device 'vfio-pci' could not be initialized

Additional info:

uname -r 3.16.0-1-mainline

The patched one from post #1

pci groups:

sh ./lsgroup ### Group 0 ### 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 01:00.0 PCI bridge: Device 1fc8:0910 (rev 80) 02:08.0 PCI bridge: Device 1fc8:0810 02:09.0 PCI bridge: Device 1fc8:0890 03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] ### Group 2 ### 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) ### Group 3 ### 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) ### Group 4 ### 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) ### Group 5 ### 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) ### Group 6 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) ### Group 7 ### 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4) ### Group 8 ### 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4) ### Group 9 ### 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) ### Group 11 ### 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4) ### Group 12 ### 00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) ### Group 13 ### 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 01) ### Group 14 ### 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 15 ### 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) ### Group 16 ### 08:00.0 VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GT] (rev a1)

can anyone help me or tell me where to read?
regards!

Offline

#2794 2014-09-22 12:44:25

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nose, I can't tell how well the pass-through of a dual graphic card does work, but this error:

nose wrote:
vfio: error opening /dev/vfio/1: No such file or directory

tells me that the vfio module might be missing. Look at the starting post, "Binding a device to vfio-pci". Within the little script:

modprobe vfio-pci

But the rest of the script is important, too, to bind your graphics card(s) to the vfio driver.

Offline

#2795 2014-09-22 13:40:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, what mauorrizze says, additionallly...

nose wrote:

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

This is an invalid config, you're assigning both 3:00.0 and 4:00.0 to the same address on the guest.  Also note that only one of the GPUs is identified as VGA (4:00.0), only that one should use x-vga.  I would start with just the VGA GPU + audio (4:00.0 & 4:00.1), bind 3:00.0 to vfio-pci, but don't expose it to the guest.  See if you can get that working first.  Then add the 2nd GPU.  There are several ways you could expose it, the easiest might be to try to expose it as function 2 on the same device.  You could also add a 2nd root port and expose it there.  QEMU also has support for PCIe switches, so you could add an upstream switch port with 2 downstream ports, just like you have on real hardware and expose VGA+audio and GPU.

Now the bad news; there's a very real possibility that the switches on this card are designed to re-route DMA between the endpoints so that peer-to-peer DMA between the GPUs never leaves the card.  That means the DMA might not be seen by the IOMMU and without ACS on the switches we don't have the ability to configure it for upstream forwarding.  If that's the case, then the only way these devices could work in a VM is if all of their address ranges were identity mapped into the guest, which is not something we have any ability to do.  Try it and see if it works, but know that it's a gamble.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2796 2014-09-22 14:27:46

nose
Member
Registered: 2014-09-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your replay, mauorrizze, i fixed my scripts!

aw wrote:

Yes, what mauorrizze says, additionallly...

nose wrote:

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

This is an invalid config, you're assigning both 3:00.0 and 4:00.0 to the same address on the guest.  Also note that only one of the GPUs is identified as VGA (4:00.0), only that one should use x-vga.  I would start with just the VGA GPU + audio (4:00.0 & 4:00.1), bind 3:00.0 to vfio-pci, but don't expose it to the guest.  See if you can get that working first.  Then add the 2nd GPU.  There are several ways you could expose it, the easiest might be to try to expose it as function 2 on the same device.  You could also add a 2nd root port and expose it there.  QEMU also has support for PCIe switches, so you could add an upstream switch port with 2 downstream ports, just like you have on real hardware and expose VGA+audio and GPU.

Now the bad news; there's a very real possibility that the switches on this card are designed to re-route DMA between the endpoints so that peer-to-peer DMA between the GPUs never leaves the card.  That means the DMA might not be seen by the IOMMU and without ACS on the switches we don't have the ability to configure it for upstream forwarding.  If that's the case, then the only way these devices could work in a VM is if all of their address ranges were identity mapped into the guest, which is not something we have any ability to do.  Try it and see if it works, but know that it's a gamble.

Tannks aw,
i tried what you said:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ > -smp 6,sockets=1,cores=6,threads=1 \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on \ > -device vfio-pci,host=04:00.1,bus=root.1

gives me:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 0 mmap unsupported. Performance may be slow

and a QEMU console window! smile but sadly the screens attached to the passthoughed card just tun off and stay black.

but at least some vm starts.. big_smile

aw, in Windows the card appears as two graphic cards, but only one appears to have connectors. And i have no idea if they redirect any DMA traffic.


UPDATE:
still no luck to get the screen on the passtroughed card working...
i noticed that in the group of the graphiccard:

.... .... ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 01:00.0 PCI bridge: Device 1fc8:0910 (rev 80) 02:08.0 PCI bridge: Device 1fc8:0810 02:09.0 PCI bridge: Device 1fc8:0890 03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] .... ....

are alot of pci bridges which surely belong to the integrated second GPU and its Crossfire ability.
better view:

# lspci -t -[0000:00]-+-00.0 +-01.0-[01-04]----00.0-[02-04]--+-[0000:04]-+-00.0 | | \-00.1 | +-[0000:03]---00.0 | \-[0000:02]-+-08.0-[00]-- | \-09.0-[00]-- +-14.0 +-16.0 +-1a.0 +-1c.0-[05]----00.0 +-1c.1-[06]----00.0 +-1c.2-[07]----00.0 +-1c.4-[08]----00.0 +-1d.0 +-1e.0-[09]-- +-1f.0 +-1f.2 \-1f.3

I tried to pass trough the whole group, but i get errors for example:

-device vfio-pci,host=01:00.0,bus=root.1: vfio: Assignment of PCIe type 0x5 devices is not currently supported

i dont want to buy a new graphiccard sad

if anybody has suggestions, feel free!

Last edited by nose (2014-09-22 15:34:47)

Offline

#2797 2014-09-22 21:34:32

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nose wrote:

Thanks for your replay, mauorrizze, i fixed my scripts!

aw wrote:

Yes, what mauorrizze says, additionallly...

nose wrote:

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

This is an invalid config, you're assigning both 3:00.0 and 4:00.0 to the same address on the guest.  Also note that only one of the GPUs is identified as VGA (4:00.0), only that one should use x-vga.  I would start with just the VGA GPU + audio (4:00.0 & 4:00.1), bind 3:00.0 to vfio-pci, but don't expose it to the guest.  See if you can get that working first.  Then add the 2nd GPU.  There are several ways you could expose it, the easiest might be to try to expose it as function 2 on the same device.  You could also add a 2nd root port and expose it there.  QEMU also has support for PCIe switches, so you could add an upstream switch port with 2 downstream ports, just like you have on real hardware and expose VGA+audio and GPU.

Now the bad news; there's a very real possibility that the switches on this card are designed to re-route DMA between the endpoints so that peer-to-peer DMA between the GPUs never leaves the card.  That means the DMA might not be seen by the IOMMU and without ACS on the switches we don't have the ability to configure it for upstream forwarding.  If that's the case, then the only way these devices could work in a VM is if all of their address ranges were identity mapped into the guest, which is not something we have any ability to do.  Try it and see if it works, but know that it's a gamble.

Tannks aw,
i tried what you said:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ > -smp 6,sockets=1,cores=6,threads=1 \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ > -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on \ > -device vfio-pci,host=04:00.1,bus=root.1

gives me:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 0 mmap unsupported. Performance may be slow

and a QEMU console window! smile but sadly the screens attached to the passthoughed card just tun off and stay black.

but at least some vm starts.. big_smile

aw, in Windows the card appears as two graphic cards, but only one appears to have connectors. And i have no idea if they redirect any DMA traffic.


UPDATE:
still no luck to get the screen on the passtroughed card working...
i noticed that in the group of the graphiccard:

.... .... ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 01:00.0 PCI bridge: Device 1fc8:0910 (rev 80) 02:08.0 PCI bridge: Device 1fc8:0810 02:09.0 PCI bridge: Device 1fc8:0890 03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] .... ....

are alot of pci bridges which surely belong to the integrated second GPU and its Crossfire ability.
better view:

# lspci -t -[0000:00]-+-00.0 +-01.0-[01-04]----00.0-[02-04]--+-[0000:04]-+-00.0 | | \-00.1 | +-[0000:03]---00.0 | \-[0000:02]-+-08.0-[00]-- | \-09.0-[00]-- +-14.0 +-16.0 +-1a.0 +-1c.0-[05]----00.0 +-1c.1-[06]----00.0 +-1c.2-[07]----00.0 +-1c.4-[08]----00.0 +-1d.0 +-1e.0-[09]-- +-1f.0 +-1f.2 \-1f.3

I tried to pass trough the whole group, but i get errors for example:

-device vfio-pci,host=01:00.0,bus=root.1: vfio: Assignment of PCIe type 0x5 devices is not currently supported

i dont want to buy a new graphiccard sad

if anybody has suggestions, feel free!


"BAR 0 mmap unsupported. Performance may be slow"

Get lspci -vvnn, find your gpu, find it's region 10 memory range
It'll look something like this:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device [1043:0459] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 18 Region 0: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 2: Memory at fe300000 (64-bit, non-prefetchable) [disabled] [size=256K] Region 4: I/O ports at e000 [disabled] [size=256] Expansion ROM at fe340000 [disabled] [size=128K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: radeon

Then, after acknowledging region 10 memory range, you go cat /proc/iomem
and look for that range, it'll look like this:

 c0000000-cfffffff : PCI Bus 0000:01 c0000000-cfffffff : 0000:01:00.0 AND HERE HIDES THE VESAFB!

(i've disabled vesa already)

I guess the GPU you're trying to pass is the first one in the system, and various uefi/bios output data on screen using it. Vesa does the same. Some motherboards have the option to change the default vga, but i don't have on, so...

To turn it off, assuming you have GRUB2:
edit /etc/default/grub
change GRUB_GFXPAYLOAD_LINUX="keep" to "text"
and append nofb to GRUB_CMDLINE_LINUX="..."
And then run grub2-mkconfig -o /boot/grub2/grub.cfg

Then reboot and check iomem again, vesa should be gone.

Vesa is the framebuffer, check the archwiki for more info on why you would or would not need it.

P.S. now there is two of us, trying to get crossfire working inside the vm.

Last edited by Duelist (2014-09-22 21:36:23)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2798 2014-09-23 06:19:26

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

seems like no more nvidia for me (both for ideology and practical aspects)

gigabyte r7 240 working using ovmf (or at least seem to - will stick with win7/vga for now)

Offline

#2799 2014-09-24 09:30:15

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone give a few pointer what benefits OVMF would give? I understand that it is basically providing an UEFI Bios to the VM, enabling stuff like secureboot etc. But if I have a working setup with the old Seabios - will using OVMF give me any advantage?

Offline

#2800 2014-09-24 09:40:15

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Can someone give a few pointer what benefits OVMF would give? I understand that it is basically providing an UEFI Bios to the VM, enabling stuff like secureboot etc. But if I have a working setup with the old Seabios - will using OVMF give me any advantage?

i wonder if you have actually read/understood anything at all...

it's gpu passthrough we are talking about here - not virtualising uefi boot process in general.
the core advantage ovmf gives is no vga legacy mess (kernel patching, etc).

p.s. i may be wrong here, but i really doubt anyone cares about secureboot (or any other weird stuff) when running 3d-accelerated vm.

Offline

#2801 2014-09-24 12:13:11

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
TripleSpeeder wrote:

Can someone give a few pointer what benefits OVMF would give? I understand that it is basically providing an UEFI Bios to the VM, enabling stuff like secureboot etc. But if I have a working setup with the old Seabios - will using OVMF give me any advantage?

i wonder if you have actually read/understood anything at all...

Well, probably not enough :-)

sinny wrote:

it's gpu passthrough we are talking about here - not virtualising uefi boot process in general.
the core advantage ovmf gives is no vga legacy mess (kernel patching, etc).

p.s. i may be wrong here, but i really doubt anyone cares about secureboot (or any other weird stuff) when running 3d-accelerated vm.

Yes, this is also what I care about. Right now I have my VM booting & working flawlessly with my passed-through vga card, without applying any patches whatsoever to kernel, qemu or SeaBios. Just using what is in the arch stable repository. I just want to understand if i would have any benefit setting up my VM using OVMF, like increased performance, better stability (I did not have a crash for several weeks now though).

So what is the vga legacy mess you are refering to? We don't need to discuss this here, just having some pointers to read would be awesome smile

Offline

#2802 2014-09-24 12:53:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

So what is the vga legacy mess you are refering to? We don't need to discuss this here, just having some pointers to read would be awesome smile

http://vfio.blogspot.com/2014/08/whats- … ation.html

If you're not affected by i915's VGA arbiter behavior and don't care for libvirt support, then just keep doing what you're doing.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2803 2014-09-24 15:16:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

Well, FWIW I now own an XFX HD7790 card.  All I can confirm so far is that it's quite broken.  VM reboots work well with the standard Windows VGA driver, but once the Catalyst drivers are loaded a reboot results in a PAGE_FAULT_IN_NONPAGED_AREA BSOD and a slew of IOMMU faults and stack traces (AMD 990fx host).  If I replace the HD7790 with an HD7850, everything works well.  I'm also a little bummed that the UEFI ROM on this card hangs, both bare metal and VM, hopefully XFX can point me to an update.  Anyway, no clues yet and not a lot of time to dig into it lately, but I share your misery.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2804 2014-09-24 18:32:42

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
anickname wrote:

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

Well, FWIW I now own an XFX HD7790 card.  All I can confirm so far is that it's quite broken.  VM reboots work well with the standard Windows VGA driver, but once the Catalyst drivers are loaded a reboot results in a PAGE_FAULT_IN_NONPAGED_AREA BSOD and a slew of IOMMU faults and stack traces (AMD 990fx host).  If I replace the HD7790 with an HD7850, everything works well.  I'm also a little bummed that the UEFI ROM on this card hangs, both bare metal and VM, hopefully XFX can point me to an update.  Anyway, no clues yet and not a lot of time to dig into it lately, but I share your misery.

Thank you, for your time and effort.

Offline

#2805 2014-09-25 03:03:24

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, guys, LTNS.

I got so long that about 2 months not follow this topic. And looks like it still powerful.

Now I'm trying New Linux 3.17.0-rc6 with new qemu-git version. And try to solve hangs while guest Linux rebooting. (Linux shutdown, but VM can't reboot. Strange is, it not happened while guest OS is Windows.)
Unfortunately it still doesn't work. For now I have to shutdown VM and reboot Host if guest Linux need to reboot.

Does anyone got solutions?

Offline

#2806 2014-09-25 06:04:24

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

http://vfio.blogspot.com/2014/08/whats- … ation.html

If you're not affected by i915's VGA arbiter behavior and don't care for libvirt support, then just keep doing what you're doing.

Well, I do use libvirt and virt-manager. Are you saying with OVMF I could setup the VM using libvirt without needing to pass additional arguments to the qemu-commandline (stuff like "<qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>")? That would be awesome!

Edit: Just found your blog entry http://vfio.blogspot.de/2014/08/primary … t-vga.html - Now this really looks interesting! Thanks smile

Last edited by TripleSpeeder (2014-09-25 08:29:19)

Offline

#2807 2014-09-27 09:03:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Hello, guys, LTNS.

I got so long that about 2 months not follow this topic. And looks like it still powerful.

Now I'm trying New Linux 3.17.0-rc6 with new qemu-git version. And try to solve hangs while guest Linux rebooting. (Linux shutdown, but VM can't reboot. Strange is, it not happened while guest OS is Windows.)
Unfortunately it still doesn't work. For now I have to shutdown VM and reboot Host if guest Linux need to reboot.

Does anyone got solutions?

Seems like your VGA doesn't have reset capabilities, or it is working not quite right, and linux can't reset it.
You can try:
doing pm-suspend and then wake the system - it'll be faster than a host reboot, while resetting the GPU hardware;
passing through only the VGA itself, without HDMI Audio(BDF: XX:00.1);
probably getting OVMF working would help, but i doubt it.

Some cards are broken - see https://bbs.archlinux.org/viewtopic.php … 6#p1459916 , some cards work like a charm, depends on the vendor(XFX?) and model.



ru?..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2808 2014-09-27 12:47:33

reper
Member
Registered: 2013-07-26
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi guys,
i was thinking of making a new pc with vga passthrough in mind, and i have some questions:
1) are the Asrock B85M Pro4 and the i5 4670 a good combination? if i'm not wrong they both support VT-D right? Also, i've never bought an asrock, are they good?
2) has someone had success passing one of the new nvidia's card?

i'm still a noob regarding virtualization and i fear to make a mistake.

Offline

#2809 2014-09-27 23:31:32

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone , first post!

This woks nicely on my ASUS X99-Deluxe + i7-5930k , passing through 1 nVidia GT610 + 1 GTX770 to two seperate VMs . You wouldn't need to patch the kernel , unless you want to passthrough the SATA controller OR the Wireless+Bluetooth controller .

Anyway , this board's back panel's USB ports are connected to a single USB 3.0 controller , so using PCI-Passthrough for USB isn't possible in my case . However , I have USB Hubs I want to use for each VM .

Is it possible to passthrough the Hub alone and thus every USB device that gets connected to it ?

Thank you .

EDIT :

lspci | grep USB :

00:14.0 USB controller: Intel Corporation Wellsburg USB xHCI Host Controller (rev 05) 00:1a.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host Controller #2 (rev 05) 00:1d.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host Controller #1 (rev 05) 0e:00.0 USB controller: ASMedia Technology Inc. Device 1142

I tried passthing-through every controller (one at a time) to a VM , they don't work . However when passing the 00:14.0 controller , all the USB ports in the back get passed-through .

Last edited by Denso (2014-09-27 23:36:01)

Offline

#2810 2014-09-28 06:04:04

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have had similar situation and had to pass

00:1a.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host Controller #2 (rev 05)
or
00:1d.0 USB controller: Intel Corporation Wellsburg USB Enhanced Host Controller #1 (rev 05)

that are connected to front usb on mbo, only that worked. Passing hubs isn't possible i think.

Offline

#2811 2014-09-28 06:29:47

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow , that makes perfect sense ...

This mobo has 2 front USB 3.0 headers , I suppose 00:1a.0 is for one and 00:1d.0 is for the other ?

Please do clarify ...

Thank you in advance !

Offline

#2812 2014-09-28 07:34:39

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
AKSN74 wrote:

Hello, guys, LTNS.

I got so long that about 2 months not follow this topic. And looks like it still powerful.

Now I'm trying New Linux 3.17.0-rc6 with new qemu-git version. And try to solve hangs while guest Linux rebooting. (Linux shutdown, but VM can't reboot. Strange is, it not happened while guest OS is Windows.)
Unfortunately it still doesn't work. For now I have to shutdown VM and reboot Host if guest Linux need to reboot.

Does anyone got solutions?

Seems like your VGA doesn't have reset capabilities, or it is working not quite right, and linux can't reset it.
You can try:
doing pm-suspend and then wake the system - it'll be faster than a host reboot, while resetting the GPU hardware;
passing through only the VGA itself, without HDMI Audio(BDF: XX:00.1);
probably getting OVMF working would help, but i doubt it.

Some cards are broken - see https://bbs.archlinux.org/viewtopic.php … 6#p1459916 , some cards work like a charm, depends on the vendor(XFX?) and model.



ru?..

Thanks for your reply.

Because I'm using NVIDIA 'official' GTX480 (that means vendor is NVIDIA, not others like XFX, ASUS, etc.)
And unfortunately, it will get error if passing through VGA only.

Besides, GTX480 is not using UEFI BIOS, so I guess OVMF is not working.
But I'll try again.

Offline

#2813 2014-09-28 07:41:57

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Wow , that makes perfect sense ...

This mobo has 2 front USB 3.0 headers , I suppose 00:1a.0 is for one and 00:1d.0 is for the other ?

Please do clarify ...

Thank you in advance !

well i don't have that mbo i got z77, but that should be correct... u need to try it  smile
(u may have 2.0 front and/or 3.0 it depends on mbo)

Last edited by slis (2014-09-28 07:57:24)

Offline

#2814 2014-09-28 16:35:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Because I'm using NVIDIA 'official' GTX480 (that means vendor is NVIDIA, not others like XFX, ASUS, etc.)
And unfortunately, it will get error if passing through VGA only.

Besides, GTX480 is not using UEFI BIOS, so I guess OVMF is not working.
But I'll try again.

Reference GTX480...
Well, i can't tell you anything more helpful.
You can do pm-suspend-wake-cycle every reboot as a workaround, i guess. Or you could experiment with guest OS - you've said you have linux as a guest.
Trying to poke nvidia support is useless, they'll say "go eat your KVM yourself" just like they did it in their latest drivers.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2815 2014-09-29 03:27:52

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
Denso wrote:

Wow , that makes perfect sense ...

This mobo has 2 front USB 3.0 headers , I suppose 00:1a.0 is for one and 00:1d.0 is for the other ?

Please do clarify ...

Thank you in advance !

well i don't have that mbo i got z77, but that should be correct... u need to try it  smile
(u may have 2.0 front and/or 3.0 it depends on mbo)

Hi .

I can confirm that the 2 USB 3.0 headers are also passed-through with the main controller (00:14.0) .

So I think I might need a PCI-E USB 3.0 card for this .

I hope there would be a way to passthrough individual hubs instead of controllers . sad

Offline

#2816 2014-09-29 03:57:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

what about 2.0? do u have 2.0 headers on mbo? try them?

Offline

#2817 2014-09-29 04:10:06

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

what about 2.0? do u have 2.0 headers on mbo? try them?

Oops ! I ignored them completely !

Will be back with results soon !

Offline

#2818 2014-09-29 04:41:36

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Here's something odd I've just encountered:
I have two motherboards: a SuperMicro X10SL7-F, and a SuperMicro X10SAT.
I  have two video cards I can pass through: an AMD Radeon R9 290, and an AMD Radeon E6760.

On the X1SL7-F, I've passed through the E6760 just fine -- no DMAR errors, and no bus-reset issues.
On the X10SAT, I've passed through the R9 290, and it works, aside from the annoying bus-reset issues.

I just replaced the R9 290 with the E6760, and started getting DMAR errors for both the GPU and the XHCI controller (one I've successfully passed through before).

I managed to get the errors to stop by detaching the HDMI audio and plugging a monitor into the second card.  (I'm not certain which of those two changes actually fixed it.)

I wonder if the card's UEFI firmware freaks out when there's no monitor attached?

EDIT: wait, maybe it's not fixed.  More experimentation to follow...

Last edited by DanaGoyette (2014-09-29 04:45:44)

Offline

#2819 2014-09-29 05:20:20

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

Here's something odd I've just encountered:
I have two motherboards: a SuperMicro X10SL7-F, and a SuperMicro X10SAT.
I  have two video cards I can pass through: an AMD Radeon R9 290, and an AMD Radeon E6760.

On the X1SL7-F, I've passed through the E6760 just fine -- no DMAR errors, and no bus-reset issues.
On the X10SAT, I've passed through the R9 290, and it works, aside from the annoying bus-reset issues.

I just replaced the R9 290 with the E6760, and started getting DMAR errors for both the GPU and the XHCI controller (one I've successfully passed through before).

I managed to get the errors to stop by detaching the HDMI audio and plugging a monitor into the second card.  (I'm not certain which of those two changes actually fixed it.)

I wonder if the card's UEFI firmware freaks out when there's no monitor attached?

EDIT: wait, maybe it's not fixed.  More experimentation to follow...

Hey .

I can confirm that I get a LOT of DMAR errors on my X99-Deluxe too when using VFIO . I doesn't happen when I use "-device kvm-pci-assign" for onboard audio and USB though . Also , when I use this method , I can't reboot Windows without rebooting the whole host , otherwise it crashes the host .

Last edited by Denso (2014-09-29 05:20:36)

Offline

#2820 2014-09-29 05:26:02

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Here are the errors I'm getting, approximately one per OVMF screen redraw (being driven via serial interface in virt-manager):

[ 330.024356] xhci_hcd 0000:08:00.0: remove, state 4 [ 330.026932] usb usb4: USB disconnect, device number 1 [ 330.037696] xhci_hcd 0000:08:00.0: USB bus 4 deregistered [ 330.041890] xhci_hcd 0000:08:00.0: remove, state 1 [ 330.044007] usb usb3: USB disconnect, device number 1 [ 330.046225] usb 3-2: USB disconnect, device number 3 [ 330.076301] xhci_hcd 0000:08:00.0: USB bus 3 deregistered [ 330.908248] vfio-pci 0000:01:00.0: enabling device (0140 -> 0143) [ 335.721419] dmar: DRHD: handling fault status reg 3 [ 335.723856] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be46c000 [ 335.723856] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.181491] dmar: DRHD: handling fault status reg 3 [ 336.183706] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be448000 [ 336.183706] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.189725] dmar: DRHD: handling fault status reg 3 [ 336.191928] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be448000 [ 336.191928] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.198007] dmar: DRHD: handling fault status reg 3 [ 336.200199] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be437000 [ 336.200199] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.206293] dmar: DRHD: handling fault status reg 3 [ 336.208468] dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr bff66000 [ 336.208468] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.360197] dmar: DRHD: handling fault status reg 3 [ 336.361816] dmar: DMAR:[DMA Write] Request device [01:00.0] fault addr bff66000 [ 336.361816] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.367985] dmar: DRHD: handling fault status reg 3 [ 336.370762] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be436000 [ 336.370762] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.376827] dmar: DRHD: handling fault status reg 3 [ 336.378948] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be436000 [ 336.378948] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 337.518236] dmar: DRHD: handling fault status reg 3 [ 337.520281] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be46c000 [ 337.520281] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 338.346795] dmar: DRHD: handling fault status reg 3 [ 338.349327] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be46c000 [ 338.349327] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 338.960278] dmar: DRHD: handling fault status reg 3 [ 338.962519] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be27a000 [ 338.962519] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 338.968614] dmar: DRHD: handling fault status reg 3 [ 338.970776] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be27a000 [ 338.970776] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 339.097760] dmar: DRHD: handling fault status reg 3 [ 339.100430] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be46c000 [ 339.100430] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 339.715753] dmar: DRHD: handling fault status reg 3 [ 339.719830] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be46c000 [ 339.719830] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.748928] dmar: DRHD: handling fault status reg 3 [ 340.751168] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.751168] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.757762] dmar: DRHD: handling fault status reg 3 [ 340.760452] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.760452] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.766677] dmar: DRHD: handling fault status reg 3 [ 340.768720] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.768720] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.775453] dmar: DRHD: handling fault status reg 3 [ 340.777950] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.777950] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.784387] dmar: DRHD: handling fault status reg 3 [ 340.787213] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.787213] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.798013] dmar: DRHD: handling fault status reg 3 [ 340.801504] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0ec000 [ 340.801504] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.807641] dmar: DRHD: handling fault status reg 3 [ 340.809757] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0ec000 [ 340.809757] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.816206] dmar: DRHD: handling fault status reg 3 [ 340.818295] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.818295] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.825173] dmar: DRHD: handling fault status reg 3 [ 340.827588] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0df000 [ 340.827588] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.833662] dmar: DRHD: handling fault status reg 3 [ 340.835817] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be3c5000 [ 340.835817] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.842135] dmar: DRHD: handling fault status reg 3 [ 340.845125] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0dd000 [ 340.845125] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.851955] dmar: DRHD: handling fault status reg 3 [ 340.854387] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0dd000 [ 340.854387] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.860749] dmar: DRHD: handling fault status reg 3 [ 340.863658] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.863658] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.869889] dmar: DRHD: handling fault status reg 3 [ 340.871905] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f6000 [ 340.871905] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.878694] dmar: DRHD: handling fault status reg 3 [ 340.881421] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f2000 [ 340.881421] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.887509] dmar: DRHD: handling fault status reg 3 [ 340.889685] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.889685] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.896034] dmar: DRHD: handling fault status reg 3 [ 340.898945] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.898945] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.905864] dmar: DRHD: handling fault status reg 3 [ 340.907925] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.907925] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.914463] dmar: DRHD: handling fault status reg 3 [ 340.916103] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.916103] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.922333] dmar: DRHD: handling fault status reg 3 [ 340.924267] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.924267] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.930355] dmar: DRHD: handling fault status reg 3 [ 340.932667] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.932667] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.938511] dmar: DRHD: handling fault status reg 3 [ 340.938516] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.938516] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.938525] dmar: DRHD: handling fault status reg 3 [ 340.938531] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.938531] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.938641] dmar: DRHD: handling fault status reg 3 [ 340.938648] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.938648] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.938664] dmar: DRHD: handling fault status reg 3 [ 340.938672] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.938672] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.941760] dmar: DRHD: handling fault status reg 3 [ 340.941773] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.941773] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.941778] dmar: DRHD: handling fault status reg 3 [ 340.941787] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.941787] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.941794] dmar: DRHD: handling fault status reg 3 [ 340.941799] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.941799] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.941815] dmar: DRHD: handling fault status reg 3 [ 340.941821] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.941821] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.941832] dmar: DRHD: handling fault status reg 3 [ 340.941835] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.941835] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.941847] dmar: DRHD: handling fault status reg 3 [ 340.941861] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.941861] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.947580] dmar: DRHD: handling fault status reg 3 [ 340.947583] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.947583] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.947599] dmar: DRHD: handling fault status reg 3 [ 340.947604] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.947604] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.947697] dmar: DRHD: handling fault status reg 3 [ 340.947704] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.947704] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.947713] dmar: DRHD: handling fault status reg 3 [ 340.947719] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.947719] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.947727] dmar: DRHD: handling fault status reg 3 [ 340.947734] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.947734] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.949772] dmar: DRHD: handling fault status reg 3 [ 340.949774] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.949774] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.950856] dmar: DRHD: handling fault status reg 3 [ 340.950863] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.950863] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.950875] dmar: DRHD: handling fault status reg 3 [ 340.950880] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.950880] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.950889] dmar: DRHD: handling fault status reg 3 [ 340.950893] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.950893] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.955757] dmar: DRHD: handling fault status reg 3 [ 340.955759] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.955759] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.956078] dmar: DRHD: handling fault status reg 3 [ 340.956086] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.956086] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.957893] dmar: DRHD: handling fault status reg 3 [ 340.957894] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.957894] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958259] dmar: DRHD: handling fault status reg 3 [ 340.958266] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.958266] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958280] dmar: DRHD: handling fault status reg 3 [ 340.958288] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.958288] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958297] dmar: DRHD: handling fault status reg 3 [ 340.958302] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.958302] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958311] dmar: DRHD: handling fault status reg 3 [ 340.958317] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.958317] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958329] dmar: DRHD: handling fault status reg 3 [ 340.958335] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.958335] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958342] dmar: DRHD: handling fault status reg 3 [ 340.958348] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.958348] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958357] dmar: DRHD: handling fault status reg 3 [ 340.958363] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958363] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958374] dmar: DRHD: handling fault status reg 3 [ 340.958379] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958379] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958389] dmar: DRHD: handling fault status reg 3 [ 340.958396] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958396] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958406] dmar: DRHD: handling fault status reg 3 [ 340.958411] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958411] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958419] dmar: DRHD: handling fault status reg 3 [ 340.958425] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958425] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958437] dmar: DRHD: handling fault status reg 3 [ 340.958443] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958443] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958450] dmar: DRHD: handling fault status reg 3 [ 340.958456] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.958456] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958466] dmar: DRHD: handling fault status reg 3 [ 340.958472] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.958472] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958483] dmar: DRHD: handling fault status reg 3 [ 340.958487] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.958487] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958498] dmar: DRHD: handling fault status reg 3 [ 340.958505] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.958505] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958512] dmar: DRHD: handling fault status reg 3 [ 340.958518] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.958518] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958529] dmar: DRHD: handling fault status reg 3 [ 340.958536] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.958536] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958630] dmar: DRHD: handling fault status reg 3 [ 340.958637] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.958637] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.958646] dmar: DRHD: handling fault status reg 3 [ 340.958651] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.958651] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.963963] dmar: DRHD: handling fault status reg 3 [ 340.963965] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.963965] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.966049] dmar: DRHD: handling fault status reg 3 [ 340.966051] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.966051] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.966266] dmar: DRHD: handling fault status reg 3 [ 340.966273] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.966273] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972292] dmar: DRHD: handling fault status reg 3 [ 340.972296] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.972296] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972311] dmar: DRHD: handling fault status reg 3 [ 340.972317] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972317] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972323] dmar: DRHD: handling fault status reg 3 [ 340.972330] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972330] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972339] dmar: DRHD: handling fault status reg 3 [ 340.972345] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972345] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972356] dmar: DRHD: handling fault status reg 3 [ 340.972363] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972363] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972370] dmar: DRHD: handling fault status reg 3 [ 340.972378] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972378] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972387] dmar: DRHD: handling fault status reg 3 [ 340.972391] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972391] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972402] dmar: DRHD: handling fault status reg 3 [ 340.972407] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972407] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972420] dmar: DRHD: handling fault status reg 3 [ 340.972425] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.972425] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972435] dmar: DRHD: handling fault status reg 3 [ 340.972440] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.972440] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972451] dmar: DRHD: handling fault status reg 3 [ 340.972463] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.972463] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.972469] dmar: DRHD: handling fault status reg 3 [ 340.972473] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.972473] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.974249] dmar: DRHD: handling fault status reg 3 [ 340.974250] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.974250] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980331] dmar: DRHD: handling fault status reg 3 [ 340.980335] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.980335] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980352] dmar: DRHD: handling fault status reg 3 [ 340.980359] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980359] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980366] dmar: DRHD: handling fault status reg 3 [ 340.980370] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980370] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980381] dmar: DRHD: handling fault status reg 3 [ 340.980386] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980386] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980395] dmar: DRHD: handling fault status reg 3 [ 340.980401] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980401] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980411] dmar: DRHD: handling fault status reg 3 [ 340.980416] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980416] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980425] dmar: DRHD: handling fault status reg 3 [ 340.980430] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980430] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980441] dmar: DRHD: handling fault status reg 3 [ 340.980446] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980446] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980456] dmar: DRHD: handling fault status reg 3 [ 340.980461] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980461] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980472] dmar: DRHD: handling fault status reg 3 [ 340.980476] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980476] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980488] dmar: DRHD: handling fault status reg 3 [ 340.980492] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980492] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980504] dmar: DRHD: handling fault status reg 3 [ 340.980510] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980510] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980518] dmar: DRHD: handling fault status reg 3 [ 340.980524] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 340.980524] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980892] dmar: DRHD: handling fault status reg 3 [ 340.980897] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980897] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980906] dmar: DRHD: handling fault status reg 3 [ 340.980912] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980912] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980923] dmar: DRHD: handling fault status reg 3 [ 340.980929] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980929] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980939] dmar: DRHD: handling fault status reg 3 [ 340.980945] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980945] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980956] dmar: DRHD: handling fault status reg 3 [ 340.980961] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980961] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980972] dmar: DRHD: handling fault status reg 3 [ 340.980977] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980977] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.980988] dmar: DRHD: handling fault status reg 3 [ 340.980994] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.980994] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981005] dmar: DRHD: handling fault status reg 3 [ 340.981011] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981011] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981021] dmar: DRHD: handling fault status reg 3 [ 340.981026] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981026] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981035] dmar: DRHD: handling fault status reg 3 [ 340.981042] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981042] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981052] dmar: DRHD: handling fault status reg 3 [ 340.981058] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981058] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981068] dmar: DRHD: handling fault status reg 3 [ 340.981074] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981074] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981085] dmar: DRHD: handling fault status reg 3 [ 340.981090] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981090] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981099] dmar: DRHD: handling fault status reg 3 [ 340.981105] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981105] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981115] dmar: DRHD: handling fault status reg 3 [ 340.981120] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.981120] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981129] dmar: DRHD: handling fault status reg 3 [ 340.981136] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.981136] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981146] dmar: DRHD: handling fault status reg 3 [ 340.981151] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.981151] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981160] dmar: DRHD: handling fault status reg 3 [ 340.981166] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.981166] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981177] dmar: DRHD: handling fault status reg 3 [ 340.981182] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 340.981182] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981272] dmar: DRHD: handling fault status reg 3 [ 340.981279] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981279] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981286] dmar: DRHD: handling fault status reg 3 [ 340.981293] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981293] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981302] dmar: DRHD: handling fault status reg 3 [ 340.981309] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981309] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981318] dmar: DRHD: handling fault status reg 3 [ 340.981323] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981323] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981333] dmar: DRHD: handling fault status reg 3 [ 340.981339] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981339] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981349] dmar: DRHD: handling fault status reg 3 [ 340.981354] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981354] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981363] dmar: DRHD: handling fault status reg 3 [ 340.981370] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981370] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981378] dmar: DRHD: handling fault status reg 3 [ 340.981385] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981385] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981395] dmar: DRHD: handling fault status reg 3 [ 340.981400] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981400] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981410] dmar: DRHD: handling fault status reg 3 [ 340.981416] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981416] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981426] dmar: DRHD: handling fault status reg 3 [ 340.981431] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981431] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981441] dmar: DRHD: handling fault status reg 3 [ 340.981447] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981447] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981456] dmar: DRHD: handling fault status reg 3 [ 340.981462] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981462] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981472] dmar: DRHD: handling fault status reg 3 [ 340.981478] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981478] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981491] dmar: DRHD: handling fault status reg 3 [ 340.981497] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981497] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981504] dmar: DRHD: handling fault status reg 3 [ 340.981509] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981509] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981519] dmar: DRHD: handling fault status reg 3 [ 340.981525] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981525] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981536] dmar: DRHD: handling fault status reg 3 [ 340.981542] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981542] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981552] dmar: DRHD: handling fault status reg 3 [ 340.981558] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fc000 [ 340.981558] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981784] dmar: DRHD: handling fault status reg 3 [ 340.981790] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981790] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981797] dmar: DRHD: handling fault status reg 3 [ 340.981803] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981803] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981812] dmar: DRHD: handling fault status reg 3 [ 340.981818] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981818] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981830] dmar: DRHD: handling fault status reg 3 [ 340.981835] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981835] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981844] dmar: DRHD: handling fault status reg 3 [ 340.981851] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981851] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981857] dmar: DRHD: handling fault status reg 3 [ 340.981862] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981862] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981873] dmar: DRHD: handling fault status reg 3 [ 340.981879] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981879] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981886] dmar: DRHD: handling fault status reg 3 [ 340.981892] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981892] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981903] dmar: DRHD: handling fault status reg 3 [ 340.981908] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981908] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981919] dmar: DRHD: handling fault status reg 3 [ 340.981925] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981925] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981933] dmar: DRHD: handling fault status reg 3 [ 340.981941] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981941] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981948] dmar: DRHD: handling fault status reg 3 [ 340.981954] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981954] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981965] dmar: DRHD: handling fault status reg 3 [ 340.981970] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981970] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981980] dmar: DRHD: handling fault status reg 3 [ 340.981986] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981986] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.981993] dmar: DRHD: handling fault status reg 3 [ 340.981999] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.981999] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982008] dmar: DRHD: handling fault status reg 3 [ 340.982014] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.982014] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982025] dmar: DRHD: handling fault status reg 3 [ 340.982031] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.982031] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982039] dmar: DRHD: handling fault status reg 3 [ 340.982045] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.982045] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982054] dmar: DRHD: handling fault status reg 3 [ 340.982061] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 340.982061] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982274] dmar: DRHD: handling fault status reg 3 [ 340.982280] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982280] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982290] dmar: DRHD: handling fault status reg 3 [ 340.982295] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982295] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982304] dmar: DRHD: handling fault status reg 3 [ 340.982312] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982312] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982319] dmar: DRHD: handling fault status reg 3 [ 340.982323] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982323] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982334] dmar: DRHD: handling fault status reg 3 [ 340.982338] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982338] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982350] dmar: DRHD: handling fault status reg 3 [ 340.982356] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982356] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982363] dmar: DRHD: handling fault status reg 3 [ 340.982370] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982370] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982636] dmar: DRHD: handling fault status reg 3 [ 340.982637] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.982637] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.982944] dmar: DRHD: handling fault status reg 3 [ 340.982950] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.982950] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.988472] dmar: DRHD: handling fault status reg 3 [ 340.988473] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.988473] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.988569] dmar: DRHD: handling fault status reg 3 [ 340.988575] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.988575] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.988584] dmar: DRHD: handling fault status reg 3 [ 340.988590] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.988590] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.990633] dmar: DRHD: handling fault status reg 3 [ 340.990634] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.990634] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.990811] dmar: DRHD: handling fault status reg 3 [ 340.990817] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.990817] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.990825] dmar: DRHD: handling fault status reg 3 [ 340.990831] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.990831] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.990841] dmar: DRHD: handling fault status reg 3 [ 340.990846] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.990846] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.990856] dmar: DRHD: handling fault status reg 3 [ 340.990862] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.990862] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.990872] dmar: DRHD: handling fault status reg 3 [ 340.990878] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.990878] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.996848] dmar: DRHD: handling fault status reg 3 [ 340.996850] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.996850] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.997144] dmar: DRHD: handling fault status reg 3 [ 340.997149] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.997149] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.997160] dmar: DRHD: handling fault status reg 3 [ 340.997165] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.997165] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999666] dmar: DRHD: handling fault status reg 3 [ 340.999667] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 340.999667] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999898] dmar: DRHD: handling fault status reg 3 [ 340.999913] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.999913] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999931] dmar: DRHD: handling fault status reg 3 [ 340.999936] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.999936] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999948] dmar: DRHD: handling fault status reg 3 [ 340.999954] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.999954] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999961] dmar: DRHD: handling fault status reg 3 [ 340.999967] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.999967] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999976] dmar: DRHD: handling fault status reg 3 [ 340.999981] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.999981] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 340.999992] dmar: DRHD: handling fault status reg 3 [ 340.999997] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 340.999997] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000006] dmar: DRHD: handling fault status reg 3 [ 341.000014] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.000014] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000021] dmar: DRHD: handling fault status reg 3 [ 341.000027] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.000027] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000038] dmar: DRHD: handling fault status reg 3 [ 341.000042] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.000042] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000055] dmar: DRHD: handling fault status reg 3 [ 341.000060] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.000060] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000067] dmar: DRHD: handling fault status reg 3 [ 341.000075] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.000075] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000082] dmar: DRHD: handling fault status reg 3 [ 341.000088] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.000088] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000100] dmar: DRHD: handling fault status reg 3 [ 341.000106] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.000106] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000114] dmar: DRHD: handling fault status reg 3 [ 341.000121] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.000121] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000130] dmar: DRHD: handling fault status reg 3 [ 341.000136] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.000136] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000144] dmar: DRHD: handling fault status reg 3 [ 341.000150] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.000150] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000164] dmar: DRHD: handling fault status reg 3 [ 341.000169] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.000169] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000177] dmar: DRHD: handling fault status reg 3 [ 341.000184] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.000184] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000271] dmar: DRHD: handling fault status reg 3 [ 341.000276] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 341.000276] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.000284] dmar: DRHD: handling fault status reg 3 [ 341.000290] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 341.000290] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.005726] dmar: DRHD: handling fault status reg 3 [ 341.005727] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 341.005727] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.005836] dmar: DRHD: handling fault status reg 3 [ 341.005840] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.005840] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.007839] dmar: DRHD: handling fault status reg 3 [ 341.007840] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.007840] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.007963] dmar: DRHD: handling fault status reg 3 [ 341.007968] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.007968] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.013906] dmar: DRHD: handling fault status reg 3 [ 341.013907] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.013907] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.016017] dmar: DRHD: handling fault status reg 3 [ 341.016019] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.016019] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.022259] dmar: DRHD: handling fault status reg 3 [ 341.022263] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.022263] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.022277] dmar: DRHD: handling fault status reg 3 [ 341.022282] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.022282] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.022289] dmar: DRHD: handling fault status reg 3 [ 341.022294] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.022294] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.022305] dmar: DRHD: handling fault status reg 3 [ 341.022310] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.022310] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.022320] dmar: DRHD: handling fault status reg 3 [ 341.022335] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.022335] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.022363] dmar: DRHD: handling fault status reg 3 [ 341.022370] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.022370] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.024202] dmar: DRHD: handling fault status reg 3 [ 341.024205] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 341.024205] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.024211] dmar: DRHD: handling fault status reg 3 [ 341.024218] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.024218] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.024225] dmar: DRHD: handling fault status reg 3 [ 341.024229] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.024229] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.024241] dmar: DRHD: handling fault status reg 3 [ 341.024247] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.024247] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.024255] dmar: DRHD: handling fault status reg 3 [ 341.024260] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.024260] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030273] dmar: DRHD: handling fault status reg 3 [ 341.030277] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f8000 [ 341.030277] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030290] dmar: DRHD: handling fault status reg 3 [ 341.030296] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 341.030296] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030303] dmar: DRHD: handling fault status reg 3 [ 341.030310] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.030310] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030320] dmar: DRHD: handling fault status reg 3 [ 341.030327] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.030327] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030336] dmar: DRHD: handling fault status reg 3 [ 341.030341] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.030341] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030351] dmar: DRHD: handling fault status reg 3 [ 341.030356] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.030356] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030369] dmar: DRHD: handling fault status reg 3 [ 341.030375] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.030375] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030381] dmar: DRHD: handling fault status reg 3 [ 341.030388] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fa000 [ 341.030388] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030397] dmar: DRHD: handling fault status reg 3 [ 341.030403] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.030403] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030415] dmar: DRHD: handling fault status reg 3 [ 341.030421] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.030421] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030429] dmar: DRHD: handling fault status reg 3 [ 341.030437] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.030437] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030444] dmar: DRHD: handling fault status reg 3 [ 341.030450] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.030450] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030459] dmar: DRHD: handling fault status reg 3 [ 341.030466] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.030466] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030476] dmar: DRHD: handling fault status reg 3 [ 341.030482] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0fb000 [ 341.030482] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030564] dmar: DRHD: handling fault status reg 3 [ 341.030570] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 341.030570] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.030578] dmar: DRHD: handling fault status reg 3 [ 341.030586] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 341.030586] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.032625] dmar: DRHD: handling fault status reg 3 [ 341.032627] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be274000 [ 341.032627] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.032734] dmar: DRHD: handling fault status reg 3 [ 341.032741] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 341.032741] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.032749] dmar: DRHD: handling fault status reg 3 [ 341.032753] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 341.032753] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.038446] dmar: DRHD: handling fault status reg 3 [ 341.038460] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 341.038460] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 341.040594] dmar: DRHD: handling fault status reg 3 [ 341.040595] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be0f9000 [ 341.040595] DMAR:[fault reason 12] non-zero reserved fields in PTE

Offline

#2821 2014-09-29 05:41:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

Here are the errors I'm getting, approximately one per OVMF screen redraw (being driven via serial interface in virt-manager):

[ 330.024356] xhci_hcd 0000:08:00.0: remove, state 4 [ 330.026932] usb usb4: USB disconnect, device number 1 [ 330.037696] xhci_hcd 0000:08:00.0: USB bus 4 deregistered [ 330.041890] xhci_hcd 0000:08:00.0: remove, state 1 [ 330.044007] usb usb3: USB disconnect, device number 1 [ 330.046225] usb 3-2: USB disconnect, device number 3 [ 330.076301] xhci_hcd 0000:08:00.0: USB bus 3 deregistered [ 330.908248] vfio-pci 0000:01:00.0: enabling device (0140 -> 0143) [ 335.721419] dmar: DRHD: handling fault status reg 3 [ 335.723856] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be46c000 [ 335.723856] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.181491] dmar: DRHD: handling fault status reg 3 [ 336.183706] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be448000 [ 336.183706] DMAR:[fault reason 12] non-zero reserved fields in PTE [ 336.189725] dmar: DRHD: handling fault status reg 3 [ 336.191928] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr be448000 [ 336.191928] DMAR:[fault reason 12] non-zero reserved fields in PTE

What host kernel is this?  I can only guess that it might be the IOMMU_CACHE attribute.  Kernel 3.15 includes support for multiple IOMMU domains within a VFIO container that should handle this situation.  What do you get from 'dmesg | grep ecap'?  If the hardware is lying about it's support for snoop control then then a patch like below should stop the error.

--- a/include/linux/intel-iommu.h +++ b/include/linux/intel-iommu.h @@ -126,7 +126,7 @@ static inline void dmar_writeq(void __iomem *addr, u64 val) #define ecap_ir_support(e) ((e >> 3) & 0x1) #define ecap_dev_iotlb_support(e) (((e) >> 2) & 0x1) #define ecap_max_handle_mask(e) ((e >> 20) & 0xf) -#define ecap_sc_support(e) ((e >> 7) & 0x1) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Snooping Control */ +#define ecap_sc_support(e) ((e >> 7) & 0x0) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Snooping Control */ /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup IOTLB_REG */ #define DMA_TLB_FLUSH_GRANU_OFFSET 60

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2822 2014-09-29 05:43:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I can confirm that I get a LOT of DMAR errors on my X99-Deluxe too when using VFIO . I doesn't happen when I use "-device kvm-pci-assign" for onboard audio and USB though . Also , when I use this method , I can't reboot Windows without rebooting the whole host , otherwise it crashes the host .

Are they the same errors as DanaGoyette reported?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2823 2014-09-29 05:51:00

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

I can confirm that I get a LOT of DMAR errors on my X99-Deluxe too when using VFIO . I doesn't happen when I use "-device kvm-pci-assign" for onboard audio and USB though . Also , when I use this method , I can't reboot Windows without rebooting the whole host , otherwise it crashes the host .

Are they the same errors as DanaGoyette reported?

Nope . They are completely different . I'll try to reproduce it now and paste them here .

EDIT :

dmar: DMAR:[DMA Read] Request device [00:14.0] fault addr 100068000 DMAR:[fault reason 06] PTE Read access is not set

Also , these go away when I use "-device kvm-pci-assign" to assign onboard audio and USB , they only appear when using VFIO + the host crashes after a while :

Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4be00 already set (to 22d5e8003 not 359200083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4c200 already set (to 22d5ea003 not 358e00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4c600 already set (to 22d5ec003 not 358a00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4ca00 already set (to 22d5ee003 not 358600083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4ce00 already set (to 22d5f0003 not 358200083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4d200 already set (to 22d5f2003 not 357e00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4d600 already set (to 22d5f4003 not 357a00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4da00 already set (to 22d5f6003 not 357600083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4de00 already set (to 22d5f8003 not 357200083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4e200 already set (to 22d5fa003 not 356e00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4e600 already set (to 22d5fc003 not 356a00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4ea00 already set (to 22d5fe003 not 356600083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4ee00 already set (to 22d600003 not 356200083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4f200 already set (to 22d602003 not 355e00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4f600 already set (to 22d604003 not 355a00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4fa00 already set (to 22d606003 not 355600083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x4fe00 already set (to 22d608003 not 355200083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x50200 already set (to 22d60a003 not 354e00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x50600 already set (to 22d60c003 not 354a00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x50a00 already set (to 22d60e003 not 354600083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x50e00 already set (to 22d610003 not 354200083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x51200 already set (to 22d612003 not 353e00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x51600 already set (to 22d614003 not 353a00083) Sep 29 00:35:28 srv1 kernel: ERROR: DMA PTE for vPFN 0x51a00 already set (to 22d616003 not 353600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x51e00 already set (to 22d618003 not 353200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x52200 already set (to 22d61a003 not 352e00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x52600 already set (to 22d61c003 not 352a00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x52a00 already set (to 22d61e003 not 352600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x52e00 already set (to 22d620003 not 352200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x53200 already set (to 22d622003 not 351e00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x53600 already set (to 22d624003 not 351a00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x53a00 already set (to 22d626003 not 351600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x53e00 already set (to 22d628003 not 351200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x54200 already set (to 22d62a003 not 350e00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x54600 already set (to 22d62c003 not 350a00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x54a00 already set (to 22d62e003 not 350600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x54e00 already set (to 22d630003 not 350200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x55200 already set (to 22d632003 not 34fe00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x55600 already set (to 22d634003 not 34fa00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x55a00 already set (to 22d636003 not 34f600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x55e00 already set (to 22d638003 not 34f200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x56200 already set (to 22d63a003 not 34ee00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x56600 already set (to 22d63c003 not 34ea00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x56a00 already set (to 22d63e003 not 34e600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x56e00 already set (to 22d640003 not 34e200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x57200 already set (to 22d642003 not 34de00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x57600 already set (to 22d644003 not 34da00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x57a00 already set (to 22d646003 not 34d600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x57e00 already set (to 22d648003 not 34d200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x58200 already set (to 22d64a003 not 34ce00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x58600 already set (to 22d64c003 not 34ca00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x58a00 already set (to 22d64e003 not 34c600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x58e00 already set (to 22d650003 not 34c200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x59200 already set (to 22d652003 not 34be00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x59600 already set (to 22d654003 not 34ba00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x59a00 already set (to 22d656003 not 34b600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x59e00 already set (to 22d658003 not 34b200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5a200 already set (to 22d65a003 not 34ae00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5a600 already set (to 22d65c003 not 34aa00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5aa00 already set (to 22d65e003 not 34a600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5ae00 already set (to 22d660003 not 34a200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5b200 already set (to 22d662003 not 349e00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5b600 already set (to 22d664003 not 349a00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5ba00 already set (to 22d666003 not 349600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5be00 already set (to 22d668003 not 349200083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5c200 already set (to 22d66a003 not 348e00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5c600 already set (to 22d66c003 not 348a00083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5ca00 already set (to 22d66e003 not 348600083) Sep 29 00:35:29 srv1 kernel: ERROR: DMA PTE for vPFN 0x5ce00 already set (to 22d670003 not 348200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5d200 already set (to 22d672003 not 347e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5d600 already set (to 22d674003 not 347a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5da00 already set (to 22d676003 not 347600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5de00 already set (to 22d678003 not 347200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5e200 already set (to 22d67a003 not 346e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5e600 already set (to 22d67c003 not 346a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5ea00 already set (to 22d67e003 not 346600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5ee00 already set (to 22d680003 not 346200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5f200 already set (to 22d682003 not 345e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5f600 already set (to 22d684003 not 345a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5fa00 already set (to 22d686003 not 345600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x5fe00 already set (to 22d688003 not 345200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x60200 already set (to 22d68a003 not 344e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x60600 already set (to 22d68c003 not 344a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x60a00 already set (to 22d68e003 not 344600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x60e00 already set (to 22d690003 not 344200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x61200 already set (to 22d692003 not 343e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x61600 already set (to 22d694003 not 343a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x61a00 already set (to 22d696003 not 343600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x61e00 already set (to 22d698003 not 343200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x62200 already set (to 22d69a003 not 342e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x62600 already set (to 22d69c003 not 342a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x62a00 already set (to 22d69e003 not 342600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x62e00 already set (to 22d6a0003 not 342200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x63200 already set (to 22d6a2003 not 341e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x63600 already set (to 22d6a4003 not 341a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x63a00 already set (to 22d6a6003 not 341600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x63e00 already set (to 22d6a8003 not 341200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x64200 already set (to 22d6aa003 not 340e00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x64600 already set (to 22d6ac003 not 340a00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x64a00 already set (to 22d6ae003 not 340600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x64e00 already set (to 22d6b0003 not 340200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x65200 already set (to 22d6b2003 not 33fe00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x65600 already set (to 22d6b4003 not 33fa00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x65a00 already set (to 22d6b6003 not 33f600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x65e00 already set (to 22d6b8003 not 33f200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x66200 already set (to 22d6ba003 not 33ee00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x66600 already set (to 22d6bc003 not 33ea00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x66a00 already set (to 22d6be003 not 33e600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x66e00 already set (to 22d6c0003 not 33e200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x67200 already set (to 22d6c2003 not 33de00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x67600 already set (to 22d6c4003 not 33da00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x67a00 already set (to 22d6c6003 not 33d600083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x67e00 already set (to 22d6c8003 not 33d200083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x68200 already set (to 22d6ca003 not 33ce00083) Sep 29 00:35:30 srv1 kernel: ERROR: DMA PTE for vPFN 0x68600 already set (to 22d6cc003 not 33ca00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x68a00 already set (to 22d6ce003 not 33c600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x68e00 already set (to 22d6d0003 not 33c200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x69200 already set (to 22d6d2003 not 33be00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x69600 already set (to 22d6d4003 not 33ba00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x69a00 already set (to 22d6d6003 not 33b600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x69e00 already set (to 22d6d8003 not 33b200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6a200 already set (to 22d6da003 not 33ae00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6a600 already set (to 22d6dc003 not 33aa00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6aa00 already set (to 22d6de003 not 33a600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6ae00 already set (to 22d6e0003 not 33a200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6b200 already set (to 22d6e2003 not 339e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6b600 already set (to 22d6e4003 not 339a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6ba00 already set (to 22d6e6003 not 339600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6be00 already set (to 22d6e8003 not 339200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6c200 already set (to 22d6ea003 not 338e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6c600 already set (to 22d6ec003 not 338a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6ca00 already set (to 22d6ee003 not 338600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6ce00 already set (to 22d6f0003 not 338200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6d200 already set (to 22d6f2003 not 337e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6d600 already set (to 22d6f4003 not 337a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6da00 already set (to 22d6f6003 not 337600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6de00 already set (to 22d6f8003 not 337200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6e200 already set (to 22d6fa003 not 336e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6e600 already set (to 22d6fc003 not 336a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6ea00 already set (to 22d6fe003 not 336600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6ee00 already set (to 22d700003 not 336200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6f200 already set (to 22d702003 not 335e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6f600 already set (to 22d704003 not 335a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6fa00 already set (to 22d706003 not 335600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x6fe00 already set (to 22d708003 not 335200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x70200 already set (to 22d70a003 not 334e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x70600 already set (to 22d70c003 not 334a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x70a00 already set (to 22d70e003 not 334600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x70e00 already set (to 22d710003 not 334200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x71200 already set (to 22d712003 not 333e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x71600 already set (to 22d714003 not 333a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x71a00 already set (to 22d716003 not 333600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x71e00 already set (to 22d718003 not 333200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x72200 already set (to 22d71a003 not 332e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x72600 already set (to 22d71c003 not 332a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x72a00 already set (to 22d71e003 not 332600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x72e00 already set (to 22d720003 not 332200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x73200 already set (to 22d722003 not 331e00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x73600 already set (to 22d724003 not 331a00083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x73a00 already set (to 22d726003 not 331600083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x73e00 already set (to 22d728003 not 331200083) Sep 29 00:35:31 srv1 kernel: ERROR: DMA PTE for vPFN 0x74200 already set (to 22d72a003 not 330e00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x74600 already set (to 22d72c003 not 330a00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x74a00 already set (to 22d72e003 not 330600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x74e00 already set (to 22d730003 not 330200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x75200 already set (to 22d732003 not 32fe00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x75600 already set (to 22d734003 not 32fa00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x75a00 already set (to 22d736003 not 32f600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x75e00 already set (to 22d738003 not 32f200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x76200 already set (to 22d73a003 not 32ee00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x76600 already set (to 22d73c003 not 32ea00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x76a00 already set (to 22d73e003 not 32e600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x76e00 already set (to 22d740003 not 32e200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x77200 already set (to 22d742003 not 32de00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x77600 already set (to 22d744003 not 32da00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x77a00 already set (to 22d746003 not 32d600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x77e00 already set (to 22d748003 not 32d200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x78200 already set (to 22d74a003 not 32ce00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x78600 already set (to 22d74c003 not 32ca00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x78a00 already set (to 22d74e003 not 32c600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x78e00 already set (to 22d750003 not 32c200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x79200 already set (to 22d752003 not 32be00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x79600 already set (to 22d754003 not 32ba00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x79a00 already set (to 22d756003 not 32b600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x79e00 already set (to 22d758003 not 32b200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7a200 already set (to 22d75a003 not 32ae00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7a600 already set (to 22d75c003 not 32aa00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7aa00 already set (to 22d75e003 not 32a600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7ae00 already set (to 22d760003 not 32a200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7b200 already set (to 22d762003 not 329e00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7b600 already set (to 22d764003 not 329a00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7ba00 already set (to 22d766003 not 329600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7be00 already set (to 22d768003 not 329200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7c200 already set (to 22d76a003 not 328e00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7c600 already set (to 22d76c003 not 328a00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7ca00 already set (to 22d76e003 not 328600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7ce00 already set (to 22d770003 not 328200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7d200 already set (to 22d772003 not 327e00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7d600 already set (to 22d774003 not 327a00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7da00 already set (to 22d776003 not 327600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7de00 already set (to 22d778003 not 327200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7e200 already set (to 22d77a003 not 326e00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7e600 already set (to 22d77c003 not 326a00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7ea00 already set (to 22d77e003 not 326600083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7ee00 already set (to 22d780003 not 326200083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7f200 already set (to 22d782003 not 325e00083) Sep 29 00:35:32 srv1 kernel: ERROR: DMA PTE for vPFN 0x7f600 already set (to 22d784003 not 325a00083) Sep 29 00:35:33 srv1 kernel: ERROR: DMA PTE for vPFN 0x7fa00 already set (to 22d786003 not 325600083) Sep 29 00:35:33 srv1 kernel: ERROR: DMA PTE for vPFN 0x7fe00 already set (to 22d788003 not 325200083)

EDIT 2 :

Just now this happened :

[17796.619734] pcieport 0000:00:01.0: AER: Corrected error received: id=0008 [17796.619746] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0008(Transmitter ID) [17796.619769] pcieport 0000:00:01.0: device [8086:2f02] error status/mask=00001000/00002000 [17796.619787] pcieport 0000:00:01.0: [12] Replay Timer Timeout [18601.941321] pcieport 0000:00:01.0: AER: Corrected error received: id=0008 [18601.941334] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0008(Transmitter ID) [18601.941358] pcieport 0000:00:01.0: device [8086:2f02] error status/mask=00001000/00002000 [18601.941376] pcieport 0000:00:01.0: [12] Replay Timer Timeout [19841.746240] pcieport 0000:00:01.0: AER: Corrected error received: id=0008 [19841.746251] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0008(Transmitter ID) [19841.746279] pcieport 0000:00:01.0: device [8086:2f02] error status/mask=00001000/00002000 [19841.746297] pcieport 0000:00:01.0: [12] Replay Timer Timeout
00:01.0 PCI bridge: Intel Corporation Haswell-E PCI Express Root Port 1 (rev 02)

Last edited by Denso (2014-09-29 07:53:17)

Offline

#2824 2014-09-29 06:23:27

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What host kernel is this?  I can only guess that it might be the IOMMU_CACHE attribute.  Kernel 3.15 includes support for multiple IOMMU domains within a VFIO container that should handle this situation.  What do you get from 'dmesg | grep ecap'?  If the hardware is lying about it's support for snoop control then then a patch like below should stop the error.

Kernel is Linux 3.16.1, with ACS override patch applied, but not enabled when using the X10SAT.
I used the ACS Override patch when testing on the X10SL7-F, because the LSI SAS controller is in a CPU-based PCIe port, and the X16 slot gets tied to it.  (I can't imagine an LSI SAS card ever directly talking to a video card...)

Lines that seem relevant:

[ 0.061450] dmar: Host address width 39 [ 0.061452] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.061460] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.061461] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.061466] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.061468] dmar: RMRR base: 0x0000006dea5000 end: 0x0000006deb3fff [ 0.061469] dmar: RMRR base: 0x0000006f000000 end: 0x0000007f1fffff [ 0.061534] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.061535] HPET id 0 under DRHD base 0xfed91000 [ 0.061536] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.061537] Your BIOS is broken and requested that x2apic be disabled. [ 0.061537] This will slightly decrease performance. [ 0.061537] Use 'intremap=no_x2apic_optout' to override BIOS request. [ 0.061649] Enabled IRQ remapping in xapic mode [ 0.061650] x2apic not enabled, IRQ remapping is in xapic mode [ 0.627093] DMAR: No ATSR found [ 0.627114] IOMMU 1 0xfed91000: using Queued invalidation [ 0.627116] IOMMU: Setting RMRR: [ 0.627126] IOMMU: Setting identity map for device 0000:00:14.0 [0x6dea5000 - 0x6deb3fff] [ 0.627145] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.627153] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.627166] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 393.579011] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)

Last edited by DanaGoyette (2014-09-29 06:36:06)

Offline

#2825 2014-09-29 16:56:46

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I get a compat_monitor0 screen similar to Slabity. However, I am using an intel IGP for the host, and an AMD 6990M for the Windows 7 guest.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0

I'm using the OP's linux-mainline package, and have also used the acs_override=downstream option with the same result.

Same result when running vga-clear vga-set.

dmesg | grep vgaarb

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.504322] vgaarb: loaded [ 0.504323] vgaarb: bridge control possible 0000:01:00.0 [ 0.504324] vgaarb: no bridge control possible 0000:00:02.0 [ 1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

Offline

#2826 2014-09-29 18:06:42

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About not being able to passthrough the onboard usb controllers, I also had that situation also with a gigabyte mobo, where apparently i could not pass any of the onboard usb controllers. None at all. What i found to be working was to disable the xhci controller entirelly, from bios. And then the passthrough was possible (for me), with either of the other remaining controllers, like a charm. Playing around with bios setup options for xhci and ehci paid off for me. Worth a try imo.

Last edited by noobman (2014-09-29 18:10:26)

Offline

#2827 2014-09-29 18:37:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:

dmesg | grep vgaarb

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.504322] vgaarb: loaded [ 0.504323] vgaarb: bridge control possible 0000:01:00.0 [ 0.504324] vgaarb: no bridge control possible 0000:00:02.0 [ 1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

Do you have host nvidia? If so - apply a patch.
http://www.mail-archive.com/qemu-devel@ … 74066.html
https://bpaste.net/show/109185


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2828 2014-09-29 19:59:07

et38
Member
Registered: 2014-09-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

About not being able to passthrough the onboard usb controllers, I also had that situation also with a gigabyte mobo, where apparently i could not pass any of the onboard usb controllers. None at all. What i found to be working was to disable the xhci controller entirelly, from bios. And then the passthrough was possible (for me), with either of the other remaining controllers, like a charm. Playing around with bios setup options for xhci and ehci paid off for me. Worth a try imo.

I was looking at the passtrough setup with ga-z97x-u3dh. If it's working, why is the motherboard not recommended in the list?

Offline

#2829 2014-09-29 20:14:13

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

If the hardware is lying about it's support for snoop control then then a patch like below should stop the error.

--- a/include/linux/intel-iommu.h +++ b/include/linux/intel-iommu.h @@ -126,7 +126,7 @@ static inline void dmar_writeq(void __iomem *addr, u64 val) #define ecap_ir_support(e) ((e >> 3) & 0x1) #define ecap_dev_iotlb_support(e) (((e) >> 2) & 0x1) #define ecap_max_handle_mask(e) ((e >> 20) & 0xf) -#define ecap_sc_support(e) ((e >> 7) & 0x1) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Snooping Control */ +#define ecap_sc_support(e) ((e >> 7) & 0x0) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Snooping Control */ /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup IOTLB_REG */ #define DMA_TLB_FLUSH_GRANU_OFFSET 60

I've applied this patch, and now the device works without errors.   Sweet!
For reference:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Whistler [Radeon E6760] [1002:6743] (prog-if 00 [VGA controller]) Subsystem: Hightech Information System Ltd. Device [1787:2323] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 51 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at eee20000 (64-bit, non-prefetchable) [size=128K] Region 4: I/O ports at e000 [size=256] Expansion ROM at eee00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00418 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: vfio-pci

EDIT: This may have also fixed passthrough of my ASMedia ASM1062 SATA controller, as well.

Last edited by DanaGoyette (2014-09-29 23:08:15)

Offline

#2830 2014-09-29 23:14:58

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Lauer wrote:

dmesg | grep vgaarb

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.504322] vgaarb: loaded [ 0.504323] vgaarb: bridge control possible 0000:01:00.0 [ 0.504324] vgaarb: no bridge control possible 0000:00:02.0 [ 1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

Do you have host nvidia? If so - apply a patch.
http://www.mail-archive.com/qemu-devel@ … 74066.html
https://bpaste.net/show/109185

Intel IGP and AMD 6990M.

Offline

#2831 2014-09-30 00:41:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
aw wrote:

If the hardware is lying about it's support for snoop control then then a patch like below should stop the error.

--- a/include/linux/intel-iommu.h +++ b/include/linux/intel-iommu.h @@ -126,7 +126,7 @@ static inline void dmar_writeq(void __iomem *addr, u64 val) #define ecap_ir_support(e) ((e >> 3) & 0x1) #define ecap_dev_iotlb_support(e) (((e) >> 2) & 0x1) #define ecap_max_handle_mask(e) ((e >> 20) & 0xf) -#define ecap_sc_support(e) ((e >> 7) & 0x1) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Snooping Control */ +#define ecap_sc_support(e) ((e >> 7) & 0x0) /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Snooping Control */ /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup IOTLB_REG */ #define DMA_TLB_FLUSH_GRANU_OFFSET 60

I've applied this patch, and now the device works without errors.   Sweet!

Please post the resulting /tmp/dmar.dsl file from:

$ sudo iasl -p /tmp/dmar -d /sys/firmware/acpi/tables/DMAR

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2832 2014-09-30 00:46:38

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Offline

#2833 2014-09-30 01:25:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2834 2014-09-30 01:27:45

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

Yes , 3.16.3 .

Offline

#2835 2014-09-30 01:37:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:
Denso wrote:

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

Yes , 3.16.3 .

Still seems potentially related to large pages.  Does using either of these options avoid it?

a) boot arg: intel_iommu=on,sp_off

b) module arg: vfio_iommu_type1 disable_hugepages=1


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2836 2014-09-30 02:24:11

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Please post the resulting /tmp/dmar.dsl file from:

$ sudo iasl -p /tmp/dmar -d /sys/firmware/acpi/tables/DMAR
/* 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Intel ACPI Component Architecture 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html AML Disassembler version 20140214-64 [Mar 29 2014] 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Copyright (c) 2000 - 2014 Intel Corporation 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Disassembly of /sys/firmware/acpi/tables/DMAR, Mon Sep 29 19:22:29 2014 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html ACPI Data Table [DMAR] 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Format: [HexOffset DecimalOffset ByteLength] FieldName : FieldValue */ [000h 0000 4] Signature : "DMAR" [DMA Remapping table] [004h 0004 4] Table Length : 000000A8 [008h 0008 1] Revision : 01 [009h 0009 1] Checksum : 57 [00Ah 0010 6] Oem ID : "INTEL " [010h 0016 8] Oem Table ID : "BDW " [018h 0024 4] Oem Revision : 00000001 [01Ch 0028 4] Asl Compiler ID : "INTL" [020h 0032 4] Asl Compiler Revision : 00000001 [024h 0036 1] Host Address Width : 26 [025h 0037 1] Flags : 03 [026h 0038 10] Reserved : 00 00 00 00 00 00 00 00 00 00 [030h 0048 2] Subtable Type : 0000 [Hardware Unit Definition] [032h 0050 2] Length : 0018 [034h 0052 1] Flags : 00 [035h 0053 1] Reserved : 00 [036h 0054 2] PCI Segment Number : 0000 [038h 0056 8] Register Base Address : 00000000FED90000 [040h 0064 1] Device Scope Entry Type : 01 [041h 0065 1] Entry Length : 08 [042h 0066 2] Reserved : 0000 [044h 0068 1] Enumeration ID : 00 [045h 0069 1] PCI Bus Number : 00 [046h 0070 2] PCI Path : 02,00 [048h 0072 2] Subtable Type : 0000 [Hardware Unit Definition] [04Ah 0074 2] Length : 0020 [04Ch 0076 1] Flags : 01 [04Dh 0077 1] Reserved : 00 [04Eh 0078 2] PCI Segment Number : 0000 [050h 0080 8] Register Base Address : 00000000FED91000 [058h 0088 1] Device Scope Entry Type : 03 [059h 0089 1] Entry Length : 08 [05Ah 0090 2] Reserved : 0000 [05Ch 0092 1] Enumeration ID : 08 [05Dh 0093 1] PCI Bus Number : F0 [05Eh 0094 2] PCI Path : 1F,00 [060h 0096 1] Device Scope Entry Type : 04 [061h 0097 1] Entry Length : 08 [062h 0098 2] Reserved : 0000 [064h 0100 1] Enumeration ID : 00 [065h 0101 1] PCI Bus Number : F0 [066h 0102 2] PCI Path : 0F,00 [068h 0104 2] Subtable Type : 0001 [Reserved Memory Region] [06Ah 0106 2] Length : 0020 [06Ch 0108 2] Reserved : 0000 [06Eh 0110 2] PCI Segment Number : 0000 [070h 0112 8] Base Address : 000000006DEA5000 [078h 0120 8] End Address (limit) : 000000006DEB3FFF [080h 0128 1] Device Scope Entry Type : 01 [081h 0129 1] Entry Length : 08 [082h 0130 2] Reserved : 0000 [084h 0132 1] Enumeration ID : 00 [085h 0133 1] PCI Bus Number : 00 [086h 0134 2] PCI Path : 14,00 [088h 0136 2] Subtable Type : 0001 [Reserved Memory Region] [08Ah 0138 2] Length : 0020 [08Ch 0140 2] Reserved : 0000 [08Eh 0142 2] PCI Segment Number : 0000 [090h 0144 8] Base Address : 000000006F000000 [098h 0152 8] End Address (limit) : 000000007F1FFFFF [0A0h 0160 1] Device Scope Entry Type : 01 [0A1h 0161 1] Entry Length : 08 [0A2h 0162 2] Reserved : 0000 [0A4h 0164 1] Enumeration ID : 00 [0A5h 0165 1] PCI Bus Number : 00 [0A6h 0166 2] PCI Path : 02,00 Raw Table Data: Length 168 (0xA8) 0000: 44 4D 41 52 A8 00 00 00 01 57 49 4E 54 45 4C 20 DMAR.....WINTEL 0010: 42 44 57 20 00 00 00 00 01 00 00 00 49 4E 54 4C BDW ........INTL 0020: 01 00 00 00 26 03 00 00 00 00 00 00 00 00 00 00 ....&........... 0030: 00 00 18 00 00 00 00 00 00 00 D9 FE 00 00 00 00 ................ 0040: 01 08 00 00 00 00 02 00 00 00 20 00 01 00 00 00 .......... ..... 0050: 00 10 D9 FE 00 00 00 00 03 08 00 00 08 F0 1F 00 ................ 0060: 04 08 00 00 00 F0 0F 00 01 00 20 00 00 00 00 00 .......... ..... 0070: 00 50 EA 6D 00 00 00 00 FF 3F EB 6D 00 00 00 00 .P.m.....?.m.... 0080: 01 08 00 00 00 00 14 00 01 00 20 00 00 00 00 00 .......... ..... 0090: 00 00 00 6F 00 00 00 00 FF FF 1F 7F 00 00 00 00 ...o............ 00A0: 01 08 00 00 00 00 02 00 ........

Offline

#2837 2014-09-30 02:55:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
aw wrote:

Please post the resulting /tmp/dmar.dsl file from:

$ sudo iasl -p /tmp/dmar -d /sys/firmware/acpi/tables/DMAR
/* [030h 0048 2] Subtable Type : 0000 [Hardware Unit Definition] [032h 0050 2] Length : 0018 [034h 0052 1] Flags : 00 [035h 0053 1] Reserved : 00 [036h 0054 2] PCI Segment Number : 0000 [038h 0056 8] Register Base Address : 00000000FED90000 [040h 0064 1] Device Scope Entry Type : 01 [041h 0065 1] Entry Length : 08 [042h 0066 2] Reserved : 0000 [044h 0068 1] Enumeration ID : 00 [045h 0069 1] PCI Bus Number : 00 [046h 0070 2] PCI Path : 02,00 [048h 0072 2] Subtable Type : 0000 [Hardware Unit Definition] [04Ah 0074 2] Length : 0020 [04Ch 0076 1] Flags : 01 [04Dh 0077 1] Reserved : 00 [04Eh 0078 2] PCI Segment Number : 0000 [050h 0080 8] Register Base Address : 00000000FED91000

Ok, as expected you have two IOMMU hardware units, the first is exclusively for the IGD (00:02.0).  We can match that to dmesg by comparing the base address:

dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a

The 2nd IOMMU hardware unit has the INCLUDE_PCI_ALL flag bit set, so everything else is handled by this IOMMU, which we can also verify by base address:

dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da

So, unless you're trying to assign IGD, we know you're exclusively using IOMMU1, which has an extended capability register value of 0xf010da.  Bit 7 in that register indicates support for Snoop Control and you've already verified that ignoring this bit fixes the problem.  When SC is supported, we can program the IOMMU page tables with the SNP bit set which causes processor caches to be snooped regardless of the NoSnoop tag on the transaction.  This effectively makes all DMA cache coherent and KVM doesn't need to emulated WBINVD to maintain coherency.

In your case the hardware tells us that SC is supported, but the IOMMU page table hardware considers it a reserved bit and throws a fault.  What CPU model are you using?  Is it perhaps a prototype?  I believe some systems have a BIOS config option to toggle IOMMU caching support, if you have such an option, disable it.  I think you're looking at either a hardware or BIOS issue.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2838 2014-09-30 03:39:47

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:
aw wrote:

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

Yes , 3.16.3 .

Still seems potentially related to large pages.  Does using either of these options avoid it?

a) boot arg: intel_iommu=on,sp_off

b) module arg: vfio_iommu_type1 disable_hugepages=1

YES !
adding : "vfio_iommu_type1.disable_hugepages=1" to boot parameters solved the issue . Now I'll see if I can reboot this VM without crashing the host !

EDIT : Aaaand , it reboots just fine !

Thank you Alex , we appreciate your kind efforts !

Last edited by Denso (2014-09-30 03:55:44)

Offline

#2839 2014-09-30 05:19:37

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So, unless you're trying to assign IGD, we know you're exclusively using IOMMU1, which has an extended capability register value of 0xf010da.  Bit 7 in that register indicates support for Snoop Control and you've already verified that ignoring this bit fixes the problem.  When SC is supported, we can program the IOMMU page tables with the SNP bit set which causes processor caches to be snooped regardless of the NoSnoop tag on the transaction.  This effectively makes all DMA cache coherent and KVM doesn't need to emulated WBINVD to maintain coherency.

In your case the hardware tells us that SC is supported, but the IOMMU page table hardware considers it a reserved bit and throws a fault.  What CPU model are you using?  Is it perhaps a prototype?  I believe some systems have a BIOS config option to toggle IOMMU caching support, if you have such an option, disable it.  I think you're looking at either a hardware or BIOS issue.

For reference, the CPU is a Xeon E3-1245v3.

I looked around in my BIOS, and there were no options for IOMMU caching -- but there WAS an option for X2APIC Opt-Out.
X2APIC Opt-Out was enabled, so I disabled it.  Now I've booted the non-patched kernel, and the errors are no longer occurring. 
Thanks for the suggestion of checking the BIOS settings.

[ 0.067534] dmar: Host address width 39 [ 0.067537] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.067544] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.067546] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.067551] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.067552] dmar: RMRR base: 0x0000006dea5000 end: 0x0000006deb3fff [ 0.067553] dmar: RMRR base: 0x0000006f000000 end: 0x0000007f1fffff [ 0.067618] IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1 [ 0.067621] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.067731] Enabled IRQ remapping in x2apic mode [ 0.067733] Enabling x2apic [ 0.067734] Enabled x2apic [ 0.067738] Switched APIC routing to cluster x2apic. [ 0.629294] DMAR: No ATSR found [ 0.629318] IOMMU 0 0xfed90000: using Queued invalidation [ 0.629319] IOMMU 1 0xfed91000: using Queued invalidation [ 0.629321] IOMMU: Setting RMRR: [ 0.629330] IOMMU: Setting identity map for device 0000:00:02.0 [0x6f000000 - 0x7f1fffff] [ 0.630530] IOMMU: Setting identity map for device 0000:00:14.0 [0x6dea5000 - 0x6deb3fff] [ 0.630547] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.630554] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Last edited by DanaGoyette (2014-09-30 05:23:06)

Offline

#2840 2014-09-30 14:00:44

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everyone .

So I'm trying to use OVMF instead of SeaBIOS , I used ovmf-svn package from AUR and added the option "-pflash /*path to ovmf_x64.bin*" to my VM aguments . Also I deleted "-vga none" AND "x-vga=on" .

It booted just fine , GPU works as primary no problem .

Except for one big issue :

It doesn't detect either my VM's HDD nor ISOs .

So it goes like this :

EFI Floppy
EFI Floppy 1
EFI Misc Device
EFI Network

then it drops me to the UEFI shell .

Any reason why it didn't pickup my HDD or ISO images to boot from ?

Offline

#2841 2014-09-30 15:10:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hey everyone .

So I'm trying to use OVMF instead of SeaBIOS , I used ovmf-svn package from AUR and added the option "-pflash /*path to ovmf_x64.bin*" to my VM aguments . Also I deleted "-vga none" AND "x-vga=on" .

It booted just fine , GPU works as primary no problem .

Except for one big issue :

It doesn't detect either my VM's HDD nor ISOs .

So it goes like this :

EFI Floppy
EFI Floppy 1
EFI Misc Device
EFI Network

then it drops me to the UEFI shell .

Any reason why it didn't pickup my HDD or ISO images to boot from ?

What interface are you using to expose them to the guest?  IDE?  AHCI?  virtio?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2842 2014-09-30 15:47:14

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

Hey everyone .

So I'm trying to use OVMF instead of SeaBIOS , I used ovmf-svn package from AUR and added the option "-pflash /*path to ovmf_x64.bin*" to my VM aguments . Also I deleted "-vga none" AND "x-vga=on" .

It booted just fine , GPU works as primary no problem .

Except for one big issue :

It doesn't detect either my VM's HDD nor ISOs .

So it goes like this :

EFI Floppy
EFI Floppy 1
EFI Misc Device
EFI Network

then it drops me to the UEFI shell .

Any reason why it didn't pickup my HDD or ISO images to boot from ?

What interface are you using to expose them to the guest?  IDE?  AHCI?  virtio?

Virtio for the HDD and IDE for the ISOs ... This is the entire command :

qemu-system-x86_64 -name main -nographic \ -enable-kvm -M q35 -m 6000 -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 2,sockets=1,cores=1,threads=2 \ -pflash /VMs/ovmf_x64.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -drive file=/VMs/Win_Main.img,if=virtio,cache=none \ -drive file=/VMs/Win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -drive file=/VMs/virtio.iso,id=isocd2 -device ide-cd,bus=ide.2,drive=isocd2 \ -net nic,model=virtio,macaddr=1c:2c:44:11:99:2d -net bridge,br=br0 \ -usb -usbdevice host:15d9:0a4f -usbdevice host:045e:0745 \ -localtime \ -monitor unix:/tmp/vm_main,server,nowait &

Last edited by Denso (2014-09-30 15:48:15)

Offline

#2843 2014-09-30 15:54:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Virtio for the HDD and IDE for the ISOs ... This is the entire command :

qemu-system-x86_64 -name main -nographic \ -enable-kvm -M q35 -m 6000 -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 2,sockets=1,cores=1,threads=2 \ -pflash /VMs/ovmf_x64.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -drive file=/VMs/Win_Main.img,if=virtio,cache=none \ -drive file=/VMs/Win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -drive file=/VMs/virtio.iso,id=isocd2 -device ide-cd,bus=ide.2,drive=isocd2 \ -net nic,model=virtio,macaddr=1c:2c:44:11:99:2d -net bridge,br=br0 \ -usb -usbdevice host:15d9:0a4f -usbdevice host:045e:0745 \ -localtime \ -monitor unix:/tmp/vm_main,server,nowait &

Any particular reason for using q35?  When you get an EFI shell, do you have any disks available?  EFI shell is just like DOS except the disk are named fs0:, fs1:, etc rather than C:, D:...  If you have disks you can try to find a a boot EFI file and execute it.  You can also try adding bootindex=# to each drive to try to force a boot order.  I haven't had time yet to try to attempt a q35+ovmf install and I have little reason to switch from 440fx other than curiosity.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2844 2014-09-30 16:03:47

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

Virtio for the HDD and IDE for the ISOs ... This is the entire command :

qemu-system-x86_64 -name main -nographic \ -enable-kvm -M q35 -m 6000 -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 2,sockets=1,cores=1,threads=2 \ -pflash /VMs/ovmf_x64.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \ -drive file=/VMs/Win_Main.img,if=virtio,cache=none \ -drive file=/VMs/Win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -drive file=/VMs/virtio.iso,id=isocd2 -device ide-cd,bus=ide.2,drive=isocd2 \ -net nic,model=virtio,macaddr=1c:2c:44:11:99:2d -net bridge,br=br0 \ -usb -usbdevice host:15d9:0a4f -usbdevice host:045e:0745 \ -localtime \ -monitor unix:/tmp/vm_main,server,nowait &

Any particular reason for using q35?  When you get an EFI shell, do you have any disks available?  EFI shell is just like DOS except the disk are named fs0:, fs1:, etc rather than C:, D:...  If you have disks you can try to find a a boot EFI file and execute it.  You can also try adding bootindex=# to each drive to try to force a boot order.  I haven't had time yet to try to attempt a q35+ovmf install and I have little reason to switch from 440fx other than curiosity.

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

Offline

#2845 2014-09-30 16:07:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \ -device vfio-pci,host=05:00.1,addr=06.1 \ -device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

Last edited by aw (2014-09-30 16:08:08)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2846 2014-09-30 16:18:56

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \ -device vfio-pci,host=05:00.1,addr=06.1 \ -device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

OK , it booted fine , but still no disks . Host is running on legacy BIOS , perhaps that's the reason ?

Offline

#2847 2014-09-30 16:24:08

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

et38 wrote:
noobman wrote:

About not being able to passthrough the onboard usb controllers, I also had that situation also with a gigabyte mobo, where apparently i could not pass any of the onboard usb controllers. None at all. What i found to be working was to disable the xhci controller entirelly, from bios. And then the passthrough was possible (for me), with either of the other remaining controllers, like a charm. Playing around with bios setup options for xhci and ehci paid off for me. Worth a try imo.

I was looking at the passtrough setup with ga-z97x-u3dh. If it's working, why is the motherboard not recommended in the list?

I had a bad experience with it, it has many issues. First off the one above, have to disable xhci entirelly from bios, from which point the 3.0 usb is not entirelly lost but become 2.0 instead, and then and only then one can passthrough other usb controllers (F7 bios). Not sure why its like that, i guess its probably due to a particular usb tree implementation. Also the latest bios i tried (F7 again) spilled some other errors so imo that bios is also bit broke. E.g. board gave "cannot find upstream pcie" an error like that, again i guess due to the pci arhitecture tree, but this last one got fixed i think in 3.17. Back few (1-2-3?) months ago i think stock kernels gave like 10+ different errors on it, researched and customized and patched one and got the number down. The uefi side threw some errors, and i disabled it also, but that is ok coz i just dont like uefi anyway. Also a number of strange errors when i iommu isolate MEI controller, something which i like to do and which should be fine but in this case wasnt. Dunno how it is now if there is any F8 bios or other fixes are in 3.17+ or not, because i made the system for a friend and its gone (so all comments are just from my memory), but otherwise its working with passthrough. Board on the overall also had many small issues which added up for me. Just to have an ideea, for example the warning for CPU max temp and fan are disabled as default settings (warnings or ctrl) so just by using bios default settings there is nothing that would prevent a catastrophe if the cpu gets to max temp and over it (e.g. if fan fails). For me those default settings show how much attention was given to the board. I think the board was among first z97 implementation and i think it simply got rushed over to launch. Quite some months ago i was looking for the latest, looked great on paper and specs, but a worse in practice so i just ended up as not happy with it. "Not recommented "is just that, a recommendation, just take it like that, you could get one and have your own and maybe different experience with it. But just saying this, if you want something specifially for passthrough and looking to buy, then simply why not get what the guy which does vfio have, like the 990FX which i think that is what aw has (?) not sure lol, but that paired with an 8 core cpu could be quite a performant option. But that is just a thought, nothing more. Mentioning coz that would be my choice right now, and most likely next time i will do exactly that.

Offline

#2848 2014-09-30 18:05:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:
Denso wrote:

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \ -device vfio-pci,host=05:00.1,addr=06.1 \ -device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

OK , it booted fine , but still no disks . Host is running on legacy BIOS , perhaps that's the reason ?

The host using legacy/UEFI BIOS is completely separate from the guest, I think it more likely has something to do with how you're specifying the disks.  I typically specify an installation and virtio ISO like this:

-drive file=/ISOs/msdn/en_windows_8.1_professional_n_vl_with_update_x64_dvd_4065208.iso,if=none,id=dvd0,media=cdrom \ -device ide-cd,drive=dvd0,bootindex=2 \ -drive file=/ISOs/virtio-win-0.1-74.iso,if=none,id=dvd1,media=cdrom \ -device ide-cd,drive=dvd1 \

The guest disk should really look more like this:

-drive file=/path/to/guest/disk.img,cache=none,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2849 2014-09-30 18:37:52

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:
aw wrote:

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \ -device vfio-pci,host=05:00.1,addr=06.1 \ -device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

OK , it booted fine , but still no disks . Host is running on legacy BIOS , perhaps that's the reason ?

The host using legacy/UEFI BIOS is completely separate from the guest, I think it more likely has something to do with how you're specifying the disks.  I typically specify an installation and virtio ISO like this:

-drive file=/ISOs/msdn/en_windows_8.1_professional_n_vl_with_update_x64_dvd_4065208.iso,if=none,id=dvd0,media=cdrom \ -device ide-cd,drive=dvd0,bootindex=2 \ -drive file=/ISOs/virtio-win-0.1-74.iso,if=none,id=dvd1,media=cdrom \ -device ide-cd,drive=dvd1 \

The guest disk should really look more like this:

-drive file=/path/to/guest/disk.img,cache=none,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \

Finally OVMF detected my Win8.iso , however trying to execute /EFI/BOOT/BOOTX64.EFI hangs the guest for 3 seconds then returns to the shell prompt .

Googling told me to execute that file to start the Windows installer  : (

Offline

#2850 2014-09-30 18:44:33

et38
Member
Registered: 2014-09-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

But just saying this, if you want something specifially for passthrough and looking to buy, then simply why not get what the guy which does vfio have, like the 990FX which i think that is what aw has (?) not sure lol, but that paired with an 8 core cpu could be quite a performant option. But that is just a thought, nothing more. Mentioning coz that would be my choice right now, and most likely next time i will do exactly that.

I've been thinking (a lot) of an 8-core Amd system, but the power consumption difference (vs. Intel 4c+HT) is just annoying. I'm just trying to see if there are any preferredly (Gigabyte/Asus/Z97/H97) motherboards offering relatively painless vga passthrough experience. Guests would probably be Steam OS / other dev distros.

Offline

#2851 2014-09-30 21:39:33

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone help me write a script to launch 3 guests (I have those 3 qemu scripts already) each with its own GPU+USB on Fedora20 after fully booting? I know it has to be in rc.local, but I want the guests to be each in its own screen. This "server" is going to serve as a multiseat PC "plugnplay"..

Thanks!

Offline

#2852 2014-09-30 22:07:36

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Can someone help me write a script to launch 3 guests (I have those 3 qemu scripts already) each with its own GPU+USB on Fedora20 after fully booting? I know it has to be in rc.local, but I want the guests to be each in its own screen. This "server" is going to serve as a multiseat PC "plugnplay"..

Thanks!

Use libvirtd for this.

Offline

#2853 2014-09-30 22:10:51

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
devianceluka wrote:

Can someone help me write a script to launch 3 guests (I have those 3 qemu scripts already) each with its own GPU+USB on Fedora20 after fully booting? I know it has to be in rc.local, but I want the guests to be each in its own screen. This "server" is going to serve as a multiseat PC "plugnplay"..

Thanks!

Use libvirtd for this.

Im kind of against libvirt because I have a perfect working guests with qemu and converting qemu cli to libvirtd is a pain...

Offline

#2854 2014-09-30 22:15:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
dwe11er wrote:
devianceluka wrote:

Can someone help me write a script to launch 3 guests (I have those 3 qemu scripts already) each with its own GPU+USB on Fedora20 after fully booting? I know it has to be in rc.local, but I want the guests to be each in its own screen. This "server" is going to serve as a multiseat PC "plugnplay"..

Thanks!

Use libvirtd for this.

Im kind of against libvirt because I have a perfect working guests with qemu and converting qemu cli to libvirtd is a pain...

Guess you better learn how to script then... wink


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2855 2014-09-30 22:16:41

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can you then post a perfect working xml so I can then try to change and abuse to my liking?

Offline

#2856 2014-09-30 22:44:05

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Can you then post a perfect working xml so I can then try to change and abuse to my liking?

http://pastie.org/private/oeqpunieatd3gu1ziyptw

Offline

#2857 2014-09-30 23:01:52

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Booting Windows 8.1 Enterprise using (EFI/BOOT/BOOTX64.EFI) returns me to the UEFI shell , what am I missing ?

Offline

#2858 2014-09-30 23:16:33

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

et38 wrote:
noobman wrote:

But just saying this, if you want something specifially for passthrough and looking to buy, then simply why not get what the guy which does vfio have, like the 990FX which i think that is what aw has (?) not sure lol, but that paired with an 8 core cpu could be quite a performant option. But that is just a thought, nothing more. Mentioning coz that would be my choice right now, and most likely next time i will do exactly that.

I've been thinking (a lot) of an 8-core Amd system, but the power consumption difference (vs. Intel 4c+HT) is just annoying. I'm just trying to see if there are any preferredly (Gigabyte/Asus/Z97/H97) motherboards offering relatively painless vga passthrough experience. Guests would probably be Steam OS / other dev distros.

Very arguable subject, amd vs intel, and all sides fanboys and marketing. But on short i think somewhere may be a confusion between the power rating and the actual power consumtion. When the cpu says its 130W or 220W, that is a power rating, take it as that, a rating only, but with a twist. The actual losses are not resistive like a light bulb, the losses are dubbed "switching" losses which means it depends on how you get it ticking, its cpu governor or generally how much work the cpu does, or what its the idle freq. So take a look and check intel's frequencies to be sort of lower than amd, then so is the rating, coz again by far they are switching looses and intel wont run freq high enough to need higher rating. Other than that all the billion gates are now pretty much the same on waffer, complementary pair, nothing new and nothing more to say about it. Just that it will take a change in technology to get any further in frequency, so for the time being the cpus evolution has to go in more cores and/or more features exactly because the frequency did hit its practical limit for the current gate technology. Remember how generations of cpus used to evolve from 33Mhz and up, they cant do that any more they are on the roof already. And as much as they go higher frequencies their thermal losses increase and become impossible to deal with. Generally i think intels stays further away from that frequency roof / hill while amd goes a bit closer. Probably some sort of product or marketing policy. I am not taking sides, so imho its all the same thing except the evolution in nr of cores and features, which both are sort of a waste if the user does not actually use them.

And those ratings to start with, on the other hand depend on many things, thermal junction from die to sink, thermal junction from sink to ambient etc. So its more like a thermal power rating, sort of speaking points at what level it will be and how much will dissipate before running into thermal protection mode. More or less like any single semiconductor datasheet, you can try check out how to interpret a power mosfet datasheet. When choosing a power mosfet every engineer will/should go with highest rating, with the mosfet that would conduct the most amps before it runs in termal shutdown. But when we talk about billions of those, in complementary pairs, the consumers are gonna panic and choose the lowest rating. They will choose the lowest wattage cpu but they will also choose the highest wattage stereo. And all that while the cpu is able to scale down, and the other things can not. smile

Offline

#2859 2014-10-01 00:02:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

But just saying this, if you want something specifially for passthrough and looking to buy, then simply why not get what the guy which does vfio have, like the 990FX which i think that is what aw has (?)

Thanks partially to my employer, I have several systems, 990FX is one of them.  It's my token AMD-Vi system though.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2860 2014-10-01 02:57:24

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for sharing.

While at it, let me thank you for ... errr ... everything else big_smile

Offline

#2861 2014-10-01 07:15:28

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Errors when rebooting VM :

[ 6061.771873] irq 28: nobody cared (try booting with the "irqpoll" option) [ 6061.771911] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 3.16.3-1-ARCH #1 [ 6061.771913] Hardware name: ASUS All Series/X99-DELUXE, BIOS 0904 09/22/2014 [ 6061.771915] 0000000000000000 b4596cd08d99296e ffff88085fc03be8 ffffffff8152b3bc [ 6061.771920] ffff88007ce16200 ffff88085fc03c10 ffffffff810d02a2 ffff88007ce16200 [ 6061.771924] 0000000000000000 000000000000001c ffff88085fc03c48 ffffffff810d0657 [ 6061.771928] Call Trace: [ 6061.771930] <IRQ> [<ffffffff8152b3bc>] dump_stack+0x4d/0x6f [ 6061.771947] [<ffffffff810d02a2>] __report_bad_irq+0x32/0xd0 [ 6061.771952] [<ffffffff810d0657>] note_interrupt+0x257/0x2a0 [ 6061.771957] [<ffffffff810cdbae>] handle_irq_event_percpu+0xae/0x1f0 [ 6061.771961] [<ffffffff810cdd2d>] handle_irq_event+0x3d/0x60 [ 6061.771966] [<ffffffff810d1281>] handle_fasteoi_irq+0x81/0x170 [ 6061.771974] [<ffffffff8101717e>] handle_irq+0x1e/0x40 [ 6061.771978] [<ffffffff81533bed>] do_IRQ+0x4d/0xe0 [ 6061.771983] [<ffffffff81531bad>] common_interrupt+0x6d/0x6d [ 6061.771989] [<ffffffff810da64e>] ? ktime_get+0x1e/0x100 [ 6061.771996] [<ffffffff813e79b1>] intel_pstate_timer_func+0x71/0x420 [ 6061.771999] [<ffffffff813e7952>] ? intel_pstate_timer_func+0x12/0x420 [ 6061.772003] [<ffffffff813e7940>] ? intel_pstate_set_pstate+0x110/0x110 [ 6061.772010] [<ffffffff8107b7f6>] call_timer_fn+0x36/0x160 [ 6061.772013] [<ffffffff813e7940>] ? intel_pstate_set_pstate+0x110/0x110 [ 6061.772017] [<ffffffff8107c224>] run_timer_softirq+0x274/0x320 [ 6061.772024] [<ffffffff810736c2>] __do_softirq+0xf2/0x2e0 [ 6061.772029] [<ffffffff81073a06>] irq_exit+0x86/0xb0 [ 6061.772033] [<ffffffff81533cc4>] smp_apic_timer_interrupt+0x44/0x50 [ 6061.772037] [<ffffffff81531f4d>] apic_timer_interrupt+0x6d/0x80 [ 6061.772039] <EOI> [<ffffffff813e88bc>] ? cpuidle_enter_state+0x4c/0xc0 [ 6061.772046] [<ffffffff813e8a17>] cpuidle_enter+0x17/0x20 [ 6061.772050] [<ffffffff810b710f>] cpu_startup_entry+0x32f/0x520 [ 6061.772056] [<ffffffff815216c4>] rest_init+0x84/0x90 [ 6061.772060] [<ffffffff818f5fc9>] start_kernel+0x45e/0x47f [ 6061.772064] [<ffffffff818f5120>] ? early_idt_handlers+0x120/0x120 [ 6061.772068] [<ffffffff818f54d7>] x86_64_start_reservations+0x2a/0x2c [ 6061.772071] [<ffffffff818f5626>] x86_64_start_kernel+0x14d/0x170 [ 6061.772073] handlers: [ 6061.772098] [<ffffffffa07693e0>] vfio_intx_handler [vfio_pci] [ 6061.772129] Disabling IRQ #28

This results in vertical tears in the screen .

Offline

#2862 2014-10-01 07:36:30

et38
Member
Registered: 2014-09-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noobman wrote:

Probably some sort of product or marketing policy. I am not taking sides, so imho its all the same thing except the evolution in nr of cores and features, which both are sort of a waste if the user does not actually use them.

I see your point and I'm not yet entirely immune to marketing. This system should be retired/repurposed at around 2020. Since there is little data to go around, one has to rely on online reviews. It's not possible to verify them, however, the ones I found describe >50% difference in system power consumption under load roughly at the same performance level. With insufficient data, I'm unable to say what that equates to, but I'm under the impression that lower thermal load will reduce noise and probability of failure.

Offline

#2863 2014-10-01 15:06:49

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:

I get a compat_monitor0 screen similar to Slabity. However, I am using an intel IGP for the host, and an AMD 6990M for the Windows 7 guest.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0

I'm using the OP's linux-mainline package, and have also used the acs_override=downstream option with the same result.

Same result when running vga-clear vga-set.

dmesg | grep vgaarb

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.504322] vgaarb: loaded [ 0.504323] vgaarb: bridge control possible 0000:01:00.0 [ 0.504324] vgaarb: no bridge control possible 0000:00:02.0 [ 1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

I solved this problem using pci-assign. The VM now starts windows, but freezes at the windows logo.

Any ideas?

Offline

#2864 2014-10-01 15:22:40

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:
Lauer wrote:

I get a compat_monitor0 screen similar to Slabity. However, I am using an intel IGP for the host, and an AMD 6990M for the Windows 7 guest.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0

I'm using the OP's linux-mainline package, and have also used the acs_override=downstream option with the same result.

Same result when running vga-clear vga-set.

dmesg | grep vgaarb

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 [ 0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.504322] vgaarb: loaded [ 0.504323] vgaarb: bridge control possible 0000:01:00.0 [ 0.504324] vgaarb: no bridge control possible 0000:00:02.0 [ 1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

I solved this problem using pci-assign. The VM now starts windows, but freezes at the windows logo.

Any ideas?

I had a similar issue , Alex solved it by adding "sp_off" to bootloader arguments after the "intel_iommu=on"

So it becomes : "intel_iommu=on,sp_off" .

That solved it for me .

Offline

#2865 2014-10-01 17:31:48

noobman
Member
Registered: 2013-12-13
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

et38 wrote:
noobman wrote:

Probably some sort of product or marketing policy. I am not taking sides, so imho its all the same thing except the evolution in nr of cores and features, which both are sort of a waste if the user does not actually use them.

I see your point and I'm not yet entirely immune to marketing. This system should be retired/repurposed at around 2020. Since there is little data to go around, one has to rely on online reviews. It's not possible to verify them, however, the ones I found describe >50% difference in system power consumption under load roughly at the same performance level. With insufficient data, I'm unable to say what that equates to, but I'm under the impression that lower thermal load will reduce noise and probability of failure.

That >50%, thats quite a strangely high number, imo its way off. You can also find quite different numbers and all kinds of numbers out there will just be used to back up some guy's opinion, thats all. No point to comment on numbers. But other than that, as i said the switching losses depend on frequency, and both of the digital gates have exact same technology, so if you actually could compare both cpus running at exact same frequency, then ought to give the same consumption. But in reality they are made to run at different frequencies both at idle and at load. And that gives the perception of being more different than they actually are. Just as a note, same frequency does not equate same load, because same load will still be ran at different frequency by each. Dunno what would be the deal with thermal dissipation vs noise and failures. Thermal dissipation goes hand in hand with frequency, which in turn goes hand in hand with cycles and performance. About noise, there is really no noise in digital realm, thats why static discipline was invented for. About failures, there are both same thing inside, silicon tech that is quite rugged, and i dont see why one would be more sensitive than the other.

Imho its just like comparing two t-shirts, if you want to get into the details there are some minor differences and can focus a comparison on that area, but in the big picture lets not forget both are t-shirts, so are ~99% same thing. They even use the same "sewing machines" industrial tools to make it and same raw materials suppliers, workforce even may jump between, and same technology which again hit its own limits years back. Therefore the efficiency differences can not be all that dramatical.

And marketing does give some interesting colors. Some time ago when cpus were evolving in frequency terms, yes everybody was looking just at that. One would upgrade an 800Mhz cpu to 1,6Ghz cpu, coz clearly that was better, everybody knew that. Now the argument goes the other way, we go with 3.2Ghz instead of 4.5Ghz, coz 3.2 its more efficient and saves electricity. I quite like this focus on efficiency, i think its great issue. But what consumes 3/4 of the energy bill in every household is the ... fridge. Even much more if there is air conditioning or heating. Coz the hardest thing to do in physics is create and maintain differences of temperatures because that has to fight law of entropy continiously, every single second. And the integral of that ofc gets huge. But is the fridge isolation a consummer concern -no. When buying the air conditioning, ppls will also look to get most watts/btu. Also high wattage in the stereo, the huget flat tv, home cinema, surround sound, etc. Did ppls commenting on cpu efficiency, what did they do for this concercn, di already replaced all light bublbs in their house with leds -probably not. Did they hermetically isolate the house with 20cm foam -probably not. So the concern with reducing power consumption is great thing, all regards towards that. But, among all the things that would reduce the power bill, the cpu is really the last on the list. The cpu itself is the most scalable and most inteligent power consumer in the house, so why pick on it. Ppls who seriously want to improve the power bill, would have to do all the other things first, and then get some fotovoltaic panels and that kind of things. Other than that is nonsense to me tbh.

I will rest coz there is no point, sort of speaking yes that dead horse was beaten too much already smile and not the point of this thread, sry everyone for the side-track of it, now getting back main course.

Last edited by noobman (2014-10-01 17:33:57)

Offline

#2866 2014-10-01 18:18:12

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I had a similar issue , Alex solved it by adding "sp_off" to bootloader arguments after the "intel_iommu=on"

So it becomes : "intel_iommu=on,sp_off" .

That solved it for me .

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on,sp_off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0

pci-assign: still freezes at windows logo.

vfio: device manager shows generic VGA card.

Offline

#2867 2014-10-01 18:28:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:
Denso wrote:

I had a similar issue , Alex solved it by adding "sp_off" to bootloader arguments after the "intel_iommu=on"

So it becomes : "intel_iommu=on,sp_off" .

That solved it for me .

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on,sp_off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0

pci-assign: still freezes at windows logo.

vfio: device manager shows generic VGA card.

I have no idea what you're trying to do.  The logo freezes on what?  The emulated display?  That's supposed to happen if you're attempting to assign the GPU as a secondary display.  My only advice for pci-assign is don't use it.  If you'd like some help with vfio-pci, please share with us how you're starting the guest to start with.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2868 2014-10-01 19:30:22

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I have no idea what you're trying to do.  The logo freezes on what?  The emulated display?  That's supposed to happen if you're attempting to assign the GPU as a secondary display.  My only advice for pci-assign is don't use it.  If you'd like some help with vfio-pci, please share with us how you're starting the guest to start with.

cat runwin-vfio

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vnc :0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/dev/md0,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk

Windows loads, but my card does not appear in device manager.

Offline

#2869 2014-10-01 19:34:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:
aw wrote:

I have no idea what you're trying to do.  The logo freezes on what?  The emulated display?  That's supposed to happen if you're attempting to assign the GPU as a secondary display.  My only advice for pci-assign is don't use it.  If you'd like some help with vfio-pci, please share with us how you're starting the guest to start with.

cat runwin-vfio

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vnc :0 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/dev/md0,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk

Windows loads, but my card does not appear in device manager.

Ugh, maybe go back to the 1st post and work on the start command.  You're not specifying -vga none, so you have an emulated VGA device, the assigned device will be secondary.  I wouldn't recommend that model unless you have a Quadro card (even Quadro doesn't work in this mode on Q35).  That also basically negates the x-vga=on option.  You're also trying to attach a piix4 (440FX) IDE device to a Q35 chipset.  There's just too many problems in that commandline to even deal with.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2870 2014-10-01 20:06:51

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Ugh, maybe go back to the 1st post and work on the start command.  You're not specifying -vga none, so you have an emulated VGA device, the assigned device will be secondary.  I wouldn't recommend that model unless you have a Quadro card (even Quadro doesn't work in this mode on Q35).  That also basically negates the x-vga=on option.  You're also trying to attach a piix4 (440FX) IDE device to a Q35 chipset.  There's just too many problems in that commandline to even deal with.

This is what I'm trying to achieve.

-vga none results in compat_monitor0 terminal, which is why I was using pci-assign. (I posted this before but nobody responded so I tried using pci-assign)

The piix4 lines are from the OP, so that's not really my fault for using them. (Should I get rid of them even though the VM boots just fine with vfio?)

Offline

#2871 2014-10-01 20:23:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:
aw wrote:

Ugh, maybe go back to the 1st post and work on the start command.  You're not specifying -vga none, so you have an emulated VGA device, the assigned device will be secondary.  I wouldn't recommend that model unless you have a Quadro card (even Quadro doesn't work in this mode on Q35).  That also basically negates the x-vga=on option.  You're also trying to attach a piix4 (440FX) IDE device to a Q35 chipset.  There's just too many problems in that commandline to even deal with.

This is what I'm trying to achieve.

Please notice that in that demonstration, the desktop window is a tigervnc client and you can see from the icon on the guest desktop that it's running TightVNC and is connected, so there's some degree of trickery here that makes it look like the VM is running in a desktop window, but it's actually no different than if you did a VNC connection to a remote system.  They also have a spice display, so presumably they do also have an emulated graphics device.  Reports like this come through every so often that someone got things to work as a secondary device w/o using VGA assignment, maybe even using pci-assign.  It depends on the guest driver working in that configuration.  If it doesn't, you're SOL on that path.

-vga none results in compat_monitor0 terminal, which is why I was using pci-assign. (I posted this before but nobody responded so I tried using pci-assign)

I didn't respond because I don't know what a "compat_monitor0 terminal" even means.  If it means you got a console window, use the -nographic option to make it go away.

The piix4 lines are from the OP, so that's not really my fault for using them. (Should I get rid of them even though the VM boots just fine with vfio?)

Seems odd to me, but whatever.  I'm encouraging people just trying to run windows to forget about Q35 and just use the default 440FX model.  It seems like you're also working with a mobile AMD GPU.  Hybrid, dual GPU laptops are wired in strange ways, so if you're expecting that you can plug a monitor into one of the outputs and get the discrete graphics output, there's a good chance it won't work.  You might be able to install everything, setup the Catalyst driver and VNC server in the guest with and emulated VGA, then add the Radeon, remove the emulated VGA, and hope you can blind boot the guest and connect.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2872 2014-10-01 23:56:10

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I didn't respond because I don't know what a "compat_monitor0 terminal" even means.  If it means you got a console window, use the -nographic option to make it go away.

Seems odd to me, but whatever.  I'm encouraging people just trying to run windows to forget about Q35 and just use the default 440FX model.  It seems like you're also working with a mobile AMD GPU.  Hybrid, dual GPU laptops are wired in strange ways, so if you're expecting that you can plug a monitor into one of the outputs and get the discrete graphics output, there's a good chance it won't work.  You might be able to install everything, setup the Catalyst driver and VNC server in the guest with and emulated VGA, then add the Radeon, remove the emulated VGA, and hope you can blind boot the guest and connect.

Running with -vga none -nographic -vnc :0 gets rid of the terminal.:) The vnc window now says "This VM has no graphics display device."

Does this change anything? Should I try the vnc in guest solution? Would that solution even work if there is no graphics display device?

Also when I plug in HDMI, arch mirrors my desktop automatically. (even with no X server running)

Offline

#2873 2014-10-02 00:24:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:
aw wrote:

I didn't respond because I don't know what a "compat_monitor0 terminal" even means.  If it means you got a console window, use the -nographic option to make it go away.

Seems odd to me, but whatever.  I'm encouraging people just trying to run windows to forget about Q35 and just use the default 440FX model.  It seems like you're also working with a mobile AMD GPU.  Hybrid, dual GPU laptops are wired in strange ways, so if you're expecting that you can plug a monitor into one of the outputs and get the discrete graphics output, there's a good chance it won't work.  You might be able to install everything, setup the Catalyst driver and VNC server in the guest with and emulated VGA, then add the Radeon, remove the emulated VGA, and hope you can blind boot the guest and connect.

Running with -vga none -nographic -vnc :0 gets rid of the terminal.:) The vnc window now says "This VM has no graphics display device."

Does this change anything? Should I try the vnc in guest solution? Would that solution even work if there is no graphics display device?

Have you installed a guest, installed a VNC server, installed Catalyst drivers, and have the VM working so that you can connect to that VNC server from the host?

Also when I plug in HDMI, arch mirrors my desktop automatically. (even with no X server running)

See comments about mobile/hybrid graphics.  That's a good hint that the discrete graphics isn't directly connected to the HDMI port.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2874 2014-10-02 08:27:33

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Errors when rebooting VM :

crashlog

This results in vertical tears in the screen .

>Hardware name: ASUS All Series/X99-DELUXE
Is it that fresh DDR4-based motherboard, requesting fresh CPUs like intel i7-5XXX? Don't they have some APICv2 or something?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2875 2014-10-02 14:15:13

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Denso wrote:

Errors when rebooting VM :

crashlog

This results in vertical tears in the screen .

>Hardware name: ASUS All Series/X99-DELUXE
Is it that fresh DDR4-based motherboard, requesting fresh CPUs like intel i7-5XXX? Don't they have some APICv2 or something?

Yes , it is . I use i7-5930k with it . It's giving a headache .

My previous Z77 board worked flawlessly , I thought X99 would be the same if not easier to setup .

Now , every time I reboot a VM with GPU passed-through to it , it ceashes the whole host . I tried the solution Alex suggested and it worked for a couple reboots , but then it returned to crashing the host .

Any help would be appreciated folks !

Offline

#2876 2014-10-02 17:42:42

GrigoryPtashko
Member
Registered: 2014-10-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello.

Here's my setup:


1. Supermicro X9DRFF-iG+/-7G+/-iTG+/-7TG+/X9DRFF-iG+/-7G+/-iTG+/-7TG+, BIOS 3.0 07/29/2013
2. Custom built kernel 3.16.3 (taken from kernel.org) + applied patches from OP
3. Cmdline intel_iommu=on,igfx_off pci_stub.ids=10de:1005,10de:0e1a4,1002:0b0c,1002:aac8 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off
4. Latest Qemu from git built by myself.
5. Latest Seabios from git built by myself.


With this I've managed to passthrough an NVIDIA GTX TITAN. Everything works fine.
Now I'm trying to passthrough the AMD FirePro W8100 (and W9100). I set up a VM with emulated VGA and with passedthrough W8100. I set up the driver. The driver sees the W8100. But it is with yellow sign since it is not primary.

And now the problem:

I turn on the VM with the W8100 as the primary adapter and I get a kernel panic on the host (!).

Here's VM:

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 16000 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -nodefaults \ -nographic \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=83:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=83:00.1,bus=root.1,addr=00.1 \ -net bridge,br=br0 -net nic,model=virtio,macaddr=52:54:00:12:34:60 \ -drive file=/vmstorage/win7test2kvm-firepro-node3.img,id=disk,format=raw,if=virtio \ -mon chardev=monitor0 \ -chardev socket,id=monitor0,path=/home/admin/tmp/win7test2kvm-firepro-node3.monitor,nowait,server \

Here's the kernel panic:


... [ 2013.702012] vfio_ecap_init: 0000:83:00.0 hiding ecap 0x19@0x270 [ 2013.707946] vfio_ecap_init: 0000:83:00.0 hiding ecap 0x1b@0x2d0 [ 2020.563698] kvm: zapping shadow pages for mmio generation wraparound [ 2028.031369] vfio-pci 0000:83:00.0: irq 146 for MSI/MSI-X [ 2050.912282] dmar: DRHD: handling fault status reg 40 .... lots of the same dmar: DRHD: handling fault status reg 40 here [ 2064.032487] dmar: DRHD: handling fault status reg 40 [ 2064.033072] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 31 [ 2064.033074] CPU: 31 PID: 4197 Comm: qemu-system-x86 Tainted: G W 3.16.3 #1 [ 2064.033075] Hardware name: Supermicro X9DRFF-iG+/-7G+/-iTG+/-7TG+/X9DRFF-iG+/-7G+/-iTG+/-7TG+, BIOS 3.0 07/29/2013 [ 2064.033077] 0000000000000000 000000006cf33090 ffff88307fde6c10 ffffffff817eb37c [ 2064.033079] ffffffff81ab62c0 ffff88307fde6c90 ffffffff817e3c66 0000000000000010 [ 2064.033080] ffff88307fde6ca0 ffff88307fde6c40 000000006cf33090 0000000000000000 [ 2064.033080] Call Trace: [ 2064.033087] <NMI> [<ffffffff817eb37c>] dump_stack+0x45/0x56 [ 2064.033089] [<ffffffff817e3c66>] panic+0xd8/0x20c [ 2064.033093] [<ffffffff81121250>] ? restart_watchdog_hrtimer+0x50/0x50 [ 2064.033095] [<ffffffff81121312>] watchdog_overflow_callback+0xc2/0xd0 [ 2064.033097] [<ffffffff8115dd8d>] __perf_event_overflow+0x9d/0x250 [ 2064.033098] [<ffffffff8115e884>] perf_event_overflow+0x14/0x20 [ 2064.033102] [<ffffffff81032bcd>] intel_pmu_handle_irq+0x1fd/0x410 [ 2064.033105] [<ffffffff811a3381>] ? unmap_kernel_range_noflush+0x11/0x20 [ 2064.033110] [<ffffffff81436974>] ? ghes_copy_tofrom_phys+0x124/0x210 [ 2064.033113] [<ffffffff81029f5b>] perf_event_nmi_handler+0x2b/0x50 [ 2064.033115] [<ffffffff81017e90>] nmi_handle+0x90/0x130 [ 2064.033116] [<ffffffff810184ae>] default_do_nmi+0xde/0x140 [ 2064.033117] [<ffffffff81018598>] do_nmi+0x88/0xc0 [ 2064.033120] [<ffffffff817f7671>] end_repeat_nmi+0x1e/0x2e [ 2064.033124] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2064.033125] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2064.033127] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2064.033129] <<EOE>> [<ffffffff81670536>] qi_flush_dev_iotlb+0x86/0xd0 [ 2064.033130] [<ffffffff81672534>] iommu_flush_dev_iotlb+0xa4/0xd0 [ 2064.033132] [<ffffffff81672612>] iommu_flush_iotlb_psi+0xb2/0xe0 [ 2064.033133] [<ffffffff81674fce>] intel_iommu_unmap+0x1ce/0x1e0 [ 2064.033135] [<ffffffff81668340>] iommu_unmap+0xb0/0x190 [ 2064.033140] [<ffffffff81599553>] vfio_remove_dma+0xc3/0x1a0 [ 2064.033142] [<ffffffff817f36e2>] ? mutex_lock+0x12/0x2f [ 2064.033144] [<ffffffff81599c71>] vfio_iommu_type1_ioctl+0x3e1/0xa20 [ 2064.033163] [<ffffffffa0005486>] ? kvm_set_memory_region+0x36/0x40 [kvm] [ 2064.033169] [<ffffffffa0005902>] ? kvm_vm_ioctl+0x472/0x730 [kvm] [ 2064.033171] [<ffffffff81597769>] vfio_fops_unl_ioctl+0x79/0x2b0 [ 2064.033174] [<ffffffff811ef660>] do_vfs_ioctl+0x2e0/0x4a0 [ 2064.033175] [<ffffffff811ef8a1>] SyS_ioctl+0x81/0xa0 [ 2064.033177] [<ffffffff8110ec66>] ? __audit_syscall_exit+0x1f6/0x2a0 [ 2064.033179] [<ffffffff817f5369>] system_call_fastpath+0x16/0x1b [ 2065.078698] Shutting down cpus with NMI [ 2065.082550] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 2065.092720] drm_kms_helper: panic occurred, switching back to text console [ 2065.402288] ------------[ cut here ]------------ [ 2065.406910] kernel BUG at mm/vmalloc.c:1320! [ 2065.411178] invalid opcode: 0000 [#1] SMP [ 2065.415312] Modules linked in: tun ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables sg rpcsec_gss_krb5 nls_utf8 mlx4_ib ib_sa ib_mad ib_core ib_addr x86_pkg_temp_thermal coretemp crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel iTCO_wdt iTCO_vendor_support ghash_clmulni_intel mgag200 syscopyarea sysfillrect sysimgblt aesni_intel ttm lrw ahci gf128mul drm_kms_helper mlx4_core glue_helper libahci mei_me drm ablk_helper sb_edac lpc_ich ioatdma cryptd ipmi_si libata edac_core pcspkr mei shpchp mfd_core i2c_i801 wmi dca ipmi_msghandler binfmt_misc kvm_intel nfsd kvm [ 2065.469011] CPU: 31 PID: 4197 Comm: qemu-system-x86 Tainted: G W 3.16.3 #1 [ 2065.476839] Hardware name: Supermicro X9DRFF-iG+/-7G+/-iTG+/-7TG+/X9DRFF-iG+/-7G+/-iTG+/-7TG+, BIOS 3.0 07/29/2013 [ 2065.487183] task: ffff882ff4e8d220 ti: ffff882fee33c000 task.ti: ffff882fee33c000 [ 2065.494665] RIP: 0010:[<ffffffff811a2980>] [<ffffffff811a2980>] __get_vm_area_node+0x150/0x160 [ 2065.503388] RSP: 0018:ffff88307fde65f8 EFLAGS: 00010006 [ 2065.508701] RAX: 0000000080110000 RBX: 00000000ffffffff RCX: ffffc90000000000 [ 2065.515830] RDX: 0000000000000022 RSI: 0000000000000001 RDI: 0000000000002000 [ 2065.522964] RBP: ffff88307fde6658 R08: ffffe8ffffffffff R09: 00000000ffffffff [ 2065.530102] R10: ffffffffa08bfe84 R11: ffff882ff2eb2218 R12: 0000000000001800 [ 2065.537236] R13: 0000000000300000 R14: 00000000000080d2 R15: ffffea0001dbd780 [ 2065.544372] FS: 00007f23dd4b69c0(0000) GS:ffff88307fde0000(0000) knlGS:0000000000000000 [ 2065.552461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2065.558210] CR2: 00007f23de4693f8 CR3: 0000002ffa139000 CR4: 00000000000427e0 [ 2065.565338] Stack: [ 2065.567348] ffffffff811a4040 ffffffff000080d2 ffffffffa05022b9 8000000000000163 [ 2065.574806] 000080d200000001 ffff88307fde675d 000000006cf33090 ffff882ff4d73de0 [ 2065.582262] ffff882ff2eb21f8 0000000000300000 0000000000000080 ffffea0001dbd780 [ 2065.589719] Call Trace: [ 2065.592163] <NMI> [ 2065.594089] [<ffffffff811a4040>] ? __vmalloc_node_range+0x80/0x280 [ 2065.600586] [<ffffffffa05022b9>] ? ttm_tt_init+0x69/0xb0 [ttm] [ 2065.606506] [<ffffffff811a4281>] __vmalloc+0x41/0x50 [ 2065.611555] [<ffffffffa05022b9>] ? ttm_tt_init+0x69/0xb0 [ttm] [ 2065.617474] [<ffffffffa05022b9>] ttm_tt_init+0x69/0xb0 [ttm] [ 2065.623233] [<ffffffffa08bfea8>] mgag200_ttm_tt_create+0x58/0x90 [mgag200] [ 2065.630204] [<ffffffffa0502a5d>] ttm_bo_add_ttm+0x8d/0xc0 [ttm] [ 2065.636213] [<ffffffffa05040e1>] ttm_bo_handle_move_mem+0x571/0x5b0 [ttm] [ 2065.643096] [<ffffffffa0504756>] ? ttm_bo_mem_space+0x116/0x340 [ttm] [ 2065.649633] [<ffffffffa0504e47>] ttm_bo_validate+0x247/0x260 [ttm] [ 2065.655919] [<ffffffff8105e959>] ? iounmap+0x79/0xa0 [ 2065.660982] [<ffffffff81050059>] ? native_safe_x2apic_wait_icr_idle+0x9/0x10 [ 2065.668119] [<ffffffffa08c0522>] mgag200_bo_push_sysram+0x82/0xe0 [mgag200] [ 2065.675160] [<ffffffffa08bba95>] mga_crtc_do_set_base.isra.8.constprop.20+0x85/0x470 [mgag200] [ 2065.683857] [<ffffffffa08bcebb>] mga_crtc_mode_set+0x103b/0x2160 [mgag200] [ 2065.690836] [<ffffffff8139b088>] ? __const_udelay+0x28/0x30 [ 2065.696508] [<ffffffffa01d6939>] drm_crtc_helper_set_mode+0x2e9/0x520 [drm_kms_helper] [ 2065.704519] [<ffffffffa01d76bf>] drm_crtc_helper_set_config+0x87f/0xaa0 [drm_kms_helper] [ 2065.712709] [<ffffffffa03ff6d1>] drm_mode_set_config_internal+0x61/0xe0 [drm] [ 2065.719935] [<ffffffffa01d9ca3>] restore_fbdev_mode+0xb3/0xe0 [drm_kms_helper] [ 2065.727243] [<ffffffffa01d9ea5>] drm_fb_helper_force_kernel_mode+0x75/0xb0 [drm_kms_helper] [ 2065.735680] [<ffffffffa01dabb9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [ 2065.743084] [<ffffffff8109b7ac>] notifier_call_chain+0x4c/0x70 [ 2065.749005] [<ffffffff8109b80a>] atomic_notifier_call_chain+0x1a/0x20 [ 2065.755534] [<ffffffff817e3c93>] panic+0x105/0x20c [ 2065.760416] [<ffffffff81121250>] ? restart_watchdog_hrtimer+0x50/0x50 [ 2065.766936] [<ffffffff81121312>] watchdog_overflow_callback+0xc2/0xd0 [ 2065.773456] [<ffffffff8115dd8d>] __perf_event_overflow+0x9d/0x250 [ 2065.779630] [<ffffffff8115e884>] perf_event_overflow+0x14/0x20 [ 2065.785552] [<ffffffff81032bcd>] intel_pmu_handle_irq+0x1fd/0x410 [ 2065.791733] [<ffffffff811a3381>] ? unmap_kernel_range_noflush+0x11/0x20 [ 2065.798428] [<ffffffff81436974>] ? ghes_copy_tofrom_phys+0x124/0x210 [ 2065.804869] [<ffffffff81029f5b>] perf_event_nmi_handler+0x2b/0x50 [ 2065.811051] [<ffffffff81017e90>] nmi_handle+0x90/0x130 [ 2065.816270] [<ffffffff810184ae>] default_do_nmi+0xde/0x140 [ 2065.821836] [<ffffffff81018598>] do_nmi+0x88/0xc0 [ 2065.826623] [<ffffffff817f7671>] end_repeat_nmi+0x1e/0x2e [ 2065.832102] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2065.837930] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2065.843765] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2065.849589] <<EOE>> [ 2065.851687] [<ffffffff81670536>] qi_flush_dev_iotlb+0x86/0xd0 [ 2065.857733] [<ffffffff81672534>] iommu_flush_dev_iotlb+0xa4/0xd0 [ 2065.863827] [<ffffffff81672612>] iommu_flush_iotlb_psi+0xb2/0xe0 [ 2065.869915] [<ffffffff81674fce>] intel_iommu_unmap+0x1ce/0x1e0 [ 2065.875835] [<ffffffff81668340>] iommu_unmap+0xb0/0x190 [ 2065.881142] [<ffffffff81599553>] vfio_remove_dma+0xc3/0x1a0 [ 2065.886796] [<ffffffff817f36e2>] ? mutex_lock+0x12/0x2f [ 2065.892103] [<ffffffff81599c71>] vfio_iommu_type1_ioctl+0x3e1/0xa20 [ 2065.898463] [<ffffffffa0005486>] ? kvm_set_memory_region+0x36/0x40 [kvm] [ 2065.905252] [<ffffffffa0005902>] ? kvm_vm_ioctl+0x472/0x730 [kvm] [ 2065.911429] [<ffffffff81597769>] vfio_fops_unl_ioctl+0x79/0x2b0 [ 2065.917428] [<ffffffff811ef660>] do_vfs_ioctl+0x2e0/0x4a0 [ 2065.922907] [<ffffffff811ef8a1>] SyS_ioctl+0x81/0xa0 [ 2065.927955] [<ffffffff8110ec66>] ? __audit_syscall_exit+0x1f6/0x2a0 [ 2065.934300] [<ffffffff817f5369>] system_call_fastpath+0x16/0x1b [ 2065.940299] Code: 00 00 00 0f bd cf 83 c1 01 83 f9 0c 0f 4c c8 b0 13 83 f9 13 0f 4f c8 49 d3 e4 e9 fd fe ff ff 4c 89 ff e8 f4 b3 01 00 31 c0 eb b3 <0f> 0b 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 [ 2065.960268] RIP [<ffffffff811a2980>] __get_vm_area_node+0x150/0x160 [ 2065.966631] RSP <ffff88307fde65f8>

I've also tried the romfile option with the BIOS dumped with ATIFlash (FreeDOS bootable USB stick with ATIFlash). But it did not change anything.

I googled but I did not find anyone's experience with W8100 and W9100 GPUs. So I'm confused on where to move.. I've read pretty much info but I'm really stuck. Any help is really appriciated.

PS I have to say that I actually don't know which set of options from my cmdline actually made the GTX TITAN passthrough possible. I've been reading this post and Alex Williamson's vfio blog and adding stuff and experimenting. After another try I've got a working VM and stopped smile

Thank you,
Grigory.

Last edited by GrigoryPtashko (2014-10-02 17:46:41)

Offline

#2877 2014-10-03 11:04:00

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am running a similar setup than OP (kvm vga passthrough vfio) based on Ubuntu 14.04 with kernel 3.13 and KVM 2.0. I have a weird similar to the one OP mentioned. My sound sometimes "lags". It is completely chopped and make the games lag. It only occurs on games. Video playback and music are fine. It happens both with audio through HDMI and with a dedicated USB sound card passed through (along with keyboard / mouse).

I noticed OP fixed this (in his 2Nd post), however the fix only applies on AMD board (I tried it, just to be sure), and I have an Intel one.

Anyone has this same issue or knows how to fix it?

Thanks in advance

PS: talking about sound, I have another issue. I am pretty sure "this is not a bug, this a feature", however I'll ask. When playing a Windows game, if I change input via an HDMI switch (eg: I switch back to linux), then when I go back to the game there is no longer sound. Sounds still works in Windows if I ALT + TAB, but nothing in game. I have to quit and relaunch. Happens also with an USB sound card when I switch back to Linux. If someone has as fix, I'll love to know about it.

EDIT: actually this issue is really software related. Not longer an issue then. For example, Watch_Dogs does it, while Middle Earth: Battle for Mordor doesn't.

Last edited by Nesousx (2014-10-03 16:34:01)

Offline

#2878 2014-10-03 16:26:14

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Hi,

I am running a similar setup than OP (kvm vga passthrough vfio) based on Ubuntu 14.04 with kernel 3.13 and KVM 2.0. I have a weird similar to the one OP mentioned. My sound sometimes "lags". It is completely chopped and make the games lag. It only occurs on games. Video playback and music are fine. It happens both with audio through HDMI and with a dedicated USB sound card passed through (along with keyboard / mouse).

I noticed OP fixed this (in his 2Nd post), however the fix only applies on AMD board (I tried it, just to be sure), and I have an Intel one.

Anyone has this same issue or knows how to fix it?

Thanks in advance

PS: talking about sound, I have another issue. I am pretty sure "this is not a bug, this a feature", however I'll ask. When playing a Windows game, if I change input via an HDMI switch (eg: I switch back to linux), then when I go back to the game there is no longer sound. Sounds still works in Windows if I ALT + TAB, but nothing in game. I have to quit and relaunch. Happens also with an USB sound card when I switch back to Linux. If someone has as fix, I'll love to know about it.


If you have a dedicated USB sound card, you can use it, well, dedicated to linux, without getting linux sound out of it. I think that should help your latest issue. And maybe the former too.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2879 2014-10-03 16:31:15

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Nesousx wrote:

Hi,

I am running a similar setup than OP (kvm vga passthrough vfio) based on Ubuntu 14.04 with kernel 3.13 and KVM 2.0. I have a weird similar to the one OP mentioned. My sound sometimes "lags". It is completely chopped and make the games lag. It only occurs on games. Video playback and music are fine. It happens both with audio through HDMI and with a dedicated USB sound card passed through (along with keyboard / mouse).

I noticed OP fixed this (in his 2Nd post), however the fix only applies on AMD board (I tried it, just to be sure), and I have an Intel one.

Anyone has this same issue or knows how to fix it?

Thanks in advance

PS: talking about sound, I have another issue. I am pretty sure "this is not a bug, this a feature", however I'll ask. When playing a Windows game, if I change input via an HDMI switch (eg: I switch back to linux), then when I go back to the game there is no longer sound. Sounds still works in Windows if I ALT + TAB, but nothing in game. I have to quit and relaunch. Happens also with an USB sound card when I switch back to Linux. If someone has as fix, I'll love to know about it.


If you have a dedicated USB sound card, you can use it, well, dedicated to linux, without getting linux sound out of it. I think that should help your latest issue. And maybe the former too.

I have been doing this for months, but it doesn't help at all. I am back with full sound over HDMI and will sell the sound card.

Offline

#2880 2014-10-03 16:33:56

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
Duelist wrote:
Nesousx wrote:

Hi,

I am running a similar setup than OP (kvm vga passthrough vfio) based on Ubuntu 14.04 with kernel 3.13 and KVM 2.0. I have a weird similar to the one OP mentioned. My sound sometimes "lags". It is completely chopped and make the games lag. It only occurs on games. Video playback and music are fine. It happens both with audio through HDMI and with a dedicated USB sound card passed through (along with keyboard / mouse).

I noticed OP fixed this (in his 2Nd post), however the fix only applies on AMD board (I tried it, just to be sure), and I have an Intel one.

Anyone has this same issue or knows how to fix it?

Thanks in advance

PS: talking about sound, I have another issue. I am pretty sure "this is not a bug, this a feature", however I'll ask. When playing a Windows game, if I change input via an HDMI switch (eg: I switch back to linux), then when I go back to the game there is no longer sound. Sounds still works in Windows if I ALT + TAB, but nothing in game. I have to quit and relaunch. Happens also with an USB sound card when I switch back to Linux. If someone has as fix, I'll love to know about it.


If you have a dedicated USB sound card, you can use it, well, dedicated to linux, without getting linux sound out of it. I think that should help your latest issue. And maybe the former too.

I have been doing this for months, but it doesn't help at all. I am back with full sound over HDMI and will sell the sound card.


Durr, i've meant, dedicated to VM, not linux. Wasn't too fast to edit the message.
So, like, linux sound getting out of HDMI, VM sound getting out of USB.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2881 2014-10-03 16:38:18

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Duelist, I have tried many combinations, with one or several (one per host and one per VM) cards... always the same bug happen (the first issue I was talking about). The sound is choppy then the game lag. I am thinking that this might not be sound related. Sound being choppy is probably just a side effect, but where to look? Found nothing relevant yet, in logs. It seems to happen randomly too.

About my 2nd issue, it is no longer an issue since it is software related. I edited my post. I have been playing one game all the time that causes the bug, and I remember I had it before on other games too and then thought it was system wide before testing for real.

Last edited by Nesousx (2014-10-03 16:48:52)

Offline

#2882 2014-10-03 17:16:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

@ Duelist, I have tried many combinations, with one or several (one per host and one per VM) cards... always the same bug happen (the first issue I was talking about). The sound is choppy then the game lag. I am thinking that this might not be sound related. Sound being choppy is probably just a side effect, but where to look? Found nothing relevant yet, in logs. It seems to happen randomly too.

Audio might be fairly sensitive to interrupt latency.  Have you tried my instructions here for making Windows use MSI interrupts for the audio device?  I've done this successfully for both AMD and NVIDIA audio functions on Win8.  Pinning vCPUs and using hugepages can also help latency issues.  You can also pin the host interrupt for the device to a host CPU not used by the VM to try to further improve latency.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2883 2014-10-03 20:20:56

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is probably a question for redger, but anyone with an idea can chime right in. 

I've had a working Win7 passthrough for quite some time, but have wanted to migrate over from qemu commandline to libvirt for a variety of reasons.  I followed the excellent guide on page 94 of the thread, but Windows 7 gets caught in the Windows cannot start, repair or reboot loop.  Interestingly, if one tries to restart the VM from inside, it bluescreens. 

I can post the details of the qemu script and the xml if needed, but there is nothing remarkable.  Using kernel 3.15.5 and qemu 2.0.0, libvirt 1.2.9.  I use the virtio drivers for the hd and network.  Other VMs (new to libvirt, not migrated) seem to work OK, so I assume this is a Windows problem having to do with a "change" in hardware between qemu commandline and libvirt.

Presumably this is a common problem with migrations and the unpalatable answer is to reinstall Windows under libvirt, but I'd rather avoid that. (If I did that I'd probably also change from q35 to 440fx and give OVMF a try.)  Is there a VM preparation maneuver to make the transition less likely to fail?  A registry hack? Google fails to enlighten in this case.

Offline

#2884 2014-10-03 20:42:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

This is probably a question for redger, but anyone with an idea can chime right in. 

I've had a working Win7 passthrough for quite some time, but have wanted to migrate over from qemu commandline to libvirt for a variety of reasons.  I followed the excellent guide on page 94 of the thread, but Windows 7 gets caught in the Windows cannot start, repair or reboot loop.  Interestingly, if one tries to restart the VM from inside, it bluescreens. 

I can post the details of the qemu script and the xml if needed, but there is nothing remarkable.  Using kernel 3.15.5 and qemu 2.0.0, libvirt 1.2.9.  I use the virtio drivers for the hd and network.  Other VMs (new to libvirt, not migrated) seem to work OK, so I assume this is a Windows problem having to do with a "change" in hardware between qemu commandline and libvirt.

Presumably this is a common problem with migrations and the unpalatable answer is to reinstall Windows under libvirt, but I'd rather avoid that. (If I did that I'd probably also change from q35 to 440fx and give OVMF a try.)  Is there a VM preparation maneuver to make the transition less likely to fail?  A registry hack? Google fails to enlighten in this case.

It should be entirely possible to convert from qemu commandline to libvirt in a compatible way, but the differences can be subtle.  Often people don't specify addresses for every device on the commandline and libvirt is likely to re-order the device declaration, so at a minimum you'd want to figure out the PCI address for each device and specify it explicitly in the xml.  Windows gets confused enough when devices move around, but if you end up changing devices at the same time, you're probably in for a world of hurt.  You probably also want to switch to a dummy disk image or snapshot so that you can make a few attempts comparing the libvirt generated commandline to your own before you risk the real VM disk image.  libvirt's restrictive support for q35 just makes the conversion all that much harder too.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2885 2014-10-03 21:24:51

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Nesousx wrote:

@ Duelist, I have tried many combinations, with one or several (one per host and one per VM) cards... always the same bug happen (the first issue I was talking about). The sound is choppy then the game lag. I am thinking that this might not be sound related. Sound being choppy is probably just a side effect, but where to look? Found nothing relevant yet, in logs. It seems to happen randomly too.

Audio might be fairly sensitive to interrupt latency.  Have you tried my instructions here for making Windows use MSI interrupts for the audio device?  I've done this successfully for both AMD and NVIDIA audio functions on Win8.  Pinning vCPUs and using hugepages can also help latency issues.  You can also pin the host interrupt for the device to a host CPU not used by the VM to try to further improve latency.

Thanks. I'll give it a shot tomorrow.

Last edited by Nesousx (2014-10-03 21:28:36)

Offline

#2886 2014-10-03 21:32:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
aw wrote:
Nesousx wrote:

@ Duelist, I have tried many combinations, with one or several (one per host and one per VM) cards... always the same bug happen (the first issue I was talking about). The sound is choppy then the game lag. I am thinking that this might not be sound related. Sound being choppy is probably just a side effect, but where to look? Found nothing relevant yet, in logs. It seems to happen randomly too.

Audio might be fairly sensitive to interrupt latency.  Have you tried my instructions here for making Windows use MSI interrupts for the audio device?  I've done this successfully for both AMD and NVIDIA audio functions on Win8.  Pinning vCPUs and using hugepages can also help latency issues.  You can also pin the host interrupt for the device to a host CPU not used by the VM to try to further improve latency.

Thanks. It looks like my device doesn't support MSI. Sorry if I missed something from your post, I am about to go to bed, and quite tired.

sudo lspci -v -s 1:00.0 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] (prog-if 00 [VGA controller]) Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+

We can notice an MSI- instead of MSI+.

The fact that it has an MSI capability means that it supports MSI.  -/+ tells you whether it's enabled.  However, what I'm suggesting is that you assign and enable MSI interrupts for the audio function in order to reduce interrupt latency and perhaps improve the choppy audio.  What you've listed here is the GPU function.  Enabling MSI for both is even better, but MSI on the GPU is typically enabled by default on AMD.

Last edited by aw (2014-10-03 21:33:29)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2887 2014-10-03 23:38:17

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw Thanks; I'll give it a try. However I'm unsure about something; you mentioned a

libvirt generated commandline

I've got the xml (which has some autogenerated PCI lines in it) eg.

<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>

for non-vfio passthrough devices, like the pci-bridge. I presume these are what you mean I should compare to their equivalents somewhere in Windows' Device Manager?

I've also got the qemu command line, eg.

 -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 

which just gets translated as

<qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1' 

which look identical so are presumably not the problem.

Offline

#2888 2014-10-04 06:36:54

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi .

I always had this question in mind :

Does using PLX chips on motherboards to expand PCI-E lanes make it harder to passthorugh devices installed in such PCI-E slots ?

I used to have ASRock Z77 Extreme 11 which has PLX chips , whenever I tried to passthrough GPUs , the whole host hanged .

I might opt for an ASUS X99-E WS , which has PLX chips as well . Can anyone confirm whether it is possible or not ?

Offline

#2889 2014-10-04 11:31:15

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Nesousx wrote:
aw wrote:

Audio might be fairly sensitive to interrupt latency.  Have you tried my instructions here for making Windows use MSI interrupts for the audio device?  I've done this successfully for both AMD and NVIDIA audio functions on Win8.  Pinning vCPUs and using hugepages can also help latency issues.  You can also pin the host interrupt for the device to a host CPU not used by the VM to try to further improve latency.

Thanks. It looks like my device doesn't support MSI. Sorry if I missed something from your post, I am about to go to bed, and quite tired.

sudo lspci -v -s 1:00.0 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] (prog-if 00 [VGA controller]) Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+

We can notice an MSI- instead of MSI+.

The fact that it has an MSI capability means that it supports MSI.  -/+ tells you whether it's enabled.  However, what I'm suggesting is that you assign and enable MSI interrupts for the audio function in order to reduce interrupt latency and perhaps improve the choppy audio.  What you've listed here is the GPU function.  Enabling MSI for both is even better, but MSI on the GPU is typically enabled by default on AMD.

I read too fast yesterday, and you replied before my edit. smile

I did some change and now MSI is enabled. I'll try and let you know.

sudo lspci -v -s 1:00.0 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Device e210 Flags: bus master, fast devsel, latency 0, IRQ 59 Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f7b00000 (64-bit, non-prefetchable) [size=256K] I/O ports at e000 [size=256] Expansion ROM at f7b40000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] #13 Capabilities: [2d0] #1b Kernel driver in use: vfio-pci

Offline

#2890 2014-10-04 13:46:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GrigoryPtashko wrote:

Hello.

Here's my setup:


1. Supermicro X9DRFF-iG+/-7G+/-iTG+/-7TG+/X9DRFF-iG+/-7G+/-iTG+/-7TG+, BIOS 3.0 07/29/2013
2. Custom built kernel 3.16.3 (taken from kernel.org) + applied patches from OP
3. Cmdline intel_iommu=on,igfx_off pci_stub.ids=10de:1005,10de:0e1a4,1002:0b0c,1002:aac8 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off
4. Latest Qemu from git built by myself.
5. Latest Seabios from git built by myself.


With this I've managed to passthrough an NVIDIA GTX TITAN. Everything works fine.
Now I'm trying to passthrough the AMD FirePro W8100 (and W9100). I set up a VM with emulated VGA and with passedthrough W8100. I set up the driver. The driver sees the W8100. But it is with yellow sign since it is not primary.

And now the problem:

I turn on the VM with the W8100 as the primary adapter and I get a kernel panic on the host (!).

Note that the quirks we currently have in place are designed and developed for Radeon cards.  We have no idea if they're relevant or correct for FirePro cards.  Quadro cards are similar, they also do not work in primary VGA mode using the GeForce quirks.  Nvidia does however support secondary assignment for those cards, so we don't bother to come up with quirks for them.

Here's the kernel panic:


... [ 2013.702012] vfio_ecap_init: 0000:83:00.0 hiding ecap 0x19@0x270 [ 2013.707946] vfio_ecap_init: 0000:83:00.0 hiding ecap 0x1b@0x2d0 [ 2020.563698] kvm: zapping shadow pages for mmio generation wraparound [ 2028.031369] vfio-pci 0000:83:00.0: irq 146 for MSI/MSI-X [ 2050.912282] dmar: DRHD: handling fault status reg 40 .... lots of the same dmar: DRHD: handling fault status reg 40 here

dmar faults are usually two lines, one indicating the status register, which you've included and another indicating the faulting device, type of fault, and address, not included.  The not included line is generally much more insightful.

[ 2064.032487] dmar: DRHD: handling fault status reg 40 [ 2064.033072] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 31 [ 2064.033074] CPU: 31 PID: 4197 Comm: qemu-system-x86 Tainted: G W 3.16.3 #1 [ 2064.033075] Hardware name: Supermicro X9DRFF-iG+/-7G+/-iTG+/-7TG+/X9DRFF-iG+/-7G+/-iTG+/-7TG+, BIOS 3.0 07/29/2013 [ 2064.033077] 0000000000000000 000000006cf33090 ffff88307fde6c10 ffffffff817eb37c [ 2064.033079] ffffffff81ab62c0 ffff88307fde6c90 ffffffff817e3c66 0000000000000010 [ 2064.033080] ffff88307fde6ca0 ffff88307fde6c40 000000006cf33090 0000000000000000 [ 2064.033080] Call Trace: [ 2064.033087] <NMI> [<ffffffff817eb37c>] dump_stack+0x45/0x56 [ 2064.033089] [<ffffffff817e3c66>] panic+0xd8/0x20c [ 2064.033093] [<ffffffff81121250>] ? restart_watchdog_hrtimer+0x50/0x50 [ 2064.033095] [<ffffffff81121312>] watchdog_overflow_callback+0xc2/0xd0 [ 2064.033097] [<ffffffff8115dd8d>] __perf_event_overflow+0x9d/0x250 [ 2064.033098] [<ffffffff8115e884>] perf_event_overflow+0x14/0x20 [ 2064.033102] [<ffffffff81032bcd>] intel_pmu_handle_irq+0x1fd/0x410 [ 2064.033105] [<ffffffff811a3381>] ? unmap_kernel_range_noflush+0x11/0x20 [ 2064.033110] [<ffffffff81436974>] ? ghes_copy_tofrom_phys+0x124/0x210 [ 2064.033113] [<ffffffff81029f5b>] perf_event_nmi_handler+0x2b/0x50 [ 2064.033115] [<ffffffff81017e90>] nmi_handle+0x90/0x130 [ 2064.033116] [<ffffffff810184ae>] default_do_nmi+0xde/0x140 [ 2064.033117] [<ffffffff81018598>] do_nmi+0x88/0xc0 [ 2064.033120] [<ffffffff817f7671>] end_repeat_nmi+0x1e/0x2e [ 2064.033124] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2064.033125] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2064.033127] [<ffffffff816700d6>] ? qi_submit_sync+0x196/0x400 [ 2064.033129] <<EOE>> [<ffffffff81670536>] qi_flush_dev_iotlb+0x86/0xd0 [ 2064.033130] [<ffffffff81672534>] iommu_flush_dev_iotlb+0xa4/0xd0 [ 2064.033132] [<ffffffff81672612>] iommu_flush_iotlb_psi+0xb2/0xe0 [ 2064.033133] [<ffffffff81674fce>] intel_iommu_unmap+0x1ce/0x1e0 [ 2064.033135] [<ffffffff81668340>] iommu_unmap+0xb0/0x190 [ 2064.033140] [<ffffffff81599553>] vfio_remove_dma+0xc3/0x1a0 [ 2064.033142] [<ffffffff817f36e2>] ? mutex_lock+0x12/0x2f [ 2064.033144] [<ffffffff81599c71>] vfio_iommu_type1_ioctl+0x3e1/0xa20 [ 2064.033163] [<ffffffffa0005486>] ? kvm_set_memory_region+0x36/0x40 [kvm] [ 2064.033169] [<ffffffffa0005902>] ? kvm_vm_ioctl+0x472/0x730 [kvm] [ 2064.033171] [<ffffffff81597769>] vfio_fops_unl_ioctl+0x79/0x2b0 [ 2064.033174] [<ffffffff811ef660>] do_vfs_ioctl+0x2e0/0x4a0 [ 2064.033175] [<ffffffff811ef8a1>] SyS_ioctl+0x81/0xa0 [ 2064.033177] [<ffffffff8110ec66>] ? __audit_syscall_exit+0x1f6/0x2a0 [ 2064.033179] [<ffffffff817f5369>] system_call_fastpath+0x16/0x1b

It looks like the watchdog kicked in while we were trying to unmap some pages from the iommu and then the mga driver exploded, as it's prone to do.  What was going on with the VM at this point?  The intel-iommu code has had some issues with stalling the kernel during large unmaps, but it shouldn't be happening for a small, 16G VM.  You can try disabling the hardware watchdog to see if this is just that the unmap takes longer than the watchdog allows, we're only looking at ~50s from VM start to lockup.

In general though, I would not expect FirePro cards to work.  AFAIK, we don't have any evidence that would suggest that they necessarily use the same set of quirks as Radeon cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2891 2014-10-04 13:52:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hi .

I always had this question in mind :

Does using PLX chips on motherboards to expand PCI-E lanes make it harder to passthorugh devices installed in such PCI-E slots ?

I used to have ASRock Z77 Extreme 11 which has PLX chips , whenever I tried to passthrough GPUs , the whole host hanged .

I might opt for an ASUS X99-E WS , which has PLX chips as well . Can anyone confirm whether it is possible or not ?

I assume you mean PLX branded PCIe switches.  If they support ACS and aren't crap, then in theory they don't pose any issue for device assignment.  If they don't support ACS, it pretty much guarantees that you'll always need the ACS override patch.  In general, more components are just more opportunities for things to be broken.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2892 2014-10-04 15:05:27

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

Hi .

I always had this question in mind :

Does using PLX chips on motherboards to expand PCI-E lanes make it harder to passthorugh devices installed in such PCI-E slots ?

I used to have ASRock Z77 Extreme 11 which has PLX chips , whenever I tried to passthrough GPUs , the whole host hanged .

I might opt for an ASUS X99-E WS , which has PLX chips as well . Can anyone confirm whether it is possible or not ?

I assume you mean PLX branded PCIe switches.  If they support ACS and aren't crap, then in theory they don't pose any issue for device assignment.  If they don't support ACS, it pretty much guarantees that you'll always need the ACS override patch.  In general, more components are just more opportunities for things to be broken.

PLX advertises that most (all?) of their PEX chips support ACS . Maybe it was the motherboard that was causing issues , and not the PLX chip itself .

I've got a kernel panic just now :

[ 42.002633] irq 28: nobody cared (try booting with the "irqpoll" option) [ 42.002633] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 3.16.3-1-ARCH #1 [ 42.002633] Hardware name: ASUS All Series/X99-DELUXE, BIOS 0904 09/22/2014 [ 42.002633] 0000000000000000 e24784535cef3a79 ffff88085fc03ad8 ffffffff8152b3bc [ 42.002633] ffff8808323f9a00 ffff88085fc03b00 ffffffff810d02a2 ffff8808323f9a00 [ 42.002633] 0000000000000000 000000000000001c ffff88085fc03b38 ffffffff810d0657 [ 42.002633] Call Trace: [ 42.002633] <IRQ> [<ffffffff8152b3bc>] dump_stack+0x4d/0x6f [ 42.002633] [<ffffffff810d02a2>] __report_bad_irq+0x32/0xd0 [ 42.002633] [<ffffffff810d0657>] note_interrupt+0x257/0x2a0 [ 42.002633] [<ffffffff810cdbae>] handle_irq_event_percpu+0xae/0x1f0 [ 42.002633] [<ffffffff810cdd2d>] handle_irq_event+0x3d/0x60 [ 42.002633] [<ffffffff810d1281>] handle_fasteoi_irq+0x81/0x170 [ 42.002633] [<ffffffff8101717e>] handle_irq+0x1e/0x40 [ 42.002633] [<ffffffff81533bed>] do_IRQ+0x4d/0xe0 [ 42.002633] [<ffffffff81531bad>] common_interrupt+0x6d/0x6d [ 42.002633] [<ffffffff810b6710>] ? __wake_up_bit+0x40/0x60 [ 42.002633] [<ffffffff8114be63>] unlock_page+0x23/0x30 [ 42.002633] [<ffffffff8114cf3e>] page_endio+0x1e/0x60 [ 42.002633] [<ffffffff811ff892>] mpage_end_io+0x42/0x60 [ 42.002633] [<ffffffff8126f4db>] bio_endio+0x6b/0xa0 [ 42.002633] [<ffffffff812766a4>] blk_update_request+0x94/0x380 [ 42.002633] [<ffffffff812769ae>] blk_update_bidi_request+0x1e/0xa0 [ 42.002633] [<ffffffff81276d01>] blk_end_bidi_request+0x21/0x60 [ 42.002633] [<ffffffff81276d50>] blk_end_request+0x10/0x20 [ 42.002633] [<ffffffffa003238d>] scsi_io_completion+0xad/0x710 [scsi_mod] [ 42.002633] [<ffffffffa00293d2>] scsi_finish_command+0xa2/0xe0 [scsi_mod] [ 42.002633] [<ffffffffa00321fe>] scsi_softirq_done+0x10e/0x130 [scsi_mod] [ 42.002633] [<ffffffff8127d5eb>] blk_done_softirq+0x8b/0xb0 [ 42.002633] [<ffffffff810736c2>] __do_softirq+0xf2/0x2e0 [ 42.002633] [<ffffffff81073a06>] irq_exit+0x86/0xb0 [ 42.002633] [<ffffffff81533bf6>] do_IRQ+0x56/0xe0 [ 42.002633] [<ffffffff81531bad>] common_interrupt+0x6d/0x6d [ 42.002633] <EOI> [<ffffffff813e88bc>] ? cpuidle_enter_state+0x4c/0xc0 [ 42.002633] [<ffffffff813e8a17>] cpuidle_enter+0x17/0x20 [ 42.002633] [<ffffffff810b710f>] cpu_startup_entry+0x32f/0x520 [ 42.002633] [<ffffffff815216c4>] rest_init+0x84/0x90 [ 42.002633] [<ffffffff818f5fc9>] start_kernel+0x45e/0x47f [ 42.002633] [<ffffffff818f5120>] ? early_idt_handlers+0x120/0x120 [ 42.002633] [<ffffffff818f54d7>] x86_64_start_reservations+0x2a/0x2c [ 42.002633] [<ffffffff818f5626>] x86_64_start_kernel+0x14d/0x170 [ 42.002633] handlers: [ 42.002633] [<ffffffffa05c43e0>] vfio_intx_handler [vfio_pci] [ 42.002633] Disabling IRQ #28

It happened when I started copying large files from my SMB share to my VM's virtio qcow2 disk file , which crashes the VM alone and not other VMs or the host .

Thank you .

Last edited by Denso (2014-10-04 15:05:53)

Offline

#2893 2014-10-04 15:32:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I've got a kernel panic just now :

[ 42.002633] irq 28: nobody cared (try booting with the "irqpoll" option) [ 42.002633] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 3.16.3-1-ARCH #1 [ 42.002633] Hardware name: ASUS All Series/X99-DELUXE, BIOS 0904 09/22/2014 [ 42.002633] 0000000000000000 e24784535cef3a79 ffff88085fc03ad8 ffffffff8152b3bc [ 42.002633] ffff8808323f9a00 ffff88085fc03b00 ffffffff810d02a2 ffff8808323f9a00 [ 42.002633] 0000000000000000 000000000000001c ffff88085fc03b38 ffffffff810d0657 [ 42.002633] Call Trace: [ 42.002633] <IRQ> [<ffffffff8152b3bc>] dump_stack+0x4d/0x6f [ 42.002633] [<ffffffff810d02a2>] __report_bad_irq+0x32/0xd0 [ 42.002633] [<ffffffff810d0657>] note_interrupt+0x257/0x2a0 [ 42.002633] [<ffffffff810cdbae>] handle_irq_event_percpu+0xae/0x1f0 [ 42.002633] [<ffffffff810cdd2d>] handle_irq_event+0x3d/0x60 [ 42.002633] [<ffffffff810d1281>] handle_fasteoi_irq+0x81/0x170 [ 42.002633] [<ffffffff8101717e>] handle_irq+0x1e/0x40 [ 42.002633] [<ffffffff81533bed>] do_IRQ+0x4d/0xe0 [ 42.002633] [<ffffffff81531bad>] common_interrupt+0x6d/0x6d [ 42.002633] [<ffffffff810b6710>] ? __wake_up_bit+0x40/0x60 [ 42.002633] [<ffffffff8114be63>] unlock_page+0x23/0x30 [ 42.002633] [<ffffffff8114cf3e>] page_endio+0x1e/0x60 [ 42.002633] [<ffffffff811ff892>] mpage_end_io+0x42/0x60 [ 42.002633] [<ffffffff8126f4db>] bio_endio+0x6b/0xa0 [ 42.002633] [<ffffffff812766a4>] blk_update_request+0x94/0x380 [ 42.002633] [<ffffffff812769ae>] blk_update_bidi_request+0x1e/0xa0 [ 42.002633] [<ffffffff81276d01>] blk_end_bidi_request+0x21/0x60 [ 42.002633] [<ffffffff81276d50>] blk_end_request+0x10/0x20 [ 42.002633] [<ffffffffa003238d>] scsi_io_completion+0xad/0x710 [scsi_mod] [ 42.002633] [<ffffffffa00293d2>] scsi_finish_command+0xa2/0xe0 [scsi_mod] [ 42.002633] [<ffffffffa00321fe>] scsi_softirq_done+0x10e/0x130 [scsi_mod] [ 42.002633] [<ffffffff8127d5eb>] blk_done_softirq+0x8b/0xb0 [ 42.002633] [<ffffffff810736c2>] __do_softirq+0xf2/0x2e0 [ 42.002633] [<ffffffff81073a06>] irq_exit+0x86/0xb0 [ 42.002633] [<ffffffff81533bf6>] do_IRQ+0x56/0xe0 [ 42.002633] [<ffffffff81531bad>] common_interrupt+0x6d/0x6d [ 42.002633] <EOI> [<ffffffff813e88bc>] ? cpuidle_enter_state+0x4c/0xc0 [ 42.002633] [<ffffffff813e8a17>] cpuidle_enter+0x17/0x20 [ 42.002633] [<ffffffff810b710f>] cpu_startup_entry+0x32f/0x520 [ 42.002633] [<ffffffff815216c4>] rest_init+0x84/0x90 [ 42.002633] [<ffffffff818f5fc9>] start_kernel+0x45e/0x47f [ 42.002633] [<ffffffff818f5120>] ? early_idt_handlers+0x120/0x120 [ 42.002633] [<ffffffff818f54d7>] x86_64_start_reservations+0x2a/0x2c [ 42.002633] [<ffffffff818f5626>] x86_64_start_kernel+0x14d/0x170 [ 42.002633] handlers: [ 42.002633] [<ffffffffa05c43e0>] vfio_intx_handler [vfio_pci] [ 42.002633] Disabling IRQ #28

It happened when I started copying large files from my SMB share to my VM's virtio qcow2 disk file , which crashes the VM alone and not other VMs or the host .

Thank you .

This is not a panic.   This is the kernel reporting that IRQ28 is continuing to fire and none of the handlers for that IRQ (vfio) are claiming the interrupt.  The kernel is therefore announcing that it is disabling that interrupt.  Since vfio is the only handler and it thinks the device is masked (otherwise it would claim the interrupt), that might mean that INTx disable on the device doesn't actually work.  You can force vfio to mask at the APIC rather than the device by adding the nointxmask=1 parameter to the vfio-pci host kernel module.  This does however require that no other devices share the interrupt, which can make configuration difficult.  If the device supports MSI, you can also try to make it use that in the guest to avoid the problem.  See my post on the blog for more info and how to do that.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2894 2014-10-04 17:36:43

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Audio bug still here after enabling MSI. sad

Offline

#2895 2014-10-04 22:08:55

dwightjl
Member
Registered: 2013-10-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm getting the following error:

vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

Here is the list of devices in that group:

/sys/bus/pci/devices/0000:02:00.0/iommu_group/devices$ dir 0000:00:01.0 0000:00:01.1 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1

I would go ahead and list all of those devices in /etc/vfio-pci.cfg, but the problem with that is 01:00.0 and 01:00.1 are my primary display adapter. If I include that device in my vfio-pci.cfg file, it crashes the primary display when I try to bind it.

Is there a way to remove my primary display adapter from the iommu_group/devices list so that I don't get this error message? I've seen other people post this error message before, but never saw any instructions on how to change iommu_group device lists.

EDIT: After some more digging, it seems like I might need to patch something. I thought patches were only for my kernel if my primary display adapter was the integrated intel. I'll keep digging.

Last edited by dwightjl (2014-10-04 22:28:14)

Offline

#2896 2014-10-04 23:25:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwightjl wrote:

I'm getting the following error:

vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

Here is the list of devices in that group:

/sys/bus/pci/devices/0000:02:00.0/iommu_group/devices$ dir 0000:00:01.0 0000:00:01.1 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1

I would go ahead and list all of those devices in /etc/vfio-pci.cfg, but the problem with that is 01:00.0 and 01:00.1 are my primary display adapter. If I include that device in my vfio-pci.cfg file, it crashes the primary display when I try to bind it.

Is there a way to remove my primary display adapter from the iommu_group/devices list so that I don't get this error message? I've seen other people post this error message before, but never saw any instructions on how to change iommu_group device lists.

EDIT: After some more digging, it seems like I might need to patch something. I thought patches were only for my kernel if my primary display adapter was the integrated intel. I'll keep digging.

http://vfio.blogspot.com/2014/08/iommu- … d-out.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2897 2014-10-04 23:57:01

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So you guys have helped me get set up and it's been runing ok, but there is still a major issue. My machine hangs when I reboot the vm. I wouldn't have a problem with just rebooting my pc when i reboot the vm, but the vm is crashing when rebooting to install windows updates which is completely corrupting the machine. I've had this happen multiple times, so I need to try to find a solution. I've tried to find a solution but I just haven't been able to find anything. I wish there was a way to do a search from this thread.

My setup

Running the patched 3.16 kernel from the first page.

lspci -nn

... 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF110 [GeForce GTX 580] [10de:1080] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GF110 High Definition Audio Controller [10de:0e09] (rev a1) ... 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06) 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06) 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba] 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1] 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0) 00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 [8086:8c94] (rev d0) 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0) 00:1c.6 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 [8086:8c9c] (rev d0) 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6] 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97 LPC Controller [8086:8cc4] 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82] 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2] ... 02:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11) 05:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 06:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 06:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 06:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 06:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184] 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04) 08:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] 09:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02) 0a:00.0 Network controller [0280]: Qualcomm Atheros AR93xx Wireless Network Adapter [168c:0030] (rev 01) 0b:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02) 0c:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1142]

from syslinux.cfg

LABEL arch-qemu MENU LABEL Arch Qemu LINUX ../vmlinuz-linux-mainline APPEND root=/dev/sdb1 rootflags=subvol=__active/rootvol intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=10de:1080,10de:8086:0c0c modeset.nouveau=0 modeset.radeon=0 rw INITRD ../initramfs-linux-mainline.img

script for booting the vm:

sudo vfio-bind 0000:01:00.0 0000:01:00.1 0000:00:03.0 #qemu-system-x86_64 -enable-kvm -m 1024 -cpu host \ #-smp 6,sockets=1,cores=2,threads=1 \ #-bios /usr/share/qemu/bios.bin -vga none \ #-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ ##-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ #-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 8196 -mem-path /dev/hugepages -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -boot menu=on \ -bios /usr/share/qemu/bios.bin -vga none\ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -nographic \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/home/anonymous/Downloads/Win.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -usb -usbdevice host:046d:c52e -usbdevice host:041e:30e0 -usbdevice host:045e:0719 \ # -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ # -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 # -device vfio-pci,host=07:00.0,bus=pcie.0 \ # -device vfio-pci,host=08:04.0,bus=pcie.0 # -device vfio-pci,host=0c:00.0,bus=pcie.0 # -drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ # -boot menu=on \ # -rtc base=localtime \

fwiw dunno if this is relevant, but when i run the vm using huge pages appears to fail

qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory

but it was having that issue before i started trying to use the huge pages.

Bus 002 Device 002: ID 8087:8001 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 007 Device 003: ID 041e:30e0 Creative Technology, Ltd Bus 007 Device 002: ID 0409:005a NEC Corp. HighSpeed Hub Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:8009 Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 007: ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 003 Device 006: ID 9886:0001 Bus 003 Device 005: ID 24f0:0137 Bus 003 Device 004: ID 174c:2074 ASMedia Technology Inc. Bus 003 Device 003: ID 046d:c52e Logitech, Inc. MK260 Wireless Combo Receiver Bus 003 Device 002: ID 045e:0719 Microsoft Corp. Xbox 360 Wireless Adapter Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

edit:

by the way my cpu is an intel 4790k and the mb is asrock z97 extreme6

Last edited by risho (2014-10-05 00:00:56)

Offline

#2898 2014-10-05 00:20:22

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Have you installed a guest, installed a VNC server, installed Catalyst drivers, and have the VM working so that you can connect to that VNC server from the host?

I did this but I can't connect to the guest VNC server from the host. I did some googling and found that the host and guest are not visible to each other on the default network.

There appear to be a lot of ways to get around this problem, none of which I understand big_smile, what method do you suggest I look into?

Offline

#2899 2014-10-05 00:25:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauer wrote:
aw wrote:

Have you installed a guest, installed a VNC server, installed Catalyst drivers, and have the VM working so that you can connect to that VNC server from the host?

I did this but I can't connect to the guest VNC server from the host. I did some googling and found that the host and guest are not visible to each other on the default network.

There appear to be a lot of ways to get around this problem, none of which I understand big_smile, what method do you suggest I look into?

Use a bridge


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2900 2014-10-05 01:52:25

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

after going back through journalctl this is the last bit of information before the crash if this helps any

Oct 04 18:42:11 arch kernel: xhci_hcd 0000:0c:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8807c0e4be88 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:0c:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8807c0e4be40 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:0c:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8807bfe96ea0 Oct 04 18:42:11 arch kernel: usb 7-2: ep 0x83 - rounding interval to 64 microframes, ep desc says 80 microframes Oct 04 18:42:11 arch kernel: usb 3-1: reset full-speed USB device number 2 using xhci_hcd Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 1. Oct 04 18:42:11 arch kernel: usb 3-1: hub failed to enable device, error -22 Oct 04 18:42:11 arch kernel: usb 3-1: reset full-speed USB device number 2 using xhci_hcd Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 1. Oct 04 18:42:11 arch kernel: usb 3-1: hub failed to enable device, error -22 Oct 04 18:42:11 arch kernel: usb 3-1: reset full-speed USB device number 2 using xhci_hcd Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88080235d048 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88080235d000 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88080235d1c8 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88080235d180 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88080235de88 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88080235de40 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2048 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2000 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2108 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa20c0 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa21c8 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2180 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2288 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2240 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2348 Oct 04 18:42:11 arch kernel: xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880801fa2300 Oct 04 18:42:32 arch kernel: BUG: unable to handle kernel paging request at 00007fff00000000 Oct 04 18:42:32 arch kernel: IP: [<00007fff00000000>] 0x7fff00000000 Oct 04 18:42:32 arch kernel: PGD 75ed0d067 PUD 0 Oct 04 18:42:32 arch kernel: Oops: 0010 [#1] PREEMPT SMP Oct 04 18:42:32 arch kernel: Modules linked in: bridge stp llc md4 md5 hmac nls_utf8 cifs dns_resolver fscache ctr ccm hid_logitech_dj uas snd_usb_audio usb_storage snd_usbmidi_lib snd_hda_codec_hdmi vfio_pci vfio_iommu_type1 vfio nct6775 hwmon_vid ext4 crc16 mbcache jbd Oct 04 18:42:32 arch kernel: i2c_designware_platform tpm e1000e i2c_designware_core i2c_core battery dw_dmac dw_dmac_core snd_timer 8250_dw gpio_lynxpoint mei_me video spi_pxa2xx_platform snd mei ptp shpchp soundcore pps_core wmi processor acpi_pad button btrfs xor raid Oct 04 18:42:32 arch kernel: CPU: 1 PID: 1110 Comm: qemu-system-x86 Not tainted 3.16.0-1-mainline #1 Oct 04 18:42:32 arch kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme6, BIOS P1.30 05/22/2014 Oct 04 18:42:32 arch kernel: task: ffff8807bfd54750 ti: ffff8800a90f0000 task.ti: ffff8800a90f0000 Oct 04 18:42:32 arch kernel: RIP: 0010:[<00007fff00000000>] [<00007fff00000000>] 0x7fff00000000 Oct 04 18:42:32 arch kernel: RSP: 0018:ffff8800a90f3d98 EFLAGS: 00010006 Oct 04 18:42:32 arch kernel: RAX: ffff8807bf3c3e38 RBX: 000000008101478d RCX: 0000000000000000 Oct 04 18:42:32 arch kernel: RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8807bf3c3e38 Oct 04 18:42:32 arch kernel: RBP: ffff8800a90f3dd8 R08: 0000000000000000 R09: 0000000000000000 Oct 04 18:42:32 arch kernel: R10: 00000000000103c0 R11: 0000000000000293 R12: ffffffff8189eb08 Oct 04 18:42:32 arch kernel: R13: 00000000004a0418 R14: 0000000000000000 R15: 0000000000000003 Oct 04 18:42:32 arch kernel: FS: 00007f1923fff700(0000) GS:ffff88082fa40000(0000) knlGS:0000000000000000 Oct 04 18:42:32 arch kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 04 18:42:32 arch kernel: CR2: 00007fff00000000 CR3: 0000000780c26000 CR4: 00000000001427e0 Oct 04 18:42:32 arch kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 04 18:42:32 arch kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Oct 04 18:42:32 arch kernel: Stack: Oct 04 18:42:32 arch kernel: ffffffff810b6425 00000000bfd54750 0000000000000000 ffffffff8189eb00 Oct 04 18:42:32 arch kernel: 0000000000000046 0000000000000003 0000000000000000 0000000000000000 Oct 04 18:42:32 arch kernel: ffff8800a90f3e10 ffffffff810b66a9 ffff880803ab0000 0000000000000000 Oct 04 18:42:32 arch kernel: Call Trace: Oct 04 18:42:32 arch kernel: [<ffffffff810b6425>] ? __wake_up_common+0x55/0x90 Oct 04 18:42:32 arch kernel: [<ffffffff810b66a9>] __wake_up+0x39/0x50 Oct 04 18:42:32 arch kernel: [<ffffffff8139aed8>] __vga_put+0x98/0x150 Oct 04 18:42:32 arch kernel: [<ffffffff8139b56f>] vga_put+0x5f/0x90 Oct 04 18:42:32 arch kernel: [<ffffffffa010d104>] vfio_pci_vga_rw+0x1f4/0x240 [vfio_pci] Oct 04 18:42:32 arch kernel: [<ffffffffa010a4e5>] vfio_pci_rw+0x35/0x80 [vfio_pci] Oct 04 18:42:32 arch kernel: [<ffffffffa010b2cf>] vfio_pci_write+0x1f/0x30 [vfio_pci] Oct 04 18:42:32 arch kernel: [<ffffffffa00492eb>] vfio_device_fops_write+0x2b/0x30 [vfio] Oct 04 18:42:32 arch kernel: [<ffffffff811c1b87>] vfs_write+0xb7/0x200 Oct 04 18:42:32 arch kernel: [<ffffffff811c29ca>] SyS_pwrite64+0x9a/0xc0 Oct 04 18:42:32 arch kernel: [<ffffffff811d48bb>] ? SyS_ioctl+0x6b/0xa0 Oct 04 18:42:32 arch kernel: [<ffffffff81530869>] system_call_fastpath+0x16/0x1b Oct 04 18:42:32 arch kernel: Code: Bad RIP value. Oct 04 18:42:32 arch kernel: RIP [<00007fff00000000>] 0x7fff00000000

and it happens either right as or a few moments after shutting down a vm.

Offline

#2901 2014-10-05 02:02:00

Lauer
Member
Registered: 2014-09-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Use a bridge

I'm able to connect to the VNC server on the guest now! smile

Device manager does not list any display devices, and generic vga-compatible device is listed in the taskbar.

runwin-vfio is:

qemu-system-x86_64 -net nic -net bridge,br=br0 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/dev/md0,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk

Offline

#2902 2014-10-05 16:20:42

3000
Member
Registered: 2013-01-18
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, is there a good how-to? I think the how-to on the first page in this thread is way too advanced and the how-to from the arch wiki is pure crap.

I have used XEN for two years passing through AMD, but now I have Nvidia and need to make KVM work and I have absolutely no clue how.

Can someone please tell me where I can find a step-by-step guide?

Offline

#2903 2014-10-05 16:32:26

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3000 wrote:

Hi, is there a good how-to? I think the how-to on the first page in this thread is way too advanced and the how-to from the arch wiki is pure crap.

I have used XEN for two years passing through AMD, but now I have Nvidia and need to make KVM work and I have absolutely no clue how.

Can someone please tell me where I can find a step-by-step guide?

places:
- in this topic (yes, reading most of it)
- web (have seen at least few situational tutorials)
- try & explore

if op post is too advanced for you - should you really bother with experimental stuff? i mean what will you do when some problems arise? go ask for more "ohhh, please do waste more of your time just to solve some trivial stuff for me" (note: not every problem arising is trivial, but judging by this thread alone, i'd say 90% of problems are trivial - already solved and discussed in this thread)

Offline

#2904 2014-10-05 16:38:11

3000
Member
Registered: 2013-01-18
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks for nothing.

Offline

#2905 2014-10-05 17:00:43

3000
Member
Registered: 2013-01-18
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oh and btw: you have of course no time to help, but time to write all that CRAP. Congratulations.

Offline

#2906 2014-10-05 17:11:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3000 wrote:

oh and btw: you have of course no time to help, but time to write all that CRAP. Congratulations.

You're not improving your chances of getting help.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2907 2014-10-05 17:17:50

3000
Member
Registered: 2013-01-18
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Again an answer. And AGAIN thanks for nothing.

Offline

#2908 2014-10-05 17:39:24

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3000 wrote:

Again an answer. And AGAIN thanks for nothing.

well, i am truly noone in this thread. but replying "thanks for no help" to aw is like... ridiculous. sure, he did no help as far as vga passthrough is concerned, you are absolutely right )

oh, btw. why do you think someone should spend their own precious time to recompile tons of info in this very thread and his own experience to enlighten some ignorant child coming with nothing but "i did this for quite some time, but now i just dont have any clue. someone go bring me step-by-step guide"?

not to mention me not being able to grasp at all how "how-to" and "experimental stuff" can coexist in someone's head unless that very head is completely broken in some way.

p.s. yup, one more stupid reply with absolutely no help. bad bad me
p.p.s. in case you are not completely hopeless, i'll just repeat myself: this very thread have really TONS of info. period.

Offline

#2909 2014-10-05 17:42:33

3000
Member
Registered: 2013-01-18
Posts: 39

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you don't want to <redacted> help? Guess what? DON'T <redacted> HELP. That simple.

moderator edit [ewaller] sanitized somewhat after a burst of frustration

Last edited by ewaller (2014-10-05 18:13:16)

Offline

#2910 2014-10-05 17:42:39

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey .

Alex , I narrowed my crashing problems to reboots only .

When I shutdown the VM cleanly then reexecute the shell file to fire it up , it works nicely , no crashes to either the host or other VMs .

It only happens when I reboot the VM .

Journalctl doesn't show any errors related to the issue .

Again , it started happening as soon as I *upgraded* to X99 platform .  : (

EDIT :

One more thing , when I reboot a VM with no VGA passthroughed to it , it reboots fine . So its related to VGA . as it is basically the same shell , the main difference between these VMs is the VGA .

EDIT 2 :

Kernel is stock ARCH , 3.16.3 .

Qemu-git , compiled two days ago .

Last edited by Denso (2014-10-05 18:31:37)

Offline

#2911 2014-10-05 18:11:01

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

While 3000 takes a day off to cool down, I remind everyone to respect each other, please do not flame.  Read the link in my signature, and remember the Arch way.

Edit: Oh, I forgot, remember that life is a two way street

Last edited by ewaller (2014-10-05 18:18:47)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#2912 2014-10-05 19:09:11

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to narrow down the problem. The host hungs on suspend/resume/reboot/shutdown only if I the video card isn't released properly and as a result I cannot unbind it. Sometimes it gets released and everything works great but I cannot predict or spot the pattern why and when. I get this in dmesg when I fail to unbind the card:

[39245.912644] BUG: unable to handle kernel NULL pointer dereference at 0000000000000411 [39245.912708] IP: [<0000000000000411>] 0x411 [39245.912757] PGD 7c511d067 PUD 80d112067 PMD 0 [39245.912801] Oops: 0010 [#1] SMP [39245.912831] Modules linked in: vfio_pci vfio_iommu_type1 vfio nfnetlink_queue nfnetlink_log nfnetlink bluetooth pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) cfg80211 vboxdrv(O) cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative binfmt_misc nls_utf8 nls_cp437 vfat fat snd_usb_audio snd_usbmidi_lib snd_rawmidi snd_seq_device joydev mxm_wmi snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap rfkill iTCO_wdt iTCO_vendor_support evdev snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic intel_powerclamp intel_rapl coretemp kvm_intel i915 snd_hda_intel efi_pstore snd_hda_controller snd_hda_codec psmouse kvm snd_hwdep drm_kms_helper snd_pcm drm serio_raw pcspkr efivars snd_timer i2c_i801 snd i2c_algo_bit soundcore i2c_core mei_me wmi lpc_ich mei mfd_core shpchp battery video button processor fuse autofs4 ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod hid_logitech_dj sg usbhid hid sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper ahci cryptd libahci libata ehci_pci xhci_hcd scsi_mod ehci_hcd r8169 mii usbcore usb_common thermal fan thermal_sys [39245.913917] CPU: 4 PID: 5417 Comm: tee Tainted: G W O 3.16.0-i915.arb.only #1 [39245.913975] Hardware name: System manufacturer System Product Name/P8Z77-V LK, BIOS 1402 03/21/2014 [39245.914039] task: ffff88082bac82d0 ti: ffff88076652c000 task.ti: ffff88076652c000 [39245.914092] RIP: 0010:[<0000000000000411>] [<0000000000000411>] 0x411 [39245.914148] RSP: 0018:ffff88076652fde8 EFLAGS: 00010206 [39245.914187] RAX: 0000000000000008 RBX: ffff88080f730098 RCX: 0000000000000000 [39245.914239] RDX: 0000000000000411 RSI: ffff88080f730098 RDI: ffff88080f730098 [39245.914291] RBP: ffff88080f730098 R08: 0000000000000246 R09: ffffffff8129c240 [39245.914343] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88080f730148 [39245.914397] R13: 0000000000000000 R14: ffff88076652ff58 R15: ffff88080d1c2758 [39245.914449] FS: 00007fd3ea278700(0000) GS:ffff88082fb00000(0000) knlGS:0000000000000000 [39245.914506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39245.914549] CR2: 0000000000000411 CR3: 000000080ea97000 CR4: 00000000001407e0 [39245.914600] Stack: [39245.914618] ffffffff813951c7 0000000000000004 ffff88080f730098 0000000000000004 [39245.914683] ffffffff813960f8 0000000000000004 ffff88080f730098 ffff88080f730148 [39245.914748] 0000000000000246 ffffffff81396244 ffff88080f730098 ffffffffa078d000 [39245.914816] Call Trace: [39245.914843] [<ffffffff813951c7>] ? __rpm_callback+0x27/0x70 [39245.914887] [<ffffffff813960f8>] ? rpm_idle+0x198/0x290 [39245.914933] [<ffffffff81396244>] ? __pm_runtime_idle+0x54/0x80 [39245.914980] [<ffffffff812d2467>] ? pci_device_remove+0x67/0xb0 [39245.915029] [<ffffffff8138d605>] ? __device_release_driver+0x75/0xf0 [39245.915079] [<ffffffff8138d699>] ? device_release_driver+0x19/0x30 [39245.915131] [<ffffffff8138ca2d>] ? unbind_store+0xed/0x150 [39245.915176] [<ffffffff812081bb>] ? kernfs_fop_write+0xdb/0x160 [39245.915225] [<ffffffff8119903d>] ? vfs_write+0xad/0x1e0 [39245.915269] [<ffffffff81198f3c>] ? vfs_read+0x11c/0x170 [39245.915313] [<ffffffff81199b4d>] ? SyS_write+0x3d/0xa0 [39245.915355] [<ffffffff814f596d>] ? system_call_fast_compare_end+0x10/0x15 [39245.915406] Code: Bad RIP value. [39245.915439] RIP [<0000000000000411>] 0x411 [39245.915481] RSP <ffff88076652fde8> [39245.915507] CR2: 0000000000000411 [39245.923907] ---[ end trace 42dbcaccec8b4e5d ]---

This is my unbind script

#!/bin/bash for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/${dev}/vendor) device=$(cat /sys/bus/pci/devices/${dev}/device) if [ -e /sys/bus/pci/devices/${dev}/driver ]; then echo "${dev}" | sudo tee /sys/bus/pci/devices/${dev}/driver/unbind > /dev/null while [ -e /sys/bus/pci/devices/${dev}/driver ]; do sleep 1 done echo "${dev}" | sudo tee /sys/bus/pci/drivers_probe > /dev/null fi done

I tried sp_off and disable_hugepages=1 as suggested here but with no luck.

Last edited by zsph (2014-10-05 20:05:54)


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#2913 2014-10-06 14:19:43

kolobok
Member
Registered: 2014-10-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!
I have AMD RS880M chipset with integrated and discrete radeons, it seems there is only GART IOMMU. (there is no option in bios)

[ 0.000000] AGP: Please enable the IOMMU option in the BIOS setup [ 1.428469] PCI-DMA: using GART IOMMU. [ 1.428471] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture [ 1.428469] PCI-DMA: using GART IOMMU. [ 2.080672] [drm] GART: num cpu pages 131072, num gpu pages 131072 [ 2.098628] [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).

When I am trying to bind vfio it fails.

[ 57.776135] VFIO - User Level meta-driver version: 0.3 [ 57.786888] vfio-pci: probe of 0000:02:00.0 failed with error -22

I have disabled radeon driver on second gpu by pci-stub

[ 2.044704] pci-stub: add 1002:68E0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.044719] pci-stub 0000:02:00.0: claimed by stub

Can you help me with this? What is 22 error code means?

Offline

#2914 2014-10-06 15:13:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kolobok wrote:

Hi!
I have AMD RS880M chipset with integrated and discrete radeons, it seems there is only GART IOMMU. (there is no option in bios)

A GART IOMMU is not sufficient for doing device assignment.  http://vfio.blogspot.com/2014/08/iommu- … d-out.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2915 2014-10-07 00:19:50

roadrunna
Member
Registered: 2014-10-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just like to thank you for this guide. I got it working on gentoo. nvidia  550 TI with intel integrated graphics. using virtio drivers from redhat. So far the only issue is when I close down the vm ,but this maybe resolved with dev eject I have yet to try. Is there a way to convert the qemu arguments to xml to be able to manage the vm using libvirt + virt manager?

Offline

#2916 2014-10-07 04:20:05

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

roadrunna wrote:

Just like to thank you for this guide. I got it working on gentoo. nvidia  550 TI with intel integrated graphics. using virtio drivers from redhat. So far the only issue is when I close down the vm ,but this maybe resolved with dev eject I have yet to try. Is there a way to convert the qemu arguments to xml to be able to manage the vm using libvirt + virt manager?

yes, there is a way - you will have to lurk the thread few dozens of pages back though

the main issue you may probably run into is audio. i resolved it by passing through dedicated pci sound card to vm while physically connecting its' front-out to host's integrated audio line-in (this way host outputs both "native" sound and sound from vm). yes, it required additional hardware (i had a spare one), but at the same time in-game fps drastically improved (it may be side-effect of libvirt's inner stuff though, like more accurate/advanced cpu pinning; but i still think it was mainly the result of escaping from emulated audio - no solid proof though)

p.s. if anyone has a way of preserving emulated audio while using libvirt/virt-manager (e.g. QEMU_AUDIO_DRV=pa) - detailed report would be highly appreciated. in my case it was always QEMU_AUDIO_DRV=none - did not bother digging too much into this though

Last edited by sinny (2014-10-07 04:23:51)

Offline

#2917 2014-10-07 12:06:49

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

p.s. if anyone has a way of preserving emulated audio while using libvirt/virt-manager (e.g. QEMU_AUDIO_DRV=pa) - detailed report would be highly appreciated. in my case it was always QEMU_AUDIO_DRV=none - did not bother digging too much into this though

Hi,

i successfully use libvirt and a emulated Soundcard via pulseaudio with this:

 <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_PA_SAMPLES' value='1024'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>

to save yourself the hassle of localhostsecurity regarding pulseaudio, add "load-module module-native-protocol-unix auth-anonymous=1" to $HOME/.pulse/default.pa, this way you dont need to copy around the cookiefile of pulseaudio.

you might need to play around with the "default-fragments" and "default-fragment-size-msec" in $HOME/.pulse/daemon.conf - i just left them as default and its working - but this depends on buffersize within your soundcards and you might need lower the defaults to get correct a/v sync.

QEMU_PA_SERVER can also be a networkserver - "localhost" works also, but on the other end (pulseaudio network server) you need to disable authentication (at least i had to).

It works - its in sync. The main requirements i had. Downside is relative high cpu usage of the pulseaudio process, as it seems qemu never "closes"(suspends?) the device, so pulseaudio is always on about 10%cpu (but only on one cpu) on my machine.

Greetings
K

Last edited by kaeptnb (2014-10-07 12:07:06)

Offline

#2918 2014-10-07 14:40:03

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:
sinny wrote:

p.s. if anyone has a way of preserving emulated audio while using libvirt/virt-manager (e.g. QEMU_AUDIO_DRV=pa) - detailed report would be highly appreciated. in my case it was always QEMU_AUDIO_DRV=none - did not bother digging too much into this though

Hi,

i successfully use libvirt and a emulated Soundcard via pulseaudio with this:

 <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_PA_SAMPLES' value='1024'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>

to save yourself the hassle of localhostsecurity regarding pulseaudio, add "load-module module-native-protocol-unix auth-anonymous=1" to $HOME/.pulse/default.pa, this way you dont need to copy around the cookiefile of pulseaudio.

you might need to play around with the "default-fragments" and "default-fragment-size-msec" in $HOME/.pulse/daemon.conf - i just left them as default and its working - but this depends on buffersize within your soundcards and you might need lower the defaults to get correct a/v sync.

QEMU_PA_SERVER can also be a networkserver - "localhost" works also, but on the other end (pulseaudio network server) you need to disable authentication (at least i had to).

It works - its in sync. The main requirements i had. Downside is relative high cpu usage of the pulseaudio process, as it seems qemu never "closes"(suspends?) the device, so pulseaudio is always on about 10%cpu (but only on one cpu) on my machine.

Greetings
K

whoa, thank you very much
this kinda makes my dual vm configuration more or less complete

p.s. would be great to have links to posts related to virt-manager setup (the one on page 94 and this one) at opening post

Offline

#2919 2014-10-07 19:39:49

dwightjl
Member
Registered: 2013-10-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3000 wrote:

Hi, is there a good how-to? I think the how-to on the first page in this thread is way too advanced

Here are some of the guides that I used:
http://www.pugetsystems.com/labs/articl … 4-KVM-585/
http://www.firewing1.com/howtos/fedora- … hrough-kvm
http://www.linux-kvm.org/page/VGA_device_assignment

This technology and process are very new, so guides are usually system-specific and are often incomplete. I had to read several guides and many pages on this forum to eventually get something that works on my system.

Speaking of which, I can confirm that the following configuration works:

Motherboard: ASRock Z77 Extreme 4-M
CPU: Intel i5 3570
Memory: 16GB DDR3 1600
Host GPU: Integrated GPU from the i5 3570
Guest GPU: EVGA 550Ti (GPU BIOS updated to the latest version)
Host OS: Ubuntu 14.04 Server updated to the 14.10 Utopic sources.
Guest OS: Windows 10 Preview
Kernel: 3.16.3 (No patches)
QEMU: v2.1 (No patches)
SeaBIOS: v1.7.5
Blacklisted drivers: nouveau, i915, intel_agp (Even though I'm using the intel integrated GPU for the host, blacklisting the drivers seems to fix GPU arbitration issues)

I tried using a spare ATI card as the host GPU but ran into the problem where both cards are included the the iommu_group and I decided to give up on that path for now. The integrated GPU doesn't have the same issue. I can use the integrated GPU without the kernel patches as long as I blacklist the intel drivers.

The reason I went with Ubuntu Server is because I plan to use my home server to stream Windows games with Steam in-home streaming, so I don't care to have a working desktop on that machine anyway. That way I can keep all my normal bare-bones server functions and use the Windows VM to stream to my Linux Gaming PC or Steam Machines around the house. I still need to convince nvidia to let me install drivers on the guest system, and that requires a QEMU patch that I'm too lazy to apply right now.

The Arch forums can be a little harsh sometimes. Please be patient with the people here. They have put a lot of time into getting this stuff to work and rightfully expect you to put in effort as well. Eventually someone will show up and help you out. Game is hard.

Last edited by dwightjl (2014-10-07 20:27:43)

Offline

#2920 2014-10-08 02:05:04

roadrunna
Member
Registered: 2014-10-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If anybody is using the 3.17 kernel using i915 ,and patching manually here are both acs override patch + i915 vga arbiter fixes into 1. http://www.mediafire.com/download/02ri1 … -kvm.patch.  I'm using gentoo and I patch manually. Thought Id make it easier for someone. smile.

Offline

#2921 2014-10-08 03:33:43

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zsph wrote:

I managed to narrow down the problem. The host hungs on suspend/resume/reboot/shutdown only if I the video card isn't released properly and as a result I cannot unbind it. Sometimes it gets released and everything works great but I cannot predict or spot the pattern why and when. I get this in dmesg when I fail to unbind the card:

[39245.912644] BUG: unable to handle kernel NULL pointer dereference at 0000000000000411 ... [39245.914843] [<ffffffff813951c7>] ? __rpm_callback+0x27/0x70 [39245.914887] [<ffffffff813960f8>] ? rpm_idle+0x198/0x290 [39245.914933] [<ffffffff81396244>] ? __pm_runtime_idle+0x54/0x80 [39245.914980] [<ffffffff812d2467>] ? pci_device_remove+0x67/0xb0 [39245.915029] [<ffffffff8138d605>] ? __device_release_driver+0x75/0xf0 [39245.915079] [<ffffffff8138d699>] ? device_release_driver+0x19/0x30 [39245.915131] [<ffffffff8138ca2d>] ? unbind_store+0xed/0x150 [39245.915176] [<ffffffff812081bb>] ? kernfs_fop_write+0xdb/0x160 [39245.915225] [<ffffffff8119903d>] ? vfs_write+0xad/0x1e0 [39245.915269] [<ffffffff81198f3c>] ? vfs_read+0x11c/0x170 [39245.915313] [<ffffffff81199b4d>] ? SyS_write+0x3d/0xa0 [39245.915355] [<ffffffff814f596d>] ? system_call_fast_compare_end+0x10/0x15 ...

I actually get something similar when I stop or start (I forgot which) a VM, if I have the AMD High-Definition Audio Controller bound to the VM.  Blacklisting snd-hda-intel on the host prevents the panic, but also prevents the integrated Realtek sound card from working on the host.

EDIT: By "similar", I mean a very similar, if not the same, backtrace.

Last edited by DanaGoyette (2014-10-08 04:40:10)

Offline

#2922 2014-10-08 10:05:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
zsph wrote:

I managed to narrow down the problem. The host hungs on suspend/resume/reboot/shutdown only if I the video card isn't released properly and as a result I cannot unbind it. Sometimes it gets released and everything works great but I cannot predict or spot the pattern why and when. I get this in dmesg when I fail to unbind the card:

[39245.912644] BUG: unable to handle kernel NULL pointer dereference at 0000000000000411 ... [39245.914843] [<ffffffff813951c7>] ? __rpm_callback+0x27/0x70 [39245.914887] [<ffffffff813960f8>] ? rpm_idle+0x198/0x290 [39245.914933] [<ffffffff81396244>] ? __pm_runtime_idle+0x54/0x80 [39245.914980] [<ffffffff812d2467>] ? pci_device_remove+0x67/0xb0 [39245.915029] [<ffffffff8138d605>] ? __device_release_driver+0x75/0xf0 [39245.915079] [<ffffffff8138d699>] ? device_release_driver+0x19/0x30 [39245.915131] [<ffffffff8138ca2d>] ? unbind_store+0xed/0x150 [39245.915176] [<ffffffff812081bb>] ? kernfs_fop_write+0xdb/0x160 [39245.915225] [<ffffffff8119903d>] ? vfs_write+0xad/0x1e0 [39245.915269] [<ffffffff81198f3c>] ? vfs_read+0x11c/0x170 [39245.915313] [<ffffffff81199b4d>] ? SyS_write+0x3d/0xa0 [39245.915355] [<ffffffff814f596d>] ? system_call_fast_compare_end+0x10/0x15 ...

I actually get something similar when I stop or start (I forgot which) a VM, if I have the AMD High-Definition Audio Controller bound to the VM.  Blacklisting snd-hda-intel on the host prevents the panic, but also prevents the integrated Realtek sound card from working on the host.

EDIT: By "similar", I mean a very similar, if not the same, backtrace.

Try using pci-stub.ids instead of blacklisting the driver


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2923 2014-10-09 11:03:09

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

Offline

#2924 2014-10-09 11:24:56

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host,hv_time,-kvm_pv_eoi,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1 -m 8192 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 6,sockets=1,cores=3,threads=2 -uuid 09d81f35-51a4-4c7d-8db0-46597c7233ba -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -drive file=/dev/storage/windows-ovmf,if=none,id=drive-virtio-disk0,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/storage/steam,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c6:b5:45,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7,rombar=0,romfile=/var/lib/libvirt/firmware/HD7870_EFI.rom -device vfio-pci,host=01:00.1,id=hostdev2,bus=pci.0,addr=0x7.0x1 -device usb-host,hostbus=2,hostaddr=17,id=hostdev3 -device usb-host,hostbus=2,hostaddr=3,id=hostdev4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -set device.virtio-disk0.scsi=off -set device.virtio-disk0.x-data-plane=on -set device.virtio-disk1.scsi=off -set device.virtio-disk1.x-data-plane=on -msg timestamp=on

It's how libvirt launches my vm, but should work anyway with few adjustments.

Offline

#2925 2014-10-09 14:01:25

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
Denso wrote:

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host,hv_time,-kvm_pv_eoi,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1 -m 8192 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 6,sockets=1,cores=3,threads=2 -uuid 09d81f35-51a4-4c7d-8db0-46597c7233ba -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -drive file=/dev/storage/windows-ovmf,if=none,id=drive-virtio-disk0,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/storage/steam,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c6:b5:45,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7,rombar=0,romfile=/var/lib/libvirt/firmware/HD7870_EFI.rom -device vfio-pci,host=01:00.1,id=hostdev2,bus=pci.0,addr=0x7.0x1 -device usb-host,hostbus=2,hostaddr=17,id=hostdev3 -device usb-host,hostbus=2,hostaddr=3,id=hostdev4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -set device.virtio-disk0.scsi=off -set device.virtio-disk0.x-data-plane=on -set device.virtio-disk1.scsi=off -set device.virtio-disk1.x-data-plane=on -msg timestamp=on

It's how libvirt launches my vm, but should work anyway with few adjustments.

Thank you very much !

I'm going to adapt these settings to my existing WIndows VM , however I couldn't find the file "windows_VARS.fd" , and Google points me to your post , if it isn't any trouble can you tell me what is it for ? is it important ? if so , how can I create one ?

Sorry for this barge of questions , I want to learn .

Offline

#2926 2014-10-09 16:50:23

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
dwe11er wrote:
Denso wrote:

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host,hv_time,-kvm_pv_eoi,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1 -m 8192 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 6,sockets=1,cores=3,threads=2 -uuid 09d81f35-51a4-4c7d-8db0-46597c7233ba -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -drive file=/dev/storage/windows-ovmf,if=none,id=drive-virtio-disk0,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/storage/steam,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c6:b5:45,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7,rombar=0,romfile=/var/lib/libvirt/firmware/HD7870_EFI.rom -device vfio-pci,host=01:00.1,id=hostdev2,bus=pci.0,addr=0x7.0x1 -device usb-host,hostbus=2,hostaddr=17,id=hostdev3 -device usb-host,hostbus=2,hostaddr=3,id=hostdev4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -set device.virtio-disk0.scsi=off -set device.virtio-disk0.x-data-plane=on -set device.virtio-disk1.scsi=off -set device.virtio-disk1.x-data-plane=on -msg timestamp=on

It's how libvirt launches my vm, but should work anyway with few adjustments.

Thank you very much !

I'm going to adapt these settings to my existing WIndows VM , however I couldn't find the file "windows_VARS.fd" , and Google points me to your post , if it isn't any trouble can you tell me what is it for ? is it important ? if so , how can I create one ?

Sorry for this barge of questions , I want to learn .

@"windows_VARS.fd":
i suspect you should read up every post at aw's blog (search in his forum sig) - especially the ones related to ovmf, even more especially for ovmf split image support

it's a bit strange that you try to find it somewhere externally even though this command line parameter declaration looks almost identical to guest's hard drive info

Offline

#2927 2014-10-10 00:42:28

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I couldn't find the file "windows_VARS.fd"

To summarize the infos from aw's blog: if you use splitted OVMF images, libvirt copies for you the vars-file, for qemu you should do it manually because this file might get modified. The original file is part or compiled of your OVMF-package. http://vfio.blogspot.de/2014/09/ovmf-sp … pport.html

Offline

#2928 2014-10-10 12:17:12

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you both !

My OVMF is exactly 2MB and according to Alex's blog post , it is a unified package so specifying a VARS file is not necessary I guess .

I don't know what is wrong with my shell script , but OVMF always drops me to the shell with no disks detected . Been trying for weeks ...

qemu-system-x86_64 -name gaming -nographic \ -enable-kvm -m 8192 -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -pflash /VMs/ovmf_x64.bin -device vfio-pci,host=02:00.0,bus=bus=pci.0,addr=addr=0x7,multifunction=on \ -device vfio-pci,host=02:00.1,bus=bus=pci.0,0x7.0x1 \ -drive file=/VMs/Win_Gaming.qed,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -drive file=/VMs/Gaming_External.qed,cache=none,if=none,id=drive1,aio=native \ -device virtio-blk-pci,drive=drive1,ioeventfd=on \ -net nic,model=virtio,macaddr=64:25:5E:87:0E:5C -net bridge,br=br0 \ -usb -usbdevice host:1c4f:0034 -usbdevice host:1c4f:0002 -usbdevice host:0a12:0001 \ -localtime \ -monitor unix:/tmp/vm_gaming,server,nowait &

Offline

#2929 2014-10-10 14:47:53

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Thank you both !

My OVMF is exactly 2MB and according to Alex's blog post , it is a unified package so specifying a VARS file is not necessary I guess .

I don't know what is wrong with my shell script , but OVMF always drops me to the shell with no disks detected . Been trying for weeks ...

qemu-system-x86_64 -name gaming -nographic \ -enable-kvm -m 8192 -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -pflash /VMs/ovmf_x64.bin -device vfio-pci,host=02:00.0,bus=bus=pci.0,addr=addr=0x7,multifunction=on \ -device vfio-pci,host=02:00.1,bus=bus=pci.0,0x7.0x1 \ -drive file=/VMs/Win_Gaming.qed,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -drive file=/VMs/Gaming_External.qed,cache=none,if=none,id=drive1,aio=native \ -device virtio-blk-pci,drive=drive1,ioeventfd=on \ -net nic,model=virtio,macaddr=64:25:5E:87:0E:5C -net bridge,br=br0 \ -usb -usbdevice host:1c4f:0034 -usbdevice host:1c4f:0002 -usbdevice host:0a12:0001 \ -localtime \ -monitor unix:/tmp/vm_gaming,server,nowait &

it always dropped me there whenever i was trying to boot something that is not uefi-aware (ISO's mostly)
if you try to switch already installed os - i guess you should reinstall (not that aware of details but it seems to me at least windows-installed boot stuff/layout is different depending on whether it's bios or uefi)

Offline

#2930 2014-10-10 17:19:09

Drakulix
Member
Registered: 2014-10-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there!
I have some question on this subject and got already pretty far on my own, although I have a very experimental hardware configuration, but I am stuck now and need some help.

First about my hardware setup. Please don't tell me right away, that VGA Passthough will not work on my setup. I know that some people might have mixed feeling, when reading the next sentence.

So I have a Macbook Pro Retina 15" with an Intel HD4000 and an nVidia GeForce 650M and yes I am trying to Passthough a Mobile Graphics Card.
My CPU (Intel(R) Core(TM) i7-3720QM) does support VT-d, I have no idea, what my mainbroad might support. It is from mid-2012, so I have at least a chance, I guess.

I am running Linux Kernel 3.17 with the acs override patch, the i915 vga arbiter fixes and some other patches. (4 cherry picks from 3.18.  First to fix broken HDMI info frames with nouveau, and some about my battery and thunderbold hotplug support on apple devices. So nothing else that should matter.)

I know that my HDMI connector is directly attached to my graphics card. My kernel also contains a self-made patch that allows me to switch my mux directly (bypassing vga_switcheroo and the ability to switch without any graphic drivers loaded). So I can decide, if my internal screen is attached to my intel card or the nvidia card. (works nicely, if someone needs help on that subject, feel free to pm me. the wiki article is not very helpful with this).

I can currently boot a Windows 7 SP1 (64-Bit) VM with VGA Passthough either on my HDMI Monitor (allowing my to use linux meanwhile) or on my internal screen (swapping the mux beforehand and after running qemu via a little bash script). The graphics card does get detected, I can install the nVidia drivers, but I get error 43 in the Device Manager.


Some more details:

I have the following in my kernel boot parameters:

modprobe.blacklist=nouveau i915.enable_hd_vgaarb=1 pci-stub.ids=10de:0fd5,10de:0e1b,8086:1547 intel_iommu=on

I am normally running the with nouveau drivers, the nvidia drivers are not installed.
The last device:vendor-id pair is from my thunderbolt connectors, which appear to be connected to the same iommu_group my graphics adapter is (probably for displayport output).
I need to bind them to vfio as well, otherwise I get an error in qemu (iommu_group not viable).
Everything shows up as loaded by pci-stub correctly, before I bind to vfio-pci.

So this is what my binding looks like:

sudo ./vfio-bind 0000:01:00.0 0000:01:00.1 0000:05:00.0 0000:06:00.0 0000:06:03.0 0000:06:04.0 0000:06:05.0 0000:06:06.0 0000:07:00.0

The first is my graphics card, the seconds is HDMI audio, the rest is thunderbolt.

I also have the following set via configs in modprobe.d:

options kvm ignore_msrs=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

I need the second option, at least thats what qemu tells me, when I try to boot without it.

Last but not atleast I run qemu like this:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=vbios.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=QEMU/windows.img,id=disk,format=qcow2 \ -device ide-hd,bus=ide.0,drive=disk -device qxl

I need to pass the romfile, otherwise qemu complains and dmesg says something like "Invalid VBIOS Contents". I am using -device qxl, since I do want to use my keyboard and trackpad.


I am using the qemu package from the arch repos, which is version 2.1.2. The qemu-git package failed for some reason, but I can try to get it to work, if that would help anything.

If you need any more informations, please feel free to ask.
I have not much ideas left, to fix the error 43. I already removed all hv_* functions and added kvm=off to my qemu cpu options, still not any better.

Offline

#2931 2014-10-11 00:18:08

Mardok45
Member
Registered: 2014-08-10
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've done a lot of switching between Xen and QEMU when researching this topic and have ultimately decided to stick with QEMU.  I have a desktop that's running both Windows 7 and Gentoo and it works almost perfectly.

There's still one problem that I haven't solved yet.  My Windows VM has a Nvidia GTX 260 exposed to it and I am able to play Dota 2.  However, Dota 2 was having framerate problems.  I know for a fact that on bare-metal, my GTX 260 chews right through Dota 2 with no problems.  I expect a performance degredation, but this is unusually slow. 

I looked at the CPU usage, and all four CPU's that were exposed to it were running at 50%.  So the CPU isn't the problem.  I looked at the memory usage and 2 gigabytes were used out of the 3 that were exposed to it.  So the memory isn't the problem.

After a lot of head-scratching, I noticed that the GTX 260 was really quiet through all this.  Usually when playing a game like Dota 2, the GPU fan gets really loud.  I think the problem is the GTX 260 isn't increasing its clock speed.

I tried a QEMU config using the Q35 machine model and the 440FX model and both have the same problem.

I tried turning on maximum performance in the NVidia control panel and Windows settings, but neither solved the problem.  Does anyone know what the problem might be?

Last edited by Mardok45 (2014-10-11 00:19:59)

Offline

#2932 2014-10-11 00:46:16

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mardok45 wrote:

I've done a lot of switching between Xen and QEMU when researching this topic and have ultimately decided to stick with QEMU.  I have a desktop that's running both Windows 7 and Gentoo and it works almost perfectly.

There's still one problem that I haven't solved yet.  My Windows VM has a Nvidia GTX 260 exposed to it and I am able to play Dota 2.  However, Dota 2 was having framerate problems.  I know for a fact that on bare-metal, my GTX 260 chews right through Dota 2 with no problems.  I expect a performance degredation, but this is unusually slow. 

I looked at the CPU usage, and all four CPU's that were exposed to it were running at 50%.  So the CPU isn't the problem.  I looked at the memory usage and 2 gigabytes were used out of the 3 that were exposed to it.  So the memory isn't the problem.

After a lot of head-scratching, I noticed that the GTX 260 was really quiet through all this.  Usually when playing a game like Dota 2, the GPU fan gets really loud.  I think the problem is the GTX 260 isn't increasing its clock speed.

I tried a QEMU config using the Q35 machine model and the 440FX model and both have the same problem.

I tried turning on maximum performance in the NVidia control panel and Windows settings, but neither solved the problem.  Does anyone know what the problem might be?

Try different cpu configs, here works best with 2 or more cores 1 thread each, and hv-time option that doesn't work with drivers after 340.xx..

Last edited by slis (2014-10-11 01:10:06)

Offline

#2933 2014-10-11 01:58:19

Mardok45
Member
Registered: 2014-08-10
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
Mardok45 wrote:

I've done a lot of switching between Xen and QEMU when researching this topic and have ultimately decided to stick with QEMU.  I have a desktop that's running both Windows 7 and Gentoo and it works almost perfectly.

There's still one problem that I haven't solved yet.  My Windows VM has a Nvidia GTX 260 exposed to it and I am able to play Dota 2.  However, Dota 2 was having framerate problems.  I know for a fact that on bare-metal, my GTX 260 chews right through Dota 2 with no problems.  I expect a performance degredation, but this is unusually slow. 

I looked at the CPU usage, and all four CPU's that were exposed to it were running at 50%.  So the CPU isn't the problem.  I looked at the memory usage and 2 gigabytes were used out of the 3 that were exposed to it.  So the memory isn't the problem.

After a lot of head-scratching, I noticed that the GTX 260 was really quiet through all this.  Usually when playing a game like Dota 2, the GPU fan gets really loud.  I think the problem is the GTX 260 isn't increasing its clock speed.

I tried a QEMU config using the Q35 machine model and the 440FX model and both have the same problem.

I tried turning on maximum performance in the NVidia control panel and Windows settings, but neither solved the problem.  Does anyone know what the problem might be?

Try different cpu configs, here works best with 2 or more cores 1 thread each, and hv-time option that doesn't work with drivers after 340.xx..

qemu-system-x86_64 \ -nodefconfig \ -monitor stdio -serial none -parallel none \ -enable-kvm \ -rtc base=localtime \ -M q35 \ -smp 4,sockets=1,cores=4,threads=1 -cpu host,kvm=off \ -m 3072 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:12:34:56:78:97 \ -drive file=/dev/system/windows,cache=none,if=none,id=drive0,aio=threads,format=raw -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=2 \ -drive file=/root/win7.iso,if=none,id=cd0,aio=threads,format=raw -device ide-cd,drive=cd0,bootindex=1 \ -drive file=/root/qemu/virtio.iso,if=none,id=cd1,aio=threads,format=raw -device ide-cd,drive=cd1,bus=ide.1 \ -device vfio-pci,host=00:12.0,bus=pcie.0 \ -device vfio-pci,host=00:12.2,bus=pcie.0 \ -device vfio-pci,host=00:14.2,bus=pcie.0 \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,bus=pcie.0 \ -vga none \ -nographic \

I'm running the 334 drivers.  I removed hv-time and it doesn't seem to help.  I'm not sure why the GPU wouldn't increase its clock speed.

Offline

#2934 2014-10-11 07:04:04

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well i got here native like perfomance in dota2, with hv-time and 2+ cores, remove that smp option maybe?


http://pastie.org/9639320

i'm using this config for streaming on headless server for last 2-3 months and with 340.82 driver it works without problem.

Offline

#2935 2014-10-11 09:36:51

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

it always dropped me there whenever i was trying to boot something that is not uefi-aware (ISO's mostly)
if you try to switch already installed os - i guess you should reinstall (not that aware of details but it seems to me at least windows-installed boot stuff/layout is different depending on whether it's bios or uefi)

Then how can one install Windows if the installation media could not be detected , I'm using Windows 8.1 ISO ?

Offline

#2936 2014-10-11 11:11:46

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can anyone who have tried using OVMF confirm that initial OVMF bootup messages and EFI shell are shown on a monitor connected to a passed-through video card? I can seem to get even this basic thing to work with OVMF.

Command line I've tried:

qemu-system-x86_64 -enable-kvm -machine pc-i440fx-2.1,accel=kvm,usb=off -device vfio-pci,host=01:00.0,bus=pci.0,addr=7,multifunction=on,romfile=/home/user/NV280MS1-EFI.rom -device vfio-pci,host=01:00.1,bus=pci.0,addr=7.1 -drive if=pflash,format=raw,readonly,file=/tmp/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd -vga none -nographic

All messages, including EFI shell, are shown on QEMU's curses console. Nothing is shown on a passed-though video card. Does it mean EFI ROM that performs video card initialization doesn't run properly? If I type `pci' command in EFI shell, I see that correct VID/PID pair is listed for the video card, thus it is clearly detected.

The ROM file I use (NV280MS1-EFI.rom) does have a EFI part:

Valid ROM signature found @400h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 0, vendor revision: 1 Valid ROM signature found @10000h, PCIR offset 1ch PCIR: type 3, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 3, vendor revision: 0 EFI: Signature Valid Last image

Video card in question is a NVidia GTX 650 (MSI N650 PE 1GD5/OC). Initially it came with hybrid (EFI/Legacy) BIOS flashed, later update was just Legacy. I'm trying to use original ROM I've fortunately backed up before reflashing.

Offline

#2937 2014-10-11 16:01:25

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
sinny wrote:

it always dropped me there whenever i was trying to boot something that is not uefi-aware (ISO's mostly)
if you try to switch already installed os - i guess you should reinstall (not that aware of details but it seems to me at least windows-installed boot stuff/layout is different depending on whether it's bios or uefi)

Then how can one install Windows if the installation media could not be detected , I'm using Windows 8.1 ISO ?

_right_ (that's the key part here) innstallation media should work without problems
what processor are you emulating (probably 64 bit)?
what iso are you trying? does it match processor bitness (it may matter)? are you sure it's uefi-bootable?

Offline

#2938 2014-10-11 16:39:48

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

_right_ (that's the key part here) innstallation media should work without problems
what processor are you emulating (probably 64 bit)?
what iso are you trying? does it match processor bitness (it may matter)? are you sure it's uefi-bootable?

1 - Processor is passed-through (-cpu host) which is an i7-5930k .

2 - I'm using the Windows 8.1 Enterprise ISO (MSDN with Update) .

I don't see why shouldn't it be UEFI-compatible , when I switch machine type to 440FX , it detects the ISO , but refuses to boot it . I tried to manually execute the file located at /EFI/BOOT/BOOTX64.EFI  but it hangs the VM for 3 seconds then return to the shell .

Offline

#2939 2014-10-11 17:43:12

Mardok45
Member
Registered: 2014-08-10
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

well i got here native like perfomance in dota2, with hv-time and 2+ cores, remove that smp option maybe?


http://pastie.org/9639320

i'm using this config for streaming on headless server for last 2-3 months and with 340.82 driver it works without problem.

I copied and edited that config and it seemed to help a bit.  It's still not running anywhere near native performance, but Dota 2 is at least playable.  I'm running a 6-core Bulldozer CPU, so maybe I just need to throw better hardware at it.  The VM is running the 334 Nvidia drivers.

Offline

#2940 2014-10-12 04:57:06

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mardok45 wrote:
slis wrote:

well i got here native like perfomance in dota2, with hv-time and 2+ cores, remove that smp option maybe?


http://pastie.org/9639320

i'm using this config for streaming on headless server for last 2-3 months and with 340.82 driver it works without problem.

I copied and edited that config and it seemed to help a bit.  It's still not running anywhere near native performance, but Dota 2 is at least playable.  I'm running a 6-core Bulldozer CPU, so maybe I just need to throw better hardware at it.  The VM is running the 334 Nvidia drivers.

try to disable any sound for vm - just drop any sound-related parameters from xml or commandline
also look at real qemu invocation parameters and look for something that is not needed or simply wrong
check that your memory allocation is normal (hugetlbfs?)
review your cpu allocation scheme (your current config binds to cpus 0-4, but are these cpus isolated from host usage?)

p.s. all this was written too many times all over the thread already, is copy-pasting someone's xml the best of your abilities?

Last edited by sinny (2014-10-12 04:58:03)

Offline

#2941 2014-10-12 11:13:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lspci -nn

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] 02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f] 02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)

dmesg: http://pastebin.com/wp56ts9Y

sudo qemu-system-x86_64 \ -boot menu=on \ -enable-kvm \ -M q35 \ -m 4096 \ -cpu host \ -net none \ -rtc base=localtime \ -smp 4,sockets=1,cores=4,threads=1 \ -netdev tap,ifname=tap0,id=tap0 \ -drive file='/mnt/hdd/qemu-vfio-win.img',id=disk,format=raw,cache=none \ -drive file='/mnt/hdd/virtio.iso',id=cdrom,format=raw,readonly=on \ -device ioh3420,addr=02.0,multifunction=on,port=1,chassis=1,id=root.0 \ -device ioh3420,addr=04.0,multifunction=on,port=1,chassis=2,id=root.1 \ -device ioh3420,addr=06.0,multifunction=on,port=1,chassis=3,id=root.2 \ -device virtio-blk-pci,bus=root.2,addr=04.0,drive=disk \ -device ide-cd,unit=0,drive=cdrom \ -device ich9-intel-hda,bus=root.2,addr=00.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device virtio-net-pci,netdev=tap0,bus=root.2,addr=01.0 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.0,addr=00.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -vga none \ -device qxl,bus=root.2,addr=02.0

BSOD 116 after catalyst install inside windows VM.
Second card doesn't start yet - device code 10.

I have patched host nvidia driver so /dev/vga_arbiter would be

count:3,PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none(0:0)

QEMU 2.1.2

AMD Catalyst 14.9

Also:

options kvm-amd npt=0 blacklist nouveau options nouveau modeset=0

Full dmesg(quite lenghty): http://pastebin.com/HiJh81Ct

ls /sys/kernel/iommu_groups/*/devices:

/sys/kernel/iommu_groups/1/devices: 0000:00:02.0 0000:01:00.0 0000:01:00.1 /sys/kernel/iommu_groups/2/devices: 0000:00:04.0 0000:02:00.0 0000:02:00.1

Full lspci: http://pastebin.com/0ZED3z7U


Any ideas how to:
- fix BSOD? disabling HDMI audio doesn't help anymore
- fix code 10 on second gpu?

Maybe i should use -M pc and -nodefaults?

UPD:Same dmesg messages appear, but for the different device, the gpu itself, when passing only GPUs without HDMI audio.

Last edited by Duelist (2014-10-12 14:03:53)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2942 2014-10-12 11:30:48

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wonder - has anyone tried and succeeded making this work on a laptop? All in all - it should work just fine on laptop with APU and dedicated GPU. What i dont know is how laptop could switch display to dedicated gpu. Anyone know if this is even possible with today's hardware?

Offline

#2943 2014-10-12 12:52:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

I wonder - has anyone tried and succeeded making this work on a laptop? All in all - it should work just fine on laptop with APU and dedicated GPU. What i dont know is how laptop could switch display to dedicated gpu. Anyone know if this is even possible with today's hardware?

I've built a pair of systems with APU and dual graphics.

It does not switch. It works the other way.

APU has it's own GPU inside, sitting on pci-e bus(thanks to UNB and integrated pci-e controller), which handles the video output.
When running dual graphics(having discrete gpu) - it's simply just crossfire between those cards.
On the desktop part, some applications may not support crossfire(or dual graphics may be turned off) and chosen device for rendering will depend from chosen output - for example, if you have your screen plugged in your motherboard then APU's GPU will render the scene and vice versa.
On the notebook part - some notebooks have video output routed physically to the dedicated GPU(usually those notebooks have non-APU versions, but that's quite rare), some have video output routed to APU.

It is not Nvidia Optimus, these notebooks just don't switch GPUs for energy saving, instead, they have zerocore technology in crossfire which cuts power to the secondary GPU.


BUT! All those shiny new("new" as 2012-new) technologies won't work with that particular technology quite right - i couldn't get fglrx working on host AND passing the second card to VM at the same time. fglrx seem to be messing around and scanning whole PC for related vendor-dev-ids and touching those devices without asking anyone.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2944 2014-10-12 22:33:21

Drakulix
Member
Registered: 2014-10-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess getting ignored means, nobody else has an idea?

Offline

#2945 2014-10-13 04:39:34

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Drakulix wrote:

So I guess getting ignored means, nobody else has an idea?

probably. you got very experimental hardware set on top of experimental softwasre feature (==experimental squared?)

plus, me personnally would not expect someone trying to achieve anything (related to the discussion here) on apple garb devices - you either get these to forget everything "complex" or stick with other hardware choices that allow more flexibility/choices

so i'm afraid you are on your own there or, if you are lucky, there will be very little pre-existing experience for you to lean on

Offline

#2946 2014-10-13 08:39:59

Drakulix
Member
Registered: 2014-10-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
Drakulix wrote:

So I guess getting ignored means, nobody else has an idea?

probably. you got very experimental hardware set on top of experimental softwasre feature (==experimental squared?)

plus, me personnally would not expect someone trying to achieve anything (related to the discussion here) on apple garb devices - you either get these to forget everything "complex" or stick with other hardware choices that allow more flexibility/choices

so i'm afraid you are on your own there or, if you are lucky, there will be very little pre-existing experience for you to lean on

Sure it is an experimental setup and sure there are probably not many users, that did try this already.
To be honest, I did not buy this hardware intentionally to do kvm-stuff on it or even install linux. I just recently switched and I am still stunned by the possibilities. Thats why I wanted to give it a try, because surprisingly enough my hardware seemed to meet all requirements. I already got quite far, that is why I want to try on.

And Code 43 seems to be a very common issue, so I did hope, there were some more hints then just "add kvm=off", "remove the hv_* functions" and "put your card in another PCIe-Port" (which would obviously not work on my setup).

I am also not afraid of looking at huge log files or even try to write patches on my own, I just don't know where to start with this weird error. So if anybody has some hints, what I could try, that would be great.

Offline

#2947 2014-10-13 09:02:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Drakulix wrote:

So I guess getting ignored means, nobody else has an idea?

Just following your wishes:

Drakulix wrote:

Please don't tell me right away, that VGA Passthough will not work on my setup.

wink

Seriously, laptops with dual graphics do strange and bizarre things and Code 43 is a pretty standard "I give up" error code from the Nvidia driver.  We don't have the source, so we can only guess what might be wrong.  Sometimes these laptops require vendor drivers for the discrete GPU to work rather than the standard Nvidia driver.  When that GPU gets extracted into a VM, it may not behave exactly like devices supported by the standard driver.  Especially on an Apple product, it's not clear that expecting the Windows driver to work is valid.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2948 2014-10-13 09:19:13

Drakulix
Member
Registered: 2014-10-10
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Drakulix wrote:

So I guess getting ignored means, nobody else has an idea?

Just following your wishes:

Drakulix wrote:

Please don't tell me right away, that VGA Passthough will not work on my setup.

wink

I think, you can guess, that I just wanted to avoid the usual "anti-apple-hate". wink


aw wrote:

Seriously, laptops with dual graphics do strange and bizarre things and Code 43 is a pretty standard "I give up" error code from the Nvidia driver.  We don't have the source, so we can only guess what might be wrong.  Sometimes these laptops require vendor drivers for the discrete GPU to work rather than the standard Nvidia driver.  When that GPU gets extracted into a VM, it may not behave exactly like devices supported by the standard driver.  Especially on an Apple product, it's not clear that expecting the Windows driver to work is valid.

I did always use the unmodified nvidia drivers from their website on a bootcamp installed system without any issue (never installed them either though apple's magical setup program or windows update). Apple isn't even using nvidia optimus, it is just a hardware mux, no weird shared buffers between the two graphic cards. I guess that is as close to desktop graphics, as it can get on a laptop.
Does anybody at least have an idea, how I can get more verbose output from the involved kernel modules and/or qemu? I start reading into the source code if necessary, I am not yet giving up. wink

Last edited by Drakulix (2014-10-13 09:19:54)

Offline

#2949 2014-10-13 17:01:59

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

We don't have the source, so we can only guess what might be wrong.

Even worse.

/* _NVRM_COPYRIGHT_BEGIN_
*
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Copyright 2011 by NVIDIA Corporation.  All rights reserved.  All
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html information contained herein is proprietary and confidential to NVIDIA
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Corporation.  Any use, reproduction, or disclosure without the written
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html permission of NVIDIA Corporation is prohibited.
*
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_11.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html _NVRM_COPYRIGHT_END_
*/

Even if we had - there would be legal issues. Because any use is prohibited. And asking nvidia something like "we want to get our KVM vga passthrough working" would be obviously pointless because... you know why.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2950 2014-10-13 19:45:48

Chetyre
Member
Registered: 2013-03-27
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgraded my kernel to 3.17, but I lost my host graphics.

Actually that happened to in 3.16, but I never mentioned it because I was thinking of messing with ovmf (which failed)

I need the VGA arbiter patch. With it on I can't see anything on host, with it off I get garbled graphics as expected. Back in 3.15 everything just worked. What could be wrong?

Offline

#2951 2014-10-14 12:53:22

dave558
Member
Registered: 2013-11-28
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all

I'm trying to pass through a graphic card to my virtual guests. Steps I've performed so far are as follows:
* Compiled kernel version 3.10 with vfio support (CONFIG_VFIO_PCI_VGA=y)
* Disabled radeon kernel modules via the blacklist in /etc/modprobe.d.
* Unbinded the graphics card from the PCI bus and assigned it the VFIO for pass through.

I have installed qemu-kvm through yum but it appears that this RPM has not been compiled for VFIO. When trying assign the graphic card to guests there is no option to assign device to vfio-pci. The partial command below results in errors:
    /usr/libexec/qemu-kvm -device  vfio-pci

I have read also that qemu 2.0+ supports VFIO and I am thinking of compiling qemu from the QEMU website. Although Qemu can run as a stand-alone emulator, I am assumming it will use the kernel-module automatically if available. Is this the case?

Also will there be any difference in performance between qemu compiled from source and qemu-kvm? Or is there anything else you can recommend? Thank you.

Offline

#2952 2014-10-14 17:03:55

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi to all...

I've got a headless server(Proxmox) with kvm/qemu git with fully working passthrough(pciex) to various vm.

I can passthrough nvidia gpu to windows vm, dvb-s pci express card to ubuntu and a pciexpress ethernet card to another ubuntu host.

My machine runs on kernel 3.16.2 with some patches (intel arbiter) and latest qemu.

I'm trying to create an openelec VM to use as media center  and thus I need intel gpu passthrough.

Basically I don't need a monitor attached because I'm administering with ssh/webinterface.


Grub Settings

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1380,10de:0fbc,4254:0952,7470:3468,8086:0152,8086:1e20"

lspci

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4) 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4) 00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1) 02:00.0 Multimedia video controller: Conexant Systems, Inc. CX23885 PCI Video and Audio Decoder (rev 04) 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

lspci -n

00:00.0 0600: 8086:0150 (rev 09) 00:01.0 0604: 8086:0151 (rev 09) 00:02.0 0300: 8086:0152 (rev 09) 00:14.0 0c03: 8086:1e31 (rev 04) 00:16.0 0780: 8086:1e3a (rev 04) 00:1a.0 0c03: 8086:1e2d (rev 04) 00:1b.0 0403: 8086:1e20 (rev 04) 00:1c.0 0604: 8086:1e10 (rev c4) 00:1c.1 0604: 8086:1e12 (rev c4) 00:1c.2 0604: 8086:1e14 (rev c4) 00:1c.4 0604: 8086:1e18 (rev c4) 00:1d.0 0c03: 8086:1e26 (rev 04) 00:1e.0 0604: 8086:244e (rev a4) 00:1f.0 0601: 8086:1e49 (rev 04) 00:1f.2 0106: 8086:1e02 (rev 04) 00:1f.3 0c05: 8086:1e22 (rev 04) 01:00.0 0300: 10de:1380 (rev a2) 01:00.1 0403: 10de:0fbc (rev a1) 02:00.0 0400: 14f1:8852 (rev 04) 03:00.0 0106: 1b21:0612 (rev 01) 04:00.0 0200: 10ec:8168 (rev 06) 05:00.0 0200: 10ec:8168 (rev 06)

KVM command

/usr/bin/kvm -id 105 -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/105.vnc,x509,password -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=2233690f-30ac-499f-ad7e-91a1e2f62792' -name Openelec -smp 'sockets=1,cores=1' -nodefaults -boot 'menu=on' -vga none -cpu 'kvm64,kvm=off,+lahf_lm,+x2apic,+sep' -k it -m 2048 -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:d4d6185a22' -drive 'file=/var/lib/vz/template/iso/ubuntu-14.04.1-desktop-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/105/vm-105-disk-1.qcow2,if=none,id=drive-ide0,format=qcow2,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:4E:D1:CD:4A:51,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35'

dmesg

Device is ineligible for IOMMU domain attach due to platform RMRR requirement.

Modprobe.d/blacklist.conf

i915 nouveau snd-hda-intel-hdmi snd-hda-codec-hdmi

Intel audio/video is correctly assigned to pci-stub and iommu group are correct but when I run I get

vm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to set iommu for container: Operation not permitted kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to setup container for group 2 kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to get group 2 kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: Device initialization failed. kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: Device 'vfio-pci' could not be initialized

I've seen this patch

https://lkml.org/lkml/2014/7/3/580 that disables devices using  rmrr.

My doubt is if I'm correctly not loading intel drivers so device is free for passthrough...

Thanks in advance

Offline

#2953 2014-10-14 17:10:23

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8


Hi aw,

i have a similar regression - but without passing a nvidia card to the guest (i pass a ati radeon hd to the guest) whichs works flawless with 3.16, but with 3.17 (with and without your patch) i get:


KVM internal error. Suberror: 1 emulation failure EAX=00000000 EBX=00000000 ECX=00000000 EDX=000106a5 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000fff2 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=90 90 eb c3 90 90 90 90 90 90 00 00 00 00 56 54 46 00 90 90 <eb> ac 90 90 90 90 90 90 90 90 90 90 90 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The only change is kernel upgrade from 3.16 to 3.17. I already tried your patch - doesn't seem to help

Greetings

Offline

#2954 2014-10-14 18:52:48

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DanaGoyette wrote:

I actually get something similar when I stop or start (I forgot which) a VM, if I have the AMD High-Definition Audio Controller bound to the VM.  Blacklisting snd-hda-intel on the host prevents the panic, but also prevents the integrated Realtek sound card from working on the host.

EDIT: By "similar", I mean a very similar, if not the same, backtrace.

Try using pci-stub.ids instead of blacklisting the driver

Thank you both. For the whole week not a single crush! I'm so happy smile


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#2955 2014-10-14 19:05:58

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

qemu-system-x86_64 \ -pflash /usr/share/qemu/bios_ovmf.bin \ -enable-kvm -cpu host,hv_time,kvm=off \ -smp cpus=4,cores=4,threads=1 \ -m 8GiB \ -name WinGamingOVMF,process=WinGamingOVMF \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -daemonize \ -rtc base=localtime \ -soundhw hda \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \ -device ide-hd,bus=ide.0,drive=disk_primary \ -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \ -device scsi-hd,bus=scsi.0,drive=disk_games \ -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb

I'm using this on Gentoo, not on Arch, but the above command should be the same. The relevant stuff is this:
Kernel: 3.16.3-gentoo (with all the patches in the OP of this thread, I will try 3.17 without any of these patches later, because iirc none of the patches should be necessary when using OVMF)
Architecture: x86_64
QEMU: GIT master on commit 2472b6c07bb50179019589af1c22f43935ab7f5c
EDK2 (OVMF): SVN trunk on revision 16195


I know that you only asked for the command, but I'd like to share some of my notes from my adventure about using OVMF (started last saturday because pretty much every other VM boot would make my host freeze and force me to power down my whole system the old-fashioned way, got it working sunday):

First, the graphics card you want to pass through absolutely must have either a hybrid, or uefi-only bios; if your graphics card has only
legacy bios (as my GTX 660 TI did), you need to flash it to hybrid / uefi-only, or OVMF won't do anything for you. If you do try
to use a legacy-bios graphics card with OVMF, your virtual machine will detect the passed-through graphics card (even by name in the device manager in case of Windows), but it will not be useable (any monitor you connect to the graphics card will not receive any signal from it / Windows will not detect any monitor connected to the graphics card).

Furthermore, if you want to use Windows (which I'm going to assume), I think you need at least Windows 8 (I tested it only with Windows 8.1 and Windows 7, but Windows 8.1 and 8 extremely similar in nature, so I'm going to assume that they behave the same in regards to this - please correct me on this if I am wrong), because when trying to boot the Windows 7 installation disk, after the usual "Windows is loading files..." the Windows 7 install freezes as soon as the Windows logo shows up.

Additionally, the newer Q35 architecture (option "-M q35") does not seem to work well with Windows 8+. While I could install Windows 8.1 perfectly fine while having q35 enabled, after the installation - at first bootup - the "Getting devices ready" freezes at 15% (on multiple tries, every time). Even if the system is setup completely without q35 (or having it only enabled until first reboot, then shutoff the machine and disable it), enabling q35 will then consistently freeze the VM at the Windows logo.

When trying to install Windows 8+, I would suggest including the relevant disc images like this:

 -drive file=/path/to/${WINDOWS_IMAGE},id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /path/to/virtio-win-0.1-81.iso`

These two should ensure that OVMF recognizes the windows disc image as a UEFI boot option, while having the windows virtio drivers available at install time. I had some trouble with QEMU+OVMF not recognizing any non-virtio disc images as UEFI-bootage (they did not show up on the UEFI shell / at the boot manager). On that note: When trying to install, you will likely automatically enter the EFI shell initially instead of the VM booting from the only available EFI boot option (the windows disc image). The OVMF EFI shell should be a bunch of yellow text (listing all pci devices you could theoretically try to boot from) on black background and a prompt. Quit the prompt (Enter "exit") and you should be taken to the UEFI bios, where you should select "Boot manager", followed by the (emulated) scsi cd drive, that contains the windows disc image.

Also, I suggest compiling OVMF yourself (The "bios_ovmf.bin" in the above command is in fact a symlink to $HOME/src/edk2/Build/OvmfX64/DEBUG_GCC48/QEMU/bios.bin), I found the instructions fairly easy to follow:
First setup and do first build like this:
http://tianocore.sourceforge.net/wiki/U … ke_systems
Afterwards build your real OVMF bios like this:
http://tianocore.sourceforge.net/wiki/How_to_build_OVMF

Lastly, if you use a consumer NVIDIA graphics card, the latest driver (imho) you should use is 340.52, because the 344.11 driver includes a new check whether or not "hv_time" is active and will make your card go to 800x600 with a "Code 43". If you want to use the latest driver, you could also remove the "hv_time" from the qemu command, but according to the QEMU changelog (http://wiki.qemu.org/ChangeLog/2.0) this option can give you a significant performance boost for exactly the type of calculations game engines tend to make heavy use of. See https://forums.geforce.com/default/topi … 00#4314345 for more information on this issue.

Alright, I think that should cover everything you need to know when switching from the usual passthrough to OVMF.
I hope the above information helps you and/or anyone else trying OVMF out, I wrote this down as more of a memo on how I did it than a proper guide, so forgive any lack in quality (I intend to make a proper gist about this when I have the time).

Last edited by Calrama (2014-10-14 19:34:16)

Offline

#2956 2014-10-14 19:23:00

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying to get this too work for a few hours, but something is somehow going wrong and I don't understand how.

The problem is that it seems I can't use pci-stub.
When I add it to /etc/mkinitcpio.conf in the MODULES section it gives me

==> ERROR: Hook 'pci-stub' cannot be found ==> WARNING: errors were encountered during the build. The image may not be complete.

The output of lspci -nn gives me

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

So i've added the following to my bootloader (Grub)

pci-stub.ids=10de:13c0,10de:0fbb

But after rebooting and running "dmesg | grep pci-stub" I only get the following output

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=a0e2f143-41fc-4a4f-905d-6a12ac5b0237 rw quiet pci-stub.ids=10de:13c0,10de:0fbb [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=a0e2f143-41fc-4a4f-905d-6a12ac5b0237 rw quiet pci-stub.ids=10de:13c0,10de:0fbb

So stub didn't actually claim my graphic card.

If I continue without stub it only gives me more errors later on.
I can't just disable my graphic card driver because both of my cards are NVIDIA.

Using the kernel provided in the first post (3.17 + acs override patch + i915 vga arbiter fixes) and a GTX 980 to passthrough combined with a i7 4790k with VT-d enabled.

Offline

#2957 2014-10-14 19:33:11

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

I've been trying to get this too work for a few hours, but something is somehow going wrong and I don't understand how.

The problem is that it seems I can't use pci-stub.
When I add it to /etc/mkinitcpio.conf in the MODULES section it gives me

==> ERROR: Hook 'pci-stub' cannot be found ==> WARNING: errors were encountered during the build. The image may not be complete.
nbhs wrote:

NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

Offline

#2958 2014-10-14 19:50:19

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So no answer then? Like I said I added pci-stub to the MODULES section, but it just gave me the error that it could not find the pci-stub hook...

Offline

#2959 2014-10-14 19:54:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

So no answer then? Like I said I added pci-stub to the MODULES section, but it just gave me the error that it could not find the pci-stub hook...

Check the hooks section then, perhaps you're missing a '"' or, delete /etc/mkinitpio.conf then reinstall it (pacman -S mkinitcpio) and try again

Offline

#2960 2014-10-14 20:19:02

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your help, that actually worked!
I removed the file, and then reinstalled mkinitcpio, added it to modules, and this time it somehow worked.
No clue what went wrong.

Anyways now when I run the test

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

it gives me the following errors:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Any clue what's happening?

Offline

#2961 2014-10-14 20:26:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Thanks for your help, that actually worked!
I removed the file, and then reinstalled mkinitcpio, added it to modules, and this time it somehow worked.
No clue what went wrong.

Anyways now when I run the test

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

it gives me the following errors:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Any clue what's happening?

Uhm yes, you need to bind your devices to vfio-pci, i suggest you re-read the guide, also if you are using an nvidia card on your host, you might need to patch it (though im using an nvidia card myself on the host, with unpatched drivers), also since your card is pretty new and it suports uefi, you might want to try OVMF

Last edited by nbhs (2014-10-14 20:29:02)

Offline

#2962 2014-10-14 20:39:09

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah I figured out I hadn't done that, my mistake, sorry.

Now I have bind my device to vfio-pci, but I get the following:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I guess my main card (the one I use in the host, a GTX 770) is also in this group, which is of course not bound to the vfio bus driver.
Is this because I need to patch my driver?

Also, my system is using BIOS, not UEFI, or is this UEFI different? I rather not reinstall my pc to EUFI when I don't have too.

EDIT: Installed nvidia-dkms, didn't help sadly, still the same error.

Last edited by PureTryOut (2014-10-14 20:48:05)

Offline

#2963 2014-10-14 21:11:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Yeah I figured out I hadn't done that, my mistake, sorry.

Now I have bind my device to vfio-pci, but I get the following:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I guess my main card (the one I use in the host, a GTX 770) is also in this group, which is of course not bound to the vfio bus driver.
Is this because I need to patch my driver?

Also, my system is using BIOS, not UEFI, or is this UEFI different? I rather not reinstall my pc to EUFI when I don't have too.

EDIT: Installed nvidia-dkms, didn't help sadly, still the same error.

That error means that you need to bind ALL the devices on that iommu group to vfio pci, also, you dont need to reinstall anything to use OVMF

ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/

Last edited by nbhs (2014-10-14 21:15:26)

Offline

#2964 2014-10-14 21:28:46

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well yes I understand that, and I would love to bind all of the devices in that group to vfio pci, but it's not possible.
The GPU I use as display in the host (a GTX 770) is also in that group:

ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ 0000:00:01.0 0000:01:00.0 0000:02:00.0 0000:00:01.1 0000:01:00.1 0000:02:00.1
lspci -nn 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

When I bind it my screen just turns black and I can't do anything anymore without restarting my pc.

Anyways I will look into OVMF.

Offline

#2965 2014-10-14 21:58:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Well yes I understand that, and I would love to bind all of the devices in that group to vfio pci, but it's not possible.
The GPU I use as display in the host (a GTX 770) is also in that group:

ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/ 0000:00:01.0 0000:01:00.0 0000:02:00.0 0000:00:01.1 0000:01:00.1 0000:02:00.1
lspci -nn 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

When I bind it my screen just turns black and I can't do anything anymore without restarting my pc.

Anyways I will look into OVMF.

That's what the acs override patch is for

Offline

#2966 2014-10-14 22:42:48

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'd like to point out one thing:

If you're about to flash card without EFI GOP with BIOS (from similar card) that support EFI - don't do this (unless you're sure). You can pass BIOS with qemu adding 'romfile=...' like this:

-device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7,rombar=0,romfile=/var/lib/libvirt/firmware/HD7870_EFI.rom

Offline

#2967 2014-10-15 11:36:44

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Oh dear, you've gone to all this trouble to compile OVMF but haven't read this post to determine why you'd want it and how to use it?  Have I missed something in describing the motivation and requirements there?  OVMF isn't a "test", it's an alternative to VGA.  Since you're not getting any output from the monitor, it seems like you have an issue with VGA routing.  Avoiding VGA by using OVMF is one method of dealing with this.

Ok, now it's a bit more clear.


aw wrote:

Regarding the parse error, how recently did you grab that copy of rom-parser?  I fixed a similar issue in the code on the 4th.  Otherwise please send me a copy of the ROM binary so I can fix the code for it.  Since you already have the edk2 tree, there's also a parser there you can use, details in the comments here.

I used latest code from your master branch.
At what address should I send you the bios?


aw wrote:

For the -bios option; you shouldn't ever need a -bios option.  The default bios that comes with QEMU is fine for seabios use and for OVMF you'd add an option like:

-drive if=pflash,format=raw,readonly,file=/path/to/OVMF.fd

Ok, added that line and I can boot in a sort of efi shell, but can't see any kind of "boot menu" or something similar.
Consider that I'm connecting through vnc so I could miss some initial boot screens.


aw wrote:

Your previous posts indicate that you're using Windows 7, so please pay particular attention to the guest operating support for UEFI.  You'd need to upgrade your guest to Windows 8 for OVMF.

Searching on the web, I found some guides about how booting/installing Windows 7 over UEFI. Can I follow such a tutorial or OVMF/VM is different?


aw wrote:

If that's not any option, we'll need to go back to debugging your VGA routing issue and 'dmesg | grep -i vga' might shed some light on what's wrong.  Please also pastebin the entire dmesg log from the host.

# dmesg | grep -i vga Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.5 root=UUID=a4837900-ca63-43d4-bc2c-cf3f37cff7c6 ro i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b quiet Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.5 root=UUID=a4837900-ca63-43d4-bc2c-cf3f37cff7c6 ro i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b quiet Console: colour VGA+ 80x25 vgaarb: setting as boot device: PCI:0000:00:02.0 vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none vgaarb: loaded vgaarb: bridge control possible 0000:01:00.0 vgaarb: no bridge control possible 0000:00:02.0 [drm] Replacing VGA console driver vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

PS: sorry for the delay, but I got some trouble.

Offline

#2968 2014-10-15 12:53:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://bbs.archlinux.org/viewtopic.php … 7#p1465127

Downgraded bios to the oldest version having IOMMU support - still getting page faults.
If i pass only one GPU without sound - windows 7 still fails to boot with BSOD 116.

I am using nvidia gpu as a host gpu, by the way.

Passing second(host:02:00.00) GPU doesn't show any video output, but there is still
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0021 address=0x0000000122dde000 flags=0x0010]
errors.

UPD: something wild and unknown happened - windows now may boot, but it'll do it with horrible memory corruption artifacts looking like horizontal stripes, and then BSOD. Cryptic messages in dmesg are gone...

Last edited by Duelist (2014-10-15 16:33:13)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2969 2014-10-15 14:56:33

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

qemu-system-x86_64 \ -pflash /usr/share/qemu/bios_ovmf.bin \ -enable-kvm -cpu host,hv_time,kvm=off \ -smp cpus=4,cores=4,threads=1 \ -m 8GiB \ -name WinGamingOVMF,process=WinGamingOVMF \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -daemonize \ -rtc base=localtime \ -soundhw hda \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \ -device ide-hd,bus=ide.0,drive=disk_primary \ -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \ -device scsi-hd,bus=scsi.0,drive=disk_games \ -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb

I'm using this on Gentoo, not on Arch, but the above command should be the same. The relevant stuff is this:
Kernel: 3.16.3-gentoo (with all the patches in the OP of this thread, I will try 3.17 without any of these patches later, because iirc none of the patches should be necessary when using OVMF)
Architecture: x86_64
QEMU: GIT master on commit 2472b6c07bb50179019589af1c22f43935ab7f5c
EDK2 (OVMF): SVN trunk on revision 16195


I know that you only asked for the command, but I'd like to share some of my notes from my adventure about using OVMF (started last saturday because pretty much every other VM boot would make my host freeze and force me to power down my whole system the old-fashioned way, got it working sunday):

First, the graphics card you want to pass through absolutely must have either a hybrid, or uefi-only bios; if your graphics card has only
legacy bios (as my GTX 660 TI did), you need to flash it to hybrid / uefi-only, or OVMF won't do anything for you. If you do try
to use a legacy-bios graphics card with OVMF, your virtual machine will detect the passed-through graphics card (even by name in the device manager in case of Windows), but it will not be useable (any monitor you connect to the graphics card will not receive any signal from it / Windows will not detect any monitor connected to the graphics card).

Furthermore, if you want to use Windows (which I'm going to assume), I think you need at least Windows 8 (I tested it only with Windows 8.1 and Windows 7, but Windows 8.1 and 8 extremely similar in nature, so I'm going to assume that they behave the same in regards to this - please correct me on this if I am wrong), because when trying to boot the Windows 7 installation disk, after the usual "Windows is loading files..." the Windows 7 install freezes as soon as the Windows logo shows up.

Additionally, the newer Q35 architecture (option "-M q35") does not seem to work well with Windows 8+. While I could install Windows 8.1 perfectly fine while having q35 enabled, after the installation - at first bootup - the "Getting devices ready" freezes at 15% (on multiple tries, every time). Even if the system is setup completely without q35 (or having it only enabled until first reboot, then shutoff the machine and disable it), enabling q35 will then consistently freeze the VM at the Windows logo.

When trying to install Windows 8+, I would suggest including the relevant disc images like this:

 -drive file=/path/to/${WINDOWS_IMAGE},id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /path/to/virtio-win-0.1-81.iso`

These two should ensure that OVMF recognizes the windows disc image as a UEFI boot option, while having the windows virtio drivers available at install time. I had some trouble with QEMU+OVMF not recognizing any non-virtio disc images as UEFI-bootage (they did not show up on the UEFI shell / at the boot manager). On that note: When trying to install, you will likely automatically enter the EFI shell initially instead of the VM booting from the only available EFI boot option (the windows disc image). The OVMF EFI shell should be a bunch of yellow text (listing all pci devices you could theoretically try to boot from) on black background and a prompt. Quit the prompt (Enter "exit") and you should be taken to the UEFI bios, where you should select "Boot manager", followed by the (emulated) scsi cd drive, that contains the windows disc image.

Also, I suggest compiling OVMF yourself (The "bios_ovmf.bin" in the above command is in fact a symlink to $HOME/src/edk2/Build/OvmfX64/DEBUG_GCC48/QEMU/bios.bin), I found the instructions fairly easy to follow:
First setup and do first build like this:
http://tianocore.sourceforge.net/wiki/U … ke_systems
Afterwards build your real OVMF bios like this:
http://tianocore.sourceforge.net/wiki/How_to_build_OVMF

Lastly, if you use a consumer NVIDIA graphics card, the latest driver (imho) you should use is 340.52, because the 344.11 driver includes a new check whether or not "hv_time" is active and will make your card go to 800x600 with a "Code 43". If you want to use the latest driver, you could also remove the "hv_time" from the qemu command, but according to the QEMU changelog (http://wiki.qemu.org/ChangeLog/2.0) this option can give you a significant performance boost for exactly the type of calculations game engines tend to make heavy use of. See https://forums.geforce.com/default/topi … 00#4314345 for more information on this issue.

Alright, I think that should cover everything you need to know when switching from the usual passthrough to OVMF.
I hope the above information helps you and/or anyone else trying OVMF out, I wrote this down as more of a memo on how I did it than a proper guide, so forgive any lack in quality (I intend to make a proper gist about this when I have the time).

Wow .. Thank you very much for this detailed walkthrough , I'm going to try ASAP and return with results !

Thank you for taking your time typing this detailed information !

EDIT !!!! :

YES ! It worked ! finally it detected my Windows 8.1 ISO ! The magic word is : SCSI .

It didn't detect regular IDE , but it detected SCSI .

Thank you Calrama !

Last edited by Denso (2014-10-15 15:13:44)

Offline

#2970 2014-10-15 16:48:16

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

That's what the acs override patch is for

Yes that is what I understand as well... I have installed your version of the kernel (3.17 + acs override patch + i915 vga arbiter fixes), but it still gives me the same error.

Offline

#2971 2014-10-15 16:51:23

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Calrama wrote:
Denso wrote:

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

qemu-system-x86_64 \ -pflash /usr/share/qemu/bios_ovmf.bin \ -enable-kvm -cpu host,hv_time,kvm=off \ -smp cpus=4,cores=4,threads=1 \ -m 8GiB \ -name WinGamingOVMF,process=WinGamingOVMF \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -daemonize \ -rtc base=localtime \ -soundhw hda \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \ -device ide-hd,bus=ide.0,drive=disk_primary \ -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \ -device scsi-hd,bus=scsi.0,drive=disk_games \ -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb

[...]

Wow .. Thank you very much for this detailed walkthrough , I'm going to try ASAP and return with results !

Thank you for taking your time typing this detailed information !

EDIT !!!! :

YES ! It worked ! finally it detected my Windows 8.1 ISO ! The magic word is : SCSI .

It didn't detect regular IDE , but it detected SCSI .

Thank you Calrama !

That's good to hear and you're welcome.

Thanks for pointing out that SCSI is the key and not VIRTIO itself, I'll take that into consideration when writing the gist.

Have you experienced any host freezes / crashes with OVMF? I'm just curious, because for me it has - so far - been the all-cure regarding that.
Since I started using OVMF I haven't had a single crash or host freeze and even better, OVMF seems to be a lot faster than the normal bios.
My Gaming VM takes only 15 seconds from the time I issue the qemu command until it's fully-booted onto the Windows desktop.

Offline

#2972 2014-10-15 17:51:18

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:
Calrama wrote:
qemu-system-x86_64 \ -pflash /usr/share/qemu/bios_ovmf.bin \ -enable-kvm -cpu host,hv_time,kvm=off \ -smp cpus=4,cores=4,threads=1 \ -m 8GiB \ -name WinGamingOVMF,process=WinGamingOVMF \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -daemonize \ -rtc base=localtime \ -soundhw hda \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \ -device ide-hd,bus=ide.0,drive=disk_primary \ -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \ -device scsi-hd,bus=scsi.0,drive=disk_games \ -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb

[...]

Wow .. Thank you very much for this detailed walkthrough , I'm going to try ASAP and return with results !

Thank you for taking your time typing this detailed information !

EDIT !!!! :

YES ! It worked ! finally it detected my Windows 8.1 ISO ! The magic word is : SCSI .

It didn't detect regular IDE , but it detected SCSI .

Thank you Calrama !

That's good to hear and you're welcome.

Thanks for pointing out that SCSI is the key and not VIRTIO itself, I'll take that into consideration when writing the gist.

Have you experienced any host freezes / crashes with OVMF? I'm just curious, because for me it has - so far - been the all-cure regarding that.
Since I started using OVMF I haven't had a single crash or host freeze and even better, OVMF seems to be a lot faster than the normal bios.
My Gaming VM takes only 15 seconds from the time I issue the qemu command until it's fully-booted onto the Windows desktop.

Hi smile

I can confirm that NOW I can reboot VM without freezing the host ! I believe that is because OVMF is legacy-free (VGA-free) .

On a side note , OVMF initializes the machine faster than SeaBIOS does . (a couple seconds faster) .

I can't thank you enough (I'm typing this reply from my new OVMF VM !) wink

Offline

#2973 2014-10-15 18:09:28

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so no one passing through intel gpu?

Offline

#2974 2014-10-15 18:21:09

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://awilliam.github.io/presentations/KVM-Forum-2014

Fresh aw's presentation works like sort of a FAQ for everybody. Now i know that i definitely need OVMF.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2975 2014-10-15 18:43:41

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

so no one passing through intel gpu?

You need KvmGT (which isn't released for wide public yet) or XenGT.

Offline

#2976 2014-10-15 18:46:41

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
evilsephiroth wrote:

so no one passing through intel gpu?

You need KvmGT (which isn't released for wide public yet) or XenGT.

So they intentionally excluded igpus for passthrough...

Now it is clear...

I think will rely on another pci-ex gpu...

Offline

#2977 2014-10-15 18:53:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
evilsephiroth wrote:

so no one passing through intel gpu?

You need KvmGT (which isn't released for wide public yet) or XenGT.

http://sched.co/1B1c3ia

Heh. Heh-heh. Cutting-edge technology.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2978 2014-10-15 20:20:20

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:

That's what the acs override patch is for

Yes that is what I understand as well... I have installed your version of the kernel (3.17 + acs override patch + i915 vga arbiter fixes), but it still gives me the same error.

Don't know if anybody else can help me...

When running the test, I get the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

In the same iommu_group are the following:

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

Of course I can't just bind my main card that is running on the host (the GTX 770) to vfio: it just crashes my system and I have to force restart by pressing the power button.

As I said, i'm running the kernel provided in the first post, so the acs patch is enabled.

Offline

#2979 2014-10-15 20:54:44

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
PureTryOut wrote:
nbhs wrote:

That's what the acs override patch is for

Yes that is what I understand as well... I have installed your version of the kernel (3.17 + acs override patch + i915 vga arbiter fixes), but it still gives me the same error.

Don't know if anybody else can help me...

When running the test, I get the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

In the same iommu_group are the following:

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

Of course I can't just bind my main card that is running on the host (the GTX 770) to vfio: it just crashes my system and I have to force restart by pressing the power button.

As I said, i'm running the kernel provided in the first post, so the acs patch is enabled.

You need to enable it with a kernel parameter, you can see it mentioned on almost every page on this thread

Offline

#2980 2014-10-15 21:25:59

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm sorry I am a newb and I feel like a newb...

Searching for a few pages the only thing I could find was this article explaining iommu groups, and the kernel parameter "acs_override=downstream", which did nothing.
I'm sorry if I am irritating you with these dumb questions, but I really want to fix this.

Could you give me the right parameter I have to use?

Offline

#2981 2014-10-15 22:14:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

I'm sorry I am a newb and I feel like a newb...

Searching for a few pages the only thing I could find was this article explaining iommu groups, and the kernel parameter "acs_override=downstream", which did nothing.
I'm sorry if I am irritating you with these dumb questions, but I really want to fix this.

Could you give me the right parameter I have to use?

i believe the correct parameter is "pcie_acs_override=downstream"

 pcie_acs_override = [PCIE] Override missing PCIe ACS support for: downstream All downstream ports - full ACS capabilties multifunction All multifunction devices - multifunction ACS subset id:nnnn:nnnn Specfic device - full ACS capabilities Specified as vid:did (vendor/device ID) in hex

Last edited by nbhs (2014-10-15 22:26:08)

Offline

#2982 2014-10-16 00:34:07

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To everyone who is interested in running OVMF :

You need a seperate OVMF.bin file for each VM you are running .

Took me 2 hours to realize that every VM modifies the "/usr/share/ovmf/x64/ovmf_x64.bin" in a way that makes it incompatible with other VMs .

So I took 3 copies of ovmf_x64.bin file and named it after each VM I had :

ovmf_main.bin
ovmf_gaming.bin
ovmf_winsrv.bin

and pointed each VM to its corresponding OVMF file . and they all worked .

If you're using a split OVMF package , you need a seperate copy of ovmf_vars_x64.bin for each VM , while the ovmf_code_x64.bin can be used for multiple VMs safely .

If you're indeed using the split package use :

-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/A_COPY_OF_ovmf_vars_x64.bin_FOR_THIS_SPECIFIC_VM.bin \

and you should be good to go .

Last edited by Denso (2014-10-16 01:21:53)

Offline

#2983 2014-10-16 15:17:45

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

i believe the correct parameter is "pcie_acs_override=downstream"

 pcie_acs_override = [PCIE] Override missing PCIe ACS support for: downstream All downstream ports - full ACS capabilties multifunction All multifunction devices - multifunction ACS subset id:nnnn:nnnn Specfic device - full ACS capabilities Specified as vid:did (vendor/device ID) in hex

Thanks! That worked perfectly, and I can now succesfully run the test.

Now up to the next newb question/problem (sorry hmm):
At this moment I have a dualboot configuration:

lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda ├─sda1 ext4 a0e2f143-41fc-4a4f-905d-6a12ac5b0237 / ├─sda2 ntfs System Reserved 8E4242894242764D │ └─md0 │ ├─md0p1 ntfs System Reserved 8E4242894242764D │ └─md0p2 ntfs 7AC4495FC4491EAF └─sda3 ntfs 7AC4495FC4491EAF └─md0 ├─md0p1 ntfs System Reserved 8E4242894242764D └─md0p2 ntfs 7AC4495FC4491EAF sdb ├─sdb1 ext4 Home 34e2c056-db9e-45a1-879d-90bd5840929f /home ├─sdb2 ext4 ExtraLinux 5b069a30-2bf2-4bc5-991b-1f3a54c03a69 /games2 └─sdb3 swap Swap 137477b2-bc03-4029-b579-a9120ed058ee [SWAP] sdc ├─sdc1 ntfs Games Windows 4DAA5B4E7792ADD1 └─sdc3 ntfs Data Windows 472386C540D36DE7 loop0 └─md0 ├─md0p1 ntfs System Reserved 8E4242894242764D └─md0p2 ntfs 7AC4495FC4491EAF

sda1 is my Linux root, sda2 is the Windows Recovery and sda3 is the Windows partition.
I would like to virtualize the physical installation, so I benefit of the SSD read and write speeds.

As you can see I succesfully setup NRAID, which I can mount and browse files on.
Now I want to add this RAID and the whole sdc disk to the VM but i'm having some problems.

I use the following command:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -cdrom "/home/bart/Documents/ISO's/32 & 64-bit Windows 7 Ultimate - Service Packet 1 - Multilanguage.iso" -boot order=d \ -drive file=/dev/md0,id=diskboot,format=raw -device ide-hd,bus=ahci.0,drive=diskboot \ -drive file=/dev/sdc,id=diskdata,format=raw -device ide-hd,bus=ahci.1,drive=diskdata

Now I guess I just don't understand really well what "bus" and "ahci" means, but it's giving me the following error:

qemu-system-x86_64: -device ide-hd,bus=ahci.0,drive=diskboot: Bus 'ahci.0' not found

If I try to run without "bus=ahci.0" (and only 1 disk otherwise it says bus supports only 1 unit), it just gives me the qemu command line in which I have no clue what to do.

I'm really sorry for bothering you with all these newbie (and probably simple) questions, but i'm almost there and it would be a shame if I drop it now.

Offline

#2984 2014-10-16 15:24:39

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:

i believe the correct parameter is "pcie_acs_override=downstream"

 pcie_acs_override = [PCIE] Override missing PCIe ACS support for: downstream All downstream ports - full ACS capabilties multifunction All multifunction devices - multifunction ACS subset id:nnnn:nnnn Specfic device - full ACS capabilities Specified as vid:did (vendor/device ID) in hex

Thanks! That worked perfectly, and I can now succesfully run the test.

Now up to the next newb question/problem (sorry hmm):
At this moment I have a dualboot configuration:

lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda ├─sda1 ext4 a0e2f143-41fc-4a4f-905d-6a12ac5b0237 / ├─sda2 ntfs System Reserved 8E4242894242764D │ └─md0 │ ├─md0p1 ntfs System Reserved 8E4242894242764D │ └─md0p2 ntfs 7AC4495FC4491EAF └─sda3 ntfs 7AC4495FC4491EAF └─md0 ├─md0p1 ntfs System Reserved 8E4242894242764D └─md0p2 ntfs 7AC4495FC4491EAF sdb ├─sdb1 ext4 Home 34e2c056-db9e-45a1-879d-90bd5840929f /home ├─sdb2 ext4 ExtraLinux 5b069a30-2bf2-4bc5-991b-1f3a54c03a69 /games2 └─sdb3 swap Swap 137477b2-bc03-4029-b579-a9120ed058ee [SWAP] sdc ├─sdc1 ntfs Games Windows 4DAA5B4E7792ADD1 └─sdc3 ntfs Data Windows 472386C540D36DE7 loop0 └─md0 ├─md0p1 ntfs System Reserved 8E4242894242764D └─md0p2 ntfs 7AC4495FC4491EAF

sda1 is my Linux root, sda2 is the Windows Recovery and sda3 is the Windows partition.
I would like to virtualize the physical installation, so I benefit of the SSD read and write speeds.

As you can see I succesfully setup NRAID, which I can mount and browse files on.
Now I want to add this RAID and the whole sdc disk to the VM but i'm having some problems.

I use the following command:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -cdrom "/home/bart/Documents/ISO's/32 & 64-bit Windows 7 Ultimate - Service Packet 1 - Multilanguage.iso" -boot order=d \ -drive file=/dev/md0,id=diskboot,format=raw -device ide-hd,bus=ahci.0,drive=diskboot \ -drive file=/dev/sdc,id=diskdata,format=raw -device ide-hd,bus=ahci.1,drive=diskdata

Now I guess I just don't understand really well what "bus" and "ahci" means, but it's giving me the following error:

qemu-system-x86_64: -device ide-hd,bus=ahci.0,drive=diskboot: Bus 'ahci.0' not found

If I try to run without "bus=ahci.0" (and only 1 disk otherwise it says bus supports only 1 unit), it just gives me the qemu command line in which I have no clue what to do.

I'm really sorry for bothering you with all these newbie (and probably simple) questions, but i'm almost there and it would be a shame if I drop it now.

I dont know where you got ahci.0 from, the correct bus using the default q35 ahci controller is ide.*

Offline

#2985 2014-10-16 15:56:51

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cryptonymous wrote:

Can anyone who have tried using OVMF confirm that initial OVMF bootup messages and EFI shell are shown on a monitor connected to a passed-through video card? I can seem to get even this basic thing to work with OVMF.

Command line I've tried:

qemu-system-x86_64 -enable-kvm -machine pc-i440fx-2.1,accel=kvm,usb=off -device vfio-pci,host=01:00.0,bus=pci.0,addr=7,multifunction=on,romfile=/home/user/NV280MS1-EFI.rom -device vfio-pci,host=01:00.1,bus=pci.0,addr=7.1 -drive if=pflash,format=raw,readonly,file=/tmp/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd -vga none -nographic

All messages, including EFI shell, are shown on QEMU's curses console. Nothing is shown on a passed-though video card. Does it mean EFI ROM that performs video card initialization doesn't run properly? If I type `pci' command in EFI shell, I see that correct VID/PID pair is listed for the video card, thus it is clearly detected.

To answer my own question since no one bothered: yes, OVMF messages are shown both on QEMU's curses console and a physical monitor once the video card is correctly initialized. Also `drivers' EFI shell command should list proper driver being loaded, e.g.:

 T D Y C I P F A DRV VERSION E G G #D #C DRIVER NAME IMAGE PATH === ======== = = = === === =================================== ========== 89 0001000B B N N 2 5 NVIDIA GPU UEFI Driver (80.07.35.00 PciRoot(0x0)/Pci(0x3,0x0)/Offset(0xFC00,0x1F9FF)

Will try to install EFI-compatible OS next.

cryptonymous wrote:

The ROM file I use (NV280MS1-EFI.rom) does have a EFI part:

Valid ROM signature found @400h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 0, vendor revision: 1 Valid ROM signature found @10000h, PCIR offset 1ch PCIR: type 3, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 3, vendor revision: 0 EFI: Signature Valid Last image

Video card in question is a NVidia GTX 650 (MSI N650 PE 1GD5/OC). Initially it came with hybrid (EFI/Legacy) BIOS flashed, later update was just Legacy. I'm trying to use original ROM I've fortunately backed up before reflashing.

It took me a while to discover that video BIOS images read and written by nvflash.exe tool can't be directly passed to QEMU's vfio-pci device `romfile=' option. The part starting with `NVGI' signature up to the first `0x55 0xaa' signature has to be skipped (that was 0x400 bytes in my case). Strange that QEMU didn't complain about invalid ROM.

Offline

#2986 2014-10-16 16:07:42

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I dont know where you got ahci.0 from, the correct bus using the default q35 ahci controller is ide.*

Ooh well I got it from your post in the "Using a physical disk or partition" section wink

Well it seems to work? Gives no errors but only shows me a console.

I've tried "boot order=d" (got it from the Arch Wiki) and "boot menu=on", none made any difference.

Offline

#2987 2014-10-17 09:37:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:

I dont know where you got ahci.0 from, the correct bus using the default q35 ahci controller is ide.*

Ooh well I got it from your post in the "Using a physical disk or partition" section wink

Well it seems to work? Gives no errors but only shows me a console.

I've tried "boot order=d" (got it from the Arch Wiki) and "boot menu=on", none made any difference.

First: I guess you should have windows' loader(NTLDR before Vista, BCD or what was it after?..) somewhere on disk. Usually it gets into MBR. You could also chainload it via GRUB. There may be major differencies if using GPT.

Second: I think you could benifit from use of virtio, either blk or scsi. But...

Third: You'll need to migrate your windows installation correctly. Windows won't boot with bsod 7B(if i remember correctly) when you change the disk controller to something else not present @ system installation. If you would like to use qemu's AHCI - windows should have drivers for it, you'll just need to activate them. If you wish to use virtio(it does improve my disk performance somewhat, but my image resides in a file on my HDD) - you must pass down virtio cd iso via most compatible interface and install drivers manually.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#2988 2014-10-17 12:24:41

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so restart from the base...
I'm getting some problems setting up a VM (using OVMF) with Windows 8.1 and a GeForce GTX 650 passed through.

Software setup:
- Debian 7 (headless pc, managed through ssh)
- kernel 3.16.6 + ACS patch + i915 arbitrer patch + VGA arbitrer patch
- boot parameters "i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b"
- qemu 2.1.2
- seabios 1.7.5
- OVMF latest-git

Hardware setup:
- MSI ZH77A-G43
- Intel i5-3470
- IGD as primary VGA (for host)
- nVidia GeForce GTX 650 as secondary VGA (for guest)

Steps I followed:
- enabled VT-d in motherboard
- installed Windows 8.1 through VNC using "-vga std" (keeping GTX 650 as secondary graphics card in guest)
- installed tightVNC inside Windows 8.1
- installed nVidia graphics driver (version 340.52)
- booted Windows 8.1 with "-vga none"
- cannot connect to VNC server inside Windows 8.1 (without any error from qemu)


Script used to start VM:

# Bind VGA DEVICES="0000:01:00.0 0000:01:00.1" for dev in $DEVICES ; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done # Start QEMU with UEFI (OVMF) qemu-system-x86_64 -enable-kvm -m 4096 -cpu host,hv_time,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -vga none -rtc base=localtime \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive file=/dev/sdc,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ -drive file=/root/Windows8.1.iso,id=isocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=isocd \ -drive if=pflash,format=raw,readonly,file=/usr/share/qemu/OVMF.fd \ -drive if=pflash,format=raw,file=/root/OVMF_vars.fd \ -k it -boot order=dc,menu=on \ -netdev tap,ifname=qemu0,id=qemu_tap -device e1000,netdev=qemu_tap,mac=00:16:3E:12:34:56 \ -usb -usbdevice tablet

Output after launching qemu:

vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 kvm: zapping shadow pages for mmio generation wraparound

Some dmesg messages about vga:

# dmesg | grep -i vga Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.6 root=UUID=a4837900-ca63-43d4-bc2c-cf3f37cff7c6 ro i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b quiet Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.6 root=UUID=a4837900-ca63-43d4-bc2c-cf3f37cff7c6 ro i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b quiet Console: colour VGA+ 80x25 vgaarb: setting as boot device: PCI:0000:00:02.0 vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none vgaarb: loaded vgaarb: bridge control possible 0000:01:00.0 vgaarb: no bridge control possible 0000:00:02.0 [drm] Replacing VGA console driver vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Output from rom-parser:

$ ./rom-parser gtx650.rom Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 0, vendor revision: 1 Valid ROM signature found @f000h, PCIR offset 1ch PCIR: type 3, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 3, vendor revision: 0 EFI: Signature Valid Last image

I'm quite sure the VM is booting because I can arping its ip, but I can't connect to its VNC server (as I said above, I installed tightVNC into Windows).
I will also try connecting a physical monitor to the GTX 650 ASAP (hoping to see some graphical output).

Any help/though is appreciated, thanks.

Offline

#2989 2014-10-17 15:29:01

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

Ok, so restart from the base...
I'm getting some problems setting up a VM (using OVMF) with Windows 8.1 and a GeForce GTX 650 passed through.

Script used to start VM:

# Start QEMU with UEFI (OVMF) qemu-system-x86_64 -enable-kvm -m 4096 -cpu host,hv_time,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -vga none -rtc base=localtime \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive file=/dev/sdc,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ -drive file=/root/Windows8.1.iso,id=isocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=isocd \ -drive if=pflash,format=raw,readonly,file=/usr/share/qemu/OVMF.fd \ -drive if=pflash,format=raw,file=/root/OVMF_vars.fd \ -k it -boot order=dc,menu=on \ -netdev tap,ifname=qemu0,id=qemu_tap -device e1000,netdev=qemu_tap,mac=00:16:3E:12:34:56 \ -usb -usbdevice tablet

Your launching code seems ok , but I don't think you need "x-vga=on" with OVMF . I'm running this VM without it . I don't know if it's related to your issue , but try to remove x-vga=on option and see if it works .

EDIT :

Also , try to configure tightVNC to accept connections without passwords , it's easier .

Last edited by Denso (2014-10-17 15:33:17)

Offline

#2990 2014-10-17 15:36:40

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I get this message, when I try to start qemu:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 3 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I use the standard kernel (3.16.4-1-ARCH) with neither acs override patch nor i915 vga arbiter fixes patch. Can this be the issue? I do not want to compile the kernel myself, unless realy needed.

zgrep CONFIG_VFIO_PCI_VGA /proc/config.gz CONFIG_VFIO_PCI_VGA=y

Offline

#2991 2014-10-17 15:40:25

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

I get this message, when I try to start qemu:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 3 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I use the standard kernel (3.16.4-1-ARCH) with neither acs override patch nor i915 vga arbiter fixes patch. Can this be the issue? I do not want to compile the kernel myself, unless realy needed.

zgrep CONFIG_VFIO_PCI_VGA /proc/config.gz CONFIG_VFIO_PCI_VGA=y

Check if IOMMU is supported by your CPU/Motherboard , and that it is probably supported/enabled in BIOS . Also make sure you added intel_iommu=on  OR  amd_iommu=on  to your boot parameters .

Offline

#2992 2014-10-17 15:46:08

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

- qemu 2.1.2

Could you please try out the latest git master instead of 2.1.2 (My qemu gives "2.1.50" on "-version")? May not help, but I have never tried this method with any of the official releases.

Ansa89 wrote:

- OVMF latest-git

Their git repo is just a mirror of the actual development subversion trunk, so it may lag behind.
If the git repo does lack behind, you might want to try the latest svn trunk.

Ansa89 wrote:

- kernel 3.16.6 + ACS patch + i915 arbitrer patch + VGA arbitrer patch
- boot parameters "i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b"

This may not be the solution, but it should reduce complexity:
First, you should not need these patches when going the UEFI way with OVMF, so please try to use a vanilla 3.16.6 kernel. I've used both unpatched 3.16.3 and 3.17.1 with OVMF successfully (unpatched meaning without these three patches, they did have Gentoo-specific patches).

Second, the boot paramater "i915.enable_hd_vgaarb=1" should also not be necessary when going the UEFI way with OVMF, I only have "intel_iommu=on" and "pci-stub.ids=[GPU and GPU AUDIO]".

Ansa89 wrote:
 -drive file=/dev/sdc,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ -drive file=/root/Windows8.1.iso,id=isocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=isocd \

Try to use SCSI instead of IDE for these two drives

Ansa89 wrote:
 -boot order=dc,menu=on \

Try to remove this. OVMF is a UEFI firmware, so IIRC it should store the UEFI boot options in your VM's individual copy of the OVMF_vars.fd template file.

Ansa89 wrote:

Output from rom-parser:

$ ./rom-parser gtx650.rom Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 0, vendor revision: 1 Valid ROM signature found @f000h, PCIR offset 1ch PCIR: type 3, vendor: 10de, device: 0fc6, class: 030000 PCIR: revision 3, vendor revision: 0 EFI: Signature Valid Last image

Please forgive the potentially stupid question, but how did you get this rom file? After I flashed my GTX 660 TI with the Asus tool to a hybrid firmware (which worked and I can use the card perfectly fine) the

# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom # cat /sys/bus/pci/devices/0000:01:00.0/rom

just returns the following for me:

cat: rom: Input/output error

So I assumed that this way doesn't work with slightly older NVIDIA cards. Or does this still work for you?

Ansa89 wrote:

I'm quite sure the VM is booting because I can arping its ip,

What does the serial console show? Your VM might hang at the EFI shell (with the emulated network card already initialised and connected to th TAP device). To see the serial console open the popup windows that starts when you execute the qemu command, and navigate to the menu item "View -> serial0". if this shows a bunch of slightly garbled yellow text on black, your VM might hang at the EFI shell.

Also, if none of the above helps you, I would like to know what the monitor your wrote you intent to connect to the graphics card shows from the moment you boot the machine.

Offline

#2993 2014-10-17 16:01:49

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Your launching code seems ok , but I don't think you need "x-vga=on" with OVMF .

You're both right and wrong about that if I understand this correctly (third to last paragraph).

You need the quirks this option enables if you have an NVIDIA card like him (and me) regardless of whether you're using OVMF or not, but if your QEMU has this commit, then you don't need to specify the option explicitly, because the commit ensures that the quirks are automatically enabled for NVIDIA cards. I haven't checked, but it might be that this commit is not in qemu 2.1.2 (in which case he needs to specify it).

Last edited by Calrama (2014-10-17 16:02:47)

Offline

#2994 2014-10-17 16:04:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:

Your launching code seems ok , but I don't think you need "x-vga=on" with OVMF .

You're both right and wrong about that if I understand this correctly (third to last paragraph).

You need the quirks this option enables if you have an NVIDIA card like him (and me) regardless of whether you're using OVMF or not, but if your QEMU has this commit, then you don't need to specify the option explicitly, because the commit ensures that the quirks are automatically enabled for NVIDIA cards. I haven't checked, but it might be that this commit is not in qemu 2.1.2 (in which case he needs to specify it).

That's right, this is why anyone trying to do nvidia+ovmf should be using qemu.git until QEMU 2.2.  I haven't used, and don't plan to try, x-vga=on with ovmf.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2995 2014-10-17 16:22:07

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

According to that commit , those quirks needed are activated regardless of x-vga=on , so anyone using a recent compiled qemu-git should be fine without it when using OVMF .

Offline

#2996 2014-10-17 16:23:06

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

According to that commit , those quirks needed are activated regardless of x-vga=on , so anyone using a recent compiled qemu-git should be fine without it when using OVMF .

Yes, but he's using qemu 2.1.2, not qemu-git, which is why I wrote

Calrama wrote:

I haven't checked, but it might be that this commit is not in qemu 2.1.2 (in which case he needs to specify it).

at the end (and also one of the reasons I wrote in my reply to him, that he should try qemu-git instead).

Last edited by Calrama (2014-10-17 16:27:17)

Offline

#2997 2014-10-17 16:40:28

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Check if IOMMU is supported by your CPU/Motherboard , and that it is probably supported/enabled in BIOS . Also make sure you added intel_iommu=on  OR  amd_iommu=on  to your boot parameters .

Thank you for your quick reply.
I have a Phenom2 x4 and a ASUS M4A89GTD PRO.
This mainboard has a AMD 890GX chipset.

If I believe this list: https://en.wikipedia.org/wiki/AMD_800_c … ries#890GX
My Chipset does not support IOMMU

Sad. Thanks for your help sad


Edit: Is there no other way expect buying a new mainboard?

Last edited by Rommy (2014-10-17 16:55:48)

Offline

#2998 2014-10-17 16:48:26

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Your launching code seems ok , but I don't think you need "x-vga=on" with OVMF . I'm running this VM without it . I don't know if it's related to your issue , but try to remove x-vga=on option and see if it works .

EDIT :

Also , try to configure tightVNC to accept connections without passwords , it's easier .

Tried both, but nothing changed.


Calrama wrote:

Could you please try out the latest git master instead of 2.1.2 (My qemu gives "2.1.50" on "-version")?

I would like to stay on "-stable", so next test will be with 2.1.3 or 2.2.0.

Calrama wrote:

Their git repo is just a mirror of the actual development subversion trunk, so it may lag behind.
If the git repo does lack behind, you might want to try the latest svn trunk.

I think it's not a real problem to be a couple of commit behind head.

Calrama wrote:

This may not be the solution, but it should reduce complexity:
First, you should not need these patches when going the UEFI way with OVMF, so please try to use a vanilla 3.16.6 kernel. I've used both unpatched 3.16.3 and 3.17.1 with OVMF successfully (unpatched meaning without these three patches, they did have Gentoo-specific patches).

Second, the boot paramater "i915.enable_hd_vgaarb=1" should also not be necessary when going the UEFI way with OVMF

If you say it's not a problem to have them enabled, I will leave them untouched (just in case I need to switch back to Windows 7).
Maybe I will try to remove "i915.enable_hd_vgaarb=1" option.

Calrama wrote:

Try to use SCSI instead of IDE

I tried installing with SCSI, but can't get drives recognized, so switched back to IDE.
Is it a big problem, or simply performance related?

Calrama wrote:

how did you get this rom file?

I got the file with the method you described:

# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom # cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/file.rom # echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

Important thing is to NOT set the card as primary graphic output.

Calrama wrote:

What does the serial console show?

It prompts a "(qemu)" terminal.

Calrama wrote:

Also, if none of the above helps you, I would like to know what the monitor your wrote you intent to connect to the graphics card shows from the moment you boot the machine.

I'll keep you informed.


Thanks for all the help.

Offline

#2999 2014-10-17 18:05:43

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
Calrama wrote:

Could you please try out the latest git master instead of 2.1.2 (My qemu gives "2.1.50" on "-version")?

I would like to stay on "-stable", so next test will be with 2.1.3 or 2.2.0.

Please try out qemu-git. The only people I know of who got OVMF to work used qemu-git. There should be no need to leave your debian stable, simply do the following as your normal user:

mkdir -p ~/qemu-git/root cd ~/qemu-git/ git clone git://git.qemu-project.org/qemu.git src cd src ./configure --prefix=~/qemu-git/root --enable-gtk --enable-sdl --enable-kvm --audio-drv-list=alsa,pa make make install

Now ~/qemu-git/root/bin/qemu-system-x86_64 -version should return "2.1.50" and you can simply use ~/qemu-git/root/bin/qemu-system-x86_64 instead of qemu-system-x86_64 for starting your VM.

Ansa89 wrote:
Calrama wrote:

Their git repo is just a mirror of the actual development subversion trunk, so it may lag behind.
If the git repo does lack behind, you might want to try the latest svn trunk.

I think it's not a real problem to be a couple of commit behind head.

If it's only a few commits over the last few days, sure. If it lags behind enough, that the latest git commit is older than last saturday, I cannot make any guess about it, because that's the earliest svn trunk I tried. You might want to check.

Ansa89 wrote:
Calrama wrote:

This may not be the solution, but it should reduce complexity:
First, you should not need these patches when going the UEFI way with OVMF, so please try to use a vanilla 3.16.6 kernel. I've used both unpatched 3.16.3 and 3.17.1 with OVMF successfully (unpatched meaning without these three patches, they did have Gentoo-specific patches).

Second, the boot paramater "i915.enable_hd_vgaarb=1" should also not be necessary when going the UEFI way with OVMF

If you say it's not a problem to have them enabled, I will leave them untouched (just in case I need to switch back to Windows 7).

You could always have two kernels, one patched, one unpatched. And just because it didn't change anything for me doesn't necessarily mean the same for you. There are enough differences in our setups that it might be worth a try.

Ansa89 wrote:
Calrama wrote:

Try to use SCSI instead of IDE

I tried installing with SCSI, but can't get drives recognized, so switched back to IDE.
Is it a big problem, or simply performance related?

Mostly perfomance, but I had some trouble with OVMF not recognising non-scsi devices, so I have everything on scsi now.

Ansa89 wrote:
Calrama wrote:

how did you get this rom file?

I got the file with the method you described:

# echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom # cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/file.rom # echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

Important thing is to NOT set the card as primary graphic output.

Okay, thanks.

Ansa89 wrote:
Calrama wrote:

What does the serial console show?

It prompts a "(qemu)" terminal.

Sorry if this sound condescending, but are you sure you're on the serial console and not the compat monitor? That's what the compat monitor should be showing, not the serial console.
You should have three views to choose from (if your qemu is compiled with GTK and SDL at least):
View -> compatmonitor0
View -> serial0                       <---- This is the one
View -> parallel0

Last edited by Calrama (2014-10-17 18:09:44)

Offline

#3000 2014-10-18 11:27:48

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:

Please try out qemu-git. The only people I know of who got OVMF to work used qemu-git. There should be no need to leave your debian stable, simply do the following as your normal user:

mkdir -p ~/qemu-git/root cd ~/qemu-git/ git clone git://git.qemu-project.org/qemu.git src cd src ./configure --prefix=~/qemu-git/root --enable-gtk --enable-sdl --enable-kvm --audio-drv-list=alsa,pa make make install

Now ~/qemu-git/root/bin/qemu-system-x86_64 -version should return "2.1.50" and you can simply use ~/qemu-git/root/bin/qemu-system-x86_64 instead of qemu-system-x86_64 for starting your VM.

Ok, I will give it a try.

Calrama wrote:

If it's only a few commits over the last few days, sure. If it lags behind enough, that the latest git commit is older than last saturday, I cannot make any guess about it, because that's the earliest svn trunk I tried. You might want to check.

Checked and git seems not too behind svn (latest commit is "ArmVirtualizationPkg: FdtPL011SerialPortLib: support UEFI_APPLICATION" which should be even with svn-r16219).

Calrama wrote:

You could always have two kernels, one patched, one unpatched. And just because it didn't change anything for me doesn't necessarily mean the same for you. There are enough differences in our setups that it might be worth a try.

Ok.

Calrama wrote:

Mostly perfomance, but I had some trouble with OVMF not recognising non-scsi devices, so I have everything on scsi now.

That's strange: I had the opposite problem.
If I want to use SCSI, will I also need "-device virtio-scsi-pci,id=scsi" option?

Calrama wrote:

Sorry if this sound condescending, but are you sure you're on the serial console and not the compat monitor? That's what the compat monitor should be showing, not the serial console.
You should have three views to choose from (if your qemu is compiled with GTK and SDL at least):
View -> compatmonitor0
View -> serial0                       <---- This is the one
View -> parallel0

My qemu is compiled without SDL nor GTK.
The explanation of this is that qemu reside on a headless pc (Xorg not installed at all) and all the work is done through ssh.

EDIT: just found this, so I can say I'm accessing the qemu monitor console (hope it can be useful).


Again thanks for all these info.

Offline

#3001 2014-10-18 12:02:06

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
Calrama wrote:

Mostly perfomance, but I had some trouble with OVMF not recognising non-scsi devices, so I have everything on scsi now.

That's strange: I had the opposite problem.
If I want to use SCSI, will I also need "-device virtio-scsi-pci,id=scsi" option?

As far as I know, yes. This option puts a virtual SCSI bus onto the PCI bus.
After you have created this SCSI bus, you can then add devices to it like this:

-drive file=${WINDOWS_INSTALL_DISK},id=iso_install,if=none \ -device scsi-cd,bus=scsi.0,drive=iso_install

or this:

-drive file=/path/to/device,if=none,id=disk_primary \ -device scsi-hd,bus=scsi.0,drive=disk_primary
Ansa89 wrote:
Calrama wrote:

Sorry if this sound condescending, but are you sure you're on the serial console and not the compat monitor? That's what the compat monitor should be showing, not the serial console.
You should have three views to choose from (if your qemu is compiled with GTK and SDL at least):
View -> compatmonitor0
View -> serial0                       <---- This is the one
View -> parallel0

My qemu is compiled without SDL nor GTK.
The explanation of this is that qemu reside on a headless pc (Xorg not installed at all) and all the work is done through ssh.

If you don't mind me asking this, what is the use-case for this? VNC (which IIRC is what you're using) isn't even fast enough to use it for high-quality video transmission inside one PC (I tried) from guest to host. Not to mention the latency and spikes.

Ansa89 wrote:

EDIT: just found this, so I can say I'm accessing the qemu monitor console (hope it can be useful).

Well, you need to see the output of the serial console. One way to do this would be to compile qemu-git like I said above (WITH sdl and gtk) and then start qemu over your ssh connection with X11 forwarding. That way, you should see the qemu window on your SSH client (with these three views) and be able to access the serial console. Read the "X11 forwarding" section of this wiki page,

Offline

#3002 2014-10-18 12:52:07

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:

After you have created this SCSI bus, you can then add devices to it like this:

-drive file=${WINDOWS_INSTALL_DISK},id=iso_install,if=none \ -device scsi-cd,bus=scsi.0,drive=iso_install

or this:

-drive file=/path/to/device,if=none,id=disk_primary \ -device scsi-hd,bus=scsi.0,drive=disk_primary

Is exactly what I did, but after initial installation screen, Windows said "cannot find installation media" (probably due to lack of necessary drivers); so I ended up using IDE.

Calrama wrote:

If you don't mind me asking this, what is the use-case for this? VNC (which IIRC is what you're using) isn't even fast enough to use it for high-quality video transmission inside one PC (I tried) from guest to host. Not to mention the latency and spikes.

You right about VNC performance, but I my idea was to use nVidia GameStream and/or Steam HomeStreaming (they had been created for this situation).

Calrama wrote:

Well, you need to see the output of the serial console. One way to do this would be to compile qemu-git like I said above (WITH sdl and gtk) and then start qemu over your ssh connection with X11 forwarding. That way, you should see the qemu window on your SSH client (with these three views) and be able to access the serial console. Read the "X11 forwarding" section of this wiki page,

Isn't there a way to access serial0 without SDL/GTK (for example "-serial stdio" option)?
I'm asking this because SDL has a ton of dependencies sad .

EDIT: quote from this (very outdated) thread:

When I try the Ctrl+Alt+2 thing in VNC, I get a screen with a display "serial0", with no "(qemu)" prompt as was implied in documentation. From what I gather from the aforementioned thread, the Ctrl+Alt+2 thing is disabled by default, won't get you to a QEMU Monitor.

I tried Ctrl+Atl+2 and the console0 switched to (or at lest become) a black screen without the "(qemu)" prompt (however can't see "serial0" written anywhere).

Last edited by Ansa89 (2014-10-18 14:41:19)

Offline

#3003 2014-10-18 15:52:56

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
Calrama wrote:

After you have created this SCSI bus, you can then add devices to it like this:

-drive file=${WINDOWS_INSTALL_DISK},id=iso_install,if=none \ -device scsi-cd,bus=scsi.0,drive=iso_install

or this:

-drive file=/path/to/device,if=none,id=disk_primary \ -device scsi-hd,bus=scsi.0,drive=disk_primary

Is exactly what I did, but after initial installation screen, Windows said "cannot find installation media" (probably due to lack of necessary drivers); so I ended up using IDE.

Yes, that is normal. That's not OVMF not finding your device, though (which would mean that the device does not show up on the UEFI shell), that's (as you suspected) Windows not having any Virtio drivers by default.
What guide did you follow? You need VirtIO drivers for Windows if you want to use VirtIO devices in a Windows guest, see here. If you want a direct download, you can grab the - currently - latest Fedora-compiled, signed VirtIO drivers for Windows here. That is an iso file, so you have to add it to your machine without VirtIO, i.e. as a normal IDE drive, so Windows can find it (Adding "-cdrom /path/to/virtio-win-0.1-81.iso" to your qemu command should do it). Once you booted from the Windows installation disc and got to the part where the installer "cannot find installation media" you should be able to select "Browse" or something similar (cannot remember exactly what it is called), at which point you select the Win8 directory on the drive that corresponds to the virtio driver iso (should be the only CDROM drive at this point) and select all the VirtIO drivers that he shows you for installation. Afterwards, you should be able to continue the Windows installation normally.

Ansa89 wrote:
Calrama wrote:

If you don't mind me asking this, what is the use-case for this? VNC (which IIRC is what you're using) isn't even fast enough to use it for high-quality video transmission inside one PC (I tried) from guest to host. Not to mention the latency and spikes.

You right about VNC performance, but I my idea was to use nVidia GameStream and/or Steam HomeStreaming (they had been created for this situation).

And in my experience they have not succeeded yet by a long shot. I've tried Steam InhomeStreaming from the guest VM directly to the Gentoo host (no real network card involved) with the Guest using a VirtIO network card (which is currently the best in performance AFAIK).
I had my left monitor show the actual output of the guest VM (connected to the passed-through graphics card) and the right monitor be the Host's Steam Inhome-Streaming client with the following results at 1080p@60Hz:
If the screen content stays nearly the same or is very slow moving, you're not going to see a difference on average. But I saw a significant difference with fast-moving content, There were many skipped frames, the FPS count plummeted from 60 down do 30. Also, regardless of whether it's fast or slow-moving content, I had spikes which had the Inhome-Streaming client lag as far behind as one second (the missing frames were then completly skipped to get to the current frames).
I have no experience with nVidia GameStream, but I doubt they have a solution for network spikes. If the above is something you can live with, then definitely go for it, but for me this was unbearable.

But regardless of the above two paragraphs, you might really want to get your passthrough to work on its own, before moving on to that stuff, because right now you're trying two things at the same time without having confirmed the first thing to be working properly.

Ansa89 wrote:
Calrama wrote:

Well, you need to see the output of the serial console. One way to do this would be to compile qemu-git like I said above (WITH sdl and gtk) and then start qemu over your ssh connection with X11 forwarding. That way, you should see the qemu window on your SSH client (with these three views) and be able to access the serial console. Read the "X11 forwarding" section of this wiki page,

Isn't there a way to access serial0 without SDL/GTK (for example "-serial stdio" option)?
I'm asking this because SDL has a ton of dependencies sad .

EDIT: quote from this (very outdated) thread:

When I try the Ctrl+Alt+2 thing in VNC, I get a screen with a display "serial0", with no "(qemu)" prompt as was implied in documentation. From what I gather from the aforementioned thread, the Ctrl+Alt+2 thing is disabled by default, won't get you to a QEMU Monitor.

I tried Ctrl+Atl+2 and the console0 switched to (or at lest become) a black screen without the "(qemu)" prompt (however can't see "serial0" written anywhere).

Sorry, I have no further information about that. Although imho you should really get the passthrough to work properly first and then move on to accessing the VM via network. What's the problem with having the SDL dependencies, though. If your machine is even remotely close to good enough to be used for virtualised gaming, you should be able to give the host a few more MB. It's not like they'll slow down your host? Anyway, what I can tell you is that when you start QEMU with OVMF, the serial console will show the EFI shell, which you can then use to boot the Windows installer or the installed windows should it in fact be stuck there. Once the VM has booted an EFI program (like Windows bootloader) it will become black. Anyway, I would suggest to try qemu-git first, before doing anything else.

Last edited by Calrama (2014-10-18 16:41:44)

Offline

#3004 2014-10-18 16:59:54

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use in-home streaming and with option FAST + Unlimited bandwith (peaks about 90+mbps) = 0 frame drops at 1200p 60fps, no matter how fast screen moves....
If u install geforce experience  and turn on / off shadowplay, that enables NVFBC for full desktop/better capture for streaming.
http://i.imgur.com/D2bWvrR.jpg u can see at bottom that display latency is below 20ms and that screen is pretty much worst case scenario, lots of action, cpu suffers smile

Last edited by slis (2014-10-18 17:10:28)

Offline

#3005 2014-10-18 20:21:17

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone had this issue where nVidia's Windows drivers crash every 20 minutes , and produce artifacts on pretty much everything that moves on the screen ?

I'm using 340.52 .

Removing ALL "hv_enhancements" EXCEPT "hv_time" seems to solve the issue + it finally allowed me to reboot the VM without crashing it .

And it seems that I didn't lose any performance too .

I will keep it running for 48 hours and see if the artifacts issues are indeed gone .

EDIT :

Nope , artifacts and horizontal tears still appear in videos . I want to update the drivers , but the new ones require the removal of "hv_time" . Will I lose THAT much of CPU performance by removing it ?

Last edited by Denso (2014-10-18 21:41:20)

Offline

#3006 2014-10-19 07:23:27

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That sounds more like hardware problem, too high oc/temp.

You will lose about 10-20% cpu performance by removing it.

Maybe u load wrong bios with romfile option?

Last edited by slis (2014-10-19 08:27:56)

Offline

#3007 2014-10-19 09:49:02

MichaelLong
Member
Registered: 2014-01-06
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Did anyone had this issue where nVidia's Windows drivers crash every 20 minutes , and produce artifacts on pretty much everything that moves on the screen ?

I'm using 340.52 .

Removing ALL "hv_enhancements" EXCEPT "hv_time" seems to solve the issue + it finally allowed me to reboot the VM without crashing it .

And it seems that I didn't lose any performance too .

I will keep it running for 48 hours and see if the artifacts issues are indeed gone .

EDIT :

Nope , artifacts and horizontal tears still appear in videos . I want to update the drivers , but the new ones require the removal of "hv_time" . Will I lose THAT much of CPU performance by removing it ?

Hi Denso,

over time I've tested my VM with three different graphic cards:

  • AMD 5850

  • NVIDIA GTX 560 TI

  • NVIDIA GTX 980.

With all of them I've got vertical colored polygon-shaped artifacs, they flicker with different intensity and often related to the direction in which I'm looking when playing a Battlefield 4 map. The problem occurs only after a few hours and can be "solved" be restarting the game. I would also rule out overheating problems because the mentioned cards are very different regarding their cooling solutions and I don't have those artifacts when playing on the same physical machine with the same hardware.

Also I found no relation to hv_*-parameters. There is no doubt a performance loss without hv_*-parameters. But my impression is that the host, more specifically the qemu-process produces more load.

Offline

#3008 2014-10-19 11:30:00

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

That sounds more like hardware problem, too high oc/temp.

You will lose about 10-20% cpu performance by removing it.

Maybe u load wrong bios with romfile option?

This is a normal slim GT610 , no exotic thing and no OCing , just a small stock fan for web browsing .

And I don't specify a ROM file at all , it just "works" for passthrough .

MichaelLong wrote:

Hi Denso,

over time I've tested my VM with three different graphic cards:

    AMD 5850

    NVIDIA GTX 560 TI

    NVIDIA GTX 980.

With all of them I've got vertical colored polygon-shaped artifacs, they flicker with different intensity and often related to the direction in which I'm looking when playing a Battlefield 4 map. The problem occurs only after a few hours and can be "solved" be restarting the game. I would also rule out overheating problems because the mentioned cards are very different regarding their cooling solutions and I don't have those artifacts when playing on the same physical machine with the same hardware.

Also I found no relation to hv_*-parameters. There is no doubt a performance loss without hv_*-parameters. But my impression is that the host, more specifically the qemu-process produces more load.

Exactly what I found today . There is no relation between my issues and hv_ parameters .

I have 2 exact VMs , one with a GTX770 and it works with no issues at all , I can reboot it , shutdown and launch it again , no issues at all .

The second one has a GT610 for my web browsing , it also works good except for this stupid reboot issue , it reboots ok until Windows logo finishes then it hangs forever . The host and other VMs are not affected though .

Offline

#3009 2014-10-19 11:43:25

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Last edited by slis (2014-10-19 11:43:50)

Offline

#3010 2014-10-19 13:31:44

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Well , a GPU-Z dump from the VM result in the exact same issue . Also , trying the way mentioned in Alex's blog results in input/output error (Invalid ROM contents) .

Any other way to obtain non-initialized ROM file ?

TechPowerUp's ROMs do not work at all and result in passthrough failure .

By the way , I tried dumping the ROM from the card while it wasn't touched by anything else other than PCI-STUB . VFIO was disabled , and VM wasn't started yet . So I don't know what was really wrong .

Last edited by Denso (2014-10-19 13:51:11)

Offline

#3011 2014-10-19 15:03:47

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

I use in-home streaming and with option FAST + Unlimited bandwith (peaks about 90+mbps) = 0 frame drops at 1200p 60fps, no matter how fast screen moves....
If u install geforce experience  and turn on / off shadowplay, that enables NVFBC for full desktop/better capture for streaming.
http://i.imgur.com/D2bWvrR.jpg u can see at bottom that display latency is below 20ms and that screen is pretty much worst case scenario, lots of action, cpu suffers smile

Wow, thank you for that bit of information:
After enabling NVFBC as the encoder on the host  AND VAAPI as the decoder on the guest (the latter of which took some fiddling on Gentoo), I only notice a slight lag between input and output on the client side (not noticeable on the monitor connected to the guest graphics card), which is still too much for FPS games. For everything else, though, this should be pretty good.
So, I have to retract my previous statement of them not getting it to work by a long shot, you just need to use a hardware encoder and a hardware decoder.

Last edited by Calrama (2014-10-19 16:06:39)

Offline

#3012 2014-10-19 15:10:52

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
slis wrote:

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Also , trying the way mentioned in Alex's blog results in input/output error (Invalid ROM contents) .

By the way , I tried dumping the ROM from the card while it wasn't touched by anything else other than PCI-STUB . VFIO was disabled , and VM wasn't started yet . So I don't know what was really wrong .

Just for reference: I cannot access my GTK 660 TIs firmware that way either. Even in pristine pci-stub stub right after booting. I'm not sure, but I think that's a thing with certain NVIDIA cards and hybrid firmware. My card came with only a legacy BIOS as its firmware, so for OVMF I had to flash it with the official ASUS tool. Before flashing - that is, when it had the legacy bios as firmware - I could dump the firmware like described in the blog you referenced, but after flashing the new firmware I get get the input/output error.

Offline

#3013 2014-10-19 21:48:26

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:
slis wrote:

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Also , trying the way mentioned in Alex's blog results in input/output error (Invalid ROM contents) .

By the way , I tried dumping the ROM from the card while it wasn't touched by anything else other than PCI-STUB . VFIO was disabled , and VM wasn't started yet . So I don't know what was really wrong .

Just for reference: I cannot access my GTK 660 TIs firmware that way either. Even in pristine pci-stub stub right after booting. I'm not sure, but I think that's a thing with certain NVIDIA cards and hybrid firmware. My card came with only a legacy BIOS as its firmware, so for OVMF I had to flash it with the official ASUS tool. Before flashing - that is, when it had the legacy bios as firmware - I could dump the firmware like described in the blog you referenced, but after flashing the new firmware I get get the input/output error.

Thank you . It wasn't surprising , as nvidia will do their best to fight this usage scenario of their cards .

This exact card was working fine on my previous system (AMD FX-8350 + 990FXA) , but with this X99 + i7-5930k it's not .

How about using a thin client + SPICE protocol instead of GPU passthrough , will it give a near-native performance ?

Offline

#3014 2014-10-19 23:45:40

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Thank you . It wasn't surprising , as nvidia will do their best to fight this usage scenario of their cards .

Suprising, no. Assholly, yes. But it works for me without artifacts so I'm happy for now. Soon enough most new AAA games will have Linux support anyway with the Unreal 4 engine and several others having official support for it, so to me all of this is mostly for legacy-game support and bridging the intermission time.

Denso wrote:

This exact card was working fine on my previous system (AMD FX-8350 + 990FXA) , but with this X99 + i7-5930k it's not .

I'm using this processor on this mainboard. Warning, [OT]Never saw any real use-case for the *k processors or the i7 series for desktop systems (or overclocking in general for that matter)[/OT]. One thing, though: I know that Intel made the recent *K models Vt-d capable (previous generations were not, only the non-*K variants IIRC), but maybe there's an issue there? Seems somewhat unlikely, but you never know with all these black boxes.

Denso wrote:

How about using a thin client + SPICE protocol instead of GPU passthrough , will it give a near-nativthin cliente performance ?

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Offline

#3015 2014-10-19 23:49:55

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

Offline

#3016 2014-10-19 23:53:24

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Calrama wrote:

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

Offline

#3017 2014-10-20 00:37:35

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:
Calrama wrote:

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Offline

#3018 2014-10-20 12:40:29

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Calrama wrote:
Denso wrote:

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Forgive me for being so blunt, but from my point of view you're still not answering my question about why you think about bringing in SPICE as a replacement for a graphics card passthrough. Please consider the following:

Setup A:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box and gets a graphics card passed through from the Linux host

Setup B:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box with an emulated graphics card (for QEMU that would be any of std,cirrus,qxl,vmware,...)

Setup C:

- Two separate hardware boxes, connected by some network (e.g. 1Gbit Ethernet, Infiniband, whatever)
- Box 1 has a native Linux running
- Box 2 has a native Windows running

These are essentially your choices (very abstract). The statement "SPICE with a low-end thin client" does not specify which, though, since the native Linux in all three setups could be such a thin client.
SPICE, VNC, or any such protocol is about how you control - in this context - the Windows machine (regardless of whether you have one or two boxes); the Windows machine (regardless of whether it is being virtualised or not) will still need access to a real, dedicated graphics card (setups A and C). I know of no way aroung that. In the far future when XenGT has become something that can actually be done with NVIDIA and AMD graphics cards, we will have a way around that by paravirtualising our graphics card analogous to how Vt-x/AMD-V paravirtualise your processor. But that's still quite a bit away.

SPICE is by definition not a replacement for a graphics card passthrough, it's about remote control and can be done regardless of whether you're doing such a passthrough, which is why I was - and still am - so confused by your posts about this.

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Last edited by Calrama (2014-10-20 12:44:41)

Offline

#3019 2014-10-20 17:08:59

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, & thanks nbhs for this tutorial.

I I tried to set up a working VGA-Passthrough on my laptop for a few days now, I wish I could do it by myself with this excellent tutorial but it seems I'm doing some things wrong. Now I'm stuck, and I call for your help, it will be very appreciated.
I know this is experimental and there's a 50/50 chance that it work for my configuration but since I'm not 100% sure I didn't forgot or misunderstood something, I prefer to ask before giving up.

I plugged in a screen with HDMI. (got VGA cable too)

Here is my setup:

model: Acer Aspire 7745G proc: intel i5-540M (not the original one, I changed to get VT-d support, intel's KB tells it is supported: http://ark.intel.com/fr/products/43544/Intel-Core-i5-540M-Processor-3M-Cache-2_53-GHz) chipset: hm55 (still refering to intel's kb, should support VT-d too: http://ark.intel.com/products/43183/Intel-BD82HM55-PCH) host gpu (IGP): Intel® HD Graphics guest gpu: ATI Mobility Radeon HD5650

In order to enable VT-d in BIOS, I had to flash it with a modded BIOS enabling Intel's hidden menu. VT-d was "disabled" ans I put it "enabled"

Question 0: First, I got a strange message in dmesg about DMAR:

# dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x00000000937EF000 0000B8 (v01 INTEL CP_DALE 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.033446] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c9008020e30272 ecap 1000 [ 0.033451] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c0000020230272 ecap 1000 [ 0.033458] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap c9008020630272 ecap 1000 -->[ 0.348199] DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics [ 0.452912] DMAR: No ATSR found [ 0.452968] IOMMU 0 0xfed90000: using Register based invalidation [ 0.452969] IOMMU 2 0xfed93000: using Register based invalidation [ 0.452971] IOMMU: Setting RMRR: [ 0.452992] IOMMU: Setting identity map for device 0000:00:1a.0 [0x936e9000 - 0x936fffff] [ 0.453055] IOMMU: Setting identity map for device 0000:00:1d.0 [0x936e9000 - 0x936fffff] [ 0.453087] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.453108] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

I can't find any understandable informations about this "shadow GTT", every google search leads me to techy stuff. Seems to be something important, I dont't want to disable IOMMU for graphics, right ? Is it important ? Or is it just a message to tell "your IGP can't be passed" ? That would be ok since I don't want to pass the IGP but the discret gpu, right ?
I couldn't find anything about this GTT in the BIOS, I suppose it should be somewhere because I unlocked the hidden intel menus, maybe with an alternative name ?

Since I couldn't find anything about this issue, I tried to go further.

What I have done:
- install a fresh arch linux, the downloaded qemu from the main repository
- compile linux-mainstream with patches

# uname -r 3.17.0.1-mainline

- set kernel parameters in bootloader

# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d28d848b-32ae-4fe9-b74e-3978972c840e rw intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68c1,1002:aa60 quiet

- blacklist radeon
- configure pci-stub

# dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d28d848b-32ae-4fe9-b74e-3978972c840e rw intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68c1,1002:aa60 quiet [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d28d848b-32ae-4fe9-b74e-3978972c840e rw intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68c1,1002:aa60 quiet [ 0.694456] pci-stub: add 1002:68C1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.694473] pci-stub 0000:01:00.0: claimed by stub [ 0.694481] pci-stub: add 1002:AA60 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.694490] pci-stub 0000:01:00.1: claimed by stub

- setting up vfio & KVM modules

# cat /etc/modprobe.d/vfio_iommu_type1.conf options vfio_iommu_type1 allow_unsafe_interrupts=1

- enable service to automatically bind the devices

- test it with the following command

/usr/bin/qemu-system-x86_64 \ -enable-kvm \ -m 2048 \ -M q35 \ -bios /usr/share/qemu/bios.bin \ -cpu host \ -vga none \ -device 'ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1' \ -device 'vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/mouton/GPUPT/vgabios.bin' \ -device 'vfio-pci,host=01:00.1,bus=root.1,addr=00.1' \ -cdrom /home/mouton/GPUPT/win7.iso 

That's where I'm stuck: qemu launches but laptop's screen show "compat_monitor0 console from qemu". I can see there is cpu-usage so I suppose emulation works fine but I just can't see the results screen.

Question 1: Removing "-vga none", it is functionning correctly, but from what I understood, the GPU is not passed then. Is it wrong ?

I tried with and without "romfile=..." result is the same. I dumped the rom from my graphic card with a liveCD, rom-parser returns the following

Valid ROM signature found @0h, PCIR offset 40h PCIR: type 0, vendor: 8086, device: 0046, class: 030000 PCIR: revision 0, vendor revision: 0 Last image

Question 2: This is the first things which surprise me: vendor/device is 8086:0046 which correspond to my IGP. So, this is the vgabios from my IGP. Is it the correct vbios for "romfile" option or should I find a way to get the passthroughed GPU vbios ? (Which sounds logical to me, but I don't get the whole picture yet, so many confusion right now...)

Question 3: I'm actually using qemu version 2.1.2 from official arch repository... I couldn't find a link in the thread for nbhs' qemu. In the tutorial, it is advised not to use the qemu-git one so I didn't. Is it important ? Where can I find nbhs' qemu and Seabios ? (couldn't find it too, all links are broken)

Question 4: Since it's my first time with vga-passthrough, I'm not sure about the cable configuration. Actually I plugged a secondary screen on the HDMI connector of my laptop. Screen is not duplicated. However, when I plug a cable in the VGA connector, screen is duplicated. Should I use VGA instead or stay on HDMI ? (I assumed, since we pass the hdmi audio device, I should use HDMI)

What makes me persevere is I achieved to send some junk datas to the secondary screen with the "echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind " command. It worked once but I am not able to reproduce it.

Question 5: Last, I got so much confused with all the threads I read on the topic that I can't understand if I should or should not install the passed-through gpu drivers ?!


If anyone could guide me in order to make my configuration work, I would greatly apreciate it.

Offline

#3020 2014-10-20 17:13:19

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK so after letting it rest for a while, and then starting to fiddle with it again, I have found what part in the command line ruined the booting (just entering "QEMU 2.1.2 monitor")...
Whenever I remove the -vga none option it boots up fine. But whenever I add it back it gives me the console again.

What is going wrong here?

Offline

#3021 2014-10-20 18:57:54

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mouton_sanglant wrote:
host gpu (IGP): Intel® HD Graphics guest gpu: ATI Mobility Radeon HD5650

Are you sure that's even feasible? In a normal PC, every graphics card has its own output ports (the IGP's ports are on the mainboard and every other graphics card has its on its own board).
In laptops - at least to the best of my knownledge - the dedicated graphics cards usually don't have any external ports, but instead operate as "calculation slaves" below the IGP, meaning that any output the dedicated graphics card produces is forwarded to the IGP through shared memory, which in turn forwards it to its own output ports (which is either the laptop's screen or the IGP's external ports). Even if you did pass through the dedicated card correctly, how would the guest OS know about using it correctly (since this card doesn't seem to be designed to be used on its own)? This is pure speculation on my part, of course, but I somewhat doubt the AMD drivers inside the guest system would be able to function properly (though I would be happy to be proven wrong).

mouton_sanglant wrote:

In order to enable VT-d in BIOS, I had to flash it with a modded BIOS enabling Intel's hidden menu. VT-d was "disabled" ans I put it "enabled"

Question 0: First, I got a strange message in dmesg about DMAR:

[...] -->[ 0.348199] DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics [...]

I can't find any understandable informations about this "shadow GTT", every google search leads me to techy stuff.

Personally I have not heard of "shadow GTT" before (speculation: it might be related to the shared memory I mentioned above; it could be that in order for the host to be able to pass through the dedicated graphics card, the bios needs to have allocated some extra memory, because the usual shared memory is in use by the host OS video driver), but a quick Google search turned up this commit, which seems to imply that if this happens, your bios contains a bug. You might want to try asking the committer (who seems to be an Intel developer, so he should really know about this) for an explanation.

mouton_sanglant wrote:

That's where I'm stuck: qemu launches but laptop's screen show "compat_monitor0 console from qemu". I can see there is cpu-usage so I suppose emulation works fine but I just can't see the results screen.

Where would you expect the passed through graphics card's output to be shown? The external ports belong to the IGP (unless of course that is different for your specific laptop?).

mouton_sanglant wrote:

Question 1: Removing "-vga none", it is functionning correctly, but from what I understood, the GPU is not passed then. Is it wrong ?

qemu-system-x86_64 --help | grep -A1 vga

returns

-vga [std|cirrus|vmware|qxl|xenfb|tcx|cg3|none] select video card type

That is the graphics card you want QEMU to emulate. In other words, if you set "std" here, the VM will get an emulated graphics card of the type "std". If you don't set this argument at all, the default (which is one of these, don't remember which) will be selected. If you select "none", the VM will not have any emulated graphics card, which is what you usually want when you passthrough a real graphics card because you don't have any use for the emulated one. But the emulated graphics card should not interfere with the passed through graphics card in any way other then it being selected as the primary card. It should not have any effect on whether your card gets passed through correctly.


mouton_sanglant wrote:

I tried with and without "romfile=..." result is the same. I dumped the rom from my graphic card with a liveCD, rom-parser returns the following

Valid ROM signature found @0h, PCIR offset 40h PCIR: type 0, vendor: 8086, device: 0046, class: 030000 PCIR: revision 0, vendor revision: 0 Last image

Question 2: This is the first things which surprise me: vendor/device is 8086:0046 which correspond to my IGP. So, this is the vgabios from my IGP. Is it the correct vbios for "romfile" option or should I find a way to get the passthroughed GPU vbios ? (Which sounds logical to me, but I don't get the whole picture yet, so many confusion right now...)

If it is the bios of the IGP, then no, it's not the correct one to be set for your dedicated card. If you want to set a romfile option for the dedicated card you need a firmware compatible with the dedicated card.

mouton_sanglant wrote:

Question 4: Since it's my first time with vga-passthrough, I'm not sure about the cable configuration. Actually I plugged a secondary screen on the HDMI connector of my laptop. Screen is not duplicated. However, when I plug a cable in the VGA connector, screen is duplicated. Should I use VGA instead or stay on HDMI ? (I assumed, since we pass the hdmi audio device, I should use HDMI)

See my reasoning above. The external ports should all belong to the IGP so I don't see how you could see any output even if you passed through your card correctly. You could try installing a VNC on the guest OS while using an emulated graphics card and see if you can connect when not using one (and passing through your dedicated card).

mouton_sanglant wrote:

What makes me persevere is I achieved to send some junk datas to the secondary screen with the "echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind " command. It worked once but I am not able to reproduce it.

I cannot explain this one without reasonable doubt and it seems to contradict what I speculated about above, so take the above with a grain of salt. You should really try to get in touch with that Intel developer.

mouton_sanglant wrote:

Question 5: Last, I got so much confused with all the threads I read on the topic that I can't understand if I should or should not install the passed-through gpu drivers ?!

In general you want to install the host's graphic card's driver in the host and the guest's graphics card's driver in the guest.

Offline

#3022 2014-10-20 21:02:57

gyrfalco
Member
Registered: 2014-09-30
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Well, FWIW I now own an XFX HD7790 card.  All I can confirm so far is that it's quite broken.  VM reboots work well with the standard Windows VGA driver, but once the Catalyst drivers are loaded a reboot results

SAPPHIRE Radeon HD 7790 1GB (non UEFI)
QEMU emulator version 2.1.50
seabios-1.7.5
q35 / -vga none

I succeeded to get a result: http://gpuz.techpowerup.com/14/10/20/2tg.png

But only once, after uninstalling Catalyst in -vga qxl mode and starting machine in -vga none. Then i decided to install fresh system and repeat that procedure, but never been able to achieve the same result.  sad

Offline

#3023 2014-10-20 22:02:37

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:
Calrama wrote:

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Forgive me for being so blunt, but from my point of view you're still not answering my question about why you think about bringing in SPICE as a replacement for a graphics card passthrough. Please consider the following:

Setup A:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box and gets a graphics card passed through from the Linux host

Setup B:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box with an emulated graphics card (for QEMU that would be any of std,cirrus,qxl,vmware,...)

Setup C:

- Two separate hardware boxes, connected by some network (e.g. 1Gbit Ethernet, Infiniband, whatever)
- Box 1 has a native Linux running
- Box 2 has a native Windows running

These are essentially your choices (very abstract). The statement "SPICE with a low-end thin client" does not specify which, though, since the native Linux in all three setups could be such a thin client.
SPICE, VNC, or any such protocol is about how you control - in this context - the Windows machine (regardless of whether you have one or two boxes); the Windows machine (regardless of whether it is being virtualised or not) will still need access to a real, dedicated graphics card (setups A and C). I know of no way aroung that. In the far future when XenGT has become something that can actually be done with NVIDIA and AMD graphics cards, we will have a way around that by paravirtualising our graphics card analogous to how Vt-x/AMD-V paravirtualise your processor. But that's still quite a bit away.

SPICE is by definition not a replacement for a graphics card passthrough, it's about remote control and can be done regardless of whether you're doing such a passthrough, which is why I was - and still am - so confused by your posts about this.

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Sorry for my lack of words (English isn't my native language smile) , and yes , I meant setup B . Where I get to interact with the VM directly from the host or through an external client . I was wondering whether SPICE can offer a big performance improvement over regular VNC or RDP , and from what I saw on YT it's still somewhat laggy . Maybe this was off topic when it comes to GPU passthrough , but it can be a solution for web surfing , printing documents , and watching videos . (Especially when you lack PCI-E lanes to add more cards as I stated previously) .

Thank you wink

Offline

#3024 2014-10-20 22:45:20

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Calrama wrote:

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Sorry for my lack of words (English isn't my native language smile) , and yes , I meant setup B . Where I get to interact with the VM directly from the host or through an external client . I was wondering whether SPICE can offer a big performance improvement over regular VNC or RDP , and from what I saw on YT it's still somewhat laggy . Maybe this was off topic when it comes to GPU passthrough , but it can be a solution for web surfing , printing documents , and watching videos . (Especially when you lack PCI-E lanes to add more cards as I stated previously) .

Thank you wink

No problem, it isn't my first language either. Alright, setup B it is. Do you mind my asking what the reason is? Web surfing, printing, watching videos, and pretty much anything else that does not expicitly depend on Windows-exclusive programs can be done on the Linux host, as well. And Intel's IGP are a lot more powerful than some people think, so even 1080p videos should be no problem at all. However, if you really want to do all of that on Windows, why even install Linux as the host? Just have Windows as the host and Linux in a VM (if you need the Linux at all, that is).

In either case, I don't think you should see stuttering from host-side if you use the "std" emulated graphics card (unless you intend to watch something really heavy, like a 1080p video). You don't need any extra VNC, or RDP there either, just use the normal monitor window that opens when you start the VM (I think it uses VNC internally, but I'm not sure). If you want to connect from a different, separate box, however, you might want to look into Steam Inhome-streaming. I've read on several pages that you can highjack it to get back at the desktop (i.e. have the Windows-desktop streamed to the client) and then you can just use any application you want through that. See this discussion for some ways to do that.

Offline

#3025 2014-10-20 22:48:18

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Calrama ! Thank you a lot for you suggestions !

I finally got it working !
The host display is on the laptop screen and the guest screen on the secondary screen. It works fine but I had some other duties to do so I couldn't finish my configuration right now but I will soon.

So the key issue was this vga bios misunderstanding. It became obvious as long as I was writting my request. So I completly disabled the IGP from the bios, then launched a live-cd, there I could dump the discrete gpu vbios. I specified it to qemu and it worked !

About the outputs ports, it's actually working on the hdmi. I will try on the VGA on next try.

The DMAR message is still there, but it doesn't seem to be an issue. Actually, there is something relative to memory in the BIOS: in the "Video (Intel IGD) Control Sub-Menu", there is a parameter "Total Graphics Memory" which can be set to 128MB, 256MB or MaxDVMT. Informations says: "Select the amount of Total Graphics Memory - Pre-Allocated + Fixed + DVMT for use by the Intel Graphics Device" As I'm actually working with OpenGL, I thought it was the framebuffer memory and nothing related to the DMA Remapping.
Well, I'm still wondering what that really means...

I'll try to get in touch with this Intel guy, he would certainly know about it.

Calrama wrote:
mouton_sanglant wrote:

I tried with and without "romfile=..." result is the same. I dumped the rom from my graphic card with a liveCD, rom-parser returns the following

Valid ROM signature found @0h, PCIR offset 40h PCIR: type 0, vendor: 8086, device: 0046, class: 030000 PCIR: revision 0, vendor revision: 0 Last image

Question 2: This is the first things which surprise me: vendor/device is 8086:0046 which correspond to my IGP. So, this is the vgabios from my IGP. Is it the correct vbios for "romfile" option or should I find a way to get the passthroughed GPU vbios ? (Which sounds logical to me, but I don't get the whole picture yet, so many confusion right now...)

If it is the bios of the IGP, then no, it's not the correct one to be set for your dedicated card. If you want to set a romfile option for the dedicated card you need a firmware compatible with the dedicated card.

Yep, that's right, it bothered me a lot !

Anyway, thank you a lot for your time. I'll post again tomorrow to tell you if it's stable but the big part is done, know I'm very satisfied and proud of my work. Gonna play with my new baby, hehe !

Have a nice day since then !

Offline

#3026 2014-10-20 23:22:02

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
Denso wrote:
Calrama wrote:

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Sorry for my lack of words (English isn't my native language smile) , and yes , I meant setup B . Where I get to interact with the VM directly from the host or through an external client . I was wondering whether SPICE can offer a big performance improvement over regular VNC or RDP , and from what I saw on YT it's still somewhat laggy . Maybe this was off topic when it comes to GPU passthrough , but it can be a solution for web surfing , printing documents , and watching videos . (Especially when you lack PCI-E lanes to add more cards as I stated previously) .

Thank you wink

No problem, it isn't my first language either. Alright, setup B it is. Do you mind my asking what the reason is? Web surfing, printing, watching videos, and pretty much anything else that does not expicitly depend on Windows-exclusive programs can be done on the Linux host, as well. And Intel's IGP are a lot more powerful than some people think, so even 1080p videos should be no problem at all. However, if you really want to do all of that on Windows, why even install Linux as the host? Just have Windows as the host and Linux in a VM (if you need the Linux at all, that is).

In either case, I don't think you should see stuttering from host-side if you use the "std" emulated graphics card (unless you intend to watch something really heavy, like a 1080p video). You don't need any extra VNC, or RDP there either, just use the normal monitor window that opens when you start the VM (I think it uses VNC internally, but I'm not sure). If you want to connect from a different, separate box, however, you might want to look into Steam Inhome-streaming. I've read on several pages that you can highjack it to get back at the desktop (i.e. have the Windows-desktop streamed to the client) and then you can just use any application you want through that. See this discussion for some ways to do that.

Well , I can't get rid of Windows altogether for work reasons . I used to run Hyper-V and a Linux VM on top of that , but it wasn't good enough to run a ZFS file server . So I switched to a Linux host + ZFS server and a Windows VM . That made me sleep at night knowing my TBs of data are safer that way smile

Plus , I want my linux box to be as headless as it can get smile

I'm pretty happy with my current GPU passthrough setup , except for the reboot issue .

I might look into Steam's streaming if my GT210 didn't work . I'm just a little lazy to open that NORCO case and switch GPUs wink

Thanks !

Offline

#3027 2014-10-20 23:25:29

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mouton_sanglant wrote:

I finally got it working !
The host display is on the laptop screen and the guest screen on the secondary screen. It works fine but I had some other duties to do so I couldn't finish my configuration right now but I will soon.

About the outputs ports, it's actually working on the hdmi. I will try on the VGA on next try.

Well, I'll be. Sorry for posting some wrong information then, I was really convinced that it wouldn't work like that. Now I'm kind of curious whether the HDMI port actually belongs to the dedicated card, or if the passed through dedicated card still communicates correctly with the IGP and the IGP then forwards the guest's video output to whatever happens to be connected as a secondary screen (the latter of which would imply some serious coding on the side of the Intel devs who wrote the driver code).

mouton_sanglant wrote:

The DMAR message is still there, but it doesn't seem to be an issue. Actually, there is something relative to memory in the BIOS: in the "Video (Intel IGD) Control Sub-Menu", there is a parameter "Total Graphics Memory" which can be set to 128MB, 256MB or MaxDVMT. Informations says: "Select the amount of Total Graphics Memory - Pre-Allocated + Fixed + DVMT for use by the Intel Graphics Device" As I'm actually working with OpenGL, I thought it was the framebuffer memory and nothing related to the DMA Remapping.
Well, I'm still wondering what that really means...

I'll try to get in touch with this Intel guy, he would certainly know about it.

Intel IGPs don't have any dedicated (V)RAM on their own (unlike the GDDR RAM of modern dedicated graphics cards), since they are integrated into the CPU-chip. So when your system boots up a small portion of your actual, normal RAM gets reserved as VRAM for the integrated graphics card. This option should set the size of that reserved portion.

mouton_sanglant wrote:

Yep, that's right, it bothered me a lot !

Anyway, thank you a lot for your time. I'll post again tomorrow to tell you if it's stable but the big part is done, know I'm very satisfied and proud of my work. Gonna play with my new baby, hehe !

Have a nice day since then !

Cool, do have fun with it, I know I have with my OVMF-Machine.

Offline

#3028 2014-10-20 23:37:31

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Plus , I want my linux box to be as headless as it can get smile

Hm, I would just use two separate boxes, one headless server box for all the data and one desktop box, but if you're happy with that, "never change a running system", I guess *toothless grin*.

Denso wrote:

I'm pretty happy with my current GPU passthrough setup , except for the reboot issue .

Have you tried OVMF? That got rid of my problems.

Offline

#3029 2014-10-20 23:48:37

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:

Have you tried OVMF? That got rid of my problems.

Yup , all of my VMs are now running on OVMF smile

I expect it to be a hardware issue , propably the X99 platform is to be blamed . But my GTX770 VM is working flawlessly , so it might also be a GT610-specific issue .

I'll nail it when I try my other GT210 and see if I still have reboot issues .

Offline

#3030 2014-10-21 03:20:05

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Calrama wrote:
Denso wrote:

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Forgive me for being so blunt, but from my point of view you're still not answering my question about why you think about bringing in SPICE as a replacement for a graphics card passthrough. Please consider the following:

Setup A:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box and gets a graphics card passed through from the Linux host

Setup B:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box with an emulated graphics card (for QEMU that would be any of std,cirrus,qxl,vmware,...)

Setup C:

- Two separate hardware boxes, connected by some network (e.g. 1Gbit Ethernet, Infiniband, whatever)
- Box 1 has a native Linux running
- Box 2 has a native Windows running

These are essentially your choices (very abstract). The statement "SPICE with a low-end thin client" does not specify which, though, since the native Linux in all three setups could be such a thin client.
SPICE, VNC, or any such protocol is about how you control - in this context - the Windows machine (regardless of whether you have one or two boxes); the Windows machine (regardless of whether it is being virtualised or not) will still need access to a real, dedicated graphics card (setups A and C). I know of no way aroung that. In the far future when XenGT has become something that can actually be done with NVIDIA and AMD graphics cards, we will have a way around that by paravirtualising our graphics card analogous to how Vt-x/AMD-V paravirtualise your processor. But that's still quite a bit away.

SPICE is by definition not a replacement for a graphics card passthrough, it's about remote control and can be done regardless of whether you're doing such a passthrough, which is why I was - and still am - so confused by your posts about this.

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Sorry for my lack of words (English isn't my native language smile) , and yes , I meant setup B . Where I get to interact with the VM directly from the host or through an external client . I was wondering whether SPICE can offer a big performance improvement over regular VNC or RDP , and from what I saw on YT it's still somewhat laggy . Maybe this was off topic when it comes to GPU passthrough , but it can be a solution for web surfing , printing documents , and watching videos . (Especially when you lack PCI-E lanes to add more cards as I stated previously) .

Thank you wink

Right now only VMware, Microsoft, Xenserver and VirtualBox provide what you want. In kvm world... Forget it... Please wait another 3 years...

Offline

#3031 2014-10-21 06:16:47

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

doubledr wrote:

[......] Please wait another 3 years...

I'm in no hurry smile My GPU passthrough setup is working good enough (for now) !

Offline

#3032 2014-10-21 16:15:56

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:
mouton_sanglant wrote:

I finally got it working !
The host display is on the laptop screen and the guest screen on the secondary screen. It works fine but I had some other duties to do so I couldn't finish my configuration right now but I will soon.

About the outputs ports, it's actually working on the hdmi. I will try on the VGA on next try.

Well, I'll be. Sorry for posting some wrong information then, I was really convinced that it wouldn't work like that. Now I'm kind of curious whether the HDMI port actually belongs to the dedicated card, or if the passed through dedicated card still communicates correctly with the IGP and the IGP then forwards the guest's video output to whatever happens to be connected as a secondary screen (the latter of which would imply some serious coding on the side of the Intel devs who wrote the driver code).

Ok, I finished to configure my virtualized system. It's working fine with drivers installed inside host, played some games, no crashes ! Performances are "passed-troughed" just as expected !

Windows detects 2 displays, so it automatically enable extended desktop. One of these 2 displays can be disabled, however, the other one can't or I won't have any signal until I hit esc (to cancel action).
I ran some additionnal testings: can't pass through via the VGA cable, it duplicates the linux screen. So it seems to confirm what you pointed about the HDMI belongs to the dedicated card. I'll try someday to install a non-virtualized Windows on another HDD, disable the discrete card and plug a screen in the hdmi port.


Now, I have to fix an issue on the host: I can't launch any Desktop Manager. I think it is because of the i915 vga arbiter patch which disable DRI. Is that right ?
Window Managers works, so I should use Mozilla and stuff like that as long as I disable any 3d acceleration stuff, right ?

nbhs wrote:

Preparing the GPU so we can bind it to vfio

If you are using an intel IGP on your host, you need the i915 vga arbiter patch (this patch is included in the kernel i provided), and you need to add this kernel parameter to your bootloader:

i915.enable_hd_vgaarb=1

NOTE: This will disable DRI for you IGP!


[Edit]: I just disabled the kernel parameter "i915.enable_hd_vgaarb=1" in the bootloader and everything works fine. Is this patch fix performances issues or things like that ?

Last edited by mouton_sanglant (2014-10-21 16:21:44)

Offline

#3033 2014-10-21 19:26:32

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Calrama wrote:

You need VirtIO drivers for Windows if you want to use VirtIO devices in a Windows guest, see here. If you want a direct download, you can grab the - currently - latest Fedora-compiled, signed VirtIO drivers for Windows here. That is an iso file, so you have to add it to your machine without VirtIO, i.e. as a normal IDE drive, so Windows can find it (Adding "-cdrom /path/to/virtio-win-0.1-81.iso" to your qemu command should do it). Once you booted from the Windows installation disc and got to the part where the installer "cannot find installation media" you should be able to select "Browse" or something similar (cannot remember exactly what it is called), at which point you select the Win8 directory on the drive that corresponds to the virtio driver iso (should be the only CDROM drive at this point) and select all the VirtIO drivers that he shows you for installation. Afterwards, you should be able to continue the Windows installation normally.

Ok, I'm now on VirtIO.

Calrama wrote:

I've tried Steam InhomeStreaming from the guest VM directly to the Gentoo host (no real network card involved) with the Guest using a VirtIO network card (which is currently the best in performance AFAIK).
I had my left monitor show the actual output of the guest VM (connected to the passed-through graphics card) and the right monitor be the Host's Steam Inhome-Streaming client with the following results at 1080p@60Hz:
If the screen content stays nearly the same or is very slow moving, you're not going to see a difference on average. But I saw a significant difference with fast-moving content, There were many skipped frames, the FPS count plummeted from 60 down do 30. Also, regardless of whether it's fast or slow-moving content, I had spikes which had the Inhome-Streaming client lag as far behind as one second (the missing frames were then completly skipped to get to the current frames).
I have no experience with nVidia GameStream, but I doubt they have a solution for network spikes. If the above is something you can live with, then definitely go for it, but for me this was unbearable.

And you are right again: I tested HomeStreaming with both e1000 and VirtIOnet and I confirm very annoying network spikes.
What about using a physical network card passed through with "vfio-pci"?
Beside that, the fact that I can stream a game means Windows can use the GTX 650 in some way (even if I'm booting with "-vga std").

Calrama wrote:

But regardless of the above two paragraphs, you might really want to get your passthrough to work on its own, before moving on to that stuff, because right now you're trying two things at the same time without having confirmed the first thing to be working properly.

You absolutely right, I will try qemu-git ASAP (sorry, but right now I'm very busy).

As always, thanks very much.

Offline

#3034 2014-10-21 23:49:03

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mouton_sanglant wrote:

Ok, I finished to configure my virtualized system. It's working fine with drivers installed inside host, played some games, no crashes ! Performances are "passed-troughed" just as expected !

Windows detects 2 displays, so it automatically enable extended desktop. One of these 2 displays can be disabled, however, the other one can't or I won't have any signal until I hit esc (to cancel action).
I ran some additionnal testings: can't pass through via the VGA cable, it duplicates the linux screen. So it seems to confirm what you pointed about the HDMI belongs to the dedicated card. I'll try someday to install a non-virtualized Windows on another HDD, disable the discrete card and plug a screen in the hdmi port.

That's quite an interesting design (thanks for the information), do post about it should go get around to it.

mouton_sanglant wrote:

Now, I have to fix an issue on the host: I can't launch any Desktop Manager. I think it is because of the i915 vga arbiter patch which disable DRI. Is that right ?
Window Managers works, so I should use Mozilla and stuff like that as long as I disable any 3d acceleration stuff, right ?

I'm afraid I cannot be of any help here, sorry. When I was still doing VGA-Passthrough, I always had full DRI-capabilities for my IGP and no problems at the host side other than sometimes my entire host system's input periphery completely freezing when booting the VM, even with the VGA-Arbitration patch and this kernel option active (despite the patch explicitly stating the opposite). But I never dug deep into this, since I moved on to OVMF to get rid of this freezing issue.

Offline

#3035 2014-10-22 00:18:13

Calrama
Member
From: Berlin
Registered: 2011-10-14
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

And you are right again: I tested HomeStreaming with both e1000 and VirtIOnet and I confirm very annoying network spikes.

One way to at least reduce that is to use hardware en- and decoding. To enable hardware encoding you need to install Geforce experience, click on the "shadow play" button in its GUI and flip the left-positioned virtual switch that they show in the new popup windows: Turn it on once, then off again. After that you will need to access the Inhome-Streaming options in Steam's settings (on the client), open up the advanced host settings, and enable hardware encoding (You might also want to set the same option on the VM's steam installation for good measure, but it shouldn't be necessary).

You might also try to set the client options to "fast" and set the bandwidth limit in the advanced client options to automatic (should it not already be set to it). I've found that even with hardware en- and decoding I could still detect a slight difference between host and guest while inside an FPS, because the camera didn't rotate completely smoothly on the guest. Just enough that it would get me killed in multiplayer. The only way I could rid of any noticeable latency when Inhome-streaming through an emulated VirtIO network card was to limit the bandwith manually to 3Mbit, at which 1080p looks like someone threw several buckets full of different-coloured paint at my screen from the inside (and 720p is unacceptable to me).

Ansa89 wrote:

What about using a physical network card passed through with "vfio-pci"?

I do not own one, but I'm going to try that out when I have enough time to make an informed decision about picking one.

Ansa89 wrote:

Beside that, the fact that I can stream a game means Windows can use the GTX 650 in some way (even if I'm booting with "-vga std").

It should. Just to be sure: You are able to stream regardless of whether your VM gets an emulated card or not (so both "-vga none" and "-vga std" work)?

Ansa89 wrote:

You absolutely right, I will try qemu-git ASAP (sorry, but right now I'm very busy).

As always, thanks very much.

Happy to help.

Last edited by Calrama (2014-10-22 00:18:28)

Offline

#3036 2014-10-22 14:45:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Suddenly, i can not boot OVMF's built-in shell or start Windows8.1 installation CD in UEFI mode. There is just blank screen and a cursor on serial0, and nothing on any VGA or video output. I need OVMF to avoid IO_PAGE_FAULTS referring to my first GPU. The windows ISO checked to be correct and it does boot on real hardware.

http://pastebin.com/9Wb3trV4

OVMF:20141021.b745.g8c83d0c
qemu 2.1.2
Fedora 21
Kernel 3.17.1-302.fc21.x86_64

Host Nvidia driver with vgaarb patch(though it shouldn't be needed anymore when using OVMF).

Two HD7750. romfile appear to be looking good, however, i couldn't update(ATIWinFlash just crashes after start, maybe i need drivers installed) it physically, so the ROMs in fact may be fake and broken. But if that's the case - asus support ticket should help me getting updated or getting the right ROMs.

Last edited by Duelist (2014-10-22 14:48:43)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3037 2014-10-22 15:09:17

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Two HD7750. romfile appear to be looking good, however, i couldn't update(ATIWinFlash just crashes after start, maybe i need drivers installed) it physically, so the ROMs in fact may be fake and broken. But if that's the case - asus support ticket should help me getting updated or getting the right ROMs.

Are you saying you have an ASUS Radeon HD7750 with UEFI support? Can I have that rom? Pretty please? My HD7750 is vga-only.

Offline

#3038 2014-10-22 17:01:53

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:

Two HD7750. romfile appear to be looking good, however, i couldn't update(ATIWinFlash just crashes after start, maybe i need drivers installed) it physically, so the ROMs in fact may be fake and broken. But if that's the case - asus support ticket should help me getting updated or getting the right ROMs.

Are you saying you have an ASUS Radeon HD7750 with UEFI support? Can I have that rom? Pretty please? My HD7750 is vga-only.

You need an Asus motherboard(Almost any, but i'm not sure about FM2(+) socket ones) and Asus HD7750. That way you are able to update it's bios.
http://www.asus.com/Graphics_Cards/HD77 … _Download/
http://www.asus.com/Graphics_Cards/HD77 … _Download/

As for exact .rom files - that will be useless because:
1.I have DCSL, they're passive cooled and they were pretty rare. The clocks, voltage and Fan profiles(yes, you've read it right, passively cooled GPU has fan profiles) are different on different models, and that may fry your card.
2.I took a diff out of two original romfiles, and we have a serial number in here. And something like CRC32-checksum in three bytes.. Don't know about the UEFI-capable ROM differences, but i've copy-pasted the serial just in case it'll be checked somehow.

Last edited by Duelist (2014-10-22 17:03:23)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3039 2014-10-22 18:47:43

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You need an Asus motherboard(Almost any, but i'm not sure about FM2(+) socket ones) and Asus HD7750. That way you are able to update it's bios.

That would be too easy. I don't have an ASUS motherboard. I downloaded the update utility. It's an .exe file. I haven't tried running it yet, but I guess it won't give me the .rom file easily. sad I just wanted to run an EFI bios without flashing it.

Duelist wrote:

As for exact .rom files - that will be useless because:
1.I have DCSL, they're passive cooled and they were pretty rare. The clocks, voltage and Fan profiles(yes, you've read it right, passively cooled GPU has fan profiles) are different on different models, and that may fry your card.
2.I took a diff out of two original romfiles, and we have a serial number in here. And something like CRC32-checksum in three bytes.. Don't know about the UEFI-capable ROM differences, but i've copy-pasted the serial just in case it'll be checked somehow.

Bummer. I certainly won't risk flashing a bios from any fanless design on my fanned HD7750-1GD5-V2. That being said, I don't think there is a serial number check in the GPU because some people even flash vbioses from different vendors (like the guy who commented on aw's blog about using Powercolor's bios on VTX3D's card).

I wish there was at least one EFI-capable vbios for HD7750 1GB DDR5 on TechPowerUP (all 51 of them are vga-only).

Offline

#3040 2014-10-23 00:32:17

dtm
Member
Registered: 2014-10-22
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Everyone,

Thanks for the fantastic thread, using it I have managed to get a fair way towards getting a working setup, at the same time breathing new life into my aging machine.

Specs:

Motherboard: Gigabyte GA-X58A-UD5 (w/VT-d enabled bios flashed)
CPU: Intel Xeon 5670
RAM: 12GB
Primary Video: Nvidia NVS 300
Passthrough Video: Nvidia GTX 570

I've installed on Debian jessie (testing) with a stock 3.17.1 kernel, to which I have added a quirk in drivers/pci/quirks.c to get my Marvell 9128 SATA3 controller working properly:

/* Declare this quirk for Marvell 9128 controller */ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9128, quirk_dma_func1_alias);

I'm running the versions of libvirt and qemu which are standard to debian jessie, i.e. qemu 2.1.2 and libvirt 1.2.8. I've actually managed to get my video and USB3 controller bound and passed succesfully to a windows 8.1 virtual machine, and thus far it all seems to work well, although I am yet to do any benchmarking or performance testing.

The problem I'm having is when I enable intel_iommu at boot, my onboard sound is disabled. I see the following messages in dmesg:

dmesg | grep -E '(IOMMU|DMAR|1b\.0)' [ 0.000000] ACPI: DMAR 0x00000000D7EDBF00 0000B0 (v01 IntelR AWRDACPI 322E3030 DRWA 00000002) [ 0.000000] Intel-IOMMU: enabled [ 0.023999] dmar: IOMMU 0: reg_base_addr fe711000 ver 1:0 cap c9008010e60262 ecap f0207a [ 0.024005] dmar: IOMMU 1: reg_base_addr fe710000 ver 1:0 cap c90780106f0462 ecap f020fe [ 0.364780] pci 0000:00:1b.0: [8086:3a3e] type 00 class 0x040300 [ 0.364794] pci 0000:00:1b.0: reg 0x10: [mem 0xfbff4000-0xfbff7fff 64bit] [ 0.364856] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold [ 0.364891] pci 0000:00:1b.0: System wakeup disabled by ACPI [ 3.854860] DMAR: No ATSR found [ 3.854903] IOMMU 0 0xfe711000: using Queued invalidation [ 3.854906] IOMMU 1 0xfe710000: using Queued invalidation [ 3.854907] IOMMU: Setting RMRR: [ 3.854921] IOMMU: Setting identity map for device 0000:00:1a.0 [0xd7ee0000 - 0xd7efffff] [ 3.854956] IOMMU: Setting identity map for device 0000:00:1a.1 [0xd7ee0000 - 0xd7efffff] [ 3.854983] IOMMU: Setting identity map for device 0000:00:1a.2 [0xd7ee0000 - 0xd7efffff] [ 3.855009] IOMMU: Setting identity map for device 0000:00:1a.7 [0xd7ee0000 - 0xd7efffff] [ 3.855035] IOMMU: Setting identity map for device 0000:00:1d.0 [0xd7ee0000 - 0xd7efffff] [ 3.855062] IOMMU: Setting identity map for device 0000:00:1d.1 [0xd7ee0000 - 0xd7efffff] [ 3.855088] IOMMU: Setting identity map for device 0000:00:1d.2 [0xd7ee0000 - 0xd7efffff] [ 3.855114] IOMMU: Setting identity map for device 0000:00:1d.7 [0xd7ee0000 - 0xd7efffff] [ 3.855132] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 3.855144] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 4.146418] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 4.146419] AMD IOMMUv2 functionality not available on this system [ 8.116548] snd_hda_intel 0000:00:1b.0: irq 42 for MSI/MSI-X [ 8.120953] dmar: DMAR:[DMA Read] Request device [00:1b.0] fault addr ffff6000 DMAR:[fault reason 02] Present bit in context entry is clear [ 9.121162] snd_hda_intel 0000:00:1b.0: Codec #2 probe error; disabling it... [ 9.125612] dmar: DMAR:[DMA Read] Request device [00:1b.0] fault addr ffff6000 DMAR:[fault reason 02] Present bit in context entry is clear [ 14.140288] snd_hda_intel 0000:00:1b.0: no AFG or MFG node found [ 14.140307] snd_hda_intel 0000:00:1b.0: no codecs initialized

If I boot without intel_iommu=on, then the codec is detected correctly and everything comes up OK. This is also the case if I boot with intel_iommu=pt, although in that case I do not see any IOMMU groups and attempting to bind devices using VFIO fails.

I was wondering if anyone had seen anything like this before or had any ideas as to what I might try to get my onboard sound working again?

Offline

#3041 2014-10-23 04:26:18

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Updated from 3.16 to  3.17.1-1 and now the vm skips. When I move the mouse or drag a window the mouse or window will freeze for a sec every few seconds. This happen to anyone else when they went to kernel 3.17?

Using X79 motherboard, 4930k cpu, and passing through an nvidia titan with a nvidia 670 on the host.

This happens using ovmf/1440FX as well as with bios/q35. Using qemu-git 2.2 and libvirt-git 1.2.9 and ovmf-svn 16214

Last edited by kristopher004 (2014-10-23 04:29:04)

Offline

#3042 2014-10-23 10:40:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:

You need an Asus motherboard(Almost any, but i'm not sure about FM2(+) socket ones) and Asus HD7750. That way you are able to update it's bios.

That would be too easy. I don't have an ASUS motherboard. I downloaded the update utility. It's an .exe file. I haven't tried running it yet, but I guess it won't give me the .rom file easily. sad I just wanted to run an EFI bios without flashing it.

Duelist wrote:

As for exact .rom files - that will be useless because:
1.I have DCSL, they're passive cooled and they were pretty rare. The clocks, voltage and Fan profiles(yes, you've read it right, passively cooled GPU has fan profiles) are different on different models, and that may fry your card.
2.I took a diff out of two original romfiles, and we have a serial number in here. And something like CRC32-checksum in three bytes.. Don't know about the UEFI-capable ROM differences, but i've copy-pasted the serial just in case it'll be checked somehow.

Bummer. I certainly won't risk flashing a bios from any fanless design on my fanned HD7750-1GD5-V2. That being said, I don't think there is a serial number check in the GPU because some people even flash vbioses from different vendors (like the guy who commented on aw's blog about using Powercolor's bios on VTX3D's card).

I wish there was at least one EFI-capable vbios for HD7750 1GB DDR5 on TechPowerUP (all 51 of them are vga-only).

TechPowerUp VBIOS database is outdated as hell. You can try asking asus support for exact ROMs. Or you can find some friends with somewhat decent new motherboard and windows on board.

As for the serial number - the driver may go crazy.

And you can also try running that asus thingie, it may give you ROM, but refuse to flash.

Last edited by Duelist (2014-10-23 11:20:01)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3043 2014-10-23 11:33:55

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Suddenly, i can not boot OVMF's built-in shell or start Windows8.1 installation CD in UEFI mode. There is just blank screen and a cursor on serial0, and nothing on any VGA or video output.

Alright, i've turned off GPU PT, got into shell, booted from file(win8.1's EFI loader), got a progress bar about "loading files" full, and then nothing anywhere. Something hints me i should have some graphics showing up, but there is no output.

After installing windows8 without OVMF, i'm still unable to boot with gpu passthrough. Seems like the ROM files were broken, and something screws up badly. I guess i'll have to search for some other ASUS motherboard to update them physically.

Last edited by Duelist (2014-10-23 12:53:34)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3044 2014-10-24 07:47:08

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
ilya80 wrote:

Hi guys!

Today I successfully hardmodded my GTX 680 to report itself as GRID K2.

When I try to passthrough the card, I get no output on either display outputs though. Nothing suspicious in the logs, just:

Nov 24 20:19:42 cave-lin kernel: [ 368.001538] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Nov 24 20:19:42 cave-lin kernel: [ 368.001719] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Nov 24 20:20:20 cave-lin kernel: [ 405.590323] kvm: zapping shadow pages for mmio generation wraparound

I havent changed anything since last time I run my config, except of device ID to assign to pci-stub during boot. Prior to that passthrough worked and I just wasnt able to install Nvidia drivers in guest.

Any advice?

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.  However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Hi, aw.

I tried it today. Passing a real(not modded) Nvidia Grid k2 results in code 12 in my windows 2008 vm... Same as a Quadro 400... Look's like "x-vga" is a must...

l have some difficulty with libvirt. No matter what I did, if I start my vm with libvirt, I always get the "Cannot allocate memory" error. "ulimit -l unlimited" doesn't help at all... Is anyone have similar issue? Now I can only start my vm manually with a long qemu command sad

Offline

#3045 2014-10-24 11:07:33

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I want to start looking into using OVMF, but before spending too much time on it a quick check: Can anyone here confirm if the XFX HD7850 (Dual Dissipation edition) has support for UEFI/GOP? I alread asked XFX, but got no answer so far...
Thanks!

Offline

#3046 2014-10-24 11:24:33

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You can try asking asus support for exact ROMs.

Great idea! I contacted them and got the rom just now. :3

./rom-parser 113-AD41300-123.bin Valid ROM signature found @0h, PCIR offset 224h PCIR: type 0, vendor: 1002, device: 683f, class: 030000 PCIR: revision 0, vendor revision: f20 Valid ROM signature found @10000h, PCIR offset 1ch PCIR: type 3, vendor: 1002, device: 683f, class: 030000 PCIR: revision 0, vendor revision: 0 EFI: Signature Valid Last image

It's time to give OVMF a try...

Offline

#3047 2014-10-24 11:25:32

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

I want to start looking into using OVMF, but before spending too much time on it a quick check: Can anyone here confirm if the XFX HD7850 (Dual Dissipation edition) has support for UEFI/GOP? I alread asked XFX, but got no answer so far...
Thanks!

Read this: http://vfio.blogspot.de/2014/08/does-my … t-efi.html

Offline

#3048 2014-10-24 12:17:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:

You can try asking asus support for exact ROMs.

Great idea! I contacted them and got the rom just now. :3

./rom-parser 113-AD41300-123.bin Valid ROM signature found @0h, PCIR offset 224h PCIR: type 0, vendor: 1002, device: 683f, class: 030000 PCIR: revision 0, vendor revision: f20 Valid ROM signature found @10000h, PCIR offset 1ch PCIR: type 3, vendor: 1002, device: 683f, class: 030000 PCIR: revision 0, vendor revision: 0 EFI: Signature Valid Last image

It's time to give OVMF a try...

Please, mark if you manage to get that cards doing something more useful than powering down when passing the ROM via romfile=.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3049 2014-10-24 13:37:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

doubledr wrote:
aw wrote:
ilya80 wrote:

Hi guys!

Today I successfully hardmodded my GTX 680 to report itself as GRID K2.

When I try to passthrough the card, I get no output on either display outputs though. Nothing suspicious in the logs, just:

Nov 24 20:19:42 cave-lin kernel: [ 368.001538] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Nov 24 20:19:42 cave-lin kernel: [ 368.001719] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Nov 24 20:20:20 cave-lin kernel: [ 405.590323] kvm: zapping shadow pages for mmio generation wraparound

I havent changed anything since last time I run my config, except of device ID to assign to pci-stub during boot. Prior to that passthrough worked and I just wasnt able to install Nvidia drivers in guest.

Any advice?

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.  However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Hi, aw.

I tried it today. Passing a real(not modded) Nvidia Grid k2 results in code 12 in my windows 2008 vm... Same as a Quadro 400... Look's like "x-vga" is a must...

Use 440FX not Q35

l have some difficulty with libvirt. No matter what I did, if I start my vm with libvirt, I always get the "Cannot allocate memory" error. "ulimit -l unlimited" doesn't help at all... Is anyone have similar issue? Now I can only start my vm manually with a long qemu command sad

If libvirt doesn't know you're doing device assignment it won't set your limits correctly.  If you use 440FX instead of Q35 you don't need any qemu:commandline sections and should be allowing libvirt to manage everything.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3050 2014-10-24 18:22:00

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Please, mark if you manage to get that cards doing something more useful than powering down when passing the ROM via romfile=.

Well, I can confirm that this rom works in vga mode. I used it to load my standard Win7 VM.
EFI, on the other hand, is a pain. First I tried to install Win8.1 just with QXL, but OVMF can't load the installation DVD (no drivers for the UDF filesystem?). I copied the DVD to a fat32 partition and managed to get the loader running, but it still fails quite soon with "can't find the media" error (maybe the installation process looks for UDF and dies because there is fat32?). Anyway, just to test EFI vbios I tried connecting the GPU annnnnd... nothing. I get a black screen and 100% one core utilisation, can't see OVMF, can't get to Windows installer.

P.S. I noticed in gpuz that this new bios (ver 015.032) bumps the default core clock to 900MHz while the original bios (ver 015.014) sets default to 820MHz.

Last edited by deniv (2014-10-24 18:26:04)

Offline

#3051 2014-10-24 18:31:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When you're trying OVMF, please make sure you're using the pure-EFI binary and not one that mentions the CSM.  We explicitly do not want the CSM, which can run the VGA BIOS and brings back all the issues with VGA arbitration.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3052 2014-10-24 21:14:26

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Suddenly, i can not boot OVMF's built-in shell or start Windows8.1 installation CD in UEFI mode. There is just blank screen and a cursor on serial0, and nothing on any VGA or video output. I need OVMF to avoid IO_PAGE_FAULTS referring to my first GPU. The windows ISO checked to be correct and it does boot on real hardware.

I think I'm hitting the same bug as you. I've tried some combinations of qemu/ovmf versions and qemu options but still got blank screen whenever OVMF touches the EFI part of the vbios (vga works as I mentioned before). So what's left? I compared EfiRom output:

Image 2 -- Offset 0x10000 ROM header contents Signature 0xAA55 PCIR offset 0x001C Signature PCIR Vendor ID 0x1002 Device ID 0x683F Length 0x0018 Revision 0x0000 DeviceListOffset 0x00 Class Code 0x030000 Image size 0xEE00 Code revision: 0x0000 MaxRuntimeImageLength 0x00 ConfigUtilityCodeHeaderOffset 0x4F47 DMTFCLPEntryPointOffset 0x2050 Indicator 0x80 (last image) Code type 0x03 (EFI image) EFI ROM header contents EFI Signature 0x0EF1 Compression Type 0x0001 (compressed) Machine type 0x8664 (unknown) Subsystem 0x000B (EFI boot service driver) EFI image offset 0x005C (@0x1005C)

with some vbioses from TechPowerUp. The result: there are no striking differences. The only lines that change between different vbioses: Device ID (obviously), Image Size, EFI image offset, and Image offset (the first line). I'm no expert, but judging from this I belive my vbios isn't broken. I can only assume it's a bug in OVMF. I've noticed aw's GT635 has "Image 2 -- Offset 0xF400" while my rom's "Image 2 -- Offset 0x10000" Could that additional digit make a difference for OVMF? In any case, 0x10000 isn't rare. Many R290 use the same offset.

P.S. Yes, I'm using pure-efi.

Offline

#3053 2014-10-24 21:58:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another sad fact of EFI is that vendors ship broken ROMs.  The XFX R7790 is one such card.  I thought I was lucky finding an EFI ROM on a 7700-series card, but it doesn't work.  If I install it as the primary graphics in my system and configure the BIOS to prefer the EFI BIOS, the system won't boot.  It behaves the same way in an OVMF VM.  I've contacted XFX support but they seem unable to do anything other than repackage a ROM provided by AMD and indicate that AMD has not provided EFI support for this card.  They seem unable to comprehend that they've shipped a card that not only doesn't support EFI, but in fact prevents using it since executing the EFI ROM seems to be the cause of the hang.  It would be interesting to trace the execution in the emulator and see where it breaks, maybe it could be patched.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3054 2014-10-24 22:39:18

gyrfalco
Member
Registered: 2014-09-30
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hello, guys.

My spec: AMD A10-6700 with integrated Radeon HD 8670D (host GPU) + Radeon HD 7870 (GV-R787OC-2GD - guest GPU).

I have problems with Catalyst driver (13.*, 14.4, 14.7, 14.8, 14.9) install: system (win 8.1) crashes during driver installation, with or without CCC + display driver installation, or just updading default win display driver in device manager. 

I tried both seabios and OVMF with same result.

qemu-2.1.50, kernel 3.17.1:

iommu=pt pci-stub.ids=1002:6818,1002:aab0

Machine starts as:

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host,hv_time,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 -vga none -boot menu=on -rtc base=localtime \ -device vfio-pci,host=01:00.0,multifunction=on \ -drive file=windows.img,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ -drive if=pflash,format=raw,readonly,file=/usr/share/qemu/efi-bios.bin

Is it possible (actually only drivers installation, because without driver VM runs smooth and fine, through second monitor) when host gpu is integrated Radeon and guest gpu is 7870?

Thanks, sorry for my English.

Last edited by gyrfalco (2014-10-26 11:28:59)

Offline

#3055 2014-10-25 06:36:55

doubledr
Member
Registered: 2013-12-26
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
doubledr wrote:
aw wrote:

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.  However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Hi, aw.

I tried it today. Passing a real(not modded) Nvidia Grid k2 results in code 12 in my windows 2008 vm... Same as a Quadro 400... Look's like "x-vga" is a must...

Use 440FX not Q35

l have some difficulty with libvirt. No matter what I did, if I start my vm with libvirt, I always get the "Cannot allocate memory" error. "ulimit -l unlimited" doesn't help at all... Is anyone have similar issue? Now I can only start my vm manually with a long qemu command sad

If libvirt doesn't know you're doing device assignment it won't set your limits correctly.  If you use 440FX instead of Q35 you don't need any qemu:commandline sections and should be allowing libvirt to manage everything.

Thx, aw. It works!

I found that my libvirt used cgroup to control memory limit, maybe that is the reason why ulimit command doesn't work.

Nov 24 20:19:42 cave-lin kernel: [ 368.001538] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Nov 24 20:19:42 cave-lin kernel: [ 368.001719] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Nov 24 20:20:20 cave-lin kernel: [ 405.590323] kvm: zapping shadow pages for mmio generation wraparound

Offline

#3056 2014-10-25 08:25:05

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Another sad fact of EFI is that vendors ship broken ROMs.

That's sad indeed. How can they screw such an easy-to-test feature? At least vgaarb patches still work.

aw wrote:

It would be interesting to trace the execution in the emulator and see where it breaks, maybe it could be patched.

Wish I could help, but it's beyond me. Please inform us if you manage to get any HD77xx card work in EFI mode.

Offline

#3057 2014-10-25 09:59:25

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First of all: my hard disk exploded (it was a bit old), so ATM I'm unable to proceed with further tests. However I managed to do a last test before it broke.


Calrama wrote:

One way to at least reduce that is to use hardware en- and decoding. To enable hardware encoding you need to install Geforce experience, click on the "shadow play" button in its GUI and flip the left-positioned virtual switch that they show in the new popup windows: Turn it on once, then off again. After that you will need to access the Inhome-Streaming options in Steam's settings (on the client), open up the advanced host settings, and enable hardware encoding (You might also want to set the same option on the VM's steam installation for good measure, but it shouldn't be necessary).

You might also try to set the client options to "fast" and set the bandwidth limit in the advanced client options to automatic (should it not already be set to it).

I will try those settings after buying a new hard disk.

Calrama wrote:

I've found that even with hardware en- and decoding I could still detect a slight difference between host and guest while inside an FPS, because the camera didn't rotate completely smoothly on the guest. Just enough that it would get me killed in multiplayer. The only way I could rid of any noticeable latency when Inhome-streaming through an emulated VirtIO network card was to limit the bandwith manually to 3Mbit, at which 1080p looks like someone threw several buckets full of different-coloured paint at my screen from the inside (and 720p is unacceptable to me).

Really not a good deal.

Calrama wrote:

I do not own one, but I'm going to try that out when I have enough time to make an informed decision about picking one.

Informed decision:
- go on an e-commerce website
- click "networking" section
- click "pci ethernet" section
- sort items by "ascending price"
- close you eyes an choose one card
I really don't know if there is one card better than others.

Calrama wrote:

It should. Just to be sure: You are able to stream regardless of whether your VM gets an emulated card or not (so both "-vga none" and "-vga std" work)?

Last test done: yes, I can stream regardless of "-vga std" or "-vga none" is passed to qemu.

Offline

#3058 2014-10-25 11:50:26

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:

Please, mark if you manage to get that cards doing something more useful than powering down when passing the ROM via romfile=.

Well, I can confirm that this rom works in vga mode. I used it to load my standard Win7 VM.
EFI, on the other hand, is a pain. First I tried to install Win8.1 just with QXL, but OVMF can't load the installation DVD (no drivers for the UDF filesystem?). I copied the DVD to a fat32 partition and managed to get the loader running, but it still fails quite soon with "can't find the media" error (maybe the installation process looks for UDF and dies because there is fat32?). Anyway, just to test EFI vbios I tried connecting the GPU annnnnd... nothing. I get a black screen and 100% one core utilisation, can't see OVMF, can't get to Windows installer.

P.S. I noticed in gpuz that this new bios (ver 015.032) bumps the default core clock to 900MHz while the original bios (ver 015.014) sets default to 820MHz.

Try installing windows 8.1 via USB flash drive. There was an official way of doing this. That should solve all controller-related issues, however, i've managed to install windows8 from virtio-scsi-pci scsi-cd to virtio-blk-pci using the virtio.iso image connected through ide-cd..
And the clock difference - that's why i warned you - my cards have 800MHz and they're pretty hot, so getting them work any faster would be dangerous.

aw wrote:

We explicitly do not want the CSM

That's an interesting remark.

aw wrote:

If I install it as the primary graphics in my system and configure the BIOS to prefer the EFI BIOS, the system won't boot.

Well, i'm going to visit my friend with ASUS-AMD-Windows system somewhere near wednesday and hopefully, i'll provide some testing information from the real hardware combination.

aw wrote:

They seem unable to comprehend that they've shipped a card that not only doesn't support EFI, but in fact prevents using it since executing the EFI ROM seems to be the cause of the hang.  It would be interesting to trace the execution in the emulator and see where it breaks, maybe it could be patched.

Well, after i get some testing information, if the VBIOS is really broken - that would be extremely fun. ASUS support seem to actually understand what the hell i am trying to ask them - last time i've asked if it's possible to combine three HD7750 with Richland or Trinity-based CPU(not APU) on one of their motherboards, and suddenly, i've got an answer.

And i've finally managed to boot windows7(not 8!) installation ISO from OVMF. Let's see what will happen.

Last edited by Duelist (2014-10-25 12:14:58)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3059 2014-10-25 17:02:20

gyrfalco
Member
Registered: 2014-09-30
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

And i've finally managed to boot windows7(not 8!) installation ISO from OVMF.

can you explain how? thanks.

Offline

#3060 2014-10-25 21:25:34

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I decided to reinstall my whole PC, this time in UEFI mode instead of the old BIOS.
Got everything up and running and was following this tutorial again but ran into some problems (again...).

This time i'm stuck at the following (again...):

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I'm 100% sure I binded the devices, by both using systemd at boot, and manually running vfio-bind 0000:02:00.0 0000:02:00.1.
I get no errors so I just assume it works, but it clearly doesn't...

Offline

#3061 2014-10-25 21:27:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

So I decided to reinstall my whole PC, this time in UEFI mode instead of the old BIOS.
Got everything up and running and was following this tutorial again but ran into some problems (again...).

This time i'm stuck at the following (again...):

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I'm 100% sure I binded the devices, by both using systemd at boot, and manually running vfio-bind 0000:02:00.0 0000:02:00.1.
I get no errors so I just assume it works, but it clearly doesn't...

Intel_iommu=on boot option


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3062 2014-10-25 21:45:23

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
PureTryOut wrote:

So I decided to reinstall my whole PC, this time in UEFI mode instead of the old BIOS.
Got everything up and running and was following this tutorial again but ran into some problems (again...).

This time i'm stuck at the following (again...):

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I'm 100% sure I binded the devices, by both using systemd at boot, and manually running vfio-bind 0000:02:00.0 0000:02:00.1.
I get no errors so I just assume it works, but it clearly doesn't...

Intel_iommu=on boot option

Aah, knew I forgot something, thanks!

Now i'm back to the initial problem: qemu only giving me a console, unless I remove the -vga none option.
I've read that several others have this problem too, but I haven't found a solution yet.

Last edited by PureTryOut (2014-10-25 21:48:45)

Offline

#3063 2014-10-26 03:27:53

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:

Please, mark if you manage to get that cards doing something more useful than powering down when passing the ROM via romfile=.

Well, I can confirm that this rom works in vga mode. I used it to load my standard Win7 VM.
EFI, on the other hand, is a pain. First I tried to install Win8.1 just with QXL, but OVMF can't load the installation DVD (no drivers for the UDF filesystem?). I copied the DVD to a fat32 partition and managed to get the loader running, but it still fails quite soon with "can't find the media" error (maybe the installation process looks for UDF and dies because there is fat32?). Anyway, just to test EFI vbios I tried connecting the GPU annnnnd... nothing. I get a black screen and 100% one core utilisation, can't see OVMF, can't get to Windows installer.

P.S. I noticed in gpuz that this new bios (ver 015.032) bumps the default core clock to 900MHz while the original bios (ver 015.014) sets default to 820MHz.

Make sure you're not using ovmf from repo, it's useless.

Offline

#3064 2014-10-26 06:50:35

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

which ovmf are you using?

Offline

#3065 2014-10-26 11:09:39

dave558
Member
Registered: 2013-11-28
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi

I've just compiled a kernel (3.10) with vfio support (CONFIG_VFIO_PCI_VGA=y).

I have installed qemu-kvm through yum but it appears that this RPM has not been compiled for VFIO. When trying assign the graphic card to guests there is no option to assign device to vfio-pci.

I have read also that qemu 2.0+ supports VFIO and I am thinking of compiling qemu from the QEMU website. Although Qemu can run as a stand-alone emulator, I am assumming it will use the kernel-module automatically if available. Is this the case?

Also will there be any difference in performance between qemu compiled from source and qemu-kvm? Or is there anything else you can recommend? Thank yoo.

NOTE: This is a CentOS system, that is why I have installed the default RPM install qemu-kvm rather than QEMU.

Offline

#3066 2014-10-26 13:55:50

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

which ovmf are you using?

ovmf-svn from AUR.

Offline

#3067 2014-10-26 15:35:32

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kristopher004 wrote:

Updated from 3.16 to  3.17.1-1 and now the vm skips. When I move the mouse or drag a window the mouse or window will freeze for a sec every few seconds. This happen to anyone else when they went to kernel 3.17?

My problem ended up being that I recompiled 3.17 and didn't recompile my nvidia driver, if anyone else has a similar problem.

Offline

#3068 2014-10-26 17:25:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gyrfalco wrote:
Duelist wrote:

And i've finally managed to boot windows7(not 8!) installation ISO from OVMF.

can you explain how? thanks.

Actually windows 7 supports GPT and UEFI natibely, you just have to force it.
http://technet.microsoft.com/en-gb/libr … 10%29.aspx
You can use the shell, or you can type exit and navigate yourself through menus.
You have to use pure-efi OVMF image.

Last edited by Duelist (2014-10-26 21:28:18)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3069 2014-10-27 01:51:32

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Offline

#3070 2014-10-27 07:43:12

gyrfalco
Member
Registered: 2014-09-30
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Actually windows 7 supports GPT and UEFI natibely, you just have to force it.
...
You have to use pure-efi OVMF image.

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Also, can you show qemu start command, which you run win7 with ovmf, thanks.

Last edited by gyrfalco (2014-10-27 08:41:59)

Offline

#3071 2014-10-27 10:26:43

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gyrfalco wrote:

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Yes, it is.

EDIT: ovmf qemu args

-drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1

Last edited by dwe11er (2014-10-27 10:33:20)

Offline

#3072 2014-10-27 10:53:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
gyrfalco wrote:

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Yes, it is.

EDIT: ovmf qemu args

-drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1

Hmm, do you explicitly need that unit=int64 parameter set if using if=pflash? It seem to be working without it, but i can't test it properly by changing some setting in UEFI setup menu - there is no settings.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3073 2014-10-27 15:41:09

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
dwe11er wrote:
gyrfalco wrote:

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Yes, it is.

EDIT: ovmf qemu args

-drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1

Hmm, do you explicitly need that unit=int64 parameter set if using if=pflash? It seem to be working without it, but i can't test it properly by changing some setting in UEFI setup menu - there is no settings.

No idea, probably not. It might be used to order pflash drives (I'm using libvirt).

Offline

#3074 2014-10-27 15:49:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3075 2014-10-28 09:01:37

Casperian
Member
Registered: 2014-10-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, been watching this thread for some time.
So I recently have set up a pass through and passed a 5450 to a Windows 7 Guest. This was easy but I had a BSOD issue caused by device reset which I worked round by not passing HDMI audio through. (BSOD 116)
From here I then tried to pass a GTX 750 through. Everything went fine, Qemu starts and gives me the warning "Bar 3 mmap unsupported" and the card is not outputting anything. I boot Qemu a second time without -vga none to see if I can find a problem and find the device is passed through, it is and I get the latest stable nvidia drivers installed to see if that fixes anything. The card is currently throwing code 12 in windows.

From what I found over the last few hours, I will check PCI-stub binding to the device tomorrow. Also thinking about trying pci-assign to see if it works.

The code I am using is;

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 12288 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \ -drive file=/home/server/window_nvidia750.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -boot menu=on 

Any advice on the issue would be awesome
I can't post anything more now as the computer is not on hand

Cheers
Casper

Last edited by Casperian (2014-10-28 09:05:00)

Offline

#3076 2014-10-28 09:49:35

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Casperian wrote:

Also thinking about trying pci-assign to see if it works.

It won't. It's kinda deprecated.

Device code 12, afair, is "not enough resources for a device to run". Since qemu says "BAR 3 MMAP unsupported" - it is not able to mmap BAR 3 - some(BAR 3 that is) region of memory is in use. Check lspci (with numerous -v and -n options) output for memory ranges, then check /proc/iomem and /proc/mtrr. Maybe you'll notice that your GPU is in use by something like vesa or nouveau, maybe you'll need to enable MTRR cleanup via kernel parameters. And what host GPU do you have?

Also you may be running into nvidia driver related problems - ensure that is not the case by disabling hyper-v extensions(or whatever it was) and hide kvm signature.

...aand you might not be needing that -bios line at all.

Last edited by Duelist (2014-10-28 09:50:16)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3077 2014-10-28 10:39:22

gyrfalco
Member
Registered: 2014-09-30
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My success story:

  • ASRock FM2A88M Extreme4+ [UEFI BIOS 2.40]

  • AMD A10-6700 with Radeon HD 8670D

  • AMD Radeon HD 7870 [Gigabyte GV-R787OC-2GD - UEFI BIOS F11]

  • gentoo-sources-3.17.1 kernel without any patches for VGA-Passthrough

  • qemu-2.1.50

  • OVMF 16056-1 from ovmf-svn

  • Windows 8.1 with Catalyst driver v.14.9

host: HD 8670D
guest: HD 7870

vfio-assign did not work for me (crash during radeon driver installation. with CCC or not, through setup.exe or Device Manager, seabios or UEFI-ovfm, with romfile or without), so I use good old pci-assign which works perfectly.

pci-stub in kernel cmdline with fix for ASRock FM2A88M Extreme4+ :

iommu=pt ivrs_ioapic[5]=00:14.0 pci-stub.ids=1002:6818,1002:aab0

Install:

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -vga qxl -rtc base=localtime -pflash ovmf_x64.bin \ -device pci-assign,host=01:00.0 -device pci-assign,host=01:00.1 \ -device virtio-scsi-pci,id=scsi -drive file=windows8.img,if=virtio,id=disk_primary,format=raw \ -drive file=windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \ -cdrom virtio-win-0.1-81.iso

Run:

qemu-system-x86_64 -enable-kvm -m 4096 \ -cpu host,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -smp 4,sockets=1,cores=4,threads=1 \ -vga none -rtc base=localtime -pflash ovmf_x64.bin \ -device pci-assign,host=01:00.0 \ -device virtio-scsi-pci,id=scsi -drive file=windows8.img,if=virtio,id=disk_primary,format=raw

LuxMark (OpenCL) benchmark results: LuxBall HDR - Only selected devices (GPU Pitcairn):

Win8.1 native: 10730
Linux qemu-kvm vga-passthrough: 9208

Difference: 14%

Not bad? Or what?

Last edited by gyrfalco (2014-10-28 10:40:20)

Offline

#3078 2014-10-28 14:51:59

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gyrfalco wrote:

My success story:

  • ASRock FM2A88M Extreme4+ [UEFI BIOS 2.40]

  • AMD A10-6700 with Radeon HD 8670D

  • AMD Radeon HD 7870 [Gigabyte GV-R787OC-2GD - UEFI BIOS F11]

  • gentoo-sources-3.17.1 kernel without any patches for VGA-Passthrough

  • qemu-2.1.50

  • OVMF 16056-1 from ovmf-svn

  • Windows 8.1 with Catalyst driver v.14.9

host: HD 8670D
guest: HD 7870

vfio-assign did not work for me (crash during radeon driver installation. with CCC or not, through setup.exe or Device Manager, seabios or UEFI-ovfm, with romfile or without), so I use good old pci-assign which works perfectly.

pci-stub in kernel cmdline with fix for ASRock FM2A88M Extreme4+ :

iommu=pt ivrs_ioapic[5]=00:14.0 pci-stub.ids=1002:6818,1002:aab0

Install:

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -vga qxl -rtc base=localtime -pflash ovmf_x64.bin \ -device pci-assign,host=01:00.0 -device pci-assign,host=01:00.1 \ -device virtio-scsi-pci,id=scsi -drive file=windows8.img,if=virtio,id=disk_primary,format=raw \ -drive file=windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \ -cdrom virtio-win-0.1-81.iso

Run:

qemu-system-x86_64 -enable-kvm -m 4096 \ -cpu host,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -smp 4,sockets=1,cores=4,threads=1 \ -vga none -rtc base=localtime -pflash ovmf_x64.bin \ -device pci-assign,host=01:00.0 \ -device virtio-scsi-pci,id=scsi -drive file=windows8.img,if=virtio,id=disk_primary,format=raw

LuxMark (OpenCL) benchmark results: LuxBall HDR - Only selected devices (GPU Pitcairn):

Win8.1 native: 10730
Linux qemu-kvm vga-passthrough: 9208

Difference: 14%

Not bad? Or what?

Surprising! But you've used OVMF anyway. I think AW will tell us some difference. I thought that pci-assign was deprecated..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3079 2014-10-28 14:59:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Surprising! But you've used OVMF anyway. I think AW will tell us some difference. I thought that pci-assign was deprecated..

Not really surprising with OVMF, Radeon cards don't require any quirks when running with EFI.  Legacy KVM device assignment is deprecated though, so it will eventually be removed.  I'd certainly rather fix vfio if it's somehow getting in the way than endorse a solution using pci-assign.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3080 2014-10-29 08:09:15

Casperian
Member
Registered: 2014-10-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sadly I'm still having problems with the passthrough. I found an issue where vesafb was caused the bar 3 error (my fault. messed up with card was primary in the bios). Thank Duelist for the suggestions, helped a lot.

I updated to qemu 2.1.2  and now my script looks like

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -usb -usbdevice host:046d:c52b \ -drive file=/home/server/windowstry.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/server/Downloads/Win.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on

I am still having a code 12 error as far as I can tell and the GPU is refusing to output anything out the back of it. My image got stuck in a repair loop so I'll see if the error has changed afterward.

Offline

#3081 2014-10-29 08:36:07

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not sure with OVMF, but maybe add -vga none, to get rid of error code 12?

Offline

#3082 2014-10-29 10:24:44

Casperian
Member
Registered: 2014-10-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, looks like I didn't save my edit...
Anyway with vga none, I am getting no output from the 750 and after ~ 30 seconds I get a kvm internal error.
Can't post the full message as I am using my phone but the cs of the message seemed to correspond to the cards memory listed in cat /proc/iomem
My mate passed a quadro through at the same time using similar hardware and following my steps. (the update was to solve his code 43 error). Do I need to get the cards rom and specify it?

Offline

#3083 2014-10-29 11:07:09

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to check aw's vfio blog, about uefi rom, maybe you need rom file or/and your gpu doesn't have uefi rom?
Again not sure, my gigabyte gtx680 4gb works either way (without rom file option), I'm using old vga way with patches, since I have 0 problems with it.

Last edited by slis (2014-10-29 11:09:23)

Offline

#3084 2014-10-29 11:42:02

Casperian
Member
Registered: 2014-10-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What patches are you running? I am running none (although a lot of fixes) cause I don't have an igpu to worry about. I'm still trying the vga way for now, just started looking at ovmf recently but am considering just putting the old way aside for now and trying it.

Last edited by Casperian (2014-10-29 11:53:24)

Offline

#3085 2014-10-29 12:28:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

I am not sure with OVMF, but maybe add -vga none, to get rid of error code 12?

Are you sure he is using OVMF? His commandline "script" doesn't do anything related. Some time ago there was even explicitly qemu's bios via -bios option.
If he's fine without OVMF - let it be. Because UEFI is tricky.

The UEFI trickiness:
1.My GPUs still have old VGA bios inside. I have a new, hybrid VGA-UEFI bios, but i'm unable to flash it hardware-wise and it appear to cause hangups on my qemu machine when using -romfile option. YMMV, but better be safe.
2.Windows. "We explicitly do not want CSM". I am not sure if you don't need to reinstall windows in order to get it working under UEFI if there was BIOS during the installation process. I guess there may be some details related to the bootloader.

When using vfio x-vga=on, you HAVE TO -vga none. Only that way you're getting output of you GPU.
When you are debugging something, like, uninstalling drivers, you can -vga std/qxl BUT YOU HAVE TO x-vga=off.

Last edited by Duelist (2014-10-29 12:38:58)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3086 2014-10-29 12:34:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use acs_override and i915 patch with 3.15 kernel.

edit:
yeah i just saw that he isn't using omvf...

unrelated (my question):

when u specify  <qemu:arg value='if=pflash,format=raw,readonly,file=/root/OVMF-pure-efi.fd'/> in xml.

Does that mean that i am 100% using efi, or?

Last edited by slis (2014-10-29 12:44:23)

Offline

#3087 2014-10-29 14:37:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

unrelated (my question):

when u specify  <qemu:arg value='if=pflash,format=raw,readonly,file=/root/OVMF-pure-efi.fd'/> in xml.

Does that mean that i am 100% using efi, or?

You will also have to connect OVMF-VARS.
Check
http://awilliam.github.io/presentations … 2014/#/4/2
this slide from alex's presentation for exact example of using OVMF split image with libvirt.

Last edited by Duelist (2014-10-29 14:38:11)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3088 2014-10-29 14:38:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
slis wrote:

I am not sure with OVMF, but maybe add -vga none, to get rid of error code 12?

Are you sure he is using OVMF? His commandline "script" doesn't do anything related. Some time ago there was even explicitly qemu's bios via -bios option.
If he's fine without OVMF - let it be. Because UEFI is tricky.

The UEFI trickiness:
1.My GPUs still have old VGA bios inside. I have a new, hybrid VGA-UEFI bios, but i'm unable to flash it hardware-wise and it appear to cause hangups on my qemu machine when using -romfile option. YMMV, but better be safe.
2.Windows. "We explicitly do not want CSM". I am not sure if you don't need to reinstall windows in order to get it working under UEFI if there was BIOS during the installation process. I guess there may be some details related to the bootloader.

When using vfio x-vga=on, you HAVE TO -vga none. Only that way you're getting output of you GPU.
When you are debugging something, like, uninstalling drivers, you can -vga std/qxl BUT YOU HAVE TO x-vga=off.

I would certainly NOT expect that you can move a BIOS guest image to EFI without doing a re-install.  These are fundamentally different boot interfaces for the guest.

Casperian, why are you using Q35?  Try 440FX.  EDIT: and if you get no VGA output, VGA arbitration is probably an issue.  Check the FAQ in the link below.

Last edited by aw (2014-10-29 14:42:35)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3089 2014-10-29 17:26:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The video for my talk is up: https://www.youtube.com/watch?v=NhZ9eIpg2nM


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3090 2014-10-29 19:30:24

kristopher004
Member
Registered: 2014-06-30
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For 3.18 what options do you guys select in kernel config to get virtual networks to work in libvirt?

Offline

#3091 2014-10-29 20:19:41

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi there! this thread is much appreciated. I am trying to passthrough my MSI GTX 980 Gaming to Windows 8.1. I followed this guide, but I get a Code 43 error. Then I found this: http://vfio.blogspot.de/2014/08/vfiovga-faq.html

In there this is stated:

Question 10:

I'm assigning an Nvidia card to a Windows guest and get a Code 43 error in device manager.

Answer:

The Nvidia driver, starting with 337.88 identifies the hypervisor and disables the driver when KVM is found.  Nvidia claims this is an unintentional bug, but has no plans to fix it.  To work around the problem, we can hide the hypervisor by adding kvm=off to the list of cpu options provided (QEMU 2.1+ required).  libvirt support for this option is currently upstream.

Note that -cpu kvm=off is not a valid incantation of the cpu parameter, a CPU model such as host, or SandyBridge must also be provided, ex: -cpu host,kvm=off.

Update: The above workaround is sufficient for drivers 337.88 and 340.52.  With 344.11 and presumably later, the Hyper-V CPUID extensions supported by KVM also trigger the Code 43 error.  Disabling these extensions appears to be sufficient to allow the 344.11 driver to work.  This includes all of the hv_* options to -cpu.  In libvirt, this includes:

     <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>

    </hyperv>

and

  <clock offset='localtime'>

    <timer name='hypervclock' present='yes'/>

  </clock>

Unfortunately removing these options will impose a performance penalty as these paravirtual interfaces are designed to improve the efficiency of virtual machines.


Now: I don't even have libvirt installed. Do I need it for Nvidia passthrough? Thanks a lot for any help! I am really excited about this project.

Offline

#3092 2014-10-29 20:37:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Now: I don't even have libvirt installed. Do I need it for Nvidia passthrough? Thanks a lot for any help! I am really excited about this project.

No, libvirt is not required, it's provided as an example, but don't forget to notice the comment about removing all the hv_* options from the -cpu parameter.  Try with just "-cpu host,kvm=off" (and make sure you're running a new enough QEMU to support the kvm=off option, QEMU 2.1 or newer).  If you're using OVMF then you need to be using qemu.git to have the Nvidia quirk turned on without x-vga=on (it will be in QEMU 2.2)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3093 2014-10-29 21:03:54

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you very much Alex! I am using qemu-git-2.2.r35796.gff0d487-1 I hope that's the right one. On my test Setup, I have no LVM, no GPT and no UEFI and I also use "-cpu-host,kvm=off", but I still have that error. This is how I start Windows 8.1:

qemu-system-x86_64 -enable-kvm -M q35 -m 8G -cpu host,kvm=off \
-smp 6,sockets=1,cores=6,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/root/windows.img,id=disk,format=raw \
-cdrom /root/windows.iso \
-usb -usbdevice host:046d:c312

is there an obvious mistake?

Offline

#3094 2014-10-29 21:05:39

Casperian
Member
Registered: 2014-10-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am using Q35 because my end goal is to have a linux based VM with access to cuda enabled GPU. Just doing it with windows for now though.

Been looking into VGA arbitration and yeah... seems like that is my problems.

Offline

#3095 2014-10-29 21:06:51

Casperian
Member
Registered: 2014-10-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you very much Alex! I am using qemu-git-2.2.r35796.gff0d487-1 I hope that's the right one. On my test Setup, I have no LVM, no GPT and no UEFI and I also use "-cpu-host,kvm=off", but I still have that error. This is how I start Windows 8.1:

qemu-system-x86_64 -enable-kvm -M q35 -m 8G -cpu host,kvm=off \
-smp 6,sockets=1,cores=6,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/root/windows.img,id=disk,format=raw \
-cdrom /root/windows.iso \
-usb -usbdevice host:046d:c312

is there an obvious mistake?

You don't seem to have -vga none, unless I am blind which is possible

Offline

#3096 2014-10-29 21:07:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you very much Alex! I am using qemu-git-2.2.r35796.gff0d487-1 I hope that's the right one. On my test Setup, I have no LVM, no GPT and no UEFI and I also use "-cpu-host,kvm=off", but I still have that error. This is how I start Windows 8.1:

qemu-system-x86_64 -enable-kvm -M q35 -m 8G -cpu host,kvm=off \
-smp 6,sockets=1,cores=6,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/root/windows.img,id=disk,format=raw \
-cdrom /root/windows.iso \
-usb -usbdevice host:046d:c312

is there an obvious mistake?

If you're using VGA then you may need host kernel patches for the VGA arbiter to work correctly.  You're also missing "-vga none".  (And I keep dis-recommending Q35, it's not needed for Windows guests)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3097 2014-10-30 06:45:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
slis wrote:

unrelated (my question):

when u specify  <qemu:arg value='if=pflash,format=raw,readonly,file=/root/OVMF-pure-efi.fd'/> in xml.

Does that mean that i am 100% using efi, or?

You will also have to connect OVMF-VARS.
Check
http://awilliam.github.io/presentations … 2014/#/4/2
this slide from alex's presentation for exact example of using OVMF split image with libvirt.

I just checked my win8 vm, i just added OVMF-pure-efi.fd, then selected boot device from "bios"/installed win8, and windows says that it's using UEFI, are OVMF-VARS needed/built? Or they are generated?

thx

Offline

#3098 2014-10-30 13:06:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

I just checked my win8 vm, i just added OVMF-pure-efi.fd, then selected boot device from "bios"/installed win8, and windows says that it's using UEFI, are OVMF-VARS needed/built? Or they are generated?

If your OVMF-pure-efi.fd is exactly 2MB, then it's a unified image that contains the code alongside the variable storage in the same image.  In this case each VM should have a writable copy of the image.  This works, but makes it difficult to upgrade the UEFI image and retain the variables.  Fortunately you likely don't care about the variables and the win8 bootloader seems to automatically regenerate them on every boot.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3099 2014-10-30 13:16:04

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Got it, thx for explanation.

Offline

#3100 2014-10-30 14:42:03

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

And I keep dis-recommending Q35, it's not needed for Windows guests

Can you write a somewhat detailed answer about the reason why Q35 is bad? Or is it covered in your speech(which i haven't watched yet)?
EDIT:Now i've watched your speech, and it doesn't solves the puzzle for me.
Okay, 440FX is just older, and it's more bug-free and stable. Is that the only reason to favor it over Q35?

Last edited by Duelist (2014-10-30 15:09:51)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3101 2014-10-30 14:59:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

And I keep dis-recommending Q35, it's not needed for Windows guests

Can you write a somewhat detailed answer about the reason why Q35 is bad? Or is it covered in your speech(which i haven't watched yet)?

Q35 isn't necessarily bad, but it adds more complication for typically no gain.  The default AHCI controller in the Q35 model has typically been horrible, both in performance and integrity.  The PCIe topology becomes a problem for anyone wanting to migrate to libvirt because libvirt support for Q35 is very restrictive, which necessitates using of <qemu:commandline> blocks, which makes libvirt unaware of the assigned devices, which causes it to not setup file permissions and memory limits, etc.  We also seem to have some unresolved Code 12 issues on Q35, for instance I cannot get a Quadro card to work on Q35 even though nvidia supports Quadro as a secondary device in a VM.  I also wouldn't be surprised if there's some relation to Q35 for the Radeon BSODs.

Maybe the better question is what do you gain from Q35?  Theoretically you gain access to extended PCI config space in the guest, but what does that get you?

There are cases with Linux guests where drivers blindly assume topology and may oops when a root port isn't present.  Those cases certainly justify using Q35 today.

Someday Q35, or it's successor, will be a better option than 440FX for Windows guests, but I don't believe that day is today.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3102 2014-10-30 19:17:24

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Awesome! thanks a lot for your help Alex! I can confirm that the brand new MSI GTX 980 Gaming has been successfully passed through Windows 8.1 with the latest NVidia drivers!

Now, I will try to install it with virtio.

One question: do you recommend the use of libvirt for performance reasons? If so, is it enough to simply install it??

Offline

#3103 2014-10-30 20:10:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Awesome! thanks a lot for your help Alex! I can confirm that the brand new MSI GTX 980 Gaming has been successfully passed through Windows 8.1 with the latest NVidia drivers!

Now, I will try to install it with virtio.

One question: do you recommend the use of libvirt for performance reasons? If so, is it enough to simply install it??

libvirt makes some aspects of performance tuning easier, especially things like vCPU pinning, network configuration, and device tuning options, but it's not required.  If you use VGA assignment you also need to deal with the lack of support for the x-vga option, which often leads people to use <qemu:commandline> blocks that hide the device assignment from libvirt and make lots of things more difficult.  I recommend instead using a wrapper script around qemu to insert the option (I provided a link to some redhat documentation that could be adapted for this a while ago).  The even better option is to use OVMF, which seems like it should be possible for you since you're running Win8.1 and have a new video card.  There are no libvirt hacks required for GPU assignment with OVMF (but you do need very new libvirt).

libvirt needs to manage the VM, so it's a bit more than simply installing it, you'll need to create/import the VM to work from libvirt.  virt-manager is a reasonable tool to get started for that, but for advanced tuning you'll likely need to edit the xml and refer to the excellent documentation on the libvirt website.  My blog below hopefully also has some useful examples.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3104 2014-10-31 01:10:19

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi @all

I have managed to passthrough my Primary Graphic Card (And the Secondary for CrossfireX).

I use a custom initramfs with a custom init script, which means that i don't know if it's important to setup this during boot.

My Computer:
CPU: Amd FX-8350
Mainboard: Asrock 970 Extreme4
Primary Graphics: PowerColor PCS+ HD7770 GHz Edition 1GB GDDR5      (vfio-pci, Windows)
Secondary Graphics: PowerColor HD7770 GHz Edition 1GB GDDR5 (V2?)  (vfio-pci, Windows)
Tertiary Graphics: PowerColor HD7750 1GB GDDR5                                     (Linux Host)
Memory: 16 GB Transcend 1333MHz Memory
Power Supply: Thermaltake Berlin 630W

I use Gentoo Linux ~amd64 with kernel 3.18.0-rc2 (Because of the OverlayFS :-), should work with 3.17.1 too) and Qemu-2.1.2 with Seabios-1.7.5 (Release)

My Kernel cmdline is:

root=/dev/mapper/root rootfstype=btrfs rw iommu=pt video=radeondrmfb:1280x1024-24@75,mtrr:3,pmipal,ywrap kvm.ignore_msrs=1 vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1b21:1042,1002:4383,1002:4391,1002:4393,1002:439c vfio-bind=0000:02:00.0,0000:02:00.1,0000:01:00.0,0000:01:00.1,0000:00:14.2,0000:00:11.0,0000:04:00.0,0000:05:00.0 hugepagesz=1GB fbcon=map:1

The vfio-bind Option is parsed by the custom init in my initramfs.

At boot i bind the devices (Like mentioned here) to the pci-stub driver, later to the vfio-pci driver. The Essential Lines from my init for this Proces are:

/sbin/vfio-bind "${vfiobond}" echo 0000:03:00.1 > /sys/bus/pci/devices/0000:03:00.1/driver/unbind # The Amd Audio device binds to pci-stub, since it has the same ids at the 7750 and the 7770, this fix this Problem. modprobe radeon # Here i Load the radeon kernel module for the framebuffer sleep 4 # Wait a little bit to be sure everything has enough time to switch echo 1 > /sys/bus/pci/drivers/vfio-pci/0000:01:00.0/remove # Here i remove the Primary adapter from the PCI bus echo 1 > /sys/bus/pci/drivers/vfio-pci/0000:01:00.1/remove echo 1 > /sys/bus/pci/rescan # Here i rescan the Pci-Bus for new devices, it find the Primary radeon and binds it automatically to the vfio-pci driver

The Last three Lines are Essential for this to work, with this config i can restart the VM, install newer Graphics Drivers, Play 3D Games (Elder Scrolls Online at the moment :-) ).

I use the rombios=/path/to/bios/file Option for the Primary Card, don't know if it is essential.

At Last here is the Qemu Startup Line from my script:


until /usr/bin/sudo /usr/bin/nice -n 10 /usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -monitor stdio -nographic -balloon none -mem-path /dev/hugepages -mem-prealloc \ -m 10240 -k de -cpu host -smp 8,sockets=1,cores=8,threads=1 -bios /usr/share/qemu/bios.bin-1.7.5 -realtime mlock=on \ -vga none -D /var/log/qemu-out.log -boot menu=on -usb -usbdevice host:046d:c517 -usbdevice host:093a:2510 \ -device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \ -device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \ -device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=2,id=root.0 \ -device ioh3420,bus=pcie.0,addr=1c.3,multifunction=on,port=2,chassis=3,id=root.1 \ -device nec-usb-xhci,multifunction=on,addr=1c.4,bus=pcie.0,id=usb3 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,romfile=/etc/qemu/vbios/Powercolor.HD7770.1024.120418.rom,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=pcie.0,multifunction=on \ -device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.6,multifunction=on \ -device vfio-pci,host=05:00.0,bus=pcie.0,addr=1c.7,multifunction=on \ -drive id=bitlocker_keys,file=/dev/loop0,if=none -device usb-storage,drive=bitlocker_keys,bus=usb3.0 \ -netdev type=tap,id=guest0,vhost=on,ifname="${IFACE}" -device virtio-net-pci,netdev=guest0,mac="${macaddr}" ${options} ; do echo "Qemu crashed with exit code $?. Respawning.." >&2 sleep 5 done

I hope it helps somebody who tries the same :-)

Last edited by Child_of_Sun (2014-11-01 00:11:24)

Offline

#3105 2014-10-31 01:14:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Child_of_Sun wrote:

I use Gentoo Linux ~amd64 with kernel 3.18.0-rc2 (Because of the OverlayFS :-), should work with 3.17.1 too) and a Patched Qemu from here: https://github.com/awilliam/qemu-vfio/releases with Seabios-1.7.5 (Release)

Please don't use random tags (or even branches) from my tree.  My tags are for pull requests to upstream and not intended to be any kind of release for general consumption.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3106 2014-10-31 07:48:13

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a dedicated monitor and keyboard and mouse for my windows guest.
i have a keyboard shortcut for starting the guest when its off.
is there a way that the host could ignore input from that mouse and keyboard except for that shortcut?
if not can you suggest other ideas so that each guest machine can be started from behind its dedicated desk and not interfere with host machine when its off?

Offline

#3107 2014-10-31 10:17:00

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

The kernel file looks like is not available anymore and when I try to patch my kernel I get some errors.

What I made:

Created a folder kernelbuild
Get into that folder
Downloaded kernel mainline: linux-3.18-rc2.tar.xz
Extract it with tar -xvJf linux-3.18-rc2.tar.xz
make mrproper (clean up (according to https://wiki.archlinux.org/index.php/Ke … ditional))
zcat /proc/config.gz > .config (to have a good .config (VIRTIO ENABLED)

Downloaded the patches from https://lkml.org/lkml/diff/2013/5/30/513/1 and https://lkml.org/lkml/diff/2014/5/9/517/1

Saved first one as i915.patch and i915.2.patch the second one.

Then the errors:

[root@arch linux-3.18-rc2]# patch -p1 < i915.patch patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2725 with fuzz 1 (offset 376 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej [root@arch linux-3.18-rc2]# patch -p1 < i915.2.patch patching file drivers/gpu/drm/i915/i915_dma.c Hunk #1 succeeded at 1318 (offset 12 lines). Hunk #2 succeeded at 1379 with fuzz 2. patching file drivers/gpu/drm/i915/i915_drv.h Hunk #1 FAILED at 2080. 1 out of 1 hunk FAILED -- saving rejects to file drivers/gpu/drm/i915/i915_drv.h.rej patching file drivers/gpu/drm/i915/i915_params.c Hunk #1 FAILED at 47. Hunk #2 FAILED at 152. 2 out of 2 hunks FAILED -- saving rejects to file drivers/gpu/drm/i915/i915_params.c.rej patching file drivers/gpu/drm/i915/intel_display.c Hunk #1 succeeded at 12936 (offset 1652 lines). Hunk #2 succeeded at 13301 (offset 1680 lines). Hunk #3 succeeded at 13553 (offset 1677 lines). patching file drivers/gpu/drm/i915/intel_drv.h Hunk #1 succeeded at 1125 (offset 191 lines). patching file include/linux/vgaarb.h [root@arch linux-3.18-rc2]# 

Can someone help me with that, please?

Thanks a lot.

Best regards,


TheArcher

Offline

#3108 2014-10-31 12:50:15

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Hello,

The kernel file looks like is not available anymore and when I try to patch my kernel I get some errors.

What I made:

Created a folder kernelbuild
Get into that folder
Downloaded kernel mainline: linux-3.18-rc2.tar.xz
Extract it with tar -xvJf linux-3.18-rc2.tar.xz
make mrproper (clean up (according to https://wiki.archlinux.org/index.php/Ke … ditional))
zcat /proc/config.gz > .config (to have a good .config (VIRTIO ENABLED)

Downloaded the patches from https://lkml.org/lkml/diff/2013/5/30/513/1 and https://lkml.org/lkml/diff/2014/5/9/517/1

Saved first one as i915.patch and i915.2.patch the second one.

Then the errors:

[root@arch linux-3.18-rc2]# patch -p1 < i915.patch patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2725 with fuzz 1 (offset 376 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej [root@arch linux-3.18-rc2]# patch -p1 < i915.2.patch patching file drivers/gpu/drm/i915/i915_dma.c Hunk #1 succeeded at 1318 (offset 12 lines). Hunk #2 succeeded at 1379 with fuzz 2. patching file drivers/gpu/drm/i915/i915_drv.h Hunk #1 FAILED at 2080. 1 out of 1 hunk FAILED -- saving rejects to file drivers/gpu/drm/i915/i915_drv.h.rej patching file drivers/gpu/drm/i915/i915_params.c Hunk #1 FAILED at 47. Hunk #2 FAILED at 152. 2 out of 2 hunks FAILED -- saving rejects to file drivers/gpu/drm/i915/i915_params.c.rej patching file drivers/gpu/drm/i915/intel_display.c Hunk #1 succeeded at 12936 (offset 1652 lines). Hunk #2 succeeded at 13301 (offset 1680 lines). Hunk #3 succeeded at 13553 (offset 1677 lines). patching file drivers/gpu/drm/i915/intel_drv.h Hunk #1 succeeded at 1125 (offset 191 lines). patching file include/linux/vgaarb.h [root@arch linux-3.18-rc2]# 

Can someone help me with that, please?

Thanks a lot.

Best regards,


TheArcher

This is not the arch's way of building the kernel, and for arch users, i provided a kernel pkg with all the patches you need on page 1.

EDIT: the kernel pkg should be up now

Last edited by nbhs (2014-10-31 13:48:39)

Offline

#3109 2014-10-31 13:00:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

This is not the arch's way of building the kernel, and for arch users, i provided a kernel pkg with all the patches you need on page 1.

While we appreciate the forum hosting, not everyone here runs arch.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3110 2014-10-31 13:02:31

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
nbhs wrote:

This is not the arch's way of building the kernel, and for arch users, i provided a kernel pkg with all the patches you need on page 1.

While we appreciate the forum hosting, not everyone here runs arch.

I know but it looks like he does

Last edited by nbhs (2014-10-31 13:48:06)

Offline

#3111 2014-10-31 13:04:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
aw wrote:
nbhs wrote:

This is not the arch's way of building the kernel, and for arch users, i provided a kernel pkg with all the patches you need on page 1.

While we appreciate the forum hosting, not everyone here runs arch.

I know but it looks like he does

touché


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3112 2014-10-31 14:00:51

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Child_of_Sun wrote:

Hi @all
I have managed to passthrough my Primary Graphic Card (And the Secondary for CrossfireX).

...post full of usefullness

I hope it helps somebody who tries the same :-)

Maaaan, i wanted to talk to you a while ago, and now you're posting the whole setup. I just love that post of yours.
But there's few strange things...
Do you really need that romfile option? You're running BIOS-setup, so you won't need to pass some new, UEFI-compatible ROM image to get it working.

Do you really-really-really need that "allow_unsafe_interrupts=1" option? aw mentioned it's needed in somewhat rare case.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3113 2014-10-31 14:12:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Do you really-really-really need that "allow_unsafe_interrupts=1" option? aw mentioned it's needed in somewhat rare case.

All AMD-Vi IOMMUs support interrupt remapping with CONFIG_IRQ_REMAP enabled, so no, they don't need it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3114 2014-10-31 14:16:56

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
thearcherblog wrote:

Hello,

The kernel file looks like is not available anymore and when I try to patch my kernel I get some errors.

What I made:

Created a folder kernelbuild
Get into that folder
Downloaded kernel mainline: linux-3.18-rc2.tar.xz
Extract it with tar -xvJf linux-3.18-rc2.tar.xz
make mrproper (clean up (according to https://wiki.archlinux.org/index.php/Ke … ditional))
zcat /proc/config.gz > .config (to have a good .config (VIRTIO ENABLED)

Downloaded the patches from https://lkml.org/lkml/diff/2013/5/30/513/1 and https://lkml.org/lkml/diff/2014/5/9/517/1

Saved first one as i915.patch and i915.2.patch the second one.

Then the errors:

[root@arch linux-3.18-rc2]# patch -p1 < i915.patch patching file Documentation/kernel-parameters.txt Hunk #1 succeeded at 2725 with fuzz 1 (offset 376 lines). patching file drivers/pci/quirks.c Hunk #1 FAILED at 3292. 1 out of 1 hunk FAILED -- saving rejects to file drivers/pci/quirks.c.rej [root@arch linux-3.18-rc2]# patch -p1 < i915.2.patch patching file drivers/gpu/drm/i915/i915_dma.c Hunk #1 succeeded at 1318 (offset 12 lines). Hunk #2 succeeded at 1379 with fuzz 2. patching file drivers/gpu/drm/i915/i915_drv.h Hunk #1 FAILED at 2080. 1 out of 1 hunk FAILED -- saving rejects to file drivers/gpu/drm/i915/i915_drv.h.rej patching file drivers/gpu/drm/i915/i915_params.c Hunk #1 FAILED at 47. Hunk #2 FAILED at 152. 2 out of 2 hunks FAILED -- saving rejects to file drivers/gpu/drm/i915/i915_params.c.rej patching file drivers/gpu/drm/i915/intel_display.c Hunk #1 succeeded at 12936 (offset 1652 lines). Hunk #2 succeeded at 13301 (offset 1680 lines). Hunk #3 succeeded at 13553 (offset 1677 lines). patching file drivers/gpu/drm/i915/intel_drv.h Hunk #1 succeeded at 1125 (offset 191 lines). patching file include/linux/vgaarb.h [root@arch linux-3.18-rc2]# 

Can someone help me with that, please?

Thanks a lot.

Best regards,


TheArcher

This is not the arch's way of building the kernel, and for arch users, i provided a kernel pkg with all the patches you need on page 1.

EDIT: the kernel pkg should be up now

Thanks a lot sir!!!

Sorry, I was trying to follow the right way and I was taking the information from the official guide.

One question: Can be done on my kernel? An in case that can be done... can you show me the right way to do it?

Thanks a lot!!!

Regards,


TheArcher

Offline

#3115 2014-10-31 14:37:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

20-30 pages back someone explained it, so look for it....

Offline

#3116 2014-10-31 14:44:45

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

20-30 pages back someone explained it, so look for it....

Truth: https://bbs.archlinux.org/viewtopic.php?id=162768&p=99

I will try later today smile

By the way... this is difficult now smile Maybe would be a good idea to edit post 1 an update relevant information for other users like an appendix smile

Anyway... thanks a lot smile


Regards,



TheArcher

Offline

#3117 2014-10-31 15:03:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
slis wrote:

20-30 pages back someone explained it, so look for it....

Truth: https://bbs.archlinux.org/viewtopic.php?id=162768&p=99

I will try later today smile

By the way... this is difficult now smile Maybe would be a good idea to edit post 1 an update relevant information for other users like an appendix smile

Anyway... thanks a lot smile


Regards,



TheArcher

tar xvzf linux-mainline-3.17.2.tar.gz cd linux-mainline makepkg -s sudo pacman -U linux-mainline-*

Offline

#3118 2014-10-31 15:39:08

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot sir smile

I will let you all know how it goes this weekend!!!

Thanks for your help smile

Offline

#3119 2014-10-31 16:37:05

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strange fact: i need to enable that
http://technet.microsoft.com/en-us/libr … 78411.aspx
registry parameter to stop my VM's keyboard lagging. When i press more than two buttons in sequence, system is having small freeze-and-go cycle.

I am using Q35 and i have -device qxl to get me a surface in qemu's graphical window, where qemu could grab host input.
I've tried using synergy but it doesn't work without a client and it doesn't get input working when windows displays you another context(like UAC).

But that is a minor problem. I typed that post while having my freshly-installed WIN7 VM without drivers installed, and got BSOD IRQL_NOT_LESS_OR_EQUAL. That was funny. I love windowns. Dmesg reports the usual io-page-faults on hdmi audio device.

EDIT: Now i'm using 440fx, and the problem is still present - bsod 116 after drivers install, with or without HDMI-audio. Oh well, back to the flashing ROM quest.

EDIT: -realtime mlock=on seemed to help address IO-PAGE-FAULTs. Benchmarking to test... Second GPU is still code 12.

EDIT: THAT. WAS. THE END! IT FINALLY WORKS! CHILD_OF_SUN, YOU ARE TRULY GODLIKE FOR YOUR SETUP, MANY THANKS!

I've got my HD7750 working on F2A55, all asus. Crossfire left. IO_PAGE_FAULTs are fixed by -realtime mlock=on switch.

Last edited by Duelist (2014-10-31 18:34:17)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3120 2014-10-31 18:51:17

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

...
EDIT: THAT. WAS. THE END! IT FINALLY WORKS! CHILD_OF_SUN, YOU ARE TRULY GODLIKE FOR YOUR SETUP, MANY THANKS!
...

The hero we deserve.

Offline

#3121 2014-10-31 19:00:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Duelist wrote:

...
EDIT: THAT. WAS. THE END! IT FINALLY WORKS! CHILD_OF_SUN, YOU ARE TRULY GODLIKE FOR YOUR SETUP, MANY THANKS!
...

The hero we deserve.

Well, i've been digging this system for two months, spending last two-three weeks on the quest of getting OVMF working. But he came and enlightened me.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3122 2014-10-31 19:05:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:
Duelist wrote:

...
EDIT: THAT. WAS. THE END! IT FINALLY WORKS! CHILD_OF_SUN, YOU ARE TRULY GODLIKE FOR YOUR SETUP, MANY THANKS!
...

The hero we deserve.

Well, i've been digging this system for two months, spending last two-three weeks on the quest of getting OVMF working. But he came and enlightened me.

I wasn't being sarcastic, this is actually the first time i see someone getting this working.

Offline

#3123 2014-10-31 19:09:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Duelist wrote:
nbhs wrote:

The hero we deserve.

Well, i've been digging this system for two months, spending last two-three weeks on the quest of getting OVMF working. But he came and enlightened me.

I wasn't being sarcastic, this is actually the first time i see someone getting this working.

You mean crossfire? There are some entries in the google-doc list. Child_Of_Sun is one of the three. I hope i may be the fourth guy with crossfire. But not yet - second gets code12.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3124 2014-10-31 19:13:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I've got my HD7750 working on F2A55, all asus. Crossfire left. IO_PAGE_FAULTs are fixed by -realtime mlock=on switch.

This would be surprising, mlock=on should do nothing when a device is assigned.  The memory is already pinned for the IOMMU, so mlock'ing it is redundant.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3125 2014-10-31 19:17:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

I've got my HD7750 working on F2A55, all asus. Crossfire left. IO_PAGE_FAULTs are fixed by -realtime mlock=on switch.

This would be surprising, mlock=on should do nothing when a device is assigned.  The memory is already pinned for the IOMMU, so mlock'ing it is redundant.

Well, there may be some quirks. But the io page faults were pointing into system ram area of /proc/iomem. And also, may be not the mlock, but allow-unsafe-interrupt thing did the job, that's to pin down for me. Definitely, it's mlock that does the trick. When i remove it - it gets BSOD 116. Or maybe i'm just hell of extremely lucky, getting memory locked in a ~safe area and not messing up anything(but HD7750 needs at least 256M as noted by lspci). Not a case either - i've survived a reboot.

UPD: putting two gpus on one root port was an interesting idea - i've got a message in DMESG about msi\msi-X mapping for the second gpu, the screen flashed a bit(messy guest drivers) and i've got bsod 7E. Changed the model back to -M pc btw, i observe no difference yet.

UPD2:Now something bizarre and exotic! My screen has VGA and DVI inputs. They can be used concurelly. !!FUN!! part is i've connected both inputs to both gpus. so my DVI output is from GPU1 and vga(via dvi-vga adapter) is from GPU2. When i turn x-vga=on on both cards sitting on one pci-e root port, i've got VGA output. When i enable only the first one, i get DVI output. That means the second card is actually working and initialised first. But in any way of interconnection, i get code 10 on the second one(which is showing me output) in devmgmt.msc. I think a driver install would do something fun. But the VGA output isn't showing me all 1920x1080, just 1440x1050.
After installing the drivers, i've got something showed up on both outputs at once(like, loading windowns on one gpu, desktop-ey loading thing on the other), and then... *gasps* my pc turned itself off. MCE? Dmesg -w was clear that moment. Glitches in virtual machine(talking to the real hardware, though) got the host shutdown itself?..

P.S. sorry for the "live" posting and many edits.

Last edited by Duelist (2014-10-31 20:29:29)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3126 2014-10-31 19:48:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:
Duelist wrote:

I've got my HD7750 working on F2A55, all asus. Crossfire left. IO_PAGE_FAULTs are fixed by -realtime mlock=on switch.

This would be surprising, mlock=on should do nothing when a device is assigned.  The memory is already pinned for the IOMMU, so mlock'ing it is redundant.

Well, there may be some quirks. But the io page faults were pointing into system ram area of /proc/iomem. And also, may be not the mlock, but allow-unsafe-interrupt thing did the job, that's to pin down for me. Definitely, it's mlock that does the trick. When i remove it - it gets BSOD 116. Or maybe i'm just hell of extremely lucky, getting memory locked in a ~safe area and not messing up anything(but HD7750 needs at least 256M as noted by lspci). Not a case either - i've survived a reboot.

Hmm, does it also work if you use hugepages?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3127 2014-10-31 20:32:08

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hmm, does it also work if you use hugepages?

I've assigned 2048 2048kb hugepages, and it spew out IO_PAGE_FAULTs at me, resulting in BSOD 116. No, it doesn't work with hugepages.
Also, dmesg is showing "kvm: zapping shadow pages for mmio generation wraparound".

I've found the difference between q35 and pc(440fx)! When using q35, only one GPU got MSI/MSI-X interrupt enabled. When using pc - both GPUs have their msi enabled. And that fact results in MANY variants of stuff. I can now repeat that interrupt-mess-and-emergency-shutdown behavior. Awesome.

Last edited by Duelist (2014-10-31 21:20:21)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3128 2014-11-01 00:12:28

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Child_of_Sun wrote:

I use Gentoo Linux ~amd64 with kernel 3.18.0-rc2 (Because of the OverlayFS :-), should work with 3.17.1 too) and a Patched Qemu from here: https://github.com/awilliam/qemu-vfio/releases with Seabios-1.7.5 (Release)

Please don't use random tags (or even branches) from my tree.  My tags are for pull requests to upstream and not intended to be any kind of release for general consumption.

Sorry for that, tested it now with qemu-2.1.2 and it works, too.

Offline

#3129 2014-11-01 12:58:51

Renfast
Member
Registered: 2013-03-04
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm having problems passing-through a Gigabyte GTX680. When I launch qemu the host starts to lag and after about 5 seconds the system freezes, having to reset.

But if I use a Sapphire 7970 it works fine.

I've tested both graphics cards in two setups, crashing in both and always using the iGPU as the host:

PC 1:
- Gigabyte G1 Sniper A88X
- AMD A10-6800K

PC 2:
- Asus A88X-Pro
- AMD A10-7850K

IOMMU is enabled in both motherboards. The sapphire card works in both, the GTX680 crashes in both.

The qemu command I use is this:

qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 1024 \ -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

And the cmdline:

root=/dev/sda1 rw iommu=pt iommu=1 amd_iommu=on kvm.ignore_msrs=1 ivrs_ioapic[0]=00:14.0 pci-stub.ids=1002:6798,1002:aaa0 pcie_acs_override=downstream initrd=../initramfs-linux-mainline.img

The ivrs_ioapic[0]=00:14.0 is only required in the first setup or else the interrupt remapping can't be enabled.

I don't know if I have to add or remove any other argument but this works for the Sapphire 7970.

Of course the pci-stub.ids are changed to properly match the connected graphic card.

In the BIOS the integrated GPU is forced and I am also blacklisting nvidia and nouveau (although those drivers aren't installed, the only one installed is xf86-video-ati).

I'm using Arch Linux and the kernel used is the one provided by the OP yesterday, which I compiled and installed.


I can't seem to find out what is causing these freezes, am I missing something?


EDIT: I just realized I changed the kernel arguments so many times that now I have the iommu argument duplicated, but I don't think that's the main cause.

Last edited by Renfast (2014-11-01 13:16:29)

Offline

#3130 2014-11-01 17:40:45

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
4kGamer wrote:

Awesome! thanks a lot for your help Alex! I can confirm that the brand new MSI GTX 980 Gaming has been successfully passed through Windows 8.1 with the latest NVidia drivers!

Now, I will try to install it with virtio.

One question: do you recommend the use of libvirt for performance reasons? If so, is it enough to simply install it??

libvirt makes some aspects of performance tuning easier, especially things like vCPU pinning, network configuration, and device tuning options, but it's not required.  If you use VGA assignment you also need to deal with the lack of support for the x-vga option, which often leads people to use <qemu:commandline> blocks that hide the device assignment from libvirt and make lots of things more difficult.  I recommend instead using a wrapper script around qemu to insert the option (I provided a link to some redhat documentation that could be adapted for this a while ago).  The even better option is to use OVMF, which seems like it should be possible for you since you're running Win8.1 and have a new video card.  There are no libvirt hacks required for GPU assignment with OVMF (but you do need very new libvirt).

libvirt needs to manage the VM, so it's a bit more than simply installing it, you'll need to create/import the VM to work from libvirt.  virt-manager is a reasonable tool to get started for that, but for advanced tuning you'll likely need to edit the xml and refer to the excellent documentation on the libvirt website.  My blog below hopefully also has some useful examples.


Hi, thanks a lot for your help. I want to do GPU assignment with OVMF. However I have a few questions left. You will see my questions are pretty elementary, so please bear with me please:

1. For OVMF: will I still need linux-mainline-3.17.2.tar.gz provided by nbhs? I do have an Intel Mainboard. And to nbhs: thank you very much for this thread and for your work!
2. Do I need to enable something in kernel configuration or is it enough to simply hit exit in the wizard and let it compile?
3. In your blog you wrote:

Ok, you're still reading, let's get started.  First you need an OVMF binary.  You can build this from source using the TianoCore EDK2 tree, but it is a massive pain.  Therefore, I recommend using a pre-built binary, like the one Gerd Hoffmann provides.  With Gerd's repo setup (or one appropriate to your distribution), you can install the edk2.git-ovmf-x64 package, which gives us the OVMF-pure-efi.fd OVMF image.

As I use arch where do I get the pure OVMF Image?
There is a ovmf-svn package in AUR but I couldn't install it. It breaks.

thanks a lot for helping me!

Offline

#3131 2014-11-01 19:30:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:
aw wrote:
4kGamer wrote:

Awesome! thanks a lot for your help Alex! I can confirm that the brand new MSI GTX 980 Gaming has been successfully passed through Windows 8.1 with the latest NVidia drivers!

Now, I will try to install it with virtio.

One question: do you recommend the use of libvirt for performance reasons? If so, is it enough to simply install it??

libvirt makes some aspects of performance tuning easier, especially things like vCPU pinning, network configuration, and device tuning options, but it's not required.  If you use VGA assignment you also need to deal with the lack of support for the x-vga option, which often leads people to use <qemu:commandline> blocks that hide the device assignment from libvirt and make lots of things more difficult.  I recommend instead using a wrapper script around qemu to insert the option (I provided a link to some redhat documentation that could be adapted for this a while ago).  The even better option is to use OVMF, which seems like it should be possible for you since you're running Win8.1 and have a new video card.  There are no libvirt hacks required for GPU assignment with OVMF (but you do need very new libvirt).

libvirt needs to manage the VM, so it's a bit more than simply installing it, you'll need to create/import the VM to work from libvirt.  virt-manager is a reasonable tool to get started for that, but for advanced tuning you'll likely need to edit the xml and refer to the excellent documentation on the libvirt website.  My blog below hopefully also has some useful examples.


Hi, thanks a lot for your help. I want to do GPU assignment with OVMF. However I have a few questions left. You will see my questions are pretty elementary, so please bear with me please:

1. For OVMF: will I still need linux-mainline-3.17.2.tar.gz provided by nbhs? I do have an Intel Mainboard. And to nbhs: thank you very much for this thread and for your work!
2. Do I need to enable something in kernel configuration or is it enough to simply hit exit in the wizard and let it compile?
3. In your blog you wrote:

Ok, you're still reading, let's get started.  First you need an OVMF binary.  You can build this from source using the TianoCore EDK2 tree, but it is a massive pain.  Therefore, I recommend using a pre-built binary, like the one Gerd Hoffmann provides.  With Gerd's repo setup (or one appropriate to your distribution), you can install the edk2.git-ovmf-x64 package, which gives us the OVMF-pure-efi.fd OVMF image.

As I use arch where do I get the pure OVMF Image?
There is a ovmf-svn package in AUR but I couldn't install it. It breaks.

thanks a lot for helping me!

OVMF-26830e8.tar.gz built from git

Last edited by nbhs (2014-11-01 19:31:26)

Offline

#3132 2014-11-01 19:37:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

1. For OVMF: will I still need linux-mainline-3.17.2.tar.gz provided by nbhs? I do have an Intel Mainboard. And to nbhs: thank you very much for this thread and for your work!

Only if you need the pcie_acs_override= option

2. Do I need to enable something in kernel configuration or is it enough to simply hit exit in the wizard and let it compile?

I would guess the default config for your distro is ok so long as the VFIO options are enabled (VFIO VGA is no longer a requirement)

3. In your blog you wrote:

Ok, you're still reading, let's get started.  First you need an OVMF binary.  You can build this from source using the TianoCore EDK2 tree, but it is a massive pain.  Therefore, I recommend using a pre-built binary, like the one Gerd Hoffmann provides.  With Gerd's repo setup (or one appropriate to your distribution), you can install the edk2.git-ovmf-x64 package, which gives us the OVMF-pure-efi.fd OVMF image.

As I use arch where do I get the pure OVMF Image?
There is a ovmf-svn package in AUR but I couldn't install it. It breaks.

Looks like nbhs already linked you up here


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3133 2014-11-01 21:14:45

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you guys! I extracted OVMF-26830e8.tar.gz and I got three files: OVMF_CODE.fd  OVMF.fd  OVMF_VARS.fd

can I use thse for libvirt too? And do I have to rename them? Is the OVMF.fd file the "pure-efi" one?

Offline

#3134 2014-11-01 21:22:26

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you guys! I extracted OVMF-26830e8.tar.gz and I got three files: OVMF_CODE.fd  OVMF.fd  OVMF_VARS.fd

can I use thse for libvirt too? And do I have to rename them? Is the OVMF.fd file the "pure-efi" one?

----> http://vfio.blogspot.com/

If by pure efi you mean without CSM, then yes

Last edited by nbhs (2014-11-01 21:27:40)

Offline

#3135 2014-11-02 03:37:07

zopilote
Member
Registered: 2014-11-02
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

When using q35, only one GPU got MSI/MSI-X interrupt enabled. When using pc - both GPUs have their msi enabled. And that fact results in MANY variants of stuff.

Hi, I'm sorry but this seems very important to point, so, if I have two virtual machines (wn8 each one with its own gpu passed through), with q35, only one of them will have its gpu with msi enabled?

Offline

#3136 2014-11-02 03:42:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zopilote wrote:
aw wrote:

When using q35, only one GPU got MSI/MSI-X interrupt enabled. When using pc - both GPUs have their msi enabled. And that fact results in MANY variants of stuff.

Hi, I'm sorry but this seems very important to point, so, if I have two virtual machines (wn8 each one with its own gpu passed through), with q35, only one of them will have its gpu with msi enabled?

I never said this, misquote.  Guests are independent of each other, more than one can use MSI at the same time.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3137 2014-11-02 03:42:59

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

do you need to be booting via efi or can you be booting via the old bios method on the host? does it matter?

Offline

#3138 2014-11-02 03:50:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

do you need to be booting via efi or can you be booting via the old bios method on the host? does it matter?

The host using EFI/BIOS is independent of the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3139 2014-11-02 22:35:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Unbelievable!
I refuse to believe that, but seems like...
...after a full guest reinstall...
...and a boot started with BSOD f4...
I'VE GOT CROSSFIREX WORKING!

The setup:
Hardware(yet again..):
MB:ASUS F2A55
CPU:Athlon X4 750K
RAM:Some generic AMD(HYNIX) 8GB.
GPUs:ASUS HD7750+HD7750(DCSL, passively cooled), GT610 silent(host, connected using pci-e x1-x16 riser into first x1 slot)
Cables connected:
GPU1->DVI
GPU2->X
(I could've connect GPU2->VGA on the same physical screen, and it would go nuts)

Versions:
BIOS(somewhat important):6501(first to support IOMMU)
Fedora 21(rawhide or what was it called?)
Kernel:3.17.1-304.fc21.x86_64
QEMU:2.1.2
Seabios:1.7.5
Host drivers:Nvidia 343.22 x64
Guest drivers:Catalyst 14.9 x64
Guest OS: Windows 7 x64.

Kernel parameters:
nofb pci-stub.ids=1002:683f,1002:aab0,1002:683f,1002:683f enable_mtrr_cleanup

/etc/modprobe.d/kvm-amd.conf:
options kvm-amd npt=1 nested=1
options kvm ignore_msrs=0
options vfio_iommu_type1=allow_unsafe_interrupts=1
#(maybe i should drop that last line as it shouldn't be needed)

Startup script:

#!/bin/bash QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa SDL_VIDEO_X11_DGAMOUSE=0 sudo qemu-system-x86_64 \ -boot menu=on \ -realtime mlock=on \ -mem-prealloc \ -enable-kvm \ -monitor stdio \ -M pc \ -m 4096 \ -cpu host \ -net none \ -rtc base=localtime \ -smp 4,sockets=1,cores=4,threads=1 \ -netdev tap,ifname=tap0,id=tap0 \ -drive file='/mnt/hdd/qemu/qemu-vfio-win.img',id=disk,format=raw,cache=none,if=none \ -drive file='/mnt/hdd/qemu/virtio.iso',id=cdrom,format=raw,readonly=on,if=none \ -drive file='/mnt/hdd/qemu/windows7.iso',id=cdrom2,format=raw,readonly=on,if=none \ -device ioh3420,addr=02.0,multifunction=on,port=1,chassis=1,id=root.0 \ -device ioh3420,addr=04.0,multifunction=on,port=1,chassis=2,id=root.1 \ -device qxl,bus=root.1,addr=02.0 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.0,addr=00.1 \ -device vfio-pci,host=02:00.0,bus=root.0,addr=01.0,multifunction=on,x-vga=off \ -device vfio-pci,host=02:00.1,bus=root.0,addr=01.1 \ -device virtio-blk-pci,bus=root.1,addr=03.0,drive=disk \ -device virtio-scsi-pci,bus=root.1,addr=04.0 \ -device ide-cd,bus=ide.1,drive=cdrom \ -device scsi-cd,drive=cdrom2 \ -device ich9-intel-hda,bus=root.1,addr=00.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device virtio-net-pci,netdev=tap0,bus=root.1,addr=01.0 \ -vga none

(Note ONE root port per two GPUs).

dmesg:

[12457.367755] vfio-pci 0000:01:00.1: Refused to change power state, currently in D3 [12457.367933] vfio_pci_disable: Failed to reset device 0000:01:00.1 (-22) [12457.378727] vfio-pci 0000:01:00.1: Refused to change power state, currently in D3 [12458.418190] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3 [12458.418367] vfio_pci_disable: Failed to reset device 0000:02:00.1 (-22) [12458.429126] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3 #A guest reboot happened here ^^ [12489.677504] device tap0 entered promiscuous mode [12489.682172] virbr0: port 2(tap0) entered listening state [12489.682182] virbr0: port 2(tap0) entered listening state [12491.681907] virbr0: port 2(tap0) entered learning state [12491.800412] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403) [12491.822020] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [12491.826594] vfio-pci 0000:01:00.1: enabling device (0400 -> 0402) [12491.851818] vfio-pci 0000:02:00.0: enabling device (0400 -> 0403) [12491.874107] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 [12491.878852] vfio-pci 0000:02:00.1: enabling device (0400 -> 0402) [12493.684777] virbr0: topology change detected, propagating [12493.684799] virbr0: port 2(tap0) entered forwarding state [12507.936987] kvm: zapping shadow pages for mmio generation wraparound [12517.733221] vfio-pci 0000:01:00.0: irq 40 for MSI/MSI-X [12517.936020] vfio-pci 0000:02:00.0: irq 41 for MSI/MSI-X #^^First boot messages without crossfire [12586.636623] vfio-pci 0000:01:00.0: irq 40 for MSI/MSI-X [12586.842463] vfio-pci 0000:02:00.0: irq 41 for MSI/MSI-X [12853.980234] vfio-pci 0000:02:00.0: irq 41 for MSI/MSI-X [12854.211315] vfio-pci 0000:01:00.0: irq 40 for MSI/MSI-X #^^Turning crossfire on, having screen blinking as usual.

/proc/:

mtrr:

reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back reg01: base=0x080000000 ( 2048MB), size= 512MB, count=1: write-back reg02: base=0x0a0000000 ( 2560MB), size= 256MB, count=1: write-back reg03: base=0x0af000000 ( 2800MB), size= 16MB, count=1: uncachable

iomem:

00000000-00000fff : reserved 00001000-0009e7ff : System RAM 0009e800-0009ffff : reserved 000a0000-000bffff : PCI Bus 0000:00 000c0000-000dffff : PCI Bus 0000:00 000c0000-000cffff : Video ROM 000e0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-ad518fff : System RAM 01000000-0174a014 : Kernel code 0174a015-01d33c7f : Kernel data 01eac000-0202cfff : Kernel bss ad519000-ad6a4fff : reserved ad6a5000-ad6b4fff : ACPI Tables ad6b5000-ada13fff : ACPI Non-volatile Storage ada14000-ae626fff : reserved ae627000-ae627fff : System RAM ae628000-ae82dfff : ACPI Non-volatile Storage ae82e000-aec5efff : System RAM aec5f000-aeff3fff : reserved aeff4000-aeffffff : System RAM af000000-afffffff : RAM buffer b0000000-ffffffff : PCI Bus 0000:00 b0000000-bfffffff : PCI Bus 0000:02 b0000000-bfffffff : 0000:02:00.0 b0000000-bfffffff : vfio-pci c0000000-cfffffff : PCI Bus 0000:01 c0000000-cfffffff : 0000:01:00.0 c0000000-cfffffff : vfio-pci d0000000-d9ffffff : PCI Bus 0000:04 d0000000-d7ffffff : 0000:04:00.0 d8000000-d9ffffff : 0000:04:00.0 da100000-da1fffff : PCI Bus 0000:06 da100000-da103fff : 0000:06:00.0 da100000-da103fff : r8169 da104000-da104fff : 0000:06:00.0 da104000-da104fff : r8169 e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff] e0000000-efffffff : pnp 00:00 fd000000-fe0fffff : PCI Bus 0000:04 fd000000-fdffffff : 0000:04:00.0 fd000000-fdffffff : nvidia fe000000-fe07ffff : 0000:04:00.0 fe080000-fe083fff : 0000:04:00.1 fe080000-fe083fff : ICH HD audio fe100000-fe1fffff : PCI Bus 0000:05 fe100000-fe107fff : 0000:05:00.0 fe100000-fe107fff : xhci_hcd fe200000-fe2fffff : PCI Bus 0000:02 fe200000-fe23ffff : 0000:02:00.0 fe200000-fe23ffff : vfio-pci fe240000-fe25ffff : 0000:02:00.0 fe260000-fe263fff : 0000:02:00.1 fe260000-fe263fff : vfio-pci fe300000-fe3fffff : PCI Bus 0000:01 fe300000-fe33ffff : 0000:01:00.0 fe300000-fe33ffff : vfio-pci fe340000-fe35ffff : 0000:01:00.0 fe360000-fe363fff : 0000:01:00.1 fe360000-fe363fff : vfio-pci fe400000-fe403fff : 0000:00:14.2 fe400000-fe403fff : ICH HD audio fe404000-fe4040ff : 0000:00:16.2 fe404000-fe4040ff : ehci_hcd fe405000-fe405fff : 0000:00:16.0 fe405000-fe405fff : ohci_hcd fe406000-fe4060ff : 0000:00:13.2 fe406000-fe4060ff : ehci_hcd fe407000-fe407fff : 0000:00:13.0 fe407000-fe407fff : ohci_hcd fe408000-fe4080ff : 0000:00:12.2 fe408000-fe4080ff : ehci_hcd fe409000-fe409fff : 0000:00:12.0 fe409000-fe409fff : ohci_hcd fe40a000-fe40a7ff : 0000:00:11.0 fe40a000-fe40a7ff : ahci feb80000-febfffff : amd_iommu fec00000-fec00fff : reserved fec00000-fec003ff : IOAPIC 0 fec10000-fec10fff : reserved fec10000-fec10fff : pnp 00:03 fed00000-fed00fff : reserved fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed61000-fed70fff : pnp 00:03 fed80000-fed8ffff : reserved fed80000-fed8ffff : pnp 00:03 fee00000-fee00fff : Local APIC fee00000-fee00fff : pnp 00:03 ff000000-ffffffff : reserved ff000000-ffffffff : pnp 00:03 100001000-24effffff : System RAM 24f000000-24fffffff : RAM buffer

interrupts

 CPU0 CPU1 CPU2 CPU3 0: 133 0 0 0 IR-IO-APIC-edge timer 1: 14205 14099 14448 15568 IR-IO-APIC-edge i8042 8: 0 1 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 IR-IO-APIC-fasteoi acpi 16: 249 301 896 452 IR-IO-APIC 16-fasteoi snd_hda_intel 17: 316 912 1305 452 IR-IO-APIC 17-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, ehci_hcd:usb3, snd_hda_intel, vfio-intx(0000:02:00.1) 18: 26813 79583 64074 22305 IR-IO-APIC 18-fasteoi ohci_hcd:usb4, ohci_hcd:usb5, ohci_hcd:usb6 19: 991 912 975 1184 IR-IO-APIC 19-fasteoi vfio-intx(0000:01:00.1) 24: 0 0 0 0 IR-PCI-MSI-edge AMD-Vi 25: 10442 160150 123 492188 IR-PCI-MSI-edge ahci0 26: 16636 16686 16897 17841 IR-PCI-MSI-edge ahci1 27: 0 0 0 0 IR-PCI-MSI-edge ahci2 28: 0 0 0 0 IR-PCI-MSI-edge ahci3 29: 0 0 0 0 IR-PCI-MSI-edge ahci4 30: 3358 3482 3569 3637 IR-PCI-MSI-edge ahci5 33: 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 34: 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 35: 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 36: 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 1043180 4 606979 6 IR-PCI-MSI-edge p34p1 39: 65 44297 63 154262 IR-PCI-MSI-edge nvidia 40: 72821 72798 73767 76769 IR-PCI-MSI-edge vfio-msi[0](0000:01:00.0) 41: 41094 41276 42254 43526 IR-PCI-MSI-edge vfio-msi[0](0000:02:00.0) NMI: 247 244 255 253 Non-maskable interrupts LOC: 3808655 3249700 4271226 3515979 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 247 244 255 253 Performance monitoring interrupts IWI: 5 0 0 2 IRQ work interrupts RTR: 0 0 0 0 APIC ICR read retries RES: 7385839 7154521 7540346 6975371 Rescheduling interrupts CAL: 2243 2057 2171 2292 Function call interrupts TLB: 36726 35621 33872 33973 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 48 48 48 48 Machine check polls THR: 0 0 0 0 Hypervisor callback interrupts

ioports:

0000-03af : PCI Bus 0000:00 0000-001f : dma1 0020-0021 : pic1 0040-0043 : timer0 0050-0053 : timer1 0060-0060 : keyboard 0061-0061 : PNP0800:00 0064-0064 : keyboard 0070-0071 : rtc0 0080-008f : dma page reg 00a0-00a1 : pic2 00c0-00df : dma2 00f0-00ff : PNP0C04:00 00f0-00ff : fpu 0290-029f : pnp 00:04 0300-031f : pnp 00:04 03b0-03df : PCI Bus 0000:00 03c0-03df : vga+ 03e0-0cf7 : PCI Bus 0000:00 03f8-03ff : serial 040b-040b : pnp 00:03 04d0-04d1 : pnp 00:03 04d0-04d1 : pnp 00:06 04d6-04d6 : pnp 00:03 0800-0803 : ACPI PM1a_EVT_BLK 0804-0805 : ACPI PM1a_CNT_BLK 0808-080b : ACPI PM_TMR 0810-0815 : ACPI CPU throttle 0820-0827 : ACPI GPE0_BLK 0900-090f : pnp 00:03 0910-091f : pnp 00:03 0b00-0b07 : piix4_smbus 0b20-0b3f : pnp 00:03 0b20-0b27 : piix4_smbus 0c00-0c01 : pnp 00:03 0c14-0c14 : pnp 00:03 0c50-0c51 : pnp 00:03 0c52-0c52 : pnp 00:03 0c6c-0c6c : pnp 00:03 0c6f-0c6f : pnp 00:03 0cd0-0cd1 : pnp 00:03 0cd2-0cd3 : pnp 00:03 0cd4-0cd5 : pnp 00:03 0cd6-0cd7 : pnp 00:03 0cd8-0cdf : pnp 00:03 0cf8-0cff : PCI conf1 0d00-ffff : PCI Bus 0000:00 b000-bfff : PCI Bus 0000:06 b000-b0ff : 0000:06:00.0 b000-b0ff : r8169 c000-cfff : PCI Bus 0000:04 c000-c07f : 0000:04:00.0 d000-dfff : PCI Bus 0000:02 d000-d0ff : 0000:02:00.0 e000-efff : PCI Bus 0000:01 e000-e0ff : 0000:01:00.0 e000-e0ff : vfio f000-f00f : 0000:00:11.0 f000-f00f : ahci f010-f013 : 0000:00:11.0 f010-f013 : ahci f020-f027 : 0000:00:11.0 f020-f027 : ahci f030-f033 : 0000:00:11.0 f030-f033 : ahci f040-f047 : 0000:00:11.0 f040-f047 : ahci fe00-fefe : pnp 00:03

sudo lspci -v

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 0459 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at fe300000 (64-bit, non-prefetchable) [size=256K] I/O ports at e000 [size=256] Expansion ROM at fe340000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Kernel driver in use: vfio-pci Kernel modules: radeon 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: ASUSTeK Computer Inc. Device aab0 Flags: bus master, fast devsel, latency 0, IRQ 19 Memory at fe360000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 0459 Flags: bus master, fast devsel, latency 0, IRQ 41 Memory at b0000000 (64-bit, prefetchable) [size=256M] Memory at fe200000 (64-bit, non-prefetchable) [size=256K] I/O ports at d000 [size=256] Expansion ROM at fe240000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [270] #19 Kernel driver in use: vfio-pci Kernel modules: radeon 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: ASUSTeK Computer Inc. Device aab0 Flags: fast devsel, IRQ 17 Memory at fe260000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 04:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 3546 Flags: bus master, fast devsel, latency 0, IRQ 39 Memory at fd000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (64-bit, prefetchable) [size=128M] Memory at d8000000 (64-bit, prefetchable) [size=32M] I/O ports at c000 [size=128] [virtual] Expansion ROM at fe000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: nvidia Kernel modules: nouveau, nvidia 04:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1) Subsystem: Gigabyte Technology Co., Ltd Device 3546 Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at fe080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel

P.S.
If only this forum had [spoiler]spoiler[/spoiler] tag.

Last edited by Duelist (2014-11-02 23:51:37)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3140 2014-11-03 01:22:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
#!/bin/bash QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa SDL_VIDEO_X11_DGAMOUSE=0 sudo qemu-system-x86_64 \ -boot menu=on \ -realtime mlock=on \ -mem-prealloc \ -enable-kvm \ -monitor stdio \ -M pc \ -m 4096 \ -cpu host \ -net none \ -rtc base=localtime \ -smp 4,sockets=1,cores=4,threads=1 \ -netdev tap,ifname=tap0,id=tap0 \ -drive file='/mnt/hdd/qemu/qemu-vfio-win.img',id=disk,format=raw,cache=none,if=none \ -drive file='/mnt/hdd/qemu/virtio.iso',id=cdrom,format=raw,readonly=on,if=none \ -drive file='/mnt/hdd/qemu/windows7.iso',id=cdrom2,format=raw,readonly=on,if=none \ -device ioh3420,addr=02.0,multifunction=on,port=1,chassis=1,id=root.0 \ -device ioh3420,addr=04.0,multifunction=on,port=1,chassis=2,id=root.1 \ -device qxl,bus=root.1,addr=02.0 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.0,addr=00.1 \ -device vfio-pci,host=02:00.0,bus=root.0,addr=01.0,multifunction=on,x-vga=off \ -device vfio-pci,host=02:00.1,bus=root.0,addr=01.1 \ -device virtio-blk-pci,bus=root.1,addr=03.0,drive=disk \ -device virtio-scsi-pci,bus=root.1,addr=04.0 \ -device ide-cd,bus=ide.1,drive=cdrom \ -device scsi-cd,drive=cdrom2 \ -device ich9-intel-hda,bus=root.1,addr=00.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device virtio-net-pci,netdev=tap0,bus=root.1,addr=01.0 \ -vga none

(Note ONE root port per two GPUs).

What a crazy config.  You're running 440FX with PCIe root ports and behind one root port you have two endpoints and while you specify -vga none you add a qxl device, so I think you've replaced the default emulated VGA device with one behind a root port.  On real hardware this is equivalent to this:

usb adapter

(EDIT: except you've managed to plug 2 USB sticks into the resulting single port)

I'm surprised that Windows scans for and finds your 2nd GPU since an OS will typically only scan device/slot 0.  Attaching a PCIe root port to a 440FX chipset is just an impossible thing on real hardware.  A pci-bridge might be a better option than the ioh3420 devices.  If it works on -M pc but not on Q35, there might be some part of the PCIe extended config space that we need to hide or handle differently.

Last edited by aw (2014-11-03 01:25:54)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3141 2014-11-03 14:43:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What a crazy config.  You're running 440FX with PCIe root ports and behind one root port you have two endpoints and while you specify -vga none you add a qxl device, so I think you've replaced the default emulated VGA device with one behind a root port.  On real hardware this is equivalent to this:

I'm surprised that Windows scans for and finds your 2nd GPU since an OS will typically only scan device/slot 0.  Attaching a PCIe root port to a 440FX chipset is just an impossible thing on real hardware.  A pci-bridge might be a better option than the ioh3420 devices.  If it works on -M pc but not on Q35, there might be some part of the PCIe extended config space that we need to hide or handle differently.

The qxl(not qxl-vga!) device is needed for... INPUT! QEMU creates a surface in it's GUI where it can grab mouse and keyboard. If i could force QEMU the other way without using crutches like that - it would be nice.
If the qxl device doesn't work - i don't care, i've got the surface. Actually it's disabled in windows due to drivers(code 28). It doesn't take any ioports or memory, so it's fine while there is no drivers for it.
The -vga none is needed to get rid of any devices which i do not specify, especially VGA ones. For example, -vga qxl seem to be an alias for -device qxl-vga as it looks the same from the guest side.

The second root port is needed for various devices like network. I haven't poked around 440FX yet, but i may move them to some other PCI bus.

It is impossible to force windows to enable MSI under Q35 via registry - it doesn't help and the following registry hack is applied by the driver install.

I still have a strong feeling that my motherboard has a TON of quirks everywhere. Last time i've had a true evidence of this is when i've tried to look my cpu's temperature, and it was negative in amd overdrive(sort of official) software. It took them half a year to release a fix to fully support my CPU.

UPD:
Doing stuff like

-device vfio-pci,host=01:00.0,addr=09.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,addr=09.1 \ -device vfio-pci,host=02:00.0,addr=0a.0,multifunction=on,x-vga=off \ -device vfio-pci,host=02:00.1,addr=0a.1 \

results in interrupt storm and host system shutdown.

P.S.
http://www.geek.com/chips/a-parallel-po … e-1276689/
Heh, what a precise analogy. Yes, it does work and looks alike. But it works.

Last edited by Duelist (2014-11-03 16:29:24)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3142 2014-11-04 10:40:35

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do you think SLI would work with such config ? smile

Offline

#3143 2014-11-04 11:56:39

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm wondering, how big impact on performance is running stuff like this in VM?

slis wrote:

Do you think SLI would work with such config ? smile

Even if it works, it's only for period of one driver release smile

Offline

#3144 2014-11-04 12:15:06

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

I'm wondering, how big impact on performance is running stuff like this in VM?

slis wrote:

Do you think SLI would work with such config ? smile

Even if it works, it's only for period of one driver release smile


What do u think by that? I really don't care about "newest drivers", cause I have gtx680... (running 340.82 for 2-3 months with 0 problems)

I tried running SLI couple months ago, and it failed with error code 10,12 or 43... on second card... so I am thinking givin it another go....

Offline

#3145 2014-11-04 14:08:42

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
dwe11er wrote:

I'm wondering, how big impact on performance is running stuff like this in VM?

slis wrote:

Do you think SLI would work with such config ? smile

Even if it works, it's only for period of one driver release smile


What do u think by that? I really don't care about "newest drivers", cause I have gtx680... (running 340.82 for 2-3 months with 0 problems)

I tried running SLI couple months ago, and it failed with error code 10,12 or 43... on second card... so I am thinking givin it another go....

nVidia seem to purposefully make changes to newest drivers to prevent GPU passthrough on all but most expensive (GRID, some Quadro) cards.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3146 2014-11-04 15:01:55

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

I'm wondering, how big impact on performance is running stuff like this in VM?

Well, as far as i remember - i've had 15FPS in opengl with 1 gpu in unigine heaven on linux. Same i have now in VM. I was trying to get this working so long, that i've lost my benchmark results and don't even remember them properly.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3147 2014-11-04 17:44:47

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello again !

I achieved to do VGA-Passthrough on another machine.
It is a desktop and more modern configuration but the GPU is quite old.

CPU: i5-4670T (Haswell family)
GPU: Nvidia 9500 GS

It's working fine, now going to do some tuning but there I have some questions first.

Question 1: While I could do VGA-PT with my laptop without applying any patches, I can't with this new config (black screen). Especially the vga-arbiter patch. From what I understood from this article from Alex Williamson (What's the deal with VGA arbitration ?), this is because my brand new CPU's IGP is too recent ?

Alex Williamson wrote:

The native IGD driver would really like to continue running when VGA routing is directed elsewhere, so the PCI method of disabling access is sub-optimal.  At some point Intel knew this and included mechanisms in the device that allowed VGA MMIO and I/O port space to be disabled separately from PCI MMIO and I/O port space.  These mechanisms are what the i915 driver uses today when it opts-out of VGA arbitration.

The problem is that modern IGD devices don't implement the same mechanisms for disabling legacy VGA.  The i915 driver continues to twiddle the old bits and opt-out of arbitration, but the IGD VGA is still claiming VGA transactions.  This is why many users who claim they don't need the i915 patch finish their proclamation of success with a note about the VGA color palate being corrupted or noting some strange drm interrupt messages on the host dmesg. [...]

So why can't we simply fix the i915 driver to correctly disable legacy VGA support?  Well, in their infinite wisdom, hardware designers have removed (or broken) that feature of IGD.

My laptop's CPU is intel 2nd-gen, I have some color problems whenever I shutdown the VM, but this doesn't bother me much. So, I guess the feature is still there and not broken, but is on my intel 4th-gen CPU, right ?

Question 2: My laptop's configuration needs a vbios rom in order to work, but this is not the case of my desktop's GPU. In fact, whenever I specify an extracted vbios it won't work: black screen (I extracted it myself from a live-cd, I strictly followed the same procedure than for my laptop vbios). My first thought was the GPU is a bit too old but I think it's a stupid idea. I used rom-parser to check the vbios and it says it's fine, reporting correct vendors ids. Where could it comes from ? Any suggestions ? It tickles me a lot !


Question 3: Now that I begin to understand PCI-PT in qemu, should I try to write a tutorial for french people ? Isn't it a bit risky since I don't have a wide knowledge of the internal mechanics and couldn't properly troubleshoot specific case issues ?

Well, thanks a lot again for the useful informations in this thread.

Offline

#3148 2014-11-04 18:01:33

Uramekus
Member
Registered: 2014-06-08
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

when i run

qemu-system-x86_64 \
        -enable-kvm \
        -M q35 \
        -m 1024 \
        -cpu host \
        -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin \
        -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
        -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

i get

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized


my grub cfg settings are

GRUB_CMDLINE_LINUX="i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1184,10de:0e0a  pcie_acs_override=downstream"

when i set intel_iommu=on i got a disgraceful bug (less than 5s start the screen got stalled code-like)
and i don't have idea what i can do to get this working
*edit*  how it looks
how it looks
please someone help me hmm

Last edited by Uramekus (2014-11-04 18:26:34)

Offline

#3149 2014-11-04 18:33:50

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You are missing intel_iommu=on

edit: forgot intel_ smile

Last edited by slis (2014-11-04 18:57:48)

Offline

#3150 2014-11-04 18:38:11

Uramekus
Member
Registered: 2014-06-08
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i should enable iommu=on and intel_iommu=on?
or just iommu=on?

Last edited by Uramekus (2014-11-04 18:41:54)

Offline

#3151 2014-11-04 18:39:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Uramekus wrote:

when i set intel_iommu=on i got a disgraceful bug (less than 5s start the screen got stalled code-like)
and i don't have idea what i can do to get this working

If you can't enable the IOMMU, you can't do PCI device assignment, end of story.  If you provide us a snapshot of some of the errors you're seeing when you enable it, maybe we can help.  You might also try a newer kernel to see if it can be enabled there, you haven't specified your host environment.

EDIT: if that screenshot is what happens when you try to boot with intel_iommu=on, try intel_iommu=on,igfx_off

Last edited by aw (2014-11-04 18:53:30)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3152 2014-11-04 18:45:44

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit: too late big_smile

Last edited by slis (2014-11-04 18:46:08)

Offline

#3153 2014-11-04 18:45:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mouton_sanglant wrote:

Hello again !

I achieved to do VGA-Passthrough on another machine.
It is a desktop and more modern configuration but the GPU is quite old.

CPU: i5-4670T (Haswell family)
GPU: Nvidia 9500 GS

It's working fine, now going to do some tuning but there I have some questions first.

Question 1: While I could do VGA-PT with my laptop without applying any patches, I can't with this new config (black screen). Especially the vga-arbiter patch. From what I understood from this article from Alex Williamson (What's the deal with VGA arbitration ?), this is because my brand new CPU's IGP is too recent ?

Alex Williamson wrote:

The native IGD driver would really like to continue running when VGA routing is directed elsewhere, so the PCI method of disabling access is sub-optimal.  At some point Intel knew this and included mechanisms in the device that allowed VGA MMIO and I/O port space to be disabled separately from PCI MMIO and I/O port space.  These mechanisms are what the i915 driver uses today when it opts-out of VGA arbitration.

The problem is that modern IGD devices don't implement the same mechanisms for disabling legacy VGA.  The i915 driver continues to twiddle the old bits and opt-out of arbitration, but the IGD VGA is still claiming VGA transactions.  This is why many users who claim they don't need the i915 patch finish their proclamation of success with a note about the VGA color palate being corrupted or noting some strange drm interrupt messages on the host dmesg. [...]

So why can't we simply fix the i915 driver to correctly disable legacy VGA support?  Well, in their infinite wisdom, hardware designers have removed (or broken) that feature of IGD.

My laptop's CPU is intel 2nd-gen, I have some color problems whenever I shutdown the VM, but this doesn't bother me much. So, I guess the feature is still there and not broken, but is on my intel 4th-gen CPU, right ?

The only IGDs that work with VGA arbitration are the ones were the GPU is on the chipset, not the CPU.  If you saw screen corruption on your laptop, even if it didn't bother you, it's the same problem.  It's probably more a difference of the GPU being assigned and whether the guest drivers are able to initialize it despite not having VGA access to it.

Question 2: My laptop's configuration needs a vbios rom in order to work, but this is not the case of my desktop's GPU. In fact, whenever I specify an extracted vbios it won't work: black screen (I extracted it myself from a live-cd, I strictly followed the same procedure than for my laptop vbios). My first thought was the GPU is a bit too old but I think it's a stupid idea. I used rom-parser to check the vbios and it says it's fine, reporting correct vendors ids. Where could it comes from ? Any suggestions ? It tickles me a lot !

Was the GPU the primary graphics device when you extracted the BIOS?  PCs will load primary graphics BIOS into a different region and it may be modified runtime.  If that's the one you get when extracted, it may not work correctly.  If QEMU/VFIO is able to get the correct ROM from the card, why do you care?

Question 3: Now that I begin to understand PCI-PT in qemu, should I try to write a tutorial for french people ? Isn't it a bit risky since I don't have a wide knowledge of the internal mechanics and couldn't properly troubleshoot specific case issues ?

Those of us who don't speak french won't be able to identify any mistakes, but more helpful documentation is always a good thing.

Last edited by aw (2014-11-04 18:47:38)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3154 2014-11-04 18:46:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[sorry, meant to edit, not re-post]

Last edited by aw (2014-11-04 18:47:10)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3155 2014-11-04 19:09:21

Uramekus
Member
Registered: 2014-06-08
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

You are missing intel_iommu=on
edit: forgot intel_

when i enable intel_iommu=on i got the image result.


*edit*
tried with intel_iommu=on,igfx_off
accly the same.

*edit2*
i'm recompiling git kernel disabling some optimizations to see how it does.
gonna post here if got problems.

*final edit*
got all working with less optimzed code, it was gcc fault

Last edited by Uramekus (2014-11-04 19:51:28)

Offline

#3156 2014-11-05 15:15:16

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mouton_sanglant wrote:

Hello again !

I achieved to do VGA-Passthrough on another machine.
It is a desktop and more modern configuration but the GPU is quite old.

CPU: i5-4670T (Haswell family)
GPU: Nvidia 9500 GS

It's working fine, now going to do some tuning but there I have some questions first.

Question 1: While I could do VGA-PT with my laptop without applying any patches, I can't with this new config (black screen). Especially the vga-arbiter patch. From what I understood from this article from Alex Williamson (What's the deal with VGA arbitration ?), this is because my brand new CPU's IGP is too recent ?

Alex Williamson wrote:

The native IGD driver would really like to continue running when VGA routing is directed elsewhere, so the PCI method of disabling access is sub-optimal.  At some point Intel knew this and included mechanisms in the device that allowed VGA MMIO and I/O port space to be disabled separately from PCI MMIO and I/O port space.  These mechanisms are what the i915 driver uses today when it opts-out of VGA arbitration.

The problem is that modern IGD devices don't implement the same mechanisms for disabling legacy VGA.  The i915 driver continues to twiddle the old bits and opt-out of arbitration, but the IGD VGA is still claiming VGA transactions.  This is why many users who claim they don't need the i915 patch finish their proclamation of success with a note about the VGA color palate being corrupted or noting some strange drm interrupt messages on the host dmesg. [...]

So why can't we simply fix the i915 driver to correctly disable legacy VGA support?  Well, in their infinite wisdom, hardware designers have removed (or broken) that feature of IGD.

My laptop's CPU is intel 2nd-gen, I have some color problems whenever I shutdown the VM, but this doesn't bother me much. So, I guess the feature is still there and not broken, but is on my intel 4th-gen CPU, right ?

The only IGDs that work with VGA arbitration are the ones were the GPU is on the chipset, not the CPU.  If you saw screen corruption on your laptop, even if it didn't bother you, it's the same problem.  It's probably more a difference of the GPU being assigned and whether the guest drivers are able to initialize it despite not having VGA access to it.

Not sure to understand what that implies. Should I care about it even if it works perfectly fine (even while gaming) ?


aw wrote:

Question 2: My laptop's configuration needs a vbios rom in order to work, but this is not the case of my desktop's GPU. In fact, whenever I specify an extracted vbios it won't work: black screen (I extracted it myself from a live-cd, I strictly followed the same procedure than for my laptop vbios). My first thought was the GPU is a bit too old but I think it's a stupid idea. I used rom-parser to check the vbios and it says it's fine, reporting correct vendors ids. Where could it comes from ? Any suggestions ? It tickles me a lot !

Was the GPU the primary graphics device when you extracted the BIOS?  PCs will load primary graphics BIOS into a different region and it may be modified runtime.  If that's the one you get when extracted, it may not work correctly.  If QEMU/VFIO is able to get the correct ROM from the card, why do you care?

I think it was, the (motherboard) BIOS got an option to choose the main Graphic Device.
Anyway, I don't need i. It was just curious.

aw wrote:

Question 3: Now that I begin to understand PCI-PT in qemu, should I try to write a tutorial for french people ? Isn't it a bit risky since I don't have a wide knowledge of the internal mechanics and couldn't properly troubleshoot specific case issues ?

Those of us who don't speak french won't be able to identify any mistakes, but more helpful documentation is always a good thing.

I'll do it.

One extra question: I saw many people videos achieving to pass the resulting output inside a window inside the host screen. How can I do that ?
I tried with xfreerdp but couldn't success. RemoteFX doesn't seem to be what I'm looking for. I noticed VNC has been mentionned many times in the previous posts but a friend told me to never use BNC because it's a total crap (slow and not free). I never used it but, is it the right way to pass the GPU output to my host ? is it time for me to make my own opinion of VNC ?

Offline

#3157 2014-11-05 15:25:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mouton_sanglant wrote:
aw wrote:
mouton_sanglant wrote:

Hello again !

I achieved to do VGA-Passthrough on another machine.
It is a desktop and more modern configuration but the GPU is quite old.

CPU: i5-4670T (Haswell family)
GPU: Nvidia 9500 GS

It's working fine, now going to do some tuning but there I have some questions first.

Question 1: While I could do VGA-PT with my laptop without applying any patches, I can't with this new config (black screen). Especially the vga-arbiter patch. From what I understood from this article from Alex Williamson (What's the deal with VGA arbitration ?), this is because my brand new CPU's IGP is too recent ?

My laptop's CPU is intel 2nd-gen, I have some color problems whenever I shutdown the VM, but this doesn't bother me much. So, I guess the feature is still there and not broken, but is on my intel 4th-gen CPU, right ?

The only IGDs that work with VGA arbitration are the ones were the GPU is on the chipset, not the CPU.  If you saw screen corruption on your laptop, even if it didn't bother you, it's the same problem.  It's probably more a difference of the GPU being assigned and whether the guest drivers are able to initialize it despite not having VGA access to it.

Not sure to understand what that implies. Should I care about it even if it works perfectly fine (even while gaming) ?

Do you care that your VM is doing reads and writes to the host GPU in ways that can corrupt its state?  I do.  I'd expect you should care too.  Besides, without VGA going to the correct device, why even bother with x-vga=on, you might as well assign the GPU as a secondary device, or at least disable VGA access to it.

One extra question: I saw many people videos achieving to pass the resulting output inside a window inside the host screen. How can I do that ?
I tried with xfreerdp but couldn't success. RemoteFX doesn't seem to be what I'm looking for. I noticed VNC has been mentionned many times in the previous posts but a friend told me to never use BNC because it's a total crap (slow and not free). I never used it but, is it the right way to pass the GPU output to my host ? is it time for me to make my own opinion of VNC ?

It's usually done through VNC but I think the videos make it look far more functional than it really is.  Conceptually it's the same as trying to play a game from a remote system over VNC, the only difference for the VM is that you can get a high performance local network connection between the host and guest.  VNC also consumes quite a bit of CPU to do the encoding and decoding, so you need extra cores for that.  I think that most people here are using a 2nd monitor for the assigned GPU output or switching inputs on their monitor.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3158 2014-11-05 16:00:24

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so I will apply the vga-arb patch when I got this remote control working out.

I don't really care about the extra cpu cycles because I won't use it often (once or twice a month). I just added "-vnc 0.0.0.0:1,mypasswd" to my qemu command but when connecting with client, the qemu console shows up. So I guess I can't go that way and I must install a vnc server inside the guest. Right ?

Offline

#3159 2014-11-05 16:13:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mouton_sanglant wrote:

Ok, so I will apply the vga-arb patch when I got this remote control working out.

I don't really care about the extra cpu cycles because I won't use it often (once or twice a month). I just added "-vnc 0.0.0.0:1,mypasswd" to my qemu command but when connecting with client, the qemu console shows up. So I guess I can't go that way and I must install a vnc server inside the guest. Right ?

yes, -vnc only works with emulated graphics devices


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3160 2014-11-05 16:34:20

snarfies
Member
Registered: 2014-11-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Our of curiosity, why is the linux-mainline-3.17.2.tar.gz (3.17.2 + acs override patch + i915 vga arbiter fixes) on the first post not in the AUR?

Offline

#3161 2014-11-05 19:24:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

snarfies wrote:

Our of curiosity, why is the linux-mainline-3.17.2.tar.gz (3.17.2 + acs override patch + i915 vga arbiter fixes) on the first post not in the AUR?

You wanna maintain it? Go ahead.

Offline

#3162 2014-11-06 15:28:57

mmm
Member
Registered: 2014-11-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm passing through two GTX 780 Ti to one Win7 guest simultaneously and they work perfectly in the guest system now (and latest drivers), without SLI though.
Now to try and get SLI working I tried a few different configurations with and without ioh3420 in between but in any case the NVidia control panel does not offer SLI settings.
As the cards are running in the VM perfectly fine as two single cards side by side I guess it's not the driver disabling it on purpose due to VM detection but rather a missing requirement on the emulated hardware.

As real hardware requires to have a SLI-enabled mainboard and possibly having SLI enabled in BIOS or through jumper settings, does anyone know what's happening there on real hardware and if any of that is implemented in qemu/seabios?

Offline

#3163 2014-11-06 15:47:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mmm wrote:

I'm passing through two GTX 780 Ti to one Win7 guest simultaneously and they work perfectly in the guest system now (and latest drivers), without SLI though.
Now to try and get SLI working I tried a few different configurations with and without ioh3420 in between but in any case the NVidia control panel does not offer SLI settings.
As the cards are running in the VM perfectly fine as two single cards side by side I guess it's not the driver disabling it on purpose due to VM detection but rather a missing requirement on the emulated hardware.

As real hardware requires to have a SLI-enabled mainboard and possibly having SLI enabled in BIOS or through jumper settings, does anyone know what's happening there on real hardware and if any of that is implemented in qemu/seabios?

From the wikipedia page:

Not all motherboards with multiple PCI-Express x16 slots support SLI. Recent motherboards as of August 2014 that support it are Intel's Z and X series chipsets (Z68, Z78, Z87, Z97, and X79) and AMD's 990FX chipset. Aside from a few exceptions, older motherboards needed the nForce series of chipsets to allow for SLI.

Sounds like it's pretty tied to specific hardware, which QEMU does not implement.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3164 2014-11-06 16:30:45

mmm
Member
Registered: 2014-11-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

From the wikipedia page:

Not all motherboards with multiple PCI-Express x16 slots support SLI. Recent motherboards as of August 2014 that support it are Intel's Z and X series chipsets (Z68, Z78, Z87, Z97, and X79) and AMD's 990FX chipset. Aside from a few exceptions, older motherboards needed the nForce series of chipsets to allow for SLI.

Sounds like it's pretty tied to specific hardware, which QEMU does not implement.

Hmm, interesting. I wonder if the chipset actually has to do any additional work since there was a hypervisor which faked it: http://www.techpowerup.com/forums/threa … ds.153046/

Also since Duelist got Crossfire working I'm rather optimistic that not a whole new emulated chipset implementation is required for SLI.

Last edited by mmm (2014-11-06 16:34:04)

Offline

#3165 2014-11-06 18:37:09

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mmm wrote:
aw wrote:

From the wikipedia page:

Not all motherboards with multiple PCI-Express x16 slots support SLI. Recent motherboards as of August 2014 that support it are Intel's Z and X series chipsets (Z68, Z78, Z87, Z97, and X79) and AMD's 990FX chipset. Aside from a few exceptions, older motherboards needed the nForce series of chipsets to allow for SLI.

Sounds like it's pretty tied to specific hardware, which QEMU does not implement.

Hmm, interesting. I wonder if the chipset actually has to do any additional work since there was a hypervisor which faked it: http://www.techpowerup.com/forums/threa … ds.153046/

Also since Duelist got Crossfire working I'm rather optimistic that not a whole new emulated chipset implementation is required for SLI.


Ahem!
First, i'm dumb. Do not forget that.
Second, there is a clarification note that i am writing now.
Third, my crossfire system DOES NOT HAVE any external bridges, it works solely through PCI-E. It is XDMA, it is very different from NVidia's SLI.I even have to enable IOMMU in bios to get that crossfire working under linux.
Fourth, NVidia intents to break everything at will. Too much software blocks. You've seen that linus torvalds reaction on nvidia? I agree with him.

Last edited by Duelist (2014-11-06 18:38:15)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3166 2014-11-06 19:20:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Awesome! Now i understand what is happening with my VM and why does it crash. But still not getting why it doesn't.

Preface: I do not know anything about linux memory management and x86 memory management and i DO NOT know how IOMMU works, so there MAY be huge misunderstandings and errors. Do not rely on my brain and train of thoughts.

A short story about how fat firefox saved the day multiple times and yet managed to show me webpages with the power of google.

I have IOMMU enabled, but working in it's default, noforce mode, without iommu=pt.
I thought it was enough just to get IOMMU working, and it'll handle the stuff itself.

So, i have rather sparse system RAM ranges in /proc/iomem, like 256k here, 256m there, 1g over here etc. and a huge chunk(~4-6g) in the end, 0x100001000-0x24EFFFFFF.

Without using -realtime mlock=on, VM gets a memory range that can be paged and moved around by kernel. PCI device drivers in guest, however, do not expect this happening, they do DMA to some address, that DMA get translated by IOMMU, it gets out of VM's memory range and BAM - IO_PAGE_FAULT, VM crash, sometimes host gets shutdown due to memory corruption caused or something.(previously i thought that there was an interrupt storm happening, but MSI gets enabled normally, and watch -n1 "cat /proc/interrupts" until host shutdown showed me no unusual stuff)

So, we have problems due to memory paging and moving and iommu not translating everything right. I pass -realtime mlock=on, qemu mlocks itself SOME, maybe even MULTIPLE chunks of memory somewhere. When i run the QEMU after the system startup, there is very little memory consumed and QEMU being one of the first processes launched, so it has memory near zero, where it is very sparse. Guest PCI device driver does DMA to some address, IOMMU remaps this again and we have a page fault again.

BUT EVENTUALLY, it starts working when ~800mb of memory is consumed or some time has passed. The following criteria was met at my first QEMU startup, it mlocked itself memory in the range where it can have large and continous chunks of memory, and guest doing DMA is fine. It can be damaged if the kernel gives QEMU sparse enough memory to mlock - we will get page faults again.


So, as i think, the problem hides in the IOMMU - it does redirect some DMA. Then i've tried reading linux kernel parameters, and there is no documentation given at all:

 iommu= [x86] off force noforce biomerge panic nopanic merge nomerge forcesac soft pt [x86, IA-64]

Yeah. What any of these mean?
So i went to google, and found this LKML message archived. Dated 2007. Better than nothing, but not approved, and there is no clarification on iommu=pt option, since there was no such option back in that time.
Awesome.
Maybe boot-options will help?
Yay, some clarification, but no iommu=pt option again. Yeah, i could've used nice, old iommu=soft, but something hints me that would be slow.

Then i went to google again, and found a part of kernel source code here with a nice comment:

 39 /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 40 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html This variable becomes 1 if iommu=pt is passed on the kernel command line. 41 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html If this variable is 1, IOMMU implementations do no DMA translation for 42 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html devices and allow every device to access to whole physical memory. This is 43 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html useful if a user wants to use an IOMMU only for KVM device assignment to 44 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html guests and not for driver dma translation. 45 */

Hmm... That seems to FIT EXACTLY FOR MY USE!

But that's not the end. Things are getting interesting.
Let's see what it looks like when VM is working right.

After a brief read of this...

We start a qemu, stop it to give us some time until windows boots and starts GPU drivers.
Then we do ps -ef | grep qemu to determine it's PID.
As root: cat /proc/QEMU-PID/maps | grep vfio
We have six ranges of memory mapped, looking like this:

7fc908000000-7fc918000000 rw-s b0000000 00:09 8529 anon_inode:[vfio-device] 7fc918000000-7fc928000000 rw-s c0000000 00:09 8529 anon_inode:[vfio-device] 7fca63b08000-7fca63b48000 rw-s fe200000 00:09 8529 anon_inode:[vfio-device] 7fca63b48000-7fca63b88000 rw-s fe300000 00:09 8529 anon_inode:[vfio-device] 7fca63c30000-7fca63c34000 rw-s fe260000 00:09 8529 anon_inode:[vfio-device] 7fca63c38000-7fca63c3c000 rw-s fe360000 00:09 8529 anon_inode:[vfio-device]

Well, as lspci -v hints us, these are really PCI device memory ranges!
But these addresses are virtual, it's how it looks like from the process' side. That isn't very interesting to us, but still interesting.

How do we translate virtual address to physical?
The power of google showed me this neat program that does that.

Copy-paste-save-gcc...

Big endian? 0 Vaddr: 0x7fc908000000, Page_size: 4096, Entry_size: 8 Reading /proc/3469/pagemap at 0x3fe4840000 [0]0xc3 [1]0x32 [2]0x1d [3]0x0 [4]0x0 [5]0x0 [6]0x0 [7]0x86 Result: 0x86000000001d32c3 PFN: 0x1d32c3

Yay, a physical address! And it fits. And VM works nice. And it doesn't crash.

Apparently, i didn't managed to crash the VM again on purpose, but when it does, it looks like...

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0021 address=0x00000001198c0000 flags=0x0010]

Yet the address is different(the example used here is very old) and it touches some other process' memory or even excedes the system ram range.

What i am not understanding fully - why does the DMA get remapped normally when it isn't out of qemu's memory bounds.
Maybe that whole story is a huge lie, heh, read the preface.

Morale:
IOMMU=PT IN KERNEL PARAMETERS OR DEATH!
Neat line in dmesg:
AMD-Vi: Initialized for Passthrough Mode

P.S.
Sorry for the double post, heh.

Last edited by Duelist (2014-11-06 19:30:00)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3167 2014-11-06 20:04:15

mmm
Member
Registered: 2014-11-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you try "-mem-prealloc"? If dynamic memory allocation is your problem, I feel like that should help.

Edit: nvm - saw it in your previous post now. Still wondering though how dynamic allocation could be a problem then.

Last edited by mmm (2014-11-06 20:28:50)

Offline

#3168 2014-11-06 21:29:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh dear, where to begin...

Duelist wrote:

So, i have rather sparse system RAM ranges in /proc/iomem, like 256k here, 256m there, 1g over here etc. and a huge chunk(~4-6g) in the end, 0x100001000-0x24EFFFFFF.

/proc/iomem is not RAM, it's device MMIO addresses

Without using -realtime mlock=on, VM gets a memory range that can be paged and moved around by kernel.

Only when an assigned device is NOT present.

PCI device drivers in guest, however, do not expect this happening, they do DMA to some address, that DMA get translated by IOMMU, it gets out of VM's memory range and BAM - IO_PAGE_FAULT, VM crash,

This is why when there's an assigned device guest memory is pinned, making the mlock redundant.

sometimes host gets shutdown due to memory corruption caused or something.(previously i thought that there was an interrupt storm happening, but MSI gets enabled normally, and watch -n1 "cat /proc/interrupts" until host shutdown showed me no unusual stuff)

So, we have problems due to memory paging and moving and iommu not translating everything right. I pass -realtime mlock=on, qemu mlocks itself SOME, maybe even MULTIPLE chunks of memory somewhere. When i run the QEMU after the system startup, there is very little memory consumed and QEMU being one of the first processes launched, so it has memory near zero, where it is very sparse. Guest PCI device driver does DMA to some address, IOMMU remaps this again and we have a page fault again.

BUT EVENTUALLY, it starts working when ~800mb of memory is consumed or some time has passed. The following criteria was met at my first QEMU startup, it mlocked itself memory in the range where it can have large and continous chunks of memory, and guest doing DMA is fine. It can be damaged if the kernel gives QEMU sparse enough memory to mlock - we will get page faults again.


So, as i think, the problem hides in the IOMMU - it does redirect some DMA. Then i've tried reading linux kernel parameters, and there is no documentation given at all:

 iommu= [x86] off force noforce biomerge panic nopanic merge nomerge forcesac soft pt [x86, IA-64]

Yeah. What any of these mean?
So i went to google, and found this LKML message archived. Dated 2007. Better than nothing, but not approved, and there is no clarification on iommu=pt option, since there was no such option back in that time.
Awesome.
Maybe boot-options will help?
Yay, some clarification, but no iommu=pt option again. Yeah, i could've used nice, old iommu=soft, but something hints me that would be slow.

Then i went to google again, and found a part of kernel source code here with a nice comment:

 39 /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 40 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html This variable becomes 1 if iommu=pt is passed on the kernel command line. 41 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html If this variable is 1, IOMMU implementations do no DMA translation for 42 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html devices and allow every device to access to whole physical memory. This is 43 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html useful if a user wants to use an IOMMU only for KVM device assignment to 44 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_12.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html guests and not for driver dma translation. 45 */

Hmm... That seems to FIT EXACTLY FOR MY USE!

But that's not the end. Things are getting interesting.
Let's see what it looks like when VM is working right.

After a brief read of this...

We start a qemu, stop it to give us some time until windows boots and starts GPU drivers.
Then we do ps -ef | grep qemu to determine it's PID.
As root: cat /proc/QEMU-PID/maps | grep vfio
We have six ranges of memory mapped, looking like this:

7fc908000000-7fc918000000 rw-s b0000000 00:09 8529 anon_inode:[vfio-device] 7fc918000000-7fc928000000 rw-s c0000000 00:09 8529 anon_inode:[vfio-device] 7fca63b08000-7fca63b48000 rw-s fe200000 00:09 8529 anon_inode:[vfio-device] 7fca63b48000-7fca63b88000 rw-s fe300000 00:09 8529 anon_inode:[vfio-device] 7fca63c30000-7fca63c34000 rw-s fe260000 00:09 8529 anon_inode:[vfio-device] 7fca63c38000-7fca63c3c000 rw-s fe360000 00:09 8529 anon_inode:[vfio-device]

Well, as lspci -v hints us, these are really PCI device memory ranges!
But these addresses are virtual, it's how it looks like from the process' side. That isn't very interesting to us, but still interesting.

How do we translate virtual address to physical?
The power of google showed me this neat program that does that.

Copy-paste-save-gcc...

Big endian? 0 Vaddr: 0x7fc908000000, Page_size: 4096, Entry_size: 8 Reading /proc/3469/pagemap at 0x3fe4840000 [0]0xc3 [1]0x32 [2]0x1d [3]0x0 [4]0x0 [5]0x0 [6]0x0 [7]0x86 Result: 0x86000000001d32c3 PFN: 0x1d32c3

Yay, a physical address! And it fits. And VM works nice. And it doesn't crash.

Um, ok.  What's PFN 0x1d32c3 tell you?  That's a Page Frame Number, so you need to multiply by the page size (4k) to get a physical address, so 0x1d32c3000 is the physical address, what does that tell us?

You can also note in this example:

01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    Region 0: [virtual] Memory at f21a0000 (64-bit, non-prefetchable) [size=16K]
                                                  ^^^^^^^^^^^
    Region 3: [virtual] Memory at f21c0000 (64-bit, non-prefetchable) [size=16K]
                                                  ^^^^^^^^^^^

7f96f35a2000-7f96f35a5000 rw-s f21c1000 00:09 7271                       anon_inode:[vfio-device]
                                               ^^^^^^^^^^
7f96f35a5000-7f96f35a9000 rw-s f21a0000 00:09 7271                       anon_inode:[vfio-device]
                                               ^^^^^^^^^^
But I'm still not sure what you're reading into that.

Apparently, i didn't managed to crash the VM again on purpose, but when it does, it looks like...

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0021 address=0x00000001198c0000 flags=0x0010]

Yet the address is different(the example used here is very old) and it touches some other process' memory or even excedes the system ram range.

adjusting the guest memory size is how I typically test whether these are stray DMAs or something is broken.  That address is just over 4.5G, so if you boot your guest with 2G of ram, do you still get these types of faults?

What i am not understanding fully - why does the DMA get remapped normally when it isn't out of qemu's memory bounds.
Maybe that whole story is a huge lie, heh, read the preface.

<ahhem> I think you're jumping to some inaccurate conclusions here.

Morale:
IOMMU=PT IN KERNEL PARAMETERS OR DEATH!

Not true at all.

What happens if you use the vfio_iommu_type1 module option disable_hugepages=1?  By default vfio will attempt to map the largest contiguous memory chunk it can find through the IOMMU.  AMD-Vi supports nearly any power of two size, but with AMD going out of favor with server vendors, I don't expect it gets much testing.  The disable_hugepages forces the vfio code to map each page individually, avoiding superpage bugs in the IOMMU driver.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3169 2014-11-07 01:36:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone !

I noticed a hiccup that occurs to every VM along with the host every half a minute .

To my surprise , rebuilding qemu-git solved the issue for me , but wait ! I did a little change :

I removed Haswell "optimizations" from makepkg.conf :

-march=core-avx2 -mtune=haswell (if I remember these values correctly) , and returned everything to default : -march=x86-64 -mtune=generic .

Rebuilt qemu-git + ovmf-svn , reboot and all hiccups are gone !

So generic gcc profile (for me at least) was "optimized" alot better than specific CPU optimizations .

Hope this helps anyone having similar issues !

Last edited by Denso (2014-11-07 01:36:47)

Offline

#3170 2014-11-07 01:47:10

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This is why when there's an assigned device guest memory is pinned, making the mlock redundant.

Then why does it helps?

aw wrote:

Um, ok.  What's PFN 0x1d32c3 tell you?  That's a Page Frame Number, so you need to multiply by the page size (4k) to get a physical address, so 0x1d32c3000 is the physical address, what does that tell us?

Yes, i've forgot to mention that this needs to multiplied by page size. I did that. That part of my bunch-of-letters is kinda useless because i haven't managed to get my VMs faulty memory chunks. Once i got a host shutdown. But the idea was simple - i determine the physical address, add 256M to it, and if compared to MMIO table it looks way to bad - that could mean that the page faults are inevitable.

aw wrote:

adjusting the guest memory size is how I typically test whether these are stray DMAs or something is broken.  That address is just over 4.5G, so if you boot your guest with 2G of ram, do you still get these types of faults?

Heh, why i didn't come up with such an idea.. But digging in the error logs - i've had different addresses, even pointing to near-zero(~<600M) system RAM ranges.

aw wrote:

<ahhem> I think you're jumping to some inaccurate conclusions here.

Preface.

aw wrote:

Morale:
IOMMU=PT IN KERNEL PARAMETERS OR DEATH!

Not true at all.

Well, it appears to be fixing page faults on my system. After some reboots mixed with complete power cycles i couldn't reproduce page faults again. Or maybe i'm just very lucky.

aw wrote:

What happens if you use the vfio_iommu_type1 module option disable_hugepages=1?  By default vfio will attempt to map the largest contiguous memory chunk it can find through the IOMMU.  AMD-Vi supports nearly any power of two size, but with AMD going out of favor with server vendors, I don't expect it gets much testing.  The disable_hugepages forces the vfio code to map each page individually, avoiding superpage bugs in the IOMMU driver.

It sounds very different from "just not using hugepages". I'll test that.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3171 2014-11-07 01:53:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hello everyone !

I noticed a hiccup that occurs to every VM along with the host every half a minute .

To my surprise , rebuilding qemu-git solved the issue for me , but wait ! I did a little change :

I removed Haswell "optimizations" from makepkg.conf :

-march=core-avx2 -mtune=haswell (if I remember these values correctly) , and returned everything to default : -march=x86-64 -mtune=generic .

Rebuilt qemu-git + ovmf-svn , reboot and all hiccups are gone !

So generic gcc profile (for me at least) was "optimized" alot better than specific CPU optimizations .

Hope this helps anyone having similar issues !

Please note, since GCC 4.9 generic optimization is now based on Intel Core and AMD Bulldozer microarchitectures' optimizations. What GCC version you were using when hiccups appeared?

GCC 4.9 Changes


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3172 2014-11-07 03:00:08

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Denso wrote:

Hello everyone !

I noticed a hiccup that occurs to every VM along with the host every half a minute .

To my surprise , rebuilding qemu-git solved the issue for me , but wait ! I did a little change :

I removed Haswell "optimizations" from makepkg.conf :

-march=core-avx2 -mtune=haswell (if I remember these values correctly) , and returned everything to default : -march=x86-64 -mtune=generic .

Rebuilt qemu-git + ovmf-svn , reboot and all hiccups are gone !

So generic gcc profile (for me at least) was "optimized" alot better than specific CPU optimizations .

Hope this helps anyone having similar issues !

Please note, since GCC 4.9 generic optimization is now based on Intel Core and AMD Bulldozer microarchitectures' optimizations. What GCC version you were using when hiccups appeared?

GCC 4.9 Changes

4.9.2

Maybe GCC knows whats best for my CPU better than me smile

Offline

#3173 2014-11-07 15:30:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Shaders? TMUs?
Shader and TM units count doubled on the slave crossfire card. Maybe that is related to video memory magic.
The cards are identical. Only differences are different heat pipe ends and a serial number. I don't know if it's a GPU-Z bug or VM errors(or maybe it actually works so), but it works and looks funny.

Well, appears like i was extremely lucky yesterday having no io page faults.

Today first VM launch resulted in a page fault, and i've recorded that. The address in question was located somewhere near 2.6Gb of MMIO address space.

Now i've disabled hugepages and the current VM instance seem to be stable.

Also, aw, modinfo kvm tells me that there is

parm: allow_unsafe_assigned_interrupts:Enable device assignment on platforms without interrupt remapping support. (bool)

which seem to be very alike to...

parm: allow_unsafe_interrupts:Enable VFIO IOMMU support for on platforms without interrupt remapping support. (bool)

in your vfio_iommu_type1 module. Am i right that it does the same thing, but for pci-assign? Or it enables interrupt remapping system-wide, regardless of method used?
Oh, and also,append a fix of that copy-paste "for on platforms" typo to your TODO list;)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3174 2014-11-07 15:45:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Also, aw, modinfo kvm tells me that there is

parm: allow_unsafe_assigned_interrupts:Enable device assignment on platforms without interrupt remapping support. (bool)

which seem to be very alike to...

parm: allow_unsafe_interrupts:Enable VFIO IOMMU support for on platforms without interrupt remapping support. (bool)

in your vfio_iommu_type1 module. Am i right that it does the same thing, but for pci-assign? Or it enables interrupt remapping system-wide, regardless of method used?
Oh, and also,append a fix of that copy-paste "for on platforms" typo to your TODO list;)

Right, those are equivalent options, one for pci-assign and one for vfio-pci.  Neither do anything to enable interrupt remapping, they just allow use of the driver without interrupt remapping enabled.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3175 2014-11-07 18:20:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Right, those are equivalent options, one for pci-assign and one for vfio-pci.  Neither do anything to enable interrupt remapping, they just allow use of the driver without interrupt remapping enabled.

But how does it works without interrupt remapping then? Using the emulated, kvm vcpu's PIC? Or something?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3176 2014-11-07 18:25:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

Right, those are equivalent options, one for pci-assign and one for vfio-pci.  Neither do anything to enable interrupt remapping, they just allow use of the driver without interrupt remapping enabled.

But how does it works without interrupt remapping then? Using the emulated, kvm vcpu's PIC? Or something?

Without interrupt remapping you simply run with the risk that a malicious guest can craft DMA writes to interrupt addresses and cause badness on the host.  With interrupt remapping, the interrupt remapping hardware only allows the guest to trigger interrupts that it has access to.  There's a difference in the format of the interrupt when enabled, but it's effectively an invisible security feature when enabled.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3177 2014-11-07 20:22:47

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible yet to passthrough Intel IGD HD4600? If yes, what more is needed compared to Radeon?

Offline

#3178 2014-11-07 20:32:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Is it possible yet to passthrough Intel IGD HD4600? If yes, what more is needed compared to Radeon?

There's some prototype code, but it would take quite a bit to assemble it into something useful.  The problem with IGD is that it's not a self contained device like a discrete GPU.  There's the PCI device at 00:02.0, but the drivers use device IDs from other parts of the chipset to identify the GPU, so it requires changing the VM host bridge and potentially even the PCH bridge IDs.  IGD also makes use of opregions and GTTs that are described by other devices and live outside the PCI resources of the 00:02.0 device itself, so we need to come up with ways to get access to those.  Effectively, while a discrete GPU like Radeon is discrete, IGD is smeared across the chipset, making assignment much more difficult.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3179 2014-11-07 20:34:20

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:

Is it possible yet to passthrough Intel IGD HD4600? If yes, what more is needed compared to Radeon?

There's some prototype code, but it would take quite a bit to assemble it into something useful.  The problem with IGD is that it's not a self contained device like a discrete GPU.  There's the PCI device at 00:02.0, but the drivers use device IDs from other parts of the chipset to identify the GPU, so it requires changing the VM host bridge and potentially even the PCH bridge IDs.  IGD also makes use of opregions and GTTs that are described by other devices and live outside the PCI resources of the 00:02.0 device itself, so we need to come up with ways to get access to those.  Effectively, while a discrete GPU like Radeon is discrete, IGD is smeared across the chipset, making assignment much more difficult.

Thanks for explanation... but still don't know if its possible or not? Whats that i915 patch for then? Thought it was for this?

https://github.com/UmbraMalison/qemu-igdvfio <-- Is this the prototype code or is there more?

Last edited by devianceluka (2014-11-07 20:35:40)

Offline

#3180 2014-11-07 20:36:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
aw wrote:
devianceluka wrote:

Is it possible yet to passthrough Intel IGD HD4600? If yes, what more is needed compared to Radeon?

There's some prototype code, but it would take quite a bit to assemble it into something useful.  The problem with IGD is that it's not a self contained device like a discrete GPU.  There's the PCI device at 00:02.0, but the drivers use device IDs from other parts of the chipset to identify the GPU, so it requires changing the VM host bridge and potentially even the PCH bridge IDs.  IGD also makes use of opregions and GTTs that are described by other devices and live outside the PCI resources of the 00:02.0 device itself, so we need to come up with ways to get access to those.  Effectively, while a discrete GPU like Radeon is discrete, IGD is smeared across the chipset, making assignment much more difficult.

Thanks for explanation... but still don't know if its possible or not? Whats that i915 patch for then? Thought it was for this?

If there's prototype code it must be possible...

i915:
http://vfio.blogspot.com/2014/08/whats- … ation.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3181 2014-11-07 20:41:44

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:
aw wrote:

There's some prototype code, but it would take quite a bit to assemble it into something useful.  The problem with IGD is that it's not a self contained device like a discrete GPU.  There's the PCI device at 00:02.0, but the drivers use device IDs from other parts of the chipset to identify the GPU, so it requires changing the VM host bridge and potentially even the PCH bridge IDs.  IGD also makes use of opregions and GTTs that are described by other devices and live outside the PCI resources of the 00:02.0 device itself, so we need to come up with ways to get access to those.  Effectively, while a discrete GPU like Radeon is discrete, IGD is smeared across the chipset, making assignment much more difficult.

Thanks for explanation... but still don't know if its possible or not? Whats that i915 patch for then? Thought it was for this?

If there's prototype code it must be possible...

i915:
http://vfio.blogspot.com/2014/08/whats- … ation.html

I know this is some kind of off-topic, but what I've quickly read on google is that it is possible to passthrough it with Xen. Is it possible to have both Xen and QEMU/KVM at the same time running, Xen for IGD, QEMU/KVM for rest?

Thanks man!

Offline

#3182 2014-11-07 20:42:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
aw wrote:
devianceluka wrote:

Thanks for explanation... but still don't know if its possible or not? Whats that i915 patch for then? Thought it was for this?

If there's prototype code it must be possible...

i915:
http://vfio.blogspot.com/2014/08/whats- … ation.html

I know this is some kind of off-topic, but what I've quickly read on google is that it is possible to passthrough it with Xen. Is it possible to have both Xen and QEMU/KVM at the same time running, Xen for IGD, QEMU/KVM for rest?

No


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3183 2014-11-08 15:46:26

mouton_sanglant
Member
Registered: 2014-10-20
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello again !

I finally successed in passing the virtualized system's output via a vnc server as Alex Williamson suggested and it's working fine.

Ok, performances and quality are pretty much altered but I achieved to get something acceptable, and I feel ok with it since I will use this only 1 or 2 days a month. I will use this virtual system to develop/compile/test games we develop on Windows while I'll be at my mate's house (of course, I could use an extra hard drive or multiple OS boot, but I feel more comfortable coding on linux).

I have tested multiple vnc solutions and the most performant was TightVNC + DFMirage mirror display Driver (Windows Guest) combined with TurboVNC (Linux Host).
I also tested UltraVNC + mirror driver (Win Guest) combined with TurboVNC (Linux Host) but the performances wern't satisfying.

I saw some "Microsoft Windows" mirror drivers too. Not tested it yet.


Next step: write an article on my experience. (hope it will help the french community)

[Edit]: Just noticed "aw" stands for Alex Williamson... Thank you for your top quality posts, it helped me a lot !

Last edited by mouton_sanglant (2014-11-08 15:51:10)

Offline

#3184 2014-11-09 14:33:50

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Now i've disabled hugepages and the current VM instance seem to be stable.

Nei, it doesn't help. VM is still suspect to crashes when launched almost immediately after host system startup.

Trying to allocate less that 4G of memory.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3185 2014-11-09 16:33:17

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys,

I have been using VGA pass through for quite some time now, with the option -cpu Haswell. Everything works fine, however now I want to change over to -cpu host. If I do this my Windows 8.1 guest will boot into repair mode, try fix stuff and then reboot. This continues endlessly.
My settings:

sudo -E qemu-system-x86_64 \ -enable-kvm \ -M q35 \ -m 8G \ -mem-path /dev/hugepages \ -mem-prealloc \ -cpu Haswell,hv-time,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -device vfio-pci,host=00:1b.0 \ -bios /usr/share/qemu/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -nographic \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/var/lib/libvirt/images/Windows8.1.img,id=disk,format=raw,cache=none \ -device ide-hd,bus=ahci.0,drive=disk \ -device vfio-pci,host=07:00.0,bus=pcie.0 \ -device vfio-pci,host=00:19.0,bus=pcie.0 \ -drive file=/dev/sdb1,id=mmo,format=raw,cache=none \ -device ide-hd,bus=ahci.1,drive=mmo \ -drive file=/home/thor/Windows/Windows-Steam.img,id=steam,format=raw,cache=none \ -drive file=/dev/sdc1,id=music,format=raw,cache=none \ -device ide-hd,bus=ahci.3,drive=music \ -net none \ -serial none \ -parallel none \ -rtc base=localtime \ -device ide-hd,bus=ahci.2,drive=steam

What is causing this? I shouldnt have to reinstall the guest for simply changing the CPU? (My physical CPU is a Haswell after all)
The reason for why I want "-cpu host" is that Nvidia checks if I have a valid CPU when it comes to use the ShadowPlay function.

Offline

#3186 2014-11-09 17:04:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

The reason for why I want "-cpu host" is that Nvidia checks if I have a valid CPU when it comes to use the ShadowPlay function.

Huh? A brand-new DICK MOVE from nvidia? Or is it really-really-really needed?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3187 2014-11-09 17:11:28

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Its not needed at all, but now I am quite interested in why -cpu host does not work!
The Nvidia cpu check for Gefore Experience came in version 340.52 I believe(the same driver where we had to use kvm=off).
The ShadowPlay function worked before with -cpu Haswell

Offline

#3188 2014-11-09 18:43:04

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone smile

For those who have only one USB controller onboard (Like me with my X99-Deluxe motherboard) , I did find a solution .

I bought this expansion card :

http://www.amazon.com/gp/product/B00K9Q … UTF8&psc=1

And it gave me FOUR SEPERATE USB controllers (each controller is a seperate 5gbps channel) :

0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 12:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

And I could assign each one to a seperate VM using VFIO , and its working like a charm !

Hope this help . smile

Offline

#3189 2014-11-09 18:51:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hi everyone smile

For those who have only one USB controller onboard (Like me with my X99-Deluxe motherboard) , I did find a solution .

I bought this expansion card :

http://www.amazon.com/gp/product/B00K9Q … UTF8&psc=1

And it gave me FOUR SEPERATE USB controllers (each controller is a seperate 5gbps channel) :

0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 12:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

And I could assign each one to a seperate VM using VFIO , and its working like a charm !

Thanks for sharing.  Does the PCIe switch on the card support ACS or do you require the ACS override for this device?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3190 2014-11-09 18:54:09

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:

Hi everyone smile

For those who have only one USB controller onboard (Like me with my X99-Deluxe motherboard) , I did find a solution .

I bought this expansion card :

http://www.amazon.com/gp/product/B00K9Q … UTF8&psc=1

And it gave me FOUR SEPERATE USB controllers (each controller is a seperate 5gbps channel) :

0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 12:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

And I could assign each one to a seperate VM using VFIO , and its working like a charm !

Thanks for sharing.  Does the PCIe switch on the card support ACS or do you require the ACS override for this device?

No it doesn't . I had to recompile a mainline kernel with ACS patch , otherwise it did go into a large group along with other onboard controllers (Ethernet and others) .

Offline

#3191 2014-11-09 18:56:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:
Denso wrote:

Hi everyone smile

For those who have only one USB controller onboard (Like me with my X99-Deluxe motherboard) , I did find a solution .

I bought this expansion card :

http://www.amazon.com/gp/product/B00K9Q … UTF8&psc=1

And it gave me FOUR SEPERATE USB controllers (each controller is a seperate 5gbps channel) :

0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 12:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

And I could assign each one to a seperate VM using VFIO , and its working like a charm !

Thanks for sharing.  Does the PCIe switch on the card support ACS or do you require the ACS override for this device?

No it doesn't . I had to recompile a mainline kernel with ACS patch , otherwise it did go into a large group along with other onboard controllers (Ethernet and others) .

Could you pastebin/pm me the full lspci -vvv from your X99 box?  I've been asking Intel whether we need ACS quirks for that chipset but haven't gotten an answer yet.  Thanks.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3192 2014-11-09 19:02:22

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Could you pastebin/pm me the full lspci -vvv from your X99 box?  I've been asking Intel whether we need ACS quirks for that chipset but haven't gotten an answer yet.  Thanks.

Sure :

http://www.pastebin.ca/2870717

smile

EDIT :

Please note that the X99-Deluxe doesn't have a PLX chip , it is a part of the USB PCI-E card mentioned earlier .

Last edited by Denso (2014-11-09 19:15:19)

Offline

#3193 2014-11-09 20:31:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

Could you pastebin/pm me the full lspci -vvv from your X99 box?  I've been asking Intel whether we need ACS quirks for that chipset but haven't gotten an answer yet.  Thanks.

Sure :

http://www.pastebin.ca/2870717

smile

EDIT :

Please note that the X99-Deluxe doesn't have a PLX chip , it is a part of the USB PCI-E card mentioned earlier .

Right, it looks like the USB card does have ACS, it's the fault of the PCH root ports on the chipset that you're getting things grouped together:

00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) (prog-if 00 [Normal decode]) Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 00:1c.3 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #4 (rev d5) (prog-if 00 [Normal decode]) Bus: primary=00, secondary=07, subordinate=0c, sec-latency=0 07:00.0 PCI bridge: ASMedia Technology Inc. Device 118f (prog-if 00 [Normal decode]) Bus: primary=07, secondary=08, subordinate=0c, sec-latency=0 08:01.0 PCI bridge: ASMedia Technology Inc. Device 118f (prog-if 00 [Normal decode]) Bus: primary=08, secondary=09, subordinate=09, sec-latency=0 09:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 08:02.0 PCI bridge: ASMedia Technology Inc. Device 118f (prog-if 00 [Normal decode]) Bus: primary=08, secondary=0a, subordinate=0a, sec-latency=0 0a:00.0 Network controller: Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter (rev 03) 08:03.0 PCI bridge: ASMedia Technology Inc. Device 118f (prog-if 00 [Normal decode]) Bus: primary=08, secondary=0b, subordinate=0b, sec-latency=0 08:04.0 PCI bridge: ASMedia Technology Inc. Device 118f (prog-if 00 [Normal decode]) Bus: primary=08, secondary=0c, subordinate=0c, sec-latency=0 00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5) (prog-if 00 [Normal decode]) Bus: primary=00, secondary=0d, subordinate=12, sec-latency=0 0d:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Bus: primary=0d, secondary=0e, subordinate=12, sec-latency=0 0e:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Bus: primary=0e, secondary=0f, subordinate=0f, sec-latency=0 Capabilities: [520 v1] Access Control Services 0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI]) 0e:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Bus: primary=0e, secondary=10, subordinate=10, sec-latency=0 Capabilities: [520 v1] Access Control Services 10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI]) 0e:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Bus: primary=0e, secondary=11, subordinate=11, sec-latency=0 Capabilities: [520 v1] Access Control Services 11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI]) 0e:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) (prog-if 00 [Normal decode]) Bus: primary=0e, secondary=12, subordinate=12, sec-latency=0 Capabilities: [520 v1] Access Control Services 12:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI]) 00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5) (prog-if 00 [Normal decode]) Bus: primary=00, secondary=13, subordinate=13, sec-latency=0 13:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller (prog-if 30 [XHCI])

That ASMedia PCIe switch does not have ACS support (not surprising), so any devices below it will always be grouped, but the PLX switch does, so if we could quirk the PCH root ports it would break up that grouping.

One more request, please report "lspci -ns 00:1c." so we have the device IDs for those root ports.  Thanks.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3194 2014-11-09 20:58:03

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

That ASMedia PCIe switch does not have ACS support (not surprising), so any devices below it will always be grouped, but the PLX switch does, so if we could quirk the PCH root ports it would break up that grouping.

One more request, please report "lspci -ns 00:1c." so we have the device IDs for those root ports.  Thanks.

Here :

00:1c.0 0604: 8086:8d10 (rev d5)
00:1c.3 0604: 8086:8d16 (rev d5)
00:1c.4 0604: 8086:8d18 (rev d5)
00:1c.6 0604: 8086:8d1c (rev d5)

I'm ready for any patch testing ! Thanks .

Offline

#3195 2014-11-09 21:18:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

That ASMedia PCIe switch does not have ACS support (not surprising), so any devices below it will always be grouped, but the PLX switch does, so if we could quirk the PCH root ports it would break up that grouping.

One more request, please report "lspci -ns 00:1c." so we have the device IDs for those root ports.  Thanks.

Here :

00:1c.0 0604: 8086:8d10 (rev d5)
00:1c.3 0604: 8086:8d16 (rev d5)
00:1c.4 0604: 8086:8d18 (rev d5)
00:1c.6 0604: 8086:8d1c (rev d5)

I'm ready for any patch testing ! Thanks .

The patch is trivial, it looks just like the one for X79, but with your device IDs obviously:

http://git.kernel.org/cgit/linux/kernel … 141fca8e23

The hard part is getting Intel to verify that the PCH root ports are isolated from one another.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3196 2014-11-09 22:11:48

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:
aw wrote:

That ASMedia PCIe switch does not have ACS support (not surprising), so any devices below it will always be grouped, but the PLX switch does, so if we could quirk the PCH root ports it would break up that grouping.

One more request, please report "lspci -ns 00:1c." so we have the device IDs for those root ports.  Thanks.

Here :

00:1c.0 0604: 8086:8d10 (rev d5)
00:1c.3 0604: 8086:8d16 (rev d5)
00:1c.4 0604: 8086:8d18 (rev d5)
00:1c.6 0604: 8086:8d1c (rev d5)

I'm ready for any patch testing ! Thanks .

The patch is trivial, it looks just like the one for X79, but with your device IDs obviously:

http://git.kernel.org/cgit/linux/kernel … 141fca8e23

The hard part is getting Intel to verify that the PCH root ports are isolated from one another.

Will we see X99 quirks added to mainline soon ?

I'm in no hurry , I can wait until 3.19 .

EDIT : Spelling .

Last edited by Denso (2014-11-09 23:41:29)

Offline

#3197 2014-11-10 08:54:42

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Can you do anything about these ?

no_acs

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GT] (rev a1) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) ### Group 6 ### 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) ### Group 7 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) ### Group 8 ### 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) ### Group 9 ### 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) ### Group 12 ### 03:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) ### Group 13 ### 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) ### Group 14 ### 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 15 ### 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 16 ### 06:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 17 ### 06:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 18 ### 06:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 19 ### 06:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 20 ### 06:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 21 ### 06:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 22 ### 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 23 ### 08:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01) ### Group 24 ### 0a:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 25 ### 0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) ### Group 26 ### 0c:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller ### Group 27 ### 0d:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 

acs

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) ### Group 2 ### 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) ### Group 3 ### 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) ### Group 9 ### 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) ### Group 10 ### 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) ### Group 11 ### 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) ### Group 12 ### 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) ### Group 13 ### 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 14 ### 02:00.0 VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GT] (rev a1) ### Group 15 ### 03:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) ### Group 16 ### 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) ### Group 17 ### 05:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 18 ### 06:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 19 ### 06:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 20 ### 06:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 21 ### 06:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 22 ### 06:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 23 ### 06:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 24 ### 06:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) ### Group 25 ### 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 26 ### 08:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01) ### Group 27 ### 0a:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 28 ### 0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) ### Group 29 ### 0c:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller ### Group 30 ### 0d:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 

lspci -nnvvv

00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0100] (rev 09) Subsystem: ASRock Incorporation Device [1849:0100] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: snb_uncore 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000efff Memory behind bridge: f6100000-f61fffff Prefetchable memory behind bridge: 00000000e0000000-00000000efffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: ASRock Incorporation Device [1849:0101] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002d8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=02 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Kernel driver in use: pcieport Kernel modules: shpchp 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0105] (rev 09) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f2000000-f50fffff Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [88] Subsystem: ASRock Incorporation Device [1849:0105] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee002f8 Data: 0000 Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #3, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <256ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #2, PowerLimit 75.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [140 v1] Root Complex Link Desc: PortNumber=03 ComponentID=01 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed19000 Kernel driver in use: pcieport Kernel modules: shpchp 00:02.0 VGA compatible controller [0300]: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller [8086:0102] (rev 09) (prog-if 00 [VGA controller]) Subsystem: ASRock Incorporation Device [1849:0102] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 40 Region 0: Memory at f5400000 (64-bit, non-prefetchable) [size=4M] Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at f000 [size=64] Expansion ROM at <unassigned> [disabled] Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00018 Data: 0000 Capabilities: [d0] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a4] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: i915 Kernel modules: i915 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04) (prog-if 30 [XHCI]) Subsystem: ASRock Incorporation Motherboard [1849:1e31] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 37 Region 0: Memory at f6200000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Address: 00000000fee00478 Data: 0000 Kernel driver in use: xhci_hcd Kernel modules: xhci_hcd 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 [8086:1e3a] (rev 04) Subsystem: ASRock Incorporation Motherboard [1849:1e3a] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 39 Region 0: Memory at f621a000 (64-bit, non-prefetchable) [size=16] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee004b8 Data: 0000 Kernel driver in use: mei_me Kernel modules: mei_me 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04) (prog-if 20 [EHCI]) Subsystem: ASRock Incorporation Motherboard [1849:1e2d] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at f6218000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller [8086:1e20] (rev 04) Subsystem: ASRock Incorporation Z77 Extreme4 motherboard [1849:1898] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 255 Region 0: Memory at f6210000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=1 ArbSelect=Fixed TC/VC=22 Status: NegoPending- InProgress- Capabilities: [130 v1] Root Complex Link Desc: PortNumber=0f ComponentID=00 EltType=Config Link0: Desc: TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+ Addr: 00000000fed1c000 Kernel driver in use: pci-stub Kernel modules: snd_hda_intel 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: f5e00000-f5ffffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us ClockPM- Surprise- LLActRep+ BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 25.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASRock Incorporation Motherboard [1849:1e10] Capabilities: [a0] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.4 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 [8086:1e18] (rev c4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 0000b000-0000bfff Memory behind bridge: f6000000-f60fffff Prefetchable memory behind bridge: 00000000f0200000-00000000f02fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us ClockPM- Surprise- LLActRep+ BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #4, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASRock Incorporation Motherboard [1849:1e18] Capabilities: [a0] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=05, subordinate=0e, sec-latency=0 I/O behind bridge: 00009000-0000afff Memory behind bridge: f5800000-f5dfffff Prefetchable memory behind bridge: 00000000f0000000-00000000f01fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #8, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us ClockPM- Surprise- LLActRep+ BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #7, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [90] Subsystem: ASRock Incorporation Motherboard [1849:1e1e] Capabilities: [a0] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport Kernel modules: shpchp 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04) (prog-if 20 [EHCI]) Subsystem: ASRock Incorporation Motherboard [1849:1e26] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 23 Region 0: Memory at f6217000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04) Subsystem: ASRock Incorporation Motherboard [1849:1e44] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:1e02] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin B routed to IRQ 38 Region 0: I/O ports at f0b0 [size=8] Region 1: I/O ports at f0a0 [size=4] Region 2: I/O ports at f090 [size=8] Region 3: I/O ports at f080 [size=4] Region 4: I/O ports at f060 [size=32] Region 5: Memory at f6216000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00498 Data: 0000 Capabilities: [70] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004 Capabilities: [b0] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller [8086:1e22] (rev 04) Subsystem: ASRock Incorporation Motherboard [1849:1e22] Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin C routed to IRQ 18 Region 0: Memory at f6215000 (64-bit, non-prefetchable) [size=256] Region 4: I/O ports at f040 [size=32] Kernel modules: i2c_i801 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] [1002:6810] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device [1043:0464] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 255 Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at f6100000 (64-bit, non-prefetchable) [size=256K] Region 4: I/O ports at e000 [size=256] Expansion ROM at f6140000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [200 v1] #15 Capabilities: [270 v1] #19 Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] #13 Capabilities: [2d0 v1] #1b Kernel driver in use: pci-stub Kernel modules: radeon 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] Subsystem: ASUSTeK Computer Inc. Device [1043:aab0] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 255 Region 0: Memory at f6160000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: pci-stub Kernel modules: snd_hda_intel 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation G84 [GeForce 8600 GT] [10de:0402] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device [10de:0505] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 255 Region 0: Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=16M] Region 1: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 3: Memory at f2000000 (64-bit, non-prefetchable) [disabled] [size=32M] Region 5: I/O ports at d000 [disabled] [size=128] Expansion ROM at f5000000 [disabled] [size=128K] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x16, ASPM not supported, Exit Latency L0s <512ns, L1 <4us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: pci-stub Kernel modules: nouveau 03:00.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS [1000:0058] (rev 08) Subsystem: Dell SAS 6/iR Adapter RAID Controller [1028:1f0e] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 255 Region 0: I/O ports at c000 [disabled] [size=256] Region 1: Memory at f5f10000 (64-bit, non-prefetchable) [disabled] [size=16K] Region 3: Memory at f5f00000 (64-bit, non-prefetchable) [disabled] [size=64K] Expansion ROM at f5e00000 [disabled] [size=1M] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [98] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [b0] MSI-X: Enable- Count=1 Masked- Vector table: BAR=1 offset=00002000 PBA: BAR=1 offset=00003000 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: pci-stub Kernel modules: mptsas 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06) Subsystem: Device [7470:3468] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 36 Region 0: I/O ports at b000 [size=256] Region 2: Memory at f6000000 (64-bit, non-prefetchable) [size=4K] Region 4: Memory at f0200000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00438 Data: 0000 Capabilities: [70] Express (v2) Endpoint, MSI 01 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b0] MSI-X: Enable- Count=4 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00000800 Capabilities: [d0] Vital Product Data Unknown small resource type 05, will not decode more. Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [160 v1] Device Serial Number 9e-3e-00-00-68-4c-e0-00 Kernel driver in use: r8169 Kernel modules: r8169 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Region 0: Memory at f5d00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=05, secondary=06, subordinate=0e, sec-latency=0 I/O behind bridge: 00009000-0000afff Memory behind bridge: f5800000-f5cfffff Prefetchable memory behind bridge: 00000000f0000000-00000000f01fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00318 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 10.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=4 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=06 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32+ WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR32 TC/VC=01 Status: NegoPending- InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [448 v1] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: f5c00000-f5cfffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00338 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #113, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=08, subordinate=09, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00358 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #116, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=0a, subordinate=0a, sec-latency=0 I/O behind bridge: 00009000-00009fff Memory behind bridge: f5b00000-f5bfffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00378 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #117, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:06.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=0b, subordinate=0b, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: f5a00000-f5afffff Prefetchable memory behind bridge: 00000000f0100000-00000000f01fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee003b8 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #118, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=0c, subordinate=0c, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: f5900000-f59fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee003d8 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+ SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #119, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=0d, subordinate=0d, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: f5800000-f58fffff Prefetchable memory behind bridge: 00000000f0000000-00000000f00fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee003f8 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #120, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=06, secondary=0e, subordinate=0e, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Address: 00000000fee00418 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #9, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #121, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 05, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport Kernel modules: shpchp 07:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:0612] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 255 Region 0: I/O ports at a050 [size=8] Region 1: I/O ports at a040 [size=4] Region 2: I/O ports at a030 [size=8] Region 3: I/O ports at a020 [size=4] Region 4: I/O ports at a000 [size=32] Region 5: Memory at f5c00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [78] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s <512ns, L1 <2us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Kernel driver in use: pci-stub Kernel modules: ahci 08:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=08, secondary=09, subordinate=09, sec-latency=32 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard [1849:1080] 0a:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0]) Subsystem: ASRock Incorporation Motherboard [1849:0612] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 255 Region 0: I/O ports at 9050 [size=8] Region 1: I/O ports at 9040 [size=4] Region 2: I/O ports at 9030 [size=8] Region 3: I/O ports at 9020 [size=4] Region 4: I/O ports at 9000 [size=32] Region 5: Memory at f5b00000 (32-bit, non-prefetchable) [size=512] Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [78] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s <512ns, L1 <2us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Kernel driver in use: pci-stub Kernel modules: ahci 0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe [14e4:16b1] (rev 10) Subsystem: ASRock Incorporation Z77 Extreme4 motherboard [1849:96b1] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 17 Region 0: Memory at f0110000 (64-bit, prefetchable) [size=64K] Region 2: Memory at f0100000 (64-bit, prefetchable) [size=64K] Expansion ROM at f5a00000 [disabled] [size=2K] Capabilities: [48] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [a0] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00000120 Capabilities: [ac] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [13c v1] Device Serial Number 00-00-bc-5f-f4-38-a3-3d Capabilities: [150 v1] Power Budgeting <?> Capabilities: [160 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Kernel driver in use: tg3 Kernel modules: tg3 0c:00.0 USB controller [0c03]: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller [1b6f:7052] (prog-if 30 [XHCI]) Subsystem: ASRock Incorporation Device [1849:7052] Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 18 Region 0: Memory at f5900000 (64-bit, non-prefetchable) [size=32K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/4 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [a0] Express (v2) Endpoint, MSI 01 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [190 v1] Device Serial Number 01-01-01-01-01-01-01-01 Kernel driver in use: pci-stub Kernel modules: xhci_hcd 0d:00.0 Ethernet controller [0200]: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe [14e4:16b1] (rev 10) Subsystem: ASRock Incorporation Z77 Extreme4 motherboard [1849:96b1] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 19 Region 0: Memory at f0010000 (64-bit, prefetchable) [size=64K] Region 2: Memory at f0000000 (64-bit, prefetchable) [size=64K] Expansion ROM at f5800000 [disabled] [size=2K] Capabilities: [48] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [a0] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00000120 Capabilities: [ac] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [13c v1] Device Serial Number 00-00-bc-5f-f4-38-a3-38 Capabilities: [150 v1] Power Budgeting <?> Capabilities: [160 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Kernel driver in use: tg3 Kernel modules: tg3

Thanks.

Offline

#3198 2014-11-10 13:18:02

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have been trying to use virsh to install Windows 8.1. However I couldn't manage to load the virtio.iso drivers. How do I need to edit the XML file so Virtio drivers are loaded too once Windows 8.1 installations starts? It doesn't ask me for the drivers ...

thanks a lot!!

Offline

#3199 2014-11-10 13:59:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:

@aw
Can you do anything about these ?

no_acs

### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GT] (rev a1)

lspci -nnvvv

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09) (prog-if 00 [Normal decode]) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0105] (rev 09) (prog-if 00 [Normal decode])

No, these are the processor based root ports.  Intel does not seem to have any intention of providing a statement regarding isolation or procedure to configure isolation of client processor root ports.  Note that while the quirk for PCH root ports is trivial, the code that it enables modifies the chipset configuration to enforce isolation between ports.  Client processors include Core i5/i7 in this case (no VT-d on i3).  Xeon processors support ACS on the processor root ports with the exception of the E3-1200 series.  Intel even went so far as to provide a specification update regarding these (http://www.intel.com/content/dam/www/pu … update.pdf)

HSW2. Intel® Virtualization Technology (Intel® VT) Clarification
Section 3.1 will be modified to include the following paragraph:
It is recommended to avoid device direct assignment to Virtual Machines in virtualized
environments with this processor due to the lack of Access Control Services (ACS)
support in PCI-Express root ports. Some Operating Systems may check for ACS support
and potentially disable direct device assignment (that is, affects SR-IOV setup/
configuration within the server) as well.

While this is for the E3-1200 v3, I tend to assume it applies to all E3-1200 processors and the root port logic from your processor is probably shared with that processor series even if it's a Core processor.

I cannot advise using the ACS override in this case because we have no indication from the hardware vendor that there is any isolation between these root ports, nor do I have any basis to quirk ACS support for these devices upstream.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3200 2014-11-10 13:59:51

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to make sure you are using virtio disk in xml, then add ide cd-rom with virtio-drivers... like here

<disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/hdd2/win8.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/virtio-win-0.1-81.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

At windows setup, when it asks where to install, choose load driver and then point it to mounted cd-rom with viritio drivers.

Last edited by slis (2014-11-10 14:01:13)

Offline

#3201 2014-11-10 22:42:18

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Child_of_Sun wrote:

Hi @all

I have managed to passthrough my Primary Graphic Card (And the Secondary for CrossfireX).

I use a custom initramfs with a custom init script, which means that i don't know if it's important to setup this during boot.

My Computer:
CPU: Amd FX-8350
Mainboard: Asrock 970 Extreme4
Primary Graphics: PowerColor PCS+ HD7770 GHz Edition 1GB GDDR5      (vfio-pci, Windows)
Secondary Graphics: PowerColor HD7770 GHz Edition 1GB GDDR5 (V2?)  (vfio-pci, Windows)
Tertiary Graphics: PowerColor HD7750 1GB GDDR5                                     (Linux Host)
Memory: 16 GB Transcend 1333MHz Memory
Power Supply: Thermaltake Berlin 630W

I use Gentoo Linux ~amd64 with kernel 3.18.0-rc2 (Because of the OverlayFS :-), should work with 3.17.1 too) and Qemu-2.1.2 with Seabios-1.7.5 (Release)

My Kernel cmdline is:

root=/dev/mapper/root rootfstype=btrfs rw iommu=pt video=radeondrmfb:1280x1024-24@75,mtrr:3,pmipal,ywrap kvm.ignore_msrs=1 vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1b21:1042,1002:4383,1002:4391,1002:4393,1002:439c vfio-bind=0000:02:00.0,0000:02:00.1,0000:01:00.0,0000:01:00.1,0000:00:14.2,0000:00:11.0,0000:04:00.0,0000:05:00.0 hugepagesz=1GB fbcon=map:1

The vfio-bind Option is parsed by the custom init in my initramfs.

At boot i bind the devices (Like mentioned here) to the pci-stub driver, later to the vfio-pci driver. The Essential Lines from my init for this Proces are:

/sbin/vfio-bind "${vfiobond}" echo 0000:03:00.1 > /sys/bus/pci/devices/0000:03:00.1/driver/unbind # The Amd Audio device binds to pci-stub, since it has the same ids at the 7750 and the 7770, this fix this Problem. modprobe radeon # Here i Load the radeon kernel module for the framebuffer sleep 4 # Wait a little bit to be sure everything has enough time to switch echo 1 > /sys/bus/pci/drivers/vfio-pci/0000:01:00.0/remove # Here i remove the Primary adapter from the PCI bus echo 1 > /sys/bus/pci/drivers/vfio-pci/0000:01:00.1/remove echo 1 > /sys/bus/pci/rescan # Here i rescan the Pci-Bus for new devices, it find the Primary radeon and binds it automatically to the vfio-pci driver

The Last three Lines are Essential for this to work, with this config i can restart the VM, install newer Graphics Drivers, Play 3D Games (Elder Scrolls Online at the moment :-) ).

I use the rombios=/path/to/bios/file Option for the Primary Card, don't know if it is essential.

At Last here is the Qemu Startup Line from my script:


until /usr/bin/sudo /usr/bin/nice -n 10 /usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -monitor stdio -nographic -balloon none -mem-path /dev/hugepages -mem-prealloc \ -m 10240 -k de -cpu host -smp 8,sockets=1,cores=8,threads=1 -bios /usr/share/qemu/bios.bin-1.7.5 -realtime mlock=on \ -vga none -D /var/log/qemu-out.log -boot menu=on -usb -usbdevice host:046d:c517 -usbdevice host:093a:2510 \ -device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \ -device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \ -device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=2,id=root.0 \ -device ioh3420,bus=pcie.0,addr=1c.3,multifunction=on,port=2,chassis=3,id=root.1 \ -device nec-usb-xhci,multifunction=on,addr=1c.4,bus=pcie.0,id=usb3 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,romfile=/etc/qemu/vbios/Powercolor.HD7770.1024.120418.rom,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=pcie.0,multifunction=on \ -device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.6,multifunction=on \ -device vfio-pci,host=05:00.0,bus=pcie.0,addr=1c.7,multifunction=on \ -drive id=bitlocker_keys,file=/dev/loop0,if=none -device usb-storage,drive=bitlocker_keys,bus=usb3.0 \ -netdev type=tap,id=guest0,vhost=on,ifname="${IFACE}" -device virtio-net-pci,netdev=guest0,mac="${macaddr}" ${options} ; do echo "Qemu crashed with exit code $?. Respawning.." >&2 sleep 5 done

I hope it helps somebody who tries the same :-)


I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

My Sartup command is now:

until /usr/bin/sudo /usr/bin/nice -n 0 /usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -monitor stdio -nographic \ -balloon none -mem-path /dev/hugepages -mem-prealloc -m 10240 -k de -cpu host -smp 8,sockets=1,cores=8,threads=1 \ -bios /usr/share/qemu/bios.bin-1.7.5 -realtime mlock=on -vga none -D /var/log/qemu-out.log -boot menu=on -usb \ -usbdevice host:046d:c517 -usbdevice host:093a:2510 -usbdevice host:1a2c:0023 \ -device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \ -device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \ -device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=1,id=root.0 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ -device vfio-pci,host=03:00.0,bus=pcie.0,addr=1c.6,multifunction=on \ -device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.7,multifunction=on \ -drive id=bitlocker_keys,file=/dev/loop0,if=none -device usb-storage,drive=bitlocker_keys \ -netdev type=tap,id=guest0,vhost=on,ifname="${IFACE}" -device virtio-net-pci,netdev=guest0,mac="${macaddr}" ${options} ; do echo "Qemu crashed with exit code $?. Respawning.." >&2 sleep 5 done

*EDIT* Since the Card have a switchable Bios (UEFI/Compatible) i wanted to tell you all that both settings work.

Last edited by Child_of_Sun (2014-11-10 22:45:00)

Offline

#3202 2014-11-11 02:11:44

s00pcan
Member
Registered: 2014-11-11
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got this working, but I'm getting a jerky mouse input.  Video is smooth and I can play games fine.  What's the usual thing to try for this?

Offline

#3203 2014-11-11 09:46:50

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
winie wrote:

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.

I have been thinking about the whole nvidia situation and came up with two possible solutions. However I don't know enough about the details of PCI-E and emulation in general to tell if they are feasible or not.

The first idea is to "softmod" a GeForce to a Quadro by tapping in the vfio driver or pcie subsystem of the linux kernel. So instead of hardmodding the pci id with a soldering iron, you provide a fake pci id to the guest.

The second idea is mostly a hack. Since the nvidia driver checks the cpuid for hv_* features, would it be possible to patch qemu to change the cpuid during runtime? So during bootup when the nvidia driver is initialized, the hv_* features are hidden and only enabled after two minutes or via the qemu console.

Maybe @aw or others could comment if one of these approaches could work and which component of the stack needs to be patched.

Offline

#3204 2014-11-11 10:13:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

The first idea is to "softmod" a GeForce to a Quadro by tapping in the vfio driver or pcie subsystem of the linux kernel. So instead of hardmodding the pci id with a soldering iron, you provide a fake pci id to the guest.

http://www.eevblog.com/forum/chat/hacki … nterparts/
For SOME cards and vendors, you also need to provide the "correct" VBIOS. Interesting idea, actually.
It gets funnier as changing the PCI ID physically just makes the firmware blob work different, providing additional stuff. The chips are identical.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3205 2014-11-11 10:14:19

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
winie wrote:

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.

I see your point. But on the other hand I hate the fact that Nvidia is crippling the possible performance to boost sales of their professional series cards...

Flyser wrote:

I have been thinking about the whole nvidia situation and came up with two possible solutions. However I don't know enough about the details of PCI-E and emulation in general to tell if they are feasible or not.

The first idea is to "softmod" a GeForce to a Quadro by tapping in the vfio driver or pcie subsystem of the linux kernel. So instead of hardmodding the pci id with a soldering iron, you provide a fake pci id to the guest.

The second idea is mostly a hack. Since the nvidia driver checks the cpuid for hv_* features, would it be possible to patch qemu to change the cpuid during runtime? So during bootup when the nvidia driver is initialized, the hv_* features are hidden and only enabled after two minutes or via the qemu console.

Maybe @aw or others could comment if one of these approaches could work and which component of the stack needs to be patched.

The first idea sounds doable, the second approach I agree seems like a flaky hack. But before trying to hack something together - Are there some numbers in which way the hyper-v enlightments increase the performance of a typical gaming machine scenario? There are some stats included in http://www.linux-kvm.org/wiki/images/0/ … hyperv.pdf which show partly significant differences, but it is not clear for me if these tests are relevant for a gaming rig?

Offline

#3206 2014-11-11 10:25:17

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Are there some numbers in which way the hyper-v enlightments increase the performance of a typical gaming machine scenario? There are some stats included in http://www.linux-kvm.org/wiki/images/0/ … hyperv.pdf which show partly significant differences, but it is not clear for me if these tests are relevant for a gaming rig?

https://bbs.archlinux.org/viewtopic.php … 0#p1383040

Offline

#3207 2014-11-11 10:56:05

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Good afternoon everyone smile

@aw

Regarding X99 quirks , I recompiled 3.18-rc4 , and instead of adding the ACS patch , I edited the file "drivers/pci/quirks.c" and add these 2 lines :

/* Wellsburg (X99) PCH */ 0x8d10, 0x8d16, 0x8d18, 0x8d1c,

I added them exactly after the X79 ones . And it broke the large group just like the ACS patch did !

VMs rebooted fine , and system is stable (uptill now) .

lsgroup :

### Group 0 ### ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02) ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02) ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02) ### Group 1 ### ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02) ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02) ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02) ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02) ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers [8086:2fe4] (rev 02) ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers [8086:2fe5] (rev 02) ### Group 2 ### ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02) ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02) ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02) ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02) ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02) ### Group 3 ### ff:10.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02) ff:10.1 Performance counters [1101]: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02) ff:10.5 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02) ff:10.6 Performance counters [1101]: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02) ff:10.7 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02) ### Group 4 ### ff:12.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02) ff:12.1 Performance counters [1101]: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02) ### Group 5 ### ff:13.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086:2fa8] (rev 02) ff:13.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086:2f71] (rev 02) ff:13.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02) ff:13.3 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02) ff:13.4 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02) ff:13.5 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02) ff:13.6 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02) ff:13.7 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02) ### Group 6 ### ff:14.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02) ff:14.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02) ff:14.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02) ff:14.3 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02) ff:14.6 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02) ff:14.7 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02) ### Group 7 ### ff:15.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02) ff:15.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02) ff:15.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02) ff:15.3 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02) ### Group 8 ### ff:16.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086:2f68] (rev 02) ff:16.6 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02) ff:16.7 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02) ### Group 9 ### ff:17.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02) ff:17.4 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02) ff:17.5 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02) ff:17.6 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02) ff:17.7 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02) ### Group 10 ### ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02) ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02) ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02) ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02) ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02) ### Group 11 ### ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02) ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02) ### Group 12 ### 00:00.0 Host bridge [0600]: Intel Corporation Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02) ### Group 13 ### 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02) ### Group 14 ### 00:02.0 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 [8086:2f04] (rev 02) ### Group 15 ### 00:02.2 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 [8086:2f06] (rev 02) ### Group 16 ### 00:03.0 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) ### Group 17 ### 00:03.2 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02) ### Group 18 ### 00:05.0 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02) 00:05.1 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02) 00:05.2 System peripheral [0880]: Intel Corporation Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02) 00:05.4 PIC [0800]: Intel Corporation Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02) ### Group 19 ### 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05) ### Group 20 ### 00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05) ### Group 21 ### 00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I218-V [8086:15a1] (rev 05) ### Group 22 ### 00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05) ### Group 23 ### 00:1b.0 Audio device [0403]: Intel Corporation C610/X99 series chipset HD Audio Controller [8086:8d20] (rev 05) ### Group 24 ### 00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5) ### Group 25 ### 00:1c.3 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #4 [8086:8d16] (rev d5) ### Group 26 ### 00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5) ### Group 27 ### 00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5) ### Group 28 ### 00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05) ### Group 29 ### 00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d47] (rev 05) 00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05) 00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05) ### Group 30 ### 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1) 05:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1) ### Group 31 ### 04:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03) ### Group 32 ### 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) ### Group 33 ### 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) 01:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1) ### Group 34 ### 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:118f] ### Group 35 ### 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:118f] 09:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) ### Group 36 ### 08:02.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:118f] 0a:00.0 Network controller [0280]: Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter [14e4:43a0] (rev 03) ### Group 37 ### 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:118f] ### Group 38 ### 08:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:118f] ### Group 39 ### 0d:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) ### Group 40 ### 0e:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) ### Group 41 ### 0e:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) ### Group 42 ### 0e:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) ### Group 43 ### 0e:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8608] (rev ba) ### Group 44 ### 0f:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) ### Group 45 ### 10:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) ### Group 46 ### 11:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) ### Group 47 ### 12:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) ### Group 48 ### 13:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]

So I think it is safe to add these to the quirks file upstream .

Thank you .

EDIT :

When rebooting the host I recieve :

genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:10:00.0)) vs. 00000000 (vfio-intx(0000:0f:00.0))

And it is related to the USB PCI-E card I installed two days ago . This error prevents my 2nd VM from launching (It shares another controller from the same USB PCI-E card)

Tried booting with "options vfio_pci nointxmask=1" , both VMs boot fine , but the sound becomes choppy and useless .

Last edited by Denso (2014-11-11 11:20:05)

Offline

#3208 2014-11-11 10:56:12

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
TripleSpeeder wrote:

Are there some numbers in which way the hyper-v enlightments increase the performance of a typical gaming machine scenario? There are some stats included in http://www.linux-kvm.org/wiki/images/0/ … hyperv.pdf which show partly significant differences, but it is not clear for me if these tests are relevant for a gaming rig?

https://bbs.archlinux.org/viewtopic.php … 0#p1383040

Doh... 20%! That's definitely worth a closer look - Thanks for the pointer.

Offline

#3209 2014-11-11 10:56:56

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah from my experience is also about 20% on cpu bound applications (dota2).

Offline

#3210 2014-11-11 11:57:40

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do i have to explicitly enable hyper-v extensions(enlightenment or what it's called), or it is automatically added if i use -cpu host?(provided i have an AMD trinity CPU)

Last edited by Duelist (2014-11-11 11:58:29)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3211 2014-11-11 12:18:53

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah u must enable it, with hv-time option (-cpu host,hv-time) if i remember correctly.

Offline

#3212 2014-11-11 12:29:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Yeah u must enable it, with hv-time option (-cpu host,hv-time) if i remember correctly.

But i can't find it in -cpu help. Does AMD support these at all? There is "hypervisor" CPUID flag in qemu-system-x86-64 -cpu help list.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3213 2014-11-11 12:36:02

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

You need to make sure you are using virtio disk in xml, then add ide cd-rom with virtio-drivers... like here

<disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/hdd2/win8.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/virtio-win-0.1-81.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

At windows setup, when it asks where to install, choose load driver and then point it to mounted cd-rom with viritio drivers.


Hi there! Thanks for posting part of your XML file. Unfortunately, it didn't work. Not only did it not ask for the virtio Drivers, it also doesn't find my storage - a physical device. This is part of my XML file where I made some changes:


<domain type='kvm'>
  <name>win8.1</name>
  <uuid>502f5be4-0126-4a32-8d63-3f8d06a1c80a</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/win8.1-OVMF.fd</loader>
  </os>
....
   <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mapper/VG-kvmwin'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/windows.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/virtio.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>

Can you please have a look? Is there something missing or wrong? Thank you very much!

Offline

#3214 2014-11-11 16:37:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
aw wrote:
winie wrote:

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.

I have been thinking about the whole nvidia situation and came up with two possible solutions. However I don't know enough about the details of PCI-E and emulation in general to tell if they are feasible or not.

The first idea is to "softmod" a GeForce to a Quadro by tapping in the vfio driver or pcie subsystem of the linux kernel. So instead of hardmodding the pci id with a soldering iron, you provide a fake pci id to the guest.

The second idea is mostly a hack. Since the nvidia driver checks the cpuid for hv_* features, would it be possible to patch qemu to change the cpuid during runtime? So during bootup when the nvidia driver is initialized, the hv_* features are hidden and only enabled after two minutes or via the qemu console.

Maybe @aw or others could comment if one of these approaches could work and which component of the stack needs to be patched.

I'm going to tread lightly here, but I will confirm that PCI config space of assigned devices is a combination of direct passthrough, emulation, and virtualization.  PCI SR-IOV devices, by the spec, don't actually have PCI vendor and device IDs, it's up to system software/VMM to provide them.  Likewise, it's a trivial matter to virtualize the PCI config space IDs - http://fpaste.org/149722/72280914/  In the vast majority of cases, modifying the vendor ID is a bad idea, but I include it to be complete.

On a completely separate note, I'll mention that PCI device IDs are also stored in the PCI ROM and the ROM includes a checksum.  In the unlikely event that you want to expose the ROM on a modified device, the device ID and checksum need to be updated.

Also note that supported Quadro devices operate in a secondary graphics mode, ie. in addition to emulated VGA.  I've also never had much luck getting them to work on Q35 machine types, but it's on my todo list to look further into the Code 12 they get there.  They work fine on a 440FX machine.

I don't think we have much opportunity to try to be creative in exposing hyper-v features, both the core guest OS and the graphics driver are going to be probing this at similar times and I doubt we have much visibility to which is calling for it.  I also tried to do some benchmarks with Unigine/3DMark so I could complain properly about the change, but I was unable to come up with a significant difference.  Perhaps this is because they're just benchmarks, maybe it's real.  We did note a while ago a fairly significant differences in Borderlands2, but that was also when we were fighting the lazy debug register use.  That 20% number might be much, much smaller now that we've optimized that path in the hypervisor.  It should be easy for someone to back down to a pre-340 driver version and test with and without hyper-v extensions for various games.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3215 2014-11-11 16:48:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Good afternoon everyone smile

@aw

Regarding X99 quirks , I recompiled 3.18-rc4 , and instead of adding the ACS patch , I edited the file "drivers/pci/quirks.c" and add these 2 lines :

/* Wellsburg (X99) PCH */ 0x8d10, 0x8d16, 0x8d18, 0x8d1c,

I added them exactly after the X79 ones . And it broke the large group just like the ACS patch did !

VMs rebooted fine , and system is stable (uptill now) .

lsgroup :

<snip>

So I think it is safe to add these to the quirks file upstream .

The groups are going to be split by that patch regardless of whether it's safe to do so.  We still need to wait for confirmation from Intel that X99 PCH root ports use the same programming algorithm and most importantly, provide the required isolation.

EDIT :

When rebooting the host I recieve :

genirq: Flags mismatch irq 17. 00000000 (vfio-intx(0000:10:00.0)) vs. 00000000 (vfio-intx(0000:0f:00.0))

And it is related to the USB PCI-E card I installed two days ago . This error prevents my 2nd VM from launching (It shares another controller from the same USB PCI-E card)

Tried booting with "options vfio_pci nointxmask=1" , both VMs boot fine , but the sound becomes choppy and useless .

When you rebooted the host you saw this error, or do you mean after you rebooted with the new quirk you ran into this problem.  The description doesn't make much sense to me.  The error indicates that both devices tried to register with no flags (00000000), which I think means exclusive IRQ, which means the INTx disable masking test failed.  By using nointxmask=1, you're telling vfio to use the same path that it should have been using based on those flags, so I'm not sure why it makes a difference.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3216 2014-11-11 18:13:14

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested couple of games on 340 driver with hyperv on, and most games and benchmarks use GPU at 100% and CPU not so much, so that might be reason why u don't see much difference, as I said before Dota2 that depends on single/dual core performance works 20% better with hyperv on.

Last edited by slis (2014-11-11 18:17:44)

Offline

#3217 2014-11-11 20:14:57

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

By the way, how do you guys enable hyperv? Im trying "-cpu host,hv-time" and its always saying that is is running as a VM in task manager?

Offline

#3218 2014-11-11 21:50:59

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

When you rebooted the host you saw this error, or do you mean after you rebooted with the new quirk you ran into this problem.  The description doesn't make much sense to me.  The error indicates that both devices tried to register with no flags (00000000), which I think means exclusive IRQ, which means the INTx disable masking test failed.  By using nointxmask=1, you're telling vfio to use the same path that it should have been using based on those flags, so I'm not sure why it makes a difference.

It happened before and after I applied the new quirk . Sorry for not mentioning that sad

It works with nointxmask=1 , but with sound being choppy and slow .

Also this might be related :

Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 73 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 73 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 74 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 73 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 74 for MSI/MSI-X Nov 10 13:31:27 srv1 kernel: vfio-pci 0000:0f:00.0: irq 75 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 73 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 67 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 68 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 69 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 70 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 71 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 72 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 73 for MSI/MSI-X Nov 10 13:31:47 srv1 kernel: vfio-pci 0000:10:00.0: irq 74 for MSI/MSI-X

Sorry for my lack of knowledge , but I see that both devices use some identical IRQs . Maybe that's the issue here ?

EDIT :

One side note , I use these modules options /etc/modprobe.d/kvm.conf :

options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Last edited by Denso (2014-11-11 21:53:42)

Offline

#3219 2014-11-11 22:52:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

By the way, how do you guys enable hyperv? Im trying "-cpu host,hv-time" and its always saying that is is running as a VM in task manager?

hv-time enables one of the hyper-v extensions, iow the guest thinks that it's running on the hyper-v hypervisor and enables some paravirtual features for the hypervisor.  KVM also supports these features.  AFAIK, nobody here is attempting to run hyper-v inside a KVM guest.  That might be possible with nested virtualization support, but it's not really relevant to the topic in this thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3220 2014-11-11 22:59:33

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:

By the way, how do you guys enable hyperv? Im trying "-cpu host,hv-time" and its always saying that is is running as a VM in task manager?

hv-time enables one of the hyper-v extensions, iow the guest thinks that it's running on the hyper-v hypervisor and enables some paravirtual features for the hypervisor.  KVM also supports these features.  AFAIK, nobody here is attempting to run hyper-v inside a KVM guest.  That might be possible with nested virtualization support, but it's not really relevant to the topic in this thread.

Yeah I'm sorry that its not relevant. What I want is nested virtualization. I thought it's easy-peasy for everyone here? It shouldn't be that hard.. I added "kvm-intel.nested=1" to grub and thats all someone can do I believe?

Offline

#3221 2014-11-11 23:03:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
aw wrote:
devianceluka wrote:

By the way, how do you guys enable hyperv? Im trying "-cpu host,hv-time" and its always saying that is is running as a VM in task manager?

hv-time enables one of the hyper-v extensions, iow the guest thinks that it's running on the hyper-v hypervisor and enables some paravirtual features for the hypervisor.  KVM also supports these features.  AFAIK, nobody here is attempting to run hyper-v inside a KVM guest.  That might be possible with nested virtualization support, but it's not really relevant to the topic in this thread.

Yeah I'm sorry that its not relevant. What I want is nested virtualization. I thought it's easy-peasy for everyone here? It shouldn't be that hard.. I added "kvm-intel.nested=1" to grub and thats all someone can do I believe?

You might need to add +vmx to your -cpu parameters for QEMU


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3222 2014-11-11 23:05:16

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:
aw wrote:

hv-time enables one of the hyper-v extensions, iow the guest thinks that it's running on the hyper-v hypervisor and enables some paravirtual features for the hypervisor.  KVM also supports these features.  AFAIK, nobody here is attempting to run hyper-v inside a KVM guest.  That might be possible with nested virtualization support, but it's not really relevant to the topic in this thread.

Yeah I'm sorry that its not relevant. What I want is nested virtualization. I thought it's easy-peasy for everyone here? It shouldn't be that hard.. I added "kvm-intel.nested=1" to grub and thats all someone can do I believe?

You might need to add +vmx to your -cpu parameters for QEMU

Oh forgot. And that yes. Those 2 things. Even with that it doesnt work.

But +vmx is already enabled if its one of the supported features by the CPU (it doesnt emulate it, but pass it through).

EDIT: I'm using i7-4790S. In ESXi theres an option for this and it then shows normally in task manager like it would be running bare-metal - with L1,L2,L3 cache sizes... Where in here it shows it is running as a VM)

Last edited by devianceluka (2014-11-11 23:07:49)

Offline

#3223 2014-11-11 23:26:37

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear all,

I've succeded in setting up PCI passthrough with a Gigabyte board, i7 3770 (integrated iGPU) and a Gigabyte GTX Titan OC card. Now, I'm trying to use the same board with a Xeon 1390v2 (no iGPU) the same Gigabyte Titan OC card and a second video card (Gigabyte GT 730). So basically two Nvidia cards installed. I would like to passthrough the Titan which sits in 0000:0100.0. Unfortunately I get the following errors in dmesg while getting black screen on the second monitor:

[  113.148867] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  113.150151] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0x98000000-0x99ffffff 64bit pref]
[  113.150980] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[  116.098894] vfio-pci 0000:01:00.0: Invalid ROM contents
[  116.099019] vfio-pci 0000:01:00.0: Invalid ROM contents

I'm using the linux-mainline package incl. ACS patch. Thx.

Last edited by dakabali (2014-11-11 23:27:59)

Offline

#3224 2014-11-11 23:35:10

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:

Dear all,

I've succeded in setting up PCI passthrough with a Gigabyte board, i7 3770 (integrated iGPU) and a Gigabyte GTX Titan OC card. Now, I'm trying to use the same board with a Xeon 1390v2 (no iGPU) the same Gigabyte Titan OC card and a second video card (Gigabyte GT 730). So basically two Nvidia cards installed. I would like to passthrough the Titan which sits in 0000:0100.0. Unfortunately I get the following errors in dmesg while getting black screen on the second monitor:

[  113.148867] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  113.150151] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0x98000000-0x99ffffff 64bit pref]
[  113.150980] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[  116.098894] vfio-pci 0000:01:00.0: Invalid ROM contents
[  116.099019] vfio-pci 0000:01:00.0: Invalid ROM contents

I'm using the linux-mainline package incl. ACS patch. Thx.

Try passing a ROM file using :

romfile=/PATH/TO/ROM/FILE.rom

Like this :

-device vfio-pci,host=01:00.0,multifunction=on,romfile=/PATH/TO/ROM/FILE.rom

You can obtain ROM files from TechPowerUp website .

Offline

#3225 2014-11-12 01:08:02

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
dakabali wrote:

Dear all,

I've succeded in setting up PCI passthrough with a Gigabyte board, i7 3770 (integrated iGPU) and a Gigabyte GTX Titan OC card. Now, I'm trying to use the same board with a Xeon 1390v2 (no iGPU) the same Gigabyte Titan OC card and a second video card (Gigabyte GT 730). So basically two Nvidia cards installed. I would like to passthrough the Titan which sits in 0000:0100.0. Unfortunately I get the following errors in dmesg while getting black screen on the second monitor:

[  113.148867] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  113.150151] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0x98000000-0x99ffffff 64bit pref]
[  113.150980] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[  116.098894] vfio-pci 0000:01:00.0: Invalid ROM contents
[  116.099019] vfio-pci 0000:01:00.0: Invalid ROM contents

I'm using the linux-mainline package incl. ACS patch. Thx.

Try passing a ROM file using :

romfile=/PATH/TO/ROM/FILE.rom

Like this :

-device vfio-pci,host=01:00.0,multifunction=on,romfile=/PATH/TO/ROM/FILE.rom

You can obtain ROM files from TechPowerUp website .

Hello,

thanks for the quick answer. Getting the card rom helped to eliminate the errors regarding the "Invalid ROM contents". However, the situation didn't change - still no signal on the second monitor. Instead of enabling my Titan card /should be something like " vfio-pci 0000:01:00.0: enabling device (0000 -> 0001)" like for the audio 01:00.1/ it drops a memory reservation error:

[  179.361851] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  179.363102] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0x98000000-0x99ffffff 64bit pref]
[  179.365305] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

Furthermore starting Qemu says: "VFIO 0000:01:00.0 BAR 3 mmap unsupported. Performance may be slow"

Any idea? Thx.

Last edited by dakabali (2014-11-12 01:08:52)

Offline

#3226 2014-11-12 01:13:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:

thanks for the quick answer. Getting the card rom helped to eliminate the errors regarding the "Invalid ROM contents". However, the situation didn't change - still no signal on the second monitor. Instead of enabling my Titan card /should be something like " vfio-pci 0000:01:00.0: enabling device (0000 -> 0001)" like for the audio 01:00.1/ it drops a memory reservation error:

[  179.361851] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  179.363102] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0x98000000-0x99ffffff 64bit pref]
[  179.365305] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

Furthermore starting Qemu says: "VFIO 0000:01:00.0 BAR 3 mmap unsupported. Performance may be slow"

Any idea? Thx.

I believe Duelist has figured out that this typically means you have something like vesafb claiming resources of the card you're trying to assign, which isn't going to work.  Look in /proc/iomem for the driver using pieces of your GPU and prevent it from happening.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3227 2014-11-12 01:33:32

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

My Sartup command is now:

until /usr/bin/sudo /usr/bin/nice -n 0 /usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -monitor stdio -nographic \ -balloon none -mem-path /dev/hugepages -mem-prealloc -m 10240 -k de -cpu host -smp 8,sockets=1,cores=8,threads=1 \ -bios /usr/share/qemu/bios.bin-1.7.5 -realtime mlock=on -vga none -D /var/log/qemu-out.log -boot menu=on -usb \ -usbdevice host:046d:c517 -usbdevice host:093a:2510 -usbdevice host:1a2c:0023 \ -device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \ -device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \ -device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=1,id=root.0 \ -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ -device vfio-pci,host=03:00.0,bus=pcie.0,addr=1c.6,multifunction=on \ -device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.7,multifunction=on \ -drive id=bitlocker_keys,file=/dev/loop0,if=none -device usb-storage,drive=bitlocker_keys \ -netdev type=tap,id=guest0,vhost=on,ifname="${IFACE}" -device virtio-net-pci,netdev=guest0,mac="${macaddr}" ${options} ; do echo "Qemu crashed with exit code $?. Respawning.." >&2 sleep 5 done

*EDIT* Since the Card have a switchable Bios (UEFI/Compatible) i wanted to tell you all that both settings work.

Only the Performance is bad :-(, so i switched back to native Windows for gaming, before with the HD7770s Unigine Heaven 3.01 (Standard Settings) had a score of ~1422 in the VM and ~1522 native, the faster Card had ~822 in the VM and ~864 native, that was good for me, because the Performance loss wasn't great.

Now with the 280x i have ~1522 Points in the VM and ~2880 Points native (Tested with the UEFI Bios from the Card), i know there are too many Variables to find the "Bottleneck" by myself, so i only wanted to tell it here in the Thread.

Offline

#3228 2014-11-12 07:27:45

anickname
Member
Registered: 2014-01-07
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Child_of_Sun wrote:

I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

Only the Performance is bad :-(, so i switched back to native Windows for gaming, before with the HD7770s Unigine Heaven 3.01 (Standard Settings) had a score of ~1422 in the VM and ~1522 native, the faster Card had ~822 in the VM and ~864 native, that was good for me, because the Performance loss wasn't great.

Now with the 280x i have ~1522 Points in the VM and ~2880 Points native (Tested with the UEFI Bios from the Card), i know there are too many Variables to find the "Bottleneck" by myself, so i only wanted to tell it here in the Thread.

I just run the test on my VM using UEFI, video card Radeon 270X. The score was:
Heaven Benchmark v3.0 Basic

FPS: 102.9
Scores: 2592
Min FPS: 41.3
Max FPS: 189.9

Hardware

Binary:
Windows 32bit Visual C++ 1600 Release Mar 7 2012

Operating system:
Windows NT 6.2 (build 9200) 64bit

CPU model:
Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

CPU flags:
3399MHz MMX SSE SSE2 SSE3 SSSE3 SSE41 SSE42 HTT

GPU model: AMD Radeon R9 200 Series 14.301.1001.0 2048Mb


Settings
Render: direct3d11
Mode: 1680x1050 fullscreen
Shaders: high
Textures: high
Filter: trilinear
Anisotropy: 4x
Occlusion: enabled
Refraction: enabled
Volumetric: enabled
Tessellation: disabled

The performance is good.

Offline

#3229 2014-11-12 08:16:48

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:
slis wrote:

You need to make sure you are using virtio disk in xml, then add ide cd-rom with virtio-drivers... like here

<disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/hdd2/win8.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/virtio-win-0.1-81.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

At windows setup, when it asks where to install, choose load driver and then point it to mounted cd-rom with viritio drivers.


Hi there! Thanks for posting part of your XML file. Unfortunately, it didn't work. Not only did it not ask for the virtio Drivers, it also doesn't find my storage - a physical device. This is part of my XML file where I made some changes:


<domain type='kvm'>
  <name>win8.1</name>
  <uuid>502f5be4-0126-4a32-8d63-3f8d06a1c80a</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/win8.1-OVMF.fd</loader>
  </os>
....
   <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mapper/VG-kvmwin'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/windows.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/virtio.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>

Can you please have a look? Is there something missing or wrong? Thank you very much!

Does anyone know why it's not finding the virtio drivers and my storage device? I also use lvm ...


any help is greatly appreciated!

Offline

#3230 2014-11-12 08:51:00

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's normal to not detect any hdd's (storage drives in windows setup), cause windows doesn't have viritio drivers, thats why u mount viritio drivers as ide cdrom, so when windows shows select where to install screen, u got there option LOAD DRIVERS, click on it, select virtio cdrom, win8, drivers then select Red Hat Virtio Storage Driver and it should show your virtio disk.

Offline

#3231 2014-11-12 14:52:15

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I believe Duelist has figured out that this typically means you have something like vesafb claiming resources of the card you're trying to assign, which isn't going to work.  Look in /proc/iomem for the driver using pieces of your GPU and prevent it from happening.

All right, I changed the Gigabyte Dual UEFI BIOS setting "legacy only" to "legacy first" regarding the priorities of my NVIDIA cards. This change seems to select my GT730 (I suppose it doesn't support UEFI anyway) as primary video card over the Titan and this helped to eliminate the BAR error. However, still no signal on the second monitor and having started the vm dmsg delivers the following one line:

[ 9650.055230] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

So I managed to decrease the number of error lines to one without coming any closer to solve the problem... "great" :-(  Any idea?

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

Last edited by dakabali (2014-11-12 15:13:44)

Offline

#3232 2014-11-12 16:50:16

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anickname wrote:
Child_of_Sun wrote:

I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

Only the Performance is bad :-(, so i switched back to native Windows for gaming, before with the HD7770s Unigine Heaven 3.01 (Standard Settings) had a score of ~1422 in the VM and ~1522 native, the faster Card had ~822 in the VM and ~864 native, that was good for me, because the Performance loss wasn't great.

Now with the 280x i have ~1522 Points in the VM and ~2880 Points native (Tested with the UEFI Bios from the Card), i know there are too many Variables to find the "Bottleneck" by myself, so i only wanted to tell it here in the Thread.

I just run the test on my VM using UEFI, video card Radeon 270X. The score was:
Heaven Benchmark v3.0 Basic

FPS: 102.9
Scores: 2592
Min FPS: 41.3
Max FPS: 189.9

Hardware

Binary:
Windows 32bit Visual C++ 1600 Release Mar 7 2012

Operating system:
Windows NT 6.2 (build 9200) 64bit

CPU model:
Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

CPU flags:
3399MHz MMX SSE SSE2 SSE3 SSSE3 SSE41 SSE42 HTT

GPU model: AMD Radeon R9 200 Series 14.301.1001.0 2048Mb


Settings
Render: direct3d11
Mode: 1680x1050 fullscreen
Shaders: high
Textures: high
Filter: trilinear
Anisotropy: 4x
Occlusion: enabled
Refraction: enabled
Volumetric: enabled
Tessellation: disabled

The performance is good.

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC big_smile.

My Actual Configuration is:
Graka: Radeon R9 280X Tri-X OC
Mainboard: Asrock 970 Extreme4
HDDs: Sandisk 64GB SSD, Seagate Barracuda 7200.12 1TB, Seagate Barracuda LP 1TB
CPU: Amd FX-8350
Burners: Optiarc DVD-RW, LG Blueray
Ram: 4x4 GB Transcend 1333MHz


The Devices i had passthrough to the VM:
-usbdevice host:046d:c517 -usbdevice host:093a:2510 -usbdevice host:1a2c:0023 \ Keyboard and Mouse ( 2 Different Keyboards, only 1 at a time)
-device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \ (Onboard AHCI Controller, only works in AHCI Mode without Bios Option Rom)
-device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \ (Onboard Audio Controller)
-device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=1,id=root.0 \ (Graphics Card Port)
-device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ (Graphics Card)
-device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ (Graphics Card Audio)
-device vfio-pci,host=03:00.0,bus=pcie.0,addr=1c.6,multifunction=on \ (Usb 3.0 Controller 1)
-device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.7,multifunction=on \ (Usb 3.0 Controller 2)

The first thing i would have tested, is  a different niceness, since i set it from 10 to 0 when i changed the Cards (setting it back to 10).

Maybe i test it another Time (I have backed up my Linux and repartitioned the Drives to use them all under Windows [except for the root drive, my second ssd, i wiped it and added it to my Mailserver as root Device, because of it's poor Performance]), but for the moment i am Happy with using Windows Native :-)

*EDIT* Some Corrections.

Last edited by Child_of_Sun (2014-11-12 16:56:36)

Offline

#3233 2014-11-12 16:56:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Child_of_Sun wrote:

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC big_smile.

devianceluka and I were talking on irc yesterday and we think that AMD ZeroCore works for devices assigned with vfio.  We still have all the same reasons that it doesn't always work on bare-metal though, in my case probably that I started using the hdmi audio.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3234 2014-11-12 17:04:55

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Child_of_Sun wrote:

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC big_smile.

devianceluka and I were talking on irc yesterday and we think that AMD ZeroCore works for devices assigned with vfio.  We still have all the same reasons that it doesn't always work on bare-metal though, in my case probably that I started using the hdmi audio.

It's mainly because of the second Graphics card for the Host (since i now have a second PC as Mailserver, i wanted to save as much as Possible [I haven't got a Onboard Graphics device] and the second Monitor. :-)

Offline

#3235 2014-11-12 17:46:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

Well, that's simple.
Your distributive has kernel mode setting or some other kind of framebuffer enabled.
It goes like this:
@ the system boot, one of the cards gets assigned as a primary one, and legacy VGA output methods are tied to it. Then the kernel boots up, claims those resources as aw said. Then you start a VM, it tries to pin the whole device, but fails, outputting you that first error line with BAR3.

Gigabyte motherboards (ASUS, for example, doesn't have this in their UEFI) allow you to manually choose which GPU will be the primary, instead of assuming that "first 16x pci-e port is primary GPU".

Actually, swapping the cards physically would also help and it would be required to do so if you wouldn't have that UEFI setting. Or you could do what aw said: look up /proc/iomem, see some vesafb or nouveau or some other KMS/framebuffer stuff sitting on VGA MMIO range of the card in question, and go disable it. Oh noes, your system would usually lack a graphical loading screen(like plymouth). I think we can sacrifice that.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3236 2014-11-12 17:49:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:

[ 9650.055230] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

BTW, this is just informational, not an error.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3237 2014-11-12 17:57:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Child_of_Sun wrote:

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC big_smile.

devianceluka and I were talking on irc yesterday and we think that AMD ZeroCore works for devices assigned with vfio.  We still have all the same reasons that it doesn't always work on bare-metal though, in my case probably that I started using the hdmi audio.

Well i can confirm that!
My crossfire setup has ULPS(ultra-low-power-state, actually it is zerocore itself) active, and it actually poweroffs the second, rarely used GPU. However, the first gpu doesn't change it's power state and doesn't adjust it's clocks, but that may be the metering issue here, as it requires some quirky way to check them.
The CPU also scales it's frequency nicely, without any problems so far. The variating frequency is seen inside the VM and on the host, yet cpu-z on VM tries to access some MSRs, which qemu doesn't pass, even when using -cpu host. That results in false multiplier and system bus clock meters, also i can't see the voltage and temperature. Neither i can do it on the host, thank you, AMD, for having such a quirky realization of thermometer, that it took A YEAR to fix native windows AMD OverDrive software.

Also, when using the whole block device(like hdd) or maybe even scsi/sata controller passed through, it is real to cast a spindown(and TRIM also) command to the block device, putting it in it's power saving mode.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3238 2014-11-12 22:03:07

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

When you rebooted the host you saw this error, or do you mean after you rebooted with the new quirk you ran into this problem.  The description doesn't make much sense to me.  The error indicates that both devices tried to register with no flags (00000000), which I think means exclusive IRQ, which means the INTx disable masking test failed.  By using nointxmask=1, you're telling vfio to use the same path that it should have been using based on those flags, so I'm not sure why it makes a difference.

It happened before and after I applied the new quirk . Sorry for not mentioning that sad

It works with nointxmask=1 , but with sound being choppy and slow .

Also this might be related :

...

Sorry for my lack of knowledge , but I see that both devices use some identical IRQs . Maybe that's the issue here ?

EDIT :

One side note , I use these modules options /etc/modprobe.d/kvm.conf :

options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

So I kept digging ! smile

And I found the problem . the BIOS does a "Full Initialization" of the USB card upon booting , which causes all these issues . Setting "USB Initialization" to "Partial Initalization" in BIOS makes everything working like a charm again .

Hope this helps ! smile

Last edited by Denso (2014-11-12 22:37:42)

Offline

#3239 2014-11-12 22:07:39

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Your distributive has kernel mode setting or some other kind of framebuffer enabled.
It goes like this:
@ the system boot, one of the cards gets assigned as a primary one, and legacy VGA output methods are tied to it. Then the kernel boots up, claims those resources as aw said. Then you start a VM, it tries to pin the whole device, but fails, outputting you that first error line with BAR3.

I see. I find it confusing that Gigabyte calls this option "Legacy only"/"Legacy first" in their BIOS - I had no clue that this setting has something to do with the video card priorities... Trial and error has worked, anyway smile

Gigabyte motherboards (ASUS, for example, doesn't have this in their UEFI) allow you to manually choose which GPU will be the primary, instead of assuming that "first 16x pci-e port is primary GPU".

Right. However, the available GPU slots are called  "PEG11" "PEG12" and "PEG60" -  I haven't found anything in the motherboard manual what they stand for. Btw any of these was selected and saved, after the next reboot it set back to "Auto"...

Actually, swapping the cards physically would also help and it would be required to do so if you wouldn't have that UEFI setting.

No way. I've got a microATM and my Titan card has got literally no space in the 2nd PIC-e smile

Something else: can anyone tell me which is the latest working Nvidia driver pack for the passthrough setup? The latest avalable 344.65 installation breaks and falls back to 1024x768 after reboot. My Asus PG278Q monitor has come with 340.52 which works okay, but the Titan card will be shown as "Unknown PCI device" in the Win8.1 device manager.

Offline

#3240 2014-11-12 22:15:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:
Duelist wrote:

Your distributive has kernel mode setting or some other kind of framebuffer enabled.
It goes like this:
@ the system boot, one of the cards gets assigned as a primary one, and legacy VGA output methods are tied to it. Then the kernel boots up, claims those resources as aw said. Then you start a VM, it tries to pin the whole device, but fails, outputting you that first error line with BAR3.

I see. I find it confusing that Gigabyte calls this option "Legacy only"/"Legacy first" in their BIOS - I had no clue that this setting has something to do with the video card priorities... Trial and error has worked, anyway smile

"Legacy only vs Legacy first" sounds more like preference of PCI option ROM execution, not ordering of VGA device.  A PCI option ROM can contain multiple images, often one for legacy BIOS and one for UEFI.  That option sounds like it's selecting which one to prefer.  Using a similar option on my Asus board is how I determined my R7790 card came from XFX with a broken UEFI ROM.  The bare metal system boots fine with Legacy First, but if I switch it to opt for the UEFI ROM the screen never turns on.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3241 2014-11-12 23:18:05

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
aw wrote:
winie wrote:

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.

I have been thinking about the whole nvidia situation and came up with two possible solutions. However I don't know enough about the details of PCI-E and emulation in general to tell if they are feasible or not.

The first idea is to "softmod" a GeForce to a Quadro by tapping in the vfio driver or pcie subsystem of the linux kernel. So instead of hardmodding the pci id with a soldering iron, you provide a fake pci id to the guest.

The second idea is mostly a hack. Since the nvidia driver checks the cpuid for hv_* features, would it be possible to patch qemu to change the cpuid during runtime? So during bootup when the nvidia driver is initialized, the hv_* features are hidden and only enabled after two minutes or via the qemu console.

Maybe @aw or others could comment if one of these approaches could work and which component of the stack needs to be patched.

What about hacking the nvidia drivers?

Offline

#3242 2014-11-13 06:39:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You guys may already know the news about microsoft getting .net core opensource and giving out free visual studio in 2015. That visual studio, due to xamarin framework and stuff, comes with an android emulator! The android emulator in question is based on HyperV.

So, it gets even more interesting to get the nested virtualization running. Windows 7 does not support HyperV, so to test the nested - you'll need to upgrade.
BUT.
Nvidia?
As it emulates the droid, we'll definitely need to run at least opengl1.1(they don't have opengl 2.0 support yet, but it isn't released in the public either), and that means we must have hardware acceleration. That means we need the drivers working.
Is it really possible to passthrough an nvidia card into the VM and have nested hyperv virtualization support without buying you-know-what?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3243 2014-11-13 08:13:39

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I'm going to tread lightly here, but I will confirm that PCI config space of assigned devices is a combination of direct passthrough, emulation, and virtualization.  PCI SR-IOV devices, by the spec, don't actually have PCI vendor and device IDs, it's up to system software/VMM to provide them.  Likewise, it's a trivial matter to virtualize the PCI config space IDs - http://fpaste.org/149722/72280914/  In the vast majority of cases, modifying the vendor ID is a bad idea, but I include it to be complete.

On a completely separate note, I'll mention that PCI device IDs are also stored in the PCI ROM and the ROM includes a checksum.  In the unlikely event that you want to expose the ROM on a modified device, the device ID and checksum need to be updated.

IT WORKS! big_smile
I set the Quadro k4000 pci ids for my GTX660, supplied the VBIOS on the qemu cmdline and the latest nvidia quadro drivers work like a charm smile

aw wrote:

Also note that supported Quadro devices operate in a secondary graphics mode, ie. in addition to emulated VGA.  I've also never had much luck getting them to work on Q35 machine types, but it's on my todo list to look further into the Code 12 they get there.  They work fine on a 440FX machine.

Thanks for the hint as I would have given up on the initial black screen^^. But why is that? From my understanding real quadro devices work as primary graphics devices. Is it possible to use them as such in qemu as well?

Last edited by Flyser (2014-11-13 08:23:25)

Offline

#3244 2014-11-13 10:35:09

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

I set the Quadro k4000 pci ids for my GTX660

Can you explain how did you achieve that without soldering iron ?

Thank you .

Offline

#3245 2014-11-13 11:21:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Flyser wrote:

I set the Quadro k4000 pci ids for my GTX660

Can you explain how did you achieve that without soldering iron ?

Thank you .

aw wrote:

aw wrote:
I'm going to tread lightly here, but I will confirm that PCI config space of assigned devices is a combination of direct passthrough, emulation, and virtualization.  PCI SR-IOV devices, by the spec, don't actually have PCI vendor and device IDs, it's up to system software/VMM to provide them.  Likewise, it's a trivial matter to virtualize the PCI config space IDs - http://fpaste.org/149722/72280914/  In the vast majority of cases, modifying the vendor ID is a bad idea, but I include it to be complete.
On a completely separate note, I'll mention that PCI device IDs are also stored in the PCI ROM and the ROM includes a checksum.  In the unlikely event that you want to expose the ROM on a modified device, the device ID and checksum need to be updated.

aw wrote:

Also note that supported Quadro devices operate in a secondary graphics mode, ie. in addition to emulated VGA.  I've also never had much luck getting them to work on Q35 machine types, but it's on my todo list to look further into the Code 12 they get there.  They work fine on a 440FX machine.

I have a quadrified geforce 470gtx --> quadro 5000 and i have no problems either using it as primary on q35 (i do get boot messages using romfile=) or secondary on 440fx

Q35 primary:

xBl8Rhx.jpg

440FX secondary:

XlJ57LW.jpg

Offline

#3246 2014-11-13 12:05:59

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmmm .. changing the device IDs to Quadros + keep Hyper-V enhancements , but is it REALLY worth it ?  I did try to disable Hyper-V enhancements and I don't see any performance hit in any real world workload (Gaming , surfing & multimedia) . So I think I'll stick with kvm=off and keep my nVidia drivers up to date .

Offline

#3247 2014-11-13 12:55:08

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I have a quadrified geforce 470gtx --> quadro 5000 and i have no problems either using it as primary on q35 (i do get boot messages using romfile=) or secondary on 440fx

Huh?
Quadro 5000 core config: 352:44:40
GTX470 core config: 448:56:40.

Ummm.. Ummmm... like who gives a damn as long as the chip architecture is the same.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3248 2014-11-13 13:44:31

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:

I have a quadrified geforce 470gtx --> quadro 5000 and i have no problems either using it as primary on q35 (i do get boot messages using romfile=) or secondary on 440fx

Huh?
Quadro 5000 core config: 352:44:40
GTX470 core config: 448:56:40.

Ummm.. Ummmm... like who gives a damn as long as the chip architecture is the same.

Now wait for NVIDIA to check core configuration in their driver...

Offline

#3249 2014-11-13 21:14:16

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

It's normal to not detect any hdd's (storage drives in windows setup), cause windows doesn't have viritio drivers, thats why u mount viritio drivers as ide cdrom, so when windows shows select where to install screen, u got there option LOAD DRIVERS, click on it, select virtio cdrom, win8, drivers then select Red Hat Virtio Storage Driver and it should show your virtio disk.

thank you! Finally I got it smile So I successfully installed Windows 8.1 after some hiccups I managed to install the "Assassin's Creed Unity"-ready Nvidia drivers (344.65) without any error code (43).

BUT then:

After a simple host reboot I got the CODE 43 back and I couldn't make it work again.

This is what I changed in the XML:

<os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/win8.1-OVMF.fd</loader>
  </os>

<features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>

  </features>

<cpu mode='host-passthrough'>
  </cpu>
  <clock offset='utc'/>

after that I ran virt-manager and added Nvidia GPU and Audio.

I use latest linux-mainline and provided by NBHS, ovmf-git, qemu-git and libvirt ...

this is how I start arch:

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=07c30ed1-3204-4c6a-8gd1-22r8ae54t1f5 rw i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream quiet


Is there something I forgot or got wrong? I am this close to game in 4k in a virtual machine, please help smile

Offline

#3250 2014-11-13 21:24:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Is there something I forgot or got wrong? I am this close to game in 4k in a virtual machine, please help smile

Assuming they haven't added anything new, any of the following will cause a Code 43:

  <features>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>

  </features>

  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
  </clock>


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3251 2014-11-13 21:34:08

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Flyser wrote:

I set the Quadro k4000 pci ids for my GTX660

Can you explain how did you achieve that without soldering iron ?

Thank you .

I applied the patch aw wrote ( http://fpaste.org/149722/72280914/ ) and added x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/path/to/quadro-k4000-bios.rom to the gpu assignment in the qemu cmdline

Offline

#3252 2014-11-13 21:38:32

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi Alex, thank you, I got every command from your blog. That's what I did and at first it worked. After some reboot it doesn't. I reinstalled Windows 4-5 times today, but I couldn't replicate the success I initially had. I assume it must be something I did, I just don't know what ...


Does it matter WHEN I remove those lines? Or when I add the

    <kvm>
      <hidden state='on'/>
    </kvm>

line?

Last edited by 4kGamer (2014-11-13 21:39:17)

Offline

#3253 2014-11-13 21:45:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

hi Alex, thank you, I got every command from your blog. That's what I did and at first it worked. After some reboot it doesn't. I reinstalled Windows 4-5 times today, but I couldn't replicate the success I initially had. I assume it must be something I did, I just don't know what ...


Does it matter WHEN I remove those lines? Or when I add the

    <kvm>
      <hidden state='on'/>
    </kvm>

line?

Not that I'm aware of.  I just updated my 440FX/OVMF/8.1/GTX750 VM to 344.65 and it still works, so apparently no new subversion.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3254 2014-11-13 21:56:51

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, thank you. I'll keep on trying. Will post as soon as it (hopefully) works.

Offline

#3255 2014-11-14 04:39:46

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

I applied the patch aw wrote ( http://fpaste.org/149722/72280914/ ) and added x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/path/to/quadro-k4000-bios.rom to the gpu assignment in the qemu cmdline

Thanks for sharing ! As I said , I saw no real life difference with Hv enhancements on or off (Maybe those who encode videos alot would) . I'll keep my stable configuration for now smile

EDIT :

@aw

I recieved this error two times uptil now :

[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018 [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID) [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: device [8086:2f08] error status/mask=00000040/00002000 [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: [ 6] Bad TLP


I didn't notice anything while using the VM or the host though . I'm using 3.18-rc4 + ACS patch .

Last edited by Denso (2014-11-14 05:23:22)

Offline

#3256 2014-11-14 10:16:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018 [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID) [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: device [8086:2f08] error status/mask=00000040/00002000 [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: [ 6] Bad TLP

I didn't notice anything while using the VM or the host though . I'm using 3.18-rc4 + ACS patch .

No surprise you didn't notice anything.
There happened an error on the pci-e bus that was corrected. The error was noted by the device ID 0018(00:18.0?), as far as i understood, and it is your cpu's root port.
That maay be related to ACS.
Don't you have the new X99 chipset, where aw haven't got the letter confirming device isolation from intel?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3257 2014-11-14 10:27:59

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

No surprise you didn't notice anything.
There happened an error on the pci-e bus that was corrected. The error was noted by the device ID 0018(00:18.0?), as far as i understood, and it is your cpu's root port.
That maay be related to ACS.
Don't you have the new X99 chipset, where aw haven't got the letter confirming device isolation from intel?

Well , there is no 00:18.0 device reported by lspci , but the error shows "00:03.0" which is :

00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) 00:03.2 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)

And yes , I have the X99 platform , and I'm by no means a coder or a kernel hacker , I just applied the ACS patch and -to me as an end user- it just "works" . lol . Host + VMs are very stable (for around 2 days uptil this moment) .

Also editing the quirks.c file and inserting the X99 device IDs manually does the same thing .

Last edited by Denso (2014-11-14 10:29:16)

Offline

#3258 2014-11-14 12:53:10

cshark
Member
Registered: 2014-11-14
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have done the basic steps described in the article and qemu works as espected when invoked with the test setup. Now I want to use virt-manager with libvirt, so I created the important xml part with

# virsh native-to-domxml qemu-args <file with args>

which gave me some xml file with the device args as qemu args. I was wondering, why the devices were "only" arguments and had no proper xml tag, but I guess it's okay. So I created a new vm with virt-manager and added the important parts to the xml file. My final xml looks like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7</name> <uuid>c344a358-33bc-4398-b223-4054d6e61680</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>3</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>destroy</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/local/bin/kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdb2'/> <target dev='vda' bus='virtio'/> <boot order='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <interface type='network'> <mac address='52:54:00:36:e7:5a'/> <source network='virtnetwork'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-enable-kvm'/> <qemu:arg value='-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

But when I try to start the vm, I get the following error (even as root):

Internal error: early end of file from monitor: possible problem: 2014-11-14T12:25:20.491803Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x0, 0x80000000, 0x7f3ce8000000) = -12 (Cannot allocate memory) 2014-11-14T12:25:20.492686Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x100000000, 0x80000000, 0x7f3d68000000) = -12 (Cannot allocate memory) 2014-11-14T12:25:20.492767Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container 2014-11-14T12:25:20.492835Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 16 2014-11-14T12:25:20.492939Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 16 2014-11-14T12:25

I searched a bit, but found nothing about this error... any ideas? Since it has to be either something with libvirt or the config, because the qemu test setup just works fine.

Last edited by cshark (2014-11-14 12:55:04)

Offline

#3259 2014-11-14 16:20:33

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Will the two patches (acs override patch + i915 vga arbiter fixes) go into the main linux packages soon? I do not want to build one myself.

Offline

#3260 2014-11-14 19:22:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

<qemu:arg value='vfio-pci,host=01:00.0,addr=07.0,multifunction=on,x-vga=on,x-vid=0x10DE,x-did=0x11BA,x-ss-vid=0x10DE,x-ss-did=0x0965,romfile=/root/k5000.rom'/>

I can also confirm it's working, 680gtx as quadro k5000/or GRIDK2 (0x11BF) with 341.05 drivers with hyperv on windows7.

edit: as grid k2 also works with geforce driver 344,65 with hyperv on.

Last edited by slis (2014-11-14 19:46:44)

Offline

#3261 2014-11-14 19:26:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Will the two patches (acs override patch + i915 vga arbiter fixes) go into the main linux packages soon? I do not want to build one myself.

You'll need to use the OVMF approach and switch hardware then, neither of those are headed upstream.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3262 2014-11-14 23:32:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cshark wrote:

I have done the basic steps described in the article and qemu works as espected when invoked with the test setup. Now I want to use virt-manager with libvirt, so I created the important xml part with

# virsh native-to-domxml qemu-args <file with args>

which gave me some xml file with the device args as qemu args. I was wondering, why the devices were "only" arguments and had no proper xml tag, but I guess it's okay. So I created a new vm with virt-manager and added the important parts to the xml file. My final xml looks like this:

 <qemu:commandline> <qemu:arg value='-enable-kvm'/> <qemu:arg value='-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/> </qemu:commandline>

But when I try to start the vm, I get the following error (even as root):

Internal error: early end of file from monitor: possible problem: 2014-11-14T12:25:20.491803Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x0, 0x80000000, 0x7f3ce8000000) = -12 (Cannot allocate memory) 2014-11-14T12:25:20.492686Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x100000000, 0x80000000, 0x7f3d68000000) = -12 (Cannot allocate memory) 2014-11-14T12:25:20.492767Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container 2014-11-14T12:25:20.492835Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 16 2014-11-14T12:25:20.492939Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 16 2014-11-14T12:25

I searched a bit, but found nothing about this error... any ideas? Since it has to be either something with libvirt or the config, because the qemu test setup just works fine.

By using <qemu:arg> you're effectively hiding the assigned device from libvirt and not allowing it setup device access or configure locked memory limits.  My suggestion would be to not use q35 and to use a wrapper script around qemu-system-x86_64 to add the x-vga option, something like:

#!/bin/sh exec qemu-system-x86_64 `echo "\$@" | sed 's|02:00.0|02:00.0,x-vga=on|g'`

Better yet, if the GPU supports UEFI, use OVMF instead and avoid all the hassles of VGA.  Some here claim this works for Windows 7 guests.  See the link in my sig for details.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3263 2014-11-15 10:03:00

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

Just for information, I successfully passed through an Asus GTX 970 Strix from Arch (configured as mentioned in first post). Guest is running Win 7 Pro 64 bits, and here is my command line:

qemu-system-x86_64 -M q35 -enable-kvm \ -m 16384 \ -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -realtime mlock=on -vga none \ -D /var/log/qemu-out.log \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/guest/Windaube_7,id=sys,format=raw -device scsi-hd,drive=sys \ -drive file=/dev/guest/win7,id=data,format=raw -device scsi-hd,drive=data \ -device vfio-pci,host=00:1d.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi2 \ -net nic,model=virtio -net bridge,br=virbr0 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I only had to disable hv-time from CPU flag in order to get rid of Code 43 error (thanks Nvidia...).

Offline

#3264 2014-11-15 17:41:54

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got rid of the Error Code 43 

I am not quite sure what it was, but I started from scratch, built the kernel provided by nbhs (linux-mainline-3.17.2.tar.gz)
and threw every command I could find into grub. This is what my command line looks like.

GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

BUT (there always seems to be a but...)

I have weird audio issues: There is some scratching in the audio. And performance suffers big time.

I am really relying on you guys to help me, cause I don't know how to fix this.

Btw.: I am not sure if it's relevant, but lspci -v gives me this output:

01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3170
        Flags: fast devsel, IRQ 16
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at f7000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3170
        Flags: fast devsel, IRQ 17
        Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

I can't remember if the last line looked always like that but it's an Nvidia Card and it says Intel, maybe that's the culprit??

Offline

#3265 2014-11-15 17:57:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

I got rid of the Error Code 43 

I am not quite sure what it was, but I started from scratch, built the kernel provided by nbhs (linux-mainline-3.17.2.tar.gz)
and threw every command I could find into grub. This is what my command line looks like.

GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

BUT (there always seems to be a but...)

I have weird audio issues: There is some scratching in the audio. And performance suffers big time.

I am really relying on you guys to help me, cause I don't know how to fix this.

Btw.: I am not sure if it's relevant, but lspci -v gives me this output:

snd_hda_intel supports audio from various vendors, that's not a problem.

Nvidia's implementation of the audio device is fairly flaky, but it usually works better if you configure the device to use MSI interrupts in the guest.  See my blog for how to do that.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3266 2014-11-15 18:00:41

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, will do, thank you very much. I'll try that right away

Offline

#3267 2014-11-15 19:53:19

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

That reminds me: My SuperMicro board has settings both for "primary display" and for which ROM to load for each slot.

If I have the Intel graphics explicitly set to primary, but also allow loading the Radeon's EFI ROM, I get really weird behavior:

* The Radeon hijacks the EFI framebuffer (completely ignoring the "primary display" option).  I seem to recall the Intel DisplayPort lighting up but being black.
* The Radeon ends up with its framebuffer address screwed up, and I get the Radeon EFI framebuffer as /dev/fb0 and the Intel KMS driver as /dev/fb1.
* Neither the Windows nor Linux Radeon drivers (native boot, even) will load.

If I switch the PCIe slot ROM options to "Legacy" or "Disabled", no such problems occur, and I can still use EFI for both host and guest.

Offline

#3268 2014-11-15 20:30:55

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:
dakabali wrote:

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

That reminds me: My SuperMicro board has settings both for "primary display" and for which ROM to load for each slot.

If I have the Intel graphics explicitly set to primary, but also allow loading the Radeon's EFI ROM, I get really weird behavior:

* The Radeon hijacks the EFI framebuffer (completely ignoring the "primary display" option).  I seem to recall the Intel DisplayPort lighting up but being black.
* The Radeon ends up with its framebuffer address screwed up, and I get the Radeon EFI framebuffer as /dev/fb0 and the Intel KMS driver as /dev/fb1.
* Neither the Windows nor Linux Radeon drivers (native boot, even) will load.

If I switch the PCIe slot ROM options to "Legacy" or "Disabled", no such problems occur, and I can still use EFI for both host and guest.

Have you tried claiming Radeon with pci-stub?

Pasting the code somewhere inside /etc/modprobe.d, and rebuilding initrd should do the job:

softdep drm pre: pci-stub softdep snd_hda_intel pre: pci-stub options pci-stub ids=1002:6818,1002:aab0 # replace vid:devid accordingly

Offline

#3269 2014-11-15 23:38:38

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Peps,

I got my setup working thanks to this forum post. However I found two issues and hope to find even more help here if possible.

1. Audio emulation seams to sometimes cut out while gaming. Or the audio sounds weird and phased.

2. I noticed a roughly 30% decrease in performance and sometimes the FPS rate can go very lower. As in 1 to 5 FPS before returning to its normal ~30ish% reduce from where it would be if I was bare metal.

Here is my hardware:
Host Card: Nvidia GTX 750 (Nvidia drivers)
Windows Card: Nvidia GTX 670
CPU: AMD FX 9590
MotherBoard: MSI 990FXA-GD80 v2

I start qemu via the command line like so:

# QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \ > qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host,kvm=off \ > -smp cpus=8,sockets=1,cores=8,threads=1 \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=06:00.1 \ > -device virtio-scsi-pci,id=scsi \ > -drive file=/dev/sdb,id=disk0,format=raw -device scsi-hd,drive=disk0 \ > -drive file=/dev/sdc,id=disk1,format=raw -device scsi-hd,drive=disk1 \ > -usb -usbdevice host:1532:0109 -usbdevice host:046d:c52b \ > -net nic,macaddr=00:16:35:AF:94:4B -net user \ > -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ > -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I also followed this post: http://forums.guru3d.com/showthread.php?t=378044 and made sure my card was using MSI. I also took the liberty of making the emulated ich9 HD audio use MSI. However that seamed to have no real effect.

I also this:

# cat /etc/modprobe.d/60-kvm-amd.conf options kvm-amd npt=0

I guess my real question is: Is this normal? Does most people using KVM/Nvidia have a ~30ish % reduction in rendering performance? And/Or times when there are large drops in performance? If not what else am I missing to help this out?

Thanks in advance. :-)

Offline

#3270 2014-11-16 03:08:57

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:

CPU: AMD FX 9590
MotherBoard: MSI 990FXA-GD80 v2

Hi . I used to have an AMD 8350 + Gigabyte 990FXA-UD5 (If I remember correctly) and it was SO SLOW in gaming and the sound was playing in SlowMo .

When I switched to an i5-3570 (Again if I remember correctly) , it was butter smooth , yes it wasn't like bare-metal but it was pretty darn close .

Now I switched to an X99 platform + GTX770 and I can play games at 2560x1440 60fps (CoD Advanced Warfare for example) no problem .

Here is my Gaming VM script if you're curious :

qemu-system-x86_64 -name gaming -nographic -mem-path /dev/hugepages \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -vga none \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_gaming.bin \ -device vfio-pci,host=02:00.0,multifunction=on \ -device vfio-pci,host=02:00.1 \ -device vfio-pci,host=10:00.0 \ -drive file=/VMs/Win_Gaming.qed,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -drive file=/VMs/Gaming_External.qed,cache=none,if=none,id=drive1,aio=native \ -device virtio-blk-pci,drive=drive1,ioeventfd=on \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -net nic,model=virtio,macaddr=64:25:5E:87:0E:5C -net bridge,br=br0 \ -localtime \ -monitor unix:/tmp/vm_gaming,server,nowait &

Note that I'm using OVMF + 440FX + HDMI Audio .

Hope this helps ! smile

Last edited by Denso (2014-11-16 03:10:29)

Offline

#3271 2014-11-16 03:55:23

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Have you tried claiming Radeon with pci-stub?

Pasting the code somewhere inside /etc/modprobe.d, and rebuilding initrd should do the job:

softdep drm pre: pci-stub softdep snd_hda_intel pre: pci-stub options pci-stub ids=1002:6818,1002:aab0 # replace vid:devid accordingly

In my EFI-Intel + EFI-Radeon case, the Radeon was claimed by the EFI firmware, preventing both radeon and vfio-pci from accessing it properly.  So, blocking the radeon driver changes nothing.
I already figured out my issue; I was just mentioning it for reference, in case somebody else sees similar behavior.

Offline

#3272 2014-11-16 07:18:12

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Denso! Thanks for your reply!

Sadly I am not sure how useful that info is. Basically it sounds like I would need to buy new hardware. Sadly I just got this AMD setup and spent a pretty penny. Instead of a whole new set of hardware I could possibly buy a new Radeon card and either set how KVM handles that or switch to using Xen.

It appears that switching to OVMF would not help gaming performance at all.. correct?

Also.. "-monitor unix:/tmp/vm_gaming,server,nowait" !? If you dont mind me asking whats going on there? It looks like your are redirecting the output too a file? Is that an socket that redirects to something?

Offline

#3273 2014-11-16 08:02:40

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:

Hello Denso! Thanks for your reply!

Sadly I am not sure how useful that info is. Basically it sounds like I would need to buy new hardware. Sadly I just got this AMD setup and spent a pretty penny. Instead of a whole new set of hardware I could possibly buy a new Radeon card and either set how KVM handles that or switch to using Xen.

It appears that switching to OVMF would not help gaming performance at all.. correct?

Also.. "-monitor unix:/tmp/vm_gaming,server,nowait" !? If you dont mind me asking whats going on there? It looks like your are redirecting the output too a file? Is that an socket that redirects to something?

Hi smile

I'm sure that you can tune your setup to gain more performance . For my case , switching to an Intel platform was night and day difference (I remember Metro 2033 game , the i5 just blew FX-8350 out of water) .

umm , I'm directing the monitor output to a file , so that I can send a shutdown command to the VM prior to the host's shutdown/reboot like this :

#!/bin/bash # This code will send an ACPI Shutdown request to our VMs then wait 10 seconds , then kill all remaining "qemu" processes # echo "system_powerdown" | socat - UNIX-CONNECT:/tmp/vm_winsrv echo "system_powerdown" | socat - UNIX-CONNECT:/tmp/vm_gaming echo "system_powerdown" | socat - UNIX-CONNECT:/tmp/vm_main sleep 10s killall -9 qemu-system-x86_64

Works like a charm ! smile

Offline

#3274 2014-11-16 08:33:27

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:

Hello Peps,

I got my setup working thanks to this forum post. However I found two issues and hope to find even more help here if possible.

1. Audio emulation seams to sometimes cut out while gaming. Or the audio sounds weird and phased.

2. I noticed a roughly 30% decrease in performance and sometimes the FPS rate can go very lower. As in 1 to 5 FPS before returning to its normal ~30ish% reduce from where it would be if I was bare metal.

Here is my hardware:
Host Card: Nvidia GTX 750 (Nvidia drivers)
Windows Card: Nvidia GTX 670
CPU: AMD FX 9590
MotherBoard: MSI 990FXA-GD80 v2

I start qemu via the command line like so:

# QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \ > qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host,kvm=off \ > -smp cpus=8,sockets=1,cores=8,threads=1 \ > -bios /usr/share/qemu/bios.bin -vga none \ > -device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \ > -device vfio-pci,host=06:00.1 \ > -device virtio-scsi-pci,id=scsi \ > -drive file=/dev/sdb,id=disk0,format=raw -device scsi-hd,drive=disk0 \ > -drive file=/dev/sdc,id=disk1,format=raw -device scsi-hd,drive=disk1 \ > -usb -usbdevice host:1532:0109 -usbdevice host:046d:c52b \ > -net nic,macaddr=00:16:35:AF:94:4B -net user \ > -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ > -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I also followed this post: http://forums.guru3d.com/showthread.php?t=378044 and made sure my card was using MSI. I also took the liberty of making the emulated ich9 HD audio use MSI. However that seamed to have no real effect.

I also this:

# cat /etc/modprobe.d/60-kvm-amd.conf options kvm-amd npt=0

I guess my real question is: Is this normal? Does most people using KVM/Nvidia have a ~30ish % reduction in rendering performance? And/Or times when there are large drops in performance? If not what else am I missing to help this out?

Thanks in advance. :-)

so, basically you did NO basic tuning, but jumped straight to some advanced sutff (e.g. msi)? smart boy, truly. read more of some random forum's post instead of digging this very thread to get all the info

no cpu pinning, no emulated sound parameter tuning (or using passed through audio card), no advanced memory mode (hugetlbfs?) - are you really expecting it to work like a charm out of the box on near-native performance level? for god's sake...

Offline

#3275 2014-11-16 20:25:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

no cpu pinning, no emulated sound parameter tuning (or using passed through audio card), no advanced memory mode (hugetlbfs?) - are you really expecting it to work like a charm out of the box on near-native performance level? for god's sake...

Aw, come on, don't be rude, man!

The only thing about CPU i have is -cpu host and -smp option.
The sound is

-device ich9-intel-hda,bus=root.1,addr=00.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \

and i honestly do not know how the hell this works, thanks to nbhs. But it works almost awesome with my host pulse audio setup(fedora, i don't know how pulse works either) - there's little stuttering occuring when there is too much sound(like, playing a really weird music from youtube).Appears like i've got the sound input working too.
The hugepages option just breaks everything apart for me.

And you know what? Performance decrease is about 10%. The only thing that bottlenecks the whole bunch is the disk image file relying on a 2FAST4U loud WD hard drive with 5 or 7 bad sectors.

The only problem i am having with the CPU performance - it's switching it's frequency too damn often, but that depends on one host UEFI setting and hits the host system too.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3276 2014-11-16 20:42:50

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinyy wrote:

so, basically you did NO basic tuning

Actually... I troubleshooted the sound bit quite a lot. First I removed all sound devices to see if it would have an effect. That became my control environment for sound. I found that the FPS in two different games increase by roughly 2 or 3... *sarcastic woo*...  I figured I continue to mess around. I had a sound card. I passed that PCI device through and it worked awesomely. Except no improvements. In the end I found that the default sound emulation will work mostly fine. Except for the aforementioned problem. I asked about it because I do not want to have to use separate speakers (or some funky mixer) and if other people don't have to perhaps there is something I can due to tweak it.  I also switched between pulse audio and direct with ALSA. In fact I did a lot of s*** but I didn't explain in detail my whole troubleshooting history. Thats not how you open a conversation for help. I stated what the environment is and what i am currently doing as a starting place.

What about hugetlbfs... well did that too. No effect. What about compiling the kernel with the voluntary... did that too.. no effect.

One thing I didn't due was CPU pinning. I left that alone because I saw other people didn't have there settings adjusted. And I figured that it likely not improve performance much. I done stuff like that before  with little effect.

It is true that I haven't read all 131 pages of this forum. But it is also true that I never expected the performance to be 100% like bare metal. (nor did I type that) In fact I deal with that on a daily basis because I work on OpenStack for customers who always complain about this or that performance wise. My question was how much of a performance drop *should* I expect. Should it be ~30 % or 10 or what? And from what I am hearing it probably shouldn't be this low.


And Duelist... what hardware is that? AMD or Intel biased?

Offline

#3277 2014-11-16 21:09:14

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:

And Duelist... what hardware is that? AMD or Intel biased?

FM2-socket based AMD.
Asus F2A55+Athlon X4 750K Trinity(there is a richland-based version).


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3278 2014-11-16 21:28:28

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
orcephrye wrote:

And Duelist... what hardware is that? AMD or Intel biased?

FM2-socket based AMD.
Asus F2A55+Athlon X4 750K Trinity(there is a richland-based version).

*sigh* So an AMD based. Wait.. sorry to ask more but is the card Radeon or Nvidia?

I got an old Radeon 5770. I am thinking of benchmarking it bare metal and then replacing my 670 to see how it fairs in the KVM and also in Xen. I hope that I only get about 10 or 15% reduction. That seams to align with what I have seen online. If that works out then I will probably by a Radeon 290x and sell my Nvidia 670 for a 100 bucks or something. :-)

Offline

#3279 2014-11-16 21:50:01

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Trying OVMF, compiled it from github and I can't get Windows 8.1 to install. Once I got it till end of the first phase but when It started to "copy files 1 out of 5?" it said it cant continue. Compiled again OVMF and now I cannot for gods sake get the DVD to boot properly. "press anything to boot" > I press > windows logo shows with spinning > reboot instantly and Im in a never ending loop. Whats the problem?

Also from Gerd's repo edk2.git-ovmf-x64 is not available anymore??

Last edited by devianceluka (2014-11-16 21:51:27)

Offline

#3280 2014-11-16 21:56:39

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you can use ovmf-svn https://aur.archlinux.org/packages/ovmf-svn/

nbhs also built it from git. You can get it on Page 126, here in the thread.

Offline

#3281 2014-11-16 21:58:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:
Duelist wrote:
orcephrye wrote:

And Duelist... what hardware is that? AMD or Intel biased?

FM2-socket based AMD.
Asus F2A55+Athlon X4 750K Trinity(there is a richland-based version).

*sigh* So an AMD based. Wait.. sorry to ask more but is the card Radeon or Nvidia?

I got an old Radeon 5770. I am thinking of benchmarking it bare metal and then replacing my 670 to see how it fairs in the KVM and also in Xen. I hope that I only get about 10 or 15% reduction. That seams to align with what I have seen online. If that works out then I will probably by a Radeon 290x and sell my Nvidia 670 for a 100 bucks or something. :-)

1. IOMMU(or something else, i haven't figured out what exactly yet) is buggy as hell. I have to mlock the memory or i'm getting IO_PAGE_FAULT. I have to do something magical or i'm getting IO_PAGE_FAULT. I have to disable hugepages or i'm getting IO_PAGE_FAULT. Beware of that - there may be bad combos.
2. Host GT610, guest 2xHD7750.
3. Dude, check this out! A whole table with stats and graphics and stuff!
https://docs.google.com/spreadsheet/ccc … _web#gid=0
4. AW has GTX750(afair), paired with intel(afair) and, you know, it works.

If possible, you should continue digging, there may be something wrong, especially when using nvidia drivers with hyper-v extensions.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3282 2014-11-16 21:58:51

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

you can use ovmf-svn https://aur.archlinux.org/packages/ovmf-svn/

nbhs also built it from git. You can get it on Page 126, here in the thread.

sorry for a silly question, been playing with this whole day, cant google anymore. How to fetch from that AUR?

Offline

#3283 2014-11-16 22:02:57

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

there are several ways, but for me the easiest is just to 1. download the tarball 2. extract it and 3. run makepkg

Offline

#3284 2014-11-16 22:03:17

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oreephrye wrote:
sinny wrote:

so, basically you did NO basic tuning

Actually... I troubleshooted the sound bit quite a lot. First I removed all sound devices to see if it would have an effect. That became my control environment for sound. I found that the FPS in two different games increase by roughly 2 or 3... *sarcastic woo*...  I figured I continue to mess around. I had a sound card. I passed that PCI device through and it worked awesomely. Except no improvements. In the end I found that the default sound emulation will work mostly fine. Except for the aforementioned problem. I asked about it because I do not want to have to use separate speakers (or some funky mixer) and if other people don't have to perhaps there is something I can due to tweak it.  I also switched between pulse audio and direct with ALSA. In fact I did a lot of s*** but I didn't explain in detail my whole troubleshooting history. Thats not how you open a conversation for help. I stated what the environment is and what i am currently doing as a starting place.

What about hugetlbfs... well did that too. No effect. What about compiling the kernel with the voluntary... did that too.. no effect.

One thing I didn't due was CPU pinning. I left that alone because I saw other people didn't have there settings adjusted. And I figured that it likely not improve performance much. I done stuff like that before  with little effect.

It is true that I haven't read all 131 pages of this forum. But it is also true that I never expected the performance to be 100% like bare metal. (nor did I type that) In fact I deal with that on a daily basis because I work on OpenStack for customers who always complain about this or that performance wise. My question was how much of a performance drop *should* I expect. Should it be ~30 % or 10 or what? And from what I am hearing it probably shouldn't be this low.

well, i have my reasons to write the stuff i write as well, in fact.
hugetlbfs as well as cpu pinning (+splitting cores intended for host from the ones used for guest(-s)) did noticeable improvement in my case.
increasing pa sample amount mostly fixed sound quality issues (stutterring, etc)
switching to separate pci audio card being passed through to guest (while connecting its' line-out to host's line-in) did HUGE performance improvement.

in the end, i got from 15-25 fps to 300+ in the same game. but that may be just my case

p.s. btw, talking about performance increases or drops (or comparing hw setups) while not specifying the task used to "measure" performance may be very wrong

Offline

#3285 2014-11-16 22:05:22

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

there are several ways, but for me the easiest is just to 1. download the tarball 2. extract it and 3. run makepkg

-bash: makepkg: command not found

which dependancy installs this? Im on Fedora...

Offline

#3286 2014-11-16 22:08:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

increasing pa sample amount mostly fixed sound quality issues (stutterring, etc)

Hmm, i seem to be misunderstanding what you did there. Can you drop an example or something? That should be quick.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3287 2014-11-16 22:08:16

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahh, I'm sorry, I use arch, I am not familiar with fedora.

Offline

#3288 2014-11-16 22:11:04

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
4kGamer wrote:

there are several ways, but for me the easiest is just to 1. download the tarball 2. extract it and 3. run makepkg

-bash: makepkg: command not found

which dependancy installs this? Im on Fedora...

That is an arch tool that generates an arch package, afair.
To do this on fedora - you'll have to build an rpm.. I'm not going to tell you how to do that, i am unable to.

We could use checkinstall... ...but it is unavailable on fedora.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3289 2014-11-16 22:14:32

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
devianceluka wrote:
4kGamer wrote:

there are several ways, but for me the easiest is just to 1. download the tarball 2. extract it and 3. run makepkg

-bash: makepkg: command not found

which dependancy installs this? Im on Fedora...

That is an arch tool that generates an arch package, afair.
To do this on fedora - you'll have to build an rpm.. I'm not going to tell you how to do that, i am unable to.

We could use checkinstall... ...but it is unavailable on fedora.

?:/
Can you atleast help me with why Windows DVD wont boot?

Last edited by devianceluka (2014-11-16 22:16:02)

Offline

#3290 2014-11-16 23:38:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

?:/
Can you atleast help me with why Windows DVD wont boot?

Well i've had some problems with windows install dvd boot, but i can't recall what was the problem.
I remember it was related to OVMF. Something like i haven't had the video output.

Something hints me that you have a peculiar disk system used in VM. Can you supply us a launching script/xml config from libvirt again?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3291 2014-11-17 01:13:41

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
devianceluka wrote:

?:/
Can you atleast help me with why Windows DVD wont boot?

Well i've had some problems with windows install dvd boot, but i can't recall what was the problem.
I remember it was related to OVMF. Something like i haven't had the video output.

Something hints me that you have a peculiar disk system used in VM. Can you supply us a launching script/xml config from libvirt again?

qemu is from github v2.1.2, kernel 3.17.2:

vfio-bind 0000:02:00.0 0000:02:00.1 /xxx/qemu-system-x86_64 \ -enable-kvm -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -vga none -nographic -balloon none \ -net nic,model=virtio,macaddr=xxx,model=virtio \ -net bridge,br=br0,helper=/xxx/qemu-bridge-helper \ -device vfio-pci,host=02:00.0,multifunction=on,romfile=/xxx/club3dr9290.rom \ -device vfio-pci,host=02:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/xxx/test.qcow2,id=disk,format=qcow2,cache=none,aio=native \ -drive if=pflash,format=raw,readonly,file=/root/OVMF/OVMF.fd \ -drive if=pflash,format=raw,file=/root/OVMF/OVMF_VARS.fd \ -cdrom /xxx/virtio.iso \ #-usb -usbdevice host:046d:c315 #-drive file=/xxx/ubuntu.iso,id=iso_install,if=none \ #-device scsi-cd,drive=iso_install \

With this latest build/compile of OVMF from ubuntu wiki I can't even grub-install EFI bootloader into EFI shell (always have to load by file)...

compiled like this:

git clone git://github.com/tianocore/edk2.git cd edk2 make -C BaseTools . edksetup.sh nano Conf/target.txt " ACTIVE_PLATFORM = OvmfPkg/OvmfPkgX64.dsc TOOL_CHAIN_TAG = GCC48 TARGET_ARCH = X64 " build

and copy it out:

cp /root/OVMF/edk2/Build/OvmfX64/DEBUG_GCC48/FV/OVMF_CODE.fd /root/OVMF/OVMF_CODE.fd cp /root/OVMF/edk2/Build/OvmfX64/DEBUG_GCC48/FV/OVMF_VARS.fd /root/OVMF/OVMF_VARS.fd

Offline

#3292 2014-11-17 05:20:56

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
sinny wrote:

increasing pa sample amount mostly fixed sound quality issues (stutterring, etc)

Hmm, i seem to be misunderstanding what you did there. Can you drop an example or something? That should be quick.

export QEMU_PA_SAMPLES=8192

Offline

#3293 2014-11-17 16:48:08

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
export QEMU_PA_SAMPLES=8192

Well that is much higher then I tried it out. I did 512. :-P Should be more bold! Although I did do a line out to line in kinda setup and that seams to be working ok.

I attempted a few more tweaks. I added 'hv-time' and removed kvm=off. Then installed the older nvidia drivers that do not freak out. All of this didn't help a whole log. Everything seams to just give tiny little performance boosts or tiny performance decreases. I would like to try out CPU pinning. But the only articles showing how to do that is doing it with the XML file created by virt-install. I tried converting my command line arguments over but the VM wont load. Is there a way to do cpu pinning via the "qemu-system-x86_64" cmd?  There is cgroups... perhaps I should do that instead?

On another note thinking maybe I will download 3.17.3. I compiled 3.16.7. I also running Qemu 2.1 perhaps 2.2.

Either case I am still stuck with crappy performance. Namely the FPS in games is ok... just ever now and again it drops to almost zero.

Offline

#3294 2014-11-17 16:52:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:

Is there a way to do cpu pinning via the "qemu-system-x86_64" cmd?

info cpus from the QEMU monitor will give the thread ID for each vCPU, use taskset to pin them to separate physical CPUs.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3295 2014-11-17 18:36:37

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
Duelist wrote:
sinny wrote:

increasing pa sample amount mostly fixed sound quality issues (stutterring, etc)

Hmm, i seem to be misunderstanding what you did there. Can you drop an example or something? That should be quick.

export QEMU_PA_SAMPLES=8192

Oh. Thanks. I forgot i've had that variable set up in the beggining of the script. Thanks for the hint again.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3296 2014-11-17 18:54:08

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
orcephrye wrote:

Is there a way to do cpu pinning via the "qemu-system-x86_64" cmd?

info cpus from the QEMU monitor will give the thread ID for each vCPU, use taskset to pin them to separate physical CPUs.

... This is why we pay you the big bucks aw.

Who 'we' is and how much is 'big' are probably interesting questions.

I used taskset before... dont know why I didn't think of it. I can combo it with isolcpus! That will keep that pesky scheduler out of my cores.  Once I get home I will play around with that.. see if it works. Not sure if I would need to get the thread_id though. I mean wouldn't something like:

# taskset -c 2,3,4,5,6,7 qemu-system-x86_64 (etc...) 

Would that get all spawned threads/subprocesses and assign them to cpus 3-8? I ask that question even though I am gonna go home and try that out.

Offline

#3297 2014-11-17 19:00:47

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:
sinny wrote:
export QEMU_PA_SAMPLES=8192

Well that is much higher then I tried it out. I did 512. :-P Should be more bold! Although I did do a line out to line in kinda setup and that seams to be working ok.

I attempted a few more tweaks. I added 'hv-time' and removed kvm=off. Then installed the older nvidia drivers that do not freak out. All of this didn't help a whole log. Everything seams to just give tiny little performance boosts or tiny performance decreases. I would like to try out CPU pinning. But the only articles showing how to do that is doing it with the XML file created by virt-install. I tried converting my command line arguments over but the VM wont load. Is there a way to do cpu pinning via the "qemu-system-x86_64" cmd?  There is cgroups... perhaps I should do that instead?

On another note thinking maybe I will download 3.17.3. I compiled 3.16.7. I also running Qemu 2.1 perhaps 2.2.

Either case I am still stuck with crappy performance. Namely the FPS in games is ok... just ever now and again it drops to almost zero.

the poor man's choice is (i have been using it - it was the most i could think of):
1) prevent host from using cpus intended for guest(-s) - "isolcpus" kernel parameter; 1 2-threaded core is more then enough in my case, taskset allows to lift this restriction when necessary
2) restrict cpus available for every qemu process - use taskset; something like "sudo -E taskset --cpu-list 1,2,5,6 qemu-system-x86_64 ..."

it's not cpu pinning, but at least it allowed to perform basic manual host cpu resource separation

later i switched to libvirt, which seems to successfully manage cpu pinning by itself - only isolcpus remains from old setup

Last edited by sinny (2014-11-17 19:02:13)

Offline

#3298 2014-11-17 19:05:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:
aw wrote:
orcephrye wrote:

Is there a way to do cpu pinning via the "qemu-system-x86_64" cmd?

info cpus from the QEMU monitor will give the thread ID for each vCPU, use taskset to pin them to separate physical CPUs.

... This is why we pay you the big bucks aw.

Who 'we' is and how much is 'big' are probably interesting questions.

I used taskset before... dont know why I didn't think of it. I can combo it with isolcpus! That will keep that pesky scheduler out of my cores.  Once I get home I will play around with that.. see if it works. Not sure if I would need to get the thread_id though. I mean wouldn't something like:

# taskset -c 2,3,4,5,6,7 qemu-system-x86_64 (etc...) 

Would that get all spawned threads/subprocesses and assign them to cpus 3-8? I ask that question even though I am gonna go home and try that out.

Sure, but that's not what you want.  You want to pin a single vCPU to a single pCPU such that the scheduler has zero migration to do and the caches are hot.  If you're benchmarking a VM, it's probably also best to set the performance governor in the host since ondemand can add latency to the guest's ability to react to load.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3299 2014-11-17 20:02:46

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
orcephrye wrote:
sinny wrote:
export QEMU_PA_SAMPLES=8192

Well that is much higher then I tried it out. I did 512. :-P Should be more bold! Although I did do a line out to line in kinda setup and that seams to be working ok.

I attempted a few more tweaks. I added 'hv-time' and removed kvm=off. Then installed the older nvidia drivers that do not freak out. All of this didn't help a whole log. Everything seams to just give tiny little performance boosts or tiny performance decreases. I would like to try out CPU pinning. But the only articles showing how to do that is doing it with the XML file created by virt-install. I tried converting my command line arguments over but the VM wont load. Is there a way to do cpu pinning via the "qemu-system-x86_64" cmd?  There is cgroups... perhaps I should do that instead?

On another note thinking maybe I will download 3.17.3. I compiled 3.16.7. I also running Qemu 2.1 perhaps 2.2.

Either case I am still stuck with crappy performance. Namely the FPS in games is ok... just ever now and again it drops to almost zero.

the poor man's choice is (i have been using it - it was the most i could think of):
1) prevent host from using cpus intended for guest(-s) - "isolcpus" kernel parameter; 1 2-threaded core is more then enough in my case, taskset allows to lift this restriction when necessary
2) restrict cpus available for every qemu process - use taskset; something like "sudo -E taskset --cpu-list 1,2,5,6 qemu-system-x86_64 ..."

it's not cpu pinning, but at least it allowed to perform basic manual host cpu resource separation

later i switched to libvirt, which seems to successfully manage cpu pinning by itself - only isolcpus remains from old setup

I just realized that my idea will not take every thread and place it on one CPU... but I am not sure if that's actually necessary:

... just refreshed and.. derp:

aw wrote:

Sure, but that's not what you want.  You want to pin a single vCPU to a single pCPU such that the scheduler has zero migration to do and the caches are hot.  If you're benchmarking a VM, it's probably also best to set the performance governor in the host since ondemand can add latency to the guest's ability to react to load.

So... perhaps a script like this:

VAR=2 && QEMUPID=`pidof /usr/bin/qemu-system-x86_64` && THREADLIST=`ps -o lwp --no-headers -p $QEMUPID -L | grep -v $QEMUPID` && for i in $THREADLIST ; do taskset -cp $VAR $i ; VAR=$((VAR+1)) ; done

Last edited by orcephrye (2014-11-17 20:03:23)

Offline

#3300 2014-11-17 20:54:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:
aw wrote:

Sure, but that's not what you want.  You want to pin a single vCPU to a single pCPU such that the scheduler has zero migration to do and the caches are hot.  If you're benchmarking a VM, it's probably also best to set the performance governor in the host since ondemand can add latency to the guest's ability to react to load.

So... perhaps a script like this:

VAR=2 && QEMUPID=`pidof /usr/bin/qemu-system-x86_64` && THREADLIST=`ps -o lwp --no-headers -p $QEMUPID -L | grep -v $QEMUPID` && for i in $THREADLIST ; do taskset -cp $VAR $i ; VAR=$((VAR+1)) ; done

Seems like it might work, the vCPUs should be the first N threads spawned by the main QEMU pid.  BTW, you seem to be starting at physical CPU2, but you were originally using -smp cpus=8,sockets=1,cores=8,threads=1 on an 8-core processor.  Are you also scaling back to a 6-core guest?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3301 2014-11-17 21:15:03

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
orcephrye wrote:
aw wrote:

Sure, but that's not what you want.  You want to pin a single vCPU to a single pCPU such that the scheduler has zero migration to do and the caches are hot.  If you're benchmarking a VM, it's probably also best to set the performance governor in the host since ondemand can add latency to the guest's ability to react to load.

So... perhaps a script like this:

VAR=2 && QEMUPID=`pidof /usr/bin/qemu-system-x86_64` && THREADLIST=`ps -o lwp --no-headers -p $QEMUPID -L | grep -v $QEMUPID` && for i in $THREADLIST ; do taskset -cp $VAR $i ; VAR=$((VAR+1)) ; done

Seems like it might work, the vCPUs should be the first N threads spawned by the main QEMU pid.  BTW, you seem to be starting at physical CPU2, but you were originally using -smp cpus=8,sockets=1,cores=8,threads=1 on an 8-core processor.  Are you also scaling back to a 6-core guest?

Yeppers. The idea is having 2 pCPUs for the host. And use isolcpus on the other 6. Use the script above to mask/pin/affinity (whatever) the threads to certain pCPUs. Starting with 2 and ending on 7. The list should spawn 7 PIDs and then one of them is remvoed with the grep command. That is the qemu parent pid which can happy stay on the host... right? I mean thats a good question shall the parent pid go over? I would assume not.. but you know what happens when people assume.

*And yes I would probably change the cmdLine to be: -smp cpus=6,sockets=1,cores=6,threads=1

Last edited by orcephrye (2014-11-17 21:16:36)

Offline

#3302 2014-11-17 21:19:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

orcephrye wrote:
aw wrote:
orcephrye wrote:

So... perhaps a script like this:

VAR=2 && QEMUPID=`pidof /usr/bin/qemu-system-x86_64` && THREADLIST=`ps -o lwp --no-headers -p $QEMUPID -L | grep -v $QEMUPID` && for i in $THREADLIST ; do taskset -cp $VAR $i ; VAR=$((VAR+1)) ; done

Seems like it might work, the vCPUs should be the first N threads spawned by the main QEMU pid.  BTW, you seem to be starting at physical CPU2, but you were originally using -smp cpus=8,sockets=1,cores=8,threads=1 on an 8-core processor.  Are you also scaling back to a 6-core guest?

Yeppers. The idea is having 2 pCPUs for the host. And use isolcpus on the other 6. Use the script above to mask/pin/affinity (whatever) the threads to certain pCPUs. Starting with 2 and ending on 7. The list should spawn 7 PIDs and then one of them is remvoed with the grep command. That is the qemu parent pid which can happy stay on the host... right? I mean thats a good question shall the parent pid go over? I would assume not.. but you know what happens when people assume.

Sure, I'm not sure how much study there's been regarding whether it's better to pin the parent process to a non-vCPU processor or just let it float around.  If you had multiple NUMA nodes you'd certainly want to pin it to the node, but otherwise I'm not sure how much it matters.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3303 2014-11-18 01:16:53

biasha
Member
Registered: 2014-11-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am trying to pass-though asus GTX980 (under ASRock x99 professional, radeon as primary card). Installing Windows 2012 R2, host Linux 3.16.0-24-generic #32-Ubuntu
I have partial success but now i've faced few problems i don't understand, perhaps someone could explain something to me or reference to useful articles, because i am very newbe to all of this.
btw, i've used this article: http://www.pugetsystems.com/labs/articl … 4-KVM-585/

1. Monitor/card refused to turn on. I've fixed it by turning off VGA ROM initialization in the bios (thanks to asrock for the option). Now it works, but (because radeon is not initialized too) i totally miss everything before X on primary card, plus i have error message from speaker at boot. I've tried "romfile=" parameter, but it does not help. Moreover it prevent monitor from turning on. So i don't understand what is "romfile=" for? It simply breaks something.
I am disabling NVIDIA by adding "pci_stub ids=" into /etc/initramfs-tools/modules. Is there any "stronger" way to prevent card's ROM from loading? Why is it actually a problem?
Disabling VGA in the BIOS works, but i don't like it, you understand.
edit: "UEFI only" bios option  works better. i guess my HD5450 does not have uefi, so nvidia works during boot, then radeon turns on with desktop. looks like non-uefi rom can't be loaded twice - into host and guest.

2. I've tried last downloadable qemu (2.1.91 aka 2.2.0rc1) (not sources) from the site, but was unable to use it due disks problem.
When i use -hda -cdrom keys Windows installer simply can't recognize disks.
I've read i need ahci for windows to recognize disks, but when i use ahci together with -drive/-device combination linked via "id/drive" fields, qemu says "this <diskname> is already defined" or something like that (can reproduce and write exactly if you want)
This has gone after downgrading to qemu 2.1.2, but i don't like it, you understand.

3. Card is connected to the virtual ioh3420. Could someone explain why? Why do i actually need ioh3420? I've tried connecting card directly to the pcie.0. It works. I feel no difference. So why do i actually need additional virtual device? I don't mind, but i don't like useless virtual device, you understand.

4. When i view "System Information" in the NVIDIA Control Panel is see "Bus: PCI Express x 1" (or even "x0" without ioh3420). This sounds scary. It supposed to be "PCIe 3.0 x16".
So do i really suffer from slow connection between Video Card and (hm, not even sure what) CPU? Or does it magically overclock this virtual PCIe 1.0 x 1 to be like physical 3.0x16?
I have not scientifically checked performances difference yet, but i don't like x1 instead of x16, you understand me.

Besides that everything works fine. No "!" marks in device manager, virtual machine restarts, starts stops without problems, latest nvidia drivers 344.65. Sound via HDMI works fine.

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /home/biasha/Downloads/bios.bin-1.7.5 -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \ -device ahci,bus=pcie.0,id=ahci \ -drive file=/home/biasha/Downloads/windows1.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ -drive file=/home/biasha/Downloads/en_windows_server_2012_r2_vl_with_update_x64_dvd_4065221.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \ -usb -usbdevice host:046d:c52b -usbdevice host:0738:1709 \ -boot menu=on

Last edited by biasha (2014-11-18 02:56:57)

Offline

#3304 2014-11-18 03:36:21

orcephrye
Member
Registered: 2014-11-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

...
All my quotes and stuff
...

Sure, I'm not sure how much study there's been regarding whether it's better to pin the parent process to a non-vCPU processor or just let it float around.  If you had multiple NUMA nodes you'd certainly want to pin it to the node, but otherwise I'm not sure how much it matters.

So... got it working. But for your information you NEED to pin the parent process with the rest of them. Without doing that I got a lot of stuttering... in everything.. audio/mouse movements/rendering. I also discovered that qemu spawns a lot more sub processes then simply the threads. So I run qemu like so:

taskset -c 2,3,4,5,6,7 qemu-system-x86_64 [....]

And then I run:

VAR=2 && QEMUPID=`pidof /usr/bin/qemu-system-x86_64` && THREADLIST=`ps -o lwp --no-headers -p $QEMUPID -L | grep -v $QEMUPID` && for i in $THREADLIST ; do taskset -cp $VAR $i ; if [ $VAR -eq 7 ] ; then VAR=1; fi ; VAR=$((VAR+1)) ; done

Notice the added if statement in the for loop. This resets the CPU counter. (I am horrible and naming variables.. shame.. shame on me). Either case now this doesn't care how many processes spawn at run time. It spreads them all out. I ran qemu several times while testing and it went from anywhere around 7 (6 cores) to 12. And while it was running it would spawn more every now and again. With the first taskset command this makes sure all processes in the future get to run on at least one of the 6 CPUs. And the fancy for loop with tasksets makes sure that each starting thread gets its own CPU. Sorta... although all the first threads that spawn are the threads for each CPU#... everytime I ran it. Which was about 10 to 15 times... i lost count. Basically its safe to assume that the first 6 threads from the one-liner above will be the qemu vCPUs.

Anywho... that had a pretty nice performance boost. Although what was even bigger was the governor being set to performance. I am use to working with RHEL 5/6 with cpuspeed service on Dell 2950s/R710s and R720s. Normally all you need to do is disable energy saving features in the bios and call it a day. So I already had done that before. But apparently I was horribly wrong. I used cpupower to set it to performance and that made pretty big diff as well. Combine that with cpu stuff and... well I am gaming and near 90% bare metal performance.


@biasha
I checked my Nvidia card which I am using on Windows and it too shows PCI x0. However I am getting good performance (now) in my games. This variable may not dictate the actual bandwidth it may just simply be displaying what it sees from the controller.

Also... I made the card that Linux runs on be the primary card on the motherboard. I did this because I had no other choice. You cannot use the card that is your primary if your bios doesn't support switching or changing the VGA settings. I have an MSI 990FXa-GD80 and it doesn't have that feature. I recommend placing the card your Windows guest will be using in a different PCI slot. Most motherboard have more then one PCI x16 slot. You would then be able to enjoy seeing the whole boot process and not wait in mystery.

I am also not using ioh3420 or any such thing. Here is my current config for your referance:

qemu-system-x86_64 -nographic -monitor telnet:127.0.0.1:1234,server,nowait \ -name gaming -enable-kvm -M q35 \ -balloon none -m 16386 -mem-path /dev/hugepages -mem-prealloc \ -cpu host,hv-time -smp cpus=4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1 \ -device vfio-pci,host=07:06.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=disk0,format=raw -device scsi-hd,drive=disk0 \ -drive file=/dev/sdc,id=disk1,format=raw -device scsi-hd,drive=disk1 \ -net nic,model=virtio,macaddr=00:16:35:AF:94:4B -net bridge,br=br0 \ -usb -usbdevice host:1532:0109 -usbdevice host:046d:c531 -usbdevice host:1a40:0101 \ -rtc base=localtime -monitor unix:/tmp/vm_gaming,server,nowait 

PS: Much thanks to aw,Denso and sinny for their help.

Last edited by orcephrye (2014-11-18 05:29:06)

Offline

#3305 2014-11-18 07:57:47

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PS: Much thanks to aw,Denso and sinny for their help.

You're welcome . smile
Pssst : stealing everyone else's CMDLines and trying them was a big help for me , also many thanks to aw and everyone else for taking their time to help ! wink

---------------------------------------------

@aw :

I noticed that ACS patch causes these errors :

[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018 [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID) [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: device [8086:2f08] error status/mask=00000040/00002000 [Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: [ 6] Bad TLP

I recompiled 3.18-rc5 without the ACS patch and added the X99 quirks manually to "/drivers/pci/quirks.c" file , and I no longer recieve them .

Last edited by Denso (2014-11-18 08:05:39)

Offline

#3306 2014-11-18 12:13:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I noticed that ACS patch causes these errors :

Seems like my guess was true. Devices aren't isolated properly. Good luck.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3307 2014-11-18 12:19:30

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

can anyone help me with why my Windows guest cannot boot anymore? Also tried ubuntu guest and the 2nd time when I reboot it wont start xinit! It looks like it didnt reset the GPU the proper way so it can reuse/reinitialize it. This is with SeaBIOS and it was working flawlessly till now. Using Fedora 3.17.2 with Seabios 1.7.5 git and Qemu 2.1.2 git

Offline

#3308 2014-11-18 12:23:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Denso wrote:

I noticed that ACS patch causes these errors :

Seems like my guess was true. Devices aren't isolated properly. Good luck.

Umm .. That sounds tragic . But my host AND VMs are working flawlessly (aside from the VM rebooting hanging the host , which is caused by Nvidia's drivers I believe) .. So I don't see the issue here ?

Last edited by Denso (2014-11-18 12:30:18)

Offline

#3309 2014-11-18 16:35:40

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I get an error, when i want to bind a device to vfio-pci.

[ 395.815169] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 395.815178] vfio-pci: probe of 0000:01:00.1 failed with error -22

Here is my cmdline

initrd=\initramfs-linux.img keydevice=UUID=daabf52e-3e81-499a-af71-e5a2887840e9:key homedevice=UUID=cc29c820-32e3-45b6-8bf4-a152bd1a29a1:home cryptdevice=UUID=16ccebd1-dadb-4180-abd3-854a23871ac0:root_enc:allow-discards cryptkey=/dev/mapper/key:0:527 root=/dev/mapper/root_enc rw pci-stub.ids=10de:06c0,10de:0be5 intel_iommu=1 i915.enable_hd_vgaarb=1

I think IOMMU is enabled:

[ 0.032533] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a [ 0.032537] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a [ 0.032607] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1

My devices get claimed by pci-stub

[ 0.412398] pci-stub: add 10DE:06C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.412414] pci-stub 0000:01:00.0: claimed by stub [ 0.412423] pci-stub: add 10DE:0BE5 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.412431] pci-stub 0000:01:00.1: claimed by stub

Intel Virtualization Tech and Intel VT-d Tech is enabled in my MSI-Z77A-G45. I use an i7-2600: http://ark.intel.com/de/products/52213/ … o-3_80-GHz

processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 42 model name : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz stepping : 7 microcode : 0x29 cpu MHz : 1599.992 cache size : 8192 KB physical id : 0 siblings : 8 core id : 0 cpu cores : 4 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt bugs : bogomips : 6802.96 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:

At boot I get the following error, i don't know if it has to do something with it:

[ 28.354419] [drm:ilk_display_irq_handler] *ERROR* Pipe B FIFO underrun [ 28.354429] [drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder B [ 28.354430] [drm:cpt_serr_int_handler] *ERROR* PCH transcoder B FIFO underrun

Any ideas?

Offline

#3310 2014-11-19 01:19:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Researching processor thermal management and thermal monitoring i've got an interesting document.
http://amd-dev.wpengine.netdna-cdn.com/ … _Guide.pdf
(i have amd athlon x4 750k FM2 socket, family 15h model 10h)
There is plenty of bugs in the IOMMU.

IOMMU Event Not Flagged when DTE Reserved Bits Are Not Zero IOMMU Event Log Not Generated for Invalid DTE GCR3 Table Root Pointer IOMMU Event Log Not Generated for Not Present Host Intermediate Page Tables Incorrect Translation with IOMMU v1 512 GB Page Table IOMMU Event Log Ordering Violation IOMMU PPR Log Ordering Violation IOMMU Interrupt May Be Lost

Well, i have nothing to do, but to read and try to understand what the hell is happening here.

But i have a note for everyone using AMD FM2 socket based C/APUs:
For CPB(turbo) to work, two things are needed: 1. TCTL must be below 74C; 2. Some(i think two) cores must be disabled by OS' power management software.
There is two problems:
1. The actual critical temperature is waaay higher than 74C, actually it is ~120C - only in that case the CPU will do THERMTRIP and(CPB is disabled) will turn off the system almost immediatly.
2. Linux does not do that yet, so your CPU will stay at it's base frequency, never working in turbo mode.

In the real case, VM won't load all 4 cores to 100%, meaning that there will be a lot of P-state transitions. Every P-state transition is either a hang-up or crash or something else nasty(look over the revision guide, P-states are buggy too), or just a minor slowdown. The slowdown effect is amplified by VM, and i've actually noticed heavy CPU performance drawbacks on dynamic CPU-load jobs, like games. On static CPU-load jobs, like mining or bruteforcing, the performance hit is not so significant. The actual numbers will come tomorrow or something, as i don't have the time to test it all yet.

EDIT:
Yep. The raw, full load CPU performance is almost the same, sometimes even doing majestic stuff.
I've ran hashcat-cliXOP.bin(.exe) with MD5 APR(-m1600) with -a3 and a huge mask of ?a?a?a?a?a on a bogus hash and i've got 37k/second inside the VM and 37k/second on the host system.
Fun part: i can run hashcat on host and guest systems simulatenously, having 27k and 27k performance meters. That feels weird.
The P-state transition part: when i am running hashcat on three or less cores instead of four, the performance decrease is not linear(same thing for plain 64, avx and avx2 versions, apart from xop), and there is a lot of P-state transitions. The actual process is fluctuating between two(or more) cores, each needs to "speed up" and we have lags.

Last edited by Duelist (2014-11-19 15:42:50)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3311 2014-11-19 06:21:49

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmmmm, after all pinning work done by orcephrye i just wonder how libvirt/virt-manager's cpu pinning works.
and whether it's more optimal (performance-wise) to use manual cpu pinning or libvirt/virt-manager's one.

p.s. for now i am totally satisfied with performance i get using "dumb" libvirt/virt-manager approach, but having more info on the matter would be great (in case someone has clear view of things)

Offline

#3312 2014-11-19 14:47:22

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To add some information: I compiled the kernel myself with the PKGBUILD. I only added the two patches, i modified them, because they did not patch correctly with actual kernel 3.17.3
i915_vga_arbiter_fixes_mod.patch:
http://sprunge.us/HWaY
acs_override_mod.patch:
http://sprunge.us/UiVa

$ lsmod | grep stub
pci_stub               12429  0

nouveau is NOT loaded. nouveau is also blacklisted.

The first time I run ./vfio-bind there are 3 messages in dmesg, after that every time I run it, there are 4 messages. I dont really get it. (I echoed X to /dev/kmesg after every vfio-bind)

[ 60.559359] usb 1-1.6: USB disconnect, device number 5 [ 892.895082] VFIO - User Level meta-driver version: 0.3 [ 892.900616] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 892.902868] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 892.902878] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 919.366858] X [ 925.888402] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 925.888414] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 925.890925] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 925.890935] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 942.445180] X [ 943.751225] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 943.751236] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 943.753664] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 943.753674] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 1013.824028] X [ 1014.636678] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 1014.636689] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 1014.639130] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 1014.639141] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 1015.453419] X [ 1016.252234] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 1016.252245] vfio-pci: probe of 0000:01:00.1 failed with error -22 [ 1016.254664] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 1016.254675] vfio-pci: probe of 0000:01:00.1 failed with error -22

Offline

#3313 2014-11-19 14:53:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

To add some information:
snip

[ 60.559359] usb 1-1.6: USB disconnect, device number 5 [ 892.895082] VFIO - User Level meta-driver version: 0.3 [ 892.900616] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 892.902868] vfio-pci: probe of 0000:01:00.0 failed with error -22 [ 892.902878] vfio-pci: probe of 0000:01:00.1 failed with error -22

-EINVAL likely means the IOMMU is not enabled.  Look at the groups in /sys/kernel/iommu_groups/  If there's nothing there, vfio can't work.  You probably need intel_iommu=on or your kernel config may be broken.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3314 2014-11-19 14:54:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

hmmmm, after all pinning work done by orcephrye i just wonder how libvirt/virt-manager's cpu pinning works.
and whether it's more optimal (performance-wise) to use manual cpu pinning or libvirt/virt-manager's one.

p.s. for now i am totally satisfied with performance i get using "dumb" libvirt/virt-manager approach, but having more info on the matter would be great (in case someone has clear view of things)

libvirt provides vcpu pinning options equivalent to what orcephrye is doing, check the libvirt domain xml spec for the correct options (I've never had any luck trying to figure out how to specify it via the virt-manager UI).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3315 2014-11-19 15:33:08

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

-EINVAL likely means the IOMMU is not enabled.  Look at the groups in /sys/kernel/iommu_groups/  If there's nothing there, vfio can't work.  You probably need intel_iommu=on or your kernel config may be broken.

Thank you for your answer.
You are right, /sys/kernel/iommu_groups is a empty directory.
I used the config.x86_64 from the PKGBUILD: https://projects.archlinux.org/svntogit … ages/linux

zcat /proc/config.gz | grep CONFIG_VFIO_PCI_VGA
CONFIG_VFIO_PCI_VGA=y

Here is the /proc/config.gz
What else must be enabled?
http://sprunge.us/XSCA

Offline

#3316 2014-11-19 15:34:56

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:
aw wrote:

You probably need intel_iommu=on

What else must be enabled?

That is a kernel boot parameter AW mentioned. Please check if it's enabled.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3317 2014-11-19 15:37:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:
intel_iommu=1

Just looked at the code, this is not valid.  It needs to be "on"


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3318 2014-11-19 16:40:49

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oh, thank you very very much!!!! :-)

Omg, dont know, where I got the 1 from :-x

Offline

#3319 2014-11-19 17:25:27

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Again: I start in X as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get this output:

Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

dmesg says

[ 1137.081643] vfio-pci 0000:01:00.0: Invalid ROM contents [ 1137.473305] kvm: zapping shadow pages for mmio generation wraparound

dwe11er says:
https://bbs.archlinux.org/viewtopic.php … 7#p1431277

The commandline option is set (with =1, not with =on): i915.enable_hd_vgaarb=1

Is the "vbios file" a file I get with nvflash and how do I have to pass it to qemu?
Is it reasonable to use rombar=0  and again: how do I have to pass it to qemu?

Offline

#3320 2014-11-19 19:09:37

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been able to get this working on my main desktop system (FX-8350, Gigabyte 990FXA-UD3, R9 290 for the guest, GTX 750 Ti for the host) but have been having more trouble with the less-friendly hardware used in my secondary/guest computer:

Motherboard: Intel DP67BG
CPU: i5-2500
Host graphics: Gigabyte HD6670
Guest graphics: Gigabyte GTX 650 Ti (GV-N65TOC-1GI rev 1.0, has the original BIOS-only firmware)

I've been able to get the guest to boot up OK with the nvidia card passed through, installed the drivers etc. (with the ACS overrides patch) but I get the notorious 'Code 43' error. I've got kvm hiding enabled, and as far as I can tell none of the hyperv extensions are enabled, but even if I use the 335 drivers I still get the same error in device manager & no output on the card.

Disabling/enabling the card in device manager removes the error for that boot, but the card still doesn't work and the nvidia control panel crashes if I attempt to open it.

I've also tried the card in both the primary & secondary x16 slots without any luck, the bios also does have an option to select which card to use as primary but this also does not help.

Any ideas, or am I SOL?



Also, assuming I'm not entirely SOL I'm interested in trying out the OVMF method, but as mentioned the card I want to pass through doesn't have EFI-capable firmware. There is a firmware update listed for it on Gigabyte's site that apparently does, but I'm not sure how to flash this from Linux...

Offline

#3321 2014-11-19 21:01:48

eric_f
Member
Registered: 2014-11-19
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys.
I made an account just so I could post my findings and what I needed to do make this work.
First following the rest of the directions did work, I got my display working pretty quickly and easily, but my problem was once Windows installed the video drivers it would blue screen, and it went away so fast I couldn't read the error.
Finally it stayed long enough, I found out it was being caused my the Windows ATI drivers.  I'm using a Radeon HD 2600 XT.  What I needed to do was boot into safe mode, uninstall the Windows drivers and use the ATI drivers I downloaded. 

Once that was done the machine booted up correctly and has been working great since.

Thanks for all the great work.

Offline

#3322 2014-11-19 21:15:37

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Congratulations @ Eric! Nice to hear more success stories. Cross your fingers for me now wink


@Alex (aw)  thanks a lot for your suggestion. Switching to MSI worked exactly as you said. Scratching  in the sound is gone! Thank you so much.

Is it possible that (in one of your setups) you have a i5-3470t?

Cause that's the CPU I am running and I couldn't manage to get the best solution regarding cpu tuning.

would you be so kind to post that certain part of code in the .xml?

that would be great

Last edited by 4kGamer (2014-11-19 21:16:37)

Offline

#3323 2014-11-19 21:15:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

eric_f wrote:

I'm using a Radeon HD 2600 XT.

Wow, I think that's the oldest Radeon card we'd heard of working.  Thanks for the report.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3324 2014-11-19 21:31:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Congratulations @ Eric! Nice to hear more success stories. Cross your fingers for me now wink


@Alex (aw)  thanks a lot for your suggestion. Switching to MSI worked exactly as you said. Scratching  in the sound is gone! Thank you so much.

Is it possible that (in one of your setups) you have a i5-3470t?

Cause that's the CPU I am running and I couldn't manage to get the best solution regarding cpu tuning.

would you be so kind to post that certain part of code in the .xml?

that would be great

I do have an i5-3470t, but all the tuning I do is pretty generic.

First you can set hugepages:

 <memoryBacking> <hugepages/> <nosharepages/> </memoryBacking>

You'll need to pre-allocate the hugepages via kernel commandline or procfs, libvirt won't do it for you (yet).

Basic CPU pinning:

 <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> </cputune>

CPU type and topology:

 <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu>

This is equivalent to -cpu host with a topology matching the physical CPU.

And if you're using AMD or have otherwise freed yourself of Nvidia restrictions:

 <features> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> </clock>

And as noted recently, use the performance CPU governor on the host for benchmarking/gaming.

The i5-3470t is a little on the under-powered side for gaming, even on bare metal.  For a VM, you don't really have any cores or even threads to spare for the host.  I think things like Steam in-home-streaming recommend a quad-core.  That doesn't stop me from using it with Nvidia and AMD VMs running simultaneously, but I either need to partition them as 1core + 1thread each, with pinning on the correct physical CPUs, or I need to serialize the workload between them to avoid choppiness.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3325 2014-11-19 21:49:44

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for posting your configuation. My settings are just about the same. Looks like the i5 is simply not good enough as you already stated, but I haven't set hugepages yet.

That I'll try next.

For 4k of course it's a no-go. I guess it needs to be the i7-4790k which is next on my agenda.

Offline

#3326 2014-11-19 22:42:00

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok with
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0

the error is gone. dmesg says only
[21593.742734] kvm: zapping shadow pages for mmio generation wraparound


But there is still no output on the monitor on the 480GTX.

Offline

#3327 2014-11-19 22:55:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Ok with
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0

the error is gone. dmesg says only
[21593.742734] kvm: zapping shadow pages for mmio generation wraparound


But there is still no output on the monitor on the 480GTX.

You need a ROM to use VGA assignment.  If we can't read it from the card, you can try to download one from the TechPowerUp database.  However, not being able to get the ROM from the card itself probably means that the host drivers aren't properly blacklisted or the assigned device is actually the boot VGA device.  Using pci-stub.ids=$VENDOR:$DEVICE is the preferred way to make sure no host drivers attach to the device.  Replace $VENDOR and $DEVICE with the PCI IDs for your card found from lspci -nn.

EDIT, I see you're using pci-stub:

Rommy wrote:

Here is my cmdline

pci-stub.ids=10de:06c0,10de:0be5

So make sure your BIOS is set to initialize the IGD, not the PCIe graphics card.

Last edited by aw (2014-11-19 23:03:06)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3328 2014-11-19 22:59:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

At boot I get the following error, i don't know if it has to do something with it:

[ 28.354419] [drm:ilk_display_irq_handler] *ERROR* Pipe B FIFO underrun [ 28.354429] [drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder B [ 28.354430] [drm:cpt_serr_int_handler] *ERROR* PCH transcoder B FIFO underrun

BTW, this is a classic indication that the i915 arbiter patch is either not applied or not enabled.  If you have Intel graphics and you get this, you'll never get VM boot output to the assigned VGA until it's fixed.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3329 2014-11-19 23:17:37

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://sprunge.us/HWaY
http://sprunge.us/UiVa

I compiled the kernel with the two patches, because the patches linked in the first post does not patch against 3.17.3.
Here is the pkgbuild:
http://sprunge.us/ciHf

Where would be the perfect place for a printk to make sure all worked out?
After
+    if (!i915.enable_hd_vgaarb || !HAS_PCH_SPLIT(dev)) {

Or add in this section
if (i915.enable_hd_vgaarb)
    printk(  ... );

Edit: Tomorrow I have time to compile again.

Last edited by Rommy (2014-11-19 23:30:37)

Offline

#3330 2014-11-20 08:48:27

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm wondering if it's possible to to passthrough a video card to a Mac OS KVM guest using the described  methods. I'm not starting a legal discussion here, just considering it from a technical point of view. Has anybody tried such a configuration yet? It could be a cost effective competitor to any commercial external PCI over Thunderbolt solution.

Offline

#3331 2014-11-20 09:35:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:

I'm wondering if it's possible to to passthrough a video card to a Mac OS KVM guest using the described  methods. I'm not starting a legal discussion here, just considering it from a technical point of view. Has anybody tried such a configuration yet? It could be a cost effective competitor to any commercial external PCI over Thunderbolt solution.

It's possible

ydNpt5S.jpg

Offline

#3332 2014-11-20 09:43:54

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
dakabali wrote:

I'm wondering if it's possible to to passthrough a video card to a Mac OS KVM guest using the described  methods. I'm not starting a legal discussion here, just considering it from a technical point of view. Has anybody tried such a configuration yet? It could be a cost effective competitor to any commercial external PCI over Thunderbolt solution.

It's possible

Cool. As your screenshot shows a delegated Quadro card: Is there any restriction to the Quadro series or would it work with any (Nvidia) card? Thanks.

Offline

#3333 2014-11-20 09:55:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dakabali wrote:
nbhs wrote:
dakabali wrote:

I'm wondering if it's possible to to passthrough a video card to a Mac OS KVM guest using the described  methods. I'm not starting a legal discussion here, just considering it from a technical point of view. Has anybody tried such a configuration yet? It could be a cost effective competitor to any commercial external PCI over Thunderbolt solution.

It's possible

Cool. As your screenshot shows a delegated Quadro card: Is there any restriction to the Quadro series or would it work with any (Nvidia) card? Thanks.

It's a quadrified GTX470, and it did work on osx before i modded it, the problem i have with it (if i use it as primary) is that i have to reboot the host every time i restart the vm, i also have to boot with PciRoot=1 or i get no output. If you plan to do this i recommend checking out which cards work on "hackintosh", for example im able to pass through my radeon 6950 but i get a white screen after it finishes booting, which also happens natively.

Last edited by nbhs (2014-11-20 09:58:44)

Offline

#3334 2014-11-20 12:51:44

janze
Member
Registered: 2014-04-12
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all.
i'm wondering why i got both kvm errors. Still i can run it..

[ 516.117869] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable [ 516.250273] pci-stub 0000:03:00.0: enabling device (0000 -> 0003) [ 517.403936] pci-stub 0000:03:00.0: kvm assign device [ 517.878698] pci-stub 0000:03:00.1: enabling device (0000 -> 0002) [ 517.878781] pci-stub 0000:03:00.1: kvm assign device [ 517.980035] pci-stub 0000:00:1d.0: kvm assign device [ 522.078331] kvm: zapping shadow pages for mmio generation wraparound [ 531.279711] pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X 

My hardware:
      intel i7 4820k
      Msi X79A-GD65-8D
      Radeon R9 290x (for windows)
      Nvidia gtx480 -> Quadro 6000 (for host)

      Kernel: 3.17.3-1-ARCH
      Qemu build from AUR  [installed: 2.2.r36230.gaf3ff19-1]

VMstart script

#!/bin/bash sudo qemu-system-x86_64 --enable-kvm -m 8192 -localtime \ -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -monitor stdio -pflash /usr/share/ovmf/x64/ovmf_x64.bin \ -usbdevice tablet -vga qxl \ -drive file=/samssd/janze/virtualpc/qemu-img/titan8 \ -device pci-assign,host=03:00.0 \ -device pci-assign,host=03:00.1 \ -device pci-assign,host=00:1d.0 \ -drive file=/tilaa/data/virtimgdata

Host boot

[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=c92efce4-a1ee-4b8b-9d49-ab570fdf7546 rw intel_iommu=on pci-stub.ids=1002:67b0,1002:aac8,8086:1d26

Offline

#3335 2014-11-20 14:02:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

janze wrote:

Hey all.
i'm wondering why i got both kvm errors. Still i can run it..

[ 516.117869] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable [ 516.250273] pci-stub 0000:03:00.0: enabling device (0000 -> 0003) [ 517.403936] pci-stub 0000:03:00.0: kvm assign device [ 517.878698] pci-stub 0000:03:00.1: enabling device (0000 -> 0002) [ 517.878781] pci-stub 0000:03:00.1: kvm assign device [ 517.980035] pci-stub 0000:00:1d.0: kvm assign device [ 522.078331] kvm: zapping shadow pages for mmio generation wraparound [ 531.279711] pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X 

What errors?  Nothing here indicates an error other than you're using legacy KVM device assignment, not VFIO.

My hardware:
      intel i7 4820k
      Msi X79A-GD65-8D
      Radeon R9 290x (for windows)
      Nvidia gtx480 -> Quadro 6000 (for host)

      Kernel: 3.17.3-1-ARCH
      Qemu build from AUR  [installed: 2.2.r36230.gaf3ff19-1]

Is this a hardware mod'd Nvidia card?  EDIT:  Doesn't matter, host

VMstart script

#!/bin/bash sudo qemu-system-x86_64 --enable-kvm -m 8192 -localtime \ -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -monitor stdio -pflash /usr/share/ovmf/x64/ovmf_x64.bin \ -usbdevice tablet -vga qxl \ -drive file=/samssd/janze/virtualpc/qemu-img/titan8 \ -device pci-assign,host=03:00.0 \ -device pci-assign,host=03:00.1 \ -device pci-assign,host=00:1d.0 \ -drive file=/tilaa/data/virtimgdata

Host boot

[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=c92efce4-a1ee-4b8b-9d49-ab570fdf7546 rw intel_iommu=on pci-stub.ids=1002:67b0,1002:aac8,8086:1d26

You haven't actually shown an error, but using pci-assign you're not going to be able to reset the bus like vfio-pci will do.  You also seem to be specifying -drive options with no interface, is the problem that the VM has no disk?

EDIT: There's no guarantee that a Radeon card will work as a secondary graphics adapter for the VM.  Some people report it works, in other cases it doesn't.  It depends on the driver in the guest and is not the focus of this forum thread.

Last edited by aw (2014-11-20 14:05:19)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3336 2014-11-20 14:09:31

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

BTW, this is a classic indication that the i915 arbiter patch is either not applied or not enabled.  If you have Intel graphics and you get this, you'll never get VM boot output to the assigned VGA until it's fixed.

Omg? Overworked? To tired? I don't know. I am sorry for wasting your time.

I compiled the kernel again with the patches and edited the arbiter patch.
I didnt want to edit the diff file so I used the same line. (For every one not knowing if the patch is working: http://sprunge.us/WfOe )

dmesg says:
[   26.297580] vgaarb: enable_hd_vgaarb = 1

But the error is NOT gone

[ 26.498934] [drm:ilk_display_irq_handler] *ERROR* Pipe B FIFO underrun [ 26.498950] [drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder B [ 26.498950] [drm:cpt_serr_int_handler] *ERROR* PCH transcoder B FIFO underrun

But now I get an output at the monitor and the qemu error is gone. I dont know what happed, I did makepkg and pacman -U paket, sorry sad


Edit: Is there an easy way I do not have to be root in X?
Edit2: Ok, I found a pdf file you linked to this topic smile

Last edited by Rommy (2014-11-20 14:18:36)

Offline

#3337 2014-11-20 14:24:23

janze
Member
Registered: 2014-04-12
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable kvm: zapping shadow pages for mmio generation wraparound 

Those are what i'm wondering.. why i get those warns?
I did try whit VFIO, but after radeon driver install i got BSoD.
Pci-assing is working for me.

Thing why i wonder this is my windows 8.1 quest system interrupt process is eating CPU usage sometimes. And host cpu usage have one thread 100% usage peak same time.

Last edited by janze (2014-11-20 14:36:31)

Offline

#3338 2014-11-20 18:03:39

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello all,

I have my kernel patched (downloaded from the first post) and installed.

my kernel options are:

pcie_aspm=force acpi_osi=Linux acpi=force acpi_enforce_resources=lax i915.enable_rc6=1 i915.enable_fbc=1 i915.lvds_downclock=1 i915.semaphores=1 drm.vblankoffdelay=1 acpi_backlight=vendor intel_pstate=disable pci-stub.ids=10de:1184,10de:030a intel_iommu=on,igfx_off rdblacklist=nouveau nouveau.modeset=0 elevator=noop intremap=no_x2apic_optout libata.force=noncq quiet loglevel=3 i915.enable_hd_vgaarb=1 libata.noacpi=1 pcie_acs_override=downstream

Running the test part works perfect:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Adding the Virtio Controller works perfect as well:

-device virtio-scsi-pci,id=scsi

Problems?? Yes... when I attach the hard drive / cdrom...

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/VM/win7.img,id=disk,format=raw -device scsi-hd,drive=disk qemu-system-x86_64: -device scsi-hd,drive=disk: Property 'scsi-hd.drive' can't take value 'disk', it's in use

What I would like, if possible?
To have the hard drive with Virtio and two cdroms, one with virtio for Windows 7.iso and the other one with SATA / IDE with the Virtio drivers...

I did it before but I do not know why is giving me this error with "Property 'scsi-hd.drive' can't take value 'disk', it's in use".

Please, can someone bring me some light on this?

Thanks a lot in advance.

Best regards,

TheArcher

Offline

#3339 2014-11-20 18:35:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Hello all,

I have my kernel patched (downloaded from the first post) and installed.

my kernel options are:

pcie_aspm=force acpi_osi=Linux acpi=force acpi_enforce_resources=lax i915.enable_rc6=1 i915.enable_fbc=1 i915.lvds_downclock=1 i915.semaphores=1 drm.vblankoffdelay=1 acpi_backlight=vendor intel_pstate=disable pci-stub.ids=10de:1184,10de:030a intel_iommu=on,igfx_off rdblacklist=nouveau nouveau.modeset=0 elevator=noop intremap=no_x2apic_optout libata.force=noncq quiet loglevel=3 i915.enable_hd_vgaarb=1 libata.noacpi=1 pcie_acs_override=downstream

Oh dear, does it also run faster if you stand on your headwink

EDIT: The ironic part is that the i915 maintainer won't take the enable_hd_vgaarb patch because he doesn't want users to need to specify module options.  BTW, with all these tuning options, you do know that enable_hd_vgaarb=1 disables host DRI, right?

Running the test part works perfect:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

What was this meant to test?  You're using -vga cirrus and x-vga=off...  so you've successfully attached a secondary GPU?  The test that I'm aware of would be to use -vga none and x-vga=on in order to see if the monitor connected to the assigned GPU lights up from seabios running the VGA ROM.

Adding the Virtio Controller works perfect as well:

-device virtio-scsi-pci,id=scsi

Problems?? Yes... when I attach the hard drive / cdrom...

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/VM/win7.img,id=disk,format=raw -device scsi-hd,drive=disk qemu-system-x86_64: -device scsi-hd,drive=disk: Property 'scsi-hd.drive' can't take value 'disk', it's in use

What I would like, if possible?
To have the hard drive with Virtio and two cdroms, one with virtio for Windows 7.iso and the other one with SATA / IDE with the Virtio drivers...

I did it before but I do not know why is giving me this error with "Property 'scsi-hd.drive' can't take value 'disk', it's in use".

disk might be a reserved word, maybe use disk0

Last edited by aw (2014-11-20 19:05:25)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3340 2014-11-20 19:59:30

Borg8401
Member
Registered: 2014-02-11
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was wondering what options people are recommending when compiling QEMU, currently I have:

./configure \ --prefix=/usr \ --enable-xen \ --enable-xen-pci-passthrough \ --enable-kvm

Anything else I should be using for stability/performance?

I'm currently using Ubuntu 14.04 and 2.2.0-rc2 off of the QEMU wiki since the included package on Ubuntu doesn't have the kvm=off option to kill code 43.

Asus Rampage IV Extreme (x79) and 3930K
ATI 4890 (host) and GTX 780 (passthrough)

Offline

#3341 2014-11-20 20:02:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borg8401 wrote:

I was wondering what options people are recommending when compiling QEMU

I typically only use --prefix for local builds.  Why do you need the xen options?  KVM support should be enabled automatically.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3342 2014-11-21 01:05:40

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you do something special to get the VGA Card working in the guest? I am not a windows pro, but I thought, I only have to install the nvidia blob. Debuging is not easy in Windows 8 :-(

Offline

#3343 2014-11-21 04:42:21

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:

Hello all,

I have my kernel patched (downloaded from the first post) and installed.

my kernel options are:

pcie_aspm=force acpi_osi=Linux acpi=force acpi_enforce_resources=lax i915.enable_rc6=1 i915.enable_fbc=1 i915.lvds_downclock=1 i915.semaphores=1 drm.vblankoffdelay=1 acpi_backlight=vendor intel_pstate=disable pci-stub.ids=10de:1184,10de:030a intel_iommu=on,igfx_off rdblacklist=nouveau nouveau.modeset=0 elevator=noop intremap=no_x2apic_optout libata.force=noncq quiet loglevel=3 i915.enable_hd_vgaarb=1 libata.noacpi=1 pcie_acs_override=downstream

Oh dear, does it also run faster if you stand on your headwink

EDIT: The ironic part is that the i915 maintainer won't take the enable_hd_vgaarb patch because he doesn't want users to need to specify module options.  BTW, with all these tuning options, you do know that enable_hd_vgaarb=1 disables host DRI, right?

About all those options is because I was checking one by one all the error on journalctl when I installed arch and found solutions for all the messages and other problems (vsync, power control,...)

Yes, I know that disables the DRI inside but I do not care if works fine smile

Running the test part works perfect:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

What was this meant to test?  You're using -vga cirrus and x-vga=off...  so you've successfully attached a secondary GPU?  The test that I'm aware of would be to use -vga none and x-vga=on in order to see if the monitor connected to the assigned GPU lights up from seabios running the VGA ROM.

Sorry, that was my fault, I tested it with vga none and x-vga=on and working fine (later I disabled it for not changing to HDMI2 on my TV all time)

Adding the Virtio Controller works perfect as well:

-device virtio-scsi-pci,id=scsi

Problems?? Yes... when I attach the hard drive / cdrom...

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/VM/win7.img,id=disk,format=raw -device scsi-hd,drive=disk qemu-system-x86_64: -device scsi-hd,drive=disk: Property 'scsi-hd.drive' can't take value 'disk', it's in use

What I would like, if possible?
To have the hard drive with Virtio and two cdroms, one with virtio for Windows 7.iso and the other one with SATA / IDE with the Virtio drivers...

I did it before but I do not know why is giving me this error with "Property 'scsi-hd.drive' can't take value 'disk', it's in use".

disk might be a reserved word, maybe use disk0

I tested with different words there but still the same result...

Anyway... thanks a lot for your quick reply!!

Regards,


TheArcher

Offline

#3344 2014-11-21 04:42:31

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Did you do something special to get the VGA Card working in the guest? I am not a windows pro, but I thought, I only have to install the nvidia blob. Debuging is not easy in Windows 8 :-(

Sorry for stepping in , but I didn't need any special configuration to make both of my Windows 8.1 VMs work . I Just installed Nvidia's drivers and they were good to go . Also Windows 10 TP worked good (although I didn't install any drivers , because I was just testing the OS itself) .

I will do a full Windows 10 installation today if only OVMF would recognize my USB controller for my keyboard to work sad . Oh wellz .

Last edited by Denso (2014-11-21 04:43:25)

Offline

#3345 2014-11-21 15:48:33

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your answer. I deltetd my drive and installed Win8 again in a new drive. This time I let Windows search for VGA driver. Now the device manager says error code: 43. I dont really know how to proceed.
I know, why I use linux -.-

Little question: How can I eject the card in a Linux guest?

Offline

#3346 2014-11-21 18:40:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Thank you for your answer. I deltetd my drive and installed Win8 again in a new drive. This time I let Windows search for VGA driver. Now the device manager says error code: 43. I dont really know how to proceed.
I know, why I use linux -.-

Little question: How can I eject the card in a Linux guest?

1. You just don't tell windowns "search for drivers yourself, you filthy slave!", you feed it with the GUI installer DL'ed from vendor's site. And if it likes it - everything will work. It might dislike the "food" or the feeding process and randomly throw up BSOD, sometimes the driver install may get completed, but after a reboot it will go BSOD all over your screen and stuff.

2.Try rmmod(or any other way you like) to unload the modules that claim the card, and you're good to go "physically" unplugging it, afair. Maybe the device needs to be powered down via sysfs or something.

EDIT:http://www.linux-kvm.org/page/Hotadd_pci_devices
You'll also need appropriate modules loaded to get pci hotplug working.

Last edited by Duelist (2014-11-21 19:02:20)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3347 2014-11-21 19:02:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Thank you for your answer. I deltetd my drive and installed Win8 again in a new drive. This time I let Windows search for VGA driver. Now the device manager says error code: 43. I dont really know how to proceed.
I know, why I use linux -.-

VFIO VGA FAQ question 10

Little question: How can I eject the card in a Linux guest?

Something in /sys I'm sure... you were saying about linux...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3348 2014-11-21 19:46:01

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any suggestion to my "it's in use" issue?

I really do not know what else to try...

Thanks wink

Offline

#3349 2014-11-21 19:54:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Any suggestion to my "it's in use" issue?

I really do not know what else to try...

Thanks wink

Add

,if=none

to your disk option, otherwise I think it's automatically getting added to the vfio-scsi-pci device before the scsi-hd device is being parsed.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3350 2014-11-21 20:15:30

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:

Any suggestion to my "it's in use" issue?

I really do not know what else to try...

Thanks wink

Add

,if=none

to your disk option, otherwise I think it's automatically getting added to the vfio-scsi-pci device before the scsi-hd device is being parsed.

Thanks smile

With that seems to load the fine the hard drive but how can I attach both cdroms (one virtio and one IDE/SATA) if all are giving me the same error?

Offline

#3351 2014-11-21 20:24:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
aw wrote:
thearcherblog wrote:

Any suggestion to my "it's in use" issue?

I really do not know what else to try...

Thanks wink

Add

,if=none

to your disk option, otherwise I think it's automatically getting added to the vfio-scsi-pci device before the scsi-hd device is being parsed.

Thanks smile

With that seems to load the fine the hard drive but how can I attach both cdroms (one virtio and one IDE/SATA) if all are giving me the same error?

I'm not sure you're reasoning for wanting a virtio cdrom really makes sense, but last time I did an install to a virtio disk I used this to have one cd for the install iso and one for the virtio iso:

-drive file=/path/to/windows.iso,if=none,id=dvd0,media=cdrom \ -device ide-cd,drive=dvd0,bootindex=0 \ -drive file=/path/to/virtio-drivers.iso,if=none,id=dvd1,media=cdrom \ -device ide-cd,drive=dvd1 \

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3352 2014-11-21 20:41:32

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:
aw wrote:

Add

,if=none

to your disk option, otherwise I think it's automatically getting added to the vfio-scsi-pci device before the scsi-hd device is being parsed.

Thanks smile

With that seems to load the fine the hard drive but how can I attach both cdroms (one virtio and one IDE/SATA) if all are giving me the same error?

I'm not sure you're reasoning for wanting a virtio cdrom really makes sense, but last time I did an install to a virtio disk I used this to have one cd for the install iso and one for the virtio iso:

-drive file=/path/to/windows.iso,if=none,id=dvd0,media=cdrom \ -device ide-cd,drive=dvd0,bootindex=0 \ -drive file=/path/to/virtio-drivers.iso,if=none,id=dvd1,media=cdrom \ -device ide-cd,drive=dvd1 \

Thanks a lot!!

I will try as soon as I arrive home smile

The reason is (please correct me if I'm wrong), Virtio devices should be faster and the IDE cdrom is for having the ISO drivers mounted so I can install Windows 7 and browse for the drivers on the installation smile

Regards,


TheArcher

Offline

#3353 2014-11-21 20:47:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

The reason is (please correct me if I'm wrong), Virtio devices should be faster and the IDE cdrom is for having the ISO drivers mounted so I can install Windows 7 and browse for the drivers on the installation smile

That sounds like excessive optimization to me.  IDE emulation is not that bad, especially for an optical drive where you're not seeking all over the platter and doing a zillion small reads and writes.  When you're backing an IDE drive by a disk image of the ISO, it's already an order of magnitude faster than real hardware.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3354 2014-11-21 21:19:52

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:

The reason is (please correct me if I'm wrong), Virtio devices should be faster and the IDE cdrom is for having the ISO drivers mounted so I can install Windows 7 and browse for the drivers on the installation smile

That sounds like excessive optimization to me.  IDE emulation is not that bad, especially for an optical drive where you're not seeking all over the platter and doing a zillion small reads and writes.  When you're backing an IDE drive by a disk image of the ISO, it's already an order of magnitude faster than real hardware.

Ok, then, recommended installation for Windows 7 would be with SATA hard drive and IDE cdrom? and in that case, can you help me with the setup, please? (I check the first post but I still have the same issue... maybe I need to sleep more and try tomorrow smile

Thanks a lot for you help, I will let you know how it goes tomorrow smile

Regards,



TheArcher

Offline

#3355 2014-11-21 21:22:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
aw wrote:
thearcherblog wrote:

The reason is (please correct me if I'm wrong), Virtio devices should be faster and the IDE cdrom is for having the ISO drivers mounted so I can install Windows 7 and browse for the drivers on the installation smile

That sounds like excessive optimization to me.  IDE emulation is not that bad, especially for an optical drive where you're not seeking all over the platter and doing a zillion small reads and writes.  When you're backing an IDE drive by a disk image of the ISO, it's already an order of magnitude faster than real hardware.

Ok, then, recommended installation for Windows 7 would be with SATA hard drive and IDE cdrom? and in that case, can you help me with the setup, please? (I check the first post but I still have the same issue... maybe I need to sleep more and try tomorrow smile

Wait a sec, I never recommended SATA for the hard disk.  The above was all about the optical drives.  I'd still use virtio for the hard drive in the VM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3356 2014-11-22 07:20:20

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:
aw wrote:

That sounds like excessive optimization to me.  IDE emulation is not that bad, especially for an optical drive where you're not seeking all over the platter and doing a zillion small reads and writes.  When you're backing an IDE drive by a disk image of the ISO, it's already an order of magnitude faster than real hardware.

Ok, then, recommended installation for Windows 7 would be with SATA hard drive and IDE cdrom? and in that case, can you help me with the setup, please? (I check the first post but I still have the same issue... maybe I need to sleep more and try tomorrow smile

Wait a sec, I never recommended SATA for the hard disk.  The above was all about the optical drives.  I'd still use virtio for the hard drive in the VM.

Good morning,

Ok, I will try now with virtio for the hard drive and sata/ide for the cdrom.

I need 2 optical drives, one for Windows iso and the second one for virtio drivers.

I will let you know in a few minutes how is going.

Thanks a lot for your help!!!

Regards,


TheArcher

Edit: Works perfect with the first DVD but when I add the second one gives me the following error:

qemu-system-x86_64: -device ide-cd,drive=dvd1: Can't create IDE unit 1, bus supports only 1 units qemu-system-x86_64: -device ide-cd,drive=dvd1: Device initialization failed. qemu-system-x86_64: -device ide-cd,drive=dvd1: Device 'ide-cd' could not be initialized

I will try to mount the second optical drive in another way smile

Thanks a lot master!

Last edited by thearcherblog (2014-11-22 07:47:42)

Offline

#3357 2014-11-22 08:28:55

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Edit: Works perfect with the first DVD but when I add the second one gives me the following error:

Hi . Try using this :

-device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Windows.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \

Change the paths to ISOs accordingly .

Hope this helps smile

Offline

#3358 2014-11-22 08:56:32

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
thearcherblog wrote:

Edit: Works perfect with the first DVD but when I add the second one gives me the following error:

Hi . Try using this :

-device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Windows.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \

Change the paths to ISOs accordingly .

Hope this helps smile

Hello smile

Thanks a lot for your suggestion smile

Seems like boots fine into the Windows 7 installation but cannot load the virtio drivers, seems like is not "mounting" the virtio.iso.

Right now my qemu command is:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/VM/win7.img,if=virtio -drive file=/home/jgomez/Downloads/X17-59186.iso,if=none,id=dvd0,media=cdrom -device scsi-cd,drive=dvd0,bootindex=0 -cdrom /home/jgomez/Downloads/virtio.iso 

Regards,


TheArcher

New way of booting... but Windows is not detecting CD/DVD or HD...:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/jgomez/VM/win7.img,if=virtio -cdrom /home/jgomez/Downloads/X17-59186.iso -drive file=/home/jgomez/Downloads/virtio.iso,index=3,media=cdrom -rtc base=localtime,clock=host -usbdevice tablet -boot menu=on

Last edited by thearcherblog (2014-11-22 10:54:12)

Offline

#3359 2014-11-22 12:55:31

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems like boots fine into the Windows 7 installation but cannot load the virtio drivers, seems like is not "mounting" the virtio.iso.

umm ... Can you please try the exact command lines I posted and see if it works ? I use those lines for all of my VMs and they are working just fine .

PS . You don't need to add "-bios /path/to/bios/bin" in recent QEMU releases .

Last edited by Denso (2014-11-22 12:56:48)

Offline

#3360 2014-11-22 14:05:35

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Seems like boots fine into the Windows 7 installation but cannot load the virtio drivers, seems like is not "mounting" the virtio.iso.

umm ... Can you please try the exact command lines I posted and see if it works ? I use those lines for all of my VMs and they are working just fine .

PS . You don't need to add "-bios /path/to/bios/bin" in recent QEMU releases .

With the following command:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/VM/win7.img -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -boot menu=on

I start the installation but gives me an error about "required CD/DVD device driver is missing..." ; if I browse looking for drivers I can see the X drive (dvd mounted with HD icon) and C and D (from the win7.img, previous installation) but I cannot see the virtio.iso or the Windows iso mounted as a DVD instead of CD/DVD.

Anyway, on the Qemu bios during the F12 boot menu I can see the AHCI hard drive and both DVD...

Please, do not hesitate to contact me to make more  testing because, hoestly, I do not know what else to try smile

Regards,


TheArcher

Last edited by thearcherblog (2014-11-22 15:36:45)

Offline

#3361 2014-11-22 15:48:38

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry for not providing the commandline for the HDD too . Here , try these :

-drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /home/jgomez/Downloads/virtio.iso \

Just copy/paste , I modified it to suit your files paths wink

Keep us posted !

Offline

#3362 2014-11-22 16:20:07

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Sorry for not providing the commandline for the HDD too . Here , try these :

-drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /home/jgomez/Downloads/virtio.iso \

Just copy/paste , I modified it to suit your files paths wink

Keep us posted !

Hello sir!!

I tried with you command attached to my the one I have:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -vga cirrus -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -boot menu=on

And boots ok from Windows CD (and the other DVD (virtio drivers) and the HD (with virtio drivers) look fine on F12 boot menu but windows does not detect anything during installation and when I browse looking for drivers, only X drive with HD icon is there.

F12 Boot Menu shows:

1.- Virtio disk PCI:0:3
2.- Legacy option rom
3.- Floppy [drive A]
4.- DVD/CD [AHCI/2: QEMU DVD-ROM ATAPI-4 DVD/CD]
5.- DVD/CD [virtio-scsi Drive QEMU QEMU CD-ROM 2.1.]]
6.- iPXE (PCI 00:02.0)

Thanks a lot for your help but still having the same issue. Maybe I have something wrong and I do not know sad

Regards,


TheArcher

Last edited by thearcherblog (2014-11-22 16:27:43)

Offline

#3363 2014-11-22 16:22:43

biasha
Member
Registered: 2014-11-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

in the examples i've seen vga card is connected via ioh3420, but i don't see difference when connect it directly.

could someone please explain which way is correct?

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

or simply

-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \

Offline

#3364 2014-11-22 16:32:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Edit: Works perfect with the first DVD but when I add the second one gives me the following error:

qemu-system-x86_64: -device ide-cd,drive=dvd1: Can't create IDE unit 1, bus supports only 1 units qemu-system-x86_64: -device ide-cd,drive=dvd1: Device initialization failed. qemu-system-x86_64: -device ide-cd,drive=dvd1: Device 'ide-cd' could not be initialized

Perhaps because you're using q35, which I no longer recommend for Windows guests.  q35 uses sata, so you only get 1 unit per "bus".  440fx emulates pata, so you get 2 units per "bus".


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3365 2014-11-22 16:33:39

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:

Edit: Works perfect with the first DVD but when I add the second one gives me the following error:

qemu-system-x86_64: -device ide-cd,drive=dvd1: Can't create IDE unit 1, bus supports only 1 units qemu-system-x86_64: -device ide-cd,drive=dvd1: Device initialization failed. qemu-system-x86_64: -device ide-cd,drive=dvd1: Device 'ide-cd' could not be initialized

Perhaps because you're using q35, which I no longer recommend for Windows guests.  q35 uses sata, so you only get 1 unit per "bus".  440fx emulates pata, so you get 2 units per "bus".

Ahhh ok, to be honest, I do not have any preference if I can pass my VGA smile

Can you share your full qemu command to start my computer using 440fx? Because I think that with the 440fx I cannot use the ioh3420 so... this is a mess in my head XD

And thanks a lot for you help and support smile

Last edited by thearcherblog (2014-11-22 16:35:42)

Offline

#3366 2014-11-22 16:35:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

biasha wrote:

in the examples i've seen vga card is connected via ioh3420, but i don't see difference when connect it directly.

could someone please explain which way is correct?

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

or simply

-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \

If you're running a Windows guest, I recommend 440fx and the latter (but note that for multifunction=on to be useful you need to specify an addr= like done in the first example, but not 0.).  Q35 and the former example are only necessary for Linux guests and drivers that assume the graphics card topology.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3367 2014-11-22 16:38:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
aw wrote:
thearcherblog wrote:

Edit: Works perfect with the first DVD but when I add the second one gives me the following error:

qemu-system-x86_64: -device ide-cd,drive=dvd1: Can't create IDE unit 1, bus supports only 1 units qemu-system-x86_64: -device ide-cd,drive=dvd1: Device initialization failed. qemu-system-x86_64: -device ide-cd,drive=dvd1: Device 'ide-cd' could not be initialized

Perhaps because you're using q35, which I no longer recommend for Windows guests.  q35 uses sata, so you only get 1 unit per "bus".  440fx emulates pata, so you get 2 units per "bus".

Ahhh ok, to be honest, I do not have any preference if I can pass my VGA smile

Can you share your full qemu command to start my computer using 440fx? Because I think that with the 440fx I cannot use the ioh3420 so... this is a mess in my head XD

And thanks a lot for you help and support smile

There's an example in the blog


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3368 2014-11-22 17:02:31

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,hv-time,kvm=off -smp sockets=1,cores=8 -vga cirrus -monitor stdio -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -boot menu=on -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off [sudo] password for jgomez: QEMU 2.1.91 monitor - type 'help' for more information (qemu) qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off: PCI: slot 1 function 0 not available for vfio-pci, in use by PIIX3 qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off: Device 'vfio-pci' could not be initialized

I do not know if I'm closer to be working or not XD

Suggestions or ideas?

Thanks a lot to all for the patience smile

Last edited by thearcherblog (2014-11-22 17:09:33)

Offline

#3369 2014-11-22 17:19:21

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,hv-time,kvm=off -smp sockets=1,cores=8 -vga cirrus -monitor stdio -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -boot menu=on -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off [sudo] password for jgomez: QEMU 2.1.91 monitor - type 'help' for more information (qemu) qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off: PCI: slot 1 function 0 not available for vfio-pci, in use by PIIX3 qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,multifunction=on,addr=1.0,x-vga=off: Device 'vfio-pci' could not be initialized

I do not know if I'm closer to be working or not XD

Suggestions or ideas?

Thanks a lot to all for the patience :)

man, are you happy with completely turning vga passthrough thread to qemu newbie tech support one (these are entirely unrelated areas of knowledge, in case you care)?

stop bugging people with your completely thread-unrelated stuff and do this:
- install windows without virtio
- add virtio dummy (e.g.: /dev/null) hdd image and virtio drivers iso (this is a good step to switch your network to virtio as well)
- boot into windows, install all the drivers it asks, shutdown
- remove virtio drivers iso, switch your main hdd to virtio
- done

in case you fail at any step - read the damned mans or use google: vga passthrough DO REQUIRE some basic knowledge

Offline

#3370 2014-11-22 17:20:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
PCI: slot 1 function 0 not available for vfio-pci, in use by PIIX3

You've picked an addr= that's in use by the chipset, please try to read and understand the error message.  The example in the blog uses 9.0 for the GPU.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3371 2014-11-22 17:24:10

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sinny:

Sorry, that was not my intention at all.

I was starting with the first post and found some problems with it. I'm not expert at all, but I'm not completely newbie.

I guess that in this thread not all the people have this BASIC KNOWLEDGE that you say but, anyway, thanks a lot.

I will try on my own way and share it after if it's a succesfull story smile

Anyway, thanks a lot to all smile

Regards,


TheArcher

Offline

#3372 2014-11-22 18:06:08

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@thearcherblog :

Here is my full command line , I'm using 440FX + OVMF :

qemu-system-x86_64 -name main -nographic -mem-path /dev/hugepages \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -vga none \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_main.bin \ -device vfio-pci,host=05:00.0,multifunction=on \ -device vfio-pci,host=05:00.1 \ -device vfio-pci,host=0f:00.0 \ -drive file=/VMs/Win_Main.qed,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win10.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -net nic,model=virtio,macaddr=64:C5:63:3C:1B:22 -net bridge,br=br0 \ -localtime \ -monitor unix:/tmp/vm_main,server,nowait &

Hope it helps !

Last edited by Denso (2014-11-22 18:06:40)

Offline

#3373 2014-11-22 20:48:14

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@thearcherblog :

Here is my full command line , I'm using 440FX + OVMF :

qemu-system-x86_64 -name main -nographic -mem-path /dev/hugepages \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -vga none \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_main.bin \ -device vfio-pci,host=05:00.0,multifunction=on \ -device vfio-pci,host=05:00.1 \ -device vfio-pci,host=0f:00.0 \ -drive file=/VMs/Win_Main.qed,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win10.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -net nic,model=virtio,macaddr=64:C5:63:3C:1B:22 -net bridge,br=br0 \ -localtime \ -monitor unix:/tmp/vm_main,server,nowait &

Hope it helps !

Hello sir!!

Good news and bad news smile

The good ones: I manage the installation using the following command (Virtio for Hard drive + DVD and IDE for the drivers, 2 DVD + 1 HD)

sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,hv-time,kvm=off -smp sockets=1,cores=8 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/X17-59186.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1 -boot menu=on -vga none

The bad ones... Error 43 XD

I read a lot about and I have the kvm=off option and my drivers are 340.52 (according to http://vfio.blogspot.no/2014/08/vfiovga-faq.html shoud be ok)

At the moment I'm using teamviewer to remote connect to that computer... I think I need to do it with VNC or similar but I also need to make a bridge network... too many things for today!

Anyway, I'm really happy and I will continue tomorrow a bit more smile

But before going to sleep... I want to give a big thanks to all of you!!!

Offline

#3374 2014-11-23 06:09:28

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi . Good to hear that you got your installation done !

Try removing "hv-time" , and see if it would work .

If it didn't work , you can try 440FX + OVMF . See my script above and adapt it to your configuration .

You'd need a UEFI capable GPU + ovmf-svn from AUR .

EDIT :

I also use VNC to setup my VMs in the first stage . But then , you must disable it for VGA passthrough to work .

To enable it , use :

-vga std \ -vnc :0 \

Then use a VNC client , point it to the host's IP:PORT , like : 10.0.1.55:0

Or even better , you can setup TightVNC server inside Windows , and use it with GPU passthrough at the same time .

Last edited by Denso (2014-11-23 08:16:28)

Offline

#3375 2014-11-23 08:50:09

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hi . Good to hear that you got your installation done !

Try removing "hv-time" , and see if it would work .

If it didn't work , you can try 440FX + OVMF . See my script above and adapt it to your configuration .

You'd need a UEFI capable GPU + ovmf-svn from AUR .

EDIT :

I also use VNC to setup my VMs in the first stage . But then , you must disable it for VGA passthrough to work .

To enable it , use :

-vga std \ -vnc :0 \

Then use a VNC client , point it to the host's IP:PORT , like : 10.0.1.55:0

Or even better , you can setup TightVNC server inside Windows , and use it with GPU passthrough at the same time .

Good morning all smile

Following you steps, I manage to install Windows 8 (7 was not working do not know why) smile

Unfortunately, I'm still with error 43. Version 340.52.

This is my startup command:

sudo qemu-system-x86_64 -name main -enable-kvm -localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin -smp sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -device vfio-pci,host=01:00.0,multifunction=on -device vfio-pci,host=01:00.1 -cdrom /home/jgomez/Downloads/virtio.iso -vga none

Connecting through Teamviewer (I will check VNC after smile)

I will investigate a bit more and if I'm success I will let you know really soon smile

Thanks a lot!

Offline

#3376 2014-11-23 10:53:36

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Unfortunately, I'm still with error 43. Version 340.52.

Maybe it's a hardware issue , did others have success passing through the same GPU as yours ?

Anyway , you're missing an extra line regarding OVMF . You need two files , please see my post :

https://bbs.archlinux.org/viewtopic.php … 9#p1466309

Best of luck smile

Offline

#3377 2014-11-23 11:34:28

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Unfortunately, I'm still with error 43. Version 340.52.

Maybe it's a hardware issue , did others have success passing through the same GPU as yours ?

Anyway , you're missing an extra line regarding OVMF . You need two files , please see my post :

https://bbs.archlinux.org/viewtopic.php … 9#p1466309

Best of luck smile

Hello sir smile

It's truth, I forgot the second bios parameter...

I made a copy of the bin fine and put it inseide my VM directory with the win7_bios.bin name

Started the whole thing with:

sudo qemu-system-x86_64 -name main -enable-kvm -localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/win7_bios.bin -smp sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -device vfio-pci,host=01:00.0,multifunction=on -device vfio-pci,host=01:00.1 -cdrom /home/jgomez/Downloads/virtio.iso -vga none

But even when the QEMU monitor starts, seems like is not booting or nothing happens (teamviewer does not see the computer).

I also tried with -vga cirrus but black screen. If I remove the "sencond" bios parameters works as usual.

Regards,


TheArcher

Offline

#3378 2014-11-23 11:34:56

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I have a working setup based on Intel CPU and Z77 chipset + GTX 970 as graphic cards. However I have one question, and one issue.

First, here is my question: whats the point using a bios dump (from graphic card) in the qemu command-line? I am not doing it yet and everything seems ok. Good performance, VM reboots ok, etc.

Then, my little issue. I have an USB sound card : Asus Xonar U7. If I use it on a USB hub (shared with my mouse + keyboard), then I have some sound issues. It stutters sometimes, and causes lag. Only happens during games; never while watching a movie / youtube / listening to music.
I decided to go with emulated audio (cf. command line below). Now i non longer have those sound / lags issue. However I have a little latency in sound. All the time. The sound occurs half a second (or even less) after the real action. I tried with various PA samples from 128 to 16384 (cf. command line). 128 to 512 is unusable, audio lag and video lag. And high values (above 4096) make the latency higher. So far, it seems that the best value is 1024.

Do you have any idea what to do?

Let me know if you need additional information.
Thanks in advance.

Edit ; HDMI audio is not a solution since my screen as no audio jack output (and on screen's speaker are just horrible).

Command line:

QEMU_PA_SAMPLES=1024 QEMU_AUDIO_DRV=pa \ qemu-system-x86_64 -name game -M q35 -enable-kvm \ -mem-path /dev/hugepages \ -balloon virtio \ -nographic \ -m 16384 \ -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -realtime mlock=on -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/guest/Windaube_7,id=sys,format=raw -device scsi-hd,drive=sys \ -drive file=/dev/guest/win7,id=data,format=raw -device scsi-hd,drive=data \ -device vfio-pci,host=00:1d.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi2 \ -net nic,model=virtio,macaddr=52:54:00:7c:3c:42 -net bridge,br=virbr0 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Last edited by Nesousx (2014-11-23 15:23:59)

Offline

#3379 2014-11-23 14:39:44

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also tried with -vga cirrus but black screen. If I remove the "sencond" bios parameters works as usual.

I think it is because you've already installed Windows . Windows writes its VARS during the installation process .

First, here is my question: whats the point using a bios dump (from graphic card) in the qemu command-line? I am not doing it yet and everything seems ok. Good performance, VM reboots ok, etc.

BIOS dumps are not needed if your VGA passthrough is working correctly . You won't notice any difference in performance if you provided a BIOS ROM file for your GPU .

Sorry , I passthrough-ed the onboard sound to my VM and it is working perfectly . Also HDMI (GPU) sound works aswell . I can't help with emulated sound cards .

Offline

#3380 2014-11-23 15:25:01

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I also tried with -vga cirrus but black screen. If I remove the "sencond" bios parameters works as usual.

I think it is because you've already installed Windows . Windows writes its VARS during the installation process .

First, here is my question: whats the point using a bios dump (from graphic card) in the qemu command-line? I am not doing it yet and everything seems ok. Good performance, VM reboots ok, etc.

BIOS dumps are not needed if your VGA passthrough is working correctly . You won't notice any difference in performance if you provided a BIOS ROM file for your GPU .

Sorry , I passthrough-ed the onboard sound to my VM and it is working perfectly . Also HDMI (GPU) sound works aswell . I can't help with emulated sound cards .

Ok, thanks for info. However, can you let me know how do you get sound in Linux then?

At some point I wanted to used onboard sound for Windows, but then I have no sound in Linux unless I plug / unplug an audio cable or buy some expensive KVM (I use DP and WQHD resolution).

Last edited by Nesousx (2014-11-23 15:27:31)

Offline

#3381 2014-11-23 15:48:37

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, thanks for info. However, can you let me know how do you get sound in Linux then?

I didn't need to , it's a headless VM + File server .

All of my gaming + daily + email needs are served from VMs . This way I kept my Arch as clean as possible .

Offline

#3382 2014-11-23 16:16:37

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello again,

Windows 8 reinstalled, using the following command:

sudo qemu-system-x86_64 -name main -enable-kvm -localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -device vfio-pci,host=01:00.0,multifunction=on -device vfio-pci,host=01:00.1 -cdrom /home/jgomez/Downloads/virtio.iso -vga none

Error Code 43 with several drivers... Feeling so close and so far XD

smile

Offline

#3383 2014-11-23 16:37:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Hello again,

Windows 8 reinstalled, using the following command:

sudo qemu-system-x86_64 -name main -enable-kvm -localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -device vfio-pci,host=01:00.0,multifunction=on -device vfio-pci,host=01:00.1 -cdrom /home/jgomez/Downloads/virtio.iso -vga none

Error Code 43 with several drivers... Feeling so close and so far XD

smile

Have you ever told us what model GPU you're trying to assign?  I'm starting to wonder if your BIOS is set correctly to make the assigned GPU a secondary device on the host, whether you've got the host drivers blacklisted or excluded from the device with pci-stub.ids, or whether dmesg reports errors loading the device ROM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3384 2014-11-23 17:07:28

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:

Hello again,

Windows 8 reinstalled, using the following command:

sudo qemu-system-x86_64 -name main -enable-kvm -localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -device vfio-pci,host=01:00.0,multifunction=on -device vfio-pci,host=01:00.1 -cdrom /home/jgomez/Downloads/virtio.iso -vga none

Error Code 43 with several drivers... Feeling so close and so far XD

smile

Have you ever told us what model GPU you're trying to assign?  I'm starting to wonder if your BIOS is set correctly to make the assigned GPU a secondary device on the host, whether you've got the host drivers blacklisted or excluded from the device with pci-stub.ids, or whether dmesg reports errors loading the device ROM.

Hello,

Thanks a lot for your reply!

I think everything is correct but I will show you all my configuration and logs if necessary smile

My processor is an intel 4790K and the motherboard is a Z97 Extreme 6 from Asrock.

Vt-d is enabled on UEFI.



My graphic card is a 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
My Linux (Arch) is configured with the Intel driver and the Kernel options are:

options root=/dev/sda2 rw pcie_aspm=force acpi_osi=Linux acpi=force acpi_enforce_resources=lax i915.enable_rc6=1 i915.enable_fbc=1 i915.lvds_downclock=1 i915.semaphores=1 drm.vblankoffdelay=1 acpi_backlight=vendor intel_pstate=disable pci-stub.ids=10de:1184,10de:030a intel_iommu=on,igfx_off rdblacklist=nouveau nouveau.modeset=0 elevator=noop intremap=no_x2apic_optout libata.force=noncq loglevel=3 i915.enable_hd_vgaarb=1 libata.noacpi=1 pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet

Kernel downloaded from the first post:

journalctl -b | grep vgaarb

Nov 23 11:59:00 myArch kernel: vgaarb: setting as boot device: PCI:0000:00:02.0 Nov 23 11:59:00 myArch kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none Nov 23 11:59:00 myArch kernel: vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none Nov 23 11:59:00 myArch kernel: vgaarb: loaded Nov 23 11:59:00 myArch kernel: vgaarb: bridge control possible 0000:01:00.0 Nov 23 11:59:00 myArch kernel: vgaarb: no bridge control possible 0000:00:02.0 Nov 23 11:59:01 myArch kernel: vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

The only error I can find, at the moment, is running dmesg:

[21438.474193] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 228fa5000 DMAR:[fault reason 12] non-zero reserved fields in PTE

Please, do not hesitate to request me any log smile

Thanks a lot and best regards,


TheArcher

Offline

#3385 2014-11-23 17:14:49

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My Linux (Arch) is configured with the Intel driver and the Kernel options are:

Wow , lots of parameters ! Why would you need all of these ? I have the same GTX 770 (EVGA SC) , and they work good . With only "intel_iommu=on" and correct "pci-stub.ids" .

Please , trim your boot parameters , I'm sure you don't need OR want all of those .

EDIT : here is my boot line :

intel_iommu=on elevator=noop modprobe.blacklist=nouveau pci-stub.ids=10de:104a,10de:0e08,10de:1184,10de:0e0a,1912:0015

The only thing I imagine you would need to add is the "i915.enable_hd_vgaarb=1" and the "pcie_acs_override=downstream" parameters .

Last edited by Denso (2014-11-23 17:23:10)

Offline

#3386 2014-11-23 19:21:36

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

My Linux (Arch) is configured with the Intel driver and the Kernel options are:

Wow , lots of parameters ! Why would you need all of these ? I have the same GTX 770 (EVGA SC) , and they work good . With only "intel_iommu=on" and correct "pci-stub.ids" .

Please , trim your boot parameters , I'm sure you don't need OR want all of those .

EDIT : here is my boot line :

intel_iommu=on elevator=noop modprobe.blacklist=nouveau pci-stub.ids=10de:104a,10de:0e08,10de:1184,10de:0e0a,1912:0015

The only thing I imagine you would need to add is the "i915.enable_hd_vgaarb=1" and the "pcie_acs_override=downstream" parameters .

Hello smile

I removed all my kernel boot parameters but still on the same situation:

intel_iommu=on,igfx_off elevator=noop modprobe.blacklist=nouveau pci-stub.ids=10de:1184,10de:030a rdblacklist=nouveau nouveau.modeset=0 i915.enable_hd_vgaarb=1 pcie_acs_override=downstream quiet

I kept only the nouveau related ones and the igfx_off to have sound on host with HDMI smile

I'm open to the next step wink

Edit: I have a question but not sure if will help smile

Seems like if I'm running the qemu with the -vga qxl then shows the nvidia graphic card well installed... Don't know for sure if that means that is working fine or not and I do not know if I can force to use the Nvidia one for gaming... Maybe I'm just crazy and I need to sleep smile

Thanks a lot and regards,


TheArcher

Last edited by thearcherblog (2014-11-23 19:30:01)

Offline

#3387 2014-11-23 20:38:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

The only error I can find, at the moment, is running dmesg:

[21438.474193] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 228fa5000 DMAR:[fault reason 12] non-zero reserved fields in PTE

That doesn't look good.  The only thing I can think that would get that particular error was fixed in v3.15, what kernel version are you using?

EDIT: And if it's the bit I'm thinking of, it could definitely contribute to the Code 43 problem

Last edited by aw (2014-11-23 20:40:52)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3388 2014-11-23 20:43:14

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems like if I'm running the qemu with the -vga qxl then shows the nvidia graphic card well installed... Don't know for sure if that means that is working fine or not and I do not know if I can force to use the Nvidia one for gaming... Maybe I'm just crazy and I need to sleep

You can passthrough it as a secondary GPU for your VM . I tried it , and it switches to the nvidia GPU as soon as Windows enters to desktop . I don't think its related to Code 43 issue .

Any special errors in dmesg when you run the VM ? Maybe ROM errors would appear when you launch the VM . Have you tried passing a ROM file along the QEMU command line ?

EDIT :

Didn't notice the DMAR error you posted earlier .

I used to recieve a similar error , and if I remember correctly , setting some modules parameters helped :

try to create a file as :

/etc/modprobe.d/qemu_kvm.conf

Inside it paste the following :

options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Honestly , there is a line among these that helped stopping the DMAR error I had with my X99 . Can you test it ? You need to reboot the host after creating the file .

Last edited by Denso (2014-11-23 20:49:24)

Offline

#3389 2014-11-23 21:55:29

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Seems like if I'm running the qemu with the -vga qxl then shows the nvidia graphic card well installed... Don't know for sure if that means that is working fine or not and I do not know if I can force to use the Nvidia one for gaming... Maybe I'm just crazy and I need to sleep

You can passthrough it as a secondary GPU for your VM . I tried it , and it switches to the nvidia GPU as soon as Windows enters to desktop . I don't think its related to Code 43 issue .

Any special errors in dmesg when you run the VM ? Maybe ROM errors would appear when you launch the VM . Have you tried passing a ROM file along the QEMU command line ?

EDIT :

Didn't notice the DMAR error you posted earlier .

I used to recieve a similar error , and if I remember correctly , setting some modules parameters helped :

try to create a file as :

/etc/modprobe.d/qemu_kvm.conf

Inside it paste the following :

options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Honestly , there is a line among these that helped stopping the DMAR error I had with my X99 . Can you test it ? You need to reboot the host after creating the file .

Hello all smile

I tried this and this is the result:

sudo qemu-system-x86_64 -name main -enable-kvm -localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -device vfio-pci,host=01:00.0,multifunction=on -device vfio-pci,host=01:00.1 -cdrom /home/jgomez/Downloads/virtio.iso -vga none qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: Error: Failed to setup INTx fd: Device or resource busy qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

BTW, my kernel (from the first post) is:

[jgomez@myArch ~]$ uname -r 3.17.2-1-mainline

I guess one of the options conflicts but I will check them tomorrow smile

I would like to give to all of you a big thanks before going to sleep... all day helping me, really thanks smile

Regards,


TheArcher

Last edited by thearcherblog (2014-11-23 21:59:48)

Offline

#3390 2014-11-24 05:10:19

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

I guess one of the options conflicts but I will check them tomorrow

It is this option :

options vfio_pci nointxmask=1

Comment it out and reboot the host . See if that resolves these errors .

Offline

#3391 2014-11-24 06:38:09

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
thearcherblog wrote:

I guess one of the options conflicts but I will check them tomorrow

It is this option :

options vfio_pci nointxmask=1

Comment it out and reboot the host . See if that resolves these errors .

Thanks a lot!

I willl test it as soon as I arrive home (near 18h (are 7:37 here now) and I will let you know smile

Thanks a lot again!!

Regards,


TheArcher

Offline

#3392 2014-11-24 07:04:25

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OMG... Today will be a party day!!!!! It's working!!!

All of you are the masters of knowledge and the kings of wisdom!!!!

There are not enough words to express my happiness right now!

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!

Amazing!!!!!! Thanks a lot!!!!!

Last edited by thearcherblog (2014-11-24 07:05:49)

Offline

#3393 2014-11-24 07:10:40

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!


https://www.archlinux.org/donate/


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#3394 2014-11-24 07:43:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've noticed that my disk subsystem is bottlenecking my game.
I'm using a virtio-blk-pci device with 60GB raw image cache=none(i thought that windows would cache disk adequately, foolish me) on my 750GB 5-years-old-with-few-badsectors-louder-than-a-CNC-mill WD caviar black hdd.

So i've created a new image sized 10GB in some directory mounted on the system SSD disk, sized 10GB. Attached it as an another virtio-blk-pci device to my windows VM and...
The disk image had 10GB of zeroes inside, so windows suggested initializing it in diskmgmt.msc, allowing me to choose between MBR and GPT(windows 7!). That again proves that i am right saying you do not need windows 8 to run on pure EFI.

So i've moved my steam library to  the new disk image, and i want to show you a neat little trick: windows has symlinks. Kind of. You can attach a drive as an empty NTFS folder, which is useful for some applications that load their data from a fixed location(like user's documents folder), but the folder must be empty.

Now the disk IO lags are gone, yay for me.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3395 2014-11-24 07:43:15

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:
thearcherblog wrote:

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!


https://www.archlinux.org/donate/

Done smile

Btw, I made a mistake on the donation... can I chat with you privately?

Offline

#3396 2014-11-24 07:48:56

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
jasonwryan wrote:
thearcherblog wrote:

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!


https://www.archlinux.org/donate/

Done smile

Btw, I made a mistake on the donation... can I chat with you privately?


Sure. Or you can just email Aaron: he looks after the donations.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#3397 2014-11-24 07:51:57

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot smile

Can you give me his mail address in a private message?

Thanks!

Offline

#3398 2014-11-24 07:53:54

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#3399 2014-11-24 07:55:39

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Perfect and thanks a lot!!!

Offline

#3400 2014-11-24 07:57:38

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

OMG... Today will be a party day!!!!! It's working!!!

All of you are the masters of knowledge and the kings of wisdom!!!!

There are not enough words to express my happiness right now!

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!

Amazing!!!!!! Thanks a lot!!!!!

So setting those options seems necessary on new Intel platforms (X99-Z97) .

Happy to see that you got it working smile

Offline

#3401 2014-11-24 08:01:26

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
thearcherblog wrote:

OMG... Today will be a party day!!!!! It's working!!!

All of you are the masters of knowledge and the kings of wisdom!!!!

There are not enough words to express my happiness right now!

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!

Amazing!!!!!! Thanks a lot!!!!!

So setting those options seems necessary on new Intel platforms (X99-Z97) .

Happy to see that you got it working smile

I'm not expert so I  cannot be sure if it's necessary on these chipsets only but now the solution is on this forum for eternity smile

Really I'm the most happy guy in the world right now and all thanks to all of you. I made my donation really happy! Please... never change! Best community ever!

Offline

#3402 2014-11-24 09:44:27

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I've noticed that my disk subsystem is bottlenecking my game.
I'm using a virtio-blk-pci device with 60GB raw image cache=none(i thought that windows would cache disk adequately, foolish me) on my 750GB 5-years-old-with-few-badsectors-louder-than-a-CNC-mill WD caviar black hdd.

So i've created a new image sized 10GB in some directory mounted on the system SSD disk, sized 10GB. Attached it as an another virtio-blk-pci device to my windows VM and...
The disk image had 10GB of zeroes inside, so windows suggested initializing it in diskmgmt.msc, allowing me to choose between MBR and GPT(windows 7!). That again proves that i am right saying you do not need windows 8 to run on pure EFI.

So i've moved my steam library to  the new disk image, and i want to show you a neat little trick: windows has symlinks. Kind of. You can attach a drive as an empty NTFS folder, which is useful for some applications that load their data from a fixed location(like user's documents folder), but the folder must be empty.

Now the disk IO lags are gone, yay for me.

Put stuff on LVM instead of file based images.

Screenshot. I've got LVM on top of dm-crypt on top of mdadm's RAID1 (2x 1TB Barracuda ES.2). All of the disks (windows and steam are separate) are:

  • raw images

  • cache set to none

  • aio set to native

  • ioeventfd set to on

  • x-data-plane set to on

  • steam drive has contiguous allocation

Last edited by dwe11er (2014-11-24 09:59:01)

Offline

#3403 2014-11-24 10:40:59

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
Duelist wrote:

I've noticed that my disk subsystem is bottlenecking my game.
I'm using a virtio-blk-pci device with 60GB raw image cache=none(i thought that windows would cache disk adequately, foolish me) on my 750GB 5-years-old-with-few-badsectors-louder-than-a-CNC-mill WD caviar black hdd.

So i've created a new image sized 10GB in some directory mounted on the system SSD disk, sized 10GB. Attached it as an another virtio-blk-pci device to my windows VM and...
The disk image had 10GB of zeroes inside, so windows suggested initializing it in diskmgmt.msc, allowing me to choose between MBR and GPT(windows 7!). That again proves that i am right saying you do not need windows 8 to run on pure EFI.

So i've moved my steam library to  the new disk image, and i want to show you a neat little trick: windows has symlinks. Kind of. You can attach a drive as an empty NTFS folder, which is useful for some applications that load their data from a fixed location(like user's documents folder), but the folder must be empty.

Now the disk IO lags are gone, yay for me.

Put stuff on LVM instead of file based images.

Screenshot. I've got LVM on top of dm-crypt on top of mdadm's RAID1 (2x 1TB Barracuda ES.2). All of the disks (windows and steam are separate) are:

  • raw images

  • cache set to none

  • aio set to native

  • ioeventfd set to on

  • x-data-plane set to on

  • steam drive has contiguous allocation


I am also using LVM and can confirm that it works really good. However, can you post your qemu command-line please? I'd like to try some the optimization you are using, and I also would like to see how you "call" your LVM parition (ie: i'm only doing this: "-drive file=/dev/guest/win7,id=data,format=raw -device scsi-hd,drive=data").

Offline

#3404 2014-11-24 10:59:58

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
dwe11er wrote:
Duelist wrote:

I've noticed that my disk subsystem is bottlenecking my game.
I'm using a virtio-blk-pci device with 60GB raw image cache=none(i thought that windows would cache disk adequately, foolish me) on my 750GB 5-years-old-with-few-badsectors-louder-than-a-CNC-mill WD caviar black hdd.

So i've created a new image sized 10GB in some directory mounted on the system SSD disk, sized 10GB. Attached it as an another virtio-blk-pci device to my windows VM and...
The disk image had 10GB of zeroes inside, so windows suggested initializing it in diskmgmt.msc, allowing me to choose between MBR and GPT(windows 7!). That again proves that i am right saying you do not need windows 8 to run on pure EFI.

So i've moved my steam library to  the new disk image, and i want to show you a neat little trick: windows has symlinks. Kind of. You can attach a drive as an empty NTFS folder, which is useful for some applications that load their data from a fixed location(like user's documents folder), but the folder must be empty.

Now the disk IO lags are gone, yay for me.

Put stuff on LVM instead of file based images.

Screenshot. I've got LVM on top of dm-crypt on top of mdadm's RAID1 (2x 1TB Barracuda ES.2). All of the disks (windows and steam are separate) are:

  • raw images

  • cache set to none

  • aio set to native

  • ioeventfd set to on

  • x-data-plane set to on

  • steam drive has contiguous allocation


I am also using LVM and can confirm that it works really good. However, can you post your qemu command-line please? I'd like to try some the optimization you are using, and I also would like to see how you "call" your LVM parition (ie: i'm only doing this: "-drive file=/dev/guest/win7,id=data,format=raw -device scsi-hd,drive=data").

I'm using libvirt to start my VMs. Anyway, here is libvirt's command-line (I've truncated not relevant stuff):

/usr/sbin/qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host,hv_time,-kvm_pv_eoi,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 8192 -realtime mlock=off -smp 6,sockets=1,cores=3,threads=2 -nographic -no-user-config -nodefaults -rtc base=localtime,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown ... -drive file=/dev/storage/windows-ovmf,if=none,id=drive-virtio-disk0,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/storage/steam,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 ... -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7 -device vfio-pci,host=01:00.1,id=hostdev2,bus=pci.0,addr=0x7.0x1 -device usb-host,hostbus=2,hostaddr=5,id=hostdev3 ... -set device.virtio-disk0.scsi=off -set device.virtio-disk0.x-data-plane=on -set device.virtio-disk1.scsi=off -set device.virtio-disk1.x-data-plane=on -msg timestamp=on

Offline

#3405 2014-11-24 11:51:07

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks dwe11er!

If I remember correctly OP (or someone else) compiled OVMF from git and gave a link to binaries. Anyone could share this link please?

Last edited by Nesousx (2014-11-24 11:51:48)

Offline

#3406 2014-11-24 12:05:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Thanks dwe11er!

If I remember correctly OP (or someone else) compiled OVMF from git and gave a link to binaries. Anyone could share this link please?


OVMF-26830e8.tar.gz

Offline

#3407 2014-11-24 14:52:46

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks nbhs. @dwe11er, after a quick test here, I have way better disk perf if I do not specify "aio=native,cache=none". I will do more tests if I have time, and report here. I just wanted to let you know, that you may be able to get better perf too. Unless those values you get are very close from your bare metal config.

Offline

#3408 2014-11-24 15:00:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
Denso wrote:
thearcherblog wrote:

OMG... Today will be a party day!!!!! It's working!!!

All of you are the masters of knowledge and the kings of wisdom!!!!

There are not enough words to express my happiness right now!

If exists a donation place I will do it right now and if not, please, let me know if any of you is in Norway and I will invite him/her to a beer!!!!

Amazing!!!!!! Thanks a lot!!!!!

So setting those options seems necessary on new Intel platforms (X99-Z97) .

Happy to see that you got it working smile

I'm not expert so I  cannot be sure if it's necessary on these chipsets only but now the solution is on this forum for eternity smile

Well, I'd prefer to have a bit more closure than instructions to apply a long list of arbitrary module options.  Any chance you could whittle down the list of options and figure out what fixed it?

Denso wrote:
options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Most of these seem completely unnecessary to me, especially the vfio options.  APICv is already enabled by default when available, so that's a no-op.  nested=1, why?  You're not doing nesting.  ignore_msrs=1 is the only thing I have set on my box.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3409 2014-11-24 15:11:08

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Thanks nbhs. @dwe11er, after a quick test here, I have way better disk perf if I do not specify "aio=native,cache=none". I will do more tests if I have time, and report here. I just wanted to let you know, that you may be able to get better perf too. Unless those values you get are very close from your bare metal config.

Might be. My main testing method was 'no sound stuttering during heavy I/O' and it's ok with these settings smile

Offline

#3410 2014-11-24 16:04:36

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I just want to share my sucess story wink.

I'm running sucessfully Arch Linux with Nvidia driver with Qemu machine with Windows 7. My CPU's i5-3550, motherboard's AsRock Z77 Extreme4, host GPU's GeForce 9800 GTX+, guest GPU's Radeon 4850. Working just great, using obviously mainline kernel from OP PKG.

Some write-up with my thoughts if anyone's interested: http://tech.draiser.net/2014/11/23/kvmq … g-machine/

Offline

#3411 2014-11-24 17:27:50

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:
Denso wrote:

So setting those options seems necessary on new Intel platforms (X99-Z97) .

Happy to see that you got it working smile

I'm not expert so I  cannot be sure if it's necessary on these chipsets only but now the solution is on this forum for eternity smile

Well, I'd prefer to have a bit more closure than instructions to apply a long list of arbitrary module options.  Any chance you could whittle down the list of options and figure out what fixed it?

Denso wrote:
options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Most of these seem completely unnecessary to me, especially the vfio options.  APICv is already enabled by default when available, so that's a no-op.  nested=1, why?  You're not doing nesting.  ignore_msrs=1 is the only thing I have set on my box.

Ok, I will test one by one to check what fixed it smile

But give me 24 hours or so smile

And thanks a lot!!!

Offline

#3412 2014-11-24 18:31:11

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Most of these seem completely unnecessary to me, especially the vfio options.  APICv is already enabled by default when available, so that's a no-op.  nested=1, why?  You're not doing nesting.  ignore_msrs=1 is the only thing I have set on my box.

I know . There was a time when I completely commented out every line of those options and the VMs booted just fine . But they weren't stable after hours of operation . Plus , with them on , both of my VMs can boot fine at the same time (2 seperate GPUs , one to each VM) .

THe masking option is needed in my case , otherwise the whole VM goes into SloMo mode . smile

I'll comment out the APICv + nested + shadow_vmcs + ept from now on though . I already commented out the disable_hugepages one for a month or so , no problem .

I didn't need any of these options when I was using Z77 .. X99 is more troublesome to setup . ALOT . And there is no way I can reboot any VM with GPU assigned to it without crashing the host sad It is the ONLY remaining issue I have now .

They reboot fine , until I install Nvidia's drivers and then they crash the host whenever I reboot them .

Last edited by Denso (2014-11-24 18:31:54)

Offline

#3413 2014-11-24 18:48:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

Most of these seem completely unnecessary to me, especially the vfio options.  APICv is already enabled by default when available, so that's a no-op.  nested=1, why?  You're not doing nesting.  ignore_msrs=1 is the only thing I have set on my box.

I know . There was a time when I completely commented out every line of those options and the VMs booted just fine . But they weren't stable after hours of operation . Plus , with them on , both of my VMs can boot fine at the same time (2 seperate GPUs , one to each VM) .

I do this too, one AMD + one Nvidia.  AFAIK, I only need ignore_msrs so that passmark performance test works

THe masking option is needed in my case , otherwise the whole VM goes into SloMo mode . smile

That's specific to a device you're assigning, not likely related to the chipset.  The option overrides our detection of whether the device supports INTxDisable masking and instead masks the interrupt at the APIC instead of the device.  The huge downside of that is that it requires the device has an exclusive interrupt because of the difference in where we mask the interrupt.  Also note:

$ modinfo vfio-pci
...
parm:           nointxmask:Disable support for PCI 2.3 style INTx masking.  If this resolves problems for specific devices, report lspci -vvvxxx to linux-pci@vger.kernel.org so the device can be fixed automatically via the broken_intx_masking flag. (bool)

Have you done that?  This option would apply automatically to just that device if you did...

I'll comment out the APICv + nested + shadow_vmcs + ept from now on though . I already commented out the disable_hugepages one for a month or so , no problem .

I didn't need any of these options when I was using Z77 .. X99 is more troublesome to setup . ALOT . And there is no way I can reboot any VM with GPU assigned to it without crashing the host sad It is the ONLY remaining issue I have now .

They reboot fine , until I install Nvidia's drivers and then they crash the host whenever I reboot them .

Aside from the two Radeon reset issues (R7790 and similar can only be used once per boot and HD8570 and similar mistakenly report NoSoftRst-) I don't know of any other GPU reset issues, nor do I see why X99 would be more or less problematic than Z77 in that respect.

EDIT: Nvidia cards have pretty terrible DisableINTx support on the audio device (esp. Quadro and even fake Quadro), but the better option there is typically to make the guest driver use MSI, not to run with nointxmask

Last edited by aw (2014-11-24 18:54:49)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3414 2014-11-24 19:56:02

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you done that?  This option would apply automatically to just that device if you did...

Reported just now wink

Will try to comment out nointxmask=1 and see if reboot issues are resolved .

nor do I see why X99 would be more or less problematic than Z77 in that respect.

I don't think its X99's fault , but rather ASUS's .

EDIT :

Confirmed . Commenting out nointxmask=1 solves the reboot issue in one VM . The second one , reboots fine and Windows loads fine until it reaches the login screen , then it goes into black screen . The good news is that the host doesn't crash anymore smile

Now . I just have to wait for my USB PCI-E card to be fixed so I wouldn't need nointxmask=1 .

EDIT 2 :

Enabling MSI in the second 8.1 VM fixed its reboots issue completely ! Thanks Alex ! One more question , I'm about to reinstall Windows 8.1 on this particular VM . can I export the "MessageSignaledInterruptProperties" key to a .reg file and just apply it to the new installed OS ? Or should I just do it manually again ? EDIT : Yes , it works when exported to a .reg file .

Last edited by Denso (2014-11-24 20:48:28)

Offline

#3415 2014-11-24 21:34:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Have you done that?  This option would apply automatically to just that device if you did...

Reported just now wink

Will try to comment out nointxmask=1 and see if reboot issues are resolved .

nor do I see why X99 would be more or less problematic than Z77 in that respect.

I don't think its X99's fault , but rather ASUS's .

EDIT :

Confirmed . Commenting out nointxmask=1 solves the reboot issue in one VM . The second one , reboots fine and Windows loads fine until it reaches the login screen , then it goes into black screen . The good news is that the host doesn't crash anymore smile

Now . I just have to wait for my USB PCI-E card to be fixed so I wouldn't need nointxmask=1 .

You can add a patch like this one to your kernel to try it:

http://git.kernel.org/cgit/linux/kernel … 3cb30b73ad

Copy the two lines of code added and replace with your vendor & device ID.

EDIT 2 :

Enabling MSI in the second 8.1 VM fixed its reboots issue completely ! Thanks Alex ! One more question , I'm about to reinstall Windows 8.1 on this particular VM . can I export the "MessageSignaledInterruptProperties" key to a .reg file and just apply it to the new installed OS ? Or should I just do it manually again ? EDIT : Yes , it works when exported to a .reg file .

Great!  And neat trick with the exported reg file, please add it as a comment on the blog.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3416 2014-11-25 06:41:04

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can add a patch like this one to your kernel to try it:

http://git.kernel.org/cgit/linux/kernel … 3cb30b73ad

Copy the two lines of code added and replace with your vendor & device ID.

Would do that when I compile the next time for sure .

Great!  And neat trick with the exported reg file, please add it as a comment on the blog.

Will do so in a minute smile

EDIT :

errrr ... I can't publish my comment to your blog because I don't have any account of those that are required to post . Can you publish it ? Thank you .

Hi .

You can export the "MessageSignaledInterruptProperties" key to a ".reg" file and use it to enable MSI easily when you reinstall Windows .

Navigate to "MessageSignaledInterruptProperties" key and right click on it > Choose "Export" , give it a name and save it anywhere you like . You will end up with a ".reg" file . you can double click on it to merge it into the registry when you do a reinstall of Windows .

You can also combine multiple ".reg" files into one big unified one to make things even easier .

http://i.imgur.com/qVY1J2e.png


EDIT 2 :

Why can't I enable MSI for Nvidia's HDMI audio device ?

Figured it out . The device path shown in device properties doesn't point to the correct path . I enabled MSI for ALL devices that have Vendor's ID of 10de (Nvidia's ID) , and now it works beautifully without "nointxmask=1" . (Yes , the culprit was Nvidia's audio device and NOT the USB card as I initially assumed) .

EDIT 3 :

Here is the ".reg" file for fixing Nvidia's GT610 audio and reboot issues (I exported each key to a seperate file then combined them into a unified file using a quick copy/paste action in notepad :

http://www.pastebin.ca/2877860

If you have GT610 and want to try it , just copy the text to a file , save it with ".reg" extention and just double click on it to merge it to your registry .

Hope this helps smile

Last edited by Denso (2014-11-25 08:16:19)

Offline

#3417 2014-11-25 09:41:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

If you have GT610 and want to try it , just copy the text to a file , save it with ".reg" extention and just double click on it to merge it to your registry .

Hope this helps smile

Huh, you're passing a GT610?
I have a GT610 too, but i'm using it for host system output. Is there any testing i could help you with? My GT610 is silent 1GB version from gigabyte.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3418 2014-11-25 12:21:16

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just to inform:

Having the cpupower ondemand or on powersave give me from 12 fps to 75 running Valley Benchmark so... be careful smile

Between ondemand and performance I didn't found any relevant difference.

Regards,


TheArcher

Last edited by thearcherblog (2014-11-25 12:33:56)

Offline

#3419 2014-11-25 14:02:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Just to inform:

Having the cpupower ondemand or on powersave give me from 12 fps to 75 running Valley Benchmark so... be careful smile

Between ondemand and performance I didn't found any relevant difference.

The powersave governor statically runs the processor at the lowest speed regardless of load, so yes, it will have a dramatic effect on performance.  The performance governor runs the processor at the highest speed regardless of load.  The ondemand governor adjusts the processor speed based on load, which is where the guest can experience response latency.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3420 2014-11-25 14:15:20

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, all is working now except sound and reboot.

I hear sound, but I get scraching/stutter. I use alsa and tried it with the params the OP mentioned it, but without "QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa" it is the same result.

Reboot:
I found this method: http://blog.ktz.me/?p=219
All is working, when I start the logon script, the screen turns black and comes back. When I reboot with the logoff script, I get Invalid ROM contents and starting the VM does not work.

Any Ideas?

Offline

#3421 2014-11-25 15:13:22

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
thearcherblog wrote:

Just to inform:

Having the cpupower ondemand or on powersave give me from 12 fps to 75 running Valley Benchmark so... be careful smile

Between ondemand and performance I didn't found any relevant difference.

The powersave governor statically runs the processor at the lowest speed regardless of load, so yes, it will have a dramatic effect on performance.  The performance governor runs the processor at the highest speed regardless of load.  The ondemand governor adjusts the processor speed based on load, which is where the guest can experience response latency.

However, this isn't true for intel_pstate (now by default for Sandy Bridge and upwards) which supports only powersave and performance governors. They behave more like ondemand, with some changes in power consumption characteristics in the long run.

Last edited by dwe11er (2014-11-25 15:41:14)

Offline

#3422 2014-11-25 16:19:56

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
aw wrote:
thearcherblog wrote:

Just to inform:

Having the cpupower ondemand or on powersave give me from 12 fps to 75 running Valley Benchmark so... be careful smile

Between ondemand and performance I didn't found any relevant difference.

The powersave governor statically runs the processor at the lowest speed regardless of load, so yes, it will have a dramatic effect on performance.  The performance governor runs the processor at the highest speed regardless of load.  The ondemand governor adjusts the processor speed based on load, which is where the guest can experience response latency.

However, this isn't true for intel_pstate (now by default for Sandy Bridge and upwards) which supports only powersave and performance governors. They behave more like ondemand, with some changes in power consumption characteristics in the long run.

It's truth but I needed to disable the intel_pstate because my processor was running all time at 4GHz...

Offline

#3423 2014-11-25 17:04:41

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a GT610 too, but i'm using it for host system output. Is there any testing i could help you with? My GT610 is silent 1GB version from gigabyte.

This card sucks . Or maybe my X99-Deluxe sucks . This card used to work with zero tweaking on my AMD 990FXA board . Now everytime I think I finally got rid of the reboot issue , a couple reboots later and BAAAM it hangs the VM again . Sigh ... I'm really tired . It works pretty darn well until I install the Nvidia drivers . Then it is the same all over again . I'm seriously thinking of running the VM in full screen on top of a Linux DE . Will QXL give me 60fps when watching movies and surfing ? Because if it would , I will happily go that route .

umm ... would it be possible to manually install JUST the graphics/audio driver in Windows without running the Nvidia installer ? Would worth a try !

Offline

#3424 2014-11-25 18:29:07

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I have a GT610 too, but i'm using it for host system output. Is there any testing i could help you with? My GT610 is silent 1GB version from gigabyte.

This card sucks . Or maybe my X99-Deluxe sucks . This card used to work with zero tweaking on my AMD 990FXA board . Now everytime I think I finally got rid of the reboot issue , a couple reboots later and BAAAM it hangs the VM again . Sigh ... I'm really tired . It works pretty darn well until I install the Nvidia drivers . Then it is the same all over again . I'm seriously thinking of running the VM in full screen on top of a Linux DE . Will QXL give me 60fps when watching movies and surfing ? Because if it would , I will happily go that route .

umm ... would it be possible to manually install JUST the graphics/audio driver in Windows without running the Nvidia installer ? Would worth a try !

uhm, running vm for surfing/video playback?

i always thought both of these areas are far more superior/comfortable while being in linux, but i just might have missed something ))

Offline

#3425 2014-11-25 18:30:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

This card sucks .

Well, you know, it's a PLUG to fill the pci-e port.


BTW, does anyone know what those two or three recent seavgabios git updates brought us? Seems like i've got something borked:
Unigine Heaven benchmark runs fine in CF mode, GPU-Z says CF is enabled and even renders, but when i start my -nodx9ex switched game which is using DX9 - it crashes with something related to access violation and then even the Unigine Heaven benchmark breaks. Yesterday it was working just fine. And there is no messages in the host dmesg or something.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3426 2014-11-25 19:24:39

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am trying to use Virtio as a standard for network. I followed the libvirt wiki page (http://wiki.libvirt.org/page/Virtio), but I couldn't make it work.

this is how my network configuration looks right now:

<interface type='direct'>
      <mac address='52:54:00:f3:27:c5'/>
      <source dev='eno1' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>

can someone please help me use Virtio for networking? Thanks a lot!

Last edited by 4kGamer (2014-11-25 19:42:23)

Offline

#3427 2014-11-25 20:24:58

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am having a reproducable race-condition resulting in multiple kernel panics/oopses under the following condition:

- Use 3.17.2 kernel from OP (need it due to i915 arbiter mess)
- use systemd service to assign VGA card to vfio:

# /etc/systemd/system/vfio.service [Unit] Description=Binds devices to vfio-pci After=syslog.target [Service] EnvironmentFile=-/etc/vfio-pci.cfg Type=oneshot RemainAfterExit=yes ExecStart=-/usr/bin/vfio-bind $DEVICES [Install] WantedBy=multi-user.target

Contents of /etc/vfio-pci.cfg:

DEVICES="0000:01:00.0 0000:01:00.1"

vfio-bind script:

#!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

Kernel freaks out when trying to start kdm service (probably the underlying X is causing the problem) in parallel with the vfio service. If i disable them and start them manually one after the other everything is fine, no matter what sequence.

Now I know there are ways to workaround this, and it is not really a big deal for me.
My actual question is rather if this indicates a real/serious bug in the kernel or if this is somehow expected behaviour with the current setup which just should be accepted/ignored? Should I try to track this down in more detail and open a bug somewhere?

Offline

#3428 2014-11-25 23:01:44

dakabali
Member
Registered: 2014-11-11
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I've made the next step in improving my configuration. After setting up a working PCI passthrough I connected a USB sound card (Creative OMNI Surround 5.1) to the virtual Windows 8.1 machine. It works, however the quality of the sound is not satisfying. It is noisy and seems to depend on the CPU because as the games start the quality decreases. Before going into the details in this forum: can it have something to do with the PCI and/or USB hardware delegations at all? Unfortunately I've got no other PC where I could try a non-virtualized setup. Any experience? Thx.

EDIT:  long story short --> the USB passthrough seems to be very noisy which is ok for mouse and keyboard connections but disturbing for HDA. The solution was to send back the OMNI Surround 5.1 and purchase a PCI Creative Sound Blaster Zx. Having used pci-stub I managed to hide the sound card in Arch and delegate it to Win8.1 in KVM. Without stub the front speakers won't work as Linux reserves the stereo channels by default. Sorry for this off-topic because - as it came out at the end - this has nothing to do with VGA PCI passthrough at all.

Last edited by dakabali (2014-12-10 17:55:33)

Offline

#3429 2014-11-26 02:31:07

snarfies
Member
Registered: 2014-11-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How does one install the linux-mainline-3.17.2.tar.gz on the front page?

I tried pacman -U, but it gave me:

error: missing package metadata in /home/erik/linux-mainline-3.17.2.tar.gz
error: '/home/erik/linux-mainline-3.17.2.tar.gz': invalid or corrupted package

Offline

#3430 2014-11-26 07:55:42

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

snarfies wrote:

How does one install the linux-mainline-3.17.2.tar.gz on the front page?

I tried pacman -U, but it gave me:

error: missing package metadata in /home/erik/linux-mainline-3.17.2.tar.gz
error: '/home/erik/linux-mainline-3.17.2.tar.gz': invalid or corrupted package

It's source package, not an actual one. You need to build it with makepkg.

4kGamer wrote:

Hi,

I am trying to use Virtio as a standard for network. I followed the libvirt wiki page (http://wiki.libvirt.org/page/Virtio), but I couldn't make it work.

this is how my network configuration looks right now:

    <interface type='direct'>
          <mac address='52:54:00:f3:27:c5'/>
          <source dev='eno1' mode='bridge'/>
          <model type='rtl8139'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
        </interface>

can someone please help me use Virtio for networking? Thanks a lot!

Change rtl8139 to virtio. Done.

Offline

#3431 2014-11-26 08:10:39

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah I tried that. I got no connection then ...

Offline

#3432 2014-11-26 09:26:26

Kironide
Member
Registered: 2014-11-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I know that VGA passthrough using Xen has some issues when AMD graphics cards are used, namely, the guest OS cannot be rebooted.

My curiosity lies in the question of whether the same issue is present when using QEMU for VGA passthrough. I would imagine not, but I felt it best to make sure.

Best regards,

Kironide

Offline

#3433 2014-11-26 09:48:03

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

go here: http://blog.ktz.me/?p=219

reboots do work under Xen

Offline

#3434 2014-11-26 10:19:00

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

I am having a reproducable race-condition resulting in multiple kernel panics/oopses under the following condition:

Contents of /etc/vfio-pci.cfg:

DEVICES="0000:01:00.0 0000:01:00.1"

This looks funny, are you sure these are correct device IDs for your graphics card and its audio when you run lspci -l on host ? Are they bound to pci-stub on system startup ?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3435 2014-11-26 10:28:58

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mine looks exactly the same

Offline

#3436 2014-11-26 10:39:54

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
4kGamer wrote:

Hi,

I am trying to use Virtio as a standard for network. I followed the libvirt wiki page (http://wiki.libvirt.org/page/Virtio), but I couldn't make it work.

this is how my network configuration looks right now:

    <interface type='direct'>
          <mac address='52:54:00:f3:27:c5'/>
          <source dev='eno1' mode='bridge'/>
          <model type='rtl8139'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
        </interface>

can someone please help me use Virtio for networking? Thanks a lot!

Change rtl8139 to virtio. Done.

Just to be more precise: Windows 8.1 DOES start, but I got no internet connection then.

Last edited by 4kGamer (2014-11-26 10:45:09)

Offline

#3437 2014-11-26 10:55:34

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Hi all,

I have a working setup based on Intel CPU and Z77 chipset + GTX 970 as graphic cards. However I have one question, and one issue.

First, here is my question: whats the point using a bios dump (from graphic card) in the qemu command-line? I am not doing it yet and everything seems ok. Good performance, VM reboots ok, etc. FIXED

Then, my little issue. I have an USB sound card : Asus Xonar U7. If I use it on a USB hub (shared with my mouse + keyboard), then I have some sound issues. It stutters sometimes, and causes lag. Only happens during games; never while watching a movie / youtube / listening to music.
I decided to go with emulated audio (cf. command line below). Now i non longer have those sound / lags issue. However I have a little latency in sound. All the time. The sound occurs half a second (or even less) after the real action. I tried with various PA samples from 128 to 16384 (cf. command line). 128 to 512 is unusable, audio lag and video lag. And high values (above 4096) make the latency higher. So far, it seems that the best value is 1024.

Do you have any idea what to do?

Let me know if you need additional information.
Thanks in advance.

Edit ; HDMI audio is not a solution since my screen as no audio jack output (and on screen's speaker are just horrible).

Command line:

QEMU_PA_SAMPLES=1024 QEMU_AUDIO_DRV=pa \ qemu-system-x86_64 -name game -M q35 -enable-kvm \ -mem-path /dev/hugepages \ -balloon virtio \ -nographic \ -m 16384 \ -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -realtime mlock=on -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/guest/Windaube_7,id=sys,format=raw -device scsi-hd,drive=sys \ -drive file=/dev/guest/win7,id=data,format=raw -device scsi-hd,drive=data \ -device vfio-pci,host=00:1d.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi2 \ -net nic,model=virtio,macaddr=52:54:00:7c:3c:42 -net bridge,br=virbr0 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Sorry to bump, but if anyone has any idea on the sound part, I'll be very happy.

Last edited by Nesousx (2014-11-26 10:55:53)

Offline

#3438 2014-11-26 11:57:19

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

First I'd like thank you for this great post and the valuable information posted here.

I'm trying to get vga passthrough work on ubuntu 14.10, I know here is a Arch Linux forum sorry if I'm being rude or something.

My setup is the follow:

Core i7 4770S
Gigabyte GA-B85M-D3PH
Nvidia Geforce GT630 1GB
AMD R7 250 2GB

I've tried to passthrough both gpus, but I can't get video signal from DVI output connected to VGAs.

I think the problem is related to missing i915 arbiter patch, I haven't found the 915 patch for 3.16 kernel which is the ubuntu 14.10 default kernel.

Does someone has the i915 arbiter patch for 3.16 kernel?

Thank you

Last edited by bpbastos (2014-11-26 11:58:44)

Offline

#3439 2014-11-26 11:58:15

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Just to be more precise: Windows 8.1 DOES start, but I got no internet connection then.

you need to install virtio network drivers form virtio drivers iso.

Offline

#3440 2014-11-26 12:01:09

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Do you have any idea what to do?



Sorry to bump, but if anyone has any idea on the sound part, I'll be very happy.

How do u pass usb devices? i have had same problem with -usb device, but passing whole INTEL usb controller with vifio worked flawlessly.

Last edited by slis (2014-11-26 12:08:24)

Offline

#3441 2014-11-26 12:05:51

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:

Hi,

First I'd like thank you for this great post and the valuable information posted here.

I'm trying to get vga passthrough work on ubuntu 14.10, I know here is a Arch Linux forum sorry if I'm being rude or something.

My setup is the follow:

Core i7 4770S
Gigabyte GA-B85M-D3PH
Nvidia Geforce GT630 1GB
AMD R7 250 2GB

I've tried to passthrough both gpus, but I can't get video signal from DVI output connected to VGAs.

I think the problem is related to missing i915 arbiter patch, I haven't found the 915 patch for 3.16 kernel which is the ubuntu 14.10 default kernel.

Does someone has the i915 arbiter patch for 3.16 kernel?

Thank you

i used this patch for 3.15
http://paste.ubuntu.com/9248971/

if something fails apply manually.

Offline

#3442 2014-11-26 12:12:30

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
4kGamer wrote:

Just to be more precise: Windows 8.1 DOES start, but I got no internet connection then.

you need to install virtio network drivers form virtio drivers iso.

thank you so much, of course!

I got it working now smile

Offline

#3443 2014-11-26 12:36:21

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
Nesousx wrote:

Do you have any idea what to do?



Sorry to bump, but if anyone has any idea on the sound part, I'll be very happy.

How do u pass usb devices? i have had same problem with -usb device, but passing whole INTEL usb controller with vifio worked flawlessly.

Unfortunately, I already pass the whole controller:

-device vfio-pci,host=00:1d.0,bus=pcie.0 \

This controller controls 2 front USB ports on my case. From here, I plug an USB hub which contains mouse + keyboard (both wireless Logitech via 1 unify adapter), a micro USB cable (for Xbox gamepad), and the sound card.

PS: I just ordered a new USB soundcard from 20€ from Amazon (free return if it doesn't suits my needs). I'll give it a try to make sure it is not some stupid hardware issue, related to my sound card.

Last edited by Nesousx (2014-11-26 12:40:07)

Offline

#3444 2014-11-26 12:39:40

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
slis wrote:
Nesousx wrote:

Do you have any idea what to do?



Sorry to bump, but if anyone has any idea on the sound part, I'll be very happy.

How do u pass usb devices? i have had same problem with -usb device, but passing whole INTEL usb controller with vifio worked flawlessly.

Unfortunately, I already pass the whole controller:

-device vfio-pci,host=00:1d.0,bus=pcie.0 \

is that intel controller? VIA caused problems for me..

Offline

#3445 2014-11-26 12:41:10

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
Nesousx wrote:
slis wrote:

How do u pass usb devices? i have had same problem with -usb device, but passing whole INTEL usb controller with vifio worked flawlessly.

Unfortunately, I already pass the whole controller:

-device vfio-pci,host=00:1d.0,bus=pcie.0 \

is that intel controller? VIA caused problems for me..

Yup:

lspci | grep 00:1d.0 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)

Offline

#3446 2014-11-26 12:45:47

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
bpbastos wrote:

Hi,

First I'd like thank you for this great post and the valuable information posted here.

I'm trying to get vga passthrough work on ubuntu 14.10, I know here is a Arch Linux forum sorry if I'm being rude or something.

My setup is the follow:

Core i7 4770S
Gigabyte GA-B85M-D3PH
Nvidia Geforce GT630 1GB
AMD R7 250 2GB

I've tried to passthrough both gpus, but I can't get video signal from DVI output connected to VGAs.

I think the problem is related to missing i915 arbiter patch, I haven't found the 915 patch for 3.16 kernel which is the ubuntu 14.10 default kernel.

Does someone has the i915 arbiter patch for 3.16 kernel?

Thank you

i used this patch for 3.15
http://paste.ubuntu.com/9248971/

if something fails apply manually.

Thank you slis!

Offline

#3447 2014-11-26 16:19:37

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello all,

After reading a lot looking why sometimes my sound is not working on the KVM (Windows 8.1, NVIDIA sound from the GTX 770 card), I think I found the solution (needs to be checked more time but seems to work for now (2 hours testing) smile

First of all... when I was running:

[jgomez@myArch ~]$ sudo lspci -v -s 01:00.0 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device 3774 Flags: bus master, fast devsel, latency 0, IRQ 49 Memory at ee000000 (32-bit, non-prefetchable) [size=16M] Memory at e0000000 (64-bit, prefetchable) [size=128M] Memory at e8000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at ef000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ -------------------> Before was MSI: Enable- Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900] #19 Kernel driver in use: vfio-pci Kernel modules: nouveau [jgomez@myArch ~]$ sudo lspci -v -s 01:00.1 //For the Sound part :) 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) Subsystem: eVga.com. Corp. Device 3774 Flags: bus master, fast devsel, latency 0, IRQ 50 Memory at ef080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ -------------------> Before was MSI: Enable- Capabilities: [78] Express Endpoint, MSI 00 Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

So, reading herehere and here that changed and seems to work for me smile

Reading all of that suggests that you try (making backup or whatever to cover your back smile):

add pci=msi on your kernel parameters --------> That changes my enabled- to enabled+
and go to your Windows guest, Control Panel, System, Device Manager
View menu, Resources by Type
Check your device (VGA and Sound in my case)
Check if the IRQ it's positive (MSI not enabled) or negative (MSI Enabled) ---------> In my case were positive so... let's go to the next step... remember to make backup, maybe your guest will not boot again!!!
Check the properties of that device, details tab, Device Instance Path and copy that value (something like PCI\VEN_10DE&DEV_1184....
Open your regedit
Go to: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Enum\PCI\ ------------- Your copied value here -----------------\DeviceParameters\Interrupt Management\

Maybe exists maybe not a key named MessageSignaledInterruptProperties (if does not exists, create it)
Create a new DWORD inside named MSISupported (value should be 1 for enabled and 0 for disabled)

Close regedit and reboot your computer

Cross your fingers and start praying smile

In my case was working and after reboot no more problems with sound (at the moment).

If you go again to the Device Manager you can check that now your IRQ for that device are negative.

If that does not happen, remove the keys on regedit and... bad luck... at least according to the post I read).

This should give us less latency so that means... better performance smile

I hope it helps for all of you!


Best regards,


TheArcher

P.S.= Thanks a lot to the authors of these blogs / forum because they are genius!

Offline

#3448 2014-11-26 19:12:57

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, could anyone tell me how exactly CPU assignment in KVM/QEMU works?
I have and i5 3550 CPU with 4 cores and no hyperthreading. I currently gave 2 cores for VM but, as I understand it, it doesn't give cores exclusively, so is it possible to pass all 4 cores and with  CPU load on host near to 0 should it perform like it has native 4 cores on guest? I hope my question is easy to understand. smile

Offline

#3449 2014-11-26 19:49:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Hi, could anyone tell me how exactly CPU assignment in KVM/QEMU works?
I have and i5 3550 CPU with 4 cores and no hyperthreading. I currently gave 2 cores for VM but, as I understand it, it doesn't give cores exclusively, so is it possible to pass all 4 cores and with  CPU load on host near to 0 should it perform like it has native 4 cores on guest? I hope my question is easy to understand. smile

vCPUs are threads and by default are handled the same as any other thread on the host, scheduling on available physical CPUs.  You can pin them to help out the scheduler and improve cache locality.  You also need to realize that there are additional threads that run in the host for I/O, so if you have a 1:1 mapping of vCPU to pCPUs and don't have extra resources for the additional threads, then that time will get stolen from the vCPUs.  It's therefore often advisable to not fully consume the available host CPUs.  cgroups can be used to attempt to further isolate where processes can run and what can run on the same pCPUs as your guest.  There's also always some overhead to virtualization, so you're never going to get 100% of native performance.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3450 2014-11-26 19:52:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Hi, could anyone tell me how exactly CPU assignment in KVM/QEMU works?
I have and i5 3550 CPU with 4 cores and no hyperthreading. I currently gave 2 cores for VM but, as I understand it, it doesn't give cores exclusively, so is it possible to pass all 4 cores and with  CPU load on host near to 0 should it perform like it has native 4 cores on guest? I hope my question is easy to understand. smile

I use cset to do this, so i have 2 cores dedicated to the host, and 6 for the vm

Offline

#3451 2014-11-26 20:01:34

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

vCPUs are threads and by default are handled the same as any other thread on the host, scheduling on available physical CPUs.  You can pin them to help out the scheduler and improve cache locality.  You also need to realize that there are additional threads that run in the host for I/O, so if you have a 1:1 mapping of vCPU to pCPUs and don't have extra resources for the additional threads, then that time will get stolen from the vCPUs.  It's therefore often advisable to not fully consume the available host CPUs.  cgroups can be used to attempt to further isolate where processes can run and what can run on the same pCPUs as your guest.  There's also always some overhead to virtualization, so you're never going to get 100% of native performance.

Thank you. I'm thinking now what would be best way for eg. Windows guest games which can effectively use more then 2 cores - to give them 2-3 cores dedicated for VM or give 4 not doing anything other on host. I know there's always some overhead, but in this combination I'd like that Windows and games  would work at their best since that games could perhaps work better when have many cores.

Last edited by dRaiser (2014-11-26 20:34:10)

Offline

#3452 2014-11-26 20:20:18

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just tried the command: /sbin/lsmod | grep virtio to check whether the virtio_net driver is being used, but it's empty? Network is working though ...

then I had a look into /var/log/libvirt/qemu/vm1.log and I found this:


2014-11-26 20:53:04.784+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name vm1 -S -machine pc-i440fx-2.2,accel=k$
Domain id=2 is tainted: host-cpu

I have to say I haven't been happy with CPU performance. Whenever I do anything - like just opening the file Explorer it CPU usage jumps to 40 %.

So basically there are two issues here:

1. It looks like I am NOT using the the virtio_net driver
2. Something weird is going on with my CPU



@aw or anyone who can help me, do you know why I get that log output?

Offline

#3453 2014-11-26 20:26:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

I just tried the command: /sbin/lsmod | grep virtio to check whether the virtio_net driver is being used, but it's empty? Network is working though ...

You ran this in the host?  The guest is the one that needs this driver, the host doesn't load it.

then I had a look into /var/log/libvirt/qemu/vm1.log and I found this:


2014-11-26 20:53:04.784+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name vm1 -S -machine pc-i440fx-2.2,accel=k$ Domain id=2 is tainted: host-cpu

I don't see what problem you're pointing out here.  libvirt taints configurations using -cpu host because it can't migrate them.  Do you care?

I have to say I haven't been happy with CPU performance. Whenever I do anything - like just opening the file Explorer it CPU usage jumps to 40 %.

So basically there are two issues here:

1. It looks like I am NOT using the the virtio_net driver
2. Something weird is going on with my CPU

@aw or anyone who can help me, do you know why I get that log output?

I don't see that you're actually reporting any problems.  What should the CPU usage go to when you open the file explorer?  Are you using virtio for both the network and the disk?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3454 2014-11-26 20:45:55

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ah, I see. So that output is expected behaviour.

Yes I am using virtio for disk and as of today also for network. According to http://wiki.libvirt.org/page/Virtio it should produce an output once the guest has loaded. That's why I was confused. ...

About the high CPU usage: whenever I load a game (even something just like Fifa 15, I have 100 % CPU usage once the game starts. That seems a tad too high?

Offline

#3455 2014-11-26 20:52:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

ah, I see. So that output is expected behaviour.

Yes I am using virtio for disk and as of today also for network. According to http://wiki.libvirt.org/page/Virtio it should produce an output once the guest has loaded. That's why I was confused. ...

From that source...

When you boot the guest (virsh start guestname), if it worked you should still have a working network, and you should see (from inside the guest) that you are using the virtio_net driver:

# /sbin/lsmod | grep virtio [shows virtio_pci, virtio_net and others loaded] # cat /sys/devices/virtio-pci/0/net/eth0/statistics/rx_bytes ...

About the high CPU usage: whenever I load a game (even something just like Fifa 15, I have 100 % CPU usage once the game starts. That seems a tad too high?

Why?  Moving data costs CPU.  Games typically move a lot of data on startup.  What are you using to back the disk image?  If you want performance you want either a non-sparse raw file, LVM volume, or physical block device behind it.  If you're using something like qcow2, you have to realize that there's a speed vs space trade-off in doing that.

Last edited by aw (2014-11-26 20:55:08)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3456 2014-11-26 20:59:14

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, that's why I am using LVM. I have created a 180 GB partition from my 240 GB SSD for Windows 8.1. The other 40 GB are for arch.

I definitely prefer performance. Isn't LVM the best option? It used to with Xen.

Last edited by 4kGamer (2014-11-26 21:01:49)

Offline

#3457 2014-11-26 21:16:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

yes, that's why I am using LVM. I have created a 180 GB partition from my 240 GB SSD for Windows 8.1. The other 40 GB are for arch.

I definitely prefer performance. Isn't LVM the best option? It used to with Xen.

A direct block device, like a partition is probably best, but LVM is nearly as good.

You can try some of the block tuning techniques from the Red Hat Virtualization Tuning and Optimization Guide


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3458 2014-11-26 21:25:28

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, thank you very much. I will try those suggestions. I will report once I get some significant improvements.

Offline

#3459 2014-11-26 23:27:02

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Reboot is now working. I extracted the vga bios with GPU-Z and passed it to qemu with romfile=

I tried to get audio working, but I failed. -soundhw hda seams the only one which is recognized from Windows8 64bit, but i get scraching/stutter.
Somebody said, he got -soundhw ac97 working, he used the Win7 driver for ac97 and installed it in under Win8 under Win7 compatibility mode, did not work for me eiter, maybe somebody else gets lucky with it.

Offline

#3460 2014-11-26 23:27:43

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:
aw wrote:

Most of these seem completely unnecessary to me, especially the vfio options.  APICv is already enabled by default when available, so that's a no-op.  nested=1, why?  You're not doing nesting.  ignore_msrs=1 is the only thing I have set on my box.

I know . There was a time when I completely commented out every line of those options and the VMs booted just fine . But they weren't stable after hours of operation . Plus , with them on , both of my VMs can boot fine at the same time (2 seperate GPUs , one to each VM) .

I do this too, one AMD + one Nvidia.  AFAIK, I only need ignore_msrs so that passmark performance test works

THe masking option is needed in my case , otherwise the whole VM goes into SloMo mode . smile

That's specific to a device you're assigning, not likely related to the chipset.  The option overrides our detection of whether the device supports INTxDisable masking and instead masks the interrupt at the APIC instead of the device.  The huge downside of that is that it requires the device has an exclusive interrupt because of the difference in where we mask the interrupt.  Also note:

$ modinfo vfio-pci
...
parm:           nointxmask:Disable support for PCI 2.3 style INTx masking.  If this resolves problems for specific devices, report lspci -vvvxxx to linux-pci@vger.kernel.org so the device can be fixed automatically via the broken_intx_masking flag. (bool)

Have you done that?  This option would apply automatically to just that device if you did...

I'll comment out the APICv + nested + shadow_vmcs + ept from now on though . I already commented out the disable_hugepages one for a month or so , no problem .

I didn't need any of these options when I was using Z77 .. X99 is more troublesome to setup . ALOT . And there is no way I can reboot any VM with GPU assigned to it without crashing the host sad It is the ONLY remaining issue I have now .

They reboot fine , until I install Nvidia's drivers and then they crash the host whenever I reboot them .

Aside from the two Radeon reset issues (R7790 and similar can only be used once per boot and HD8570 and similar mistakenly report NoSoftRst-) I don't know of any other GPU reset issues, nor do I see why X99 would be more or less problematic than Z77 in that respect.

EDIT: Nvidia cards have pretty terrible DisableINTx support on the audio device (esp. Quadro and even fake Quadro), but the better option there is typically to make the guest driver use MSI, not to run with nointxmask

I have a Linux VM (XBMCBuntu) with passed through GT620 which works well except for reboots cause it to fail completely. I believe it is something to do with the Nvidia card as I have other VMs passing through raid controller and Sat card with VFIO that can reboot fine.

Interrupts within the VM looks a bit odd (The passed through Intel VF uses MSI but the Geforce does not) and config is shown below. Can anyone help with what might be wrong or how I can collect needed debug information (Qemu log for the VM shows nothing other than on intial startup of the VM).

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>XBMC</name>
  <uuid>f702c5eb-83d4-4b50-b909-8da878e08d36</uuid>
  <memory unit='KiB'>1036288</memory>
  <currentMemory unit='KiB'>1036288</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/VM/XBMC.img'/>
      <target dev='sda' bus='sata'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/ISO/XBMC13.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
<controller type='usb' index='0' />
<hostdev mode='subsystem' type='usb'>
      <source>
        <vendor id='0x413c'/>
        <product id='0x2106'/>
      </source>
    </hostdev>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=09:10.2,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0a:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0a:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>
</domain>

Offline

#3461 2014-11-27 08:28:19

kevinxucs
Member
Registered: 2014-09-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just got an NVIDIA Geforce GTX 970 working using vfio-pci.

Initially, when I was using SeaBios (1.7.5 from Debian), it doesn't seem to have any output. After I switched to using EFI boot (following tutorial here), my 970 works perfectly fine.

Also, kvm=off still works with 970 and the latest driver 344.75 (for resolving code 43 issue with NVIDIA's driver).

Last edited by kevinxucs (2014-11-27 08:33:23)

Offline

#3462 2014-11-27 09:33:36

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding OVMF/UEFI: I heard rumors that this also works with Windows7. Can anybody here confirm that a Windows 7 VM is working with OVMF?

Offline

#3463 2014-11-27 09:47:46

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again. Anyone has solution for suspend/hibernation problem (both fail - suspend sometimes, hibernation always) with kernel from OP and Nvidia-340xx-dkms drivers?

Offline

#3464 2014-11-27 11:27:36

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Hi again. Anyone has solution for suspend/hibernation problem (both fail - suspend sometimes, hibernation always) with kernel from OP and Nvidia-340xx-dkms drivers?

I think this is overall problem of NVIDIA cards, not kernel in particular. At least I had such problems.

Offline

#3465 2014-11-27 11:59:38

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well I haven't had this problem for a long time when using standard Arch kernel. smile

Offline

#3466 2014-11-27 14:00:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Regarding OVMF/UEFI: I heard rumors that this also works with Windows7. Can anybody here confirm that a Windows 7 VM is working with OVMF?

Well, i can confirm that. Windows 7 supports UEFI, but you need a somewhat special way to install it(obviously, the cd should have something like .EFI loader file). Mine installation ISO supported that. But i won't provide it to you because i'm a filthy pirate and should be punished for using windows.
I've got to install and boot it, but i couldn't get vfio/pci passthrough working because, well, my asus GPUs needed to be physically flashed - i couldn't boot myself with romfile= pointing to my card's UEFI-compatible VBIOS update. So YMMV. Also, there is GPT support if you're curious.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3467 2014-11-27 14:01:47

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Gotta love redhat for that document. Pure awesome.

Last edited by Duelist (2014-11-27 14:02:17)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3468 2014-11-27 16:42:09

wikavalier
Member
Registered: 2013-01-22
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

First of all, thank you a lot for the post. It is awesome and really well documented. 10/10.

Even me, with my rudimentary linux skills, I was able to make the passthrough to the KVM, after facing all kind of issues. Yes, I had all of them!.

- My box is a:
intel i7
gigabyte mobo
nvidia gtx780
and the damn:
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 10)

- I am running:
Linux nexus2 3.17.2-1-mainline #1 SMP PREEMPT Mon Nov 24 17:50:40 CET 2014 x86_64 GNU/Linux

And when I activate intel_iommu=on, the raid6 which is partly held by the pcie-SATA marvell controller, stops working. So I have to decide whether I want vfio or my raid storage

I dive a little bit, and I found that the hardware-monster (compliment here!) known as Alex Williamson (AKA aw) has also a patch for my problem here:
https://github.com/awilliam/linux-vfio/ … a-alias-v4

I cloned the git but I downloaded again a new kernel. But with my poor debugging skills, I cant tell if that fork will include the patches from nbhs and the fear of starting over again is killing me. Step by step checking everything.

Yeah, I have read about the help-vampire. But, after my duly apologizes, can anyone be so kind to tell me how to apply the dma-alias to the modified kernel of nbhs? For dummies, if i can abuse.

Thanks in advance,


EDIT, if this can be interesting to anyone, below you may see the steps that I took to make it work. It is been a pain!

https://bbs.archlinux.org/viewtopic.php?id=162768 http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm asume a fresh arch install copy linux-mainline tar xzf linux-mainline-3.17.2.tar.gz cd linux-mainline* makepkg -s --asroot (option 15 haswell) (esc esc to exit) (2hours) pacman -U linux-mainline-* lspci: 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller 01:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX 780 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1) lspci -n 00:1b.0 0403: 8086:8ca0 <--- AUDIO mobo to be pci_stub 01:00.0 0300: 10de:100a (rev a1) <-- NVIDIA graphics to be pci_stub 01:00.1 0403: 10de:0e1a (rev a1) <--- NVIDIA audio to be pci_stub nano /etc/default/grub GRUB_CMDLINE_LINUX="pci-stub.ids=8086:8ca0,10de:100a,10de:0e1a i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream" grub-mkconfig -o /boot/grub/grub.cfg nano /etc/mkinitcpio.conf MODULES="i915 pci-stub vfio-pci" mkinitcpio -p linux-mainline lsusb Bus 001 Device 005: ID 0738:1713 Mad Catz, Inc. Bus 001 Device 002: ID 03f0:0324 Hewlett-Packard SK-2885 keyboard Bus 001 Device 007: ID 0781:5571 SanDisk Corp. Cruzer Fit pacman -S mesa-libgl qemu seabios echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf nano /usr/bin/vfio-bind #!/bin/bash modprobe vfio-pci for dev in "$@"; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done chmod +x /usr/bin/vfio-bind nano /etc/systemd/system/vfio-bind.service [Unit] Description=Binds devices to vfio-pci After=syslog.target [Service] EnvironmentFile=-/etc/vfio-pci.cfg Type=oneshot RemainAfterExit=yes ExecStart=-/usr/bin/vfio-bind $DEVICES [Install] WantedBy=multi-user.target nano /etc/vfio-pci.cfg DEVICES="0000:00:1b.0 0000:01:00.0 0000:01:00.1" debug vfio-bind 0000:00:1b.0 0000:01:00.0 0000:01:00.1 ls -l /sys/bus/pci/drivers/vfio-pci/ find /sys/kernel/iommu_groups/ -type l dmesg | grep -e DMAR -e IOMMU launch VM vifo #detach usb echo 0 > /sys/bus/usb/devices/1-2/authorized echo 0 > /sys/bus/usb/devices/1-5/authorized echo 0 > /sys/bus/usb/devices/1-7/authorized # bind the graphics, sound and mobo sound to vfio vfio-bind 0000:00:1b.0 0000:01:00.0 0000:01:00.1 #allow unsafe remapping modprobe vfio_iommu_type1 modprobe kvm #-M q35 is the emulated chipset # smp how many cpus will be shown #load emulated bios and set no emulated cirrus card #create a pcie root port to attach the gpu called root.1 #attach the nvidia card to root.1 #attach the mobo soundcard to pcie.0 #use a phisical partition attach to q35 (ide.0) #passthrough usbs qemu-system-x86_64 -enable-kvm -M q35 -m 8024 -cpu host \ -smp 6,sockets=1,cores=3,threads=2 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1b.0,bus=pcie.0,addr=1b.0 \ -drive file=/dev/sda4,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/test/windowsos.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -usb -usbdevice host:03f0:0324 -usbdevice host:0738:1713 \ -nographic

EDIT 2,

I found that appending a .patch at the end of the url i can download the branch as a patch. Then i edited the PKGBUILD and update the signature with updpkgsums. Sadly it doesnt work... it doesnt compile

I also tried the v4.1 which seems to be the new way to do so, using devfn. It also explicitly uses the vendor ID of my sata controller (9230) But i cant find it in github. And when i try to copy/paste from the mail list, it doesnt work:

patching file drivers/pci/quirks.c patch: 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html malformed patch at line 16: DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_RICOH, 0xe832, quirk_dma_func0_alias); ==> ERROR: A failure occurred in prepare(). Aborting...

EDIT 3,
I used this as a patch
http://www.jarzebski.pl/files/patch/dma … ices.patch

and after many trials and errors i ended up putting it as the last patch like this in the PKGBUILD

 # Extra GCC optimizations patch -p1 -i "${srcdir}/enable_additional_cpu_optimizations_for_gcc_v4.9+_kernel_v3.15+.patch" # patches Marvell sata patch -p1 -i "${srcdir}/dma-alias-devfn.patch"

but after a good looking begining it end up like always

 CC drivers/pci/quirks.o drivers/pci/quirks.c:3535:13: error: redefinition of 'quirk_dma_func1_alias' static void quirk_dma_func1_alias(struct pci_dev *dev) ^ drivers/pci/quirks.c:3470:13: note: previous definition of 'quirk_dma_func1_alias' was here static void quirk_dma_func1_alias(struct pci_dev *dev) ^ scripts/Makefile.build:257: recipe for target 'drivers/pci/quirks.o' failed make[2]: 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html [drivers/pci/quirks.o] Error 1 scripts/Makefile.build:404: recipe for target 'drivers/pci' failed make[1]: 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html [drivers/pci] Error 2 Makefile:929: recipe for target 'drivers' failed make: 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_13.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html [drivers] Error 2 ==> ERROR: A failure occurred in build(). Aborting...

So, my hacking skills doesnt work to create a proper patch.

EDIT 4,
Ok, after lots of trial and error, I found out that I tried to patch something that was already included....

And my raid card is not compatible.... great!. Googling for a new one...

Any suggestion for a SAT 4xports cheap and linux friendly? For a storage raid



Any good Samaritan?

Last edited by wikavalier (2014-11-28 20:33:45)

Offline

#3469 2014-11-27 17:42:04

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

this was extremely valuable find for me

so i'm leaving it here, in case someone relied purely on "static" usb passthrough but was not very happy with it

Offline

#3470 2014-11-28 12:34:27

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all.

About my audio issue (stuttering audio using an USB Xonar U7 sound card). It is more or less fixed. I bought a new USB sound card : Hercules LT3, and it now works as expected. I passthrough the whole USB controller (including sound card, and input devices), and it works fine!

So  far it seems it was only an hardware related issue. I have had this issue for months using Xen (iirc), and KVM under Ubuntu and Arch. If I can give people a piece of advice : dont buy this Xonar U7 for this usage.

Last edited by Nesousx (2014-11-28 15:37:43)

Offline

#3471 2014-11-28 17:24:10

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
TripleSpeeder wrote:

Regarding OVMF/UEFI: I heard rumors that this also works with Windows7. Can anybody here confirm that a Windows 7 VM is working with OVMF?

Well, i can confirm that. Windows 7 supports UEFI, but you need a somewhat special way to install it(obviously, the cd should have something like .EFI loader file). Mine installation ISO supported that. But i won't provide it to you because i'm a filthy pirate and should be punished for using windows.
I've got to install and boot it, but i couldn't get vfio/pci passthrough working because, well, my asus GPUs needed to be physically flashed - i couldn't boot myself with romfile= pointing to my card's UEFI-compatible VBIOS update. So YMMV. Also, there is GPT support if you're curious.

Hmm, I do have the installation iso containing UEFI loader etc. But even with a minimum setup (no pass-through devices, just the iso loaded as the only CDROM, no additional HDs) i don't have much success. I can launch the windows bootloader from the UEFI shell, it starts with the black/white screen "loading Files", followed by the graphical "Starting Windows" screen. 1-2 seconds later i get a black screen again with some kind of black/white progress bar (?) on top, which is not moving. THere it stops, with one CPU core running at 100%. I left it for +10 minutes, with nothing happening.

Just for testing I removed the emulated graphics and passed through my GTX970, which was working flawlessly, except for the installation progress stopping at the exact same place :-(

Could you share your qmeu commandline / libvirt xml config?

Thanks!

Edit: I used ovmf-svn from AUR, compiled yesterday.

Last edited by TripleSpeeder (2014-11-28 17:24:49)

Offline

#3472 2014-11-28 19:36:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Hmm, I do have the installation iso containing UEFI loader etc. But even with a minimum setup (no pass-through devices, just the iso loaded as the only CDROM, no additional HDs) i don't have much success. I can launch the windows bootloader from the UEFI shell, it starts with the black/white screen "loading Files", followed by the graphical "Starting Windows" screen. 1-2 seconds later i get a black screen again with some kind of black/white progress bar (?) on top, which is not moving. THere it stops, with one CPU core running at 100%. I left it for +10 minutes, with nothing happening.

Just for testing I removed the emulated graphics and passed through my GTX970, which was working flawlessly, except for the installation progress stopping at the exact same place :-(

Could you share your qmeu commandline / libvirt xml config?

Thanks!

Edit: I used ovmf-svn from AUR, compiled yesterday.

OVMF git repo build(i use fedora) 20141128.b817.gb04a63a

Seems like windows installation iso loses it's boot drive, but i got you a pair of screenshots getting past your point of breakage.

Here, have an imgur slideshow.

qemu-system-x86_64 \ -boot menu=on \ -enable-kvm \ -monitor stdio \ -M q35 \ -m 1024 \ -cpu host \ -net none \ -rtc base=localtime \ -smp 1,sockets=1,cores=1,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/mnt/hdd/qemu/OVMF_VARS.fd \ -drive file='/mnt/hdd/qemu/qemu-uefi-win7.img',id=disk,format=raw,aio=native,cache=none,if=none \ -drive file='/mnt/hdd/qemu/virtio.iso',id=cdrom,format=raw,readonly=on,if=none \ -drive file='/mnt/hdd/qemu/windows7.iso',id=cdrom2,format=raw,readonly=on,if=none \ -device ioh3420,addr=04.0,multifunction=on,port=1,chassis=2,id=root.1 \ -device virtio-blk-pci,bus=root.1,addr=03.0,drive=disk \ -device virtio-scsi-pci,bus=root.1,addr=05.0 \ -device ide-cd,bus=ide.1,drive=cdrom \ -device scsi-cd,drive=cdrom2 \ -vga std

And here's a loading script. Just a config based on my vfio setup, so it shouldn't work because i'm lazy to fix it, but it boots and tries to install.
Notice the pure-efi image of OVMF used.

EDIT:
seems like i've figured out why it fails to find it's boot drive. There is no virtio-scsi-pci drivers in the windows installation system, and the cdrom2 drive, which is windows7.iso, is connected to it. Changing that virtio-scsi-pci to something more generic should help the issue here.

Last edited by Duelist (2014-11-28 20:42:54)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3473 2014-11-29 15:10:52

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot Duelist! I found the root cause that stopped windows setup from continuing: It is the -cpu parameter.
This is the qemu command-line as provided by libvirt:

-cpu qemu64,hv_time

The moment I remove parameter hv_time, windows setup continues. And this problem only happens with smp=2 or higher. With only one core assigned to the VM it boots also with hv_time.

Any ideas out there what might be the root cause?

Edit: Note that this is still without any device pass-through, just using the default emulated vga. Trying to get setup running completely with passed-through NVidia card is the next step smile

Last edited by TripleSpeeder (2014-11-29 15:15:19)

Offline

#3474 2014-11-29 15:30:54

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now this is REALLY odd ...

Using "pci-assign" instead of "vfio-pci" to assign my devices to the VM seems to solves the reboot issue with GT610 GPU . This is using OVMF + 440FX by the way .

The VFIO still claims those devices at boot from the PCI-STUB . I don't know how much will this success last , but I rebooted the VM twice and it reboots fine . I thought the VFIO was supposed to fix the devices reset issues ?

dmesg :

[Sat Nov 29 18:17:31 2014] vfio-pci 0000:05:00.0: kvm assign device [Sat Nov 29 18:17:31 2014] vfio-pci 0000:05:00.1: kvm assign device [Sat Nov 29 18:17:32 2014] vfio-pci 0000:0f:00.0: kvm assign device [Sat Nov 29 18:17:32 2014] vfio-pci 0000:00:1b.0: kvm assign device

CMDLine :

qemu-system-x86_64 -name main -nographic -mem-path /dev/hugepages \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 2,sockets=1,cores=1,threads=2 \ -vga none \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_main.bin \ -device pci-assign,host=05:00.0,multifunction=on \ -device pci-assign,host=05:00.1 \ -device pci-assign,host=0f:00.0 \ -device pci-assign,host=00:1b.0 \ -drive file=/VMs/Win_Main.qcow2,cache=writeback,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -net nic,model=virtio,macaddr=64:C5:63:3C:1B:22 -net bridge,br=br0 \ -usb -usbdevice host:045e:0745 \ -localtime \ -monitor unix:/tmp/vm_main,server,nowait &

Let's see if this is going to be the solution for my ongoing issues with GT610 .

EDIT :

Installed Nvidia's drivers 344.75 (the latest I think) , reboot 2 times and it works perfectly . I'm happy , yet confused . smile

Last edited by Denso (2014-11-29 15:43:19)

Offline

#3475 2014-11-29 15:40:53

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Thanks a lot Duelist! I found the root cause that stopped windows setup from continuing: It is the -cpu parameter.
This is the qemu command-line as provided by libvirt:

-cpu qemu64,hv_time

The moment I remove parameter hv_time, windows setup continues. And this problem only happens with smp=2 or higher. With only one core assigned to the VM it boots also with hv_time.

Any ideas out there what might be the root cause?

Edit: Note that this is still without any device pass-through, just using the default emulated vga. Trying to get setup running completely with passed-through NVidia card is the next step smile

I experienced crashes during the windows installation in a non-vfio setup as well. The root cause were missing kernel config options for libvirt, but I guess you are using the archlinux kernel, right?

Offline

#3476 2014-11-29 16:22:35

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
TripleSpeeder wrote:

Thanks a lot Duelist! I found the root cause that stopped windows setup from continuing: It is the -cpu parameter.
This is the qemu command-line as provided by libvirt:

-cpu qemu64,hv_time

The moment I remove parameter hv_time, windows setup continues. And this problem only happens with smp=2 or higher. With only one core assigned to the VM it boots also with hv_time.

Any ideas out there what might be the root cause?

Edit: Note that this is still without any device pass-through, just using the default emulated vga. Trying to get setup running completely with passed-through NVidia card is the next step smile

I experienced crashes during the windows installation in a non-vfio setup as well. The root cause were missing kernel config options for libvirt, but I guess you are using the archlinux kernel, right?

Yes, i tried both current Arch linux kernel and patched kernel from OP.

Offline

#3477 2014-12-01 03:06:44

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone knows how convert xen disk image into kvm . I used dd of lvm to get img of partition. When I try to boot img under kvm I get error: file /grub/i386-pc/normal.mod not found
I tried to reinstall grub but I can get the error to go away

Offline

#3478 2014-12-01 17:04:25

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

Does anyone knows how convert xen disk image into kvm . I used dd of lvm to get img of partition. When I try to boot img under kvm I get error: file /grub/i386-pc/normal.mod not found
I tried to reinstall grub but I can get the error to go away

how the hell is your question related to this thread's topic?

not to mention that judging by error message alone it's obvious that it may be architecture mismatch

Last edited by sinny (2014-12-01 17:04:43)

Offline

#3479 2014-12-01 21:09:56

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://imageshack.com/i/p8J5tWtEp
https://imageshack.com/i/ipmipYYxp
I finaly succeeded to launch OSX 10.9.5 Mavericks on my Arch + Qemu 2.1.2

Last edited by mastersplinter777 (2014-12-01 21:10:37)

Offline

#3480 2014-12-01 21:18:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mastersplinter777 wrote:

https://imageshack.com/i/p8J5tWtEp
https://imageshack.com/i/ipmipYYxp
I finaly succeeded to launch OSX 10.9.5 Mavericks on my Arch + Qemu 2.1.2

Cool.  And since running a VM with GPU acceleration in a window always brings the newbies out, let's just point out right now that that's a VNC connection to the VM, guest-based VNC server.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3481 2014-12-01 21:28:15

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mastersplinter777 wrote:

https://imageshack.com/i/p8J5tWtEp
https://imageshack.com/i/ipmipYYxp
I finaly succeeded to launch OSX 10.9.5 Mavericks on my Arch + Qemu 2.1.2

Cool.  And since running a VM with GPU acceleration in a window always brings the newbies out, let's just point out right now that that's a VNC connection to the VM, guest-based VNC server.

VNC connection is the only more or less comfotable way to connect graphically to VM if I have one physical display with different inputs... Besides I also use VNC connection with my Win8.1 VM and it is rather fast, so I can even play some 3d games or chat using my webcam in ooVoo

Offline

#3482 2014-12-01 21:29:32

sumpfmensch
Member
Registered: 2014-06-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there,

after trying the whole passthrough journey with ubuntu, I finally gave up and tried to follow the guide in the first post with a new arch installation but I just can't get to pass through my VTX R9 280 3GB to a guest on my arch host. The mobo is the Gigabyte Z97 UD3H BK and the Processor the new 4690k (which supports VT-d).

Passing through an old 8800 GTX seems to work (i.e. gives me the seabios "No boot device found" output) using the same commands and setup. My R9 280 however does not want to output any signal at all (neither via HDMI, DVI nor DVI->VGA).

Since I'm all out of ideas (reading and trying since 3 days almost fulltime now) I'd like to ask for any further advice. The huge amount of posts I skimmed through in this thread usually seem to have get an error message when receiving no signal. I don't see anything problematic in dmesg and no output from qemu at all.

Here is my current setup which seems to work with the Nvidia card (removing the entries for 0000:01:00.1 in the vfio-bind script and the qemu call).

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=eed2a69c-c80f-4820-92bb-2b2afe2b7ad5 rw quiet pci-stub.ids=1002:679a,1002:aaa0 i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream

Here i added pcie_acs_override=downstream later on although I think it should not be needed having both 0000:01:00.0 and 0000:01:00.0 on iommu group 13 (and no other devices there).

The script for vfio binding: cat vfio-bind

DEVICES="0000:01:00.0 0000:01:00.1" modprobe vfio-pci for dev in $DEVICES; do vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id done

ls /sys/bus/pci/drivers/vfio-pci/                 

0000:01:00.0 0000:01:00.1 bind module new_id remove_id uevent unbind

The final command I use to start the machine:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -vga none -nographic\ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/root/ati_r9_280.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

(I tried it with and without the romfile i got from /sys/.../rom - no difference - played around with the pc chipset - or should I continue trying to relocate the devices?)

cat /etc/modprobe.d/blacklist.conf

blacklist radeon blacklist nouveau 

dmesg

... [ 221.927937] VFIO - User Level meta-driver version: 0.3 [ 350.698684] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 350.722340] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 350.722345] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 [ 353.236170] kvm: zapping shadow pages for mmio generation wraparound

Do you have any Ideas or pointers where I may continue my quest for passthrough? (I can't switch to OVMF since I got two windows 7 licences I want to run on VMs which I do not want to throw away.)

I would just love to see that bios screen - from that point on I already have Ideas how to proceed.

Offline

#3483 2014-12-01 21:52:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sumpfmensch wrote:

Hello there,

after trying the whole passthrough journey with ubuntu, I finally gave up and tried to follow the guide in the first post with a new arch installation but I just can't get to pass through my VTX R9 280 3GB to a guest on my arch host. The mobo is the Gigabyte Z97 UD3H BK and the Processor the new 4690k (which supports VT-d).

Passing through an old 8800 GTX seems to work (i.e. gives me the seabios "No boot device found" output) using the same commands and setup. My R9 280 however does not want to output any signal at all (neither via HDMI, DVI nor DVI->VGA).

Since I'm all out of ideas (reading and trying since 3 days almost fulltime now) I'd like to ask for any further advice. The huge amount of posts I skimmed through in this thread usually seem to have get an error message when receiving no signal. I don't see anything problematic in dmesg and no output from qemu at all.

Here is my current setup which seems to work with the Nvidia card (removing the entries for 0000:01:00.1 in the vfio-bind script and the qemu call).

When you switch from the Nvidia card to the Radeon, are you physically swapping them?  If not I'd suspect something with primary VGA or the nvidia driver getting in there and locking the VGA arbiter.  If so, I dunno...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3484 2014-12-01 22:21:08

sumpfmensch
Member
Registered: 2014-06-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

When you switch from the Nvidia card to the Radeon, are you physically swapping them?  If not I'd suspect something with primary VGA or the nvidia driver getting in there and locking the VGA arbiter.  If so, I dunno...

Yes, I shut down the machine, pull the card and the power connectors out and insert the other one. My Primary VGA is indeed the Intel card but I hoped I'd have solved the arbiter problems with the kernel provided in post 1.


Maybe the following is interesting, too:

When I boot the vm with an ubuntu live disk, the attached ATI card AND the default std ouput, lspci shows me that the ATI card is present ... While the radeon driver gets loaded assigned to the card, it is not activated by default. It does neither show up in the display settings not output anything. When I boot the linux image on bare metal, (with the Intel card set as primary) it was used for a second screen.

EDIT: I'm using the -M pc currently since I have problems booting the image ("no boot device") with q35
EDIT2: Ok - getting used to the different drive syntax for q35 - however i cannot get the radeon to output anything on the virtualized ubuntu. Already starting to consider switching to xen and starting all over again or even selling the card hmm

Hope somebody can think of anything to try next.

Last edited by sumpfmensch (2014-12-02 00:48:42)

Offline

#3485 2014-12-02 04:27:08

consty
Member
Registered: 2014-12-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

wikavalier wrote:

Hi all,

First of all, thank you a lot for the post. It is awesome and really well documented. 10/10.

Even me, with my rudimentary linux skills, I was able to make the passthrough to the KVM, after facing all kind of issues. Yes, I had all of them!.

- My box is a:
intel i7
gigabyte mobo
nvidia gtx780
and the damn:
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 10)

- I am running:
Linux nexus2 3.17.2-1-mainline #1 SMP PREEMPT Mon Nov 24 17:50:40 CET 2014 x86_64 GNU/Linux

And when I activate intel_iommu=on, the raid6 which is partly held by the pcie-SATA marvell controller, stops working. So I have to decide whether I want vfio or my raid storage

I was in the same boat as you with the marvell 9230. I found that booting with intel_iommu=on AND iommu=pt allowed me to passthrough hardware while still having access to my RAID controller.

I don't know if this causes any system instability or if I'm just a special case where this works for only me.

Offline

#3486 2014-12-02 06:08:25

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I  wanna buy a new CPU and motherboard.

Bearing in mind, we get round about 15-20 % less performance in a VM, I'd like to compensate that with stronger, better hardware. My guest OS is Windows 8.1 and I own a MSI GTX 980

I just bought the i7-4790k http://www.newegg.com/Product/Product.a … 6819117369

But I am not 100 % sure I should stay with it (it's still sealed and I haven't decided on the motherboard yet)

There are two other interesting options:

the  I7-5820K http://www.newegg.com/Product/Product.a … 6819117402

and the I7-5930K http://www.newegg.com/Product/Product.a … 6819117403


I just don't know if it's worth going for the more expensive and more power hungry options. What would you guys do if we forget the cost for a moment? It's all about 4k gaming really

Last edited by 4kGamer (2014-12-02 07:37:53)

Offline

#3487 2014-12-02 08:04:56

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi, I  wanna buy a new CPU and motherboard.

Bearing in mind, we get round about 15-20 % less performance in a VM, I'd like to compensate that with stronger, better hardware. My guest OS is Windows 8.1 and I own a MSI GTX 980

I just bought the i7-4790k http://www.newegg.com/Product/Product.a … 6819117369

But I am not 100 % sure I should stay with it (it's still sealed and I haven't decided on the motherboard yet)

There are two other interesting options:

the  I7-5820K http://www.newegg.com/Product/Product.a … 6819117402

and the I7-5930K http://www.newegg.com/Product/Product.a … 6819117403


I just don't know if it's worth going for the more expensive and more power hungry options. What would you guys do if we forget the cost for a moment? It's all about 4k gaming really

Hello,

To be honest I have a 4790k and the performance is really good. No problem at all.

I think the 4790k can be faster (and cheaper) for gaming and the other one probably is better for doing more multitasking (according to cores). Anyway, the clock is faster on 4790k.

About playing games with 4K resolution... is up to your graphic cards really and I'm not expert on that smile But I guess that you probably need more than one if you really want it in Ultra mode smile

So, my two cents... 4790K and a Z97 motherboard / X99 (check for vt-d support (I have the Extreme 6 Z97 with good results)) and use the internal IGP for host and two graphic cards for passthrough (two NVIDIA GTX 770 should be enough and I know for sure that works fine smile).

I hope it helps smile

Regards,


TheArcher

Offline

#3488 2014-12-02 08:50:14

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks a lot for your input! Really appreciated. Yeah, the i7-4790k is faster than the other ones. I think I will stick with it then. And thank you for suggesting a motherboard. I have seen so many that I don't know which one I should buy. But it's great to hear the ASRock Z97 EXTREME6 is doing a fine job in a VM.

One more question: what CPU cooler do you use? And are you happy with it?

Last edited by 4kGamer (2014-12-02 08:50:26)

Offline

#3489 2014-12-02 09:56:55

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thanks a lot for your input! Really appreciated. Yeah, the i7-4790k is faster than the other ones. I think I will stick with it then. And thank you for suggesting a motherboard. I have seen so many that I don't know which one I should buy. But it's great to hear the ASRock Z97 EXTREME6 is doing a fine job in a VM.

One more question: what CPU cooler do you use? And are you happy with it?

A pleasure if I helped you smile

Is not the proper thread for this but, Noctua is a good brand for normal coolers. If you look for water cooling... H110 is a good one for non super geeks smile

Anyway, it also depends of your case. In my case I chose a not too big one to have my computer at the living room. I have Noctua fans and I'm happy about the noise but I think can be better results.

My idle temperatures are around 34C (room is near 20) and running the VM is around 37-38C.

When I play dragon age, graphic card goes to 80C and the computer is on 70-72C.

Can be better? yes. My though was to have the H110 but there was not enough room on my case for it.

I didn't notice any difference between running the VM or Windows as a baremetal to be honest.

I hope it helps smile

Regards,


TheArcher

Last edited by thearcherblog (2014-12-02 10:05:17)

Offline

#3490 2014-12-02 10:08:55

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I read somewhere about the h110. I will have a look into that. You have been really helpful. Now I can make some decisions smile

thank you!

Offline

#3491 2014-12-02 10:36:38

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

I think I read somewhere about the h110. I will have a look into that. You have been really helpful. Now I can make some decisions smile

thank you!

You welcome smile

Offline

#3492 2014-12-02 20:46:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi, I  wanna buy a new CPU and motherboard.

Sounds like you're already committed to a pricey upgrade, but FWIW I decided to go a more mild route upgrading from the i5-3470T.  I found my motherboard is compatible with a Xeon E3-1245v2, found a reasonable deal for one on ebay, and now I've got 2x the cores and a 500MHz boost.  Including a Cooler Master Hyper 212 Plus, I'm in it for well below the price of an i7-4790k alone, and it seems to have enough juice to run Steam in-home-streaming with the GTX750 (4-cores for the VM, leaving 4 threads for the host).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3493 2014-12-02 22:09:23

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah, but you know what you're doing smile I am just following guides without the knowledge wink Seriously, I am very grateful for your input and I am sure I will need it again once I get my new motherboard and run the new CPU. Hopefully towards the weekend smile

Offline

#3494 2014-12-02 22:30:23

MatthiasArch
Member
Registered: 2014-12-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi There!
I'm very happy this thread exists as everything is working (more or less) out of the box smile
I have an Intel i5 4670 on an ASRock B85M-ITX and a Gigabyte Nvidia GTX750TI.
Running Archlinux default

Linux loewenbraeu 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux

with current arch qemu/kvm/libvirt with Windows 8.1 Pro as OVMF uefi booted guest and Nvidia card passed through. Everything works like a charm UNTIL I shut down the VM!

Then Hell kicks in:

[ 1164.246920] ------------[ cut here ]------------ [ 1164.246935] WARNING: CPU: 3 PID: 303 at drivers/gpu/drm/i915/i915_gem_gtt.c:1492 gen6_ggtt_insert_entries+0x146/0x150 [i915]() [ 1164.246936] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec [ 1164.246957] nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio [ 1164.246975] CPU: 3 PID: 303 Comm: Xorg.bin Tainted: G O 3.17.4-1-ARCH #1 [ 1164.246976] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014 [ 1164.246977] 0000000000000000 000000008bc678ec ffff8800a5cffc68 ffffffff81537b0e [ 1164.246978] 0000000000000000 ffff8800a5cffca0 ffffffff8107078d 00000000fc8c9000 [ 1164.246980] 00000000ffffffff ffffc9000598b424 ffff8800bb417a58 ffffc90005989100 [ 1164.246981] Call Trace: [ 1164.246985] [<ffffffff81537b0e>] dump_stack+0x4d/0x6f [ 1164.246988] [<ffffffff8107078d>] warn_slowpath_common+0x7d/0xa0 [ 1164.246989] [<ffffffff810708ba>] warn_slowpath_null+0x1a/0x20 [ 1164.246994] [<ffffffffa0531786>] gen6_ggtt_insert_entries+0x146/0x150 [i915] [ 1164.246999] [<ffffffffa05309b9>] ggtt_bind_vma+0xc9/0xf0 [i915] [ 1164.247005] [<ffffffffa053b3cb>] i915_gem_object_set_cache_level+0x29b/0x360 [i915] [ 1164.247009] [<ffffffffa053b53d>] i915_gem_set_caching_ioctl+0xad/0xf0 [i915] [ 1164.247015] [<ffffffffa03affef>] drm_ioctl+0x1df/0x680 [drm] [ 1164.247017] [<ffffffff811da5f0>] do_vfs_ioctl+0x2d0/0x4b0 [ 1164.247019] [<ffffffff811e4cae>] ? __fget+0x6e/0xb0 [ 1164.247021] [<ffffffff811da851>] SyS_ioctl+0x81/0xa0 [ 1164.247023] [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b [ 1164.247024] ---[ end trace 59b5489690ced9e7 ]--- [ 1164.247025] ------------[ cut here ]------------ [ 1164.247030] WARNING: CPU: 3 PID: 303 at drivers/gpu/drm/i915/intel_uncore.c:47 assert_device_not_suspended.isra.7+0x43/0x50 [i915]() [ 1164.247031] Device suspended [ 1164.247032] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec [ 1164.247049] nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio [ 1164.247061] CPU: 3 PID: 303 Comm: Xorg.bin Tainted: G W O 3.17.4-1-ARCH #1 [ 1164.247062] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014 [ 1164.247062] 0000000000000000 000000008bc678ec ffff8800a5cffbc0 ffffffff81537b0e [ 1164.247064] ffff8800a5cffc08 ffff8800a5cffbf8 ffffffff8107078d ffff8800bb410000 [ 1164.247065] 0000000000101008 ffff8800bb410068 ffff8800bb417a58 0000000000000001 [ 1164.247067] Call Trace: [ 1164.247068] [<ffffffff81537b0e>] dump_stack+0x4d/0x6f [ 1164.247070] [<ffffffff8107078d>] warn_slowpath_common+0x7d/0xa0 [ 1164.247071] [<ffffffff8107080c>] warn_slowpath_fmt+0x5c/0x80 [ 1164.247077] [<ffffffffa0556423>] assert_device_not_suspended.isra.7+0x43/0x50 [i915] [ 1164.247082] [<ffffffffa0559de7>] hsw_write32+0x37/0x160 [i915] [ 1164.247086] [<ffffffffa0531716>] gen6_ggtt_insert_entries+0xd6/0x150 [i915] [ 1164.247091] [<ffffffffa05309b9>] ggtt_bind_vma+0xc9/0xf0 [i915] [ 1164.247096] [<ffffffffa053b3cb>] i915_gem_object_set_cache_level+0x29b/0x360 [i915] [ 1164.247100] [<ffffffffa053b53d>] i915_gem_set_caching_ioctl+0xad/0xf0 [i915] [ 1164.247104] [<ffffffffa03affef>] drm_ioctl+0x1df/0x680 [drm] [ 1164.247106] [<ffffffff811da5f0>] do_vfs_ioctl+0x2d0/0x4b0 [ 1164.247108] [<ffffffff811e4cae>] ? __fget+0x6e/0xb0 [ 1164.247109] [<ffffffff811da851>] SyS_ioctl+0x81/0xa0 [ 1164.247111] [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b [ 1164.247112] ---[ end trace 59b5489690ced9e8 ]--- [ 1164.247114] [drm:hsw_unclaimed_reg_detect] *ERROR* Unclaimed register detected. Please use the i915.mmio_debug=1 to debug this problem. [ 1164.247115] ------------[ cut here ]------------ [ 1164.247120] WARNING: CPU: 3 PID: 303 at drivers/gpu/drm/i915/intel_uncore.c:47 assert_device_not_suspended.isra.7+0x43/0x50 [i915]() [ 1164.247120] Device suspended [ 1164.247121] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec [ 1164.247138] nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio [ 1164.247150] CPU: 3 PID: 303 Comm: Xorg.bin Tainted: G W O 3.17.4-1-ARCH #1 [ 1164.247151] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014 [ 1164.247151] 0000000000000000 000000008bc678ec ffff8800a5cffbb8 ffffffff81537b0e [ 1164.247153] ffff8800a5cffc00 ffff8800a5cffbf0 ffffffff8107078d ffff8800bb410068 [ 1164.247154] 0000000000101008 ffffc9000598b424 0000000000101008 ffff8800bb410000 [ 1164.247155] Call Trace: [ 1164.247157] [<ffffffff81537b0e>] dump_stack+0x4d/0x6f [ 1164.247158] [<ffffffff8107078d>] warn_slowpath_common+0x7d/0xa0 [ 1164.247160] [<ffffffff8107080c>] warn_slowpath_fmt+0x5c/0x80 [ 1164.247165] [<ffffffffa0556423>] assert_device_not_suspended.isra.7+0x43/0x50 [i915] [ 1164.247169] [<ffffffffa05592d5>] gen6_read32+0x35/0x150 [i915] [ 1164.247174] [<ffffffffa0531729>] gen6_ggtt_insert_entries+0xe9/0x150 [i915] [ 1164.247178] [<ffffffffa05309b9>] ggtt_bind_vma+0xc9/0xf0 [i915] [ 1164.247183] [<ffffffffa053b3cb>] i915_gem_object_set_cache_level+0x29b/0x360 [i915] [ 1164.247188] [<ffffffffa053b53d>] i915_gem_set_caching_ioctl+0xad/0xf0 [i915] [ 1164.247191] [<ffffffffa03affef>] drm_ioctl+0x1df/0x680 [drm] [ 1164.247193] [<ffffffff811da5f0>] do_vfs_ioctl+0x2d0/0x4b0 [ 1164.247195] [<ffffffff811e4cae>] ? __fget+0x6e/0xb0 [ 1164.247196] [<ffffffff811da851>] SyS_ioctl+0x81/0xa0 [ 1164.247198] [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b [ 1164.247199] ---[ end trace 59b5489690ced9e9 ]--- [ 1164.248115] input: Logitech USB Gaming Mouse as /devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3:1.0/0003:046D:C049.0007/input/input31 [ 1164.248166] hid-generic 0003:046D:C049.0007: input,hidraw2: USB HID v1.11 Mouse [Logitech USB Gaming Mouse] on usb-0000:00:14.0-3/input0 [ 1164.249614] hid-generic 0003:046D:C049.0008: hiddev0,hidraw3: USB HID v1.11 Device [Logitech USB Gaming Mouse] on usb-0000:00:14.0-3/input1 [ 1165.942981] kernel tried to execute NX-protected page - exploit attempt? (uid: 0) [ 1165.943032] BUG: unable to handle kernel paging request at ffff880045645000 [ 1165.943074] IP: [<ffff880045645000>] 0xffff880045645000 [ 1165.943108] PGD 2b2c067 PUD 80000000400001e3 [ 1165.943136] Oops: 0011 [#1] PREEMPT SMP [ 1165.943162] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec [ 1165.943620] nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio [ 1165.943846] CPU: 0 PID: 464 Comm: libvirtd Tainted: G W O 3.17.4-1-ARCH #1 [ 1165.943894] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014 [ 1165.943952] task: ffff88021acd4670 ti: ffff8802221e4000 task.ti: ffff8802221e4000 [ 1165.943998] RIP: 0010:[<ffff880045645000>] [<ffff880045645000>] 0xffff880045645000 [ 1165.944046] RSP: 0018:ffff8802221e7d70 EFLAGS: 00010286 [ 1165.944078] RAX: 0000000000000088 RBX: ffff880222b4d098 RCX: 0000000000000000 [ 1165.944121] RDX: ffff8802154dab88 RSI: ffff880222b4d098 RDI: ffff880222b4d098 [ 1165.944165] RBP: ffff8802221e7d90 R08: 0000000000000246 R09: ffffea00029e7c00 [ 1165.944208] R10: ffffffffa06c91b8 R11: 0000000000000000 R12: ffff880222b4d146 [ 1165.944251] R13: ffff880045645000 R14: 0000000000000000 R15: fffffffffffffff2 [ 1165.944295] FS: 00007f6c8235c780(0000) GS:ffff88022e200000(0000) knlGS:0000000000000000 [ 1165.944344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1165.944379] CR2: ffff880045645000 CR3: 000000021dcb2000 CR4: 00000000001407f0 [ 1165.944422] Stack: [ 1165.944434] ffffffff813b5772 ffff880222b4d098 ffff880222b4d098 0000000000000004 [ 1165.944480] ffff8802221e7dc0 ffffffff813b65c9 ffff880222b4d098 0000000000000004 [ 1165.944526] ffff880222b4d146 0000000000000246 ffff8802221e7df0 ffffffff813b6703 [ 1165.944573] Call Trace: [ 1165.944590] [<ffffffff813b5772>] ? __rpm_callback+0x32/0x70 [ 1165.944626] [<ffffffff813b65c9>] rpm_idle+0x259/0x340 [ 1165.944658] [<ffffffff813b6703>] __pm_runtime_idle+0x53/0x70 [ 1165.944695] [<ffffffff812e6fd4>] pci_device_remove+0x74/0xc0 [ 1165.944731] [<ffffffff813aca4f>] __device_release_driver+0x7f/0xf0 [ 1165.944770] [<ffffffff813acae3>] device_release_driver+0x23/0x30 [ 1165.944808] [<ffffffff813ab82d>] unbind_store+0xed/0x150 [ 1165.944842] [<ffffffff813aaaa5>] drv_attr_store+0x25/0x40 [ 1165.944877] [<ffffffff8123ec7a>] sysfs_kf_write+0x3a/0x50 [ 1165.944911] [<ffffffff8123e1ee>] kernfs_fop_write+0xee/0x180 [ 1165.944946] [<ffffffff811c7967>] vfs_write+0xb7/0x200 [ 1165.944978] [<ffffffff811c85d9>] SyS_write+0x59/0xd0 [ 1165.945010] [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b [ 1165.945047] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1165.945190] RIP [<ffff880045645000>] 0xffff880045645000 [ 1165.945224] RSP <ffff8802221e7d70> [ 1165.948044] CR2: ffff880045645000 [ 1165.969330] ---[ end trace 59b5489690ced9ea ]---

This seems a little bit like those VGA arbitration missing issues - except that I thought to avoid that problematic with using UEFI in guest? The host archlinux is also booted using UEFI, CSM is disabled in bios. (Booting using CSM makes no difference, though).
nouveau is blacklisted and the nvidia card set to pci stub. Bootparams:

intel_iommu=on pci-stub.ids=01:00.0,01:00.1

After VM shutdown, lspci hangs forever, libvirt crashed and a system shutdown is not possible as some process hangs...

Any idea where to dig into?

Thanks a lot!

Offline

#3495 2014-12-02 22:43:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MatthiasArch wrote:
intel_iommu=on pci-stub.ids=01:00.0,01:00.1

Nak, pci-stub.ids takes vendor and device IDs, not bus address.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3496 2014-12-03 05:50:12

kidchunks
Member
Registered: 2014-12-03
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been cracking at this for about a week (off and on) and get stuck with the same problem on multiple setups. Before I talk about my problem, I tested the card by installing windows 8.1 on the host and everything worked fine.

Hardware Specs:

Test 1:
Mobo: Supermicro X8DAH+-F
CPU: Xeon X5670
Video Card: Sapphire r9 290x Tri-X OC
OS: Debian Wheezy with latest kernel release from alex william and qemu-kvm from wheezy backports (v2.1)

Test 2:
Same hardware as above
OS: Ubuntu 14.04 LTS (everything stock)

Now, I've done all the necessary changes from post 1 (vfio binding, kernel command, stubbing and etc). I didn't do any patches given I don't think I need them.

Using the following to launch start the VM:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 8,sockets=1,cores=8,threads=1 \ -bios /usr/share/seabios/bios.bin -vga none -rtc base=localtime -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=85:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ -drive file=/root/win7.img,cache=none,if=virtio \ -cdrom /root/win7.iso \ -usb -usbdevice host:046d:c068 \ -no-reboot 

Devices:
85:00.0 <-- video card
85:00.1 <-- audio for card

When I start the VM, I can install the OS no problem through a monitor connected to the video card (no drivers installed). As soon as I install the drivers (any catalyst driver! I tested 13.12 through 14.11.2).

I get the following after the start screen (window logo):

vfio-pci 0000:85:00.0: irq 89 for MSI/MSI-X

the irq number changes each time

Immediately following that message..the VM process is killed. I tested both windows 7 and windows 8.1..same issue.

I'm not sure how to trouble this because dmesg doesn't provide anything other than the irq message. I've also tried searching this thread but seems no one has run into this edge case.

If anyone has any ideas, let me know! smile

EDIT: Updated error message

Last edited by kidchunks (2014-12-03 14:53:32)

Offline

#3497 2014-12-03 07:30:50

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

try without -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ ?
i don't have amd, but reading this thread i think you need to remove amd audio device

edit: and try 440fx chipset not q35

Last edited by slis (2014-12-03 10:57:11)

Offline

#3498 2014-12-03 09:27:52

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

try without -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ ?
i don't have amd, but reading this thread i think you need to remove amd audio device

edit: and try fx440 chipset not q35

It actually depends on card (or maybe mobo's ACPI?). My HD7870 works fine with HD Audio assigned as function 0x1 on 440FX.

Last edited by dwe11er (2014-12-03 09:29:11)

Offline

#3499 2014-12-03 09:41:04

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
4kGamer wrote:

Hi, I  wanna buy a new CPU and motherboard.

Sounds like you're already committed to a pricey upgrade, but FWIW I decided to go a more mild route upgrading from the i5-3470T.  I found my motherboard is compatible with a Xeon E3-1245v2, found a reasonable deal for one on ebay, and now I've got 2x the cores and a 500MHz boost.  Including a Cooler Master Hyper 212 Plus, I'm in it for well below the price of an i7-4790k alone, and it seems to have enough juice to run Steam in-home-streaming with the GTX750 (4-cores for the VM, leaving 4 threads for the host).

You do know that xeon lacks integrated video core.
He may be needing it to get host output. However, sometimes buying a cheap-as-cardboard GT610 can be more profitable than buying i7 for integrated video.

Whoa, just noted that this post is #3499.

Last edited by Duelist (2014-12-03 12:46:49)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3500 2014-12-03 13:40:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You do know that xeon lacks integrated video core.
He may be needing it to get host output. However, sometimes buying a cheap-as-cardboard GT610 can be more profitable than buying i7 for integrated video.

Whoa, just noted that this post is #3499.

Check ark, E3-1245v2 has HD P4000 video.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3501 2014-12-03 16:19:19

kidchunks
Member
Registered: 2014-12-03
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

try without -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ ?
i don't have amd, but reading this thread i think you need to remove amd audio device

edit: and try 440fx chipset not q35


I removed the audio device:

qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host \ -smp 8,sockets=1,cores=4,threads=4 \ -bios /usr/share/seabios/bios.bin -vga none \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=85:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/root/qemu/win7.img,cache=none,if=virtio \ -usb -usbdevice host:046d:c068 \ -vnc :1 \ -usbdevice tablet \ -daemonize \ -no-reboot

Still the same error:

vfio-pci 0000:85:00.0: irq 89 for MSI/MSI-X

I tried with pc (440fx) and got the same error.

qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host \ -smp 8,sockets=1,cores=4,threads=4 \ -bios /usr/share/seabios/bios.bin -vga none \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=85:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/root/qemu/win7.img,cache=none,if=virtio \ -usb -usbdevice host:046d:c068 \ -vnc :1 \ -usbdevice tablet \ -daemonize \ -no-reboot

Here is pc with pci-assign (no vfio):
qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host \
-smp 8,sockets=1,cores=4,threads=4 \
-bios /usr/share/seabios/bios.bin -vga none \
-device pci-assign,host=85:00.0 \
-drive file=/root/qemu/win7.img,cache=none,if=virtio \
-usb -usbdevice host:046d:c068 \
-vnc :1 \
-usbdevice tablet \
-daemonize \
-no-reboot

Same error but this pci-stub throwing it.

pci-stub 0000:85:00.0: irq 89 for MSI/MSI-X

Last edited by kidchunks (2014-12-03 16:20:23)

Offline

#3502 2014-12-03 16:26:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kidchunks wrote:
slis wrote:

try without -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ ?
i don't have amd, but reading this thread i think you need to remove amd audio device

edit: and try 440fx chipset not q35


I removed the audio device:

qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host \ -smp 8,sockets=1,cores=4,threads=4 \ -bios /usr/share/seabios/bios.bin -vga none \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=85:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/root/qemu/win7.img,cache=none,if=virtio \ -usb -usbdevice host:046d:c068 \ -vnc :1 \ -usbdevice tablet \ -daemonize \ -no-reboot

Still the same error:

vfio-pci 0000:85:00.0: irq 89 for MSI/MSI-X

This is not an error.  Not everything the kernel prints is an error.  This is simply indicating that the driver vfio-pci allocated irq 89 for the device for MSI/X.  That's it.

I tried with pc (440fx) and got the same error.

qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host \ -smp 8,sockets=1,cores=4,threads=4 \ -bios /usr/share/seabios/bios.bin -vga none \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=85:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/root/qemu/win7.img,cache=none,if=virtio \ -usb -usbdevice host:046d:c068 \ -vnc :1 \ -usbdevice tablet \ -daemonize \ -no-reboot

ioh3402 should not be mixed with 440FX, remove that line, remove the bus= on the vfio-pci line, and set addr to something like 9.0 since 0.0 won't be available on the host bridge.

Here is pc with pci-assign (no vfio):
qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host \
-smp 8,sockets=1,cores=4,threads=4 \
-bios /usr/share/seabios/bios.bin -vga none \
-device pci-assign,host=85:00.0 \
-drive file=/root/qemu/win7.img,cache=none,if=virtio \
-usb -usbdevice host:046d:c068 \
-vnc :1 \
-usbdevice tablet \
-daemonize \
-no-reboot

Same error but this pci-stub throwing it.

pci-stub 0000:85:00.0: irq 89 for MSI/MSI-X

Again, not an error.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3503 2014-12-03 20:41:49

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

You do know that xeon lacks integrated video core.
He may be needing it to get host output. However, sometimes buying a cheap-as-cardboard GT610 can be more profitable than buying i7 for integrated video.

Whoa, just noted that this post is #3499.

Check ark, E3-1245v2 has HD P4000 video.

Whoa! That was a surprise. There's only four versions in whole E3 v2 line-up, 1225, 1245 and 1275. And one with 2000 graphics.
Well, E3-1240v2 is a more popular choice, and it lacks graphics. Attention must be paid when choosing the CPU, as it more likely will not have it.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3504 2014-12-04 00:45:27

tatsu
Member
Registered: 2014-12-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I'd like to know how to apply the kernel patches. the two links simply lead to a forum archive with text posts containing what seems to be some kernel files.

I have no idea how to use this. some help please?

Offline

#3505 2014-12-04 07:33:18

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tatsu wrote:

Hi all!

I'd like to know how to apply the kernel patches. the two links simply lead to a forum archive with text posts containing what seems to be some kernel files.

I have no idea how to use this. some help please?

You don't need those; just the first file which is a .tar.gz file
Download and extract it a folder and go to that folder and run these lines:
sudo pacman -S base-devel
makepkg -s

After the kernel compile is finnished few .pkg.tar.xz files will be created. save them if you want for next time.
Install them by running:
sudo pacman -U *.pkg.tar.xz

Offline

#3506 2014-12-04 08:07:06

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The kvm version of intel vgt is out:
http://www.phoronix.com/vr.php?view=MTg1MzQ

Offline

#3507 2014-12-04 09:50:01

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:

The kvm version of intel vgt is out:
http://www.phoronix.com/vr.php?view=MTg1MzQ

Good news.

Now only we need aw or someone smart enough to make it use each output for each vm  (probably not possible smile)

Offline

#3508 2014-12-04 11:37:40

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
winie wrote:

The kvm version of intel vgt is out:
http://www.phoronix.com/vr.php?view=MTg1MzQ

Good news.

Now only we need aw or someone smart enough to make it use each output for each vm  (probably not possible smile)

Actually that's what they're doing. Each vm gets a virtual monitor which can be connected to a physical output or not.

Offline

#3509 2014-12-04 11:38:55

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm... i tried xen, and i couldn't get 2 vms at same time to output to different outputs at same time. Am i missing something?

Imagine if we could output 3 windows vm's to 3 different outputs at same time, so we could have 3 separate vm's with 3d acceleration on single host with igpu...

Last edited by slis (2014-12-04 11:47:06)

Offline

#3510 2014-12-04 12:14:20

tatsu
Member
Registered: 2014-12-03
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:

The kvm version of intel vgt is out:
http://www.phoronix.com/vr.php?view=MTg1MzQ

OMG wow thanks for sharing the love, man!

winie wrote:

You don't need those; just the first file which is a .tar.gz file
Download and extract it a folder and go to that folder and run these lines:
sudo pacman -S base-devel
makepkg -s
After the kernel compile is finnished few .pkg.tar.xz files will be created. save them if you want for next time.
Install them by running:
sudo pacman -U *.pkg.tar.xz

I installed a newer kernel? Do I really have to use his?

Offline

#3511 2014-12-04 14:04:30

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tatsu wrote:

I installed a newer kernel? Do I really have to use his?

If using ovmf then probably not.
The patches on the first post are for vga arbitration, acs override and radeon related fixes.
Booting vm with ovmf  in efi mode solves the first one. 
acs isn't an issue if you have only one pcie graphic card or you passthrough all to the same vm

Offline

#3512 2014-12-04 14:36:27

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:
slis wrote:
winie wrote:

The kvm version of intel vgt is out:
http://www.phoronix.com/vr.php?view=MTg1MzQ

Good news.

Now only we need aw or someone smart enough to make it use each output for each vm  (probably not possible smile)

Actually that's what they're doing. Each vm gets a virtual monitor which can be connected to a physical output or not.

Whoa, this is cool. I was just wondering exactly how this sharing of a GPU between different VMs would work, as there is only 1 physical screen attached. Being able to use the different outputs for different VMs seems so impossible smile Do you have any sources/pointers to more info on this topic?

Offline

#3513 2014-12-04 17:55:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Do you have any sources/pointers to more info on this topic?

http://lists.freedesktop.org/archives/i … 56652.html
Check out the ending - there is links to the presentations and there's links for sources.
Sadly, i am unable to check out these sources myself, so i can't help you anymore further.


whole github is banned in my country for awesome reasons, and i haven't got a VPN up yet


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3514 2014-12-04 19:30:01

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone succeeded to use qemu + vfio without sudo/root? I tried to chown /dev/vfio/1 and /dev/vfio/vfio and increased ulimit to 9000000 (8GB dedicated to qemu + ~1GB for safety) but I'm gettings such errors:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3b7ad1ce60, 0x0, 0x80000000, 0x7f3944000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3b7ad1ce60, 0x100000000, 0x180000000, 0x7f39c4000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container

I don't use libvirt btw, just qemu.


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#3515 2014-12-04 19:36:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zsph wrote:

Has anyone succeeded to use qemu + vfio without sudo/root? I tried to chown /dev/vfio/1 and /dev/vfio/vfio and increased ulimit to 9000000 (8GB dedicated to qemu + ~1GB for safety) but I'm gettings such errors:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3b7ad1ce60, 0x0, 0x80000000, 0x7f3944000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3b7ad1ce60, 0x100000000, 0x180000000, 0x7f39c4000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container

I don't use libvirt btw, just qemu.

How are you setting ulimit -l?  It's rather tricky


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3516 2014-12-04 20:03:21

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

How are you setting ulimit -l?  It's rather tricky

First I calculated it as (8 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1024 + 20) 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1024 = 8409088, but I was getting same errors. So I though that extra 20mb was not enough and went with 9000000.

I do it as

sudo -s ulimit -l 9000000

Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#3517 2014-12-04 20:14:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zsph wrote:
aw wrote:

How are you setting ulimit -l?  It's rather tricky

First I calculated it as (8 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1024 + 20) 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 1024 = 8409088, but I was getting same errors. So I though that extra 20mb was not enough and went with 9000000.

I do it as

sudo -s ulimit -l 9000000

And then what?  At this point you either run qemu directly, which means your running it as root and the ulimit was unnecessary, you ^d back to yourself and you've lost the ulimit, or you su back to yourself.  The last option is the only one that keeps the ulimit.  If you want to make a user that has elevated limits then look at /etc/security/limits.conf, or however your distro does it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3518 2014-12-04 21:49:06

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did someone try getting Intels GVT-g / KVMGT to work with KVM yet?
Heard it's ready now after already working in Xen for a while now.

Offline

#3519 2014-12-05 12:14:01

zsph
Member
Registered: 2014-09-04
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

And then what?

That was my mistake perhaps, I was doing ^d. But then I also tried su - <myusername> and still received memory allocation error.

aw wrote:

If you want to make a user that has elevated limits then look at /etc/security/limits.conf, or however your distro does it.

I will look, thanks.


Asus P8Z77-V LK, Core i7 3770, Asus NVIDIA GTX 770 DirectCU II OC, Debian x64, EFI, Kernel 3.16 + i915 arbiter patch.
3DMark11: bare metal P10231 vs vfio P10039: 98% of performance!

Offline

#3520 2014-12-05 14:16:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zsph wrote:
aw wrote:

And then what?

That was my mistake perhaps, I was doing ^d. But then I also tried su - <myusername> and still received memory allocation error.

That should have worked if you set the limit high enough:

$ sudo -s # ulimit -l 1000 # ulimit -l 1000 # su - alwillia $ ulimit -l 1000

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3521 2014-12-06 03:44:29

kidchunks
Member
Registered: 2014-12-03
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This is not an error.  Not everything the kernel prints is an error.  This is simply indicating that the driver vfio-pci allocated irq 89 for the device for MSI/X.  That's it.

Thanks for the clarification regarding irq!

aw wrote:

ioh3402 should not be mixed with 440FX, remove that line, remove the bus= on the vfio-pci line, and set addr to something like 9.0 since 0.0 won't be available on the host bridge.

Removed ioh3402 and changed addr card:

qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu host \ -smp 8,sockets=1,cores=4,threads=4 \ -bios /usr/share/seabios/bios.bin -vga none \ -device vfio-pci,host=85:00.0,addr=08.0,multifunction=on,x-vga=on \ -device vfio-pci,host=85:00.0,addr=08.1 \ -drive file=/root/qemu/win7.img,cache=none,if=virtio \ -drive file=/root/qemu/win7.iso,index=0,media=cdrom \ -drive file=/root/qemu/virtio-win-0.1-81.iso,index=1,media=cdrom \ -usb -usbdevice host:046d:c068 \ -vnc :1 \ -usbdevice tablet \ -daemonize

I was able to install the drivers but upon reboot, got stuck in a boot loop.

Figured it could be driver related so I tried the latest catalyst beta drivers (14.11.2) and it worked! Audio and video are passing through with no problems. smile

The above also fixed an issue I had with video card freezing (stuck image) on reboot/shutdown so no need for no-reboot.

The only problem I have now is that I can't see the bios(seabios) screen but that's minor given it still boots into whatever drive is first. Strangely enough, OVMF(UEFI) works fine. Maybe something with the card.

In any case, glad to get it working after a week!

Thanks AW and others for the help.

To those running an r9 290x...use the 14.11.2 drivers!

Last edited by kidchunks (2014-12-06 03:49:43)

Offline

#3522 2014-12-06 19:05:17

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally succeeded to passthrough my Zotac GT630 to OSX 10.9.5 with working 3D
https://imagizer.imageshack.us/v2/1152x … u1Qwht.png

Offline

#3523 2014-12-07 07:05:05

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mastersplinter777 wrote:

Finally succeeded to passthrough my Zotac GT630 to OSX 10.9.5 with working 3D
https://imagizer.imageshack.us/v2/1152x … u1Qwht.png

I didn't get os x 10.9.5 working in qemu. I get a fast bootloop from seabios.
Can you post your command line?

Offline

#3524 2014-12-07 07:18:45

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:
mastersplinter777 wrote:

Finally succeeded to passthrough my Zotac GT630 to OSX 10.9.5 with working 3D
https://imagizer.imageshack.us/v2/1152x … u1Qwht.png

I didn't get os x 10.9.5 working in qemu. I get a fast bootloop from seabios.
Can you post your command line?

I second this -- also, has anyone written an Ubuntu tutorial or pre-compiled everything for 14.04?

Offline

#3525 2014-12-07 07:22:52

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so i switched out my gtx 580 that was  running via bios and vga which was running fine to a radeon r9 290x in hopes of being able to get it set up using ovmf. I can get it to output a display by using

sudo vfio-bind 0000:01:00.0 0000:01:00.1 #0000:00:03.0 qemu-system-x86_64 -enable-kvm -M q35 -m 8196 -mem-path /dev/hugepages -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -boot menu=on \ -bios /usr/share/qemu/bios.bin -vga none\ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -nographic \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \

on vga and seabios but on ovmf it won't work. it shows no output. i accidentially left the command running over night and it appears it output "3h" every once and a while after many many lines. this is the command i was running to try to get it running with ovmf

sudo vfio-bind 0000:01:00.0 0000:01:00.1 #0000:00:03.0 qemu-system-x86_64 -enable-kvm -m 8196 -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -boot menu=on \ -drive if=pflash,format=raw,readonly,file=/home/anonymous/kvm/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/home/anonymous/kvm/OVMF_VARS.fd \ -device vfio-pci,host=01:00.0 -nographic -vga none \ -device vfio-pci,host=01:00.1 \

i tried running aw's rom parser but it would output a 0 byte file even on a fresh restart. does that mean my card doesnt support efi?

when i ran with "multifunction=on,x-vga=on" it would show uefi interactive shell inside of my terminal but still wouldnt show anyhting on my graphics card screen.

UEFI Interactive Shell v2.1 EDK II UEFI v2.40 (EDK II, 0x00010000) Mapping table BLK2: Alias(s): PciRoot(0x0)/Pci(0x1,0x1)/Ata(0x0) BLK0: Alias(s): PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x0) BLK1: Alias(s): PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x1) Press ESC in 1 seconds to skip startup.nsh or any other key to continue. Shell> 

Last edited by risho (2014-12-07 07:31:00)

Offline

#3526 2014-12-07 07:43:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

i tried running aw's rom parser but it would output a 0 byte file even on a fresh restart. does that mean my card doesnt support efi?

romparser software needs a file to check. But you're not passing romfile=SOMEFILE.rom option to your device, so you don't need to check, and also you have a very fresh radeon card, which 95% supports UEFI.


Also, try installing windowns(or whatever you need) using the qemu's graphics(like qxl or something), and then plugging in the card as a secondary.
By the way, you shouldn't need x-vga=on while using OVMF. Appending multifunction=on is needed when there is HDMI-audio device present as a function of the GPU itself, so you should always leave it on.

And you appear to be giving us only partial command-lines, and running qemu without sudo(you might have fixes for that, but usually they are needed).


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3527 2014-12-07 07:50:47

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist, dozens of pages ago you posted a message on successfully installing windows 7 using ovmf

could you please specify all related details (qemu/libvirt configuration, win 7 iso "type", steps taken, etc)?

already had 4 "try sessions" with no result

Last edited by sinny (2014-12-07 07:52:27)

Offline

#3528 2014-12-07 07:51:33

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yea i've removed all drives currently i'm just trying to get it to output to my graphics card's monitor. i'll reattatch drives once i get that working.

and yea i'm running that as a script that i'm executing using sudo.

i had windows up and running perfectly using my old card and bios, but i can't figure out what is missing to get it to output using ovmf and my new card.

would i not see some sort of output on the screen from the uefi/ovmf running like i would in seabios?

Last edited by risho (2014-12-07 07:55:13)

Offline

#3529 2014-12-07 09:36:30

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:
mastersplinter777 wrote:

Finally succeeded to passthrough my Zotac GT630 to OSX 10.9.5 with working 3D
https://imagizer.imageshack.us/v2/1152x … u1Qwht.png

I didn't get os x 10.9.5 working in qemu. I get a fast bootloop from seabios.
Can you post your command line?

This is my launch script:

#/bin/sh #sleep 10 export QEMU_AUDIO_DRV=alsa #export QEMU_AUDIO_TIMER_PERIOD=100 export QEMU_AUDIO_DAC_FIXED_SETTINGS=1 export QEMU_AUDIO_DAC_FIXED_FREQ=48000 #export QEMU_AUDIO_DAC_FIXED_FMT=S16 #export QEMU_AUDIO_DAC_FIXED_CHANNELS=2 #export QEMU_AUDIO_DAC_VOICES=1 #export QEMU_AUDIO_DAC_TRY_POLL=0 export QEMU_AUDIO_ADC_FIXED_SETTINGS=1 export QEMU_AUDIO_ACD_FIXED_FREQ=48000 #export QEMU_AUDIO_ADC_FIXED_FMT=S16 #export QEMU_AUDIO_ADC_FIXED_CHANNELS=1 #export QEMU_AUDIO_ADC_VOICES=1 #export QEMU_AUDIO_ADC_TRY_POLL=0 export QEMU_ALSA_DAC_PERIOD_SIZE=170 export QEMU_ALSA_DAC_BUFFER_SIZE=512 #export QEMU_PA_SAMPLES=480 #<qemu:env name="QEMU_AUDIO_DRV" value="pa"></qemu:env> #<qemu:env name="QEMU_AUDIO_TIMER_PERIOD" value="100"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_SETTINGS" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_FREQ" value="48000"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_FMT" value="S16"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_CHANNELS" value="2"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_VOICES" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_TRY_POLL" value="0"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_FIXED_SETTINGS" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_ACD_FIXED_FREQ" value="48000"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_FIXED_FMT" value="S16"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_FIXED_CHANNELS" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_VOICES" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_TRY_POLL" value="0"></qemu:env> #<qemu:env name="QEMU_PA_SAMPLES" value="480"></qemu:env> #export QEMU_AUDIO_DRV=pa #export QEMU_PA_SAMPLES=128 #export QEMU_AUDIO_DRV=alsa #export QEMU_ALSA_DAC_PERIOD_SIZE=170 #export QEMU_ALSA_DAC_BUFFER_SIZE=512 spicec --host 127.0.0.1 --port 5902 --title="OSX" --hotkeys release-cursor=ctrl+alt & #spicy --host 127.0.0.1 --port 5902 --title="OSX" & sudo qemu-system-x86_64 \ -enable-kvm \ -smp 2,sockets=1,cores=2,threads=1 \ -chardev spicevmc,id=vdagent,name=vdagent \ -spice port=5902,disable-ticketing,seamless-migration=on \ -M q35 \ -m 4096 \ -cpu core2duo \ -rtc base=localtime \ -vga none \ -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x12 \ -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \ -device virtio-serial,addr=0x8 \ -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \ -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x9 \ -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x1.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x1 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.2,addr=0x1.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.2,addr=0x1.0x2 \ -chardev spicevmc,name=usbredir,id=usbredirchardev1 \ -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \ -chardev spicevmc,name=usbredir,id=usbredirchardev2 \ -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \ -chardev spicevmc,name=usbredir,id=usbredirchardev3 \ -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \ -chardev spicevmc,name=usbredir,id=usbredirchardev4 \ -device usb-redir,chardev=usbredirchardev4,id=usbredirdev4 \ -net nic,macaddr=00:00:00:FF:FF:02,model=virtio,id=netkvm,addr=0x7 \ -usbdevice host:046d:c52e \ -net bridge \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file='/vm/QEMU/OSX/osx.img',if=none,media=disk,id=drive-sata0-0-2,format=raw -device ide-hd,bus=ide.1,drive=drive-sata0-0-2,id=sata0-0-2

This is my boot options:

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>EthernetBuiltIn</key> <string>Yes</string> <key>GUI</key> <string>No</string> <key>Graphics Mode</key> <string>1280x800x32</string> <key>HDAEnabler</key> <string>Yes</string> <key>Legacy Logo</key> <string>Yes</string> <key>Timeout</key> <string>5</string> <key>USBBusFix</key> <string>Yes</string> <key>UseKernelCache</key> <string>No</string></dict> </plist>

smbios profile: MacPro3,1

Qemu version: 2.1.2
Kernel: 3.17.4-99-libre with applied patches from this topic
HW: ASRock Z87 Pro4 with latest bios, 16 Gb RAM, Intel Core i5 4440, 1GPU: Intel HD 4600 (host), 2GPU: NVIDIA GT630 1Gb (Zotac) (osx), 3GPU: AMD 5450 (ASUS) (win8.1)

Offline

#3530 2014-12-07 10:50:41

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mastersplinter777 wrote:
winie wrote:
mastersplinter777 wrote:

Finally succeeded to passthrough my Zotac GT630 to OSX 10.9.5 with working 3D
https://imagizer.imageshack.us/v2/1152x … u1Qwht.png

I didn't get os x 10.9.5 working in qemu. I get a fast bootloop from seabios.
Can you post your command line?

This is my launch script:

#/bin/sh #sleep 10 export QEMU_AUDIO_DRV=alsa #export QEMU_AUDIO_TIMER_PERIOD=100 export QEMU_AUDIO_DAC_FIXED_SETTINGS=1 export QEMU_AUDIO_DAC_FIXED_FREQ=48000 #export QEMU_AUDIO_DAC_FIXED_FMT=S16 #export QEMU_AUDIO_DAC_FIXED_CHANNELS=2 #export QEMU_AUDIO_DAC_VOICES=1 #export QEMU_AUDIO_DAC_TRY_POLL=0 export QEMU_AUDIO_ADC_FIXED_SETTINGS=1 export QEMU_AUDIO_ACD_FIXED_FREQ=48000 #export QEMU_AUDIO_ADC_FIXED_FMT=S16 #export QEMU_AUDIO_ADC_FIXED_CHANNELS=1 #export QEMU_AUDIO_ADC_VOICES=1 #export QEMU_AUDIO_ADC_TRY_POLL=0 export QEMU_ALSA_DAC_PERIOD_SIZE=170 export QEMU_ALSA_DAC_BUFFER_SIZE=512 #export QEMU_PA_SAMPLES=480 #<qemu:env name="QEMU_AUDIO_DRV" value="pa"></qemu:env> #<qemu:env name="QEMU_AUDIO_TIMER_PERIOD" value="100"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_SETTINGS" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_FREQ" value="48000"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_FMT" value="S16"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_FIXED_CHANNELS" value="2"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_VOICES" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_DAC_TRY_POLL" value="0"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_FIXED_SETTINGS" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_ACD_FIXED_FREQ" value="48000"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_FIXED_FMT" value="S16"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_FIXED_CHANNELS" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_VOICES" value="1"></qemu:env> #<qemu:env name="QEMU_AUDIO_ADC_TRY_POLL" value="0"></qemu:env> #<qemu:env name="QEMU_PA_SAMPLES" value="480"></qemu:env> #export QEMU_AUDIO_DRV=pa #export QEMU_PA_SAMPLES=128 #export QEMU_AUDIO_DRV=alsa #export QEMU_ALSA_DAC_PERIOD_SIZE=170 #export QEMU_ALSA_DAC_BUFFER_SIZE=512 spicec --host 127.0.0.1 --port 5902 --title="OSX" --hotkeys release-cursor=ctrl+alt & #spicy --host 127.0.0.1 --port 5902 --title="OSX" & sudo qemu-system-x86_64 \ -enable-kvm \ -smp 2,sockets=1,cores=2,threads=1 \ -chardev spicevmc,id=vdagent,name=vdagent \ -spice port=5902,disable-ticketing,seamless-migration=on \ -M q35 \ -m 4096 \ -cpu core2duo \ -rtc base=localtime \ -vga none \ -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x12 \ -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \ -device virtio-serial,addr=0x8 \ -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \ -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x9 \ -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x1.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x1 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.2,addr=0x1.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.2,addr=0x1.0x2 \ -chardev spicevmc,name=usbredir,id=usbredirchardev1 \ -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \ -chardev spicevmc,name=usbredir,id=usbredirchardev2 \ -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \ -chardev spicevmc,name=usbredir,id=usbredirchardev3 \ -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \ -chardev spicevmc,name=usbredir,id=usbredirchardev4 \ -device usb-redir,chardev=usbredirchardev4,id=usbredirdev4 \ -net nic,macaddr=00:00:00:FF:FF:02,model=virtio,id=netkvm,addr=0x7 \ -usbdevice host:046d:c52e \ -net bridge \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file='/vm/QEMU/OSX/osx.img',if=none,media=disk,id=drive-sata0-0-2,format=raw -device ide-hd,bus=ide.1,drive=drive-sata0-0-2,id=sata0-0-2

This is my boot options:

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>EthernetBuiltIn</key> <string>Yes</string> <key>GUI</key> <string>No</string> <key>Graphics Mode</key> <string>1280x800x32</string> <key>HDAEnabler</key> <string>Yes</string> <key>Legacy Logo</key> <string>Yes</string> <key>Timeout</key> <string>5</string> <key>USBBusFix</key> <string>Yes</string> <key>UseKernelCache</key> <string>No</string></dict> </plist>

smbios profile: MacPro3,1

Qemu version: 2.1.2
Kernel: 3.17.4-99-libre with applied patches from this topic
HW: ASRock Z87 Pro4 with latest bios, 16 Gb RAM, Intel Core i5 4440, 1GPU: Intel HD 4600 (host), 2GPU: NVIDIA GT630 1Gb (Zotac) (osx), 3GPU: AMD 5450 (ASUS) (win8.1)

UPDATE:
change:
-net nic,macaddr=00:00:00:FF:FF:02,model=virtio,id=netkvm,addr=0x7 \
-net bridge \

to:
-netdev bridge,id=br0 \
-device e1000-82545em,netdev=br0,id=vnet0,bus=pcie.0,addr=15 \

as it solves os freeze problems while copying large files

Offline

#3531 2014-12-07 14:00:07

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

Duelist, dozens of pages ago you posted a message on successfully installing windows 7 using ovmf

could you please specify all related details (qemu/libvirt configuration, win 7 iso "type", steps taken, etc)?

already had 4 "try sessions" with no result

https://bbs.archlinux.org/viewtopic.php … 9#p1480029

Since i've got strange problems with passing the bios via romfile option to my GPU(the UEFI freezes!), i reverted to BIOS and i dropped any thoughts about running OVMF with VFIO, so i've done setting it up just with qemu's emulated graphics.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3532 2014-12-07 16:17:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://www.extremetech.com/gaming/15646 … d-analyzed

According to Microsoft, the Xbox One runs three different operating systems. There’s a core operating system that’s based on Microsoft’s Hyper-V hypervisor technology, which boots up when you turn the console on. This hypervisor then boots up two further operating systems: Xbox OS, which runs the games, and an OS that’s based on Windows 8, which runs the apps (Skype, TV, etc.)

The Xbox OS and Windows-based OS run side-by-side, on hardware that’s virtualized by the hypervisor. Both OSes are permanently switched on and constantly rendering their video output, to enable instant switching/snapping between games and apps/TV. The Xbox OS is rebooted whenever you load a game, but the Windows-based OS is persistent until you turn the console off. It isn’t clear how the hardware resources are split between the two virtualized OSes, but hopefully the Xbox OS (games) gets most of the RAM and GPU time.

WUT.
So it seems like hyper-v can do not only the GPU-passthrough, but divide it's performance! Isn't that VERY MUCH alike intel gvt-g but on AMD SoC?
Surprising fact. Now i can boast that my machine runs very alike to how xbox one runs - three OS at the same time.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3533 2014-12-07 16:51:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
risho wrote:

i tried running aw's rom parser but it would output a 0 byte file even on a fresh restart. does that mean my card doesnt support efi?

romparser software needs a file to check. But you're not passing romfile=SOMEFILE.rom option to your device, so you don't need to check, and also you have a very fresh radeon card, which 95% supports UEFI.

Some Radeon cards have been reported to have a switch which enables an EFI capable ROM.  Failure to dump the ROM from sysfs can sometimes be attributed to the card being the primary host graphics.  Is vfio able to get the ROM or do you get an invalid ROM message in dmesg on guest startup?  You may want to check the techpowerup database for an alternate ROM.

Also, try installing windowns(or whatever you need) using the qemu's graphics(like qxl or something), and then plugging in the card as a secondary.
By the way, you shouldn't need x-vga=on while using OVMF. Appending multifunction=on is needed when there is HDMI-audio device present as a function of the GPU itself, so you should always leave it on.

multifunction=on changes how the device is exposed to the guest, it has nothing to do with the physical device.  In fact, you can combine functions from multiple physical and emulated devices into a single franken-multifunction-device if you want, though I wouldn't recommend it.  If you're not creating multiple functions on the slot, as exposed to the guest, there's no point in using multifunction=on, but doing so gratuitously is generally harmless.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3534 2014-12-07 18:53:19

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
risho wrote:

i tried running aw's rom parser but it would output a 0 byte file even on a fresh restart. does that mean my card doesnt support efi?

romparser software needs a file to check. But you're not passing romfile=SOMEFILE.rom option to your device, so you don't need to check, and also you have a very fresh radeon card, which 95% supports UEFI.

Some Radeon cards have been reported to have a switch which enables an EFI capable ROM.  Failure to dump the ROM from sysfs can sometimes be attributed to the card being the primary host graphics.  Is vfio able to get the ROM or do you get an invalid ROM message in dmesg on guest startup?  You may want to check the techpowerup database for an alternate ROM.

Also, try installing windowns(or whatever you need) using the qemu's graphics(like qxl or something), and then plugging in the card as a secondary.
By the way, you shouldn't need x-vga=on while using OVMF. Appending multifunction=on is needed when there is HDMI-audio device present as a function of the GPU itself, so you should always leave it on.

multifunction=on changes how the device is exposed to the guest, it has nothing to do with the physical device.  In fact, you can combine functions from multiple physical and emulated devices into a single franken-multifunction-device if you want, though I wouldn't recommend it.  If you're not creating multiple functions on the slot, as exposed to the guest, there's no point in using multifunction=on, but doing so gratuitously is generally harmless.

First - he has a problem, not me. We don't know what he used as an input to romparser. Is it possible to feed that sysfs image of rom into romparser without catting it to some file first?
Also, i haven't been able to get the rom out of my secondary card, not the primary one, but that's me. He has a single GPU(r9 290x is single, right? r9 295x2 is the dual one, right?).

Second - i think if the HDMI audio device is exposed to the guest as an independent device, the driver installer software MIGHT go nuts.
I've noted that multifunction option just in case he thinks that changing it would do anything closely related to OVMF or UEFI, which isn't true.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3535 2014-12-07 21:33:42

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it's a sapphire tri x oc 290x. i checked newegg, the exact one i bought and it said it has uefi support, and then i checked sapphire's website and it also said it supports uefi, so i'm assuming it's fine.

i have it pci stubbed in my bootloader.

[anonymous@arch kvm]$ dmesg | grep stub [ 0.000000] Command line: BOOT_IMAGE=../vmlinuz-linux-mainline root=/dev/sdb1 rootflags=subvol=__active/rootvol intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=1002:67b0,1002:aac8 modeset.nouveau=0 modeset.radeon=0 rw initrd=../initramfs-linux-mainline.img [ 0.000000] Kernel command line: BOOT_IMAGE=../vmlinuz-linux-mainline root=/dev/sdb1 rootflags=subvol=__active/rootvol intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=1002:67b0,1002:aac8 modeset.nouveau=0 modeset.radeon=0 rw initrd=../initramfs-linux-mainline.img [ 0.606900] pci-stub: add 1002:67B0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.606905] pci-stub 0000:01:00.0: claimed by stub [ 0.606908] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.606920] pci-stub 0000:01:00.1: claimed by stub

i'm running the kernel + patches from page 1. i'm running the script with sudo and it gives me this inside the shell with no information being pushed through to the monitor plugged into the card

UEFI Interactive Shell v2.1 EDK II UEFI v2.40 (EDK II, 0x00010000) Mapping table BLK2: Alias(s): PciRoot(0x0)/Pci(0x1,0x1)/Ata(0x0) BLK0: Alias(s): PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x0) BLK1: Alias(s): PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x1) Press ESC in 1 seconds to skip startup.nsh or any other key to continue. Shell> 

this is the script

test3.sh sudo vfio-bind 0000:01:00.0 0000:01:00.1 #0000:00:03.0 #qemu-system-x86_64 -enable-kvm -m 1024 -cpu host \ #-smp 6,sockets=1,cores=2,threads=1 \ #-bios /usr/share/qemu/bios.bin -vga none \ #-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ ##-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ #-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 qemu-system-x86_64 -enable-kvm -m 8196 -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -boot menu=on \ -drive if=pflash,format=raw,readonly,file=/home/anonymous/kvm/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/home/anonymous/kvm/OVMF_VARS.fd \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -nographic -vga none \ -device vfio-pci,host=01:00.1 \ # -drive file=/home/anonymous/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ # -drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ # -usb -usbdevice host:046d:c52e -usbdevice host:041e:30e0 -usbdevice host:045e:0719 \ # -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ # -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 # -device vfio-pci,host=07:00.0,bus=pcie.0 \ # -device vfio-pci,host=08:04.0,bus=pcie.0 # -device vfio-pci,host=0c:00.0,bus=pcie.0 # -drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \ # -boot menu=on \ # -rtc base=localtime \

a lot of comments from my trial and error. i've removed all drives and everything that isn't absolutely necessary. I'm just trying to get a respone out of my monitor to show that it is working before i go through and try to set up drives and installing the os.

by the way i was able to get it to output when i was running it wish seabios and vga i just want to get it working with uefi since i can't update windows since restarts cause the guest to crash and also have a very high chance of hanging the host.

when i run the the script dmesg outputs

[ 64.293733] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 64.293841] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 64.293845] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

Offline

#3536 2014-12-08 00:26:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

when i run the the script dmesg outputs

[ 64.293733] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 64.293841] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 64.293845] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

That is perfectly normal, no need to be worried by these messages.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3537 2014-12-08 08:21:43

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

do you know any way for me to troubleshoot why i'm not getting any output on my monitor?

Offline

#3538 2014-12-08 09:53:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

do you know any way for me to troubleshoot why i'm not getting any output on my monitor?

First i'd try to boot windows installation iso from the serial-console of the OVMF menu(hint: type exit to get into somewhat menu-based UI)

Also, i'd install windowns with emulated graphics first, and then tried to attach the GPU. It would be working like secondary, but then, if all goes well, i'd remove the emulated graphics.

Something hints me that you shouldn't even have the output from OVMF in the first place since there is no VGA. But on the other hand, there is graphics-output-protocol or something, i don't know much about it. And i've got output on my emulated graphics when i booted the windows ISO.

But remember, i'm dumb and somewhat crazy.

Also, if you had it working with conventional VGA BIOS, why not just leave it working so?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3539 2014-12-08 09:58:35

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

risho wrote:

do you know any way for me to troubleshoot why i'm not getting any output on my monitor?

It might be OVMF itself (for me it worked with older releases, stopped for few weeks and now its working again; it was only during early bootup, Windows's working fine when driver kicks it). Which version are you using?

Last edited by dwe11er (2014-12-08 10:15:31)

Offline

#3540 2014-12-08 15:59:07

risho
Member
Registered: 2011-09-06
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
risho wrote:

do you know any way for me to troubleshoot why i'm not getting any output on my monitor?

First i'd try to boot windows installation iso from the serial-console of the OVMF menu(hint: type exit to get into somewhat menu-based UI)

Also, i'd install windowns with emulated graphics first, and then tried to attach the GPU. It would be working like secondary, but then, if all goes well, i'd remove the emulated graphics.

Something hints me that you shouldn't even have the output from OVMF in the first place since there is no VGA. But on the other hand, there is graphics-output-protocol or something, i don't know much about it. And i've got output on my emulated graphics when i booted the windows ISO.

But remember, i'm dumb and somewhat crazy.

Also, if you had it working with conventional VGA BIOS, why not just leave it working so?


you are a genius. i installed it via spice, downloaded the drivers and then booted with the graphics card and it's up and running.

the reason I stopped using vga bios is because i literally could not update the guest because shutting down/rebooting the system wouldn't happen correctly. it would just hang, most of the time actually causing the host to hang in the process.  i also am hoping to be able to stop using this specialized kernel. i can do it and all, but being able to just have the distro maintain the kernel for me is just way better. i don't want to have to update it manually every single time to avoid vulnerabilities etc.

edit: I just rebooted. I'm running an unpatched default arch kernel and it booted. this is absolutely amazing.

Last edited by risho (2014-12-08 16:02:16)

Offline

#3541 2014-12-09 05:07:52

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone know how to enable the virtio drivers after you've already installed WIndows 8? Whenever I change my disk interface to virtio Windows explodes on boot.

Offline

#3542 2014-12-09 05:32:47

jyni
Member
Registered: 2014-09-01
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:

Does anyone know how to enable the virtio drivers after you've already installed WIndows 8? Whenever I change my disk interface to virtio Windows explodes on boot.

The technique to do this is to add another dummy disk in your configuration with virtio enabled. When you manage to get dummy disk working with virtio drivers you can shutdown and change to virtio for your system disk too.

Offline

#3543 2014-12-09 05:37:01

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
sinny wrote:

Duelist, dozens of pages ago you posted a message on successfully installing windows 7 using ovmf

could you please specify all related details (qemu/libvirt configuration, win 7 iso "type", steps taken, etc)?

already had 4 "try sessions" with no result

https://bbs.archlinux.org/viewtopic.php … 9#p1480029

Since i've got strange problems with passing the bios via romfile option to my GPU(the UEFI freezes!), i reverted to BIOS and i dropped any thoughts about running OVMF with VFIO, so i've done setting it up just with qemu's emulated graphics.

oh, thanks. i somehow missed that post meaning, even though read it before.

well, seems like i'm in for a pain train (ovmf+uefi win7 install).
one lost day and i got to the point where setup begs me for any drivers - seems like it does not see any disks to install to... which is pretty weird since diskpart in setup's recovery command prompt lists the disk image i'm supposed to install to (tried all available hdd modes in virt-manager, including virtio)

anthonym52 wrote:

Does anyone know how to enable the virtio drivers after you've already installed WIndows 8? Whenever I change my disk interface to virtio Windows explodes on boot.

google or search maybe? (googling "switch windows to virtio" or finding this post for example)

Last edited by sinny (2014-12-09 05:49:19)

Offline

#3544 2014-12-09 14:30:55

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jyni wrote:
anthonym52 wrote:

Does anyone know how to enable the virtio drivers after you've already installed WIndows 8? Whenever I change my disk interface to virtio Windows explodes on boot.

The technique to do this is to add another dummy disk in your configuration with virtio enabled. When you manage to get dummy disk working with virtio drivers you can shutdown and change to virtio for your system disk too.

Haha, I was just thinking this would work too. (and it did!)

One more question, is everyone still using vfio-bind or is there a better way to do this now?

Here is my script, please help me optimize my settings.

#!/bin/bash -x # # Notes: Survives reboots, works where others have failed # inspired by kidchunks & aw on bbs.archlinux.org configfile=/etc/vfio-pci.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done GPU=$(lspci -D | grep VGA | grep -i gm107 | awk '{print $1}') AUDIO=$(lspci -D |grep -i gm107 -A1|awk 'END{print $1}') MEM=8192 HUGE="-mem-path /dev/hugepages" DISK=/media/ntfs/windows8.raw QEMU=/usr/local/bin/qemu-system-x86_64 NEED=$(( $MEM / 2 )) TOTAL=$(grep HugePages_Total /proc/meminfo | awk '{print $NF}') WANT=$(( $TOTAL + $NEED )) echo $WANT > /proc/sys/vm/nr_hugepages AVAIL=$(grep HugePages_Free /proc/meminfo | awk '{print $NF}') if [ $AVAIL -lt $NEED ]; then echo $TOTAL > /proc/sys/vm/nr_hugepages HUGE= fi QEMU_PA_SAMPLES=8192 QEMU_AUDIO_DRV=pa \ $QEMU \ -enable-kvm -cpu host,kvm=off \ -m $MEM $HUGE -smp 8,sockets=1,cores=4,threads=4 \ -bios /home/anthony/git/seabios/out/bios.bin -vga none \ -device intel-hda -device hda-output \ -device vfio-pci,host=$GPU,addr=03.0,multifunction=on,x-vga=on \ -device vfio-pci,host=$AUDIO,addr=03.1 \ -netdev tap,id=br0,vhost=on \ -device virtio-net-pci,mac=02:12:34:56:78:91,netdev=br0 \ -drive file=$DISK,cache=none,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -vnc :0 -k en-us TOTAL=$(grep HugePages_Total /proc/meminfo | awk '{print $NF}') RELEASE=$(( $TOTAL - $NEED )) echo $RELEASE > /proc/sys/vm/nr_hugepages

Other notes:
Kernel: 3.13.0-40-generic #69-Ubuntu SMP
Guest GPU: Nvidia 750TI with latest drivers
Grub boot opts: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 hugepages=1"
qemu version = 2.1.0, which seems to get a lot of "KVM internal error. Suberror: 1" crashes
Also, check out my setup running on my LG 34UM95-P with PBP: http://youtu.be/EFkLbEtj9cg

Last edited by anthonym52 (2014-12-09 15:36:48)

Offline

#3545 2014-12-09 14:37:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

well, seems like i'm in for a pain train (ovmf+uefi win7 install).
one lost day and i got to the point where setup begs me for any drivers - seems like it does not see any disks to install to... which is pretty weird since diskpart in setup's recovery command prompt lists the disk image i'm supposed to install to (tried all available hdd modes in virt-manager, including virtio)

Well, that seems strange. I'll try to dig it later. The main point is - windows 7 can be installed on bare metal hardware with UEFI and using GPT. If it lacks the drivers for disk controllers or OVMF is doing something strange - that's not windows' problem. Hint: try booting something like livecd based on windows 7.

sinny wrote:
anthonym52 wrote:

Does anyone know how to enable the virtio drivers after you've already installed WIndows 8? Whenever I change my disk interface to virtio Windows explodes on boot.

google or search maybe? (googling "switch windows to virtio" or finding this post for example)

Ah, come on there, don't blame people for asking, you do stuff like that too;)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3546 2014-12-09 17:42:58

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
sinny wrote:

well, seems like i'm in for a pain train (ovmf+uefi win7 install).
one lost day and i got to the point where setup begs me for any drivers - seems like it does not see any disks to install to... which is pretty weird since diskpart in setup's recovery command prompt lists the disk image i'm supposed to install to (tried all available hdd modes in virt-manager, including virtio)

Well, that seems strange. I'll try to dig it later. The main point is - windows 7 can be installed on bare metal hardware with UEFI and using GPT. If it lacks the drivers for disk controllers or OVMF is doing something strange - that's not windows' problem. Hint: try booting something like livecd based on windows 7.

i tried using i440fx - either setup is going mad about something different or microsoft completely dropped some old hardware drivers in win7-x64 (some doubt here)
or maybe problem is i am trying to install from self-made usb image attached as usb drive (iso that was converted to this usb image somehow refuses to be recognized by ovmf)

Duelist wrote:
sinny wrote:
anthonym52 wrote:

Does anyone know how to enable the virtio drivers after you've already installed WIndows 8? Whenever I change my disk interface to virtio Windows explodes on boot.

google or search maybe? (googling "switch windows to virtio" or finding this post for example)

Ah, come on there, don't blame people for asking, you do stuff like that too;)

well, i'm pretty sure i mostly ask either on-topic things or not-that-easy-to-find-answer-to (or at least i want to think i do)

Last edited by sinny (2014-12-09 17:44:19)

Offline

#3547 2014-12-10 09:18:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
Duelist wrote:
sinny wrote:

well, seems like i'm in for a pain train (ovmf+uefi win7 install).
one lost day and i got to the point where setup begs me for any drivers - seems like it does not see any disks to install to... which is pretty weird since diskpart in setup's recovery command prompt lists the disk image i'm supposed to install to (tried all available hdd modes in virt-manager, including virtio)

Well, that seems strange. I'll try to dig it later. The main point is - windows 7 can be installed on bare metal hardware with UEFI and using GPT. If it lacks the drivers for disk controllers or OVMF is doing something strange - that's not windows' problem. Hint: try booting something like livecd based on windows 7.

i tried using i440fx - either setup is going mad about something different or microsoft completely dropped some old hardware drivers in win7-x64 (some doubt here)
or maybe problem is i am trying to install from self-made usb image attached as usb drive (iso that was converted to this usb image somehow refuses to be recognized by ovmf)

Duelist wrote:
sinny wrote:

google or search maybe? (googling "switch windows to virtio" or finding this post for example)

Ah, come on there, don't blame people for asking, you do stuff like that too;)

well, i'm pretty sure i mostly ask either on-topic things or not-that-easy-to-find-answer-to (or at least i want to think i do)

I tried installing windows 7 using ovmf but i could never get it to work, i believe it needs the csm present, so i gave up and installed windows 8 with a patched ovmf to allow it to boot directly from the amd sata controller.
OVMF-sata.tar.gz

Also, once you're done installing windows 8 disable fast boot

Last edited by nbhs (2014-12-10 10:13:26)

Offline

#3548 2014-12-10 14:44:17

kidchunks
Member
Registered: 2014-12-03
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:

One more question, is everyone still using vfio-bind or is there a better way to do this now?

I've done testing using pc/pci-assign and while it worked, I needed a virtual vga attached.

Example:

qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu host \ -smp 8,sockets=1,cores=4,threads=4 \ -bios /usr/share/seabios/bios.bin -vga cirrus \ -device pci-assign,host=85:00.0,addr=08.0,multifunction=on,prefer_msi=off \ -device pci-assign,host=85:00.1,addr=08.1 \ -drive file=/root/qemu/win8.img,cache=none,if=virtio \ -drive file=/root/qemu/win812.iso,index=0,media=cdrom \ -drive file=/root/qemu/virtio-win-0.1-81.iso,index=1,media=cdrom \ -usb -usbdevice host:046d:c068 \ -vnc :1 \ -usbdevice tablet \ -daemonize

Stick with vfio-bind smile

Offline

#3549 2014-12-11 06:06:01

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
sinny wrote:

...

I tried installing windows 7 using ovmf but i could never get it to work, i believe it needs the csm present, so i gave up and installed windows 8 with a patched ovmf to allow it to boot directly from the amd sata controller.
OVMF-sata.tar.gz

Also, once you're done installing windows 8 disable fast boot

i got working win 8 install - the problem is i want win 7 uefi vm (win 8 is a bit... inconvenient? not sure about the exact word).

nbhs wrote:

... i believe it needs the csm present ...

win 7 64 in uefi mode seems to support only installations to gpt, so i am not so sure csm (mbr compatibility fallback, as far as i understand it) will really do the trick

UPD:
ok, seems like i am wrong afterall and csm may really be a requirement (source0, source1)

will try using OVMF_{CODE,VARS}-with-csm.fd instead of OVMF_{CODE,VARS}-pure-efi.fd when i get some time to play with my setup

Last edited by sinny (2014-12-11 06:44:52)

Offline

#3550 2014-12-11 07:28:56

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

UPD:
ok, seems like i am wrong afterall and csm may really be a requirement (source0, source1)

will try using OVMF_{CODE,VARS}-with-csm.fd instead of OVMF_{CODE,VARS}-pure-efi.fd when i get some time to play with my setup

Source1 in particular, speaks about "legacy OEM video drivers" - i guess that is related to VGA. Source0 - the writer says the same, he doesn't have "high-res UEFI setup" anymore, so that is VGA too.

Alright, today i have a free day dedicated to testing. But i'm a little bit afraid of the fact that i've got windows 7 installed on real UEFI hardware some time ago. I'm not 100% sure if there was any compatibility issues, but i doubt that CSM was enabled on that system. Also, it wasn't a notebook, so no OEM unupdateable hardware.

As far as i understand it - the CSM actually provides VGA, aw told us a while ago about the fact that we explicitly do not want it.

Last edited by Duelist (2014-12-11 07:29:46)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3551 2014-12-11 10:10:16

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

As far as i understand it - the CSM actually provides VGA, aw told us a while ago about the fact that we explicitly do not want it.

yep, i already thought about it being vga fallback as well. it won't do if it really is...
still, i think that win 7 setup running is a good sign - it's just that some kind of trick is needed to make it going. the hard part is uncovering this trick (and for now, i don't even have a clue on where to look for to get more info that may prove useful - i hope installer have some kind of logging) )

Offline

#3552 2014-12-11 11:09:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
Duelist wrote:

As far as i understand it - the CSM actually provides VGA, aw told us a while ago about the fact that we explicitly do not want it.

yep, i already thought about it being vga fallback as well. it won't do if it really is...
still, i think that win 7 setup running is a good sign - it's just that some kind of trick is needed to make it going. the hard part is uncovering this trick (and for now, i don't even have a clue on where to look for to get more info that may prove useful - i hope installer have some kind of logging) )

I have install running right now.

qemu-system-x86_64 \ -boot menu=on \ -enable-kvm \ -monitor stdio \ -M q35 \ -m 1024 \ -cpu host \ -net none \ -rtc base=localtime \ -smp 1,sockets=1,cores=1,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/mnt/hdd/qemu/OVMF_VARS.fd \ -drive file='/mnt/hdd/qemu/qemu-uefi-win7.img',id=disk,format=raw,aio=native,cache=none,if=none \ -drive file='/mnt/hdd/qemu/virtio.iso',id=cdrom,format=raw,readonly=on,if=none \ -drive file='/mnt/hdd/qemu/windows7.iso',id=cdrom2,format=raw,readonly=on,if=none \ -drive file='/dev/null',id=dummy \ -device virtio-scsi-pci \ -device virtio-blk-pci,drive=disk \ -device ide-cd,drive=cdrom \ -device scsi-cd,drive=cdrom2 \ -vga std

Well, i could've forced windowns instadamnation to use GPT, but... i just pressed "next" on the disk select menu, and it actually installs.

It MIGHT not detect the disks from the first time, saying something like "driver was installed, but no new devices appeared" - you just have to tell it "try again, silly".

Wonderful fact: OVMF doesn't recognize IDE-CD and shows virtio-blk-pci as EFI Misc. device(seems like we'll need to use SCSI anyway, to force OVMF to at least try booting from it), BUT windows installation system recognizes IDE-CD and loads drivers from it just fine.

I doubt the use of q35, maybe with -M pc it will work too.

Isn't windows 7 is using legacy MBR-based bootloader? We MAY have to force the use of GPT to make it choose the right, EFI-compatible bootloader.

EDIT: Whoa, hasty update! It finished copying installation files on the disk, and hung. But CTRL-C thrown at qemu, and starting it over again actually made it booting the installer. It means we're booted from virtio-blk-pci. But it hung again.

Booting from the device(virtio-blk-pci,that is), after "starting windows" animation, results in a black screen, some disk activity(on the host!) and zero cpu consumption. I've pushed the reset, windows said "ow, that did hurt. safe mode?" - booting in safe mode results in a message that "windows installation won't continue in safe-mode, restart the computer normally".
That means:
1.IT BOOTS.Kind of.
2.Maybe resetting everything wasn't a good idea.
3.Try again!

Oh, i get it. Guest reboots just don't work.

It uses GPT
It actually uses GPT. So we don't have to force it. The install is possible, you just need to do it right. Surprisingly enough, even a zero-ed out OVMF-based VM fails to reset. Maybe that's OVMF's problem.

Last edited by Duelist (2014-12-11 11:29:55)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3553 2014-12-11 14:03:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
sinny wrote:
Duelist wrote:

As far as i understand it - the CSM actually provides VGA, aw told us a while ago about the fact that we explicitly do not want it.

yep, i already thought about it being vga fallback as well. it won't do if it really is...
still, i think that win 7 setup running is a good sign - it's just that some kind of trick is needed to make it going. the hard part is uncovering this trick (and for now, i don't even have a clue on where to look for to get more info that may prove useful - i hope installer have some kind of logging) )

I have install running right now.

qemu-system-x86_64 \ -boot menu=on \ -enable-kvm \ -monitor stdio \ -M q35 \ -m 1024 \ -cpu host \ -net none \ -rtc base=localtime \ -smp 1,sockets=1,cores=1,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/mnt/hdd/qemu/OVMF_VARS.fd \ -drive file='/mnt/hdd/qemu/qemu-uefi-win7.img',id=disk,format=raw,aio=native,cache=none,if=none \ -drive file='/mnt/hdd/qemu/virtio.iso',id=cdrom,format=raw,readonly=on,if=none \ -drive file='/mnt/hdd/qemu/windows7.iso',id=cdrom2,format=raw,readonly=on,if=none \ -drive file='/dev/null',id=dummy \ -device virtio-scsi-pci \ -device virtio-blk-pci,drive=disk \ -device ide-cd,drive=cdrom \ -device scsi-cd,drive=cdrom2 \ -vga std

Well, i could've forced windowns instadamnation to use GPT, but... i just pressed "next" on the disk select menu, and it actually installs.

It MIGHT not detect the disks from the first time, saying something like "driver was installed, but no new devices appeared" - you just have to tell it "try again, silly".

Wonderful fact: OVMF doesn't recognize IDE-CD and shows virtio-blk-pci as EFI Misc. device(seems like we'll need to use SCSI anyway, to force OVMF to at least try booting from it), BUT windows installation system recognizes IDE-CD and loads drivers from it just fine.

I doubt the use of q35, maybe with -M pc it will work too.

Isn't windows 7 is using legacy MBR-based bootloader? We MAY have to force the use of GPT to make it choose the right, EFI-compatible bootloader.

EDIT: Whoa, hasty update! It finished copying installation files on the disk, and hung. But CTRL-C thrown at qemu, and starting it over again actually made it booting the installer. It means we're booted from virtio-blk-pci. But it hung again.

Booting from the device(virtio-blk-pci,that is), after "starting windows" animation, results in a black screen, some disk activity(on the host!) and zero cpu consumption. I've pushed the reset, windows said "ow, that did hurt. safe mode?" - booting in safe mode results in a message that "windows installation won't continue in safe-mode, restart the computer normally".
That means:
1.IT BOOTS.Kind of.
2.Maybe resetting everything wasn't a good idea.
3.Try again!

Oh, i get it. Guest reboots just don't work.

https://i.imgur.com/a5MviRd.png
It actually uses GPT. So we don't have to force it. The install is possible, you just need to do it right. Surprisingly enough, even a zero-ed out OVMF-based VM fails to reset. Maybe that's OVMF's problem.

Have you tried integrating your amd/nvidia drivers on the install disk to get past the black screen?

Offline

#3554 2014-12-11 15:41:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK i got windows 7 working using OVMF, so what i did was integrate redhat virtio drivers and qxl drivers into the windows installation disk using win-toolkit then i created a 4GB GPT fat32 formatted disk and copied the installation files there, then i launched QEMU with -vga qxl, a raw disk image, the windows installation image, also using -device virtio-scsi-pci ...

Offline

#3555 2014-12-11 16:43:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

OK i got windows 7 working using OVMF, so what i did was integrate redhat virtio drivers and qxl drivers into the windows installation disk using win-toolkit then i created a 4GB GPT fat32 formatted disk and copied the installation files there, then i launched QEMU with -vga qxl, a raw disk image, the windows installation image, also using -device virtio-scsi-pci ...

So you're running with -vga qxl and it still needs drivers? Strange. I was running with -vga std, and it needed drivers too! That's weird. But it's nice to see that i was right, and windows7+pure EFI+GPT is really possible. Guess i'll have to include -vga std drivers myself. Actually, maybe someone who made my(YARR) iso threw these drivers out.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3556 2014-12-11 16:49:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:

OK i got windows 7 working using OVMF, so what i did was integrate redhat virtio drivers and qxl drivers into the windows installation disk using win-toolkit then i created a 4GB GPT fat32 formatted disk and copied the installation files there, then i launched QEMU with -vga qxl, a raw disk image, the windows installation image, also using -device virtio-scsi-pci ...

So you're running with -vga qxl and it still needs drivers? Strange. I was running with -vga std, and it needed drivers too! That's weird. But it's nice to see that i was right, and windows7+pure EFI+GPT is really possible. Guess i'll have to include -vga std drivers myself. Actually, maybe someone who made my(YARR) iso threw these drivers out.

There are no -vga std drivers AFAIK, windows will use its standar vga driver which i believe is causing the problems, plus the only way you can install windows using EFI is with a GPT partitioned disk

Last edited by nbhs (2014-12-11 16:49:51)

Offline

#3557 2014-12-11 16:53:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Duelist wrote:
nbhs wrote:

OK i got windows 7 working using OVMF, so what i did was integrate redhat virtio drivers and qxl drivers into the windows installation disk using win-toolkit then i created a 4GB GPT fat32 formatted disk and copied the installation files there, then i launched QEMU with -vga qxl, a raw disk image, the windows installation image, also using -device virtio-scsi-pci ...

So you're running with -vga qxl and it still needs drivers? Strange. I was running with -vga std, and it needed drivers too! That's weird. But it's nice to see that i was right, and windows7+pure EFI+GPT is really possible. Guess i'll have to include -vga std drivers myself. Actually, maybe someone who made my(YARR) iso threw these drivers out.

There are no -vga std drivers AFAIK, windows will use its standar vga driver which i believe is causing the problems, plus the only way you can install windows using EFI is with a GPT partitioned disk

Isn't CSM helping when using MBR+UEFI?

Well, windows' default VGA driver works in safe mode and during installation, that's a little bit borked.
Seems like it actually loads something different and it's broken. If i'll trick it to use the fall-back-default one, which is rendering in safemode and during boot-up animation..

Last edited by Duelist (2014-12-11 16:54:58)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3558 2014-12-11 17:01:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:
Duelist wrote:

So you're running with -vga qxl and it still needs drivers? Strange. I was running with -vga std, and it needed drivers too! That's weird. But it's nice to see that i was right, and windows7+pure EFI+GPT is really possible. Guess i'll have to include -vga std drivers myself. Actually, maybe someone who made my(YARR) iso threw these drivers out.

There are no -vga std drivers AFAIK, windows will use its standar vga driver which i believe is causing the problems, plus the only way you can install windows using EFI is with a GPT partitioned disk

Isn't CSM helping when using MBR+UEFI?

Well, windows' default VGA driver works in safe mode and during installation, that's a little bit borked.
Seems like it actually loads something different and it's broken. If i'll trick it to use the fall-back-default one, which is rendering in safemode and during boot-up animation..

Actually maybe it doesnt load anything at all

Offline

#3559 2014-12-11 17:29:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Duelist wrote:
nbhs wrote:

There are no -vga std drivers AFAIK, windows will use its standar vga driver which i believe is causing the problems, plus the only way you can install windows using EFI is with a GPT partitioned disk

Isn't CSM helping when using MBR+UEFI?

Well, windows' default VGA driver works in safe mode and during installation, that's a little bit borked.
Seems like it actually loads something different and it's broken. If i'll trick it to use the fall-back-default one, which is rendering in safemode and during boot-up animation..

Actually maybe it doesnt load anything at all

That is very strange, because safe mode works. I'll try to grab a bootlog from a safe-mode boot to see which driver actually gets loaded in it.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3560 2014-12-11 17:31:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:
Duelist wrote:

Isn't CSM helping when using MBR+UEFI?

Well, windows' default VGA driver works in safe mode and during installation, that's a little bit borked.
Seems like it actually loads something different and it's broken. If i'll trick it to use the fall-back-default one, which is rendering in safemode and during boot-up animation..

Actually maybe it doesnt load anything at all

That is very strange, because safe mode works. I'll try to grab a bootlog from a safe-mode boot to see which driver actually gets loaded in it.

Yup like i said, its because it not loading the vga driver, go into safe mode and disable the the the display adapter

Offline

#3561 2014-12-11 17:55:19

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, nice to see people got it working - now i only need to somehow catch up.

for now i'm getting weird issues - i definitely can not uefi-boot from any of 3 isos i got (all 3 are not recognized as FS's, onnly as BLK devices)
and installing from self-assembled image failed requiring drivers. still need more testing using all the data you assembled

i also tried using virt-manager plenty of time, but it seemed less manageable (due to lack of fine-grained control)

anyways, thanks for all the work - it's really valuable, considering the mess win8 is

Offline

#3562 2014-12-11 18:05:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Duelist wrote:
nbhs wrote:

Actually maybe it doesnt load anything at all

That is very strange, because safe mode works. I'll try to grab a bootlog from a safe-mode boot to see which driver actually gets loaded in it.

Yup like i said, its because it not loading the vga driver, go into safe mode and disable the the the display adapter

The problem is - i'll have to reinvent the bicycle: safe mode isn't functional. So it's either you're pushing all the needed drivers in the ISO at the very beginning, or you're booting a livecd system to fix the not-yet-installed system. Thankfully, that livecd may be MSDART, which is included in my installation ISO, so i can even fire up the explorer to have more-or-less GUI representation of what i'll need to do to revive the system. Or i could just shuffle through all available display devices and find one that works nice.

BTW, are you using -M pc or -M q35(like i am)?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3563 2014-12-11 18:14:59

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:
Duelist wrote:

That is very strange, because safe mode works. I'll try to grab a bootlog from a safe-mode boot to see which driver actually gets loaded in it.

Yup like i said, its because it not loading the vga driver, go into safe mode and disable the the the display adapter

The problem is - i'll have to reinvent the bicycle: safe mode isn't functional. So it's either you're pushing all the needed drivers in the ISO at the very beginning, or you're booting a livecd system to fix the not-yet-installed system. Thankfully, that livecd may be MSDART, which is included in my installation ISO, so i can even fire up the explorer to have more-or-less GUI representation of what i'll need to do to revive the system. Or i could just shuffle through all available display devices and find one that works nice.

BTW, are you using -M pc or -M q35(like i am)?

Both seem to work just fine, i suggest you try the method i mentioned before, it will save you a lot of time, perhaps its possible to load the qxl driver during the installation, idk and i havent tried, i also mentioned booting with -vga std ( this is of course post install ) and disabling it in safe mode also seems to allow windows to boot, as of trying another vga adapter, i dont think it will work since windows always try load its standard vga driver, another thing i tested is passing though my GPU ( R9 280x ) and it will work but the drivers have to be installed first ( booting with -vga qxl then installing the catalyst driver )

Last edited by nbhs (2014-12-11 18:19:54)

Offline

#3564 2014-12-11 18:22:23

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Duelist wrote:
nbhs wrote:

Yup like i said, its because it not loading the vga driver, go into safe mode and disable the the the display adapter

The problem is - i'll have to reinvent the bicycle: safe mode isn't functional. So it's either you're pushing all the needed drivers in the ISO at the very beginning, or you're booting a livecd system to fix the not-yet-installed system. Thankfully, that livecd may be MSDART, which is included in my installation ISO, so i can even fire up the explorer to have more-or-less GUI representation of what i'll need to do to revive the system. Or i could just shuffle through all available display devices and find one that works nice.

BTW, are you using -M pc or -M q35(like i am)?

Both seem to work just fine, i suggest you try the method i mentioned before, it will save you a lot of time, perhaps its possible to load the qxl driver during the installation, idk and i havent tried, i also mentioned booting with -vga std ( this is of course post install ) and disabling it in safe mode also seems to allow windows to boot, as of trying another vga adapter, i dont think it will work since windows always loads its standard vga driver, another thing i tested is passing though my GPU ( R9 280x ) and it will work but the drivers have to be installed first ( booting with -vga qxl then installing the catalyst driver )

But i've installed the whole system using only -vga std and no other stuff! Tried changing it on that undead system to something else(-vga qxl for example) - and no, the install won't finish because there is no graphics.

BTW, great to know that even VFIO works with that tricky setup.

Also a thing i've remembered: do you have guest reboots working? Because my test-system hangs in the OVMF after a reset, showing an underscore somewhere in the middle and nothing else. Only ctrl-c to qemu saves the day.

Last edited by Duelist (2014-12-11 20:16:33)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3565 2014-12-11 18:24:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Also a think i've remembered: do you have guest reboots working? Because my test-system hangs in the OVMF after a reset, showing an underscore somewhere in the middle and nothing else. Only ctrl-c to qemu saves the day.

Looks like an issue with OVMF + q35

Offline

#3566 2014-12-11 18:27:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

But i've installed the whole system using only -vga std and no other stuff! Tried changing it on that undead system to something else(-vga qxl for example) - and no, the install won't finish because there is no graphics.

Then i believe the only solution is integrating either amd/nvidia drivers or the qxl drivers, or perhaps replacing windows 7 vga driver with windows 8's one (if its possible at all)

Last edited by nbhs (2014-12-11 18:27:59)

Offline

#3567 2014-12-12 21:04:27

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I got my new hardware - including a new 1 TB SSD from Samsung. So I have one SSD for archlinux (240 GB) and the new Samsung SSD for virtualized Windows 8.1.

NOW: How do I add the SSD as a direct block device as AW suggested? I have done it like this:

<devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>

      <source dev='/dev/sdb'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>

It did install Windows like that. But I am not sure it's correct. I really want as much performance as possible in a VM.


Thank you guys for your help!

Offline

#3568 2014-12-13 11:43:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi, I got my new hardware - including a new 1 TB SSD from Samsung. So I have one SSD for archlinux (240 GB) and the new Samsung SSD for virtualized Windows 8.1.


Offtopic question:
1TB SSD?! Entirely for windows?! Why would you need linux in that case?
I'm fine with my 60+10(SSD) setup, as it hosts some games on steam and i'm using linux as a main system, not the secondary. I'd understand if you would use all the info from something like redhat's VM security guide or fiddled with iptables so hard, that even skype wouldn't work, just for security reasons, but setting up vfio entirely for gaming..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3569 2014-12-13 13:13:37

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need linux as a backend for Live TV. I know, there are easier and more efficient ways like building a second minimal server for it, but I like the idea of having one system that can do it all.
And regarding the 1TB SSD: there's been a great offer last week. And having all my games on one SSD isn't too bad right? wink

btw: I did some testing and I am sure the disk performance isn't right. Please help guys.

Last edited by 4kGamer (2014-12-13 13:36:08)

Offline

#3570 2014-12-13 14:28:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't have a spare ssd large enough for my VM, instead I use a small one in front of a spinning disk with bcache.  I'm currently using multi-queue virtio-scsi for the guest and it seems pretty quick.

 <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/mnt/store/vm/Steam.raw'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <driver queues='4'/> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller>

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3571 2014-12-13 20:43:27

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

multi-queue virtio-scsi

Can you hint on how to do that via command-line?
BTW, it seems somewhat strange that virtio-blk is slower than virtio-scsi with scsi disk attached.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3572 2014-12-13 20:57:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

multi-queue virtio-scsi

Can you hint on how to do that via command-line?
BTW, it seems somewhat strange that virtio-blk is slower than virtio-scsi with scsi disk attached.

I think these are the relevant lines from the libvirt log:

-device virtio-scsi-pci,id=scsi0,num_queues=4,bus=pci.0,addr=0x5 \ -drive file=/mnt/store/vm/Steam.raw,if=none,id=drive-scsi0-0-0-0,format=raw,cache=none,aio=native \ -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1

I don't know which is faster between virtio-blk and virtio-scsi, try it and see.  I just picked up multi-queue virtio-scsi based on the virt tuning and optimization guide.  There's also x-data-plane=on support in virtio-blk-pci which you can google about.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3573 2014-12-14 16:08:22

oegat
Member
Registered: 2014-12-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to pass through an MSI geforce GTX 970 as a secondary adapter, to a Windows 8.1 vm. I get the Code 43 error in device manager, even though I seem to have followed aw:s instructions on how to work around nvidia's "bugs".  Can anyone see an explanation in what follows?

The card is a drop-in replacement for a radeon 7790, which was working with vfio passthrough just before. So my problem is arguably device-specific. Note that I'm trying secondary passthrough, so no extra vga logic involved (unless I misunderstood things and this misunderstanding is the reason for my problems).

System: Opteron magny-cours (two 6134 cpu:s), on Supermicro H8DG6
Numa turned off in cmos while testing.
Host vga: amd firepro v4900 (radeon driver)

ubuntu 14.10 (upgraded from 14.04 during testing this, in order to get support for kvm=off)
qemu 2.1/2.2 (same result)
libvirt 1.2.8
virt-manager 1.0.1
Nvidia driver: 244.11 / 244.75 (same result with both, cannot try older since the gpu is too new)

Passed thru adapters:
Geforce GTX 970 (gpu+audio): 42:00.{01} (giving code 43 error)
TI USB 73x0: 41:00.0 (usb controller for kb+mouse, works fine)

I run libvirt for help with CPU-pinning, using 'virsh edit' for settings not understood by virt-manager. The qemu command line it gives rise to is this one:

qemu-system-x86_64 -enable-kvm -name Win8 -S -machine pc-i440fx-2.2,accel=kvm,usb=off,mem-merge=off -cpu host,kvm=off -m 4096 -mem-prealloc -mem-path /hugepages/libvirt/qemu -realtime mlock=on -smp 1,sockets=1,cores=1,threads=1 -uuid xxx -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Win8.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device ahci,id=ahci0,bus=pci.0,addr=0x6 -drive file=/dev/sdb,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/dev/zvol/data/win8d,if=none,id=drive-ide0-0-1,format=raw -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive file=/opt/images/en-gb_windows_8_n_x64_dvd_915799.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=24,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:46:9f:a9,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1 -device e1000,netdev=hostnet1,id=net1,mac=52:54:00:25:10:ce,bus=pci.0,addr=0x9 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -device VGA,id=video0,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,addr=0x8,rombar=1 -device vfio-pci,host=42:00.1,id=hostdev1,bus=pci.0,addr=0xa,rombar=1 -device vfio-pci,host=41:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

I replaced the real UUID above with xxx, and when running qemu 2.1 the machinetype was 'pc-i440fx-trusty'. Otherwise was everything the same. So the question is, am I missing something, or could it be some other previously undocumented quirk in my setup which is detected by the nvidia driver?

Unless I am mistaken that a Geforce 970 can be passed through as secondary this easily. I have assumed that options such as x-vga=on and/or custom firmware won't be needed when doing it this way. I also tried turing MSI on and off in windows, but without luck.

(As you see I am running ubuntu as host, not arch. I'm posting here anyway because I think it is a good thing to keep the collected research on vfio passthrough in one place at this stage, and because the level of abstraction of the techniques discussed here is anyway above the workings of specific package managers and service daemons. I hope I am not breaking any forum rule or such.)

Edit: broke up the qemu command line for readability.

Last edited by oegat (2014-12-14 16:22:25)

Offline

#3574 2014-12-14 16:13:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oegat wrote:

Unless I am mistaken that a Geforce 970 can be passed through as secondary this easily.

You are mistaken


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3575 2014-12-14 16:44:26

oegat
Member
Registered: 2014-12-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You are mistaken

Thanks, then I must have taken the wrong path somewhere. Explains a lot.

Offline

#3576 2014-12-14 17:05:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oegat wrote:

Nvidia driver: 244.11 / 244.75 (same result with both, cannot try older since the gpu is too new)

You've meant 344.11 and 344.75, right?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3577 2014-12-14 17:16:15

oegat
Member
Registered: 2014-12-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
oegat wrote:

Nvidia driver: 244.11 / 244.75 (same result with both, cannot try older since the gpu is too new)

You've meant 344.11 and 344.75, right?

Yes, sorry.

Offline

#3578 2014-12-15 09:15:01

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

Just to inform that with the new kernel (3.17.6-1 ARCH), I do not need to patch it, just works perfect smile

My kernel options are now:

rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:1184,10de:030a,8086:15a1 intel_pstate=disable intel_iommu=on,igfx_off quiet

And performance is the same, or even a bit more now smile

Regards,


TheArcher

Offline

#3579 2014-12-15 10:31:54

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks TheArcher, that is great news. I have something to share too:

Ok, everyone with the intention of buying a new motherboard and owners of a motherboard with DUAL LAN, listen up:

I found out something exciting and I want to share it with you. I realize this may be no news whatsoever for the majority here. It was a big one for me though.

I was never happy with the performance of the VM. Even having virtio for disk and network didn't seem to do the trick. Then I tried to passtrough one of the LAN ports as a pci device - and it worked wonders. Disk usage went down to 0% in idle and 4k gaming is finally possible for me in a virtualized Windows 8.1.

Like I said this might not be news and (it is kind of common sense) but I can only advise people taking full advantage of that second LAN port.

A big thanks to aw and nbhs for their help. I am not done configuring, so many more questions to come wink


btw: I am using direct block device as aw suggested. It has another huge advantage: When I need to I can boot directly into native Windows 8.1 - the same disk I use as my VM.


Hope it helps some of you smile

Last edited by 4kGamer (2014-12-15 12:25:16)

Offline

#3580 2014-12-15 12:51:09

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

smile

I have 2 ethernet (Asrock Z97 Extreme 6) as well and I was passing the intel one without problems but with the realtek one... was not able to work fine... (drivers downloaded from the official website)... so, if you have two and passthrough one and is not working... try the other one XD

Regards,


TheArcher

Offline

#3581 2014-12-15 13:10:28

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah, I have the same motherboard! I sent you a mail, did you get it? Need uefi boot working, which still doesn't. Just have a look into your mail account.

thanks a lot!

Offline

#3582 2014-12-15 13:18:56

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

yeah, I have the same motherboard! I sent you a mail, did you get it? Need uefi boot working, which still doesn't. Just have a look into your mail account.

thanks a lot!

Hello smile

I've got you mail right now, but I will you reply here to benefit of all the community (please, moderators, move it to the proper place if necessary).

To boot with UEFI you just need to choose on the boot menu (F11) the UEFI option for your USB / DVD boot device.

Normally you will have the device two times (one with UEFI and one without smile).

Please, let us know if solved wink

Anyway, this forum is plenty of this information on the newbie corner (I need to go there almost everyday to find solutions smile).

Regards,


TheArcher

Offline

#3583 2014-12-15 13:37:05

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, thank you. The problem is, it is NOT showing in BIOS. So it must be something else. I use standard grub ...

I think my problem is the actual path of the bootx64.efi/grubx64.efi

thanks a lot!

Last edited by 4kGamer (2014-12-15 13:42:50)

Offline

#3584 2014-12-15 13:43:53

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm not expert on this but I remember that I had a few problems with it when booting (booting from USB pen drive).

Just check that you don't have secureboot enabled or fastboot (gives a lot of problems).

I hope it helps smile

Offline

#3585 2014-12-15 13:48:11

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't have secure boot or fast boot enabled. What would really help:  can you please tell me 2 things:

1. where is your grub.cfg located and
2. where is your .efi file located? (and what name does it have? grubx64.efi or what?)

That would solve my problem, I am sure

Offline

#3586 2014-12-15 13:59:48

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok smile

I'm not using grub, I like gummibot, I just followed the instructions on the Wiki (https://wiki.archlinux.org/index.php/be … guide#GRUB) but:

My EFI file is on: /boot/EFI/Boot/BOOTX64.EFI

I love Gummiboot smile

Last edited by thearcherblog (2014-12-15 14:19:17)

Offline

#3587 2014-12-15 14:04:46

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

all right, thanks a lot!. I will have a look into that later on. I think I can solve it now smile My bootx64.efi is located under /boot/efi/EFI/boot

so that really might be it smile

Last edited by 4kGamer (2014-12-15 14:05:17)

Offline

#3588 2014-12-15 14:23:57

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

Hello,
My kernel options are now:

rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:1184,10de:030a,8086:15a1 intel_pstate=disable intel_iommu=on,igfx_off quiet

And performance is the same, or even a bit more now smile

What is the benefit of "intel_pstate=disable"?

Offline

#3589 2014-12-15 14:25:54

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:
thearcherblog wrote:

Hello,
My kernel options are now:

rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:1184,10de:030a,8086:15a1 intel_pstate=disable intel_iommu=on,igfx_off quiet

And performance is the same, or even a bit more now smile

What is the benefit of "intel_pstate=disable"?

To be honest, the problem I had was that my CPU (4790K) was all time running at 4.4GHz so I was looking for some information and when I disabled it, then works quite good (it's at 800MHz when it's in idle mode and changing to 4.4GHz when need it. Then my computer's temperature is back to normal smile I've checked it today with pstate enabled and temperatures and core speeds are back to normal smile I also run my gummibot with the microcode update before initram... maybe that made the trick smile


I hope it helps to you smile

Last edited by thearcherblog (2014-12-16 09:17:56)

Offline

#3590 2014-12-15 15:37:47

Rholk
Member
Registered: 2014-12-15
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer:

Hi, is it possible that multiplayer games are slow down by network traffic?

I have 1 nic, a 4790k cpu and using 6 cores for the guest (cpu pinned, but not using cpuset shield, only taskset). While i have good performance in a couple of games (solo), i have really bad performance on "simple games" like Heroes of the storm. Where i barely reach 50fps (native windows was at 80...).
I use a tap bridge, my network card uses virtio.

Can you give some numbers of the gain between kvm network and the new mode?

I also thougt of performance problem with emulated sound, but it doesn't seems to affect this so much. (made a game with sound card disabled, performance wasn't better)

Last edited by Rholk (2014-12-15 15:38:11)

Offline

#3591 2014-12-15 17:10:14

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rholk wrote:

4kGamer:

Hi, is it possible that multiplayer games are slow down by network traffic?

I have 1 nic, a 4790k cpu and using 6 cores for the guest (cpu pinned, but not using cpuset shield, only taskset). While i have good performance in a couple of games (solo), i have really bad performance on "simple games" like Heroes of the storm. Where i barely reach 50fps (native windows was at 80...).
I use a tap bridge, my network card uses virtio.

Can you give some numbers of the gain between kvm network and the new mode?

I also thougt of performance problem with emulated sound, but it doesn't seems to affect this so much. (made a game with sound card disabled, performance wasn't better)

anything might factor - it highly depends on the way application is designed/coded
80 to 50 fps is not that huge of a performance drop, i would say (significant though)

in my case, audio and memory allocations were main slowdowns (jumped to 200-300 fps in lol, from 50-60 fps)
oh, i'm using basic virtio network by the way

Last edited by sinny (2014-12-15 17:10:49)

Offline

#3592 2014-12-15 23:30:08

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Rholk:

I can't give you an exact answer. I haven't had the chance to test thoroughly. I can tell you more in a few days, but based on my experiences so far, I would believe performance does depend on network usage. You can also give multi-queue virtio-scsi a try. aw wrote about it a page back.

Offline

#3593 2014-12-16 01:23:40

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Thanks TheArcher, that is great news. I have something to share too:

Ok, everyone with the intention of buying a new motherboard and owners of a motherboard with DUAL LAN, listen up:

I found out something exciting and I want to share it with you. I realize this may be no news whatsoever for the majority here. It was a big one for me though.

I was never happy with the performance of the VM. Even having virtio for disk and network didn't seem to do the trick. Then I tried to passtrough one of the LAN ports as a pci device - and it worked wonders. Disk usage went down to 0% in idle and 4k gaming is finally possible for me in a virtualized Windows 8.1.

Like I said this might not be news and (it is kind of common sense) but I can only advise people taking full advantage of that second LAN port.

A big thanks to aw and nbhs for their help. I am not done configuring, so many more questions to come wink


btw: I am using direct block device as aw suggested. It has another huge advantage: When I need to I can boot directly into native Windows 8.1 - the same disk I use as my VM.


Hope it helps some of you smile

For those of you with supported Intel Nics I find using VF devices also performs better than bridge too (You can pass through the spawned VFs using VFIO).

Offline

#3594 2014-12-16 01:29:20

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can anyone help me with an example of using cpuset and cgroups to fix the host to cores 0-3, VM1 to cores 4-5, VM2 to core 6 and VM3 to core 7?

Offline

#3595 2014-12-16 04:34:12

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Thanks TheArcher, that is great news. I have something to share too:

Ok, everyone with the intention of buying a new motherboard and owners of a motherboard with DUAL LAN, listen up:

I found out something exciting and I want to share it with you. I realize this may be no news whatsoever for the majority here. It was a big one for me though.

I was never happy with the performance of the VM. Even having virtio for disk and network didn't seem to do the trick. Then I tried to passtrough one of the LAN ports as a pci device - and it worked wonders. Disk usage went down to 0% in idle and 4k gaming is finally possible for me in a virtualized Windows 8.1.

Like I said this might not be news and (it is kind of common sense) but I can only advise people taking full advantage of that second LAN port.

A big thanks to aw and nbhs for their help. I am not done configuring, so many more questions to come wink


btw: I am using direct block device as aw suggested. It has another huge advantage: When I need to I can boot directly into native Windows 8.1 - the same disk I use as my VM.


Hope it helps some of you smile

How do you pass a single ethernet port as a PCI device? Do you have benchmarks ?

Offline

#3596 2014-12-16 06:19:28

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:
4kGamer wrote:

Thanks TheArcher, that is great news. I have something to share too:

Ok, everyone with the intention of buying a new motherboard and owners of a motherboard with DUAL LAN, listen up:

I found out something exciting and I want to share it with you. I realize this may be no news whatsoever for the majority here. It was a big one for me though.

I was never happy with the performance of the VM. Even having virtio for disk and network didn't seem to do the trick. Then I tried to passtrough one of the LAN ports as a pci device - and it worked wonders. Disk usage went down to 0% in idle and 4k gaming is finally possible for me in a virtualized Windows 8.1.

Like I said this might not be news and (it is kind of common sense) but I can only advise people taking full advantage of that second LAN port.

A big thanks to aw and nbhs for their help. I am not done configuring, so many more questions to come wink


btw: I am using direct block device as aw suggested. It has another huge advantage: When I need to I can boot directly into native Windows 8.1 - the same disk I use as my VM.


Hope it helps some of you smile

How do you pass a single ethernet port as a PCI device? Do you have benchmarks ?

Hello,

I do not have benchmark but is quite confortable to me using the second ethernet device as a native one on Windows because then both computers (windows and linux) are on the same LAN without bridge/NAT.

This is the command I use to start the qemu:

qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp 4,sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:19.0 -usb -usbdevice host:046d:c52e -net none -vga none -nographic

Regards,


TheArcher

Offline

#3597 2014-12-16 11:09:01

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Krobar wrote:

Can anyone help me with an example of using cpuset and cgroups to fix the host to cores 0-3, VM1 to cores 4-5, VM2 to core 6 and VM3 to core 7?

Just use cset

cset shield -c 4-5
cset shield --shield --pid $PID1
cset shield --shield --pid $PID2


Then pin with taskset:

taskset -pc 4 $PID1
taskset -pc 5 $PID2

You can get each VCPU pid by running qemu with:

-qmp unix:./qmp-sock,server,nowait

And getting the VCPU'S pids

echo "{ \"execute\": \"qmp_capabilities\" }" "{ \"execute\": \"query-cpus\" }" | socat UNIX-CONNECT:./qmp-sock STDIO

After you're done

cset shield --reset

And here's the cset build for arch cpuset.tar.gz

Last edited by nbhs (2014-12-16 11:25:30)

Offline

#3598 2014-12-16 15:35:05

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:

This is the command I use to start the qemu:

qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp 4,sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:19.0 -usb -usbdevice host:046d:c52e -net none -vga none -nographic

Regards,


TheArcher


I can't seem to get the PCI stub ID claimed, both interfaces have the same ID, do your IDs match as well?

05:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
06:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]


[anthony:/home/anthony] 7s $ dmesg |grep -i claim
[    0.332788] pci 0000:00:1f.0: quirk: [io  0x0500-0x053f] claimed by ICH6 GPIO
[    4.332685] pci-stub 0000:03:00.0: claimed by stub
[    4.332702] pci-stub 0000:03:00.1: claimed by stub

Offline

#3599 2014-12-16 15:51:00

apostmax
Member
Registered: 2014-12-16
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any news on passing through a Intel IGD?
Is it possible at all? I am interested in passing through a HD 4600 (Intel i7 4790)

Offline

#3600 2014-12-16 16:00:20

magemaster
Member
From: Brasil
Registered: 2010-11-24
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Someone else using with arch kernel?

uname -a

Linux 3.17.6-1-ARCH #1 SMP PREEMPT Sun Dec 7 23:43:32 UTC 2014 x86_64 GNU/Linux

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=e9f18698-c70a-41f6-b5c6-c84a8e7eac85 rw quiet rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:10c3,10de:0be3 intel_pstate=disable i915.enable_hd_vgaarb=1 intel_iommu=on,igfx_off quiet

Tried this way,  based on thearcherblog, gave me no error, but no screen

sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -cpu host,kvm=off -m 2048 -smp 2,sockets=1,cores=2 -serial none -parallel none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/ovmf_x64.bin \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/user/windows.img,cache=writeback,if=none,id=drive0,aio=native \ -drive file=/home/user/Win81.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /home/user/Downloads/virtio.iso \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -net none \ -vga none \ -boot menu=on \ -nographic

and this way, based on nbhs, gave me this error: KVM internal error. Suberror: 1 emulation failure

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Offline

#3601 2014-12-17 05:41:23

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:
thearcherblog wrote:

This is the command I use to start the qemu:

qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp 4,sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:19.0 -usb -usbdevice host:046d:c52e -net none -vga none -nographic

Regards,


TheArcher


I can't seem to get the PCI stub ID claimed, both interfaces have the same ID, do your IDs match as well?

05:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
06:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]


[anthony:/home/anthony] 7s $ dmesg |grep -i claim
[    0.332788] pci 0000:00:1f.0: quirk: [io  0x0500-0x053f] claimed by ICH6 GPIO
[    4.332685] pci-stub 0000:03:00.0: claimed by stub
[    4.332702] pci-stub 0000:03:00.1: claimed by stub


I got this working by using: echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts and using pci-assign (the NICs are in the same IOMMU group but I have disabled eth1 as much as possible on the host -- is this safe?)

Also, I'm trying OVMF to see if I can improve reliability with boots -- what is 'OVMF_VARS-pure-efi.fd ' and why do I need it? Also how do I get my HDD booted -- I end up at the UEFI interactive shell each time:

QEMU_PA_SAMPLES=8192 QEMU_AUDIO_DRV=pa \
$QEMU \
-boot menu=on \
-drive if=pflash,format=raw,readonly,file=/home/anthony/ovmf-x64/OVMF-pure-efi.fd \
-enable-kvm -cpu host,kvm=off -rtc base=localtime \
-m $MEM $HUGE -smp 8,sockets=1,cores=4,threads=4 \
-vga std \
-device intel-hda -device hda-output \
-device vfio-pci,host=$GPU,addr=04.0,multifunction=on,x-vga=on \
-device vfio-pci,host=$AUDIO,addr=04.1 \
-device pci-assign,host=06:00.0 \
-drive file=$DISK,cache=none,if=none,id=drive0,aio=threads \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-net none \
-vnc :0 -k en-us

Offline

#3602 2014-12-17 06:00:34

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:
thearcherblog wrote:

This is the command I use to start the qemu:

qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp 4,sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:19.0 -usb -usbdevice host:046d:c52e -net none -vga none -nographic

Regards,


TheArcher


I can't seem to get the PCI stub ID claimed, both interfaces have the same ID, do your IDs match as well?

05:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
06:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]


[anthony:/home/anthony] 7s $ dmesg |grep -i claim
[    0.332788] pci 0000:00:1f.0: quirk: [io  0x0500-0x053f] claimed by ICH6 GPIO
[    4.332685] pci-stub 0000:03:00.0: claimed by stub
[    4.332702] pci-stub 0000:03:00.1: claimed by stub

Yes its normal for them to match, I use 82574Ls and they are independent controllers so it should work. I'm a virsh user but this is all I do to bind an 82574L on 04:00.0 (It definitely works independently as the other controller is used by the host:
echo "8086 10d3" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:04:00.0 > /sys/bus/pci/devices/0000:04:00.0/driver/unbind
echo 0000:04:00.0 > /sys/bus/pci/drivers/pci-stub/bind
vfio-bind 0000:04:00.0
virsh create ~/VM/Router.xml

Router.xml:
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/>

For those who are interested I switched the GPU to an R5-230 Radeon and my reboot issue disappeared.

Offline

#3603 2014-12-17 06:02:33

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:
thearcherblog wrote:

This is the command I use to start the qemu:

qemu-system-x86_64 -enable-kvm -rtc base=localtime -m 8192 -cpu host,kvm=off -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin -drive if=pflash,format=raw,file=/home/jgomez/VM/ovmf_x64.bin -smp 4,sockets=1,cores=4 -serial none -parallel none -nodefconfig -drive file=/home/jgomez/VM/win7.img,cache=writeback,if=none,id=drive0,aio=native -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 -device virtio-scsi-pci,id=scsi -drive file=/home/jgomez/Downloads/win8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install -cdrom /home/jgomez/Downloads/virtio.iso -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:19.0 -usb -usbdevice host:046d:c52e -net none -vga none -nographic

Regards,


TheArcher


I can't seem to get the PCI stub ID claimed, both interfaces have the same ID, do your IDs match as well?

05:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
06:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]


[anthony:/home/anthony] 7s $ dmesg |grep -i claim
[    0.332788] pci 0000:00:1f.0: quirk: [io  0x0500-0x053f] claimed by ICH6 GPIO
[    4.332685] pci-stub 0000:03:00.0: claimed by stub
[    4.332702] pci-stub 0000:03:00.1: claimed by stub

No, I have different ID's...

[jgomez@mycolonialone ~]$ lspci | grep Eth 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)

Offline

#3604 2014-12-17 06:05:09

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

magemaster wrote:

Someone else using with arch kernel?

uname -a

Linux 3.17.6-1-ARCH #1 SMP PREEMPT Sun Dec 7 23:43:32 UTC 2014 x86_64 GNU/Linux

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=e9f18698-c70a-41f6-b5c6-c84a8e7eac85 rw quiet rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:10c3,10de:0be3 intel_pstate=disable i915.enable_hd_vgaarb=1 intel_iommu=on,igfx_off quiet

Tried this way,  based on thearcherblog, gave me no error, but no screen

sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -cpu host,kvm=off -m 2048 -smp 2,sockets=1,cores=2 -serial none -parallel none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/ovmf_x64.bin \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/user/windows.img,cache=writeback,if=none,id=drive0,aio=native \ -drive file=/home/user/Win81.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /home/user/Downloads/virtio.iso \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -net none \ -vga none \ -boot menu=on \ -nographic

and this way, based on nbhs, gave me this error: KVM internal error. Suberror: 1 emulation failure

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I also got the black screen but someone suggested me to have this (and works for me smile):

[jgomez@mycolonialone etc]$ cat /etc/modprobe.d/qemu_kvm.conf options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 #options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Regards smile

Offline

#3605 2014-12-17 06:14:52

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My geekbench results are near native after passing through my NIC controller.. not sure why it's making a difference -- but I'm really just posting to show off this: u4fJxyt.jpg

Offline

#3606 2014-12-17 06:26:21

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Glad to hear it works well. Sorry about the confusion I had gotten your config and posts muddled with anthonym52.

Offline

#3607 2014-12-17 06:26:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:

My geekbench results are near native after passing through my NIC controller.. not sure why it's making a difference

So now you're limiting your NIC bandwidth to the wire speed and going out to an external switch whereas you could have had 10s of gigabits through an internal bridge to have seamless window mode of your Windows VM onto your Linux desktop?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3608 2014-12-17 06:30:54

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
anthonym52 wrote:

My geekbench results are near native after passing through my NIC controller.. not sure why it's making a difference

So now you're limiting your NIC bandwidth to the wire speed and going out to an external switch whereas you could have had 10s of gigabits through an internal bridge to have seamless window mode of your Windows VM onto your Linux desktop?

That is a good point, I just wanted to try it out -- seems to preform a bit better but I need to do more benchmarks. BTW, this isn't 'seamless window mode' it's picture-by-picture (a hardware feature of my monitor).

Edit: (This is the reason for the off screen camera image)

Last edited by anthonym52 (2014-12-17 06:33:05)

Offline

#3609 2014-12-17 09:34:23

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:

My geekbench results are near native after passing through my NIC controller.. not sure why it's making a difference -- but I'm really just posting to show off this: http://i.imgur.com/u4fJxyt.jpg

Damn, this looks AWESOME!
Especially when you match the wallpaper.
What's the model number?
It's awesomely better than having two separate screens(1280x1024 and 1920x1080). Can you press a button and make that screen appear "double-wide" to the system, running windows(or linux) on native resolution?

Two full-featured(almost) PCs in one case and two screens in one. I think i need a two-in-one keyboard, with a hardware KVM switch built into:D


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3610 2014-12-17 09:43:51

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
anthonym52 wrote:

My geekbench results are near native after passing through my NIC controller.. not sure why it's making a difference -- but I'm really just posting to show off this: http://i.imgur.com/u4fJxyt.jpg

Damn, this looks AWESOME!
Especially when you match the wallpaper.
What's the model number?
It's awesomely better than having two separate screens(1280x1024 and 1920x1080). Can you press a button and make that screen appear "double-wide" to the system, running windows(or linux) on native resolution?

Two full-featured(almost) PCs in one case and two screens in one. I think i need a two-in-one keyboard, with a hardware KVM switch built into:D

Can be the new Philips Monitor presented this summer? smile Looks pretty cool smile

Offline

#3611 2014-12-17 11:05:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
Duelist wrote:
anthonym52 wrote:

My geekbench results are near native after passing through my NIC controller.. not sure why it's making a difference -- but I'm really just posting to show off this: http://i.imgur.com/u4fJxyt.jpg

Damn, this looks AWESOME!
Especially when you match the wallpaper.
What's the model number?
It's awesomely better than having two separate screens(1280x1024 and 1920x1080). Can you press a button and make that screen appear "double-wide" to the system, running windows(or linux) on native resolution?

Two full-featured(almost) PCs in one case and two screens in one. I think i need a two-in-one keyboard, with a hardware KVM switch built into:D

Can be the new Philips Monitor presented this summer? smile Looks pretty cool smile

The photo clearly says LG on the screen's bottom.
In fact, it's no new feature - my old(somewhere around 2002, i think)philips CRT TV can do that with television channels, it's a variation of traditional picture-by-picture.
I'm interested in that particular ultra-wide screen.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3612 2014-12-17 11:59:44

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The photo clearly says LG on the screen's bottom.
In fact, it's no new feature - my old(somewhere around 2002, i think)philips CRT TV can do that with television channels, it's a variation of traditional picture-by-picture.
I'm interested in that particular ultra-wide screen.

http://www.lg.com/us/monitors/lg-34UM95 … de-monitor

I knew you guys would appreciate how cool this looks -- I've been lurking this thread for awhile and when I came across this monitor I thought it would be rather novel to give this setup a try (haven't been this excited about Linux since the compiz days!).

Now I have a few questions about OVMF:

Why doesn't it boot my virtual disk? Secondly, what is OVMF_varsxxx.fd for?

Here is my franken-fig (there are other bits to it, but this is the meat and potatoes):

QEMU_PA_SAMPLES=8192 QEMU_AUDIO_DRV=pa \ $QEMU \ -boot menu=on \ -drive if=pflash,format=raw,readonly,file=/home/anthony/ovmf-x64/OVMF-pure-efi.fd \ -drive if=pflash,format=raw,file=/home/anthony/ovmf-x64/OVMF_VARS-pure-efi.fd \ -enable-kvm -cpu host,kvm=off -rtc base=localtime \ -m $MEM $HUGE -smp 8,sockets=1,cores=4,threads=4 \ -vga std \ -device intel-hda -device hda-output \ -device vfio-pci,host=$GPU,addr=04.0,multifunction=on,x-vga=on \ -device vfio-pci,host=$AUDIO,addr=04.1 \ -device pci-assign,host=06:00.0 \ -drive file=$DISK,cache=none,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -net none \ -vnc :0 -k en-us

Offline

#3613 2014-12-17 12:15:02

magemaster
Member
From: Brasil
Registered: 2010-11-24
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
magemaster wrote:

Someone else using with arch kernel?

uname -a

Linux 3.17.6-1-ARCH #1 SMP PREEMPT Sun Dec 7 23:43:32 UTC 2014 x86_64 GNU/Linux

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=e9f18698-c70a-41f6-b5c6-c84a8e7eac85 rw quiet rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:10c3,10de:0be3 intel_pstate=disable i915.enable_hd_vgaarb=1 intel_iommu=on,igfx_off quiet

Tried this way,  based on thearcherblog, gave me no error, but no screen

sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -cpu host,kvm=off -m 2048 -smp 2,sockets=1,cores=2 -serial none -parallel none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/ovmf_x64.bin \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/user/windows.img,cache=writeback,if=none,id=drive0,aio=native \ -drive file=/home/user/Win81.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /home/user/Downloads/virtio.iso \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -net none \ -vga none \ -boot menu=on \ -nographic

and this way, based on nbhs, gave me this error: KVM internal error. Suberror: 1 emulation failure

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I also got the black screen but someone suggested me to have this (and works for me smile):

[jgomez@mycolonialone etc]$ cat /etc/modprobe.d/qemu_kvm.conf options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 #options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Regards smile

Thank you, I installed with the emulated vga as primary and the passthrough as secondary to see what happens, could install nvidia drivers but got Code 43 error, which appears to be fixed with a patch from the 1st post.

PS: your qemu_kvm.conf has "intel_kvm" but in my machine the module name is "kvm_intel"

Offline

#3614 2014-12-17 13:20:13

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

magemaster wrote:
thearcherblog wrote:
magemaster wrote:

Someone else using with arch kernel?

uname -a

Linux 3.17.6-1-ARCH #1 SMP PREEMPT Sun Dec 7 23:43:32 UTC 2014 x86_64 GNU/Linux

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=e9f18698-c70a-41f6-b5c6-c84a8e7eac85 rw quiet rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:10c3,10de:0be3 intel_pstate=disable i915.enable_hd_vgaarb=1 intel_iommu=on,igfx_off quiet

Tried this way,  based on thearcherblog, gave me no error, but no screen

sudo qemu-system-x86_64 -enable-kvm -rtc base=localtime -cpu host,kvm=off -m 2048 -smp 2,sockets=1,cores=2 -serial none -parallel none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/ovmf_x64.bin \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/user/windows.img,cache=writeback,if=none,id=drive0,aio=native \ -drive file=/home/user/Win81.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /home/user/Downloads/virtio.iso \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -net none \ -vga none \ -boot menu=on \ -nographic

and this way, based on nbhs, gave me this error: KVM internal error. Suberror: 1 emulation failure

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I also got the black screen but someone suggested me to have this (and works for me smile):

[jgomez@mycolonialone etc]$ cat /etc/modprobe.d/qemu_kvm.conf options kvm ignore_msrs=1 options intel_kvm emulate_invalid_guest_state=0 options intel_kvm nested=1 options intel_kvm enable_shadow_vmcs=1 options intel_kvm enable_apicv=1 options intel_kvm ept=1 #options vfio_pci nointxmask=1 options vfio_iommu_type1 disable_hugepages=1 options vfio_iommu_type1 allow_unsafe_interrupts=1

Regards smile

Thank you, I installed with the emulated vga as primary and the passthrough as secondary to see what happens, could install nvidia drivers but got Code 43 error, which appears to be fixed with a patch from the 1st post.

PS: your qemu_kvm.conf has "intel_kvm" but in my machine the module name is "kvm_intel"

Happy that works for you smile

To be honest, I do not know anything about the qemu_kvm.conf and, even, when I made a few tests, is a mess because sometimes I get the error 43 and then I need to remove the driver and install it again (because setting the options back does not help until reinstall drivers)... so.... I'm not expert at all; probably some of the Gurus here can give you real information about what makes every option smile

Offline

#3615 2014-12-17 14:56:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
thearcherblog wrote:
Duelist wrote:

Damn, this looks AWESOME!
Especially when you match the wallpaper.
What's the model number?
It's awesomely better than having two separate screens(1280x1024 and 1920x1080). Can you press a button and make that screen appear "double-wide" to the system, running windows(or linux) on native resolution?

Two full-featured(almost) PCs in one case and two screens in one. I think i need a two-in-one keyboard, with a hardware KVM switch built into:D

Can be the new Philips Monitor presented this summer? smile Looks pretty cool smile

The photo clearly says LG on the screen's bottom.
In fact, it's no new feature - my old(somewhere around 2002, i think)philips CRT TV can do that with television channels, it's a variation of traditional picture-by-picture.
I'm interested in that particular ultra-wide screen.

My Dell U2913WM can do this too, but I've never used it.  You have to ask yourself if two 1280x1080 desktops, side-by-side w/o a seam is really all that useful for you.  In my case I do all my work on a Linux desktop and Windows is a test/toy.  I wouldn't sacrifice primary real estate for Windows.  Maybe the LG shown is 4k, but it still wouldn't make sense for my use case, no matter how pretty is is.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3616 2014-12-17 16:58:22

ebol4
Member
Registered: 2014-12-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I spent most of yesterday getting QEMU with KVM set up on my system. I compiled an up-to-date version of QEMU on my own. I'm getting Code 43 in Windows 7's device manager after installing the NVIDIA drivers (version 335.23, to avoid the KVM detection issues).

Specs:
ASUS z97-AR
i5 4690k
GTX 660

The HDMI audio device I passed through with the VGA device appears to work just fine! No yellow bang next to the audio device. IOMMU and Vt-d are supported by my motherboard and processor, so no issues there.

sudo /home/ian/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -M pc \ -m 4096 \ -cpu host,kvm=off \ -bios /usr/share/qemu/bios.bin -vga cirrus \ -smp cores=4,threads=1,sockets=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -hda /home/ian/windows.img -cdrom /home/ian/windows.iso \ -boot menu=on
BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=db818228-48ec-49b8-997b-163fa46c6ea0 ro quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:11c0,10de:0e0b vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7

Any ideas? Using QEMU 2.2.50.

Last edited by ebol4 (2014-12-17 17:00:05)

Offline

#3617 2014-12-17 17:01:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ebol4 wrote:

I spent most of yesterday getting QEMU with KVM set up on my system. I compiled an up-to-date version of QEMU on my own. I'm getting Code 43 in Windows 7's device manager after installing the NVIDIA drivers (version 335.23, to avoid the KVM detection issues).

Specs:
ASUS z97-AR
i5 4690k
GTX 660

The HDMI audio device I passed through with the VGA device appears to work just fine! No yellow bang next to the audio device. IOMMU and Vt-d are supported by my motherboard and processor, so no issues there.

sudo /home/ian/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -M pc \ -m 4096 \ -cpu host,kvm=off \ -bios /usr/share/qemu/bios.bin -vga cirrus \ -smp cores=4,threads=1,sockets=1 \ -device pci-assign,host=01:00.0 \ -device pci-assign,host=01:00.1 \ -hda /home/ian/windows.img -cdrom /home/ian/windows.iso \ -boot menu=on
BOOT_IMAGE=/boot/vmlinuz-3.13.0-32-generic root=UUID=db818228-48ec-49b8-997b-163fa46c6ea0 ro quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:11c0,10de:0e0b vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7

Any ideas? Using QEMU 2.2.50.

Why are you even expecting this to work?  Where are the instructions that say ignore everything in this thread, and just assign GeForce cards as secondary graphics with legacy KVM device assignment?  Has this ever worked?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3618 2014-12-17 17:45:19

magemaster
Member
From: Brasil
Registered: 2010-11-24
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thearcherblog wrote:
magemaster wrote:
thearcherblog wrote:

I also got the black screen but someone suggested me to have this (and works for me smile):

Linux 3.17.6-1-ARCH #1 SMP PREEMPT Sun Dec 7 23:43:32 UTC 2014 x86_64 GNU/Linux

Regards smile

Thank you, I installed with the emulated vga as primary and the passthrough as secondary to see what happens, could install nvidia drivers but got Code 43 error, which appears to be fixed with a patch from the 1st post.

PS: your qemu_kvm.conf has "intel_kvm" but in my machine the module name is "kvm_intel"

Happy that works for you smile

To be honest, I do not know anything about the qemu_kvm.conf and, even, when I made a few tests, is a mess because sometimes I get the error 43 and then I need to remove the driver and install it again (because setting the options back does not help until reinstall drivers)... so.... I'm not expert at all; probably some of the Gurus here can give you real information about what makes every option smile

BOOT_IMAGE=/vmlinuz-linux root=UUID=e9f18698-c70a-41f6-b5c6-c84a8e7eac85 rw quiet rdblacklist=nouveau nouveau.modeset=0 pci-stub.ids=10de:10c3,10de:0be3 intel_pstate=disable i915.enable_hd_vgaarb=1 intel_iommu=on,igfx_off quiet

Thank you, but It didn't work,
what I could do was boot using the cirrus emulated card and see the nvidia card as a secondary, then installed the latest driver and got a 43 error sad

I will try the OP way, patching and compilling stuff

Last edited by magemaster (2014-12-17 17:46:08)

Offline

#3619 2014-12-17 17:51:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
thearcherblog wrote:

Can be the new Philips Monitor presented this summer? smile Looks pretty cool smile

The photo clearly says LG on the screen's bottom.
In fact, it's no new feature - my old(somewhere around 2002, i think)philips CRT TV can do that with television channels, it's a variation of traditional picture-by-picture.
I'm interested in that particular ultra-wide screen.

My Dell U2913WM can do this too, but I've never used it.  You have to ask yourself if two 1280x1080 desktops, side-by-side w/o a seam is really all that useful for you.  In my case I do all my work on a Linux desktop and Windows is a test/toy.  I wouldn't sacrifice primary real estate for Windows.  Maybe the LG shown is 4k, but it still wouldn't make sense for my use case, no matter how pretty is is.


The actual screen is 3440x1440. So it makes it 1720x1440. Well, despite the weird aspect ratio, it's almost fullHD.
Comparing to the current setup...
I have 1280x1024 D-SUB-Only attached to GT610 fitted inside my case using MAGIC, and 1920x1080 DVI and D-SUB compatible.
Some time ago i used fullhd's video inputs for testing both passed GPUs(i've got two in crossfire, if you forgot), but it produced really weird glitches. So, right now my D-SUB input on fullhd is unplugged. I can't plug it to GT610 because... there is no damn space for it in the case. I can do some OTHER magic, but that would require a lot of mana.(changing the motherboard or designing and making a whole new case from scratch)
And so, i have my fullhd screen offline when there is no VM running, i can't watch some cinema or series or anime or youtube or whatever on it. Watching 1280x720p is fine for me, but it still makes me sad a little. And the other way, i can't plug my square screen into one of PT'd CF'd GPUs to have 3200x1080 "eyefinity" experience(i did it previously when i had real windows running on physical hardware, and it was fun).

Making two 1280x1080 screens and having the ability to quickly stretch them to the native resolution would be good. Having two 1720x1440 screens would be even more awesome, and having the ability to switch it to full 3440x1440 for cinema or work is totally awesome.

Also, it'd fill up my desk completely, as there is about 5cm gap between the upper bezel and the upper "deck" or whatever it's called. I can't use the wall mount because the square screens lacks VESA(it's ~10 years old), and i've got no proper wall behind them.

So, the question is: can you turn on/turn off that picture-by-picture feature quickly, switching the output to native resolution?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3620 2014-12-17 18:08:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

So, the question is: can you turn on/turn off that picture-by-picture feature quickly, switching the output to native resolution?

This seems to be pretty annoying on the Dell.  You can change one of the buttons to PBP (Picture-By-Picture), but it enables it and allows you to switch inputs.  AFAICT there's no way to turn it back off w/o going through the menu.  It also doesn't seem to do anything automatic with the screen resolution.  When I enable it my desktop just gets squashed to half the screen.  I suppose it would need to appear more like a monitor disconnected+reconnect with different EDID to do anything automatic.  No idea what the LG does obviously.  That resolution certainly opens up the possibilities a bit though.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3621 2014-12-17 18:29:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

So, the question is: can you turn on/turn off that picture-by-picture feature quickly, switching the output to native resolution?

This seems to be pretty annoying on the Dell.  You can change one of the buttons to PBP (Picture-By-Picture), but it enables it and allows you to switch inputs.  AFAICT there's no way to turn it back off w/o going through the menu.  It also doesn't seem to do anything automatic with the screen resolution.  When I enable it my desktop just gets squashed to half the screen.  I suppose it would need to appear more like a monitor disconnected+reconnect with different EDID to do anything automatic.  No idea what the LG does obviously.  That resolution certainly opens up the possibilities a bit though.

I expected exactly "EDID Change" behaviour. Well, sad thing that it doesn't work that way, just squishes.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3622 2014-12-17 23:32:17

ebol4
Member
Registered: 2014-12-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does Code 12 in Device Manager for an Nvidia card have a fix? Or could that be caused by a lot of different things?

edit: I also don't see any output on the attached monitor, including seabios output, which leads me to believe that it's an issue with my configuration, not Windows.

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ahci,id=ahci0 \
-drive file=/home/ian/windows.img,id=disk,if=none -device ide-drive,bus=ahci0.0,drive=disk \
-drive file=/home/ian/windows.iso,id=isocd,if=none -device ide-cd,bus=ahci0.1,drive=isocd \
-boot menu=on

Last edited by ebol4 (2014-12-17 23:45:18)

Offline

#3623 2014-12-18 00:13:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ebol4 wrote:

Does Code 12 in Device Manager for an Nvidia card have a fix? Or could that be caused by a lot of different things?

edit: I also don't see any output on the attached monitor, including seabios output, which leads me to believe that it's an issue with my configuration, not Windows.

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ahci,id=ahci0 \
-drive file=/home/ian/windows.img,id=disk,if=none -device ide-drive,bus=ahci0.0,drive=disk \
-drive file=/home/ian/windows.iso,id=isocd,if=none -device ide-cd,bus=ahci0.1,drive=isocd \
-boot menu=on

That's looking better, though personally I still discourage q35 for Windows guests.  You showed in the first post that you're using the i915.enable_hd_vgaarb=1 option, did you actually patch your kernel to include that patch?  Primary VGA assignment still doesn't work with Intel IGD as host without it.  If you want to avoid patching anything, use a GPU and guest with UEFI support and go the OVMF route.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3624 2014-12-18 00:24:30

ebol4
Member
Registered: 2014-12-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

That's looking better, though personally I still discourage q35 for Windows guests.  You showed in the first post that you're using the i915.enable_hd_vgaarb=1 option, did you actually patch your kernel to include that patch?  Primary VGA assignment still doesn't work with Intel IGD as host without it.  If you want to avoid patching anything, use a GPU and guest with UEFI support and go the OVMF route.

I did not patch my kernel. I'm pretty sure my GPU doesn't support UEFI though, so it looks like I'm going to have to go that way. This involves pulling the kernel source and rebuilding it completely, right? Or am I just rebuilding one module?
edit: I got it figured out. Building kernel now.

Also, you said you discourage q35 for Windows guests. Is there another one you recommend instead?

Last edited by ebol4 (2014-12-18 01:10:41)

Offline

#3625 2014-12-18 01:18:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ebol4 wrote:
aw wrote:

That's looking better, though personally I still discourage q35 for Windows guests.  You showed in the first post that you're using the i915.enable_hd_vgaarb=1 option, did you actually patch your kernel to include that patch?  Primary VGA assignment still doesn't work with Intel IGD as host without it.  If you want to avoid patching anything, use a GPU and guest with UEFI support and go the OVMF route.

I did not patch my kernel. I'm pretty sure my GPU doesn't support UEFI though, so it looks like I'm going to have to go that way. This involves pulling the kernel source and rebuilding it completely, right? Or am I just rebuilding one module?
edit: I got it figured out. Building kernel now.

Also, you said you discourage q35 for Windows guests. Is there another one you recommend instead?

The default machine type is 440FX, there's an example of using it here:

http://vfio.blogspot.com/2014/08/dual-v … adeon.html

Q35 really buys you nothing but complication and poor disk performance with AHCI.  Also, if you ever intend to migrate to using libvirt to manage the guest, it gets even worse.  I think I gave an example within the last few weeks of a simple wrapper script for QEMU that can be used with libvirt if all you need to do is add the x-vga=on option to vfio-pci VGA device.  With q35 people end up requiring <qemu:arg> extensions in the guest xml that hides the assigned device and setup of resources for it from libvirt.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3626 2014-12-18 02:08:50

ebol4
Member
Registered: 2014-12-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh my god, aw, it works. I've been trying to get this running for so long. I get BIOS output and it boots into Windows. Checking to make sure the video drivers are functional... edit: YES THEY ARE!

In case anyone is having a similar issue with Code 12 with Intel Host/Nvidia Guest -- PATCH THE KERNEL!!!!

Now I just need to figure out how to get some decent input available on it... I'm currently doing "sendkey ____" in the QEMU monitor through TightVNC to navigate Windows....

Thank you so much for your help.

Last edited by ebol4 (2014-12-18 02:37:46)

Offline

#3627 2014-12-18 10:34:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ebol4 wrote:

Oh my god, aw, it works. I've been trying to get this running for so long. I get BIOS output and it boots into Windows. Checking to make sure the video drivers are functional... edit: YES THEY ARE!

In case anyone is having a similar issue with Code 12 with Intel Host/Nvidia Guest -- PATCH THE KERNEL!!!!

Now I just need to figure out how to get some decent input available on it... I'm currently doing "sendkey ____" in the QEMU monitor through TightVNC to navigate Windows....

Thank you so much for your help.

You can add -device qxl somewhere in the middle to make qemu show you an SDL surface where it can grab keyboard and mouse. But be aware, it lags seriously in source-based games, sometimes mod-keys(shift, ctrl, etc.) stick and the mouse may stop responding if you press some weird buttons(like win+alt+smth). I've been using this BAD input for a month or so because i'm lazy as hell.

Another options are:
VNC. I think there was a way of getting it working via qemu's built-in, so you could rule the keyboard while it's in BIOS menu, so no software required on guest.
Synergy(it doesn't work in another context, like UAC windows popping up)
Spice(my fedora doesn't have a client in the repos)
Separate keyboard-mouse pair connected through passed-through USB controller(or just as usb devices).
Hardware KVM(heh!)-switch.

Last edited by Duelist (2014-12-18 10:35:59)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3628 2014-12-18 13:16:43

kidchunks
Member
Registered: 2014-12-03
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Setup: Debian, 3.14 Kernel with qemu 2.1.2

Doing some testing with libvirt (xml) config and I'm getting the following when trying to start my guest using virsh.

2014-12-18T13:11:47.567800Z qemu-system-x86_64: -device vfio-pci,host=85:00.0,addr=03.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe5acea33b0, 0x100000000, 0x40000000, 0x7fe555800000) = -12 (Cannot allocate memory)

In dmesg (a few of htese):

[ 350.663041] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

I looked around the thread and saw others have similar issues when trying to run as a diff user other than root. The problem is that I am as root and still cannot launch the guest.

I checked the RLIMIT using ulimit and got the following:

root@local:~# ulimit unlimited

/etc/libvirt/qemu.conf has the user/group set as well as the cgroup_device_acl.

libvirt sample config:

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=85:00.0,addr=09.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=85:00.1,addr=09.1'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/seabios/bios.bin'/> </qemu:commandline>

Again, not sure why even as root, I cannot run the guest using virsh. Running qemu directly works.

Last edited by kidchunks (2014-12-18 13:40:50)

Offline

#3629 2014-12-18 15:44:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kidchunks wrote:

libvirt sample config:

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=85:00.0,addr=09.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=85:00.1,addr=09.1'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/seabios/bios.bin'/> </qemu:commandline>

You're causing your own problems with this <qemu:arg> crap.  The bios option is not needed.  The vfio-pci devices should be created as <hostdev> options.  To add the x-vga=on option, use a wrapper script like shown in this post: https://bbs.archlinux.org/viewtopic.php … 1#p1475541  Set the <emulator> to the wrapper.  All the problems you're having are because you're hiding the assigned device from libvirt and therefore needing to hack the libvirt setup to allow the VM the permissions that libvirt doesn't know it needs.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3630 2014-12-18 20:53:29

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was looking for the method to build the kernel from first post. Here is a little reminder if anyone had forgotten how to do it like myself.

Download archive, extract it and cd into it, then:

makepkg -s pacman -U linux-mainline-*

Imho, it would be great if OP could add it to his first post.

Last edited by Nesousx (2014-12-18 21:09:24)

Offline

#3631 2014-12-18 21:31:06

kidchunks
Member
Registered: 2014-12-03
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You're causing your own problems with this <qemu:arg> crap.  The bios option is not needed.  The vfio-pci devices should be created as <hostdev> options.  To add the x-vga=on option, use a wrapper script like shown in this post: https://bbs.archlinux.org/viewtopic.php … 1#p1475541  Set the <emulator> to the wrapper.  All the problems you're having are because you're hiding the assigned device from libvirt and therefore needing to hack the libvirt setup to allow the VM the permissions that libvirt doesn't know it needs.

Thanks for the clarification and link to the wrapper script. I tried searching for a way to build the script after seeing this post from you but came up short. It's strange that this worked using ubuntu 14.04 (qemu args in xml file). It could be that ubuntu does something different regarding the implementation.

Anyways, I'll stick to using hostdev for libvirt and remove the qemu args. smile

Last edited by kidchunks (2014-12-18 21:40:14)

Offline

#3632 2014-12-18 22:56:54

magemaster
Member
From: Brasil
Registered: 2010-11-24
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just to be clear, the people who can do passthrough with stock kernel are not using intel onboard video as primary, right?
Tried a lot with arch stock kernel(3.17) and only was able to have image on the second monitor after
patching a kernel(3.18-ck + i915 vga arbiter patch).

Offline

#3633 2014-12-18 22:59:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

magemaster wrote:

Just to be clear, the people who can do passthrough with stock kernel are not using intel onboard video as primary, right?

Or avoiding the whole VGA problem by using OVFM+UEFI


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3634 2014-12-19 06:01:14

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
magemaster wrote:

Just to be clear, the people who can do passthrough with stock kernel are not using intel onboard video as primary, right?

Or avoiding the whole VGA problem by using OVFM+UEFI

Yes, I can confirm this.

Mine is working with intel on host and NVIDIA on guest using stock kernel but OVFM and UEFI and windows 8.

Offline

#3635 2014-12-19 13:13:31

magemaster
Member
From: Brasil
Registered: 2010-11-24
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My vga does not support UEFI and I can't change the card

8400GS
Valid ROM signature found @0h, PCIR offset 188h
    PCIR: type 0, vendor: 10de, device: 10c3, class: 030000
    PCIR: revision 0, vendor revision: 1
    Last image

Offline

#3636 2014-12-19 13:57:59

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If followed the guide, installed windows 8, and it kinda worked, execept the nividia driver. I enabled "kvm=off" on the cpu parameter, and installed the driver (from a cirrus vga adapter, with the card passthroug as secondary). It installed fine, but now i'm getting BSOD's from the VM. The error is "SYSTEM_SERVICE_EXCEPTION" and it's originating from the nvidia driver.

I then tried installing windows 7, but that's just giving me this whenever i try to boot. the installation went fine (this is without any passthrough).

KVM internal error. Suberror: 1 emulation failure EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080 ESI=0025db2a EDI=0007db2a EBP=00007c00 ESP=00000200 EIP=000000ca EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0020 00000200 0000ffff 00009300 CS =b000 000b0000 0000ffff 00009f00 SS =0020 00000200 0000ffff 00009300 DS =0020 00000200 0000ffff 00009300 FS =0020 00000200 0000ffff 00009300 GS =0020 00000200 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 002b0000 0000001f IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

Here's my script for launching the vm:

vfio-bind 0000:03:00.0 0000:03:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga cirrus \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -drive file=/home/delusional/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/delusional/vm/install.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -usb -usbdevice host:1038:1361 -usbdevice host:1532:010d \ -net nic -net bridge,br=br0 \ -boot menu=on

I am aware that this would not enable the passthrough anyway (seeing as the nividia card is secondary) but at least it should boot right?

Offline

#3637 2014-12-19 14:01:13

magemaster
Member
From: Brasil
Registered: 2010-11-24
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

If followed the guide, installed windows 8, and it kinda worked, execept the nividia driver. I enabled "kvm=off" on the cpu parameter, and installed the driver (from a cirrus vga adapter, with the card passthroug as secondary). It installed fine, but now i'm getting BSOD's from the VM. The error is "SYSTEM_SERVICE_EXCEPTION" and it's originating from the nvidia driver.

I then tried installing windows 7, but that's just giving me this whenever i try to boot. the installation went fine (this is without any passthrough).

KVM internal error. Suberror: 1 emulation failure EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080 ESI=0025db2a EDI=0007db2a EBP=00007c00 ESP=00000200 EIP=000000ca EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0020 00000200 0000ffff 00009300 CS =b000 000b0000 0000ffff 00009f00 SS =0020 00000200 0000ffff 00009300 DS =0020 00000200 0000ffff 00009300 FS =0020 00000200 0000ffff 00009300 GS =0020 00000200 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 002b0000 0000001f IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

Here's my script for launching the vm:

vfio-bind 0000:03:00.0 0000:03:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga cirrus \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -drive file=/home/delusional/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/delusional/vm/install.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -usb -usbdevice host:1038:1361 -usbdevice host:1532:010d \ -net nic -net bridge,br=br0 \ -boot menu=on

I am aware that this would not enable the passthrough anyway (seeing as the nividia card is secondary) but at least it should boot right?


I was having the same error before I patched the kernel.

Offline

#3638 2014-12-20 10:41:15

ailcp
Member
Registered: 2014-12-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am trying to passthrough the GPU of the following computer:

CPU: Athlon x4 860k (no iGPU)
Motherboard: ASRock FM2A88M Extreme4+
Primary GPU for host: ATI Rage XL (in PCI slot)
GPU for passthrough: MSI GTX 750 Ti OC v1 (in PCIE 3.0 slot)
Host OS: Fedora 21 (Stock Kernel 3.17.4)
Guest OS: Windows 8.1 64 bit

Kernel boot parameter in grub:

pci-stub.ids=10de:1380,10de:0fbc iommu=pt

Libvirt XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>88b0dc9a-8831-11e4-b116-123b93f75cba</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> <bootmenu enable='yes'/> <smbios mode='host'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/Windows81.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/disk/by-id/ata-WDC_WD10EALX-759BA1_WD-WCATR8239826'/> <target dev='hdb' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='pci' index='1' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='28:8F:A0:07:98:C4'/> <source bridge='br0'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc246'/> <address bus='1' device='3'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52e'/> <address bus='1' device='4'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/> </qemu:commandline> </domain>

Symptoms:
When I boot the guest the first time, the display can output the OVMF screen and the whole boot process successfully.  After booting into Windows, I installed Nvidia driver 334.89 and hope to avoid code 43 error.  After installation finished, the name "GTX 750 Ti" was shown correctly with a yellow exclamation mark in Device Manager and still come with code 43 error.  The screen resolution was stuck at 800x600 and the option to change resolution was greyed out.

I have looked at the google doc for reporting GPU passthrough results.  It seems that there is no success case of passing through a Nvidia GPU on FM2/FM2+ motherboard + CPU combo.  If someone had successfully passthrough a Nvidia GPU on FM2/FM2+ CPU, would you please share your results.  Since there is some success case of passthrough AMD GPU on FM2/FM2+ CPU, should I just give up on the GTX 750 Ti and buy a AMD GPU instead?  Please give me some advice?  Thank you very much!

Offline

#3639 2014-12-20 11:08:22

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I encountered a problem while running my VM: it freezed and when I tried to stop it with Ctrl+C, I got this message in syslog:

WARNING: CPU: 3 PID: 2679 at fs/block_dev.c:67 bdev_inode_switch_bdi+0x77/0x80() Modules linked in: xt_mac xt_iprange xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_connmark xt_physdev xt_state xt_multiport CPU: 3 PID: 2679 Comm: qemu-system-x86 Tainted: G D O 3.18.1 #1 Hardware name: MSI MS-7758/ZH77A-G43 (MS-7758), BIOS V1.6 10/25/2012 0000000000000000 0000000000000009 ffffffff81a33879 0000000000000000 ffffffff8112cadb ffff88031cc891b0 ffff88031cc89238 ffffffff81ddc400 0000000000000000 000000000006001f ffffffff81286367 ffff88031cc890c0 Call Trace: [<ffffffff81a33879>] ? dump_stack+0x41/0x51 [<ffffffff8112cadb>] ? warn_slowpath_common+0x8b/0xc0 [<ffffffff81286367>] ? bdev_inode_switch_bdi+0x77/0x80 [<ffffffff8128672c>] ? __blkdev_put+0x6c/0x1a0 [<ffffffff812869d5>] ? blkdev_close+0x25/0x30 [<ffffffff81250fad>] ? __fput+0xcd/0x1f0 [<ffffffff8114807f>] ? task_work_run+0x9f/0xe0 [<ffffffff8105bbb9>] ? do_notify_resume+0x69/0xa0 [<ffffffff81250e9b>] ? fput+0x4b/0x90 [<ffffffff81a3cba0>] ? int_signal+0x12/0x17

Host info:
- kernel 3.18.1 (without patches)
- IGD as primary graphic device
- qemu 2.2.0
- OVMF (latest svn)
- LVM used for guest disk

Guest info:
- Windows 8.1 UEFI
- nVidia geForce GTX 650 passed as primary graphic device
- virtio drivers for disk

Script to launch qemu:

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host,hv_relaxed,hv_time,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -vga none -rtc base=localtime -k it \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/qemu/windows,id=disk,format=raw,if=none -device scsi-hd,bus=scsi.0,drive=disk \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/root/OVMF_VARS_Win8.1.fd

Any thoughts about the kernel oops?

Offline

#3640 2014-12-20 21:39:50

maxqia
Member
Registered: 2014-12-20
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

[*]Use pci-stub

In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub.

NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

mkinitcpio -p linux-mainline

lspci

07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

lspci -n

07:00.0 0300: 1002:6719 <-- radeon 6950 07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

[/*]

By the Way, "lspci" & "lspci -n" Can be replaced with a single "lspci -nn" command

Offline

#3641 2014-12-21 18:04:13

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ailcp wrote:

Hi,

I am trying to passthrough the GPU of the following computer:

CPU: Athlon x4 860k (no iGPU)
Motherboard: ASRock FM2A88M Extreme4+
Primary GPU for host: ATI Rage XL (in PCI slot)
GPU for passthrough: MSI GTX 750 Ti OC v1 (in PCIE 3.0 slot)
Host OS: Fedora 21 (Stock Kernel 3.17.4)
Guest OS: Windows 8.1 64 bit

I have looked at the google doc for reporting GPU passthrough results.  It seems that there is no success case of passing through a Nvidia GPU on FM2/FM2+ motherboard + CPU combo.  If someone had successfully passthrough a Nvidia GPU on FM2/FM2+ CPU, would you please share your results.  Since there is some success case of passthrough AMD GPU on FM2/FM2+ CPU, should I just give up on the GTX 750 Ti and buy a AMD GPU instead?  Please give me some advice?  Thank you very much!

1. I really like your CPU. I want that CPU, actually. Can you tell me if there's PCI-E 3.0 support there? Google tells me that there isn't one. What does your lspci tell about it?
2. I have Athlon X4 750K, trinity-based FM2-socket paired with F2A55 motherboard. I am passing through TWO HD7750 with VGA and it works. As a host GPU i have nvidia gt610. Due to ASUS, there is MAJOR problems with IOMMU realization and memory accesses, but it works as long as i have my firefox running during the startup of VM. It reboots perfectly too. There appear to be some slight performance drawbacks related to bandwidth-issues, but i can't find the reasons or any info on how the hell crossfire works. I can post my qemu's launching script if you're interested, but it's awful. Oh, and i have fedora 21 too. Check if you have git-repo of OVMF, it helped me a lot when i fiddled with win7+UEFI+GPT setup.
3. Continue digging.
4. Awesome host gpu, i definitely should dig into old ATI cards, since only PCI ones i have right now are various S3 cards, which won't work.

P.S. I'm really-really tired right now, so my language may be a little bit messy, since my head's contents are messy.

Last edited by Duelist (2014-12-21 18:05:29)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3642 2014-12-22 03:47:57

ailcp
Member
Registered: 2014-12-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

1. I really like your CPU. I want that CPU, actually. Can you tell me if there's PCI-E 3.0 support there? Google tells me that there isn't one. What does your lspci tell about it?

Hi Duelist,

I think there is PCI-E 3.0 support.  lspci shows:
LnkCap: Port #0, Speed 8GT/s, Width x16

Actually I have read your success case about passing through the HD7750, so I am thinking to buy a single slot XFX R7 250e which should be equivalent to the HD7750.  However, I really want to make the GTX 750 Ti work because I really like its low power consumption.

Since I am a Linux Newbie, hopefully someone with the FM2/FM2+ CPU and Nvidia GPU combo can share their experience on resolving this dreadful code 43 error.  Thank you.

Offline

#3643 2014-12-22 04:26:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ailcp wrote:

I have looked at the google doc for reporting GPU passthrough results.  It seems that there is no success case of passing through a Nvidia GPU on FM2/FM2+ motherboard + CPU combo.  If someone had successfully passthrough a Nvidia GPU on FM2/FM2+ CPU, would you please share your results.  Since there is some success case of passthrough AMD GPU on FM2/FM2+ CPU, should I just give up on the GTX 750 Ti and buy a AMD GPU instead?  Please give me some advice?  Thank you very much!

I don't know why you're getting a Code 43 and don't have or want any experience on an FM2/+ system, but a few comments on your config:

a) Your <qemu:arg> options can be avoided, libvirt should support OVMF on F21, it certainly does on F20 w/ the virt-preview repo enabled - http://vfio.blogspot.com/2014/09/libvir … -ovmf.html

b) Why are you defining extra PCI bridges in slots 3 & 4?

c) Why aren't you using your distro version of QEMU?  Again, F20 + virt preview repo works just fine on my Intel box with GTX750 and OVMF

d) <smbios mode='host'/>  Why?

I don't know if any of that will help, but better than nothing I suppose.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3644 2014-12-22 05:44:54

ailcp
Member
Registered: 2014-12-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I don't know why you're getting a Code 43 and don't have or want any experience on an FM2/+ system, but a few comments on your config:

Thank you so much for aw's advice.  I guess I should give up on this Nvidia GPU and maybe buy the HD7750 to try.

Offline

#3645 2014-12-22 12:33:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ailcp wrote:
aw wrote:

I don't know why you're getting a Code 43 and don't have or want any experience on an FM2/+ system, but a few comments on your config:

Thank you so much for aw's advice.  I guess I should give up on this Nvidia GPU and maybe buy the HD7750 to try.

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?


BTW, aw, does libvirt gets a copy of OVMF_VARS when ovmf is specified via qemu:arg section? Wasn't there some other way?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3646 2014-12-22 14:15:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

BTW, aw, does libvirt gets a copy of OVMF_VARS when ovmf is specified via qemu:arg section? Wasn't there some other way?

Not the way it's specified there, that's using the unified OVMF image that contains both CODE and VARS.  The guest doesn't get to persistently store any EFI variables.  It is possible to specify separate CODE and VARS with <qemu:arg>, but libvirt won't automatically do it for you.  libvirt doesn't parse anything specified via <qemu:arg>, it just blindly passes it through.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3647 2014-12-22 17:54:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

BTW, aw, does libvirt gets a copy of OVMF_VARS when ovmf is specified via qemu:arg section? Wasn't there some other way?

Not the way it's specified there, that's using the unified OVMF image that contains both CODE and VARS.  The guest doesn't get to persistently store any EFI variables.

But that's not so crucial. Or is it?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3648 2014-12-22 17:57:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:
Duelist wrote:

BTW, aw, does libvirt gets a copy of OVMF_VARS when ovmf is specified via qemu:arg section? Wasn't there some other way?

Not the way it's specified there, that's using the unified OVMF image that contains both CODE and VARS.  The guest doesn't get to persistently store any EFI variables.

But that's not so crucial. Or is it?

Depends on the guest and whether it can manage to boot without them.  In most cases I've seen, it works fine without VARs.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3649 2014-12-23 08:53:28

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys! Thanks for this helpful thread.

I managed to start VM with such hardware:

OS: Debian Wheezy with no patches (3.18.1 self compiled kernel)
MB: GA-Z97M-WIFI
Host gpu: Intel HD (unplugged all actions through ssh)
Guest: Asus GTX 670

qemu-system-x86_64 -nographic -enable-kvm -M q35 -m 1024 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -usb -usbdevice host:099a:0721 -usbdevice host:09da:9090 \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/root/windows.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk

But i struggling with several issues. At first i used Q35 sata controller but where was to many BSODs. So i switched to piix4-ide. VM becomes more stable and even faster, but in both cases i cannot install nvidia video driver VM simply freezes. I am using Windows 7 x64. Also i tried to install driver manually but windows cannot find specific file for gtx 670 but it managed to identify video card correctly. I need some help here smile Please smile


Also a have a lot of such messages but they are refers for some problems with my usb devices. For some reason my keyboard resets many times and shifts picture on monitor to right. After several resets and shifts on monitor all calms down.

Dec 23 03:15:21 hypervisor kernel: [17266.255391] usb 1-10: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes Dec 23 03:15:21 hypervisor kernel: [17266.408915] usb 1-7: reset full-speed USB device number 2 using xhci_hcd Dec 23 03:15:21 hypervisor kernel: [17266.409015] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 1. Dec 23 03:15:21 hypervisor kernel: [17266.409111] usb 1-7: hub failed to enable device, error -22 Dec 23 03:15:21 hypervisor kernel: [17266.561631] usb 1-7: reset full-speed USB device number 2 using xhci_hcd Dec 23 03:15:21 hypervisor kernel: [17266.561743] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 1. Dec 23 03:15:21 hypervisor kernel: [17266.561852] usb 1-7: hub failed to enable device, error -22 Dec 23 03:15:21 hypervisor kernel: [17266.714487] usb 1-7: reset full-speed USB device number 2 using xhci_hcd Dec 23 03:15:21 hypervisor kernel: [17266.726624] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88041a0cd840 Dec 23 03:15:21 hypervisor kernel: [17266.726747] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff88041a0cd7c0

Offline

#3650 2014-12-23 10:05:47

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to use OVMF, or patch kernel.

That dmesg, looks normal when u pass usb devices like that.

Last edited by slis (2014-12-23 10:06:14)

Offline

#3651 2014-12-23 10:28:38

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

You need to use OVMF, or patch kernel.


Thanks! What patches i should install ?

Offline

#3652 2014-12-23 11:49:51

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://lkml.org/lkml/2014/5/9/517

This one.

Are you having latest qemu? whezzy's is too old i guess.

Offline

#3653 2014-12-23 11:52:46

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everybody,

I was watching this thread from its first post, bud had very little success passing through my cards in the beginning. Also didn't beg for help because I'm doing this on ubuntu (please, don't laugh... too much) and tried to figure things out on my own. Thanks to so many useful advices in this thread I have learnt a lot, and also tried a few successful configurations. So thanks all for the contributions.
My actual working configuration:

Host:
- FX-8350, Sabertooth 990FX, 2xNV660Ti 3GB, 32GB RAM
- SSD Samsung 840Pro, Adaptec 2405 RAID0, And a few various big drives
- Ubuntu 15.04
- Self compiled kernel (based on arch config) 3.18.1, 1000HZ, voluntary preemption, builtin pci-back, and other tweaks; works also with stock ubuntu kernel if pci-back is in initramfs, but everything is slower

BOOT_IMAGE=/boot/vmlinuz-3.18.1-custom root=UUID=7ca2c85e-f29e-404d-91c7-92ad1e087465 ro splash ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 iommu=pt iommu=1 xen-pciback.hide=(06:00.0)(06:00.1) default_hugepagesz=1G hugepagesz=1G panic=10 hugepages=6 isolcpus=2-7 vt.handoff=7

- qemu 2.2.50 with DEVID patch
- propietery nvidia drivers for primary card on host

Guest(s):
Almost completely through libvirt (thanks to aws wrapper script its pretty neat), I have just the pci-back > vfio-pci in init script (probably can be done in libvirt pre-start script)
Im also using taskset in libvirt event script for compacting all running processes to cores 0-1 and releasing them to all cores after guest shutdown
aws qemu wrapper script was the last thing to tidy up libvirt config and solve the privilege mess, genial:

dzon@xenhost:~$ cat /usr/bin/qemu-vfio #!/bin/sh exec qemu-system-x86_64 `echo "\$@" | sed 's|06:00.0|06:00.0,x-vga=on,x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/usr/share/qemu/NVIDIA.QuadroK4000.3072.120813.rom|g'`

Win8.1, libvirt, xml:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Win81</name> <uuid>b61f94a6-08de-455b-9073-95c0c3316501</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <memoryBacking> <hugepages/> </memoryBacking> <vcpu placement='static' cpuset='0-7'>6</vcpu> <iothreads>2</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='7'/> <emulatorpin cpuset='0-1'/> <iothreadpin iothread='1' cpuset='0-1'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader readonly='yes' type='pflash'>/mnt/home/dzon/App/virt/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/Win81_VARS.fd</nvram> <bios useserial='yes' rebootTimeout='0'/> </os> <features> <acpi/> <apic eoi='on'/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> <pvspinlock state='on'/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='6' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>destroy</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-vfio</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/> <source file='/home/VMSSD/Win81.qcow2'/> <target dev='sda' bus='scsi'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/dev/sda2'/> <target dev='sdc' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/home/VMSSD/Win81Data.qcow2'/> <target dev='sdd' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/home/dzon/VM/Win81q.raw'/> <target dev='sde' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='4'/> </disk> <controller type='usb' index='0' model='nec-xhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <driver queues='6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:a5:e1:5a'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='8192' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_PA_SAMPLES' value='1024'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/> </qemu:commandline> </domain>
dzon@xenhost:~$ cat /etc/libvirt/hooks/qemu #!/bin/bash logPath=/home/dzon/libvirt_start echo "skript_start" $1 $2 >> $logPath if [ $2 == "start" ] ; then echo "start" >> $logPath for i in `pgrep -v rcu`; do taskset -pc 0-1 $i >> $logPath; done fi if [ $2 == "release" ] ; then echo "shutdown" >> $logPath for i in `pgrep -v rcu`; do taskset -pc 0-7 $i >> $logPath; done fi exit 0

What works:
- primary spice graphics with keyboard/mouse grab and usb redirection
- usb redirection (I'm using a sript forr ssh-ing into host through putty and disconnect my redirected keyb/mouse from guset with doubleclick, the same in host for redirection back in live guest)
- secondary passed through NV660Ti quadrified to K4000 with latest drivers in guest
- GPU performance 100% (weirdly sometimes above this as compared to native win install, probably some timing issues)
- CPU performance allmost 100% (6 cores), depends on applications
- HDD performace is great, also using partition passthrough, but not as boot drive - there are references in this thread on how to achieve that
- audio through pulseaudio daemon without a need to have connected client, also the HD audio on card is working fine
- no BSODs or lockups at all, very stable

What doesnt:
- in latest OMVF config the machine isn’t able to reboot, instead a new boot it just shuts down (correctly though)
- one anomaly is that without quadryfying my card the guest thinks that it is in pcieX1 mode, although on host I see it in full x16, after changing IDs and ROM for guest everything is ok
- there is a lot of host kernel time in some games during loading (not iowait) which causes temporary lagging, it seems according to perf that it is caused by _raw_spin_lock which could hint on multiple guest threads racing for timer gets - here I'm stuck but its not a game breaker. According to some of the posts here this could be AMD specific. Still looking for solution, if its possible.

Also tried and worked combinations:
linux guests, Win7, i440fx, q35, qemu cmd line, few low-end AMD and Nvidia cards

Some of the config parts maybe leftovers from previous tests, not everything is necessary for working configuration
Hope some of this info is usefull. Thanks all.

edit:
corrected module name to pci-back - thats what I'm using because of the two identical cards, added (dirty) hook script

Last edited by JohnyPea (2014-12-23 12:36:36)

Offline

#3654 2014-12-23 16:10:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:

  <on_reboot>destroy</on_reboot>
- in latest OMVF config the machine isn’t able to reboot, instead a new boot it just shuts down (correctly though)

Errr, isn't that's the way it's supposed to work? Also, great setup, great info. Thanks. I think i'm going to adapt some of your settings for my system.

By the way, what benchmarks did you use to check CPU and GPU performance?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3655 2014-12-23 18:15:10

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
JohnyPea wrote:

  <on_reboot>destroy</on_reboot>
- in latest OMVF config the machine isn’t able to reboot, instead a new boot it just shuts down (correctly though)

Errr, isn't that's the way it's supposed to work? Also, great setup, great info. Thanks. I think i'm going to adapt some of your settings for my system.

By the way, what benchmarks did you use to check CPU and GPU performance?

OMFG, i'm an idiot. Time to learn how to read again. Thank you very much.
I tried a lot of benchmarks, but basic synthetic ones were SuperPI, Furmark, Fluidmark. For drive performance ATTO disk benchmak. In Win7, I was comparing also builtin performance evaluation. (its possible in Win8, but its too much hassle for me). Also for guite a while I was using benchmark with Metro: Last Light - when the physics engine was acting weird, it is heavy on resources and most errors in machine config manifested even when synthetic benchmarks were running fine.
Noticed just now, that I left a Virtio SCSI disk types in the mentioned guest config. This way the iothreads config part isn't working (only with virtio). AFAIK the SCSI is goot only for trim operations in win guest and single virtual controller for multiple drives for now. Not performance gains. Virtio type is probably a better choice.
If you find anything usefull, I'm glad I could help.
edit: added ATTO benchmark

Last edited by JohnyPea (2014-12-23 18:36:24)

Offline

#3656 2014-12-24 06:47:15

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to install latest nvidia drivers (344.75) on Windows 7 (32 bit) and everything looks ok. Still trying to install driver on Windows 7 64 bit. Still using unpatched kernel and qemu (2.1) from debian wheezy-backports repository.

P.S. Simple tutorial how to compile kernel with optimisations for vm in debian  http://edencomputing.com/index.php/2014 … ebian-way/

Last edited by kainet (2014-12-24 07:06:03)

Offline

#3657 2014-12-24 07:04:58

ailcp
Member
Registered: 2014-12-20
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Offline

#3658 2014-12-24 10:17:55

qquark
Member
Registered: 2014-12-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I'm thinking of doing a similar setup, but before starting messing with my system, I wanted to make sure that I'm not wasting my time; I have a few questions that someone here might be able to answer, as I wasn't able to google the answers unfortunately:


I currently have a P6T motherboard with shitty DMAR tables, changing it is not an option, and I sent a support request to Asus, but I don't have any satisfying answer yet (and probably won't get one). There are rumors of a "beta" bios working, but trying to find any info about it seems like a dead end from what I saw for now.

I noticed that apparently one of the RMRR entries is correct, and the other one is not.
Since this is very easy to do, I enabled intel_iommu=on to see what happens, and Linux tells me that while setting up identity mappings it detected that my bios is broken (true, true ...); *however*, after a lengthy log full of call stacks about the devices it tried to map at the second RMRR entry, I get a log showing success (or at least no error) for those same devices at the first RMRR entry.

My first question from this result is whether I have a chance of getting VGA passthrough working in those conditions (?).
Can Linux manage to work with only one entry by ignoring the error in the other ? (assuming the first one is correct AND working) ? And Is there a way to override this entry (I do not know what the RMRR actually represents, but I suspect the answer would be negative).
Also, in my dmesg log, I only find this identity mapping happening for the USB controller devices (or subdevices ?), should I also expect to see my current graphics card there ?


Second topic: I see that most people are generally using setups with {intel,ATI}+{nvidia,ATI}, but I would like to know if there is any problem with an nvidia+nvidia setup (?), as I only have a spare 9800GT I can try in addition to my GTX660, and do not have any recent extra ATI card, nor an integrated intel GP (i7 920). From what I thought I understood, just being able to unbind/rebind the appropriate card should be enough to remove any conflict between host and guest, why are people also blacklisting drivers ? Do they interfere even though they're not bound to the card ? Or is it just an easier way to do boot-time non-binding ?


Thanks for any help in advance !

Offline

#3659 2014-12-24 10:32:12

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can only help you with second topic.
You can use any combination, but you need to patch nvidia host driver for nvidia+nvidia etc... (same/similar goes for radeon).
People blacklist driver instead of using pci-stub I guess to achieve same effect easier (works if u don't have both cards that use same driver).

Offline

#3660 2014-12-24 10:48:34

qquark
Member
Registered: 2014-12-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I see, thanks !

That's a step in the right direction smile

Offline

#3661 2014-12-24 11:50:02

Jimaklas
Member
Registered: 2014-12-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there, congratulations on the excellent advice and information found in this thread, it is by far the most helpful corner of the net as far as VGA Passthrough with KVM is concerned! Following the guide in the OP and solving most of my problems by searching the thread i managed to successfully pass my PCIe VGA adapter to a VM. By "successfully" i mean that there are no error messages issued during VM script execution. My problem is that i get no VGA output on the passthrough-ed card. Let me be more specific:

Motherboard: Intel DQ45CB (supports Intel's virtualization extensions + VT-d)
CPU: Intel Core2 Quad Q9450 @ 2.66GHz (supports Intel's virtualization extensions + VT-d)
GPU1: on chip Intel GMA 4500 (set as primary in BIOS)
GPU2: Nvidia GTX 550 Ti (passed-through in the VM)
RAM: 8GB Dual Channel DDR2
OS: openSuSE 13.2 x86_64

qemu: 2.1.0
seabios: 1.7.5 (also tried the git version)
kernel: 3.18.1 with acs override + i915 vga arbiter patches applied

I have patched the kernel with the patches provided in the OP (as found in linux-mainline-3.18.0.tar.gz), patch process went fine although i use kernel 3.18.1 instead of 3.18.0.
Some relevant kernel configuration parameters:

dimitris@openSuSE:~> cat /proc/config.gz | gzip -d | grep -e CONFIG_VIRTUALIZATION= -e CONFIG_KVM= -e CONFIG_KVM_INTEL= -e CONFIG_PCI_STUB= -e CONFIG_VFIO= -e CONFIG_VFIO_PCI= -e CONFIG_VFIO_PCI_VGA= -e CONFIG_VHOST_NET= -e CONFIG_VIRTIO_PCI= -e CONFIG_VIRTIO_BALLOON= -e CONFIG_VGA_ARB= CONFIG_PCI_STUB=y CONFIG_VHOST_NET=m CONFIG_VGA_ARB=y CONFIG_VFIO=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_VIRTIO_PCI=m CONFIG_VIRTIO_BALLOON=m CONFIG_VIRTUALIZATION=y CONFIG_KVM=y CONFIG_KVM_INTEL=y

I am booting the kernel with:

dimitris@openSuSE:~> cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.18.1-2.kvmpcipass-desktop root=UUID=4b1fbf1a-5770-4424-8609-a2b384ce6c41 resume=/dev/disk/by-uuid/2b281131-8310-48d9-94a0-9f81531469f3 splash=silent quiet showopts drm_kms_helper.edid_firmware=VGA-1:edid/lg-L1982U.bin pci-stub.ids=10de:1244,10de:0bee intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1

I am using a VGA bios from techpowerup site which compares equal to the VGA bios that i extracted from the card itself.

I am using the following script to boot the VM:

#!/bin/bash /usr/bin/qemu-system-x86_64 \ -enable-kvm -M q35 -m 4096 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /home/dimitris/Documents/build/seabios/out/bios.bin \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/dimitris/Gigabyte.GTX550Ti.1024.111121.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

and i am getting no output on the monitor attached to the Nvidia card (tried VGA, DVI and HDMI connections).

All searches lead me to the answer: "patch your kernel with i915 vga arbiter patch and enable it in the kernel command line of grub".
As far as i can understand, since the patch is already applied (CONFIG_VGA_ARB=y) and the grub command line includes i915.enable_hd_vgaarb=1, i should already have VGA output on my passed-through card.

I also tried to pass the card as secondary adapter:

#!/bin/bash /usr/bin/qemu-system-x86_64 \ -enable-kvm -M q35 -m 4096 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /home/dimitris/Documents/build/seabios/out/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/dimitris/Gigabyte.GTX550Ti.1024.111121.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/mapper/os-win7,id=disk,format=raw -device scsi-hd,drive=disk \ -drive file=/home/dimitris/Downloads/GRMCPRXFREO_EL_DVD.iso,id=isocd -device scsi-cd,drive=isocd \ -drive file=/home/dimitris/Downloads/virtio-win-0.1-94.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd

in which case the VM booted normally, Windows installation completed and the card could be seen in device manager. The Nvidia driver installation finished successfully (tried both 331.82 and 344.75) but the card was giving Error Code 12 mentioning that there are not enough resources to allocate to it. Then i removed used the "-vga none" parameter and rebooted with no output from the card. This problem (Error Code 12 in windows device manager) was caused to another user of this thread because he hadn't applied the i915 vga arbiter patch to his kernel and was solved after the patch was applied. It seems the patch doesn't get activated in my situation and i am clueless as to why this could be happening. Any ideas?

Last edited by Jimaklas (2014-12-24 14:01:50)

Offline

#3662 2014-12-24 15:17:04

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ailcp wrote:
Duelist wrote:

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Try -realtime mlock=on and having firefox(or chrome or whatever) eating enough memory, just about 1.5G.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3663 2014-12-24 15:22:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jimaklas wrote:

All searches lead me to the answer: "patch your kernel with i915 vga arbiter patch and enable it in the kernel command line of grub".
As far as i can understand, since the patch is already applied (CONFIG_VGA_ARB=y) and the grub command line includes i915.enable_hd_vgaarb=1, i should already have VGA output on my passed-through card.

As far as i know, i915 vgaarb patch isn't upstream yet, so you'll need to MANUALLY patch the kernel with, you know, patch command.
Config option isn't much related to it, it just enables any vga arbitration, and you need fixes from the patch to get i915's vgaarb logic working.

BTW, nice to see that this tech works on older, socket775 platforms too.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3664 2014-12-24 16:16:40

Jimaklas
Member
Registered: 2014-12-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Jimaklas wrote:

All searches lead me to the answer: "patch your kernel with i915 vga arbiter patch and enable it in the kernel command line of grub".
As far as i can understand, since the patch is already applied (CONFIG_VGA_ARB=y) and the grub command line includes i915.enable_hd_vgaarb=1, i should already have VGA output on my passed-through card.

As far as i know, i915 vgaarb patch isn't upstream yet, so you'll need to MANUALLY patch the kernel with, you know, patch command.
Config option isn't much related to it, it just enables any vga arbitration, and you need fixes from the patch to get i915's vgaarb logic working.

BTW, nice to see that this tech works on older, socket775 platforms too.

I am sorry i wasn't clear enough in the 1st post. Indeed, the i915 vgaarb patch isn't upstream yet, so i have actually patched the kernel myself. The patch process went like a breeze and i have also checked that every relevant file is patched according to the diffs (i also diffed the patched files against the originals and got the diffs i applied with the above patch). Is there a way to tell if the patch indeed does it's job? Here is dmesg output in case it helps:

dimitris@openSuSE:~> dmesg | grep -i vgaarb [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.18.1-2.kvmpcipass-desktop root=UUID=4b1fbf1a-5770-4424-8609-a2b384ce6c41 resume=/dev/disk/by-uuid/2b281131-8310-48d9-94a0-9f81531469f3 splash=silent quiet showopts drm_kms_helper.edid_firmware=VGA-1:edid/lg-L1982U.bin pci-stub.ids=10de:1244,10de:0bee intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.18.1-2.kvmpcipass-desktop root=UUID=4b1fbf1a-5770-4424-8609-a2b384ce6c41 resume=/dev/disk/by-uuid/2b281131-8310-48d9-94a0-9f81531469f3 splash=silent quiet showopts drm_kms_helper.edid_firmware=VGA-1:edid/lg-L1982U.bin pci-stub.ids=10de:1244,10de:0bee intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1 [ 0.115067] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 0.115067] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.115067] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.115067] vgaarb: loaded [ 0.115067] vgaarb: bridge control possible 0000:01:00.0 [ 0.115067] vgaarb: no bridge control possible 0000:00:02.0 [ 4.249183] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=io+mem

Yes KVM passthrough works on socket 775 platforms and is effectively a quite cheap solution with very decent performance wink

Offline

#3665 2014-12-24 16:45:04

Silar
Member
Registered: 2014-12-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys! I just start playing around with VM and KVM  VGA-Passthrough
My Hardware
OS: Latest Arch Linux with patched kernel from the first post (linux-mainline-3.18.0)
MB: Asrock Z68 Gen3 Extreme 3
CPU: i7 3770
Host GPU: Intel HD
Guest: GPU: nVidia GTX560Ti

I start qemu with the following parameters:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root. -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1a.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/windows.img,id=windisk,format=raw,if=virtio

The first time i start the VM everything is ok. But after the guest was shutdown and start again I get this error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

dmesg also says:

[ 3496.604688] vfio-pci 0000:00:1a.0: enabling device (0400 -> 0402) [ 3496.707482] vfio_cap_init: 0000:00:1a.0 hiding cap 0xa [ 3500.034452] vfio-pci 0000:01:00.0: Invalid ROM contents [ 3500.883294] kvm: zapping shadow pages for mmio generation wraparound

Does anybody know this error and can help me to fix it?

A second question:

I use qemu-git from AUR. How can i use -cpu kvm=off to hide the hypervisor to install latest nvidia driver?

Offline

#3666 2014-12-24 16:49:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qquark wrote:

I noticed that apparently one of the RMRR entries is correct, and the other one is not.
Since this is very easy to do, I enabled intel_iommu=on to see what happens, and Linux tells me that while setting up identity mappings it detected that my bios is broken (true, true ...); *however*, after a lengthy log full of call stacks about the devices it tried to map at the second RMRR entry, I get a log showing success (or at least no error) for those same devices at the first RMRR entry.

My first question from this result is whether I have a chance of getting VGA passthrough working in those conditions (?).
Can Linux manage to work with only one entry by ignoring the error in the other ? (assuming the first one is correct AND working) ? And Is there a way to override this entry (I do not know what the RMRR actually represents, but I suspect the answer would be negative).
Also, in my dmesg log, I only find this identity mapping happening for the USB controller devices (or subdevices ?), should I also expect to see my current graphics card there ?

RMRRs (Reserved Memory Region Reporting) entries in the DMAR table are things that should see very little use, typically only USB and integrated graphics.  They tell the IOMMU driver to configure a persistent identity map for the specified memory range and device(s).  Google'ing for the problem on this system seems to indicate that the RMRR points to a memory region that isn't listed as reserved by the e820 table.  If that's the only problem, you could fix it using this following boot option:

 memmap=nn[KMG]$ss[KMG] [KNL,ACPI] Mark specific memory as reserved. Region of memory to be reserved is from ss to ss+nn. Example: Exclude memory from 0x18690000-0x1869ffff memmap=64K$0x18690000 or memmap=0x10000$0x18690000

Second topic: I see that most people are generally using setups with {intel,ATI}+{nvidia,ATI}, but I would like to know if there is any problem with an nvidia+nvidia setup (?), as I only have a spare 9800GT I can try in addition to my GTX660, and do not have any recent extra ATI card, nor an integrated intel GP (i7 920). From what I thought I understood, just being able to unbind/rebind the appropriate card should be enough to remove any conflict between host and guest, why are people also blacklisting drivers ? Do they interfere even though they're not bound to the card ? Or is it just an easier way to do boot-time non-binding ?

Don't expect that you can bind/unbind graphics cards from their driver as easily as you can a NIC.  That's why people generally use pci-stub.ids= to make the stub driver claim devices rather than the host graphics driver.  As long as the two cards have different device IDs, which they obviously will in this case, there's no problem using a host and assigned GPU by the same vendor.  Note that there have been an maybe still are issues with using nvidia in the host regardless of the second card due to VGA arbiter locking.  nouveau shouldn't be an issue.  Ideally with a GTX660 you can get a UEFI ROM for it and use OVMF.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3667 2014-12-24 17:00:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jimaklas wrote:

Hello there, congratulations on the excellent advice and information found in this thread, it is by far the most helpful corner of the net as far as VGA Passthrough with KVM is concerned! Following the guide in the OP and solving most of my problems by searching the thread i managed to successfully pass my PCIe VGA adapter to a VM. By "successfully" i mean that there are no error messages issued during VM script execution. My problem is that i get no VGA output on the passthrough-ed card. Let me be more specific:

Motherboard: Intel DQ45CB (supports Intel's virtualization extensions + VT-d)
CPU: Intel Core2 Quad Q9450 @ 2.66GHz (supports Intel's virtualization extensions + VT-d)
GPU1: on chip Intel GMA 4500 (set as primary in BIOS)
GPU2: Nvidia GTX 550 Ti (passed-through in the VM)
RAM: 8GB Dual Channel DDR2
OS: openSuSE 13.2 x86_64

qemu: 2.1.0
seabios: 1.7.5 (also tried the git version)
kernel: 3.18.1 with acs override + i915 vga arbiter patches applied

You may be in a unique situation with this dinosaur.  The i915 patch fixes VGA arbitration for CPU integrated Intel graphics.  Your graphics is integrated into the chipset, not the CPU, so I don't expect the i915 or kernel boot option is going to help you.  If you're not getting VGA output, then it's possible that Intel broke VGA routing even prior to the switch from PCH to CPU based IGD.  IOW, you're probably out of luck unless you're able to do some debugging on your own.  Your options with this system would be to disable integrated graphics and use a discrete card for the host or use an assigned GPU and guest capable of going the UEFI/OVMF route.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3668 2014-12-24 17:01:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
ailcp wrote:
Duelist wrote:

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Try -realtime mlock=on and having firefox(or chrome or whatever) eating enough memory, just about 1.5G.

This sounds an awful lot like voodoo


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3669 2014-12-24 17:05:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Silar wrote:

Hi guys! I just start playing around with VM and KVM  VGA-Passthrough
My Hardware
OS: Latest Arch Linux with patched kernel from the first post (linux-mainline-3.18.0)
MB: Asrock Z68 Gen3 Extreme 3
CPU: i7 3770
Host GPU: Intel HD
Guest: GPU: nVidia GTX560Ti

I start qemu with the following parameters:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root. -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device vfio-pci,host=00:1a.0,bus=pcie.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/windows.img,id=windisk,format=raw,if=virtio

The first time i start the VM everything is ok. But after the guest was shutdown and start again I get this error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=

dmesg also says:

[ 3496.604688] vfio-pci 0000:00:1a.0: enabling device (0400 -> 0402) [ 3496.707482] vfio_cap_init: 0000:00:1a.0 hiding cap 0xa [ 3500.034452] vfio-pci 0000:01:00.0: Invalid ROM contents [ 3500.883294] kvm: zapping shadow pages for mmio generation wraparound

Does anybody know this error and can help me to fix it?

Dump the ROM or get it from techpowerup database and use the romfile= option to provide the ROM to the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3670 2014-12-24 17:19:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
ailcp wrote:

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Try -realtime mlock=on and having firefox(or chrome or whatever) eating enough memory, just about 1.5G.

This sounds an awful lot like voodoo

More like cargo-cult.

mlock is default:on, btw. But it works.
I still have no idea what's broken. Any ideas where do i dig to investigate the page faults? I'm not so familiar with deep hardware fiddling yet.

Also, there are performance drawbacks on some applications.

My system needs a full overhaul.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3671 2014-12-24 18:15:15

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any hint about this?

Offline

#3672 2014-12-24 22:33:04

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

Any hint about this?

As a "better than nothing" hint, i'll tell you: check whatever you use for your vm's disk. Be it filesystem's(you use file as a storage) health or actual hardware disk.
It seems to me like a kernel's problem, not a VM one.
Found this bugreport, seems somewhat related:
https://bugzilla.redhat.com/show_bug.cgi?id=1167001

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3673 2014-12-25 14:07:20

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone run into an issue with black lines flickering on the host system when the secondary GPU is being passed through? (Edit below) It disappears when I shut the guest down but comes back when the guest starts up. I otherwise have things working, though haven't tested any games yet.

I'm using two Radeon cards, R5 230 for host and R9 270X for guest with the open source drivers but have the second card bound to pci-stub. My google-fu hasn't found anything relevant yet, but if it's an issue that's already been posted that someone remembers I can work through earlier posts, perhaps with some targeted direction.

If people feel I need to provide more information to try determine what might be the cause, I'm happy to do so.

Edit: Okay, I've tested a GTX460 (nouveau) and an R7 250 (radeon) and don't get flickering on either of them, so it looks like it's related to the R5 230 I'm using. Will try to replace this with another card in the upcoming week.

Last edited by Myranti (2014-12-28 09:12:40)

Offline

#3674 2014-12-25 15:31:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Myranti wrote:

Has anyone run into an issue with black lines flickering on the host system when the secondary GPU is being passed through? It disappears when I shut the guest down but comes back when the guest starts up. I otherwise have things working, though haven't tested any games yet.

I'm using two Radeon cards, R5 230 for host and R9 270X for guest with the open source drivers but have the second card bound to pci-stub. My google-fu hasn't found anything relevant yet, but if it's an issue that's already been posted that someone remembers I can work through earlier posts, perhaps with some targeted direction.

If people feel I need to provide more information to try determine what might be the cause, I'm happy to do so.

I don't know the solution, but i think motherboard and cpu information would be handy.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3675 2014-12-25 16:26:41

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I don't know the solution, but i think motherboard and cpu information would be handy.

The motherboard i'm using is an Asus P9D WS (VT-d and virtualization options enabled in BIOS of course) and the CPU is a Xeon E3-1241v3.

Last edited by Myranti (2014-12-25 16:27:54)

Offline

#3676 2014-12-25 20:19:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm feeling like my GPU's vendor dumped me.
Some months ago i've tried to update my card's VBIOS on UEFI, it downloaded the image but failed to apply it due to lack of drivers or some other stuff - it wasn't my pc, so i didn't want to mess with owner's windows system like installing drivers.

But i got the ROM. For one card only, but that's fine.
Tried passing it wth romfile and OVMF just hanged, aw suggested to test it on real hardware to see if it's just poorly written and broken. So i gave up for some time.

Now, i've installed windows system on a bare metal system using UEFI+GPT and even secureboot was working. But ASUS UEFI Update tools now says that... my GPU doesn't support UEFI. Considering the fact i have the actual ROM... It seems strange.

So, it's time to begin my journey on writing messages to their support team.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3677 2014-12-26 10:05:08

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

check whatever you use for your vm's disk. Be it filesystem's(you use file as a storage) health or actual hardware disk.
It seems to me like a kernel's problem, not a VM one.

I'm using a dedicated physical disk through lvm (I created a virtual disk using the whole physical disk and passed it to qemu).
It's a kernel oops for sure, but the process that generated it was qemu (for what I can understand from logs).


Duelist wrote:

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

Offline

#3678 2014-12-26 11:50:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

Also, you appear to be using Q35 with it's built-in AHCI(SATA) controller, which is known to have some issues.

Hope that helps.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3679 2014-12-26 14:18:09

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

Thanks, I'll try ASAP.
What are the benefits of using virtio-blk instead of virtio-scsi?


Duelist wrote:

Also, you appear to be using Q35 with it's built-in AHCI(SATA) controller, which is known to have some issues.

I don't think so: as you can see from the script I'm using to launch qemu (posted previously), I don't specify any "-m" option and looking at the output of "qemu-system-x86_64 -machine help" I found this:

... pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-2.2) pc-i440fx-2.2 Standard PC (i440FX + PIIX, 1996) (default) ...

What is that makes you think I'm using q35?

Offline

#3680 2014-12-26 14:28:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
Duelist wrote:

check whatever you use for your vm's disk. Be it filesystem's(you use file as a storage) health or actual hardware disk.
It seems to me like a kernel's problem, not a VM one.

I'm using a dedicated physical disk through lvm (I created a virtual disk using the whole physical disk and passed it to qemu).

Maybe it's a collision of the host and guest LVM since you're using a whole disk and it will be visible to both.  Is the guest volume group active in the host?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3681 2014-12-26 14:52:01

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's posible to use the intended GPU to passthrough in first slot and tell the host to use the second slot GPU? In most motherboards the first slot has x16 and the second one has x8...

Offline

#3682 2014-12-26 15:22:51

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

It's posible to use the intended GPU to passthrough in first slot and tell the host to use the second slot GPU? In most motherboards the first slot has x16 and the second one has x8...

Yes.
Moreover, i have nvidia GT610 plugged in into pci-e x1 slot. Bus-Device-Function is 04:00.0
Passing through two HD7750 in x16 and x8(somehow runs as x4, but wired as x8, may be a driver's problem).
It works.
BUT!
1. You have to plunge the GPUs to pci-stub BEFORE ANYTHING.
2. Nvidia driver fails to detect the card without BusID option.

Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GT 610" BusID "PCI:04:00:00" EndSection

EDIT:
Oh, the only downside is - there will be no emergency console available on ctrl-alt-F[1-12] since there is no VGA output.
Grub will show you your custom fonts, but things like plymouth, using kernel mode setting, won't work.
There is a bright side in it - usually VFIO-bind service is loaded after X, so if X fails to load due to bad drivers(like after a kernel rebuild, which should be worked around using package manager, but i'm lazy and sort of know what i'm doing) - primary GPU won't be hooked to pci-stub and VGA will work and you'll get to login in the console.

Also, gigabyte motherboards may offer a switch, which changes the primary GPU, solving everything. My ASUS doesn't have that feature.

Last edited by Duelist (2014-12-26 16:19:07)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3683 2014-12-26 17:21:57

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The console thing maybe can be solved by using fbcon, for example adding fbcon=map:111111 in bootline
and thanks for confirming the posibility of first slot

Last edited by Cubex (2014-12-26 17:22:36)

Offline

#3684 2014-12-27 04:45:10

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I though that I should post what I've done so far that has enabled me to get my passthrough working and the issues I've run into along the way. Note: I'm still having at least 1 issue, which is an issue with some flickering black lines on the host side of the passthrough.

Specifications:
Host OS: Arch Linux - I've manually patched the current kernel (from ABS) with the ACS patch. This was an interesting experience required some modification to patch the correct lines.
Guest OS: Windows 8.1 Pro
Motherboard: Asrock C226-WS (Not the C226-WS+)
CPU: Intel Xeon E3-1241v3
Host GPU: Radeon R5 230
Guest GPU: Radeon R9 270X
(I'm using the open source radeon driver, not catalyst)
Note: I'm also using OVMF (which is actually in the Arch repos in Extra), but apart from manually patching the kernel, all other core software involved (qemu, libvirt, virt-manager) is from Arch repos, not from the AUR.

Edit: I've replaced the original motherboard (Asus P9D WS) with an Asrock C226-WS which allows me to put my passthrough VGA card in the top (closest to CPU) slot, since the video seems to initialise as per slot number since the full size slots are PCIe Slot 1, 3 and 6, with 6 being the top slot. This has given me a little extra performance (going from PCIe 3 x4 to x8) as well, bringing me closer to around 98% performance (somewhere around 98.5%), up from around 95%/96%, which is good enough for me.

The devices that I am passing through to the guest include the USB 3 card, the onboard audio (using a PCIe sound card on the host), one of the onboard LAN ports (Dual LAN, both Intel I210) and the R9 270X of course. At this point, I will mention that the original USB 3 card I was using ended up causing issues as the device wouldn't reset properly once it was released from the VM the first time leading to error's along the line of:

vfio-pci 0000:01:00.0: Refused to change power state, currently in D3 vfio_pci_disable: Failed to reset device 0000:01:00.0 (-22)

This also meant I could only pass through the device once from a fresh boot and then have to reboot to do it again. It would also cause irq issues like the following conflict with the sound card (I did remove the sound card, thinking that was the issue but it conflicted with other devices):

genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (snd_oxygen_lib)

I had other cards on hand and replaced the USB 3 card with one that only partially worked (over-current conditions, leaving 2 ports not working) and then finally with one that would reset properly when released by the VM. I mention all this so that it might provide some insight for those that may be having these issues. The other main issue was with IOMMU groups (libvirt/qemu couldn't bind the devices being passed through without binding other devices in the IOMMU group) and required the ACS patch and the addition of pcie_acs_override=downstream to the kernel parameters in GRUB.

Kernel Parameters (through GRUB):

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:6810,1002:aab0,1106:3432,8086:8c20

I will also note here that the id for the dual LAN port hasn't been bound to pci-stub here as both devices share the same id, but it appears to work fine, perhaps due to it being an Intel NIC.

Kernel Modules (in initramfs via /etc/mkinicpio.conf) - except ahci, which was already in my modules line before this:

MODULES="ahci pci-stub vfio-pci vfio-iommu-type1"

I also have /etc/modprobe.d/kvm-qemu.conf

options kvm ignore_msrs=1

And /etc/modprobe.d/vfio-iommu.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

Edit: I was using the vfio-bind script, but I was able to just assign devices to pci-stub and let libvirt manage this (managed='yes' hostdev entries). The only issue was that both LAN ports on this dual LAN board are the same vendor:model in lspci and my network scripts were adding any en* devices to the network bridge and so my host box wouldn't connect to the network. This has since been fixed but only assigning the one I want to a network and leaving the other disconnected.

At this point, I diverged slightly from others and used libvirt along with virt-manager to configure setup and configure VM and the passthrough.

I used virt-manager to create a new virtual machine, specifying Windows 8.1 as the guest OS, created a temporary hdd image and completed the setup without customizing the install, leaving it as an i440fx system (after watching Alex Williamson's KVM Forum presentation, seems like there's still potential issues with q35 and i440fx works fine). I then went through and removed some of the extra devices I didn't need, leaving the IDE, PCI and USB controllers in place and ended up just using VNC and cirrus video to get the guest installed, rather than having all the spice stuff in place. I modified a few of the settings, told it to copy the hosts cpu config, adjusted the cpu topology and then used virsh to edit the xml file of the VM. I specifically did not pass through the HDMI for the GPU being passed through in the end, as it causes the host screen to go fuzzy and it doesn't release/reset properly once the guest VM shuts down.

Yet another note: I didn't pass the GPU through until I had installed the OS using the VNC/cirrus video and had ensured that my other devices were working correctly and that the VM would startup and shutdown repeatedly without issue without the GPU bieng passed through.

Of the changes via virsh, one of them was the addition of the OVMF loader, specifying a whole physical hard drive, rather than an image and modifying some of the hostdev entries with the vfio driver (I'll point these out further down).

Edit: I modified the VM after it being noted that I wasn't using Virtio for the disk (virtio is definitely faster for me), so I went through the process to get the driver installed then switched the disk to virtio initially, then to scsi (using virtio-scsi controller) and added the discard="unmap" to the driver line for the disk since I'm using an SSD.

This is how the updated xml file for my VM looks:

<domain type='kvm'> <name>windows</name> <uuid>b49fc4cc-149e-4904-8a5c-e74fb5a4c417</uuid> <memory unit='KiB'>12582912</memory> <currentMemory unit='KiB'>12582912</currentMemory> <memoryBacking> <hugepages/> <nosharepages/> </memoryBacking> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='6' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='invpcid'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pbe'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='smx'/> <feature policy='require' name='abm'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='fma'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='tm2'/> <feature policy='require' name='avx2'/> <feature policy='require' name='ss'/> <feature policy='require' name='bmi1'/> <feature policy='require' name='bmi2'/> <feature policy='require' name='pcid'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='movbe'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/> <source dev='/dev/sdd'/> <target dev='sda' bus='scsi'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sde'/> <target dev='sdb' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='nec-xhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <driver queues='6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </controller> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

The relevant extra changes I made are:
The OVMF loader, which I copied to my libvirt images directory (which is a symlink anyway) as it complained that it didn't have permissions and I was too lazy to mess around with it.

 <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader>

The disk type for the system drive (SSD), which I specified as block and specified the source drive (/dev/sdd in this case). As noted earlier, this was changed as I'm using virtio-scsi and unmap to pass through trim/discard commands from the guest OS to the drive:

 <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/> <source dev='/dev/sdd'/> <target dev='sda' bus='scsi'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk>

And the addition of the vfio device driver name for the host devices being passed through (just added between the hostdev lines):

 <driver name='vfio'/>

There was a bit of fiddling amongst all this obviously (as per some of my notes above for issues I ran into) but ultimately it is, for the most part, working. I haven't yet tested the system on bare metal to get an indication of the performance when it's not in the VM, but comparing some limited benchmarking to what I can find equivalents of online (Unigine Valley benchmark), I think I'm within about 5% of bare metal performance. It might be more like within 10%, but I have yet to do more testing to determine how it compares to relatively equivalent hardware.

Hopefully some of this is useful to others with their attempts and I think I've covered most of the work I've done to get it to this point.

Edit: I removed the seperate pci-stub conf and softdep on drm and put my pci-stub line as a kernel parameter. I also added a second drive for storage (steam backups, etc). I've also enabled hugepages (using the Arch wiki guide), though this required a restart to work properly without throwing errors from libvirt.

Last edited by Myranti (2015-01-20 11:43:49)

Offline

#3685 2014-12-27 20:15:42

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

All,

I wanted to report my successful setup so that others may have another data point trying to do the same. I also need others to see what setup works, because before embarking on trying to do this and sinking down a significant amount of time, I didn't know whether I would succeed or not. I tried Googling my motherboard and the setup with vfio but didn't find any with my setup, so I didn't have 100% confidence that I would be able to get my system running.

Host OS: Fedora 21 (64bit, 3.17.7 kernel)
Guest OS: Windows 7 (64bit)
CPU: i7-4790K (one of the very few K versions that supports Vt-d)
Motherboard: Asus Sabertooth Z97 mark 1 (had to enable virtualization and Vt-d in BIOS)
Host GPU: Geforce 970 GTX (Nvidia's proprietary driver v346.22)
Guest GPU: Geforce 560Ti (Nvidia's driver v347.09)
System RAM: DDR3-1600 4x8GB (Crucial Ballistix Sport)
Emulator/Hypervisor: Running qemu-kvm, which is managed by libvirtd

I believe I ran into every conceivable problem that has been reported by others in this very long thread. Some of the issues and how I resolved them:

1. Windows reported Code 43 - had to hide from Windows (more specifically from Nvidia's driver) that it was running as a VM - info here (modified the -cpu switch that was being sent by libvirt, include kvm=off and eliminate all hv_* options) - I am starting to see why Linus infamously gave Nvidia the finger on one of the videos I found on the internet. Why would Nvidia actively engage in locking out their card from being run in VM? I doubt that would increase the sales of their Quadro line of cards or to decrease their support line load because of guys like me trying to get this to work with their cards. Instead they decided to turn this into a game of cat and mouse hide and seek. I don't understand their decision other than because they have such a dominant position that they think they can get away with it. AMD isn't providing any competition to Nvidia as I am reading that Radeon cards have their own set of issues.
2. Windows reported Code 12 - Initially ran without any video or graphics for the VM in libvirt to bypass this issue, but I couldn't figure out how to pipe the sound out - ended up using "Xen" for video card in libvirt. That allowed the sound the come through and eliminated Code 12 in Windows.
3. qemu reported that it couldn't lock memory - I played with ilimit and editing /etc/security/limits.conf, but none of that worked (makes sense, I was running qemu-kvm as root - no limit) - it turned out that selinux was blocking ipc_lock operation for qemu-kvm. I used the excellent Fedora's selinux debugging tool to enable it and now it works.
4. I had issues with iommu grouping. The two Nvidia cards were being grouped into the same group and qemu refused to work because the iommu grouping was "not a viable grouping." I had initially assumed that the PCIe ACS patch from Alex Williamson patch here has either gone upstream or Fedora team incorporated the patch by now, but no amount of tweaking on the boot command pcie_acs_override=downstream would work. But in few posts, Alex says that this patch will not go upstream due to the nature of the patch and the bug report on fedora bugzilla report here makes it clear that it hadn't made it even as a patch on Fedora. I also downloaded the source RPM and checked out quirks.c file and realized that it did not have the patch (I'm at the latest kernel version 3.17.7 for FC21). I wanted to do this the "right" way and went about creating a custom RPM from the source RPM (plus the ACS patch). I used the custom kernel compiling guide here . The patch also didn't work out of the box, I had to manually patch it in for 3.17.7. I also had to create a link for python-config in /usr/local/bin. The stock installation doesn't seem to have the link and it breaks the kernel compile.
5. I had to put the following options to grub boot up - "intel_iommu=on pci-stub.ids=10de:1200,10de:0e0c pcie_acs_override=downstream"

I've had other issues too, but many of them were caused by not following some of the nicely written guides on the net.

Here's what I used as a base guide here .
I didn't rebuild qemu, as I was already at version 2.1.2. I also didn't have to worry about nosnoop and i915 patch. I installed the stock qemu and kvm from Fedora. I did install the vfio-bind service as the guide indicated - this is important as we need bring the device into the /dev/vfio/ directory. I did update /etc/libvirt/qemu.conf to allow qemu to run as root and included the new cgroup_device_acl options. Instead of creating the libvirt xml from scratch as the guide suggested, I actually use libvirt to create a VM first. I then continued to install the guest OS and the virtio stuff needed to get it working. Then I started to modify the xml using "virsh edit [domain name]" to make updates first to use vfio-pci then any additional modifications as needed. The tricky part of editing the xml file was making sure I had the "xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'" line at the top. Otherwise, I kept losing the <qemu:commandline> options.

Windows 7 indicates that I'm at level 7.0 for "Windows Experience Index"
I didn't run any 3D benchmarks, but did run Portal using Steam - no problems seen there and very good performance as far as I can tell, certainly better than a Intel Core 2 Duo E8500 setup I had previously dedicated for gaming.

Ongoing problems - after shutting down the VM, I can't get it to come up again the second time (or the third time, etc). I used the suggested trick of putting the host in Suspend mode/wake up and launching again - that works. But occasionally, I get an interrupt issue that I can't get out of and the whole machine needs rebooting. Also the audio isn't quite right. I'm using ICH6 to emulate and piping it to the host via the Xen setup, but I'm getting fairly frequent glitches/stuttering in audio from the guest.

This capability is a bit of a game-changer for me. I've always had two machines - one for my mission critical stuff in Linux, and another for gaming in Windows. Steam's effort to bring more gaming to Linux is helping, but we're not quite there yet. I think that this setup may be the first time where I feel that I can actually consolidate two machines into one. A big thank you for everyone who allowed this to happen and many people debugging this.

My libvirt xml file:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win7-gaming</name> <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/win7.qcow2'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/virtio-win-0.1-94.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:cb:a5:3b'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='xen' vram='4096' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0424'/> <product id='0x2514'/> <address bus='5' device='2'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x06a3'/> <product id='0x8021'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> <address bus='5' device='4'/> </source> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.1,bus=pci.0'/> <qemu:arg value='-bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> <qemu:arg value='-cpu'/> <qemu:arg value='Haswell,kvm=off'/> </qemu:commandline> </domain>

The patch file for kernel 3.17.7:

diff -rupN kernel-3.17.fc21/linux-3.17.7-300.pkim.fc21.x86_64/Documentation/kernel-parameters.txt kernel-3.17.fc21.new/linux-3.17.7-300.pkim.fc21.x86_64/Documentation/kernel-parameters.txt --- a/Documentation/kernel-parameters.txt 2014-12-26 00:24:21.607861638 -0800 +++ b/Documentation/kernel-parameters.txt 2014-12-25 23:56:13.554340620 -0800 @@ -2655,6 +2655,16 @@ bytes respectively. Such letter suffixes nomsi Do not use MSI for native PCIe PME signaling (this makes all PCIe root ports use INTx for all services). + pcie_acs_override = + [PCIE] Override missing PCIe ACS support for: + downstream + All downstream ports - full ACS capabilties + multifunction + All multifunction devices - multifunction ACS subset + id:nnnn:nnnn + Specfic device - full ACS capabilities + Specified as vid:did (vendor/device ID) in hex + pcmv= [HW,PCMCIA] BadgePAD 4 pd_ignore_unused diff -rupN kernel-3.17.fc21/linux-3.17.7-300.pkim.fc21.x86_64/drivers/pci/quirks.c kernel-3.17.fc21.new/linux-3.17.7-300.pkim.fc21.x86_64/drivers/pci/quirks.c --- a/drivers/pci/quirks.c 2014-12-26 00:24:21.157861498 -0800 +++ b/drivers/pci/quirks.c 2014-12-26 00:02:46.714460917 -0800 @@ -3583,6 +3583,108 @@ struct pci_dev *pci_get_dma_source(struc return pci_dev_get(dev); } +static bool acs_on_downstream; +static bool acs_on_multifunction; + +#define NUM_ACS_IDS 16 +struct acs_on_id { + unsigned short vendor; + unsigned short device; +}; +static struct acs_on_id acs_on_ids[NUM_ACS_IDS]; +static u8 max_acs_id; + +static __init int pcie_acs_override_setup(char *p) +{ + if (!p) + return -EINVAL; + + while (*p) { + if (!strncmp(p, "downstream", 10)) + acs_on_downstream = true; + if (!strncmp(p, "multifunction", 13)) + acs_on_multifunction = true; + if (!strncmp(p, "id:", 3)) { + char opt[5]; + int ret; + long val; + + if (max_acs_id >= NUM_ACS_IDS - 1) { + pr_warn("Out of PCIe ACS override slots (%d)\n", + NUM_ACS_IDS); + goto next; + } + + p += 3; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].vendor = val; + + p += strcspn(p, ":"); + if (*p != ':') { + pr_warn("PCIe ACS invalid ID\n"); + goto next; + } + + p++; + snprintf(opt, 5, "%s", p); + ret = kstrtol(opt, 16, &val); + if (ret) { + pr_warn("PCIe ACS ID parse error %d\n", ret); + goto next; + } + acs_on_ids[max_acs_id].device = val; + max_acs_id++; + } +next: + p += strcspn(p, ","); + if (*p == ',') + p++; + } + + if (acs_on_downstream || acs_on_multifunction || max_acs_id) + pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n"); + + return 0; +} +early_param("pcie_acs_override", pcie_acs_override_setup); + +static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags) +{ + int i; + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Never override ACS for legacy devices or devices with ACS caps */ + if (!pci_is_pcie(dev) || + pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS)) + return -ENOTTY; + + for (i = 0; i < max_acs_id; i++) + if (acs_on_ids[i].vendor == dev->vendor && + acs_on_ids[i].device == dev->device) + return 1; + + switch (pci_pcie_type(dev)) { + case PCI_EXP_TYPE_DOWNSTREAM: + case PCI_EXP_TYPE_ROOT_PORT: + if (acs_on_downstream) + return 1; + break; + case PCI_EXP_TYPE_ENDPOINT: + case PCI_EXP_TYPE_UPSTREAM: + case PCI_EXP_TYPE_LEG_END: + case PCI_EXP_TYPE_RC_END: + if (acs_on_multifunction && dev->multifunction) + return 1; + } + + return -ENOTTY; +} + + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html AMD has indicated that the devices below do not support peer-to-peer 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_14.html topic_15.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html in any system where they are found in the southbridge with an AMD @@ -3696,6 +3798,7 @@ static const struct pci_dev_acs_enabled { PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs }, { PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_intel_pch_acs }, + { PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides }, { 0 } };

the patch for kernel.spec:

--- kernel.spec.orig 2014-12-25 23:12:35.649583082 -0800 +++ kernel.spec 2014-12-26 01:44:27.301825643 -0800 @@ -42,7 +42,7 @@ Summary: The Linux kernel # For non-released -rc kernels, this will be appended after the rcX and # gitX tags, so a 3 here would become part of release "0.rcX.gitX.3" # -%global baserelease 300 +%global baserelease 301 %global fedora_build %{baserelease} # base_sublevel is the kernel version we're starting with and patching @@ -648,6 +648,9 @@ Patch26099: deal-with-deadlock-in-d_walk # git clone ssh://git.fedorahosted.org/git/kernel-arm64.git, git diff master...devel Patch30000: kernel-arm64.patch +# pcie_acs_override +Patch90000: PCIE-ACS-override.patch + # END OF PATCH DEFINITIONS %endif @@ -1412,6 +1415,9 @@ ApplyPatch arm64-fix-xgene_enet_process_ %endif %endif +# pcie_acs_override +ApplyPatch PCIE-ACS-override.patch + # END OF PATCH APPLICATIONS %endif

Last edited by pkim (2014-12-27 20:28:55)

Offline

#3686 2014-12-27 20:52:18

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you guys for posting detailed descriptions!

one question for Myranti: I noticed you aren't using Virtio for your disk. Did you do that on purpose? Is it faster that way? Do you also use a spare SSD for Windows 8.1?

Last edited by 4kGamer (2014-12-27 20:55:52)

Offline

#3687 2014-12-27 21:59:43

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you guys for posting detailed descriptions!

one question for Myranti: I noticed you aren't using Virtio for your disk. Did you do that on purpose? Is it faster that way? Do you also use a spare SSD for Windows 8.1?

I had tested virtio and piix4-ide for me Windows shows the same 6.9 points for disk performance. I suppose the most performant way is to passthrough sata controller.

Offline

#3688 2014-12-28 00:04:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pkim wrote:

All,

I wanted to report my successful setup so that others may have another data point trying to do the same. I also need others to see what setup works, because before embarking on trying to do this and sinking down a significant amount of time, I didn't know whether I would succeed or not. I tried Googling my motherboard and the setup with vfio but didn't find any with my setup, so I didn't have 100% confidence that I would be able to get my system running.

Congratulations... unfortunately I'm not even sure where to start correcting your config.

a) Z97 (Lynx Point) is supported by kernel ACS quirks, so all you need to do is move your GPU to a difference slot to fix the IOMMU grouping problem, no custom kernel required.  Put it behind a PCH root port slot.

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

c) Your config uses ioh3420 root ports on 440fx chipset.  Stop it, just stop it.  Move the vfio-pci options to proper hostdev options.

d) The whole xen primary graphics, stop that too.

e) libvirt supports kvm=off, why are you specifying it via qemu:args?!  You've also still got various hyper-v extensions enabled in your xml.  See my blog for how to do all of this properly.

f) You're specifying a bios, stop that too.  Completely unnecessary.

What can you expect to achieve by making all these changes?  You don't need the i915 patch and you don't need the ACS patch, so I expect you can make this work with an unmodified kernel, qemu, and libvirt, having libvirt fully manage your VM (no vfio-bind), with just a qemu-kvm wrapper, your pci-stub.ids line, and an selinux module for the script.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3689 2014-12-28 00:55:21

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

one question for Myranti: I noticed you aren't using Virtio for your disk. Did you do that on purpose? Is it faster that way? Do you also use a spare SSD for Windows 8.1?

Good question! Probably because I didn't think of doing it, but I see there's a guide on on linux-kvm.org, so I'll try it out. It wasn't specifically on purpose, I just hadn't really considered it and now that I've had a look, I realise I'd probably need to do it after the initial install anyway.

I'm not sure if it's faster, but I'll do a test before the change and after to see if it has a noticeable impact. Yes, I am using an SSD, not so much a spare, more of an additional drive, since this system was upgraded/built specifically for the purposes of doing this.

Edit: Yes, it does appear to be faster to use virtio.

Last edited by Myranti (2014-12-28 02:29:40)

Offline

#3690 2014-12-28 07:21:23

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex,

Thanks for the good feedback -

a) Z97 (Lynx Point) is supported by kernel ACS quirks, so all you need to do is move your GPU to a difference slot to fix the IOMMU grouping problem, no custom kernel required.  Put it behind a PCH root port slot.

This is a consequence of the motherboard size, case design, power supply installed at the bottom and the fact that both Nvidia cards are double width. The motherboard has 3 16-lane PCIe slots and the two cards can only be fitted into two of them. In fact, the first thing I tried was to try to relocate one of the two cards to the 3rd slot based on your advice - not physically possible. So rather than buying another bigger case, I decided to try the ACS patch. (I would venture to guess that my situation is not unique.)

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

I think I got this working using a wrapper script that you provided. But I still don't know how to do it without the vfio-bind script - are you suggesting that I do the binding inside the wrapper as well?

c) Your config uses ioh3420 root ports on 440fx chipset.  Stop it, just stop it.  Move the vfio-pci options to proper hostdev options.

Ok, figured it out. Not too many examples out there. A user named "teekay" talked about trying it out and failing on this thread here - but it seems to work for me.

d) The whole xen primary graphics, stop that too.

One of your posts on your blog mentioned using qxl and disabling it so I can use it at least for the sound here. But no matter how I tried, I just don't know how to "disable" the qxl video and eliminate the Code 12 error. I need this so that I can use the host sound.

e) libvirt supports kvm=off, why are you specifying it via qemu:args?!  You've also still got various hyper-v extensions enabled in your xml.  See my blog for how to do all of this properly.

I actually saw this while I was on the hunt to solve this, I remember seeing that you've submitted it to git for inclusion like 3-months ago - I incorrectly assumed that the feature was not included in qemu 2.1.2. I was wrong. I cleaned up my xml - thanks!

f) You're specifying a bios, stop that too.  Completely unnecessary.

I didn't know that. I deleted it and it works fine without it.

Here's my latest xml, free of any qemu:args (still on xen - couldn't figure out how to use qxl):

<domain type='kvm'> <name>win7-gaming</name> <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/win7.qcow2'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/virtio-win-0.1-94.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:cb:a5:3b'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='xen' vram='4096' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0424'/> <product id='0x2514'/> <address bus='5' device='2'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x06a3'/> <product id='0x8021'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> <address bus='5' device='4'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

By the way, I also backed out of the three settings in /etc/libvirt/qemu.conf (commented them and let libvirt use the defaults - the VM is very stable and also the sound works better now):

user = "root" group = "root" clear_emulator_capabilities = 0

Last edited by pkim (2014-12-28 08:44:06)

Offline

#3691 2014-12-28 10:27:12

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Myranti. I wasn't so sure myself. Glad if it helped you.

And @kainet: It does make sense what you say. I wonder whether sata performance would surpass virtio. Did someone make the comparison?

Last edited by 4kGamer (2014-12-28 12:17:54)

Offline

#3692 2014-12-28 11:24:41

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@4kGamer: Yeah, I just hadn't considered it. I ended up going with SCSI and using virito-scsi with discard='unmap' so that any trim/discard is passed through.

If I was using another motherboard with additional sata controllers I could try that to see how it went, but alas!

Offline

#3693 2014-12-28 15:55:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pkim wrote:

Alex,

Thanks for the good feedback -

a) Z97 (Lynx Point) is supported by kernel ACS quirks, so all you need to do is move your GPU to a difference slot to fix the IOMMU grouping problem, no custom kernel required.  Put it behind a PCH root port slot.

This is a consequence of the motherboard size, case design, power supply installed at the bottom and the fact that both Nvidia cards are double width. The motherboard has 3 16-lane PCIe slots and the two cards can only be fitted into two of them. In fact, the first thing I tried was to try to relocate one of the two cards to the 3rd slot based on your advice - not physically possible. So rather than buying another bigger case, I decided to try the ACS patch. (I would venture to guess that my situation is not unique.)

Too bad, so close to being able to work unmodified.  I have the same problem on my P8H67-M PRO, but I use integrated graphics for the host and my primary guest uses a dual slot GTX750 from the CPU root ports.  My secondary guest uses an HD8570, single width slot off the PCH root port.  In fact I've managed to make that card fanless by cutting down a CoolVivaZ1 so that it still squeezes in above the P/S.

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

I think I got this working using a wrapper script that you provided. But I still don't know how to do it without the vfio-bind script - are you suggesting that I do the binding inside the wrapper as well?

No, libvirt will do it for you, that's what the managed="yes" option in the xml means.

c) Your config uses ioh3420 root ports on 440fx chipset.  Stop it, just stop it.  Move the vfio-pci options to proper hostdev options.

Ok, figured it out. Not too many examples out there. A user named "teekay" talked about trying it out and failing on this thread here - but it seems to work for me.

ioh3420 is a PCIe root port, so the frankenstein configuration that creates is to have a PCIe root port sourced on a conventional PCI bus.  It's not something you'd ever see on real hardware.

d) The whole xen primary graphics, stop that too.

One of your posts on your blog mentioned using qxl and disabling it so I can use it at least for the sound here. But no matter how I tried, I just don't know how to "disable" the qxl video and eliminate the Code 12 error. I need this so that I can use the host sound.

Right, that qxl config is using OVMF and Win8, which is a much more amenable config for simultaneously supporting both assigned and emulated graphics.  With Win7 and VGA, the VM will stop using the emulated graphics once the Nvidia driver loads due to resource conflicts of the VGA space.  I don't know that that would affect audio over spice though.  Theoretically libvirt can support connecting the emulated audio to your local pulseaudio server independent of the emulated graphics, but I've never figured out how to do this.  I'm also suspicious whether things like Steam In-home-streaming work with anything other than the GPU audio device.

e) libvirt supports kvm=off, why are you specifying it via qemu:args?!  You've also still got various hyper-v extensions enabled in your xml.  See my blog for how to do all of this properly.

I actually saw this while I was on the hunt to solve this, I remember seeing that you've submitted it to git for inclusion like 3-months ago - I incorrectly assumed that the feature was not included in qemu 2.1.2. I was wrong. I cleaned up my xml - thanks!

There's also the virt-preview repo for Fedora if you want more bleeding edge QEMU/libvirt.

f) You're specifying a bios, stop that too.  Completely unnecessary.

I didn't know that. I deleted it and it works fine without it.

Here's my latest xml, free of any qemu:args (still on xen - couldn't figure out how to use qxl):

<domain type='kvm'> <name>win7-gaming</name> <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/win7.qcow2'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/virtio-win-0.1-94.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:cb:a5:3b'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='xen' vram='4096' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0424'/> <product id='0x2514'/> <address bus='5' device='2'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x06a3'/> <product id='0x8021'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> <address bus='5' device='4'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

By the way, I also backed out of the three settings in /etc/libvirt/qemu.conf (commented them and let libvirt use the defaults - the VM is very stable and also the sound works better now):

user = "root" group = "root" clear_emulator_capabilities = 0

Great, that's a much cleaner config IMHO.  If you wanted you could add multifunction="on" to the GPU hostdev config and then you could move the assigned audio device to function 1 of the same slot.  It makes the card appear to the guest more like it actually is, but I haven't found that it functionally behaves any different.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3694 2014-12-28 21:38:52

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone help me? I don't know how to passthrough a sata controller.

this is what lspci | gep SATA outputs:

00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)


I want to install Windows on a SATA Disk. I have to passthrough a SATA controller for that right?

First I bound the controller to vfio. But when I try to start the installation, i get this error:

Error starting domain: Requested operation is not valid: PCI device 0000:07:00.0 is not assignable



This is what my XML looks like

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


I googled it, but I am lost here. Please help

Offline

#3695 2014-12-28 21:48:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Can someone help me? I don't know how to passthrough a sata controller.

this is what lspci | gep SATA outputs:

00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)


I want to install Windows on a SATA Disk. I have to passthrough a SATA controller for that right?

First I bound the controller to vfio. But when I try to start the installation, i get this error:

Error starting domain: Requested operation is not valid: PCI device 0000:07:00.0 is not assignable



This is what my XML looks like

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


I googled it, but I am lost here. Please help

You cannot both assign the device and attach the disk behind it using an emulated controller.  One or the other.  The only case where I consider assigning a SATA controller to be a valid option is if you want to boot the image both on bare metal and a VM, and even there it's questionable whether it's really an advantage.  If you're doing this only for performance reasons then you haven't taken virtio tuning as far as it can go.  There's very, very little performance to gain by doing this.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3696 2014-12-28 22:02:48

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you Alex, I won't bother with it, back to Virtio then. You're right. I am still not comfortable with virtio tuning, the documentation is great, but I still haven't had success with it. I will try again and probably have more questions for you smile

Offline

#3697 2014-12-28 22:14:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you Alex, I won't bother with it, back to Virtio then. You're right. I am still not comfortable with virtio tuning, the documentation is great, but I still haven't had success with it. I will try again and probably have more questions for you smile

FWIW, I use this on my Steam VM:

 <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/mnt/store/vm/Steam.raw'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <driver queues='4'/> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller>

The raw disk image actually lives on a relatively slow spinning disk, but I use bcache to cache it across a small SSD.  This setup doesn't always benchmark well, but for real world performance launching a single game, it does fantastic.

EDIT: queues=4 enables multi-queue virtio-scsi.  I use 4 because that's the number of vCPUs provided to the guest.

Last edited by aw (2014-12-28 22:16:07)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3698 2014-12-29 04:01:27

Chaohong Hu
Member
Registered: 2014-12-29
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have successfully passed through graphics card using vfio.

The problem is vfio need an additional monitor connected to the graphics card to show vm,
but i only have one monitor(and two gfxs of course),so i am searching a solution which can
share the only monitor between host's gfx and vm's gfx. I mean when i launch vm the monitor
start to display output from passthru's card and when i shutdown vm, monitor display output from
host's gfx again.

Any idea about this?

Offline

#3699 2014-12-29 06:42:40

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Alex: thank you, I will try your config!


@Chao: You need some sort of switch or hub. If your GPUs both have HDMI outputs, you can use an HDMI switch. In any case a device with one output and two inputs would work.

I use an AV-R, because I use my setup with a TV. But there are other options.

Like a KVM switch  (the abbreviations are coincidental).

Look here: https://www.youtube.com/watch?v=VGxzHkI8rXo

Last edited by 4kGamer (2014-12-29 08:09:39)

Offline

#3700 2014-12-29 09:30:00

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

I think I got this working using a wrapper script that you provided. But I still don't know how to do it without the vfio-bind script - are you suggesting that I do the binding inside the wrapper as well?

No, libvirt will do it for you, that's what the managed="yes" option in the xml means.

Confirmed! letting libvirt to do all the managing seems to be much smoother and easier (once you figure out the correct xml configuration).

Great, that's a much cleaner config IMHO.  If you wanted you could add multifunction="on" to the GPU hostdev config and then you could move the assigned audio device to function 1 of the same slot.  It makes the card appear to the guest more like it actually is, but I haven't found that it functionally behaves any different.

Agreed. I'll try out the multifunction="on" later just for the sake of making it cleaner.

So basically what I needed it to work is this:

Host OS: Fedora 21 (64bit, 3.17.7 kernel, CUSTOM compiled with PCIe ACS patch from Alex Williamson)
Guest OS: Windows 7 (64bit)
CPU: i7-4790K (one of the very few K versions that supports Vt-d)
Motherboard: Asus Sabertooth Z97 mark 1 (had to enable virtualization and Vt-d in BIOS)
Host GPU: Geforce 970 GTX (Nvidia's proprietary driver v346.22)
Guest GPU: Geforce 560Ti (Nvidia's driver v347.09)
System RAM: DDR3-1600 4x8GB (Crucial Ballistix Sport)
QEMU-KVM: 2.1.2, stock, no configuration changes.
Libvirt: 1.2.9, stock

And the following xml (it is possible to do everything without qemu:args! - also note that I used a wrapper script to insert the x-vga option):

<domain type='kvm'> <name>win7-gaming</name> <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/win7-gaming.qcow2'/> <target dev='vdc' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:cb:a5:3b'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='xen' vram='4096' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x06a3'/> <product id='0x8021'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc066'/> </source> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

/usr/libexec/qemu-kvm-nvidia-vga:

#!/bin/sh exec /bin/qemu-kvm \ `echo "$@" | sed 's|02:00.0|02:00.0,x-vga=on|g'`

Remaining problems:
1. Once the first VM shuts down, the subsequent booting doesn't work properly. Work-around: suspend the host OS to reset the GPU. There is an annoying problem with losing the background with the suspend/resume on the Nvidia cards. Alt-F2 then r to reset in Gnome.
2. I can't figure out how to pipe only the sound through spice. Work-around: use Xen for video card type.

Last edited by pkim (2014-12-29 12:26:59)

Offline

#3701 2014-12-29 09:32:50

Ansa89
Member
Registered: 2014-08-30
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Ansa89 wrote:

I'm using a dedicated physical disk through lvm (I created a virtual disk using the whole physical disk and passed it to qemu).

Maybe it's a collision of the host and guest LVM since you're using a whole disk and it will be visible to both.  Is the guest volume group active in the host?

1) LVM is used only on the host to create a virtual disk for the guest (while the guest doesn't use LVM and sees the virtual disk as a normal one).
2) The physical disk is visible only on the host; the virtual disk is visible to both host and guest (again: the guest sees it as a normal disk).
3) The LVM virtual disk is used only by the guest (and by qemu on the host).


PS: what are the benefits of using virtio-blk instead of virtio-scsi?

Offline

#3702 2014-12-29 11:59:33

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pkim wrote:

2. I can't figure out how to pipe only the sound through spice. Work-around: use Xen for video card type.

It works when you have the VM client open and connected (virt-manager, spicy). This requires virtual graphics, which can be disabled in quest, but I think a better solution is to use pulseaudio server directly. I have found information on this somewhere earlier in this thread:

/etc/pulse/default.pa: .. load-module module-native-protocol-unix auth-anonymous=1 ..

and with added configuration in VM xml:

 <qemu:commandline> <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_PA_SAMPLES' value='1024'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/> </qemu:commandline>

In guest, you are then using virtual sound card configured in xml the standard way.
It doesnt work if you are running qemu as root (or not dropping capabilities through libvirt..?), but thanks to aws qemu wrapper script it works for me perfectly (permissions problems without it). You have to have logged in session on host, too.
I didn't dive into security of pulseaudio server after this, there could be some problems.

Offline

#3703 2014-12-29 12:09:37

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:

PS: what are the benefits of using virtio-blk instead of virtio-scsi?

With virtio-blk you can use x-data-plane=on, which increases IOPS in guest and lowers overhead in host. In windows guest I wasn't able to pass trim operations through virtio-blk as opposed to virtio-scsi. Later I found a workaround to this by using detect-zeroes=on with discard='unmap' and writing zeros to desired sectors (zero out unused disk space inside guest).

Offline

#3704 2014-12-29 20:28:57

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have a weird issue. I haven't been able to solve this. When I start Windows 8.1 a message pops up in Arch. It says:

vcpu 2 ignored rdmsr kvm
vcpu 4 ignored rdmsr kvm
vcpu 6 ignored rdmsr kvm

This repeats itself for as long as I am running the VM.

I have a i7-4790k and here is some more info about it:

lscpu:

Architecture:          x86_64
CPU op-mode(s):             32-bit, 64-bit
Byte Order:                       Little Endian
CPU(s):                            8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):                  1
NUMA node(s):          1
Vendor ID:                 GenuineIntel
CPU family:               6
Model:                       60
Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Stepping:                  3
CPU MHz:                4392.031
CPU max MHz:        4400.0000
CPU min MHz:         800.0000
BogoMIPS:               8000.34
Virtualization:            VT-x
L1d cache:                32K
L1i cache:                 32K
L2 cache:                  256K
L3 cache:                  8192K
NUMA node0 CPU(s):     0-7

virsh capabilities shows this

<topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>12061780</memory>
          <pages unit='KiB' size='4'>3015445</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='8'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,7'/>
            <cpu id='4' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='5' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='6' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='7' socket_id='0' core_id='3' siblings='3,7'/>
          </cpus>
        </cell>
      </cells>
    </topology>


this is what my XML file looks like. 


<vcpu current="7" placement="static">8</vcpu>
-<cputune>
<vcpupin cpuset="0" vcpu="0"/>
<vcpupin cpuset="1" vcpu="1"/>
<vcpupin cpuset="2" vcpu="2"/>
<vcpupin cpuset="3" vcpu="3"/>
<vcpupin cpuset="4" vcpu="4"/>
<vcpupin cpuset="5" vcpu="5"/>
<vcpupin cpuset="6" vcpu="6"/>
</cputune>

I also tried different settings like only this:

<vcpu placement='static'>7</vcpu>

or like this:

<vcpu placement='static' current='4'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0,4'/>
    <vcpupin vcpu='1' cpuset='1,5'/>
    <vcpupin vcpu='2' cpuset='2,6'/>
    <vcpupin vcpu='3' cpuset='3,7'/>
  </cputune>

I have to say - and maybe you guys can even tell from my post, I am not even sure what's relevant here and what not. So if you need any more info just tell me what you need.
I am really grateful for your help.

Last edited by 4kGamer (2014-12-29 20:51:49)

Offline

#3705 2014-12-29 21:48:07

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi, I have a weird issue. I haven't been able to solve this. When I start Windows 8.1 a message pops up in Arch. It says:

vcpu 2 ignored rdmsr kvm
vcpu 4 ignored rdmsr kvm
vcpu 6 ignored rdmsr kvm

It says that virtual CPU number X didn't reply to command "read MSR(model specific register) KVM". Do you passthrough an nvidia card? If so, i suppose, you have kvm=off somewhere, hiding KVM? Something hints me that it is related to your errors. BTW, that seems like just a warning, so you can safely ignore it.

And a thing to test:

nbhs wrote:

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

That may help too.

When i've launched some software like CPU-Z, i'd get some of these messages, but generally they're harmless.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3706 2014-12-29 22:43:45

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I do have an Nvidia Card. The MSI 980 GTX.

I'm relieved, if that error message is harmless.


I do have the option kvm ignore_msrs=1. I also have other options (like vfio_iommu_type1 disable_hugepages=1), but they are all under one file: /etc/modprobe.d/modprobe.conf

Does it matter what name the .conf file in /etc/modprobe.d/ has?

Offline

#3707 2014-12-30 10:10:05

adejour
Member
Registered: 2014-04-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Whaaaa !
Incredible..

I'm stuck since 1 year on this no-reboot issue, and with this driver, it works (I'm not passing audio for the moment) !!

kidchunks wrote:

I was able to install the drivers but upon reboot, got stuck in a boot loop.

Figured it could be driver related so I tried the latest catalyst beta drivers (14.11.2) and it worked! Audio and video are passing through with no problems. smile

The above also fixed an issue I had with video card freezing (stuck image) on reboot/shutdown so no need for no-reboot.

The only problem I have now is that I can't see the bios(seabios) screen but that's minor given it still boots into whatever drive is first. Strangely enough, OVMF(UEFI) works fine. Maybe something with the card.

In any case, glad to get it working after a week!

Thanks AW and others for the help.

To those running an r9 290x...use the 14.11.2 drivers!

Just a little resumé of my configuration :
MB : ASUS H87M-PLUS
GPU : Radeon R7 260X
RAM : 8Gb (4Gb passed to guest)
CPU : i5 4570
Kernel : 3.15.1 compiled locally for pci passthrough
Guest : Windows 7 Home Prem. x64
CCC : 14.11.2 beta drivers  (17/11/2014)

And I have to notice that the last 14.12 drivers (they say "14.501.1003" Display Driver ver. - 9/12/2014) don't work (still the no-reboot problems)!!!!

So, I would advice to everyone having (reboot) issue to passthrough R7 & R9 to try these drivers.

Furthermore, the Windows Benchmark gives some good results I guess :
CPU : 5.9
Memory : 7.3
Graphic : 7.7
Gaming graphic : 7.7
HD : 7.9

Remaing issue for the moment :

- sometimes (can't reproduce), I have a blue screen "MEMORY MANAGEMENT"
- tried to launch DOTA 2, but CPU get to 100% and the screen of the game stay black (I can come back to the desktop & kill the dota 2 process)

Here is my config, I saw it is quite different than kidchunks one (https://bbs.archlinux.org/viewtopic.php … 5#p1481075)  :
If you have any advice :

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1 # Radeon R7
/usr/bin/vfio-bind 0000:00:1d.0 # USB on back side
qemu-system-x86_64 \
-M q35 -enable-kvm -balloon none -rtc clock=host -cpu host \
-m 4G -smp 1,sockets=1,cores=1,threads=1 \
-bios /usr/share/qemu/bios.bin \
-acpitable file=/sys/firmware/acpi/tables/SLIC \
-serial null -parallel null \
-net nic,model=virtio,macaddr=xx:xx:xx:xx:xx:xx \
-net nic -net tap,ifname=tap2,script=no,downscript=no \
-monitor none -display none \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=VM/win7.img,id=disk0,format=raw -device scsi-hd,drive=disk0 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1d.0,bus=pcie.0 \
-drive file=iso/Win7HomePrem64.X15-65739.iso,id=isocd -device scsi-cd,drive=isocd \
-drive file=iso/virtio-win-0.1-94.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd

Offline

#3708 2014-12-30 11:00:33

Silar
Member
Registered: 2014-12-24
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Dump the ROM or get it from techpowerup database and use the romfile= option to provide the ROM to the guest.

I tried both, dumping die ROM and get it from techpowerup. Now i get no more error messages and qemu starts but i get no video signal from the guests graphic card. In addition the host freeze after a few seconds. And i don't know what happens and how to fix this problem.

Offline

#3709 2014-12-30 11:30:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Silar wrote:
aw wrote:

Dump the ROM or get it from techpowerup database and use the romfile= option to provide the ROM to the guest.

I tried both, dumping die ROM and get it from techpowerup. Now i get no more error messages and qemu starts but i get no video signal from the guests graphic card. In addition the host freeze after a few seconds. And i don't know what happens and how to fix this problem.

I had a similar problem with my GTX470, the first run worked great, but after a guest reboot i had the same problem as you do.

What i did was, reboot the host, start the guest up with the GTX470 as primary, dump the rom with gpu-z, copy it back to your host, shut down the guest, then load the gpu rom it like this:

-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/path/to/your/gpu/bios.bin

I found out that in my case, if i ran the guest for a second time WITHOUT the romfile, i wouldnt be able to start it again even if i actually loaded the romfile.

Last edited by nbhs (2014-12-30 11:32:17)

Offline

#3710 2014-12-30 14:12:54

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need a bit of help here. My system is Supermicro X9DA7 with two Xeon E5-2630v2 . Kernel version 3.18.1 mainline with no patches, qemu version 2.2.0 (with no patches either), seabios 1.7.5-2

Relevant dmesg excerpt:

[bronek@gdansk ~]$ dmesg | grep -iE "iommu|stub" [ 0.000000] Command line: BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw quiet edac_core.edac_mc_panic_on_ue=1 elevator=noop pci-stub.ids=1002:6749,1002:aa90,1002:67b0,1002:aac8 intel_iommu=on spl.spl_hostid=0xbb51d4d8 initrd=../initramfs-linux.img [ 0.000000] Kernel command line: BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw quiet edac_core.edac_mc_panic_on_ue=1 elevator=noop pci-stub.ids=1002:6749,1002:aa90,1002:67b0,1002:aac8 intel_iommu=on spl.spl_hostid=0xbb51d4d8 initrd=../initramfs-linux.img [ 0.000000] Intel-IOMMU: enabled [ 0.086314] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de [ 0.086320] dmar: IOMMU 1: reg_base_addr bfffc000 ver 1:0 cap d2078c106f0466 ecap f020de [ 0.086451] IOAPIC id 3 under DRHD base 0xfbffe000 IOMMU 0 [ 0.086453] IOAPIC id 0 under DRHD base 0xbfffc000 IOMMU 1 [ 0.086454] IOAPIC id 2 under DRHD base 0xbfffc000 IOMMU 1 [ 1.135393] IOMMU 0 0xfbffe000: using Queued invalidation [ 1.135394] IOMMU 1 0xbfffc000: using Queued invalidation [ 1.135400] IOMMU: Setting RMRR: [ 1.135423] IOMMU: Setting identity map for device 0000:00:1a.0 [0x7f23d000 - 0x7f24bfff] [ 1.135470] IOMMU: Setting identity map for device 0000:00:1d.0 [0x7f23d000 - 0x7f24bfff] [ 1.135507] IOMMU: Setting identity map for device 0000:07:00.0 [0x7f23d000 - 0x7f24bfff] [ 1.135555] IOMMU: Setting identity map for device 0000:08:00.0 [0x7f23d000 - 0x7f24bfff] [ 1.135608] IOMMU: Setting identity map for device 0000:83:00.0 [0x7f23d000 - 0x7f24bfff] [ 1.135686] IOMMU: Setting identity map for device 0000:84:00.0 [0x7f23d000 - 0x7f24bfff] [ 1.135742] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 1.135760] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 1.151027] pci-stub: add 1002:6749 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.151034] pci-stub 0000:03:00.0: claimed by stub [ 1.151042] pci-stub: add 1002:AA90 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.151059] pci-stub 0000:03:00.1: claimed by stub [ 1.151067] pci-stub: add 1002:67B0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.151090] pci-stub 0000:82:00.0: claimed by stub [ 1.151107] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.151137] pci-stub 0000:82:00.1: claimed by stub

As you can see, I have two GPU cards from AMD, both claimed by pci-stub

[bronek@gdansk ~]$ lspci -nn | grep 1002 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Turks GL [FirePro V4900] [1002:6749] 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Turks/Whistler HDMI Audio [Radeon HD 6000 Series] [1002:aa90] 82:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X] [1002:67b0] 82:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aac8]

Also both cards are passed successfully to vfio-pci by script presented in the first post on this thread:

[bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:03:00.1/driver lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:03:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci [bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:03:00.0/driver lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:03:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci [bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:82:00.1/driver lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:82:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci [bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:82:00.0/driver lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:82:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci

When trying VGA passthrough I get the following:

[root@gdansk bronek]# cat trypc.sh #!/bin/bash qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host -smp 6 \ -bios /usr/share/qemu/bios.bin \ -nographic -vga none \ -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,addr=08.1 [root@gdansk bronek]# ./trypc.sh qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: No available IOMMU models qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 18 qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to get group 18 qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I receive the same error for both cards and also when using -M q35 (with ioh3420). In case that matters, this machine is intended as headless VDI server and has no X windows installed. Radeon driver is blacklisted. I do not have ACS patch as it does not seem necessary in this case:

[bronek@gdansk ~]$ ls /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices 0000:03:00.0 0000:03:00.1

Also, I am not using IGP thus no arb patch and also no option allow_unsafe_interrupts=1, as it seems not necessary for my motherboard. I did try it though, with no effect.

I was following this thread from start (yes, started last year!) and I did try this feature previously using older kernel and older qemu version, with little more success (was able to install Linux guest, but not Windows). This time I'm stuck at the very beginning. Perhaps if I downgraded qemu or kernel it might help a little ?

Last edited by Bronek (2014-12-30 14:14:54)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3711 2014-12-30 14:19:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
[root@gdansk bronek]# cat trypc.sh #!/bin/bash qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host -smp 6 \ -bios /usr/share/qemu/bios.bin \ -nographic -vga none \ -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,addr=08.1 [root@gdansk bronek]# ./trypc.sh qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: No available IOMMU models qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 18 qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to get group 18 qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What does 'grep VFIO /boot/config-`uname -r`' show?  lsmod | grep vfio?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3712 2014-12-30 14:30:16

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Alex thanks for prompt reply. I think ArchLinux does not save a config copy to /boot , but I have this:

[bronek@gdansk ~]$ zcat /proc/config.gz | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y [bronek@gdansk ~]$ lsmod | grep vfio vfio_pci 31566 0 vfio 18519 1 vfio_pci

Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3713 2014-12-30 14:42:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

Hi Alex thanks for prompt reply. I think ArchLinux does not save a config copy to /boot , but I have this:

[bronek@gdansk ~]$ zcat /proc/config.gz | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y [bronek@gdansk ~]$ lsmod | grep vfio vfio_pci 31566 0 vfio 18519 1 vfio_pci

So somehow vfio_iommu_type1 is not getting auto-loaded.  The way it's supposed to work is that vfio-pci depends on vfio, so loading vfio-pci always brings in vfio.  vfio then does a module request for any available iommu drivers to load those.  This is where vfio_iommu_type1 is supposed to get loaded.  Maybe something has broken in your userspace that the module request is not being serviced?  Anyway, you should be able to manually `modprobe vfio_iommu_type1`


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3714 2014-12-30 14:55:21

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

This is where vfio_iommu_type1 is supposed to get loaded.  Maybe something has broken in your userspace that the module request is not being serviced?  Anyway, you should be able to manually `modprobe vfio_iommu_type1`

Thank you, this has helped!

[root@gdansk bronek]# modprobe vfio_iommu_type1 [root@gdansk bronek]# lsmod | grep vfio vfio_iommu_type1 17163 0 vfio_pci 31566 0 vfio 18519 2 vfio_iommu_type1,vfio_pci

I will add this module to /etc/mkinitcpio.conf to ensure it is always automatically loaded.

Also,  I suppose I need your other arb patch to which you refer in your  blog http://vfio.blogspot.co.uk/2014/08/vfiovga-faq.html question 4  - since after closing qemu (SIGTERM from other session) I got a bit of mess on my VGA screen.

EDIT Never mind, I see your patch is already applied in my kernel version (3.18.1) ! I guess the mess I've got is simply result of qemu being cautions on handling SIGTERM

Last edited by Bronek (2014-12-30 15:32:29)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3715 2014-12-30 15:27:13

adejour
Member
Registered: 2014-04-29
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

About my last post (vga passthrough working with beta ATI drivers 14.11.2) https://bbs.archlinux.org/viewtopic.php … 2#p1488792

I succeed to make audio working adding  -device vfio-pci,host=01:00.1,bus=pcie.0 to the command line.
Now, all games are working well (tested dota 2 and bioshock infinite).

thank you all !

Offline

#3716 2014-12-30 23:06:39

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:
pkim wrote:

2. I can't figure out how to pipe only the sound through spice. Work-around: use Xen for video card type.

It works when you have the VM client open and connected (virt-manager, spicy). This requires virtual graphics, which can be disabled in quest, but I think a better solution is to use pulseaudio server directly. I have found information on this somewhere earlier in this thread:

/etc/pulse/default.pa: .. load-module module-native-protocol-unix auth-anonymous=1 ..

and with added configuration in VM xml:

 <qemu:commandline> <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/> <qemu:env name='QEMU_PA_SAMPLES' value='1024'/> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/> <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/> </qemu:commandline>

In guest, you are then using virtual sound card configured in xml the standard way.
It doesnt work if you are running qemu as root (or not dropping capabilities through libvirt..?), but thanks to aws qemu wrapper script it works for me perfectly (permissions problems without it). You have to have logged in session on host, too.
I didn't dive into security of pulseaudio server after this, there could be some problems.

OK, I tried doing what you suggested, but I couldn't get any sound once I deleted the graphics and video section in the xml file.

On the other hand, I did figure out the glitch/stutter/click issues on the sound. It turns out others are having the same issue with ICH6 and ICH9 - here.

I set up the audio for AC'97 in virt-manager and installed the AC'97 driver in Windows 7 - worked without any glitch! :-)

The only "real" issue now is the PCIe reset - can't reboot without suspending/resuming the host.

Offline

#3717 2014-12-31 01:05:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I'm feeling like my GPU's vendor dumped me.
Some months ago i've tried to update my card's VBIOS on UEFI, it downloaded the image but failed to apply it due to lack of drivers or some other stuff - it wasn't my pc, so i didn't want to mess with owner's windows system like installing drivers.

But i got the ROM. For one card only, but that's fine.
Tried passing it wth romfile and OVMF just hanged, aw suggested to test it on real hardware to see if it's just poorly written and broken. So i gave up for some time.

Now, i've installed windows system on a bare metal system using UEFI+GPT and even secureboot was working. But ASUS UEFI Update tools now says that... my GPU doesn't support UEFI. Considering the fact i have the actual ROM... It seems strange.

So, it's time to begin my journey on writing messages to their support team.

I've got some news i've decided to share with you, folks.

I wrote to the support team, but they're being all "jelly-brained" and stuff. I think that is related to holidays.
Anyway, they've requested a screenshot of main GPU-Z window and provided a VBIOS image that was lacking serial number information but was the same as original. It hints that this information is left for decoration purposes only(or maybe some other non-essential stuff), the main meat-and-bones of ROM will work without it.

That yet remains open my two questions - why the hell their tool won't work anymore, and is it right to flash the bios or it's considered buggy. The journey shall continue.


But i definitely can tell one important thing:
If any of you are trying to use OVMF with a card that "may be updated to support UEFI" - think twice before doing that, updating it may not be so easy.

Last edited by Duelist (2014-12-31 01:06:03)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3718 2015-01-02 13:35:46

Hoeze
Member
Registered: 2014-12-30
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
how can I disable/re-enable my GPU after stopping a guest?
I'd like to disable my GPU until it's needed to save power...

Last edited by Hoeze (2015-01-02 13:45:26)

Offline

#3719 2015-01-03 02:01:09

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've got a couple questions regarding performance optimizations:

1) Neste Page Tables: I found suggestions to disable NPT (also in this thread). Now why would I want to do this? NPT should speed up memory access. Or is there some interaction between NPT and PCI passthrough that makes it less performant?
2) VCPU pinning: Again, I found suggestions to pin each QEMU thread to a single CPU core. I guess it might in fact help to pin each VCPU thread to a core. But QEMU uses more threads than VCPUS (I guess for IO and additional worker threads). So a naive approach might pin multiple VCPU threads to a single core. Is this suggestion then plane wrong, or am I missing something here? Can I determine which threads are VCPU threads? (I'm talking SMP here, not NUMA, so the effect might be negligible anyway).

Offline

#3720 2015-01-03 02:28:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rawuza wrote:

Hi,

I've got a couple questions regarding performance optimizations:

1) Neste Page Tables: I found suggestions to disable NPT (also in this thread). Now why would I want to do this? NPT should speed up memory access. Or is there some interaction between NPT and PCI passthrough that makes it less performant?

The suggestions has been do disable nesting support, ie. the ability to run a hypervisor within the guest, not to disable three dimensional paging.  I haven't seen actual benchmarks from doing this, so I'm not sure how worthwhile it is.

2) VCPU pinning: Again, I found suggestions to pin each QEMU thread to a single CPU core. I guess it might in fact help to pin each VCPU thread to a core. But QEMU uses more threads than VCPUS (I guess for IO and additional worker threads). So a naive approach might pin multiple VCPU threads to a single core. Is this suggestion then plane wrong, or am I missing something here? Can I determine which threads are VCPU threads? (I'm talking SMP here, not NUMA, so the effect might be negligible anyway).

Pinning all vCPUs to the same single core would be the worst optimization ever.  The suggestion is to pin each vCPU to a separate pCPU, which gives you cache locality and makes the host scheduler's job easier.  info cpus lists the vCPU threads.  libvirt also has the ability to do vCPU pinning.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3721 2015-01-03 02:31:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hoeze wrote:

Hi,
how can I disable/re-enable my GPU after stopping a guest?
I'd like to disable my GPU until it's needed to save power...

Unless you have power controllable slots, the best suggestion might be to leave the guest running.  The guest driver can probably put the card in a fairly low power state when it's idle.  I don't expect to see a lot of power savings from D3 in the host, which is about the best we can offer without device specific drivers.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3722 2015-01-03 02:56:54

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The suggestions has been do disable nesting support, ie. the ability to run a hypervisor within the guest, not to disable three dimensional paging.

So that would be the "nested" parameter of kvm_amd. That makes more sense. The comments I found suggested to set "npt" to 0.

Pinning all vCPUs to the same single core would be the worst optimization ever.

Well, yes, I meant pinning each vCPU to a separate core.

info cpus lists the vCPU threads.

That's exactly what I need. Thanks a lot.

PS: I previously had trouble with my old GPU. I've got a new one and it finally works. Quite impressive, I must say.

Offline

#3723 2015-01-03 21:04:13

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I'd like to ask Alex and others here about something I discovered. I ran Open Hardware Monitor and under section "Clocks" I get no value for CPU Core #1 till #4.

It's not only for benchmarking, but my case has a screen where I can follow stuff like that constantly. And it's annoying that it says 4x0 Mhz.

Is this expected behaviour or is something wrong?

Offline

#3724 2015-01-03 21:08:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi, I'd like to ask Alex and others here about something I discovered. I ran Open Hardware Monitor and under section "Clocks" I get no value for CPU Core #1 till #4.

It's not only for benchmarking, but my case has a screen where I can follow stuff like that constantly. And it's annoying that it says 4x0 Mhz.

Is this expected behaviour or is something wrong?

Personally I wouldn't necessarily expect a hardware monitor to work in a virtual machine.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3725 2015-01-03 21:14:07

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it works pretty well. It's just the CPU Clock. So something is preventing it from showing the right value.

Offline

#3726 2015-01-03 22:44:20

TwosComplement
Member
Registered: 2010-10-16
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to everyone who contributed to this thread, but I need a bit of assistance. I've been trying to get this working for hours, but I can't get this setup working no matter which modules or options I use. There's no errors when I bind the card using the script in the OP or when I run QEMU, but I get no signal.

Hardware:
Motherboard: Intel DB85FL
Processor: Intel i7-4770
Graphics (guest):  EVGA GeForce GTX 760
Graphics (host): Intel

I'm using the latest kernel in the repo at the moment (3.17.6), as I am under the impression that the patches in the OP are now included in the official arch kernel. Is this correct, or did I skip the most important step?

Here's some command outputs.

$ zcat /proc/config | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y
$ cat /proc/cmdline \boot\vmlinuz-linux root=/dev/sda2 rootfstype=ext4 rw intel_iommu=on i915.enable_hd_vgaarb=1 initrd=boot\initramfs-linux.img
$ cat /etc/modprobe.d/vgapassthru.conf blacklist nouveau blacklist nvidia options kvm ignore_msrs=1 options pci-stub ids=10de:1187,10de:0e0a options vfio_iommu_type1 allow_unsafe_interrupts=1
$ lsmod | grep vfio vfio_iommu_type1 17118 0 vfio_pci 35525 0 vfio 18477 2 vfio_iommu_type1,vfio_pci
sudo qemu-system-x86_64 \ -enable-kvm -M q35 -m 8G -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -cpu kvm=off \ -mem-prealloc \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

The monitor is connected to the graphics card via a DVI cable, if that's relevant.

Anyone have any ideas on how to get my card working? I would very much appreciate it as I would like to finally boot Arch full-time.

Last edited by TwosComplement (2015-01-03 22:47:07)

Offline

#3727 2015-01-03 22:52:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TwosComplement wrote:

Thanks to everyone who contributed to this thread, but I need a bit of assistance. I've been trying to get this working for hours, but I can't get this setup working no matter which modules or options I use. There's no errors when I bind the card using the script in the OP or when I run QEMU, but I get no signal.

Hardware:
Motherboard: Intel DB85FL
Processor: Intel i7-4770
Graphics (guest):  EVGA GeForce GTX 760
Graphics (host): Intel

I'm using the latest kernel in the repo at the moment (3.17.6), as I am under the impression that the patches in the OP are now included in the official arch kernel. Is this correct, or did I skip the most important step?

<ding, ding, ding> We have a winner!  The i915 patch is not upstream and is not destined to ever be upstream.  If you want to use an unmodified kernel, you either need to get rid of Intel graphics on the host or use OVMF in the guest, which your video card should support.

Here's some command outputs.

$ zcat /proc/config | grep VFIO CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_KVM_VFIO=y
$ cat /proc/cmdline \boot\vmlinuz-linux root=/dev/sda2 rootfstype=ext4 rw intel_iommu=on i915.enable_hd_vgaarb=1 initrd=boot\initramfs-linux.img
$ cat /etc/modprobe.d/vgapassthru.conf blacklist nouveau blacklist nvidia options kvm ignore_msrs=1 options pci-stub ids=10de:1187,10de:0e0a options vfio_iommu_type1 allow_unsafe_interrupts=1

This vfio option is probably not needed.  pci-stub is a poor choice to make a module, you generally want it builtin so that it can claim devices before host drivers.  Shouldn't be an issue in this case since you're blacklisting the relevant host drivers.

sudo qemu-system-x86_64 \ -enable-kvm -M q35 -m 8G -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -cpu kvm=off \ -mem-prealloc \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

You don't specify your guest OS, but I'll repeat for the billionth time, if it's Windows, don't bother with q35, just use 440fx.  You also don't need to specify a bios and -mem-prealloc does nothing in this case.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3728 2015-01-03 23:18:31

TwosComplement
Member
Registered: 2010-10-16
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you so much for your help! I really need to be more careful about reading, sorry about the misunderstanding. I finally have it working with OVMF and 440fx. Do you accept tips?

Offline

#3729 2015-01-04 03:42:12

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw: I'm curious if you have any feedback on my post, with the setup I'm using. I've made a few edits since I originally posted it, based on more reading and some suggestions relating to the changes others have made.

Offline

#3730 2015-01-04 05:39:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Myranti wrote:

@aw: I'm curious if you have any feedback on my post, with the setup I'm using. I've made a few edits since I originally posted it, based on more reading and some suggestions relating to the changes others have made.

The vfio-bind script is unnecessary in your configuration.  With managed="yes" libvirt should automatically bind the devices to vfio-pci when you start the VM.

It looks like you motherboard has 4 x16 slots, the top 3 sourced from the CPU, the bottom (and the x1 slot and PCI slots via the crappy ASM1083) sourced from the PCH.  If that's true, then the top 3 x16 slots will always be grouped together because Intel doesn't feel like supporting ACS on Xeon E3-1200 series.  If you could find a GPU that can fit in that bottom slot for the host, then you could at least run a test w/o needing the ACS override hackery.  Asus boards are pretty poor about letting you pick the primary graphics though.  I wouldn't rule out lack of isolation between GPUs as a source of the black lines, there is a reason the ACS patch is not upstream.  Maybe there are Crossfire settings in the BIOS that can be disabled.  Trying a non-Radeon card for host or guest would also be a good experiment.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3731 2015-01-04 06:52:33

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The vfio-bind script is unnecessary in your configuration.  With managed="yes" libvirt should automatically bind the devices to vfio-pci when you start the VM.

Ah, that's easier if I don't need to do the extra to bind it as well since I've already done the unbinding and binding over to pci-stub.

aw wrote:

It looks like you motherboard has 4 x16 slots, the top 3 sourced from the CPU, the bottom (and the x1 slot and PCI slots via the crappy ASM1083) sourced from the PCH.  If that's true, then the top 3 x16 slots will always be grouped together because Intel doesn't feel like supporting ACS on Xeon E3-1200 series. If you could find a GPU that can fit in that bottom slot for the host, then you could at least run a test w/o needing the ACS override hackery.  Asus boards are pretty poor about letting you pick the primary graphics though.  I wouldn't rule out lack of isolation between GPUs as a source of the black lines, there is a reason the ACS patch is not upstream.  Maybe there are Crossfire settings in the BIOS that can be disabled.  Trying a non-Radeon card for host or guest would also be a good experiment.

Yeah, I found there was a large IOMMU group with the PCIe slots and would require too many devices to be passed through. What are Asrock boards like as an alternative, at least from the perspective of picking primary graphics? I've considered replacing the board with the Asrock C226-WS. I'll see if I can find a better setup/arrangement in bios, but another issue was that the cooler I'm using goes ever so slightly into the first x16 slot, so I ended up using it for the USB 3 card. Perhaps a different board with different layout would give me flexibility to move the cards elsewhere without needing to use the ACS patch.

As for the black lines, I tested a GTX 460 and an R7 250 and didn't have any issues with lines, only with the R5 230, so I'm not entirely sure what the issue is there.

Thanks for the feedback though, it gives me some stuff to work with and see what I can change or what might work better!

Last edited by Myranti (2015-01-04 06:53:19)

Offline

#3732 2015-01-04 12:58:33

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

I tried installing windows 7 using ovmf but i could never get it to work, i believe it needs the csm present, so i gave up and installed windows 8 with a patched ovmf to allow it to boot directly from the amd sata controller.
OVMF-sata.tar.gz
Also, once you're done installing windows 8 disable fast boot


Hi, I found your post from early December. Can you tell me how you patched the ovmf file with the amd sata controller? I have 3 Sata Controllers available and I'd like to try the Intel one and boot from it. How can I patch the ovmf myself with the Intel sata controller drivers? Is this easy?

Thank you very much for your help!

Last edited by 4kGamer (2015-01-04 13:34:59)

Offline

#3733 2015-01-04 13:57:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:
nbhs wrote:

I tried installing windows 7 using ovmf but i could never get it to work, i believe it needs the csm present, so i gave up and installed windows 8 with a patched ovmf to allow it to boot directly from the amd sata controller.
OVMF-sata.tar.gz
Also, once you're done installing windows 8 disable fast boot


Hi, I found your post from early December. Can you tell me how you patched the ovmf file with the amd sata controller? I have 3 Sata Controllers available and I'd like to try the Intel one and boot from it. How can I patch the ovmf myself with the Intel sata controller drivers? Is this easy?

Thank you very much for your help!

Just google reza sata patch v3, and it's not just for amd controllers, you can also use qemu's emulated one, and no it not that easy, you could use the binaries i provided

Last edited by nbhs (2015-01-04 13:59:36)

Offline

#3734 2015-01-04 14:11:26

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you!

one question: since it's not that easy, can I just use the patched ovmf you provided and boot to Windows with it and afterwards install the Intel sata drivers? I am not sure if I'd lose performance that way.

Offline

#3735 2015-01-04 15:07:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you!

one question: since it's not that easy, can I just use the patched ovmf you provided and boot to Windows with it and afterwards install the Intel sata drivers? I am not sure if I'd lose performance that way.

You need to use the patched binaries if you plan on booting from the controller, without the patch ovmf wont see your drives and you simply cant boot from it unless you dump/get an intel UEFI (not the windows driver) driver for your controller

Here's a guide on how to dump and uefi driver from memory: https://cs.nyu.edu/~gazzillo/xeniac/ext … river.html

But you wont need to do this if you're using my binaries

Last edited by nbhs (2015-01-04 15:13:30)

Offline

#3736 2015-01-04 15:26:48

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess it just doesn't Impact performance if I use those binaries and update SATA Drivers in Windows... Thank you for the link. But I will just use yours. And it worked! Thank you!

Now what I realised is that the only issue I am having are low IOPS (30.000+ compared to bare metal 90.000+).

Is there something I can do about it or is SATA passthrough simply not good enough?

Last edited by 4kGamer (2015-01-04 16:59:37)

Offline

#3737 2015-01-04 15:59:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Myranti wrote:

Yeah, I found there was a large IOMMU group with the PCIe slots and would require too many devices to be passed through. What are Asrock boards like as an alternative, at least from the perspective of picking primary graphics? I've considered replacing the board with the Asrock C226-WS. I'll see if I can find a better setup/arrangement in bios, but another issue was that the cooler I'm using goes ever so slightly into the first x16 slot, so I ended up using it for the USB 3 card. Perhaps a different board with different layout would give me flexibility to move the cards elsewhere without needing to use the ACS patch.

Gigabyte boards seem to be the most configurable for setting primary graphics in the BIOS.  Asrock annoyed me by being more eager to tell me they don't support Linux than to answer my technical question back when I was looking for an AMD-Vi system, so I don't have any experience with them.  I doubt the USB3 card in the top slot is really hurting anything, it's just not making very good use of the bandwidth available in that slot.  Are you currently ordering the cards to make the host GPU the primary graphics?

As for the black lines, I tested a GTX 460 and an R7 250 and didn't have any issues with lines, only with the R5 230, so I'm not entirely sure what the issue is there.

Could certainly be some peer-to-peer enabled when multiple Radeon cards are installed.  Good for enabling Crossfire, bad for isolation with a VM.  Maybe this is why Intel won't support isolation on "client" CPU root ports.  You might look at motherboards that would allow you to install the guest GPU in a PCH root port.  These are often identifiable as the PCIe2.0 slots rather than PCIe3.0.  Assigning the onboard USB3 card to the guest and using the plugin card for the host may also provide more flexibility.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3738 2015-01-05 09:36:03

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Gigabyte boards seem to be the most configurable for setting primary graphics in the BIOS.  Asrock annoyed me by being more eager to tell me they don't support Linux than to answer my technical question back when I was looking for an AMD-Vi system, so I don't have any experience with them.  I doubt the USB3 card in the top slot is really hurting anything, it's just not making very good use of the bandwidth available in that slot.  Are you currently ordering the cards to make the host GPU the primary graphics?

Yeah, the cards are ordered so that the host GPU (The R5 230) is primary graphics in the next usable slot after the one being taken up by the USB 3 card.

Could certainly be some peer-to-peer enabled when multiple Radeon cards are installed.  Good for enabling Crossfire, bad for isolation with a VM.  Maybe this is why Intel won't support isolation on "client" CPU root ports.  You might look at motherboards that would allow you to install the guest GPU in a PCH root port.  These are often identifiable as the PCIe2.0 slots rather than PCIe3.0.  Assigning the onboard USB3 card to the guest and using the plugin card for the host may also provide more flexibility.

I think it might be worth seeing what alternatives there are for motherboards. The main concern is still getting enough PCIe slots, seeing as I'll have no use for PCI slots with my setup.

Edit: It looks like Supermicro have some workable boards with Primary Display selection (looking at the X10SAT), although I from what I can see there may very well be the same sort of issue with IOMMU groups as what I have now.

Last edited by Myranti (2015-01-06 13:27:21)

Offline

#3739 2015-01-06 10:41:33

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all i want to ask. Are there some performance benefits when i use kernel 3.18 instead my 3.15? or another reasons?

Offline

#3740 2015-01-06 15:21:09

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

magemaster wrote:
DelusionalLogic wrote:

If followed the guide, installed windows 8, and it kinda worked, execept the nividia driver. I enabled "kvm=off" on the cpu parameter, and installed the driver (from a cirrus vga adapter, with the card passthroug as secondary). It installed fine, but now i'm getting BSOD's from the VM. The error is "SYSTEM_SERVICE_EXCEPTION" and it's originating from the nvidia driver.

I then tried installing windows 7, but that's just giving me this whenever i try to boot. the installation went fine (this is without any passthrough).

KVM internal error. Suberror: 1 emulation failure EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080 ESI=0025db2a EDI=0007db2a EBP=00007c00 ESP=00000200 EIP=000000ca EFL=00010002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0020 00000200 0000ffff 00009300 CS =b000 000b0000 0000ffff 00009f00 SS =0020 00000200 0000ffff 00009300 DS =0020 00000200 0000ffff 00009300 FS =0020 00000200 0000ffff 00009300 GS =0020 00000200 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 002b0000 0000001f IDT= 00000000 000003ff CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

Here's my script for launching the vm:

vfio-bind 0000:03:00.0 0000:03:00.1 qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga cirrus \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -drive file=/home/delusional/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/delusional/vm/install.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -usb -usbdevice host:1038:1361 -usbdevice host:1532:010d \ -net nic -net bridge,br=br0 \ -boot menu=on

I am aware that this would not enable the passthrough anyway (seeing as the nividia card is secondary) but at least it should boot right?


I was having the same error before I patched the kernel.


Which patch fixed it? I can't see the VGA arbiter patch having much effect since I'm not using the i915 driver. Also what OS are you running in the VM?

Last edited by DelusionalLogic (2015-01-06 15:22:19)

Offline

#3741 2015-01-06 16:54:25

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got my Win10 guest to boot with my GTX 760 passed through, though I am unable to get USB working:

qemu-system-x86_64: util/qemu-option.c:387: qemu_opt_get_bool_helper: Assertion `opt->desc && opt->desc->type == QEMU_OPT_BOOL' failed.

Searching for this error, it appears the latest commit of QEMU introduced this error - recent results from Google in last two weeks. Originally my host was setup for XEN, but I disabled those features and booted off your linux-mainline kernel. On installing QEMU-GIT it had to unisntall regular QEMU. Does pulling down QEMU-GIT also grab the latest QEMU source? Could this issue be from an unknown failure of the uninstall?

Offline

#3742 2015-01-06 18:01:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

I've got my Win10 guest to boot with my GTX 760 passed through, though I am unable to get USB working:

qemu-system-x86_64: util/qemu-option.c:387: qemu_opt_get_bool_helper: Assertion `opt->desc && opt->desc->type == QEMU_OPT_BOOL' failed.

Searching for this error, it appears the latest commit of QEMU introduced this error - recent results from Google in last two weeks. Originally my host was setup for XEN, but I disabled those features and booted off your linux-mainline kernel. On installing QEMU-GIT it had to unisntall regular QEMU. Does pulling down QEMU-GIT also grab the latest QEMU source? Could this issue be from an unknown failure of the uninstall?

If you're unwilling to deal with breakages, you should probably be building from a released tag in qemu.git and not just the current development head.  git checkout v2.2.0


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3743 2015-01-06 18:43:38

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How did you all compile the modified kernel? The "acs override" patch doesn't seem to be compatible with the 3.17 "quirks.c"

Offline

#3744 2015-01-06 18:50:07

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
The_Moves wrote:

I've got my Win10 guest to boot with my GTX 760 passed through, though I am unable to get USB working:

qemu-system-x86_64: util/qemu-option.c:387: qemu_opt_get_bool_helper: Assertion `opt->desc && opt->desc->type == QEMU_OPT_BOOL' failed.

Searching for this error, it appears the latest commit of QEMU introduced this error - recent results from Google in last two weeks. Originally my host was setup for XEN, but I disabled those features and booted off your linux-mainline kernel. On installing QEMU-GIT it had to unisntall regular QEMU. Does pulling down QEMU-GIT also grab the latest QEMU source? Could this issue be from an unknown failure of the uninstall?

If you're unwilling to deal with breakages, you should probably be building from a released tag in qemu.git and not just the current development head.  git checkout v2.2.0

Yeah, that makes sense. I'll need to recompile with the correct build. I've only been using Arch for three days now and it is a lot different from what i'm used to, RHEL 5/6 Solaris AIX. I'm actually liking it quite a lot.

I assume i'm supposed to do v2.2.0, but how do I pull that specific version instead of automatically getting the latest and greatest?

Last edited by The_Moves (2015-01-06 18:50:37)

Offline

#3745 2015-01-06 18:52:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
aw wrote:
The_Moves wrote:

I've got my Win10 guest to boot with my GTX 760 passed through, though I am unable to get USB working:

qemu-system-x86_64: util/qemu-option.c:387: qemu_opt_get_bool_helper: Assertion `opt->desc && opt->desc->type == QEMU_OPT_BOOL' failed.

Searching for this error, it appears the latest commit of QEMU introduced this error - recent results from Google in last two weeks. Originally my host was setup for XEN, but I disabled those features and booted off your linux-mainline kernel. On installing QEMU-GIT it had to unisntall regular QEMU. Does pulling down QEMU-GIT also grab the latest QEMU source? Could this issue be from an unknown failure of the uninstall?

If you're unwilling to deal with breakages, you should probably be building from a released tag in qemu.git and not just the current development head.  git checkout v2.2.0

Yeah, that makes sense. I'll need to recompile with the correct build. I've only been using Arch for three days now and it is a lot different from what i'm used to, RHEL 5/6 Solaris AIX. I'm actually liking it quite a lot.

I assume i'm supposed to do v2.2.0, but how do I pull that specific version instead of automatically getting the latest and greatest?

Maybe you shouldn't be using git.  http://wiki.qemu.org/Download  I assume there are ways to get what you need from arch too.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3746 2015-01-06 18:55:32

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, you are correct. I was making it harder than it was supposed to be

Offline

#3747 2015-01-06 22:24:46

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So i applied the "ACS override" patch and tried running it again (That was a lot of work). Sadly it changed nothing.

To reiterate the issue:
I installed windows 8 and the nvidia drivers using the cirrus drivers with my nvidia card as a secondary passthrough. That part works. The problem occurs when I start the VM with the nvidia drivers installed (the card is still passed through as secondary). It just immediately causes a BSOD with a "SYSTEM_SERVICE_EXCEPTION".

I have the ACS override patch applied and IOMMU enables (with some custom patches to the bios)

My hardware is:
Intel I7 920
Asus P6T deluxe V2 (Yes, i know it's broken. I patched the bios, and it IOMMU does enable)
Nvidia GTX 460 (Passthrough)
Nvidia GTX 660ti (Main)

Offline

#3748 2015-01-06 22:30:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

So i applied the "ACS override" patch and tried running it again (That was a lot of work). Sadly it changed nothing.

To reiterate the issue:
I installed windows 8 and the nvidia drivers using the cirrus drivers with my nvidia card as a secondary passthrough. That part works. The problem occurs when I start the VM with the nvidia drivers installed (the card is still passed through as secondary). It just immediately causes a BSOD with a "SYSTEM_SERVICE_EXCEPTION".

I have the ACS override patch applied and IOMMU enables (with some custom patches to the bios)

My hardware is:
Intel I7 920
Asus P6T deluxe V2 (Yes, i know it's broken. I patched the bios, and it IOMMU does enable)
Nvidia GTX 460 (Passthrough)
Nvidia GTX 660ti (Main)

Where is it claimed that GeForce cards work as secondary in the guest?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3749 2015-01-06 22:59:52

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DelusionalLogic wrote:

So i applied the "ACS override" patch and tried running it again (That was a lot of work). Sadly it changed nothing.

To reiterate the issue:
I installed windows 8 and the nvidia drivers using the cirrus drivers with my nvidia card as a secondary passthrough. That part works. The problem occurs when I start the VM with the nvidia drivers installed (the card is still passed through as secondary). It just immediately causes a BSOD with a "SYSTEM_SERVICE_EXCEPTION".

I have the ACS override patch applied and IOMMU enables (with some custom patches to the bios)

My hardware is:
Intel I7 920
Asus P6T deluxe V2 (Yes, i know it's broken. I patched the bios, and it IOMMU does enable)
Nvidia GTX 460 (Passthrough)
Nvidia GTX 660ti (Main)

Where is it claimed that GeForce cards work as secondary in the guest?

Ohh, so I'm an idiot... The good news is that it actually works. I got the long unsupported nonworking asus board to work.

Thanks.

EDIT:

I was a bit too quick. It boots fine, but the graphics performance is absolutely abysmal. GPU-Z reports that the driver is installed and loaded (forceware 347), it's also reporting the right card (Gefore 460). The problem is that it's reporting 0 MHz clock rate, and 0 MB memory. This obviously means that it's not communicating correctly with the card. What could cause that?

EDIT2: Indeed, even the control panel wont launch. I guess the driver is completely borked. (Exception code 0x40000015)

Last edited by DelusionalLogic (2015-01-07 00:37:52)

Offline

#3750 2015-01-07 09:50:43

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ansa89 wrote:
Duelist wrote:

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

Duelist wrote:

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

JohnyPea wrote:
Ansa89 wrote:

PS: what are the benefits of using virtio-blk instead of virtio-scsi?

With virtio-blk you can use x-data-plane=on, which increases IOPS in guest and lowers overhead in host. In windows guest I wasn't able to pass trim operations through virtio-blk as opposed to virtio-scsi. Later I found a workaround to this by using detect-zeroes=on with discard='unmap' and writing zeros to desired sectors (zero out unused disk space inside guest).

Can someone please share with me how I can enable virtio-blk-data-plane for the Libvirt Domain XML?

I couldn't transform these posts into my XML.

I also found some slides from redhat, but they don't work either. How does the relevant code look like in XML?

I am trying for some days now without luck.

thank you very much.

Last edited by 4kGamer (2015-01-07 10:27:35)

Offline

#3751 2015-01-07 13:06:09

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:
Ansa89 wrote:
Duelist wrote:

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

Duelist wrote:

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

JohnyPea wrote:
Ansa89 wrote:

PS: what are the benefits of using virtio-blk instead of virtio-scsi?

With virtio-blk you can use x-data-plane=on, which increases IOPS in guest and lowers overhead in host. In windows guest I wasn't able to pass trim operations through virtio-blk as opposed to virtio-scsi. Later I found a workaround to this by using detect-zeroes=on with discard='unmap' and writing zeros to desired sectors (zero out unused disk space inside guest).

Can someone please share with me how I can enable virtio-blk-data-plane for the Libvirt Domain XML?

I couldn't transform these posts into my XML.

I also found some slides from redhat, but they don't work either. How does the relevant code look like in XML?

I am trying for some days now without luck.

thank you very much.

I didn't tried before but seems like google has the answer: http://blog.vmsplice.net/2013/03/new-in … irtio.html

Regards,


TheArcher

Offline

#3752 2015-01-07 15:17:16

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you TheArcher! Always there when needed smile

I found that page too. But on that page it doesn't mention anything about detect-zeroes=on and how do I "zero out unused disk space inside guest"?

Offline

#3753 2015-01-07 16:32:49

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

and it says "Red Hat VirtIO SCSI Disk Device" in Device Manager in Windows. Is this correct for virtio-blk-data-plane?


and these are the relevant parts of my XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/sda'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </disk>

<qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.scsi=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.config-wce=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
  </qemu:commandline>

Last edited by 4kGamer (2015-01-07 17:30:16)

Offline

#3754 2015-01-07 19:35:44

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success story, it took me few days to get it right by seems pretty stable. The biggest obstacle was ZFS on Linux as root partition (still causing hiccups - obviously not related to topic at all) and few silly mistakes in network setup etc. Other issue was selection of hardware passed-thru eventually I settled for either of my two GPUs (AMD Turks ie. FirePro V4900 and Hawaii ie. 290X - both work nicely), all USB controllers (but one) including PCIe cards and keyboard (attached to only USB controller remaining to host). From the start the intent was to not to leave any graphic capability to host, and both GPU cards are meant for virtual machines. For this I simply blacklisted radeon and configured pci-stub to take the ownership of both cards.

Following Alex's advice I set to use FX440 rather than Q35, also right from the start I installed Windows on Virtio thus being spared abysmal perfomance of emulated disk. Also I do not use HDMI audio output and do not plan on using it, my audio is from DAC attached to one of passed-thru USB controllers. I had BSODs or other problems if I explicitly declared CPU type to something different than qemu64 (or host), tried SandyBridge and Nehalem. I also run almost exactly same command to start Linux guest (mint 17.1) which also runs nicely - obvious differences being -rtc base=utc, different -drive for disk backing store and virtio network card rather than e1000 (I might try virtio on Windows again actually, there may be better versions avialable).

My goal is to split my machine into 3 machines - one "visible" Windows that you will see after starting the computer, "invisible" host which provides zfs for user files shared over virtual network and another "invisible" linux machine for general hacking and experimentation. Both linux machines to be accessed through shh "as if they were over network". I might eventually split the machine to run concurrently two "visible" Windows since it has two GPUs , enough CPU power and enough RAM for that. Only a question of finding space on desk for another monitor and keyboard.

Here is my qemu command:

#!/bin/bash qemu-system-x86_64 -enable-kvm \ -M pc -m 16G -mem-path /dev/hugepages \ -smp cpus=8,cores=4,threads=2,sockets=2 \ -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -bios /usr/share/qemu/bios.bin \ -rtc base=localtime,clock=host \ -nodefaults \ -nographic -vga none \ -monitor stdio \ -boot order=c \ -device vfio-pci,host=02:00.0,addr=08.0,romfile=ROM/Hawaii-1.rom,x-vga=on \ -device vfio-pci,host=00:1b.0,addr=09.0 \ -device vfio-pci,host=00:1d.0,addr=0a.0 \ -device vfio-pci,host=00:1a.0,addr=0b.0 \ -device vfio-pci,host=81:00.0,addr=0e.0 \ -device vfio-pci,host=83:00.0,addr=0f.0 \ -usb -usbdevice host:04D9:2011 \ -drive file=/dev/zvol/zdata/image1,id=disk0,format=raw,if=virtio \ -drive file=/home/bronek/windows7.iso,id=isocd0,if=none -device ide-cd,drive=isocd0 \ -drive file=/home/bronek/virtio-win-0.1-94.iso,id=isocd1,if=none -device ide-cd,drive=isocd1\ -netdev bridge,br=br0,id=net0 -net nic,netdev=net0,macaddr=52:54:01:23:45:f5,model=e1000

My software versions are qemu-2.2.0 (from testing), linux-3.17.7 (modified linux 3.17.6 package with newer version and own config) and zfs 0.6.3-r1.2 + few patches from zfsonlinux (modified archzfs package).

I also switched from systemd-networkd to netctl because I liked its configuration better and enabled sysctl for definition of iptables overrides for bridge and for initialisation of huge pages:

root@gdansk ~ # cat /etc/sysctl.d/80-hugepages.conf # Reserve 40GB in 2MB pages for virtual machines vm.nr_hugepages = 20480; root@gdansk ~ # cat /etc/sysctl.d/81-bridge.conf # Disable ip tables on bridge frames net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0

Didnt run any games yet - this machine is actually not "for gaming" although I will certainly allow my children to play  games.

Next step is to replace qemu command with libvirt (for CPU pinning, automatic start and managment) but without GUI managment part since the host is headless and I only access it through SSH (and I like command line in Linux better than GUI tools). I also plan to setup smb sharing of zfs setup on the host, with the guest. Tempted to make it Kerberos with help of samba4 running on host but not sure, I might not have time for that.

Last part I was struggling to find qemu guest agent for Windows binaries (for VSS support) prepared by someone but even without it I can take backups of my guest, the only difference being I need to do it offline. Building qemu guest agent myself, from qemu sources proved to be bit more difficult than I expected it to be. Although I might try it again later.

Last edited by Bronek (2015-01-07 19:38:34)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3755 2015-01-07 19:45:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

Success story, it took me few days to get it right by seems pretty stable.

Thanks for the report and congrats!  Tiny tip, add -serial none -parallel none to your QEMU commandline to avoid the terminal needing a reset after guest shutdown.  I think some recent upstream changes preventing multiple things being muxed to stdio will require this for the next release anyway.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3756 2015-01-07 20:05:24

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding virtio Ethernet adapter for Windows, version virtio-win-0.1-94 didn't work for me, but version virtio-win-0.1-81 did, so I'm no longer using e1000. Virtio SCSI driver worked from both versions. I'm running Windows 7 Ultimate.

Now to that libvirt setup .... hope it wont be that hard.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3757 2015-01-07 20:09:52

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Thanks for the report and congrats!  Tiny tip, add -serial none -parallel none to your QEMU commandline to avoid the terminal needing a reset after guest shutdown.  I think some recent upstream changes preventing multiple things being muxed to stdio will require this for the next release anyway.

Thanks for the tip, I'm using -nodefaults since that seemed the only way to get rid of bogus floppy drive and also to get -monitor stdio working (wont need it after migration to libvirt, but now its very useful) but I will add the above nevertheless smile

Any other tips will be hugely appreciated smile


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3758 2015-01-08 06:18:26

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

thank you TheArcher! Always there when needed smile

I found that page too. But on that page it doesn't mention anything about detect-zeroes=on and how do I "zero out unused disk space inside guest"?

Hello sir,

According to http://qemu.11.n7.nabble.com/is-x-data- … ml#a293049, dataplane is deprecated and will be removed on QEMU 2.2.

I do not know exactly the problem you have with the performance but, in my case, with normal image (raw), my windows 8.1 boots in 2 seconds on a SSD non dedicated one. I think is enough performance for me smile

Best regards,


TheArcher

Offline

#3759 2015-01-08 07:08:01

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hello! I didn't know that. Thank you very much. This is valuable information for me. Some of us here are obviously confused with what's best as a storage device.

To tell you the truth I don't even know how to rate my performance. I guess all the Windows benchmark tools for assessing hardware cannot be regarded as reliable.

What benchmark tools do you use or can you recommend? Especially for storage (IOPS and 4k random read/write in particular)

Last edited by 4kGamer (2015-01-08 07:14:51)

Offline

#3760 2015-01-08 08:00:24

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I never use benchmark utilities because normally are synthetic ones and are just numbers that can be really different from the real performance I'm getting. Well, to be honest I run them on the beginning... you know, the normal ones, some for graphic card and some for general... but I have been too many years working with computers to know that best benchmark utilities and results does not mean that is better performance... talking about phones... look antutu results for different phones and read the feelings... No one is agree! smile

For me is just a feeling, I mean, if I can run my games and boots ok, can copy files in a reasonable time, etc... is ok. Sometimes I notice that is slower or not giving me the right performance and then I look for some information about but, talking about the QEMU/KVM... I'm so happy about how it works. Performance is really good even when I'm not checking it with benchmarks.

Cheers,


TheArcher

Offline

#3761 2015-01-08 08:12:32

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah you're definitely right! I completely agree.

I am not after certain numbers. It's only that sometimes I experience issues in Games and I wonder if it's related to poor disk performance. I therefore need to make sure that I have the right tools to test NOW so I don't have to worry LATER.

So it's only to get the configuration right. And I am happy you made me aware that I was trying a depriaciated technique. Before I also tried SATA passthrough which worked fine for most parts, but then I had issues in Games and I still don't know why ...

some tools would definitely help for now smile

Offline

#3762 2015-01-08 08:23:32

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But... what kind of issues do you have?

As I'm not expert, maybe if you put your hardware configuration, config files and so on, and describe as detailed as possible the problems... maybe a guru from here can help you smile

Cheers smile

Offline

#3763 2015-01-08 09:00:07

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. (Normally my CPU usage is about 20-30 % when I play Fifa 15 at 4k resolution)
I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

Last edited by 4kGamer (2015-01-08 09:03:03)

Offline

#3764 2015-01-08 09:04:41

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

Don't know for sure but can be several things:

- As is on playing internet... can be the ethernet? Are you passing it through or emulating?
- Can be NVIDIA drivers? (you know, loading an effect or similar)
- If it's not happening with any other game... can be the game? smile

Hope we can solve it together smile


Cheers

Offline

#3765 2015-01-08 09:08:42

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am passing through my Ethernet. And it happened offline when I played with friends. I couldn't test thoroughly with other games. I barely played on it so far. A few days back we had a Soccer session here and it happened a few times. Never before and I am familiar with the game.

So my guess: It could be Nvidia Drivers or the CPU or the disk. But maybe it's something completely different smile

I will post my complete settings in a few hours. I am very grateful for your help!

Offline

#3766 2015-01-08 09:18:51

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

I am passing through my Ethernet. And it happened offline when I played with friends. I couldn't test thoroughly with other games. I barely played on it so far. A few days back we had a Soccer session here and it happened a few times. Never before and I am familiar with the game.

So my guess: It could be Nvidia Drivers or the CPU or the disk. But maybe it's something completely different smile

I will post my complete settings in a few hours. I am very grateful for your help!

Ok smile

As you have quite similar hadrware as me... Are you passing through the intel LAN isn't it? I know that the realtek ones give me problems smile

Anyway... yes, can be but... my suggestion to troubleshoot this would be:

Try different things (different stadiums, weather conditions,... to try to find if there is any pattern)
Different NVIDIA drivers... sometimes the last version is not the best, talking by experience
I do not know how many vCPU you have assigned or if you have different setup. In my case I assign 4 because if I assign 8 the heat increase really high and I didn't notice any performance improvement.

Anyway, I will wait for your config files smile

Cheers.

Offline

#3767 2015-01-08 13:47:52

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So here is my setup and config files:

Motherboard:ASROCK Z97 Extreme6
CPU: i7-4790k
GPU: MSI GTX 980 Gaming 4G
RAM: 12 GB RAM
Storage for Windows 8.1: Samsung 840 EVO Basic 1TB (/dev/sda)
Linux Kernel: linux-3.17.6-1 (untouched)

Packages installed:
qemu 2.1
libvirt
virt-manager

cat /proc/cmdline
"rw root=/dev/mapper/VG-root rdblacklist=nouveau nouveau.modeset=0 elevator=noop vfio_iommu_type1 intel_iommu=on,igfx_off panic=10 pci-stub.ids=10de:13c0,10de:0fbb,8086:15a1,8086:8cb1 intel_pstate=disable pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"


/etc/modprobe.d/modprobe.conf:

options kvm ignore_msrs=1
options intel_kvm emulate_invalid_guest_state=0
options intel_kvm nested=1
options intel_kvm enable_shadow_vmcs=1
options intel_kvm enable_apicv=1
options intel_kvm ept=1
#options vfio_pci nointxmask=1
options vfio_iommu_type1 disable_hugepages=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/libvirt/qemu/vm1.xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit vm1
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>vm1</name>
  <uuid>2459c3f0-4269-4823-55d4-43528ff0d991</uuid>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <vcpu placement='static' cpuset='0-7'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0,4'/>
    <vcpupin vcpu='1' cpuset='1,5'/>
    <vcpupin vcpu='2' cpuset='2,6'/>
    <vcpupin vcpu='3' cpuset='3,7'/>
    <vcpupin vcpu='4' cpuset='0,4'/>
    <vcpupin vcpu='5' cpuset='1,5'/>
    <vcpupin vcpu='6' cpuset='2,6'/>
    <vcpupin vcpu='7' cpuset='3,7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/VM1-OVMF.fd</loader>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/root/virtio-win-0.1-94.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/root/windows.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='1' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Last edited by 4kGamer (2015-01-08 14:00:15)

Offline

#3768 2015-01-08 13:49:07

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If I forgot something just tell me and I'll provide it smile

oh and yes I pass through the Intel LAN

and I forgot to post this:

lsblk

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
04:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
0a:00.0 Multimedia controller: Digital Devices GmbH Octopus DVB Adapter
0b:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

the bold ones are the pci devices I passed through (successfully)

Last edited by 4kGamer (2015-01-08 13:58:10)

Offline

#3769 2015-01-08 14:12:27

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As you make CPU pin with the libvirt.... is happening the same using the command line? and not using them? I don't use any CPU pin smile

Offline

#3770 2015-01-08 14:16:33

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I haven't tried without CPU Pin, but that is something I am not really sure if I did it correctly. Also hugepages, which are disabled in my setup. I think I have to review both. And like you know I am really uncertain of the storage device. smile

Offline

#3771 2015-01-08 14:50:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

So here is my setup and config files:

Motherboard:ASROCK Z97 Extreme6
CPU: i7-4790k
GPU: MSI GTX 980 Gaming 4G
RAM: 12 GB RAM
Storage for Windows 8.1: Samsung 840 EVO Basic 1TB (/dev/sda)
Linux Kernel: linux-3.17.6-1 (untouched)

Packages installed:
qemu 2.1
libvirt
virt-manager

cat /proc/cmdline
"rw root=/dev/mapper/VG-root rdblacklist=nouveau nouveau.modeset=0 elevator=noop vfio_iommu_type1 intel_iommu=on,igfx_off panic=10 pci-stub.ids=10de:13c0,10de:0fbb,8086:15a1,8086:8cb1 intel_pstate=disable pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

If you're using an "untouched" kernel, the pcie_acs_override option isn't doing anything.  You also shouldn't need the vfio option here.

/etc/modprobe.d/modprobe.conf:

options kvm ignore_msrs=1
options intel_kvm emulate_invalid_guest_state=0
options intel_kvm nested=1
options intel_kvm enable_shadow_vmcs=1
options intel_kvm enable_apicv=1
options intel_kvm ept=1
#options vfio_pci nointxmask=1
options vfio_iommu_type1 disable_hugepages=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

Could you not find any more random option to add?  You should not need any of the vfio ones, and by disabling huge pages, you're making the IOMMU work harder.  BTW, the Intel kvm module is called kvm_intel, not intel_kvm, so all of those options are not doing anything.  I don't think you want to be touching any of those option anyway.

/etc/libvirt/qemu/vm1.xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit vm1
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>vm1</name>
  <uuid>2459c3f0-4269-4823-55d4-43528ff0d991</uuid>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <vcpu placement='static' cpuset='0-7'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0,4'/>
    <vcpupin vcpu='1' cpuset='1,5'/>
    <vcpupin vcpu='2' cpuset='2,6'/>
    <vcpupin vcpu='3' cpuset='3,7'/>
    <vcpupin vcpu='4' cpuset='0,4'/>
    <vcpupin vcpu='5' cpuset='1,5'/>
    <vcpupin vcpu='6' cpuset='2,6'/>
    <vcpupin vcpu='7' cpuset='3,7'/>

You really want to pin to a single CPU, so I'd eliminate the a,b options, pick one.  I recognize that you're trying to pin to a core, which should share cache, but I don't think this is helping you.  Also, you're assigning 100% of your CPU power to the guest.  Are you running a very minimal host?  How is the host supposed to do anything without stealing CPU from the guest?  In my setup I assign the cores to the guest and leave the threads for the host.  Huge pages are another tuning option here.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3772 2015-01-08 15:19:31

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. (Normally my CPU usage is about 20-30 % when I play Fifa 15 at 4k resolution)
I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

You really should try to pin your vcpus 1:1 to real cpus; and keep hyper-threading in mind to prevent cache-trashing (which very well could be the cause for your micro-stutters);

With a i7 920 Host CPU (8 virtual cpus; hyperthreading) i use this:

 <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='6'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-1,4-5'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader readonly='yes' type='pflash'>/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram template='/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/win_VARS.fd</nvram> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu>

Looking forward to hear if that helps!
Greetings

Last edited by kaeptnb (2015-01-08 15:21:24)

Offline

#3773 2015-01-08 15:28:11

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex, thank you very much for having a closer look. I will adjust accordingly. I didn't know many of the things you wrote. I have assigned all CPUs to guest to make sure I have enough power. It's just for now when I am still trying to find out what I really need. I will leave 25 - 50 % to the host as I am not doing anything other than using the console at the moment - but that might change in time smile


Alex, is this okay?
       

<vcpu placement='static' cpuset='0-5'>8</vcpu>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>

You said you only assign cores to the guest. How is this done within the XML?

Offline

#3774 2015-01-08 15:37:52

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:
4kGamer wrote:

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. (Normally my CPU usage is about 20-30 % when I play Fifa 15 at 4k resolution)
I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

You really should try to pin your vcpus 1:1 to real cpus; and keep hyper-threading in mind to prevent cache-trashing (which very well could be the cause for your micro-stutters);

With a i7 920 Host CPU (8 virtual cpus; hyperthreading) i use this:

 <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='6'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-1,4-5'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader readonly='yes' type='pflash'>/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram template='/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/win_VARS.fd</nvram> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu>

Looking forward to hear if that helps!
Greetings

Thank you for sharing your configs! I will definitely pin 1:1 from now on.

About hyperv: If I understand correctly I can't enable hyper-threading because I use an Nvidia GPU. It would give me Code 43 on my GTX 980.

Last edited by 4kGamer (2015-01-08 15:39:12)

Offline

#3775 2015-01-08 15:41:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Alex, thank you very much for having a closer look. I will adjust accordingly. I didn't know many of the things you wrote. I have assigned all CPUs to guest to make sure I have enough power. It's just for now when I am still trying to find out what I really need. I will leave 25 - 50 % to the host as I am not doing anything other than using the console at the moment - but that might change in time smile


Alex, is this okay?
       

<vcpu placement='static' cpuset='0-5'>8</vcpu>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>

You said you only assign cores to the guest. How is this done within the XML?

I don't think you have the right idea about what a thread is.  A thread is not a full execution unit, so your example here has vcpus 4 & 5 partially sharing vcpus 0 & 1.  I don't know how you'd represent that to the guest.  You also need to realize that a thread only adds maybe 10-20% performance, so if you assign 4 cores (without threads), you're already providing the VM with perhaps 75%+ of the processing power of the socket.  I do something more like kaeptnb, although I don't know how they came up with those cpuset values.  AFAIK, cpusets are the same as processor numbers, where the cores are generally enumerated before the threads.  Thus on a 4-core w/ threads processor, 0-3 are cores and 4-7 are threads, where 4 is the thread sibling of 0, 5 of 1, etc.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3776 2015-01-08 15:50:56

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I see. So more than 4 isn't gonna work.

I will use 4 then:

 <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> </cputune>

that looks simple. Hopefully I understood correctly.

Last edited by 4kGamer (2015-01-08 15:52:20)

Offline

#3777 2015-01-08 15:51:37

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sorry, double post

Last edited by 4kGamer (2015-01-08 15:51:55)

Offline

#3778 2015-01-08 15:57:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And change

 <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu>

to threads='1'


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3779 2015-01-08 16:01:14

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ah yes, of course. Thank you Alex. smile

Offline

#3780 2015-01-08 16:22:36

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

... although I don't know how they came up with those cpuset values.  AFAIK, cpusets are the same as processor numbers, where the cores are generally enumerated before the threads.  Thus on a 4-core w/ threads processor, 0-3 are cores and 4-7 are threads, where 4 is the thread sibling of 0, 5 of 1, etc.

this whole core/socket/thread/sibling really gave me a headache...
i found out - as you described, that 0/4 is actually my first core, 1/5 2nd and so on;
Its new to me that e.g. 4 is only "25%" of the power - relative to 0 ... but anyways it didn't matter to me because my goal was to represent my real cpu as close as possible to the guest.

hence the  threads='2' and the exact pinning as i posted (2-6-3-7) - that way i was quite sure that windows also sees that 2 of the cores are "thread" cores - and hopefully does "the right thing" regarding its internal scheduling.

benchmarks backed this: guest with vcpus pinned this way:
root@pc:/usr/local/sbin> cat vcpu-2637.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 6
virsh vcpupin windows7 --vcpu 2 3
virsh vcpupin windows7 --vcpu 3 7

was faster in "wprime" thread benchmarking than guest with vcpus pinned that way:

root@pc:/usr/local/sbin> cat vcpu-2367.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 3
virsh vcpupin windows7 --vcpu 2 6
virsh vcpupin windows7 --vcpu 3 7


so i concluded, a) windows "knows" about thread-cores b) my pinning was ok

HTH
Greetings

Last edited by kaeptnb (2015-01-08 16:28:25)

Offline

#3781 2015-01-08 17:00:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:
aw wrote:

... although I don't know how they came up with those cpuset values.  AFAIK, cpusets are the same as processor numbers, where the cores are generally enumerated before the threads.  Thus on a 4-core w/ threads processor, 0-3 are cores and 4-7 are threads, where 4 is the thread sibling of 0, 5 of 1, etc.

this whole core/socket/thread/sibling really gave me a headache...
i found out - as you described, that 0/4 is actually my first core, 1/5 2nd and so on;
Its new to me that e.g. 4 is only "25%" of the power - relative to 0 ... but anyways it didn't matter to me because my goal was to represent my real cpu as close as possible to the guest.

hence the  threads='2' and the excact pinning as i posted - that way i was quite sure that windows also sees that 2 of the cores are "thread" cores - and hopefully does "the right thing" regarding its internal scheduling.

Aha, I missed that you used threads=2, so that makes a bit more sense.  So you're actually trying to represent exactly half of your CPU to the guest.

benchmarks backed this: guest with vcpus pinned this way:
root@pc:/usr/local/sbin> cat vcpu-2637.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 6
virsh vcpupin windows7 --vcpu 2 3
virsh vcpupin windows7 --vcpu 3 7

So let's look at your actual mapping:

cpuset: core,thread
2: 2,0
6: 2,1
3: 3,0
7: 3,1

ie. core/thread/core/thread

was faster in "wprime" thread benchmarking than guest with vcpus pinned that way:

root@pc:/usr/local/sbin> cat vcpu-2367.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 3
virsh vcpupin windows7 --vcpu 2 6
virsh vcpupin windows7 --vcpu 3 7


so i concluded, a) windows "knows" about thread-cores b) my pinning was ok

Here we've got:

2: 2,0
3: 3,0
6: 2,1
7: 3,1

ie. core/core/thread/thread

I'd expect the first mode would be what the guest expects.  Were you using the same cpu topology for these tests, cores=2/threads=2?

My host system shows:

$ egrep "^processor|^core id" /proc/cpuinfo processor : 0 core id : 0 processor : 1 core id : 1 processor : 2 core id : 2 processor : 3 core id : 3 processor : 4 core id : 0 processor : 5 core id : 1 processor : 6 core id : 2 processor : 7 core id : 3

So bare metal is core/core/core/core/thread/thread/thread/thread, which I think is pretty standard.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3782 2015-01-08 17:11:28

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, both vcpu-pinnings were tried with host-passthrough cpu;
and always with "cores=2, threads=2", yes.

as i said - my goal was to represent my real cpu as close as possible.

here my host-system cpuinfo:

root@pc:~> egrep "^processor|^core id" /proc/cpuinfo processor : 0 core id : 0 processor : 1 core id : 1 processor : 2 core id : 2 processor : 3 core id : 3 processor : 4 core id : 0 processor : 5 core id : 1 processor : 6 core id : 2 processor : 7 core id : 3

all of this is only a "problem" with intel hyperthreading cpu's - afaik amd "only" ships "real" cores, no hyperthreading, right?

P.S.: now i get your core/thread vs. core/core argument ; i'll try to redo my benchmarks, but iirc (its some time ago...) 2637 was superior to 2367...
It's afaik not documented what exactly libvirt does with "threads=2" config stanza - if it picks the "threads" itself and if it "tells" the guests these are "just" threading-cores... i don't know, so i benchmarked it; but - let me do it again smile

Greetings!

Last edited by kaeptnb (2015-01-08 17:32:00)

Offline

#3783 2015-01-08 17:41:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

all of this is only a "problem" with intel hyperthreading cpu's - afaik amd "only" ships "real" cores, no hyperthreading, right?

Depends how you define "problem".  I find non-competitive performance to be a "problem".  Threading is a common way to try to make the most effective use of the pipeline.  It requires the scheduler, and in this case the user, know that not all the processors are completely independent of each other.  If you don't agree to that, disable it in your BIOS.  Back in the P4 days threads were of questionable benefit for quite a few workloads, but I think they've come a long way since then.

P.S.: now i get your core/thread vs. core/core argument ; i'll try to redo my benchmarks, but iirc (its some time ago...) 2637 was superior to 2367...
It's afaik not documented what exactly libvirt does with "threads=2" config stanza - if it picks the "threads" itself and if it "tells" the guests these are "just" threading-cores... i don't know, so i benchmarked it; but - let me do it again smile

It mirrors the QEMU args:

-smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,sockets=sockets] set the number of CPUs to 'n' [default=1] maxcpus= maximum number of total cpus, including offline CPUs for hotplug, etc cores= number of CPU cores on one socket threads= number of threads on one CPU core sockets= number of discrete sockets in the system

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3784 2015-01-08 17:47:38

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, a quick re-run of wprime with both (2367 and 2637) configs didn't reveal any visible difference ...

this either means, the cache-trashing effect is negliable - or libvirt/qemu does the "right" thing, and tells the guest that cpu #2 and #3 (with 2367) are threading-cores, and #0 and #1 are "real" cores ...

I think its the later smile

Offline

#3785 2015-01-08 18:02:59

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

are you using wprime in Windows or do you run it from host?

Offline

#3786 2015-01-08 18:05:03

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok this is bothering me, i re-did my benchmarks, this time more often for each case and under the very same conditions (guest was settled a few minutes after boot, so not too much interfered with the cpu-benchmark).
i did 6 benchmarks with wprime, 6 times the "32m" test;
here the results:

2367 setup:

1: 18,943
2: 20,157
3: 19,93
4: 19,058
5: 19,003
6: 18,95

2637 setup:

1: 18,892
2: 18,861
3: 18,943
4: 18,9
5: 19,011
6: 18,984

so - i retract my conclusion from post #3784 and say its better to tell qemu/libvirt what to do, it seems it doesn't do the right thing itself.

Last edited by kaeptnb (2015-01-08 18:09:00)

Offline

#3787 2015-01-08 18:06:39

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4k: yes within the guest; you want to know which vcpu-pinning is better - for the guest. The host doesn't care much about your kvm-cpu pinnings

Greetings

Offline

#3788 2015-01-08 18:18:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

Ok this is bothering me, i re-did my benchmarks, this time more often for each case and under the very same conditions (guest was settled a few minutes after boot, so not too much interfered with the cpu-benchmark).
i did 6 benchmarks with wprime, 6 times the "32m" test;
here the results:

2367 setup:

1: 18,943
2: 20,157
3: 19,93
4: 19,058
5: 19,003
6: 18,95

2637 setup:

1: 18,892
2: 18,861
3: 18,943
4: 18,9
5: 19,011
6: 18,984

so - i retract my conclusion from post #3784 and say its better to tell qemu/libvirt what to do, it seems it doesn't do the right thing itself.

I don't think libvirt makes any attempt to re-interpret pinning.  You tell it to pin a vCPU to one or more CPU and that's what it does.  It doesn't care if the pinning target is in the correct order compared to the topology you're describing to the guest.  So I think you've reconfirmed the expected result here, right?  If a topology with theads is described to the guest, the first half of the vCPUs enumerate the cores with one thread, the second half re-enumerate the cores with a different thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3789 2015-01-08 18:20:59

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It mirrors the QEMU args:

yes, so far so good; let me rephrase my question then: its (at least to me) unknown, what qemu does with this config stanza.

best case would be (with threads=>1), qemu searches for thread-cpu's, picks one of these as thread-cpus for the guest and TELLS the guest that this is a threading-cpu;
OTOH - we're defining the pinning ourselfs, so the rationale is i guess that if you do that, you know what you're doing, and nothing happens automatically.

well; i'll stick with my 2637 setup here, but your milage may vary.

aw wrote:

So I think you've reconfirmed the expected result here, right?  If a topology with theads is described to the guest, the first half of the vCPUs enumerate the cores with one thread, the second half re-enumerate the cores with a different thread.

yes this is what i expected;

Rule of thumb: if you have a hyperthreading CPU and want 1:1 vcpu-pinning: take care of your thread-cores  smile

Greetings!

Last edited by kaeptnb (2015-01-08 18:25:23)

Offline

#3790 2015-01-08 18:30:03

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@4kgamer: also remember to define "    <emulatorpin cpuset='0-1,4-5'/>" - it takes a lot of cpu-load (which qemu itself generates to manage your guest) away from your guest-cpus

What i also have active, but never measured the concrete impact regarding benchmarks is
" isolcpus=2,3,6,7 nohz_full=2,3,6,7 " in my kernel-cmdline;

and "CPUAffinity=0 1 4 5" in my systemd's system.conf; this and "isocpus" helps to free up cpu's 2,3,6,7 for only kvm guests;

nohz_full _should_ enable the host-kernel to lower the "Local timer interrupts" rate on these cpu's which in turn lowers the cpu's work it's doing without the guest knowing...
( look at http://www.breakage.org/2013/11/15/nohz_fullgodmode/ regarding nohz)

To free the kvm-cpus up from even more work, at boot i run this script:

#!/bin/bash # 33 = 0,1,4,5 # 11 = 0,4 # 22 = 1,5 # 01 = 0 # 02 = 1 # 10 = 4 # 20 = 5 # irq balancing doesnt work well for more than one CPUs, so just one: irqmask1=02 #for some devices other interrupts: irqmask2=20 procmask=33 #rcuomask=$procmask rcuomask=22 #Interrupts: echo $irqmask1 >/proc/irq/default_smp_affinity for irq in $(ls -1 /proc/irq/ ) ; do [ -w /proc/irq/$irq/smp_affinity ] && echo $irqmask1 >/proc/irq/$irq/smp_affinity 2>/dev/null done # some interrupt-intensive hardware: for i in $(grep -i ahci /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done for i in $(grep -i eth /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done for i in $(grep -i nouveau /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done for i in $(grep -i xhci /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done # for mdadm echo "$procmask" >/sys/devices/virtual/workqueue/writeback/cpumask echo "$procmask" >/sys/devices/virtual/workqueue/raid5wq/cpumask echo "$procmask" >/sys/bus/workqueue/devices/writeback/cpumask echo 0 > /sys/bus/workqueue/devices/writeback/numa # adjusts rcuo kernel threads - see : https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt for pid in $(pgrep rcuo) ; do taskset -p $rcuomask $pid >/dev/null done


Greetings!

Last edited by kaeptnb (2015-01-08 19:02:17)

Offline

#3791 2015-01-08 18:44:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:
aw wrote:

It mirrors the QEMU args:

yes, so far so good; let me rephrase my question then: its (at least to me) unknown, what qemu does with this config stanza.

best case would be (with threads=>1), qemu searches for thread-cpu's, picks one of these as thread-cpus for the guest and TELLS the guest that this is a threading-cpu;
OTOH - we're defining the pinning ourselfs, so the rationale is i guess that if you do that, you know what you're doing, and nothing happens automatically.

First, let's clear something up, what we're calling a "core" here is really just the first thread on the core.  If the other thread has no work, the first thread should get roughly the full core.  If the second thread has work and the first thread doesn't, the second thread should get roughly the whole core.  It's only when both have work to do that we have an interaction.  It's that interaction at the hardware level that we're trying to describe to the guest.  As far as QEMU/KVM is concerned these are all just configuration bits for the guest that don't have any substantive effect on the vCPU.  A vCPU is just a thread and it's up to the host scheduler where to run that thread.  By pinning the vCPUs, we know we're going to have an interaction at the hardware level that the guest needs to be aware of, otherwise it will assume each vCPU is completely independent and make poor scheduling decisions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3792 2015-01-08 18:50:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

@4kgamer: also remember to define "    <emulatorpin cpuset='0-1,4-5'/>" - it takes a lot of cpu-load (which qemu itself generates to manage your guest) away from your guest-cpus

FWIW, you're the only one I've seen using this option.  It makes sense, 4k would want to use 4-7, but I'm not sure how beneficial it is.

What i also have active, but never measured the concrete impact regarding benchmarks is
" isolcpus=2,3,6,7 nohz_full=2,3,6,7 " in my kernel-cmdline;

This _should_ enable the host-kernel to lower the "Local timer interrupts" rate on these cpu's which in turn lowers the cpu's work it's doing without the guest knowing...
( look at http://www.breakage.org/2013/11/15/nohz_fullgodmode/ regarding nohz)

Please be careful with isolcpus.  This option excludes the listed CPUs from the general scheduler, so they won't be available to normal host processes without manual scheduling, regardless of whether a guest is currently running.  If you're running the guest full time and want to reduce the host impact on it, this is a good option to use.

EDIT: correction, 4-7

Last edited by aw (2015-01-08 19:13:50)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3793 2015-01-08 18:56:22

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

right, all these hints are coming from someone who's kvm-guest is running 24/7 - as soon as the host boots up;

OTOH i am able to use all CPUs within the host - i just created a cpuset:

cgcreate -t kaeptnb -g cpuset:/allcpus cgset -r cpuset.cpus=0-7 allcpus

and when i want to run a task on the host which needs full cpu i just do e.g.

cgexec -g cpuset:/allcpus mpv 4kmovie.mkv

Last edited by kaeptnb (2015-01-08 19:00:10)

Offline

#3794 2015-01-08 19:12:23

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you guys, I will try <emulatorpin cpuset='0-1,4-5'/>" and compare. Really grateful for this thread. Everyone that's helping: THANK YOU. I wouldn't be able to run the VM without you.

Offline

#3795 2015-01-08 19:27:00

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

FWIW, you're the only one I've seen using this option.  It makes sense, 4k would want to use 4-7, but I'm not sure how beneficial it is.


I just took a look:

ps -Leo cmd,pid,lwp,cputime,psr 

shows for the qemu processes:

qemu-system-x86_64 -enable- 23109 23109 00:01:23   5
qemu-system-x86_64 -enable- 23109 23112 00:06:22   2
qemu-system-x86_64 -enable- 23109 23113 00:05:31   6
qemu-system-x86_64 -enable- 23109 23114 00:05:15   3
qemu-system-x86_64 -enable- 23109 23115 00:06:52   7
qemu-system-x86_64 -enable- 23109 23122 00:00:06   2
qemu-system-x86_64 -enable- 23109 23123 00:00:01   2
qemu-system-x86_64 -enable- 23109 23124 00:00:24   3
qemu-system-x86_64 -enable- 23109 23125 00:00:00   7
qemu-system-x86_64 -enable- 23109 23126 00:00:00   3

so - only one thread is outside the vcpu-range, the first one;


the first and the last 5 threads seem to be the emulator-threads,
i expected more - all but 4 vcpu-threads to be exact - to be on cpu 0,1,4 or 5.

So you may be right, it's not that much helpful - but OTOH its not bad, at least one thread is "offloaded" from the guest-cpus;

Last edited by kaeptnb (2015-01-08 19:35:39)

Offline

#3796 2015-01-08 19:34:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

So you may be right, it's not that much helpful - but OTOH its not bad, at least one thread is "offloaded" from the guest-cpus;

Theoretically, if you're using isolcpu, 2,3,6,7 shouldn't be available for general scheduling for the emulator anyway.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3797 2015-01-08 19:36:42

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well i'm surprised as well and can't explain why only one of the 6 emulator-threads are on non-guest cpus... strange

Offline

#3798 2015-01-08 19:51:10

kaeptnb
Member
Registered: 2013-09-04
Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, i set the emulatorpin to just one cpu (    <emulatorpin cpuset='1'/> ), restarted the guest, ran the "1024m" wprime test (just to see which threads are the vcpu threads and which are the emulator threads)

watch "ps -Leo cmd,pid,lwp,cputime,psr | grep qemu" 

reveals, that the emulator-threads are NOT pinned - they change their cpu from time to time.
And i still see changing them to cpu's which should be "protected" by isol_cpu... strange

Last edited by kaeptnb (2015-01-08 19:53:44)

Offline

#3799 2015-01-08 20:26:27

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After conversion to libvirt I have this:

<domain type='kvm'> <name>windows1</name> <uuid>de644f73-fd03-4e6a-81dd-eb3e8cdd5305</uuid> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <hugepages> <page size='2048' unit='KiB'/> </hugepages> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8'/> <vcpupin vcpu='1' cpuset='9'/> <vcpupin vcpu='2' cpuset='10'/> <vcpupin vcpu='3' cpuset='11'/> <vcpupin vcpu='4' cpuset='20'/> <vcpupin vcpu='5' cpuset='21'/> <vcpupin vcpu='6' cpuset='22'/> <vcpupin vcpu='7' cpuset='23'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader type='rom'>/usr/share/qemu/bios.bin</loader> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='4096'/> </hyperv> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Penryn</model> <topology sockets='1' cores='4' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64.xvga</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/zvol/zdata/windows1'/> <target dev='vda' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/bronek/virtio-win-0.1-81.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='0' model='pci-root'/> <interface type='bridge'> <mac address='52:54:01:34:56:f7'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04d9'/> <product id='0x2011'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x83' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x82' slot='0x00' function='0x0'/> </source> <rom file='/home/bronek/Turks.vga.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

Note "fancy" <emulator>/usr/bin/qemu-system-x86_64.xvga</emulator>, which is:

root@gdansk ~ # cat /usr/bin/qemu-system-x86_64.xvga #!/bin/sh exec /usr/bin/qemu-system-x86_64 `echo "$@" | sed 's/vga.rom/vga.rom,x-vga=on/g'`

It works briliantly,  thanks aw for your hard work on this feature! Best of all, since it's virsh and I've setup core isolation on my kernel, I now know that this virtual machine is guaranteed full access to selected PCI, CPU and RAM resources - and it starts automatically when I switch on my (metal) machine - it's really fast smile

Small nigglees: there is now long list of devices under "safely remove" icon including all my passed thru PCI devices and Virtio ones as well. I hope I can find a way to somehow convince Windows that these should not be listed. Also, I had to "downgrade" VCPU from host, SandyBridge and Nehalem to Penryn (another stable choice was qemu64)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3800 2015-01-08 20:36:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...long story short...
I've finally flashed my cards with UEFI-compatible "secret"(i can't get it anywhere anymore) firmware, but appears like windows7 doesn't really works right with it. I turn off CSM, boot up the windows, it shows the splash logo and then image freezes. Yay for windows7. So, OVMF/UEFI and windows 7 is possible but troublesome, so most folks shouldn't bother about it.

P.S. if anyone needs UEFI-combatible VBIOS file for ASUS HD7750(silent, 800mhz one, but i think it's possible to edit it for any card) - send me an e-mail, because i seem to be unable to get it officially from support anymore.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3801 2015-01-09 10:55:10

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to set up a OVMF VM so I can also use the Intel iGPU for a Linux workstation, but I seem to get no video output at all. I know it's booting from my test Linux ISO (the serial output shows something).

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly,file=/usr/share/qemu/OVMF-pure-efi.fd -vga none -device vfio-pci,host=08:00.0 -device vfio-pci,host=08:00.1 -drive file=CentOS-7.0-1406-x86_64-DVD.iso,id=virtiocd

No video, zilch. Am I missing something obvious?

In dmesg, I see:

[ 1755.997185] vfio-pci 0000:08:00.0: enabling device (0400 -> 0403) [ 1756.019017] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x270 [ 1756.021067] vfio-pci 0000:08:00.1: enabling device (0400 -> 0402) [ 1757.395393] kvm: zapping shadow pages for mmio generation wraparound

Which I assume is good?

EDIT: And of course, tell me what I can do to debug/what more information if any is needed.

Last edited by MacCoaster (2015-01-09 11:01:10)

Offline

#3802 2015-01-09 13:47:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:

I'm trying to set up a OVMF VM so I can also use the Intel iGPU for a Linux workstation, but I seem to get no video output at all. I know it's booting from my test Linux ISO (the serial output shows something).

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly,file=/usr/share/qemu/OVMF-pure-efi.fd -vga none -device vfio-pci,host=08:00.0 -device vfio-pci,host=08:00.1 -drive file=CentOS-7.0-1406-x86_64-DVD.iso,id=virtiocd

No video, zilch. Am I missing something obvious?

In dmesg, I see:

[ 1755.997185] vfio-pci 0000:08:00.0: enabling device (0400 -> 0403) [ 1756.019017] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x270 [ 1756.021067] vfio-pci 0000:08:00.1: enabling device (0400 -> 0402) [ 1757.395393] kvm: zapping shadow pages for mmio generation wraparound

Which I assume is good?

EDIT: And of course, tell me what I can do to debug/what more information if any is needed.

The graphics card needs to support UEFI.  Does it?

http://vfio.blogspot.com/2014/08/does-m … t-efi.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3803 2015-01-09 14:46:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:

I'm trying to set up a OVMF VM so I can also use the Intel iGPU for a Linux workstation, but I seem to get no video output at all. I know it's booting from my test Linux ISO (the serial output shows something).

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly,file=/usr/share/qemu/OVMF-pure-efi.fd -vga none -device vfio-pci,host=08:00.0 -device vfio-pci,host=08:00.1 -drive file=CentOS-7.0-1406-x86_64-DVD.iso,id=virtiocd

No video, zilch. Am I missing something obvious?

In dmesg, I see:

[ 1755.997185] vfio-pci 0000:08:00.0: enabling device (0400 -> 0403) [ 1756.019017] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x270 [ 1756.021067] vfio-pci 0000:08:00.1: enabling device (0400 -> 0402) [ 1757.395393] kvm: zapping shadow pages for mmio generation wraparound

Which I assume is good?

EDIT: And of course, tell me what I can do to debug/what more information if any is needed.

1. Tell us about your hardware - what GPU are you trying to passthrough.
2. Are you sure that the guest OS(CentOS) does support pure EFI and works with GOP and stuff? I'm sorry, i'm a bit unfamiliar with CentOS(while using fedora, heh) on UEFI.
3. Try booting with CSM on - maybe it'll get you video output, but there will be no gains - it will use VGA.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3804 2015-01-09 15:01:54

longyny
Member
Registered: 2015-01-09
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I need some assistance, please. My rig has an Asus P9X79E-WS with Xeon E5-2670, 32gb of ram and 4x AMD R9 280x. What I want to do is to passthrough 2 GPU to the guest OS, but I don't know how to blacklist only 2 of them without blacklisting all 4 GPUs. I want to leave 2 GPUs for the Host OS  with its own HDMI Audio Device, and the other pair for the Guest OS with its HDMI Audio Device.

I don't want to go the ESXi way because I want to have direct access by Linux without any virtualization layer to a pool of 28 hdds that I have attached to an HP SAS Expander Card

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]

Offline

#3805 2015-01-09 15:59:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

longyny wrote:

Hi,

I need some assistance, please. My rig has an Asus P9X79E-WS with Xeon E5-2670, 32gb of ram and 4x AMD R9 280x. What I want to do is to passthrough 2 GPU to the guest OS, but I don't know how to blacklist only 2 of them without blacklisting all 4 GPUs. I want to leave 2 GPUs for the Host OS  with its own HDMI Audio Device, and the other pair for the Guest OS with its HDMI Audio Device.

Somewhere in this thread, within the last couple weeks, is an example of someone using Xen's pci-back driver (not Xen itself) in place of pci-stub because it has the ability to claim based on bus address.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3806 2015-01-09 18:00:44

MacCoaster
Member
Registered: 2014-01-06
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The graphics card needs to support UEFI.  Does it?

http://vfio.blogspot.com/2014/08/does-m … t-efi.html

Hi aw, thanks for the quick response.

[root@localhost 0000:08:00.0]# cat rom > /tmp/image.rom cat: rom: Input/output error [root@localhost 0000:08:00.0]# (snip) [root@localhost 0000:08:00.0]# dmesg (snip) [ 117.891895] pci-stub 0000:08:00.0: Invalid ROM contents [root@localhost 0000:08:00.0]#

Can't seem to extract the ROM. Any other way? Maybe I have to do GPU-Z?

Duelist wrote:

1. Tell us about your hardware - what GPU are you trying to passthrough.

ASUS PD9-WS, with Radeon R7 240 in the last PCI-E port (PCH).

08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240] (prog-if 00 [VGA controller]) Subsystem: PC Partner Limited / Sapphire Technology Device e263 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 11 Region 0: Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 2: Memory at f0000000 (64-bit, non-prefetchable) [disabled] [size=256K] Region 4: I/O ports at a000 [disabled] [size=256] Expansion ROM at f0040000 [disabled by cmd] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [270 v1] #19 Kernel driver in use: pci-stub Kernel modules: radeon
Duelist wrote:

2. Are you sure that the guest OS(CentOS) does support pure EFI and works with GOP and stuff? I'm sorry, i'm a bit unfamiliar with CentOS(while using fedora, heh) on UEFI.

Fair question -- but wouldn't I also get EFI BIOS output on the GPU too? I tried to use the Windows 8.1 ISO but it says boot failed (in the serial console) from a CD/DVD. :\

Duelist wrote:

3. Try booting with CSM on - maybe it'll get you video output, but there will be no gains - it will use VGA.

I see several options for that in the BIOS. Right now it's set to [Enabled].

Offline

#3807 2015-01-09 18:29:28

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I got new hardware in hands, a 5820k with a Asrock X99 Extreme 6, I can confirm that GT430 in second slot works but when I try to pass the GTX970 at first slot I get

vfio-pci 0000:02:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]

in dmesg and similar warning in QEMU console warning about performance, the gpu resets but then stays black, any suggestion?

Offline

#3808 2015-01-09 20:41:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Hello, I got new hardware in hands, a 5820k with a Asrock X99 Extreme 6, I can confirm that GT430 in second slot works but when I try to pass the GTX970 at first slot I get

vfio-pci 0000:02:00.0: BAR 3: can't reserve [mem 0xe0000000-0xe1ffffff 64bit pref]

in dmesg and similar warning in QEMU console warning about performance, the gpu resets but then stays black, any suggestion?

Check if the GPU is used by some splash-screens. Do cat /proc/iomem and search the given memory range. I bet there is something like vesafb.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3809 2015-01-09 20:42:34

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MacCoaster wrote:
Duelist wrote:

3. Try booting with CSM on - maybe it'll get you video output, but there will be no gains - it will use VGA.

I see several options for that in the BIOS. Right now it's set to [Enabled].

By saying "booting with CSM on" i've meant booting the VM with OVMF-with-csm image. My OVMF would always output on the serial instead of any graphics given, QXL, STD or passed through GPU.

About the rom... My suggestion would be placing the GPU in the first pci-e slot, booting with it and extracting the ROM. I can't extract ROM from my second card resting in pci-e x8(02:00.0) slot.

Last edited by Duelist (2015-01-09 20:51:11)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3810 2015-01-09 20:49:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
longyny wrote:

Hi,

I need some assistance, please. My rig has an Asus P9X79E-WS with Xeon E5-2670, 32gb of ram and 4x AMD R9 280x. What I want to do is to passthrough 2 GPU to the guest OS, but I don't know how to blacklist only 2 of them without blacklisting all 4 GPUs. I want to leave 2 GPUs for the Host OS  with its own HDMI Audio Device, and the other pair for the Guest OS with its HDMI Audio Device.

Somewhere in this thread, within the last couple weeks, is an example of someone using Xen's pci-back driver (not Xen itself) in place of pci-stub because it has the ability to claim based on bus address.

https://bbs.archlinux.org/viewtopic.php … 9#p1486869

JohnyPea wrote:
BOOT_IMAGE=/boot/vmlinuz-3.18.1-custom root=UUID=7ca2c85e-f29e-404d-91c7-92ad1e087465 ro splash ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 iommu=pt iommu=1 xen-pciback.hide=(06:00.0)(06:00.1) default_hugepagesz=1G hugepagesz=1G panic=10 hugepages=6 isolcpus=2-7 vt.handoff=7

xen-pciback.hide=(06:00.0)(06:00.1)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3811 2015-01-10 03:14:48

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Check if the GPU is used by some splash-screens. Do cat /proc/iomem and search the given memory range. I bet there is something like vesafb.

Wow thanks, really, I was quite lost and comtemplating posiblity of changing the pc case because the 970 doesn't fit in the second slot
The culprit was "efifb" even it was not showing in lsmod, just recompiling the kernel without vesafb,efifb... did the trick

EDIT:
Seems that trying to boot host with CSM/legacy vbios causes Invalid ROM error, but booting host without CSM/Using UEFI vbios just works fine, interesting... (guest booting in legacy/uefi mode has the same outcome, seems that host booting mode matters)

Last edited by Cubex (2015-01-10 03:28:19)

Offline

#3812 2015-01-10 11:10:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Seems that trying to boot host with CSM/legacy vbios causes Invalid ROM error, but booting host without CSM/Using UEFI vbios just works fine, interesting... (guest booting in legacy/uefi mode has the same outcome, seems that host booting mode matters)

Turn on CSM on host, try booting VM without CSM(pure-efi). What happens? It is obvious that if you turn off CSM on host, it won't work in VM while not using emulated graphics that have VGA(qxl-vga for example).
Turn on CSM on host, try booting VM with csm(with-csm). What happens? Invalid ROM contents error? Try passing the ROM via romfile= option to the vfio-pci device. You can get the ROM file on techpowerup vbios database(i personally don't like that approach, since my asus cards had serial number inside that can be omitted harmlessly, but you can't know for sure if it's all the same) or plugging the card into the first PCI-E slot(making it your ONLY host GPU) and dumping the ROM via sysfs as told in aw's blog post. The NASTY way of getting ROM out of the card is using GPU-Z or ATIWinFlash(or just ATIFlash) on windows, but it works with cards not-in-the-first-slot.
Also, it may turn out that your GPU(R7 240, yes?my HD7750 didn't originally, and they are rebadged R7 250E) doesn't support UEFI at all. In that case you can check your vendor's site(like asus.com) for bios updates - i did so and updated my cards' VBIOS to make them support UEFI and GOP. Though there may be problems with some vendors(again, like asus) having a broken update tool.

Last edited by Duelist (2015-01-10 11:11:28)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3813 2015-01-10 14:11:50

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Booting host with UEFI vbios or disabling CSM its the only way it works, the guest can be run using bios or ovmf
If I boot host with legacy vbios there is no way running the guest, yesterday tried with romfile too but nothing happens, anyways csm on host is not very relevant to me

Offline

#3814 2015-01-10 14:42:52

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Booting host with UEFI vbios or disabling CSM its the only way it works, the guest can be run using bios or ovmf
If I boot host with legacy vbios there is no way running the guest, yesterday tried with romfile too but nothing happens, anyways csm on host is not very relevant to me

Oh my! I think i've mistaken you with the other guy.

Forget all that i've said there
https://bbs.archlinux.org/viewtopic.php … 9#p1491799
Sorry.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3815 2015-01-10 22:37:03

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:
Duelist wrote:

Check if the GPU is used by some splash-screens. Do cat /proc/iomem and search the given memory range. I bet there is something like vesafb.

Wow thanks, really, I was quite lost and comtemplating posiblity of changing the pc case because the 970 doesn't fit in the second slot
The culprit was "efifb" even it was not showing in lsmod, just recompiling the kernel without vesafb,efifb... did the trick

EDIT:
Seems that trying to boot host with CSM/legacy vbios causes Invalid ROM error, but booting host without CSM/Using UEFI vbios just works fine, interesting... (guest booting in legacy/uefi mode has the same outcome, seems that host booting mode matters)

That sounds a lot like the issue I had on my SuperMicro board (X10SAT).
My board has BIOS options for choosing primary dispaly device, and for what mode to use when loading the ROM from a given PCIe slot.

If I have the Intel card set as always-enabled and primary, I must also make sure the PCIe slot is set to "legacy" or "disabled" ROM loading.
Otherwise, even though it's not the primary device, the Radeon card will hijack the boot display.  If I remember correctly, the Radeon became fb0 and the Intel became fb1.
Worse than that, though, is the fact that the device's framebuffer base address gets screwed up.  This breaks everything, including the native-boot Windows Catalyst driver!

You may still be able to EFI boot the host with framebuffer if your BIOS lets you disable loading the discrete card's ROM.

Last edited by DanaGoyette (2015-01-10 22:37:20)

Offline

#3816 2015-01-11 04:21:57

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The problem is that my CPU doesn't have any IGPU so using a secondary 430 for host does solve, the good part about this new system is that I don't need to apply the override ACS patch, iommu grouping works fine out of box.

Seems that booting guest with seabios causes a host crash after ~8-10 secs, but using OVMF seems to work fine.

Offline

#3817 2015-01-11 07:52:13

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

...long story short...
I've finally flashed my cards with UEFI-compatible "secret"(i can't get it anywhere anymore) firmware, but appears like windows7 doesn't really works right with it. I turn off CSM, boot up the windows, it shows the splash logo and then image freezes. Yay for windows7. So, OVMF/UEFI and windows 7 is possible but troublesome, so most folks shouldn't bother about it.

Have you had any further success with UEFI Windows 7? I'm stuck at this flash screen as well, unfortunately.

No errors or messages, just a pulsing "Starting Windows" logo indefinitely on both the installation disk and a boot disk.

Offline

#3818 2015-01-11 13:58:20

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

need a little help here:

PC:

Processor: Amd FX-8320
Motherboard: M597 Pro
8 GB ram
Video1: Asus Radeon R9 270
Video2: Sapphire Radeon HD 6850

System:

arch linux, linux-uksm-ck 3.18.2

For example, I want to passthrough video2

i have in my grub:

GRUB_CMDLINE_LINUX_DEFAULT="root=/dev/mapper/lvmVolume-lvmRoot discard rw iommu=pt iommu=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 amd_iommu_dump=1 pci-stub.ids=1002:6739,1002:aa88,1002:43a0"

because:

lspci

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739]
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88]

and

walkindude@Desktop> ls /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/                                        ~
0000:00:15.0  0000:03:00.0  0000:03:00.1
walkindude@Desktop>

so we need also

00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]

on start pci-stub gives me:

[    2.983932] pci-stub: add 1002:6739 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.983954] pci-stub 0000:03:00.0: claimed by stub
[    2.983960] pci-stub: add 1002:AA88 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.983969] pci-stub 0000:03:00.1: claimed by stub
[    2.983973] pci-stub: add 1002:43A0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

my windows.xml (windows 10 9888):

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows</name>
  <uuid>c8bc2373-6271-4810-991e-8618a0ee01d2</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/usr/share/ovmf/ovmf_x64.bin</loader>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/virtio-win-0.1-94.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/brasero.iso'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='ide' index='1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0cf3'/>
        <product id='0x9271'/>
      </source>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <qemu:commandline>
    <qemu:arg value='-vga'/>
    <qemu:arg value='none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=03:00.0,x-vga=on'/>
    <qemu:arg value='-runas'/>
    <qemu:arg value='root'/>
  </qemu:commandline>
</domain>

after

sudo vfio-bind 0000:03:00.0 0000:03:00.1 0000:00:15.0

and

sudo virsh start windows

i get this:

error: Failed to start domain windows
error: internal error: process exited while connecting to monitor: 2015-01-11T13:57:15.773249Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12

2015-01-11T13:57:15.773322Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f4e40656020, 0x0, 0xa0000, 0x7f4d58000000) = -12 (Cannot allocate memory)
2015-01-11T13:57:15.773402Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12

2015-01-11T13:57:15.773429Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f4e40656020, 0xc0000, 0x20000, 0x7f4e23400000) = -12 (Cannot allocate memory)
2015-01-11T13:57:15.773481Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12

2015-01-11T13:57:15.773505Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f4e40656020, 0xe0000, 0x20000, 0x7f4e23600000) = -12 (Cannot allocate memory)
2015-01-11T13:57:15.773541Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12

and in dmesg:

walkindude@Desktop> sudo dmesg | tail                                                                                ~
[ 2727.433870] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2727.433929] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2727.433932] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570647] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570655] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570807] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570810] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570880] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570945] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 2733.570947] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
walkindude@Desktop>

Last edited by walkindude (2015-01-11 14:25:41)

Offline

#3819 2015-01-11 14:13:09

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe I sould have said this earlier, AFAIK Windows 7 boots in UEFI mode, however doesn't support UEFI vbios. I remenber getting a boot error on bare metal when disabling CSM even If i was booting using UEFI mode.
If you can't use seabios or ovmf with csm (like me), use Win8+  hmm

Offline

#3820 2015-01-11 14:53:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

need a little help here:

PC:

Processor: Amd FX-8320 Motherboard: M597 Pro 8 GB ram Video1: Asus Radeon R9 270 Video2: Sapphire Radeon HD 6850

System:

arch linux, linux-uksm-ck 3.18.2

For example, I want to passthrough video2

i have in my grub:

GRUB_CMDLINE_LINUX_DEFAULT="root=/dev/mapper/lvmVolume-lvmRoot discard rw iommu=pt iommu=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 amd_iommu_dump=1 pci-stub.ids=1002:6739,1002:aa88,1002:43a0"

because:

lspci

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88]

and

walkindude@Desktop> ls /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices/ ~ 0000:00:15.0 0000:03:00.0 0000:03:00.1 walkindude@Desktop> 

so we need also

00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]

No, the root ports never get bound to vfio-pci, only endpoints.  There's a patch that just went upstream, and tagged for stable, that will prevent binding root ports, bridges, and switches to vfio-pci.

on start pci-stub gives me:

[ 2.983932] pci-stub: add 1002:6739 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.983954] pci-stub 0000:03:00.0: claimed by stub [ 2.983960] pci-stub: add 1002:AA88 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.983969] pci-stub 0000:03:00.1: claimed by stub [ 2.983973] pci-stub: add 1002:43A0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

my windows.xml (windows 10 9888):

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>windows</name> <uuid>c8bc2373-6271-4810-991e-8618a0ee01d2</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader type='pflash'>/usr/share/ovmf/ovmf_x64.bin</loader> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/walkindude/virtio-win-0.1-94.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/walkindude/brasero.iso'/> <target dev='hdd' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vm1.qcow2'/> <target dev='vda' bus='virtio'/> <boot order='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='ide' index='1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='scsi' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </controller> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0cf3'/> <product id='0x9271'/> </source> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0,x-vga=on'/> <qemu:arg value='-runas'/> <qemu:arg value='root'/> </qemu:commandline> </domain>

Well, you got 440fx and ovmf right, but all of your <qemu:arg> stuff needs to go away.  -vga none doesn't get specified here, you do that in xml by deleting the <video> and <graphics> entries.  You don't need x-vga=on because you're using ovmf, and trying to specify the -runas option here is pointless and futile.  Use a proper <hostdev> entry with managed="yes" to assign the GPU.

after

 sudo vfio-bind 0000:03:00.0 0000:03:00.1 0000:00:15.0

This becomes unnecessary when you properly define the device in the XML.

and

sudo virsh start windows

i get this:

error: Failed to start domain windows error: internal error: process exited while connecting to monitor: 2015-01-11T13:57:15.773249Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12 2015-01-11T13:57:15.773322Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f4e40656020, 0x0, 0xa0000, 0x7f4d58000000) = -12 (Cannot allocate memory) 2015-01-11T13:57:15.773402Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12 2015-01-11T13:57:15.773429Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f4e40656020, 0xc0000, 0x20000, 0x7f4e23400000) = -12 (Cannot allocate memory) 2015-01-11T13:57:15.773481Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12 2015-01-11T13:57:15.773505Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f4e40656020, 0xe0000, 0x20000, 0x7f4e23600000) = -12 (Cannot allocate memory) 2015-01-11T13:57:15.773541Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: VFIO_MAP_DMA: -12

and in dmesg:

walkindude@Desktop> sudo dmesg | tail ~ [ 2727.433870] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2727.433929] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2727.433932] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570647] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570655] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570807] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570810] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570880] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570945] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded [ 2733.570947] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded walkindude@Desktop>

Both of these are because you're hiding the define from libvirt in <qemu:arg> options.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3821 2015-01-11 15:37:00

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
ok, thanks

so, i deleted video and graphics tags, and add this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>

for

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739]

Is it right?

Now vm boots, i see spikes in virt-manager. I've already installed windows, everything works in there, except catalyst always gives me VIDEO_TDR_FAILURE, that's why I tried to solve with qemu:args.

Stupid question: now, as I haven't any graphics in xml config, how do I see whats going on wm?
Tried to connect hdmi cable from gpu to my second monitor (it's a tv really, multiple hdmi outs) - nothing happens.
Previously I did it with the help of virt-manager, through VNC.

Last edited by walkindude (2015-01-11 15:37:27)

Offline

#3822 2015-01-11 15:43:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@aw
ok, thanks

so, i deleted video and graphics tags, and add this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>

for

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739]

Is it right?

Now vm boots, i see spikes in virt-manager. I've already installed windows, everything works in there, except catalyst always gives me VIDEO_TDR_FAILURE, that's why I tried to solve with qemu:args.

Stupid question: now, as I haven't any graphics in xml config, how do I see whats going on wm?
Tried to connect hdmi cable from gpu to my second monitor (it's a tv really, multiple hdmi outs) - nothing happens.
Previously I did it with the help of virt-manager, through VNC.

<qemu:args> is almost never the right answer.  Have you checked whether your HD 6850 actually supports UEFI in the ROM?  You should be getting OVMF splash screen as well as the Windows loading animation on the connected screen.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3823 2015-01-11 16:06:55

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
When I was using graphics VNC, I indeed see OVMF splash screen (TianoCore) and after it Windows logo, than system boots as expected.
You had said that I need to delete <graphics> from xml, which I did. So I plug hdmi cable from 6850 to hdmi-out of tv-screen. It's also connect by another hdmi to first card, R9 and I have properly configured dual screen.
When I start the VM I tried to switch to another out of my tv, but it says only connecting... and then logo of LG appears - means nothing here.

I've tried this videocard on the same hardware with the same windows UEFI - catalyst works. Also, before I bought this R9 I was running arch on UEFI with this HD 6850 - no problems.

Offline

#3824 2015-01-11 16:14:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@aw
When I was using graphics VNC, I indeed see OVMF splash screen (TianoCore) and after it Windows logo, than system boots as expected.

This means the emulated graphics support UEFI and proves nothing about your 6850.

You had said that I need to delete <graphics> from xml, which I did. So I plug hdmi cable from 6850 to hdmi-out of tv-screen. It's also connect by another hdmi to first card, R9 and I have properly configured dual screen.
When I start the VM I tried to switch to another out of my tv, but it says only connecting... and then logo of LG appears - means nothing here.

Not sounding good for UEFI support.

I've tried this videocard on the same hardware with the same windows UEFI - catalyst works. Also, before I bought this R9 I was running arch on UEFI with this HD 6850 - no problems.

Bare-meta UEFI systems almost always support the Compatibility Support Module (CSM), which allows support for legacy x86 option ROMs and VGA.  So, this doesn't really prove anything either.  To avoid VGA and all the issues that come with it, we explicitly do not want a CSM for our VM, which is why we use the pure-EFI OVMF image.  There's a post on my blog (link below) to determine whether the card supports UEFI.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3825 2015-01-11 16:28:53

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Thanks again for explanations. Now it gives me, when the driver for this card is pci-stub (lspci -kn)

cat rom > /tmp/image.rom                                               /sys/bus/pci/devices/0000:03:00.0
cat: rom: Input/output error

Maybe I just try with the latest bios on http://www.techpowerup.com/ ...

walkindude@Desktop> ./rom-parser ~/Sapphire.HD6850.1024.110620.rom                                        ~/rom-parser
Valid ROM signature found @0h, PCIR offset 1f4h
    PCIR: type 0, vendor: 1002, device: 6739, class: 030000
    PCIR: revision 0, vendor revision: d0c
    Last image

latest from this site.. meanging no UEFI with this card?

and if I disable OVMF, and boot with seabios, should I see anything on screen which is connected to hd 6850 by hdmi? For now, it's nothing there.

Last edited by walkindude (2015-01-11 17:52:03)

Offline

#3826 2015-01-11 18:02:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@aw
Thanks again for explanations. Now it gives me, when the driver for this card is pci-stub (lspci -kn)

cat rom > /tmp/image.rom                                               /sys/bus/pci/devices/0000:03:00.0
cat: rom: Input/output error

Maybe I just try with the latest bios on http://www.techpowerup.com/ ...

walkindude@Desktop> ./rom-parser ~/Sapphire.HD6850.1024.110620.rom                                        ~/rom-parser
Valid ROM signature found @0h, PCIR offset 1f4h
    PCIR: type 0, vendor: 1002, device: 6739, class: 030000
    PCIR: revision 0, vendor revision: d0c
    Last image

latest from this site.. meanging no UEFI with this card?

and if I disable OVMF, and boot with seabios, should I see anything on screen which is connected to hd 6850 by hdmi? For now, it's nothing there.

Yep, no UEFI support in that ROM.  If you use seabios then you need x-vga=on, but search back a couple weeks and find the wrapper script method of adding it, don't use the <qemu:arg> abomination.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3827 2015-01-11 19:58:08

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

tried with

<domain type='kvm'>
  <name>windows</name>
  <uuid>c8bc2373-6271-4810-991e-8618a0ee01d2</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-wrapper</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/virtio-win-0.1-94.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/brasero.iso'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='ide' index='1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0cf3'/>
        <product id='0x9271'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
</domain>

modified wrapper to

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|03:00.0|03:00.0,x-vga=on|g'`

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850]

start the VM, switch to hmdi which connected to HD 6850 - no connection

Offline

#3828 2015-01-11 20:04:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

start the VM, switch to hmdi which connected to HD 6850 - no connection

Do you get the "Invalid ROM contents" message in dmesg when starting the VM?  If so, then you need to pass a rom with the <rom file="$FILE"/> option (double check that syntax)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3829 2015-01-11 20:22:35

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have:

[    4.355351] radeon 0000:01:00.0: Invalid ROM contents

but this is for

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao PRO [Radeon R9 270]

which works nice in linux with the latest mesa-git drivers

hmmm, sometimes it shows Seabios on screen, sometimes screen is completely dark.. but with no "no connection".. it's like it's waiting for something

in dmesg I have:

[  740.315287] vfio-pci 0000:03:00.0: enabling device (0400 -> 0403)

[  744.276043] kvm: zapping shadow pages for mmio generation wraparound

Stupid me: should add tag bootmenu to os

Now I need to passthrough my second, ps2 keyboard to actually press F12. How?)

Last edited by walkindude (2015-01-11 23:07:15)

Offline

#3830 2015-01-12 00:25:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

Now I need to passthrough my second, ps2 keyboard to actually press F12. How?)

QEMU doesn't support ps/2 passthrough afaik.  Probably too much BIOS involved in ps/2.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3831 2015-01-12 00:37:09

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Damn, and my other, bluetooth keyboard doesn't work too. Not detected before any system has been booted.
Time to borrow some from friend)

Last edited by walkindude (2015-01-12 00:37:39)

Offline

#3832 2015-01-12 01:49:12

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It really is a bummer that the E3-12xx series lacks isolation on the CPU root ports.
Motherboards that use PLX chips (which have ACS) should help a lot.

http://www.anandtech.com/show/6170/four … s-and-evga

The Gigabyte diagram shows all CPU lanes connected directly into the switch, which is the ideal case.
The EVGA diagram shows the first x16 slot directly connected to the CPU, and all the other slots behind the switch.
In that situation, is it possible for the PLX switch to provide isolation between that directly-connected slot and all the other slots?  I'd hope it would be able to force all transactions to go directly upstream.

Offline

#3833 2015-01-12 01:54:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

It really is a bummer that the E3-12xx series lacks isolation on the CPU root ports.
Motherboards that use PLX chips (which have ACS) should help a lot.

http://www.anandtech.com/show/6170/four … s-and-evga

The Gigabyte diagram shows all CPU lanes connected directly into the switch, which is the ideal case.
The EVGA diagram shows the first x16 slot directly connected to the CPU, and all the other slots behind the switch.
In that situation, is it possible for the PLX switch to provide isolation between that directly-connected slot and all the other slots?  I'd hope it would be able to force all transactions to go directly upstream.

Nope, switches downstream of root ports don't help, the full topology must support ACS.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3834 2015-01-12 03:58:59

anthonym52
Member
Registered: 2014-12-07
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does anyone have any tips on reducing audio latency?

Offline

#3835 2015-01-12 05:35:00

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

anthonym52 wrote:

does anyone have any tips on reducing audio latency?

best option is to not use an emulated sound device in geust.

instead pasthrough a cheap usb sound card. you can find one for a few dollars.
or passthrough the graphic cards built in hdmi audio device.

Last edited by winie (2015-01-12 05:35:37)

Offline

#3836 2015-01-12 12:21:58

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:
anthonym52 wrote:

does anyone have any tips on reducing audio latency?

best option is to not use an emulated sound device in geust.

instead pasthrough a cheap usb sound card. you can find one for a few dollars.
or passthrough the graphic cards built in hdmi audio device.

or passthrough ODAC USB device (like this http://www.amazon.co.uk/Epiphany-Acoust … B00G36MTT0 - that's mine BTW) or even better, whole USB controller to which such device is attached.

also, use libvirt vcpupin or some other means of pinning VCPU.

finally, only if the computer is dedicated  to running virtual machines, you can use isolcpus kernel option to ensure that some cores are left by kernel scheduler for exclusive use of the virtual machine.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3837 2015-01-12 12:27:15

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yep, no UEFI support in that ROM.  If you use seabios then you need x-vga=on, but search back a couple weeks and find the wrapper script method of adding it, don't use the <qemu:arg> abomination.


... or only two pages back  : https://bbs.archlinux.org/viewtopic.php … 1#p1491451 wink


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3838 2015-01-12 19:33:06

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek
Already done that, see https://bbs.archlinux.org/viewtopic.php … 7#p1492217 at the end. Just forgot to set bootmenu timeout.
Tried you wrapper: no connection.

But after I had seen Seabios load screen, pressed F12, tried to boot from Windows DVD / Arch Live CD, see "Booting from CD/DVD" or "ISOLINUX ..." caption for about a second and then screen goes dark. Tried to wait several minutes: nothing, just blackness.

My current xml is:

<domain type='kvm'>
  <name>windows</name>
  <uuid>c8bc2373-6271-4810-991e-8618a0ee01d2</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='rom'>/usr/share/qemu/bios.bin</loader>
    <bootmenu enable='yes' timeout='60000'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-wrapper</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='directsync' io='threads'/>
      <source file='/var/lib/libvirt/images/windows.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='ide' index='1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x09da'/>
        <product id='0x0260'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x09da'/>
        <product id='0x000e'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

@Bronek
Of course, I know why your script didn't work for me: didn't load rom file for my card.

But when I try this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/home/walkindude/hd6850.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x15' function='0x0'/>
    </hostdev>

and it's definitely there:

walkindude@Desktop> ls /home/walkindude/hd6850.rom                                                                                                                           ~
/home/walkindude/hd6850.rom

it gives me

walkindude@Desktop> sudo virsh start windows                                                                                                                                 ~
error: Failed to start domain windows
error: internal error: early end of file from monitor: possible problem:
2015-01-12T20:25:41.102985Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: pci_add_option_rom: failed to find romfile "/home/walkindude/hd6850.rom"
2015-01-12T20:25:41.235373Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device initialization failed.
2015-01-12T20:25:41.235447Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device 'vfio-pci' could not be initialized

Last edited by walkindude (2015-01-12 20:26:27)

Offline

#3839 2015-01-12 22:46:11

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@Bronek
Of course, I know why your script didn't work for me: didn't load rom file for my card.

But when I try this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/home/walkindude/hd6850.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x15' function='0x0'/>
    </hostdev>

and it's definitely there:

walkindude@Desktop> ls /home/walkindude/hd6850.rom                                                                                                                           ~
/home/walkindude/hd6850.rom

it gives me

walkindude@Desktop> sudo virsh start windows                                                                                                                                 ~
error: Failed to start domain windows
error: internal error: early end of file from monitor: possible problem:
2015-01-12T20:25:41.102985Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: pci_add_option_rom: failed to find romfile "/home/walkindude/hd6850.rom"
2015-01-12T20:25:41.235373Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device initialization failed.
2015-01-12T20:25:41.235447Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device 'vfio-pci' could not be initialized


ok so my script uses "vga.rom" suffix in the filename to detect that VGA ROM is being loaded and by this it "knows" to append x-vga=on. If your filename suffix is different, or if you do not need to load VGA ROM (e.g because your computer's BIOS alread did this) then you can use some other distinguishing feature of VGA PCI device. Just run "ps lfxww | grep qemu"  to find what your qemu options look like and then you will know what the script should be looking for

Last edited by Bronek (2015-01-12 22:47:01)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3840 2015-01-12 23:09:10

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

walkindude@Desktop> ps lfxww | grep qemu                                                                             ~
0  1000  5944  1447   5   0  14256  2240 pipe_w S+   pts/0      0:00  |       \_ grep qemu

No qemu options there...

If I look in /var/log/libvirt/qemu/windows.log, I see

2015-01-12 23:01:43.682+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-wrapper -name windows -S -machine pc-i440fx-2.2,accel=kvm,usb=off -bios /usr/share/qemu/bios.bin -m 3072 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid c8bc2373-6271-4810-991e-8618a0ee01d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,splash-time=60000,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x8 -device ahci,id=ahci0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/lib/libvirt/images/windows.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x15 -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device usb-host,hostbus=4,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=o

As I wrote before, I use:

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|03:00.0|03:00.0,x-vga=on|g'`

It indeed shows Seabios info on desired screen, I can choose boot device (so keyboard also working), but then turns black.

Offline

#3841 2015-01-13 00:49:08

Japanlinux
Member
Registered: 2010-05-18
Posts: 173

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've tried this process several times without the kernel+patches mentioned in the OP and I have determined that I do need the ACS patch to make this work. I then proceeded to download the kernel, makepkg and pacman -U the mainline, kernel, and headers xz files that it generated 2 hours later. However, I do not have that kernel installed (still using the same default one with no option to use the new during boot). So I must have done something wrong here. I'd appreciate some help.

Offline

#3842 2015-01-13 02:30:35

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Japanlinux
Maybe you should do

grub-mkconfig -o /boot/grub/grub.cfg

?

Or something similar, if you have different bootloader

Last edited by walkindude (2015-01-13 02:32:56)

Offline

#3843 2015-01-13 09:26:19

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@Bronek

walkindude@Desktop> ps lfxww | grep qemu                                                                             ~
0  1000  5944  1447   5   0  14256  2240 pipe_w S+   pts/0      0:00  |       \_ grep qemu

No qemu options there...

If I look in /var/log/libvirt/qemu/windows.log, I see

2015-01-12 23:01:43.682+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-wrapper -name windows -S -machine pc-i440fx-2.2,accel=kvm,usb=off -bios /usr/share/qemu/bios.bin -m 3072 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid c8bc2373-6271-4810-991e-8618a0ee01d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,splash-time=60000,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x8 -device ahci,id=ahci0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/lib/libvirt/images/windows.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x15 -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device usb-host,hostbus=4,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=o

As I wrote before, I use:

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|03:00.0|03:00.0,x-vga=on|g'`

It indeed shows Seabios info on desired screen, I can choose boot device (so keyboard also working), but then turns black.

I suppose you do not see all processes in your ps lfxww command because you are not running it as root. Make it "ps alfxww" to see your effective qemu command. Above shows that the call to wrapper is correct and I checked your wrapper, it should work. So you should see "qemu-system-x86_64 . . . -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15 . . ." reported by ps

Looking at what you wrote earlier I would hazard a guess that you are not setting up permissions of libvirt relaxed enough to access all the devices it needs and the ROM file as well - this could explain "pci_add_option_rom: failed to find romfile "/home/walkindude/hd6850.rom" . See https://bbs.archlinux.org/viewtopic.php … 6#p1438076


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3844 2015-01-13 19:14:41

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have difficuties enabling hugepages. I followed the arch wiki.


First step ist to add this line to /etc/fstab:

hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0

First Problem. It says,

gid must match that of the kvm group

. How do I find out if the gids match? I have no clue ...

Then I mounted /dev/hugepages. No problem.

grep Hugepagesize /proc/meminfo shows:

[root@archkvm ~]# grep Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB

Now: I want my guest Windows 8.1 to run with 10 GB RAM. If I undersood correctly, I should add half to /etc/sysctl.d/40-hugepage.conf. So I added this:

vm.nr_hugepages = 5120

I am not sure about this though. My thought was: 10 GB equals 10*1024=10240. Half of it is 5120.

But when I start Windows and use the command "grep HugePages /proc/meminfo" it shows this:

[root@archkvm ~]# grep HugePages /proc/meminfo
AnonHugePages:    366592 kB
HugePages_Total:    5120
HugePages_Free:        0

HugePages_Rsvd:        0
HugePages_Surp:        0

I think it's pretty obvious I made a mistake. I think the allocation is wrong. Why does it show 0 for HugePages_Free?


oh, and this is what I added to my XML:

<memoryBacking>
    <hugepages/>
    <nosharepages/>
  </memoryBacking>

I am also not sure about those line. Especially the nosharepages.

Can someone please help me?


thank you very much.

Offline

#3845 2015-01-13 19:20:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
Duelist wrote:

...long story short...
I've finally flashed my cards with UEFI-compatible "secret"(i can't get it anywhere anymore) firmware, but appears like windows7 doesn't really works right with it. I turn off CSM, boot up the windows, it shows the splash logo and then image freezes. Yay for windows7. So, OVMF/UEFI and windows 7 is possible but troublesome, so most folks shouldn't bother about it.

Have you had any further success with UEFI Windows 7? I'm stuck at this flash screen as well, unfortunately.

No errors or messages, just a pulsing "Starting Windows" logo indefinitely on both the installation disk and a boot disk.

My test Win7 VM booted and worked on OVMF-pure-efi image with emulated graphics.
But when i try to passthrough my GPU with romfile option, it doesn't work.

Cubex wrote:

Maybe I sould have said this earlier, AFAIK Windows 7 boots in UEFI mode, however doesn't support UEFI vbios. I remenber getting a boot error on bare metal when disabling CSM even If i was booting using UEFI mode.
If you can't use seabios or ovmf with csm (like me), use Win8+  hmm

Windows 7 boots and shows me that "starting windows" logo, BUT when it's time for some high-res video output through GOP, the image freezes on my bare metal hardware(with CSM disabled, of course) while system continues to boot. I can't make any valuable conclusions from that because i haven't tested booting Win8 on that hardware and my vbios is "illegal" - you can't get it anywhere anymore(yay for asus). But hang on a second, i've got an idea..

There should be no "boot errors", it's just the image output part.

It may be not so obvious, but to install windows on UEFI, you need to boot the UEFI-setup and use GPT on your disk. Only that way the windows installation cd will provide you EFI loader.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3846 2015-01-13 19:27:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi, I have difficuties enabling hugepages. I followed the arch wiki.


First step ist to add this line to /etc/fstab:

hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0

First Problem. It says,

gid must match that of the kvm group

. How do I find out if the gids match? I have no clue ...

Then I mounted /dev/hugepages. No problem.

I doubt the above is necessary, but then I don't use arch, so I don't know for sure.  On Fedora hugetlbfs is already mounted and works, all that's necessary is to make sure there are some available.

grep Hugepagesize /proc/meminfo shows:

[root@archkvm ~]# grep Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB

Now: I want my guest Windows 8.1 to run with 10 GB RAM. If I undersood correctly, I should add half to /etc/sysctl.d/40-hugepage.conf. So I added this:

vm.nr_hugepages = 5120

I am not sure about this though. My thought was: 10 GB equals 10*1024=10240. Half of it is 5120.

But when I start Windows and use the command "grep HugePages /proc/meminfo" it shows this:

[root@archkvm ~]# grep HugePages /proc/meminfo
AnonHugePages:    366592 kB
HugePages_Total:    5120
HugePages_Free:        0

HugePages_Rsvd:        0
HugePages_Surp:        0

I think it's pretty obvious I made a mistake. I think the allocation is wrong. Why does it show 0 for HugePages_Free?

Looks like what I would expect.  You have 5120 hugepages, 0 are free, therefore you've configured the system for exactly the number you need.  Congrats.  If you want further confirmation, look in top.  The resident memory for the qemu process should be much smaller now since it doesn't include hugepages.

oh, and this is what I added to my XML:

<memoryBacking>
    <hugepages/>
    <nosharepages/>
  </memoryBacking>

I am also not sure about those line. Especially the nosharepages.

Matches what I have.

Last edited by aw (2015-01-13 19:28:15)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3847 2015-01-13 19:33:18

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, that's it then? smile Great, I didn't know how to check. Very happy then smile

thank you!

Offline

#3848 2015-01-13 20:17:57

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

Did as you'd told me. Now it doesn't complain about rom file, but black screen after "Booting from CD/DVD" remains.

walkindude@Desktop> sudo ps alfxww | grep qemu                                                                       ~
0  1000  6602  1492   4   0  14256  2244 pipe_w S+   pts/0      0:00          |   |   \_ grep qemu
6     0  5830     1   1   0 4001136 3175140 poll_s SLl ?        0:52 qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.2,accel=kvm,usb=off -bios /usr/share/qemu/bios.bin -m 3072 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid c8bc2373-6271-4810-991e-8618a0ee01d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,splash-time=60000,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x8 -device ahci,id=ahci0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/lib/libvirt/images/windows.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device usb-host,hostbus=4,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

in dmesg

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

[  450.266194] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)

Tried with rom which I got from running Atiflash and also some roms from techPowerUp. Every time: black screen after Seabios and boot device choosing.

It's like I'm missing something stupid or something big, dunno

Last edited by walkindude (2015-01-13 20:57:01)

Offline

#3849 2015-01-13 21:09:45

Nanori
Member
Registered: 2015-01-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I follow this topic for a long time in order to manage a VGA pass-through on an Nvidia GTX 760.

At this time, I'm able to start my VM without errors from qemu but my screen seems to receve no signal from the card... :'(
I've perform a successful try with an ATI Radeon 5770 so I hope that isn't mean my GTX 760 doesn't support pass-through hmm

Here is my configuration :
   Host software configuraton (Fedora 21) :
         kernel 3.17-300 without any patch (I've try 3.18 with i915 patch but I've get the save result)
         cmdline :

 GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet intel_iommu=on \ vfio_iommu_type1.allow_unsafe_interrupts=1 \ kvm_intel.enable_apicv=1 \ kvm_intel.emulate_invalid_guest_state=0 \ kvm.allow_unsafe_assigned_interrupts=1 \ pci-stub.ids=10de:1187,10de:0e0a"

         qemu 2.2.0

   Host GPU (IGD):

 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller

   Guest GPU and attached audio device (GTX 760) :

 01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] 01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)

To run my VM, I use those commands :

./vfio-bind 0000:01:00.0 0000:01:00.1
qemu-system-x86_64 \ -enable-kvm -M q35 \ -vga none \ -m 4096 \ -boot menu=on \ -cpu host -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=${HOME}/MSI.GTX760.2048.140225.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -hda ${HOME}/windowsDisk"

After running this command, dmesg display those messages :

[ 3843.261116] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 3849.124426] kvm: zapping shadow pages for mmio generation wraparound

Can someone give me a hand ?

Offline

#3850 2015-01-13 21:24:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/me beats head on desk...

If you're using Intel host graphics and VGA mode for the device, you need the i915 patch.  If it didn't work for you, it's probably because you didn't enable it via the kernel commandline.

Most of your current kernel options are unnecessary.

And... since you have a Windows guest, don't use q35!

Better yet, use Windows8+ and OVMF.

Doesn't seem like you've been following the thread very carefully...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3851 2015-01-13 22:24:52

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@Bronek

Tried with rom which I got from running Atiflash and also some roms from techPowerUp. Every time: black screen after Seabios and boot device choosing.

It's like I'm missing something stupid or something big, dunno

Sorry, I'm out of ideas. Perhaps aw will be able to help. Also, which versions of software are your running? Did you try using ROM from your own card?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3852 2015-01-13 22:25:14

Nanori
Member
Registered: 2015-01-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

/me beats head on desk...

If you're using Intel host graphics and VGA mode for the device, you need the i915 patch.  If it didn't work for you, it's probably because you didn't enable it via the kernel commandline.

Even if I use this patch (and specify the i915.enable_hd_vgaarb=1 in command line), the Nvidia card doesn't send any signal to the screen. Without this patch, the ATI card work perfectly. For each test I make, I try the tow kernels (3.17 not patched and 3.18 patched) to optimize the chance of success

aw wrote:

Most of your current kernel options are unnecessary.

Yes, but I've try them while reading some other topics about VGA Passthrough (nothing ventured, nothing gained big_smile ). Remove those unnecessary options from cmdline doen't solved the issue

aw wrote:

And... since you have a Windows guest, don't use q35!

Better yet, use Windows8+ and OVMF.

Doesn't seem like you've been following the thread very carefully...

At this time, I'm trying to have a Windows 7 guest (preferably without using an UEFI bios). Moreover, the "windowsDisk" I use dosen't contain any data (no OS). So I should be able to see the bios on my screen when I boot my VM but it isn't the case.
Does a 440fx is realy better ?

Offline

#3853 2015-01-13 22:55:23

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

Kernel: https://aur.archlinux.org/packages/linux-uksm-ck/

walkindude@Desktop> uname -a                                                                                         ~
Linux Desktop 3.18.2-1-uksm-ck #1 SMP PREEMPT Wed Jan 14 01:23:45 MSK 2015 x86_64 GNU/Linux

Also patched it with ACS patch

Host video driver (radeonsi):

walkindude@Desktop> glxinfo | grep version                                                                           ~
server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.5.0-devel (git-0e6f0ee)
OpenGL core profile shading language version string: 3.30
OpenGL version string: 3.0 Mesa 10.5.0-devel (git-0e6f0ee)
OpenGL shading language version string: 1.30
OpenGL ES profile version string: OpenGL ES 3.0 Mesa 10.5.0-devel (git-0e6f0ee)
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.0

walkindude@Desktop> qemu-x86_64 -version                                                                             ~
qemu-x86_64 version 2.2.50, Copyright (c) 2003-2008 Fabrice Bellard

walkindude@Desktop> libvirtd -v                                                                                     
2015-01-13 22:40:53.819+0000: 9697: info : libvirt version: 1.2.1

Yes, I'd booted from FreeDos, and backed up ROM (by ATIflash utility) of my own card - same result.

What I see on desired screen is
http://rghost.net/60334498.view

Whatever option I choose -> dark screen.

Could it be because of (one string for one launch)?

[  140.296537] kvm: zapping shadow pages for mmio generation wraparound
[ 1134.175319] kvm: zapping shadow pages for mmio generation wraparound

Last edited by walkindude (2015-01-13 23:25:51)

Offline

#3854 2015-01-13 23:32:39

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nanori wrote:

So I should be able to see the bios on my screen when I boot my VM but it isn't the case.

Maybe you should set

-boot menu=on,splash-time=60000,strict=on

or some other preferable time [1...65535] ?

Offline

#3855 2015-01-13 23:43:59

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried the Win10 preview with passthrough?
I assume it's a bit early, but I'm waiting for it as I have no 8.1 license and failed with Win7 & nvidia.

Offline

#3856 2015-01-13 23:50:01

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wimma77
I'm trying to do it the right way (build 9888), read my messages wink
But for now, I cannot go further than selecting the boot device.
Previously I'd installed it successfully but with VNC and Cirrus, and it gave me a lot of VIDEO_TDR_FAILURE after I installed Catalyst.

Last edited by walkindude (2015-01-13 23:56:56)

Offline

#3857 2015-01-14 00:11:15

Wimma77
Member
Registered: 2014-08-12
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@walkindude
Yeah, trying to keep up with progress, but been a while. I did have success with a HD7770 (way back around page 80/90 days), but struggled with my GTX750ti so gave it a break. Hoping that it get a little easier as things mature. Want to try OVMF, and have looked through aw's blog, but I don't have Win8 so can't try yet.
Also my server is in full use at home, so don't have the play time I used to, so need to be confident before I start messing around again.
Think I need another test box ...

Last edited by Wimma77 (2015-01-14 00:11:44)

Offline

#3858 2015-01-14 02:39:07

dam5h
Member
Registered: 2011-08-04
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Would this be possible with a haswell laptop with only the integrated graphics?  It would be okay that once the guest boots and takes control of the GPU you could no longer access the host until you shut down the guest.

Thanks.

Offline

#3859 2015-01-14 02:40:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dam5h wrote:

Would this be possible with a haswell laptop with only the integrated graphics?  It would be okay that once the guest boots and takes control of the GPU you could no longer access the host until you shut down the guest.

Thanks.

No


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3860 2015-01-14 05:19:50

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dam5h wrote:

Would this be possible with a haswell laptop with only the integrated graphics?  It would be okay that once the guest boots and takes control of the GPU you could no longer access the host until you shut down the guest.

Thanks.

use xengt:
https://github.com/01org/XenGT-Preview-kernel
it does the gpu equivalent  of vt-x for the cpu and uses the integrated graphics for host and guest at the same time smile

Offline

#3861 2015-01-14 06:41:00

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

or maybe kvmGT? did anyone try it?

Offline

#3862 2015-01-14 09:11:21

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

@Bronek
Host video driver (radeonsi):

walkindude@Desktop> glxinfo | grep version                                                                           ~
server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
OpenGL core profile version string: 3.3 (Core Profile) Mesa 10.5.0-devel (git-0e6f0ee)
OpenGL core profile shading language version string: 3.30
OpenGL version string: 3.0 Mesa 10.5.0-devel (git-0e6f0ee)
OpenGL shading language version string: 1.30
OpenGL ES profile version string: OpenGL ES 3.0 Mesa 10.5.0-devel (git-0e6f0ee)
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.0


OK so you have versions of everything recent enough. How many GPUs have you got? Are you sure that the one you want passed through is not being claimed by radeon driver? I have blacklisted radeon on my machine, and it's being run headless. You can use lspci -vnn -s 03:00.0 for this, you may also need to check /proc/iomem  as suggested by aw on page 130 of this thread.

EDIT: also, capabilities of driver loaded in host are totally irrelevant for passthrough. The whole idea of VGA passthrough is to make the card  unavailable for the host and available for the guest instead. Not some part of the card, but all of it, starting from emulated PCI channel. The only thing that hosts own drivers need to be able to do, is to sod off. Some drivers are not very good in this and will actually claim the card you want to passthrough, just because it has recognized chip type. It might be that the module you have loaded in the host is this kind of a driver.

Last edited by Bronek (2015-01-14 09:19:08)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3863 2015-01-14 09:39:29

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@walkindude, actually you reminded me I wanted to ask aw something : which video modules are known to cause trouble/ignore VGA arbitration? Assuming guest OS is not EFI compatible (i.e. OVMF is not available) and recent kernel is being used on host. I think the following options are normally available : vga16fb , vesafb , radeon/nouveau, radeonfb/nvidiafb (i915 is not an option for reasons you explained often enough, but what about intelfb?). Some users may run fbcon on top of it, or run a non-free driver instead of regular kernel module.

Also from reading http://vfio.blogspot.co.uk/2014/08/what … ation.html I understand that Xorg does not play (at all) with VGA arbitration, but it not does not imply that another GPU cannot be passed through, does it?

Last edited by Bronek (2015-01-14 10:35:04)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3864 2015-01-14 12:12:52

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After reading a number of the posts here about hugepages I figured I'd give it a shot and have set it up. I'm not sure if there's much of a performance improvement, but I haven't really tested it yet.

I also took the time to finally remove the need for the vfio-bind script since I was still binding a network adapter because I had been too lazy to fix my network config files, which was adding all en* interfaces to my bridge and causing the network on the host not to start. The less messing around with extra scripts that I can do, the better.

I have to say that I'm pretty happy to have everything working well, taking me from 2 separate computers (1 for linux, 1 for windows gaming) down to 1.

Offline

#3865 2015-01-14 13:05:55

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@walkindude , one more thing, show us the result of

$ dmesg | grep -E "0000:03:00|radeon|stub|vfio"

Last edited by Bronek (2015-01-14 13:20:07)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3866 2015-01-14 16:57:20

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

walkindude@Desktop> sudo dmesg | grep -E "0000:03:00|radeon|stub|vfio"                                               ~
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-uksm-ck root=UUID=7d535c2f-90b6-4f8d-b39f-ff79dd6dfedf rw root=/dev/mapper/lvmVolume-lvmRoot discard rw iommu=pt iommu=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 amd_iommu_dump=1 pci-stub.ids=1002:6739,1002:aa88 pcie_acs_override=downstream
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-uksm-ck root=UUID=7d535c2f-90b6-4f8d-b39f-ff79dd6dfedf rw root=/dev/mapper/lvmVolume-lvmRoot discard rw iommu=pt iommu=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 amd_iommu_dump=1 pci-stub.ids=1002:6739,1002:aa88 pcie_acs_override=downstream
[    0.502020] pci 0000:03:00.0: [1002:6739] type 00 class 0x030000
[    0.502040] pci 0000:03:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.502053] pci 0000:03:00.0: reg 0x18: [mem 0xfe820000-0xfe83ffff 64bit]
[    0.502061] pci 0000:03:00.0: reg 0x20: [io  0xc000-0xc0ff]
[    0.502075] pci 0000:03:00.0: reg 0x30: [mem 0xfe800000-0xfe81ffff pref]
[    0.502160] pci 0000:03:00.0: supports D1 D2
[    0.502224] pci 0000:03:00.1: [1002:aa88] type 00 class 0x040300
[    0.502243] pci 0000:03:00.1: reg 0x10: [mem 0xfe840000-0xfe843fff 64bit]
[    0.502330] pci 0000:03:00.1: supports D1 D2
[    0.505789] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=none,locks=none
[    0.505792] vgaarb: bridge control possible 0000:03:00.0
[    0.512441] pci 0000:03:00.0: can't claim BAR 0 [mem 0xc0000000-0xcfffffff 64bit pref]: no compatible bridge window
[    0.528611] pci 0000:03:00.0: BAR 0: no space for [mem size 0x10000000 64bit pref]
[    0.528614] pci 0000:03:00.0: BAR 0: trying firmware assignment [mem size 0x10000000 64bit pref]
[    0.528616] pci 0000:03:00.0: BAR 0: assigned [mem 0xc0000000-0xcfffffff 64bit pref]
[    3.000492] pci-stub: add 1002:6739 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.000505] pci-stub 0000:03:00.0: claimed by stub
[    3.000511] pci-stub: add 1002:AA88 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.000526] pci-stub 0000:03:00.1: claimed by stub
[    4.984586] [drm] radeon kernel modesetting enabled.
[    4.984674] fb: switching to radeondrmfb from EFI VGA
[    4.989974] radeon 0000:01:00.0: Invalid ROM contents
[    4.990438] radeon 0000:01:00.0: VRAM: 2048M 0x0000000000000000 - 0x000000007FFFFFFF (2048M used)
[    4.990441] radeon 0000:01:00.0: GTT: 1024M 0x0000000080000000 - 0x00000000BFFFFFFF
[    4.990648] [drm] radeon: 2048M of VRAM memory ready
[    4.990650] [drm] radeon: 1024M of GTT memory ready.
[    5.011140] [drm] radeon: dpm initialized
[    5.023619] radeon 0000:01:00.0: WB enabled
[    5.023623] radeon 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00 and cpu addr 0xffff880001cdfc00
[    5.023625] radeon 0000:01:00.0: fence driver on ring 1 use gpu addr 0x0000000080000c04 and cpu addr 0xffff880001cdfc04
[    5.023628] radeon 0000:01:00.0: fence driver on ring 2 use gpu addr 0x0000000080000c08 and cpu addr 0xffff880001cdfc08
[    5.023630] radeon 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c and cpu addr 0xffff880001cdfc0c
[    5.023633] radeon 0000:01:00.0: fence driver on ring 4 use gpu addr 0x0000000080000c10 and cpu addr 0xffff880001cdfc10
[    5.024698] radeon 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000000075a18 and cpu addr 0xffffc90011135a18
[    5.024704] radeon 0000:01:00.0: radeon: MSI limited to 32-bit
[    5.024742] radeon 0000:01:00.0: irq 30 for MSI/MSI-X
[    5.024756] radeon 0000:01:00.0: radeon: using MSI.
[    5.024789] [drm] radeon: irq initialized.
[    6.162222] fbcon: radeondrmfb (fb0) is primary device
[    6.208766] radeon 0000:01:00.0: fb0: radeondrmfb frame buffer device
[    6.208768] radeon 0000:01:00.0: registered panic notifier
[    6.216932] [drm] Initialized radeon 2.40.0 20080528 for 0000:01:00.0 on minor 0
[  523.806082] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[  523.879271] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)

walkindude@Desktop> sudo lspci -vnn -s 03:00.0 
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739] (prog-if 00 [VGA controller])
    Subsystem: PC Partner Limited / Sapphire Technology Device [174b:e174]
    Flags: fast devsel, IRQ 16
    Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
    Memory at fe820000 (64-bit, non-prefetchable) [disabled] [size=128K]
    I/O ports at c000 [disabled] [size=256]
    Expansion ROM at fe800000 [disabled] [size=128K]
    Capabilities: [50] Power Management version 3
    Capabilities: [58] Express Legacy Endpoint, MSI 00
    Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
    Capabilities: [150] Advanced Error Reporting
        Kernel driver in use: pci-stub
    Kernel modules: radeon

walkindude@Desktop> cat /proc/iomem                                                                                                            ~
00000000-00000fff : reserved
00001000-0009ffff : System RAM
000a0000-000bffff : PCI Bus 0000:00
000c0000-000dffff : PCI Bus 0000:00
  000c0000-000cffff : Video ROM
000f0000-000fffff : System ROM
00100000-bda58fff : System RAM
  03000000-0345329d : Kernel code
  0345329e-038e57ff : Kernel data
  03a0a000-03b7afff : Kernel bss
bda59000-bdaa9fff : ACPI Non-volatile Storage
bdaaa000-bdab2fff : ACPI Tables
bdab3000-bdab3fff : ACPI Non-volatile Storage
bdab4000-bdad8fff : reserved
bdad9000-bdae9fff : ACPI Non-volatile Storage
bdaea000-bdaebfff : reserved
bdaec000-bdafcfff : reserved
bdafd000-bdafdfff : reserved
bdafe000-bdafefff : reserved
bdaff000-bdb00fff : ACPI Non-volatile Storage
bdb01000-bdb01fff : reserved
bdb02000-bdb02fff : reserved
bdb03000-bdb09fff : reserved
bdb0a000-bdb0ffff : ACPI Non-volatile Storage
bdb10000-bdb1cfff : reserved
bdb1d000-bdb5efff : reserved
bdb5f000-bdb6efff : reserved
bdb6f000-bdb71fff : reserved
bdb72000-bdd74fff : ACPI Non-volatile Storage
bdd75000-bdefffff : System RAM
bdf00000-bfffffff : RAM buffer
c0000000-cfffffff : 0000:03:00.0
d0000000-ffffffff : PCI Bus 0000:00
  d0000000-dfffffff : PCI Bus 0000:01
    d0000000-dfffffff : 0000:01:00.0
  e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
    e0000000-efffffff : pnp 00:00
  f8000000-fbffffff : reserved
  fe800000-fe8fffff : PCI Bus 0000:03
    fe800000-fe81ffff : 0000:03:00.0
    fe820000-fe83ffff : 0000:03:00.0
    fe840000-fe843fff : 0000:03:00.1
  fe900000-fe9fffff : PCI Bus 0000:02
    fe900000-fe91ffff : 0000:02:06.0
    fe920000-fe9200ff : 0000:02:06.0
      fe920000-fe9200ff : r8169
  fea00000-feafffff : PCI Bus 0000:01
    fea00000-fea3ffff : 0000:01:00.0
    fea40000-fea5ffff : 0000:01:00.0
    fea60000-fea63fff : 0000:01:00.1
      fea60000-fea63fff : ICH HD audio
  feb00000-feb000ff : 0000:00:16.2
    feb00000-feb000ff : ehci_hcd
  feb01000-feb01fff : 0000:00:16.0
    feb01000-feb01fff : ohci_hcd
  feb02000-feb02fff : 0000:00:14.5
    feb02000-feb02fff : ohci_hcd
  feb03000-feb030ff : 0000:00:13.2
    feb03000-feb030ff : ehci_hcd
  feb04000-feb04fff : 0000:00:13.0
    feb04000-feb04fff : ohci_hcd
  feb05000-feb050ff : 0000:00:12.2
    feb05000-feb050ff : ehci_hcd
  feb06000-feb06fff : 0000:00:12.0
    feb06000-feb06fff : ohci_hcd
  feb07000-feb073ff : 0000:00:11.0
    feb07000-feb073ff : ahci
  feb20000-feb23fff : amd_iommu
  fec00000-fec00fff : reserved
    fec00000-fec003ff : IOAPIC 0
  fec10000-fec10fff : reserved
    fec10000-fec10fff : pnp 00:01
  fec20000-fec20fff : reserved
    fec20000-fec203ff : IOAPIC 1
  fed00000-fed00fff : reserved
    fed00000-fed003ff : HPET 0
      fed00000-fed003ff : PNP0103:00
  fed61000-fed70fff : reserved
    fed61000-fed70fff : pnp 00:01
  fed80000-fed8ffff : reserved
    fed80000-fed8ffff : pnp 00:01
      fed80b00-fed80b07 : SB800 TCO
  fee00000-fee00fff : Local APIC
    fee00000-fee00fff : pnp 00:01
  fef00000-ffffffff : reserved
    ffc00000-ffffffff : pnp 00:01
100001000-23effffff : System RAM
23f000000-23fffffff : RAM buffer

Two GPUs:
HD6850
R9 270

Mobo:
M5A97 Pro

Processor:
Amd FX-8320

Last edited by walkindude (2015-01-14 17:00:25)

Offline

#3867 2015-01-14 20:23:44

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since qemu 2.2.0-1 I get the following error:
qemu-system-x86_64: -device ide-hd,bus=ide.0,drive=disk: Property 'ide-hd.drive' can't take value 'disk', it's in use

Is there a workaround?

Offline

#3868 2015-01-14 20:35:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Since qemu 2.2.0-1 I get the following error:
qemu-system-x86_64: -device ide-hd,bus=ide.0,drive=disk: Property 'ide-hd.drive' can't take value 'disk', it's in use

Is there a workaround?

Don't name your drive "disk"?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3869 2015-01-14 20:46:01

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried diska, before posting.
Now I tried 'win', same errors.

Edit:
My full command is:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/rommy/vms/win8/GF100.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -usb -usbdevice host:0079:0006 -usbdevice host:062a:0252 -usbdevice host:04f3:0103 -drive file=/home/rommy/vms/win8/win8_vga_rw.qcow2,id=win,format=qcow2 -device ide-hd,bus=ide.0,drive=win -nographic

Last edited by Rommy (2015-01-14 20:48:38)

Offline

#3870 2015-01-14 20:47:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

I tried diska, before posting.
Now I tried 'win', same errors.

That's the only guess I'm going to give without the rest of your command line.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3871 2015-01-14 20:49:08

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry, I edited it. See above

Offline

#3872 2015-01-14 20:50:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

I tried diska, before posting.
Now I tried 'win', same errors.

Edit:
My full command is:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/rommy/vms/win8/GF100.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -usb -usbdevice host:0079:0006 -usbdevice host:062a:0252 -usbdevice host:04f3:0103 -drive file=/home/rommy/vms/win8/win8_vga_rw.qcow2,id=win,format=qcow2 -device ide-hd,bus=ide.0,drive=win -nographic

add "if=none"

...,drive=win,if=none

qemu 2.2 should support -cdrom -hda, etc using the q35 chipset

Offline

#3873 2015-01-14 20:57:23

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/rommy/vms/win8/GF100.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -usb -usbdevice host:0079:0006 -usbdevice host:062a:0252 -usbdevice host:04f3:0103 -drive file=/home/rommy/vms/win8/win8_vga_rw.qcow2,id=win,format=qcow2 -device ide-hd,bus=ide.0,drive=win,if=none -nographic

Error:

qemu-system-x86_64: -device ide-hd,bus=ide.0,drive=win,if=none: Property 'ide-hd.drive' can't take value 'win', it's in use

Offline

#3874 2015-01-14 21:01:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/rommy/vms/win8/GF100.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -usb -usbdevice host:0079:0006 -usbdevice host:062a:0252 -usbdevice host:04f3:0103 -drive file=/home/rommy/vms/win8/win8_vga_rw.qcow2,id=win,format=qcow2 -device ide-hd,bus=ide.0,drive=win,if=none -nographic

Error:

qemu-system-x86_64: -device ide-hd,bus=ide.0,drive=win,if=none: Property 'ide-hd.drive' can't take value 'win', it's in use

Yeah sorry it should be like this:

... -drive file=/home/rommy/vms/win8/win8_vga_rw.qcow2,id=win,format=qcow2,if=none -device ide-hd,bus=ide.0,drive=win ...

EDITED

Last edited by nbhs (2015-01-14 21:03:41)

Offline

#3875 2015-01-14 21:03:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think if=none goes on the -drive option...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3876 2015-01-14 21:05:49

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you :-) It works!

Offline

#3877 2015-01-14 21:06:05

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I think if=none goes on the -drive option...

Yeah sorry, i edited it

Offline

#3878 2015-01-15 14:29:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mmmay be what we need is a new FAQ/OP-post made up. Any opinions/suggestions about that?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3879 2015-01-16 05:49:08

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've followed the steps and attempted to read through a lot of the comments looking for anything relating to my problem but I haven't found anything to help.

First, my specs

CPU: i7 4790k
Mobo: g1.sniper h6
Main GPU: GTX 970
Passthrough GPU: GTX 260

I currently see no errors in dmesg or from qemu. I am using this to start the VM

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -vga none -device vfio-pci,host=06:00.0,x-vga=on

The QEMU 2.2.0 monitor console pops up and when I switch inputs on my monitor all I see is "entering powersave mode" and no signal seems to be passed to it.

The only dmesg info that shows up is this

[ 2283.841755] vfio-pci 0000:06:00.0: enabling device (0400 -> 0403)

I am using the kernel supplied in OP's post which was built using makepkg. The card is picked up by pci-stub in dmesg, the vfio-bind command works without error.

Last edited by tully (2015-01-16 06:09:22)

Offline

#3880 2015-01-16 06:35:48

ceri
Member
Registered: 2013-10-12
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Mmmay be what we need is a new FAQ/OP-post made up. Any opinions/suggestions about that?

It would be nice to have everything that's known all in one place, as opposed to scattered all around this thread with no decent way of searching (besides awkward google searches). I think putting it on the arch wiki would be best since it would be editable by all.

I actually considered starting one but I don't have time to maintain it. Here is a collection of useful links I found for my personal usage:
http://pastebin.com/sdkZPykz

(for some links, you have to scroll down to a *reply* with the useful info)

Offline

#3881 2015-01-16 07:21:25

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ tully
try using rom file for your gpu

Offline

#3882 2015-01-16 08:43:37

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:

I've followed the steps and attempted to read through a lot of the comments looking for anything relating to my problem but I haven't found anything to help.

First, my specs

CPU: i7 4790k
Mobo: g1.sniper h6
Main GPU: GTX 970
Passthrough GPU: GTX 260

I currently see no errors in dmesg or from qemu. I am using this to start the VM

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -vga none -device vfio-pci,host=06:00.0,x-vga=on

The QEMU 2.2.0 monitor console pops up and when I switch inputs on my monitor all I see is "entering powersave mode" and no signal seems to be passed to it.

The only dmesg info that shows up is this

[ 2283.841755] vfio-pci 0000:06:00.0: enabling device (0400 -> 0403)

I am using the kernel supplied in OP's post which was built using makepkg. The card is picked up by pci-stub in dmesg, the vfio-bind command works without error.

there is a note in OP:

NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this.

i beleive there was a note in now-missing ISSUES section about the need to patch nvidia drivers if both host and guest are using nvidia gpus (or something like that - cant remember for sure)

Last edited by sinny (2015-01-16 08:43:49)

Offline

#3883 2015-01-16 11:42:53

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:

I've followed the steps and attempted to read through a lot of the comments looking for anything relating to my problem but I haven't found anything to help.

First, my specs

CPU: i7 4790k...

I suggest you to use the intel integrated gpu as main gpu and use OVMF uefi in guest instead of bios

Offline

#3884 2015-01-16 13:18:11

zlice
Member
Registered: 2015-01-16
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

don't see a way to search just this thread (156 pages)

i have 2 cards - problem is the card i'd like to passthrough is seen by my mobo(asus) as primary, guessing because it's in slot1, but i can't swap it because the thing's so big

is there any way to switch the pci bus? i've seen people talk about modding kernel code but i don't really know where i'd start to look if that's even the answer


edit:
Asus M5A99FX PRO R20
pci-stub is in dmesg but the card is still what displays bios and OS
even trying to run Xorg on the card i want, it grabs pci:1:0:0 even though it's not in my xorg.conf
both ati/amd cards (5870 and 290)

Last edited by zlice (2015-01-16 13:23:03)

Offline

#3885 2015-01-16 15:12:30

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:
tully wrote:

I've followed the steps and attempted to read through a lot of the comments looking for anything relating to my problem but I haven't found anything to help.

First, my specs

CPU: i7 4790k...

I suggest you to use the intel integrated gpu as main gpu and use OVMF uefi in guest instead of bios


I have three monitors. I can't extend to 3 monitors with IGP.

Offline

#3886 2015-01-16 16:18:17

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

@ tully
try using rom file for your gpu

I pulled the ROM with gpu-z in windows and tried a few from techpowerup. Here is the output when I try to use it.

root@tully ~]# qemu-system-x86_64 -enable-kvm -M q35 -m 4G -cpu host -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -k en-us -no-reboot -vga none -device qxl -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/root/GT200.rom -boot menu=on dma: command 84 not supported dma: command 84 not supported dma: command b4 not supported dma: command b4 not supported dma: command 84 not supported dma: command 84 not supported dma: command 78 not supported dma: command c not supported dma: command 48 not supported dma: command 48 not supported dma: command 4c not supported dma: command c not supported

Offline

#3887 2015-01-16 16:37:43

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zlice wrote:

i have 2 cards - problem is the card i'd like to passthrough is seen by my mobo(asus) as primary, guessing because it's in slot1, but i can't swap it because the thing's so big

is there any way to switch the pci bus? i've seen people talk about modding kernel code but i don't really know where i'd start to look if that's even the answer

Unless there's an option in your bios to actually choose which PCIe slot to boot primary graphics from, which from a look at the manual doesn't seem to be the case and not something I have on my own board (Asus P9D WS), so your only other option is to swap the cards around.

When you say you can't swap because it's too big, is it because it wouldn't fit in that spot in the case or because it's taking up another slot that you're using or due to the physical constraints of the case? It looks like, from the manual, that only two of the PCIe/PCI slots are PCIe 2 x16 slots on that board as well (1 and 4).

Last edited by Myranti (2015-01-16 16:38:06)

Offline

#3888 2015-01-16 16:54:23

zlice
Member
Registered: 2015-01-16
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Myranti wrote:
zlice wrote:

i have 2 cards - problem is the card i'd like to passthrough is seen by my mobo(asus) as primary, guessing because it's in slot1, but i can't swap it because the thing's so big

is there any way to switch the pci bus? i've seen people talk about modding kernel code but i don't really know where i'd start to look if that's even the answer

Unless there's an option in your bios to actually choose which PCIe slot to boot primary graphics from, which from a look at the manual doesn't seem to be the case and not something I have on my own board (Asus P9D WS), so your only other option is to swap the cards around.

When you say you can't swap because it's too big, is it because it wouldn't fit in that spot in the case or because it's taking up another slot that you're using or due to the physical constraints of the case? It looks like, from the manual, that only two of the PCIe/PCI slots are PCIe 2 x16 slots on that board as well (1 and 4).

ya there's nothing in bios about video but switching PEG/PCI which does nothing.
and yes the card it too big, i'd have to cut up the HD holder.

i may try to unbind/bind and ssh to see if i can get anything to happen...then i'm going to take it out of the case to make sure slot1 is what's making it display (probably why)

if i can get it working out of the case...i'll probably burn another $70...

Offline

#3889 2015-01-16 17:31:47

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:
winie wrote:
tully wrote:

I've followed the steps and attempted to read through a lot of the comments looking for anything relating to my problem but I haven't found anything to help.

First, my specs

CPU: i7 4790k...

I suggest you to use the intel integrated gpu as main gpu and use OVMF uefi in guest instead of bios


I have three monitors. I can't extend to 3 monitors with IGP.

well, if you gonna ignore possible need to patch nvidia drivers, you could try twisting your setup a bit:
- [ host ] virtualization host, nothing more (minimal install)
- [ guest0 ] takes the role of your current triple-monitor host
- [ guest1 ] your currently desired guest (i assume windows)

Offline

#3890 2015-01-16 18:47:54

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone , long time no see .

Does anyone experience soft lockups of VMs once every couple of minutes ?

I always experience such lockups that lasts for a second or less in every Windows 8.1 VM I have . This is on my X99 host by the way .

CMD line if anyone would have an idea what's causing them :

qemu-system-x86_64 -name main -nographic \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -vga none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_main.bin \ -device vfio-pci,host=05:00.0,multifunction=on \ -device vfio-pci,host=05:00.1 \ -device vfio-pci,host=0f:00.0 \ -device vfio-pci,host=00:1b.0 \ -drive file=/VMs/Win_Main.img,cache=none,format=raw,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -localtime \ -net nic,model=virtio,macaddr=33:29:21:5F:C4:7D -net bridge,br=br0 \ -monitor unix:/tmp/vm_main,server,nowait &

I believe that heavy HDD activity causes it more often . These VMs are sitting on a 4 disk RAID0 . Disks are 7200 ones .

Offline

#3891 2015-01-16 19:01:00

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:
tully wrote:
winie wrote:

I suggest you to use the intel integrated gpu as main gpu and use OVMF uefi in guest instead of bios


I have three monitors. I can't extend to 3 monitors with IGP.

well, if you gonna ignore possible need to patch nvidia drivers, you could try twisting your setup a bit:
- [ host ] virtualization host, nothing more (minimal install)
- [ guest0 ] takes the role of your current triple-monitor host
- [ guest1 ] your currently desired guest (i assume windows)

Would you mind elaborating on that setup? I'm assuming you are suggesting that I passthrough my 970 to a linux VM and then the 260 to a windows VM. I would think that if the 260 doesn't currently work with this setup that OP provided that it wouldn't work under those circumstances either. I'm probably missing something though which is why I'm asking for clarification.

I wasn't attempting to ignore patching the nvidia drivers, I simply haven't found any up to date information on doing it. The last I saw was about two years ago and I'd prefer to be on a more recent driver.

If I can't get the 260 to work I think the next recourse is to just bit the bullet and run two monitors off the 260 and one off the IGP. From my understanding I can't have 3 monitors extended and able to pass applications back and forth with that setup. But, I should be able to use OVMF with the 970 I think. Do you know if the host gpu has to support OVMF as well or is it only the GPU that is getting passed through?

Thanks

Offline

#3892 2015-01-16 19:23:36

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do you know if the host gpu has to support OVMF as well or is it only the GPU that is getting passed through?

Only the GPU you're passing-through . OVMF has nothing to  do with your host GPUs .

Offline

#3893 2015-01-16 21:29:21

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried to switch GPUs on my mobo:

so, now we have

HD 6850, x16, blue
R9 270, x4, black

after reboot can't load to my kernel, it just stuck on
grub complaining about diskfilter writes (system's on lvm, just wait or press any key -> it goes away, and system boots as expected)

It doesn't matter if I removed pci-ids or not.

But I managed to boot in ARCH-kernel, and now I get in dmesg:

walkindude@Desktop> sudo dmesg | grep -i AMD-Vi                                                                                ~
[sudo] password for walkindude:
[    0.107871] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
[    0.107872] AMD-Vi:        mmio-addr: 00000000feb20000
[    0.107883] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:00.0 flags: 00
[    0.107885] AMD-Vi:   DEV_RANGE_END         devid: 00:00.2
[    0.107886] AMD-Vi:   DEV_SELECT             devid: 00:02.0 flags: 00
[    0.107887] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 01:00.0 flags: 00
[    0.107888] AMD-Vi:   DEV_RANGE_END         devid: 01:00.1
[    0.107889] AMD-Vi:   DEV_SELECT             devid: 00:11.0 flags: 00
[    0.107890] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:12.0 flags: 00
[    0.107891] AMD-Vi:   DEV_RANGE_END         devid: 00:12.2
[    0.107892] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:13.0 flags: 00
[    0.107893] AMD-Vi:   DEV_RANGE_END         devid: 00:13.2
[    0.107894] AMD-Vi:   DEV_SELECT             devid: 00:14.0 flags: d7
[    0.107895] AMD-Vi:   DEV_SELECT             devid: 00:14.3 flags: 00
[    0.107896] AMD-Vi:   DEV_SELECT             devid: 00:14.4 flags: 00
[    0.107898] AMD-Vi:   DEV_ALIAS_RANGE         devid: 02:00.0 flags: 00 devid_to: 00:14.4
[    0.107898] AMD-Vi:   DEV_RANGE_END         devid: 02:1f.7
[    0.107906] AMD-Vi:   DEV_SELECT             devid: 00:14.5 flags: 00
[    0.107907] AMD-Vi:   DEV_SELECT             devid: 00:15.0 flags: 00
[    0.107908] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 03:00.0 flags: 00
[    0.107909] AMD-Vi:   DEV_RANGE_END         devid: 03:00.1
[    0.107910] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:16.0 flags: 00
[    0.107911] AMD-Vi:   DEV_RANGE_END         devid: 00:16.2
[    0.107912] AMD-Vi:   DEV_SPECIAL(IOAPIC[0])        devid: 00:14.0
[    0.107914] AMD-Vi:   DEV_SPECIAL(HPET[0])        devid: 00:14.0
[    0.107915] AMD-Vi:   DEV_SPECIAL(IOAPIC[255])        devid: 00:00.1
[    2.908990] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    2.908992] AMD-Vi: Interrupt remapping enabled
[    2.909114] AMD-Vi: Initialized for Passthrough Mode
[    3.045906] AMD-Vi: Completion-Wait loop timed out
[    3.156248] AMD-Vi: Completion-Wait loop timed out
[    3.266509] AMD-Vi: Completion-Wait loop timed out
[    3.376798] AMD-Vi: Completion-Wait loop timed out
[    3.487084] AMD-Vi: Completion-Wait loop timed out
[    3.597349] AMD-Vi: Completion-Wait loop timed out
[    3.707628] AMD-Vi: Completion-Wait loop timed out
[    3.817917] AMD-Vi: Completion-Wait loop timed out
[    3.928191] AMD-Vi: Completion-Wait loop timed out
[    4.017795] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e88e0]
[    4.017805] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e8900]
[    4.017810] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e8930]

If I switch GPUs back,my kernel boots, but during this process I'm seing something strange (can't find it later in logs, so I shot it by my phone camera)
http://rghost.net/private/60392304/15d7 … e5413.view
These two line were present on all my attempts (on this and previous pages of current topic)

In dmesg after

AMD-Vi: Initialized for Passthrough Mode

nothing, no lines about

Completion-Wait loop timed out or

Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e8900]

UEFI bios on my mobo (M5A97 PRO) is the latest available: 1604. I'd flashed it a long time ago, during the installation of FX CPU.

What is wrong here? Should I buy the second revision or completely another AM3+ motherboard? or smth wrong in my settings?

Offline

#3894 2015-01-16 22:08:57

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Do you know if the host gpu has to support OVMF as well or is it only the GPU that is getting passed through?

Only the GPU you're passing-through . OVMF has nothing to  do with your host GPUs .

Thanks. I've reconfigured my rig now. My gtx 260 is in the primary pcie slot, gtx 970 in the second. I've got all the monitors working and extended with xrandr. Time to give OVMF a shot.

Offline

#3895 2015-01-16 22:58:41

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Hi everyone , long time no see .

Does anyone experience soft lockups of VMs once every couple of minutes ?

I always experience such lockups that lasts for a second or less in every Windows 8.1 VM I have . This is on my X99 host by the way .

CMD line if anyone would have an idea what's causing them :

qemu-system-x86_64 -name main -nographic \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -vga none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_main.bin \ -device vfio-pci,host=05:00.0,multifunction=on \ -device vfio-pci,host=05:00.1 \ -device vfio-pci,host=0f:00.0 \ -device vfio-pci,host=00:1b.0 \ -drive file=/VMs/Win_Main.img,cache=none,format=raw,if=none,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -localtime \ -net nic,model=virtio,macaddr=33:29:21:5F:C4:7D -net bridge,br=br0 \ -monitor unix:/tmp/vm_main,server,nowait &

I believe that heavy HDD activity causes it more often . These VMs are sitting on a 4 disk RAID0 . Disks are 7200 ones .

It's probably because you're using images instead of block devices (and if your fs is btrfs then can be even worse. Also, how much RAM did you left for your host?

Offline

#3896 2015-01-16 23:09:25

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

It's probably because you're using images instead of block devices (and if your fs is btrfs then can be even worse. Also, how much RAM did you left for your host?

I went the RAW images path because they're easy to manage , move around , duplicate and delete . Those soft lock-ups used to happen alot when I was using QCOW2 images , they are alot less now with RAW but not gone completely . The RAID0 which all of these images are is an MDRAID (RAID0) + EXT4 . About the ram , now the whole system is consuming 18GB out of 32GB total , so It's not about the RAM I guess .

Offline

#3897 2015-01-17 03:08:56

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Update: We have video! Using the 970 as the passthrough and OVMF I was able to get output to the screen. However, I'm unable to get the VM to boot from the windows iso. When trying it I see "Boot failed. EFI DVD/CDROM 1", it tries to pxe boot, then it kicks over to the UEFI Interactive shell v2.1.

Here is my qemu launch script

qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off -smp 2,cores=2,threads=1 \ -vga none \ -drive if=pflash,format=raw,file=/usr/share/ovmf/ovmf_x64.bin \ -device virtio-scsi-pci,id=scsi \ -device vfio-pci,host=06:00.0,multifunction=on \ -device vfio-pci,host=06:00.1 \ -drive file=/VMs/win.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/VMs/win8.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/VMs/virt.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -boot menu=on

Also, this is probably pretty stupid but I haven't found an answer anywhere else. How exactly are you guys using your keyboard and mouse in the guest? I saw a youtube video of someone using synergy so that seems like a good option, but other than plugging in a separate keyboard and passing it through as well I'm at a loss as to how I'm supposed to share the keyboard and mouse without them potentially getting stuck in the guest. Thanks for all of your help.

Offline

#3898 2015-01-17 08:30:33

zlice
Member
Registered: 2015-01-16
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so after messing around for quite some time. i ended up with an old 8600gts and my r290...

maybe it's an issue someone came across on this thread, but can someone tell me how my video output is from my passthrough but device manager/amd doesn't know it's an amd card?
changed my qemu command a few times, maybe i have to redo the VM.

Offline

#3899 2015-01-17 10:27:56

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:

Update: We have video! Using the 970 as the passthrough and OVMF I was able to get output to the screen. However, I'm unable to get the VM to boot from the windows iso. When trying it I see "Boot failed. EFI DVD/CDROM 1", it tries to pxe boot, then it kicks over to the UEFI Interactive shell v2.1.

Use this to assign the ISO to the VM :

-device virtio-scsi-pci,id=scsi \ -drive file=/VMs/win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \

Good luck !

Offline

#3900 2015-01-17 16:10:41

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:

How exactly are you guys using your keyboard and mouse in the guest? I saw a youtube video of someone using synergy so that seems like a good option, but other than plugging in a separate keyboard and passing it through as well I'm at a loss as to how I'm supposed to share the keyboard and mouse without them potentially getting stuck in the guest.

My setup is less common I assume, but I actually have essentially a dumb* USB switch setup (well, 2 of them actually), that I soldered together with some 4 Pole Double Throw (4PDT), On-Off-On switches and some short USB connectors and then have a separate PCIe USB 3 card passed through to the guest, giving me dedicated USB ports on the guest with separate USB connections going to the host and guest (via the USB 3 card) and then switch between them with the switches. You could also get some sort of basic USB switch if you were using something like a dedicated USB card, the kind of thing that's designed to share printers for instance (like an Aten US221A), though depending on the keyboard it might not pass through multimedia keys or other special keys (like the Roccat keyboard and mouse I used to use).

* When I say dumb, I mean that it's not like the Aten US221A that I mentioned which normally keeps the devices enumerated on each system whether connected to a particular system or not. In my setup it's dumb and actually results in disconnection of the electrical circuit, akin to unplugging and plugging the cable back in, also meaning that the keyboard and mouse have to re-enumerate each time you switch systems.

Offline

#3901 2015-01-17 17:55:54

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:

How exactly are you guys using your keyboard and mouse in the guest? I saw a youtube video of someone using synergy so that seems like a good option, but other than plugging in a separate keyboard and passing it through as well I'm at a loss as to how I'm supposed to share the keyboard and mouse without them potentially getting stuck in the guest.

Don't know if will help but in my case, if I know that I will not use Linux when I'm running the Guest (Windows), I pass the USB for my Logitech keyboard and mouse and works like a charm. As soon as I turn off the Windows VM then I recover my keyboard and mouse on Linux.

In case that I need to go to my Linux host, then I have ssh and in the worst case, Teamviewer and both works fine from the Windows setup.

Sometimes I'm on Linux and my wife is connecting to the Windows machine from the iPad too watch movies / play simple games and then I'm running same script but without passing the USB for keyboard and mouse so I have full control on Linux and she is connecting with Teamviewer to the Windows computer.

Working with Photoshop I connect from the Linux computer using rdesktop that works pretty cool.

By the way, I know some people uses synergy but I tried myself and didn't get as good results as passing the USB directly (0 delay smile)

Hope it helps for you.

Best regards,



TheArcher

Last edited by thearcherblog (2015-01-17 17:56:17)

Offline

#3902 2015-01-17 18:47:15

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
tully wrote:

Update: We have video! Using the 970 as the passthrough and OVMF I was able to get output to the screen. However, I'm unable to get the VM to boot from the windows iso. When trying it I see "Boot failed. EFI DVD/CDROM 1", it tries to pxe boot, then it kicks over to the UEFI Interactive shell v2.1.

Use this to assign the ISO to the VM :

-device virtio-scsi-pci,id=scsi \ -drive file=/VMs/win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \

Good luck !

I managed to boot an arch cd using that. Thanks. Just need to figure out why my windows 8 iso is refusing to boot.

Thanks Myranti and archer for your usb help. For now I'm using

"-vga none -device secondary-vga"

This just gives me a black box that will capture input until I ctrl+alt. So it's a good temp solution for now. Will explore both options to find a suitable method once I've got everything stable.

Offline

#3903 2015-01-17 23:50:15

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Okay, so i got everything working (i forgot kvm=off in the cpu parameter last).

The only problem now seems to be that i can't reboot the VM. vfio complains that the bios is corrupted. I'm guessing that's because the windows VM shuts the GFX card off. What can i do about this?

To reiterate: It's a GTX 460



thearcherblog wrote:
tully wrote:

How exactly are you guys using your keyboard and mouse in the guest? I saw a youtube video of someone using synergy so that seems like a good option, but other than plugging in a separate keyboard and passing it through as well I'm at a loss as to how I'm supposed to share the keyboard and mouse without them potentially getting stuck in the guest.

Don't know if will help but in my case, if I know that I will not use Linux when I'm running the Guest (Windows), I pass the USB for my Logitech keyboard and mouse and works like a charm. As soon as I turn off the Windows VM then I recover my keyboard and mouse on Linux.

In case that I need to go to my Linux host, then I have ssh and in the worst case, Teamviewer and both works fine from the Windows setup.

Sometimes I'm on Linux and my wife is connecting to the Windows machine from the iPad too watch movies / play simple games and then I'm running same script but without passing the USB for keyboard and mouse so I have full control on Linux and she is connecting with Teamviewer to the Windows computer.

Working with Photoshop I connect from the Linux computer using rdesktop that works pretty cool.

By the way, I know some people uses synergy but I tried myself and didn't get as good results as passing the USB directly (0 delay smile)

Hope it helps for you.

Best regards,

TheArcher

I'm using synergy, and it's working well. I'm using a bridged connection with netctl, and connecting with the hostnames. No noticeable delay anywhere in the system. It's great because it allows me to have one monitor windows and the other linux (I disable one monitor on linux with xrandr in my VM startup script).

Last edited by DelusionalLogic (2015-01-17 23:54:40)

Offline

#3904 2015-01-18 07:07:19

tully
Member
Registered: 2015-01-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Good news everyone! I'm up and running. I wasn't able to get the windows 8 iso to load. I managed to get the windows 10 preview to load and that works just fine. I tried a few windows 8 isos so no idea what that was about. Anyway, I used a gtx 970 FTW with a i7 4970k. Nvidia drivers worked without a hitch, gpu shows up just fine in dev manager, and I have no problem rebooting the machine. I ended up using OVMF as the older method wasn't cutting it for me. I also used the kernel OP provided although I'm not exactly sure how much of the patches were needed.

Here's some code

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="pcie_acs_override=downstream i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb"

and my script to start the vm

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off -smp 4,cores=4,threads=1 \ -pflash /usr/share/ovmf/x64/ovmf_x64.bin \ -vga none \ -device vfio-pci,host=06:00.0,multifunction=on \ -device vfio-pci,host=06:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/tully/VMs/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \ -drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/home/tully/VMs/virt.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -boot menu=on

I'm writing directly to a 300GB velociraptor that I had lying around. I'm not exactly sure the % hit I took from using kvm but it's negligible. Insurgency was at 180fps maxed out and league was locked at 60fps and never dipped down once. I'll try out some more demanding games in a few days when I can but for now it looks to be performing wonderfully. Synergy doesn't exactly do well with certain games, but it does well in others. I'll just need to find out the quirks and work around them. Thanks everyone for helping me get this up and running.

Offline

#3905 2015-01-18 20:53:54

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tully wrote:

Good news everyone! I'm up and running. I wasn't able to get the windows 8 iso to load. I managed to get the windows 10 preview to load and that works just fine. I tried a few windows 8 isos so no idea what that was about. Anyway, I used a gtx 970 FTW with a i7 4970k. Nvidia drivers worked without a hitch, gpu shows up just fine in dev manager, and I have no problem rebooting the machine. I ended up using OVMF as the older method wasn't cutting it for me. I also used the kernel OP provided although I'm not exactly sure how much of the patches were needed.

Here's some code

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="pcie_acs_override=downstream i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb"

and my script to start the vm

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off -smp 4,cores=4,threads=1 \ -pflash /usr/share/ovmf/x64/ovmf_x64.bin \ -vga none \ -device vfio-pci,host=06:00.0,multifunction=on \ -device vfio-pci,host=06:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/tully/VMs/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \ -drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/home/tully/VMs/virt.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -boot menu=on

I'm writing directly to a 300GB velociraptor that I had lying around. I'm not exactly sure the % hit I took from using kvm but it's negligible. Insurgency was at 180fps maxed out and league was locked at 60fps and never dipped down once. I'll try out some more demanding games in a few days when I can but for now it looks to be performing wonderfully. Synergy doesn't exactly do well with certain games, but it does well in others. I'll just need to find out the quirks and work around them. Thanks everyone for helping me get this up and running.


Firstly, How did you get past the interactive shell? Even with a parameter-list that looks functionally identical to yours i can't get it past the PXE boot.

Secondly. What i was doing (I'm trying OVMF now) was passing the usb keyboard/mouse through, but then running a synergy server ON THE VM, with a client on the host. so the vm sees them as native input devices, and the linux host sees them as synergy virtual stuff.

EDIT: Figured it out. If you want to fix it you need to get a windows8 iso that support UEFI (most iso's seem to have that stripped).

Last edited by DelusionalLogic (2015-01-18 23:06:09)

Offline

#3906 2015-01-18 22:01:29

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

Offline

#3907 2015-01-19 11:40:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kainet wrote:

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

Technically 770 is 680 with overclocked memory, so there shouldn't be major differences between those cards.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#3908 2015-01-19 12:53:41

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kainet wrote:

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

I have had same problems on 660ti, 680 and 770... (560ti works as i remember), after some time (random 1-60mins) sound would go crazy, like it's lagging (slowed down) or something... I couldn't get rid of it (changed mbo and cpu=same reslut), so i bought usb sound card.

Offline

#3909 2015-01-19 13:44:25

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
kainet wrote:

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

I have had same problems on 660ti, 680 and 770... (560ti works as i remember), after some time (random 1-60mins) sound would go crazy, like it's lagging (slowed down) or something... I couldn't get rid of it (changed mbo and cpu=same reslut), so i bought usb sound card.

With 670 i solved this by such a way. For me sound goes crazy before 10 min after start. If that did not happened then i have normal sound till i shut down VM. Otherwise i shutdown (restart in VM not helps) VM and start it again, usually it takes 2-5 restarts to achieve stable sound smile With 770 sound seems is normal but i start vm only few times.

Offline

#3910 2015-01-19 15:43:19

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wish i could suspend the graphics card when it's not in use without keeping the vm on at all times.

I've been unbinding the vfio-pci driver when i stop the vm, which does make sysfs report the device as suspended. The problem is that the graphics card still has the fan on at full power, because it's recieving juice. Can i somehow properly suspend the device?

EDIT: I can't seem to get libvirt to boot the VM correctly either. Is there any catch to running the VM with libvirt compared to qemu?

Last edited by DelusionalLogic (2015-01-19 19:05:18)

Offline

#3911 2015-01-19 18:42:22

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

journalctl -f:

Jan 19 21:35:37 localhost sudo[20006]: : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=./vfio-bind 0000:01:00.0 0000:01:00.1 Jan 19 21:35:37 localhost sudo[20022]: : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=/usr/bin/losetup /dev/loop1 bootL.mgr Jan 19 21:35:37 localhost sudo[20025]: : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=/usr/bin/mdadm --build /dev/md1 --level=linear --force --raid-devices=2 /dev/loop1 /dev/sda2 Jan 19 21:35:37 localhost sudo[20028]: : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=/usr/bin/qemu-system-x86_64 -M q35 -m 6148 -enable-kvm -cpu host -boot c -vga none -display none -monitor telnet:10.10.10.92:4441,server,nowait -smp 4,sockets=1,cores=2,threads=2 -device ahci,bus=pcie.0,id=ahci -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -device vfio-pci,host=01:00.1,bus=pcie.0 -drive file=/dev/md1,id=dataDiskL,format=raw,if=none -device ide-hd,drive=dataDiskL -net nic,macaddr=52:54:00:12:34:56 -net bridge,br=br0 Jan 19 21:35:37 localhost kernel: device tap0 entered promiscuous mode Jan 19 21:35:37 localhost kernel: br0: port 2(tap0) entered forwarding state Jan 19 21:35:37 localhost kernel: br0: port 2(tap0) entered forwarding state Jan 19 21:35:37 localhost kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Jan 19 21:35:38 localhost ntpd[1076]: bind(31) AF_INET6 fe80::fc43:e2ff:fefd:e905%92%92#123 flags 0x11 failed: Cannot assign requested address Jan 19 21:35:38 localhost ntpd[1076]: unable to create socket on tap0 (88) for fe80::fc43:e2ff:fefd:e905%92#123 Jan 19 21:35:38 localhost ntpd[1076]: failed to init interface for address fe80::fc43:e2ff:fefd:e905%92 Jan 19 21:35:40 localhost ntpd[1076]: new interface(s) found: waking up resolver Jan 19 21:35:52 localhost kernel: br0: port 2(tap0) entered forwarding state

dmesg:

[2087126.096421] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [2087140.619697] br0: port 2(tap0) entered forwarding state [2087205.484345] br0: port 2(tap0) entered disabled state [2087205.484415] device tap0 left promiscuous mode [2087205.484432] br0: port 2(tap0) entered disabled state [2087307.712013] device tap0 entered promiscuous mode [2087307.712105] br0: port 2(tap0) entered forwarding state [2087307.712113] br0: port 2(tap0) entered forwarding state [2087308.213505] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [2087322.757144] br0: port 2(tap0) entered forwarding state

Execution:

sudo qemu-system-x86_64 -M q35 -m 6148 -enable-kvm -cpu host -boot c -vga none -display none -monitor telnet:10.10.10.92:4441,server,nowait \ -smp 4,sockets=1,cores=2,threads=2 \ -device ahci,bus=pcie.0,id=ahci \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \ -device vfio-pci,host=01:00.1,bus=pcie.0 \ -drive file=/dev/md1,id=dataDiskL,format=raw,if=none -device ide-hd,drive=dataDiskL \ -net nic,macaddr=52:54:00:12:34:56 -net bridge,br=br0

running without vga=none:

dmesg:

[2087884.573891] kvm: zapping shadow pages for mmio generation wraparound [2087893.724076] kvm [24007]: vcpu0 unhandled rdmsr: 0x1c9 [2087893.724565] kvm [24007]: vcpu0 unhandled rdmsr: 0x1a6 [2087893.725027] kvm [24007]: vcpu0 unhandled rdmsr: 0x1a7 [2087893.725463] kvm [24007]: vcpu0 unhandled rdmsr: 0x3f6 [2087894.981414] kvm [24007]: vcpu0 unhandled rdmsr: 0x606 [2087895.836766] br0: port 2(tap0) entered forwarding state [2087896.384502] kvm [24007]: vcpu0 unhandled rdmsr: 0x611 [2087896.384932] kvm [24007]: vcpu0 unhandled rdmsr: 0x639 [2087896.385350] kvm [24007]: vcpu0 unhandled rdmsr: 0x641 [2087896.385764] kvm [24007]: vcpu0 unhandled rdmsr: 0x619 [2087896.397773] vfio-pci 0000:01:00.1: irq 30 for MSI/MSI-X [2087896.405814] kvm [24007]: vcpu0 unhandled rdmsr: 0x611 [2087904.772580] vfio-pci 0000:01:00.0: irq 33 for MSI/MSI-X

Last edited by myweb (2015-01-19 18:47:54)

Offline

#3912 2015-01-19 18:47:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3913 2015-01-19 18:50:26

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
myweb wrote:

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

Yes, I do not see the reason in logs, monitor is without signal and Guest is not available via network.
Could you please specify which log I should take a look?

Without vga=none - guest booted.

qemu-system-x86_64 --version QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

uname -a

Linux localhost 3.18.0-1-mainline #1 SMP PREEMPT Wed Dec 10 19:09:20 FET 2014 x86_64 GNU/Linux

qemu 2.2.0-1
seabios 1.7.5-2

Last edited by myweb (2015-01-19 18:52:56)

Offline

#3914 2015-01-19 18:51:31

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I fixed the sound problem by switching the hdmi audio of my gtx 660 to use MSI.

Offline

#3915 2015-01-19 19:01:38

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:
myweb wrote:

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

Yes, I do not see the reason in logs, monitor is without signal and Guest is not available via network.
Could you please specify which log I should take a look?

Without vga=none - guest booted.

qemu-system-x86_64 --version QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

uname -a

Linux localhost 3.18.0-1-mainline #1 SMP PREEMPT Wed Dec 10 19:09:20 FET 2014 x86_64 GNU/Linux

qemu 2.2.0-1
seabios 1.7.5-2

Without vga=none I am able to boot and use my Nvidia GT64 card with legacy 403 driver. But with new 346.45 I got:

nvidia-smi -q

Unable to determine the device handle for GPU 0000:01:00.0: Unknown Error

dmesg

[ 11.797417] nvidia 0000:01:00.0: irq 28 for MSI/MSI-X [ 11.799246] NVRM: RmInitAdapter failed! (0x23:0x56:606) [ 11.799251] NVRM: rm_init_adapter failed for device bearing minor number 0 [ 11.799381] NVRM: nvidia_frontend_open: minor 0, module->open() failed, error -5

Xorg

[ 11.283] (**) NVIDIA(0): ConnectedMonitor string: "DFP-1" [ 11.285] (EE) NVIDIA(GPU-0): The NVIDIA GPU at PCI:1:0:0 is not supported by the 346.35 [ 11.285] (EE) NVIDIA(GPU-0): NVIDIA driver. [ 11.285] (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA graphics device! [ 11.285] (EE) NVIDIA(0): Failing initialization of X screen 0

Offline

#3916 2015-01-19 19:11:01

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

I fixed the sound problem by switching the hdmi audio of my gtx 660 to use MSI.

Could you please explain this. Not fully understand you.

Offline

#3917 2015-01-19 19:13:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:
myweb wrote:

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

Yes, I do not see the reason in logs, monitor is without signal and Guest is not available via network.
Could you please specify which log I should take a look?

Without vga=none - guest booted.

qemu-system-x86_64 --version QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

uname -a

Linux localhost 3.18.0-1-mainline #1 SMP PREEMPT Wed Dec 10 19:09:20 FET 2014 x86_64 GNU/Linux

qemu 2.2.0-1
seabios 1.7.5-2

Sounds like a VGA arbitration problem... if only we knew your hardware config and had some idea what the last version was that worked, someone might be able to help.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3918 2015-01-19 20:14:27

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to get libvirt to boot my vm. I have it working in pure Qemu from the commandline. I'm using OVMF with a 660ti.

For some reason when i boot through libvirt the nvidia driver fails (in the same way that it did before i started using OVMF, exception code 0x40000015). It doesn't crash, the GPU just doesn't seem to be passed through properly. I have no idea what's causing it, although i suspect it's not booting OVMF/EFI.

Here's my current config:

<domain type='kvm'> <name>OVMF-Gaming</name> <uuid>523502ec-9ff4-46de-9935-6582c468dc25</uuid> <memory unit='KiB'>8392704</memory> <currentMemory unit='KiB'>8392704</currentMemory> <vcpu placement='static'>6</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdb'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:f1:a0:4b'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <sound model='ac97'> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </sound> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> </domain>

Offline

#3919 2015-01-19 20:18:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>

    <timer name='hypervclock' present='yes'/>

The above lines need to be removed for nvidia


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3920 2015-01-19 20:28:34

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Sounds like a VGA arbitration problem... if only we knew your hardware config and had some idea what the last version was that worked, someone might be able to help.

The last working kernel for me was 3.15
qemu-git-2.0.r32368.ge2da502-1-x86_64.pkg.tar.xz

My configuration:
i5-3470
AsRock Z77 pro4
Nvidia GT640 (01:00.0) - Arch Linux - vga=none does not work
ATI HD 7750 (02:00.0) - Windows 8 guest - vga=none works

Curent kernel command line:

Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=b1f615cd-86ea-4c6a-ba12-b6e0e4b2f3a4 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:0fc1,10de:0e1b,1002:683f,1002:aab0 i915.enable_hd_vgaarb=1 quiet systemd.unit=multi-user.target

Last edited by myweb (2015-01-19 20:30:41)

Offline

#3921 2015-01-19 20:31:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:

Sounds like a VGA arbitration problem... if only we knew your hardware config and had some idea what the last version was that worked, someone might be able to help.

The last working kernel for me was 3.15
qemu-git-2.0.r32368.ge2da502-1-x86_64.pkg.tar.xz

My configuration:
i5-3470
AsRock Z77 pro4
Nvidia GT640 (01:00.0) - Arch Linux - vga=none does not work
ATI HD 7750 (02:00.0) - Windows 8 guest - vga=none works

Are you using the i5 Intel graphics for the host?  Did you apply the i915 patch to your kernel?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3922 2015-01-19 20:35:02

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Are you using the i5 Intel graphics for the host?  Did you apply the i915 patch to your kernel?

Yes, I am using i5 Intel graphics for the host.
i915_317.patch patch was used in linux-mainline-3.18.0.tar.gz (3.18.0 + acs override patch + i915 vga arbiter fixes)

I added the following kernel option:i915.enable_hd_vgaarb=1 and now I see in logs:

Jan 19 23:40:56 localhost kernel: device tap1 entered promiscuous mode Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state Jan 19 23:40:57 localhost ntpd[1641]: bind(32) AF_INET6 fe80::fc1e:56ff:fe2b:fdb2%13%13#123 flags 0x11 failed: Cannot assign requested address Jan 19 23:40:57 localhost ntpd[1641]: unable to create socket on tap1 (17) for fe80::fc1e:56ff:fe2b:fdb2%13#123 Jan 19 23:40:57 localhost ntpd[1641]: failed to init interface for address fe80::fc1e:56ff:fe2b:fdb2%13 Jan 19 23:40:57 localhost kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Jan 19 23:40:59 localhost ntpd[1641]: Listen normally on 18 tap1 [fe80::fc1e:56ff:fe2b:fdb2%13]:123 Jan 19 23:40:59 localhost ntpd[1641]: new interface(s) found: waking up resolver Jan 19 23:41:01 localhost kernel: kvm: zapping shadow pages for mmio generation wraparound Jan 19 23:41:11 localhost kernel: br0: port 3(tap1) entered forwarding state Jan 19 23:41:13 localhost kernel: kvm_get_msr_common: 3 callbacks suppressed Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1c9 Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a6 Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a7 Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x3f6 Jan 19 23:41:14 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x606 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x639 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x641 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x619 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611 Jan 19 23:41:16 localhost kernel: vfio-pci 0000:01:00.1: irq 33 for MSI/MSI-X

But display is still black and guest does not boot

Last edited by myweb (2015-01-19 20:45:32)

Offline

#3923 2015-01-19 20:49:33

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
DelusionalLogic wrote:

    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>

    <timer name='hypervclock' present='yes'/>

The above lines need to be removed for nvidia

Fantastic! it's working perfectly now. Thank you.

Screw you asus, linux wasn't the problem, your shoddy IOMMU table was. If anyone still cares (after so long), the Asus P6T deluxe V2 board has a broken IOMMU bios. The fix is as easy as dumping the table, decompiling it then changing a single hex char and recompiling. Inserting it into the initramfs was a breeze since the relevant compile flag is already enabled on arch-mainline.

The functionality that enables this (boot time overriding of ACPI tables) is https://www.kernel.org/doc/Documentatio … erride.txt
The only problem with it is that i have to run my injection script every time i run mkinitcpio, but that's a small price to pay.

Offline

#3924 2015-01-19 20:58:28

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
aw wrote:

Are you using the i5 Intel graphics for the host?  Did you apply the i915 patch to your kernel?

Yes, I am using i5 Intel graphics for the host.
i915_317.patch patch was used in linux-mainline-3.18.0.tar.gz (3.18.0 + acs override patch + i915 vga arbiter fixes)

I added the following kernel option:i915.enable_hd_vgaarb=1 and now I see in logs:

Jan 19 23:40:56 localhost kernel: device tap1 entered promiscuous mode Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state Jan 19 23:40:57 localhost ntpd[1641]: bind(32) AF_INET6 fe80::fc1e:56ff:fe2b:fdb2%13%13#123 flags 0x11 failed: Cannot assign requested address Jan 19 23:40:57 localhost ntpd[1641]: unable to create socket on tap1 (17) for fe80::fc1e:56ff:fe2b:fdb2%13#123 Jan 19 23:40:57 localhost ntpd[1641]: failed to init interface for address fe80::fc1e:56ff:fe2b:fdb2%13 Jan 19 23:40:57 localhost kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Jan 19 23:40:59 localhost ntpd[1641]: Listen normally on 18 tap1 [fe80::fc1e:56ff:fe2b:fdb2%13]:123 Jan 19 23:40:59 localhost ntpd[1641]: new interface(s) found: waking up resolver Jan 19 23:41:01 localhost kernel: kvm: zapping shadow pages for mmio generation wraparound Jan 19 23:41:11 localhost kernel: br0: port 3(tap1) entered forwarding state Jan 19 23:41:13 localhost kernel: kvm_get_msr_common: 3 callbacks suppressed Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1c9 Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a6 Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a7 Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x3f6 Jan 19 23:41:14 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x606 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x639 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x641 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x619 Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611 Jan 19 23:41:16 localhost kernel: vfio-pci 0000:01:00.1: irq 33 for MSI/MSI-X

But display is still black and guest does not boot

I added i915 module to black list and now I see the following information in dmesg:

[ 41.665779] br0: port 2(tap0) entered forwarding state [ 41.665790] br0: port 2(tap0) entered forwarding state [ 42.167464] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 42.167597] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 42.171227] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002) [ 45.893325] kvm: zapping shadow pages for mmio generation wraparound [ 56.689435] br0: port 2(tap0) entered forwarding state [ 57.805957] kvm [1248]: vcpu0 unhandled rdmsr: 0x1c9 [ 57.807847] kvm [1248]: vcpu0 unhandled rdmsr: 0x1a6 [ 57.809960] kvm [1248]: vcpu0 unhandled rdmsr: 0x1a7 [ 57.812242] kvm [1248]: vcpu0 unhandled rdmsr: 0x3f6 [ 58.991381] kvm [1248]: vcpu0 unhandled rdmsr: 0x606 [ 60.451997] kvm [1248]: vcpu0 unhandled rdmsr: 0x611 [ 60.453933] kvm [1248]: vcpu0 unhandled rdmsr: 0x639 [ 60.455845] kvm [1248]: vcpu0 unhandled rdmsr: 0x641 [ 60.457879] kvm [1248]: vcpu0 unhandled rdmsr: 0x619 [ 60.740637] vfio-pci 0000:01:00.1: irq 31 for MSI/MSI-X [ 74.566563] saa7146: interrupt_hw: warning: interrupt enabled, but not handled properly.(0xe7fcfbff) [ 74.569760] saa7146: interrupt_hw: disabling interrupt source(s)! [ 100.167145] saa7146: saa7146 (0): unexpected i2c irq: isr e7fffbff psr 00040420 ssr 00810000 [ 100.167149] saa7146: interrupt_hw: warning: interrupt enabled, but not handled properly.(0xe7fcfbff) [ 100.171143] saa7146: interrupt_hw: disabling interrupt source(s)! [ 118.367566] saa7146: saa7146 (0): unexpected i2c irq: isr e7fffbff psr 00040420 ssr 00810000 [ 118.367570] saa7146: interrupt_hw: warning: interrupt enabled, but not handled properly.(0xe7fcfbff) [ 118.371014] saa7146: interrupt_hw: disabling interrupt source(s)!

old nvidia driver without vga=none works fine:
nvidia-smi -q

==============NVSMI LOG============== Timestamp : Tue Jan 20 00:15:22 2015 Driver Version : 304.125 Attached GPUs : 1 GPU 0000:01:00.0 Product Name : GeForce GT 640 Display Mode : N/A Persistence Mode : Disabled Driver Model Current : N/A Pending : N/A Serial Number : N/A GPU UUID : GPU-ae36e4e0-8e34-e7d2-e571-159b9bfe3020 VBIOS Version : 80.07.26.00.5B Inforom Version Image Version : N/A OEM Object : N/A ECC Object : N/A Power Management Object : N/A GPU Operation Mode Current : N/A Pending : N/A PCI Bus : 0x01 Device : 0x00 Domain : 0x0000 Device Id : 0x0FC110DE Bus Id : 0000:01:00.0 Sub System Id : 0x840E1043 GPU Link Info PCIe Generation Max : N/A Current : N/A Link Width Max : N/A Current : N/A Fan Speed : 37 % Performance State : N/A Clocks Throttle Reasons : N/A Memory Usage Total : 1023 MB Used : 77 MB Free : 946 MB Compute Mode : Default Utilization Gpu : N/A Memory : N/A Ecc Mode Current : N/A Pending : N/A ECC Errors Volatile Single Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Total : N/A Double Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Total : N/A Aggregate Single Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Total : N/A Double Bit Device Memory : N/A Register File : N/A L1 Cache : N/A L2 Cache : N/A Texture Memory : N/A Total : N/A Temperature Gpu : 44 C Power Readings Power Management : N/A Power Draw : N/A Power Limit : N/A Default Power Limit : N/A Min Power Limit : N/A Max Power Limit : N/A Clocks Graphics : N/A SM : N/A Memory : N/A Applications Clocks Graphics : N/A Memory : N/A Max Clocks Graphics : N/A SM : N/A Memory : N/A Compute Processes : N/A

Last edited by myweb (2015-01-19 21:20:00)

Offline

#3925 2015-01-19 21:07:35

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Am i right in assuming i can't enable Spice at all on nividia passthrough? I'd love to hotplug usb devices to the VM.

Offline

#3926 2015-01-19 21:17:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

Am i right in assuming i can't enable Spice at all on nividia passthrough? I'd love to hotplug usb devices to the VM.

Why do you need spice to have hotplug usb devices?  Do you mean hotplug usb from remote clients?  AFAIK, spice requires adding a QXL video head.  If you're using Win8+/OVMF, then that's possible and you can even have the GPU render into the QXL display.  However, you might as well be doing software 3D with the speed you'll get.  You can also disable the QXL display in the guest, I've done this to experiment with piping the audio back to the host over spice.  I don't see why USB wouldn't work the same.  However, I wonder what the value is of doing USB passthrough via spice when you're sitting within a monitor cable length of the host and therefore able to just hotplug usb via virt-manager.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3927 2015-01-19 21:28:17

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Why do you need spice to have hotplug usb devices?  Do you mean hotplug usb from remote clients?  AFAIK, spice requires adding a QXL video head.  If you're using Win8+/OVMF, then that's possible and you can even have the GPU render into the QXL display.  However, you might as well be doing software 3D with the speed you'll get.  You can also disable the QXL display in the guest, I've done this to experiment with piping the audio back to the host over spice.  I don't see why USB wouldn't work the same.  However, I wonder what the value is of doing USB passthrough via spice when you're sitting within a monitor cable length of the host and therefore able to just hotplug usb via virt-manager.

I was under the impression that you would have to reboot the VM to attach a new USB device to it. Is that wrong? If so, then how would i go about hotplugging passthrough devices into the vm. It's all on my local qemu:///system.

I tried virsh attach-device usb-dev.xml (with a correctly formatted xml file specifying my phone vendor and product id) sadly that had no effect on my vm.

I see no option in virt-manager to do this.

EDIT: Okay, so it works fine with virsh attach-device (the problem was that qualcomm sneakily changes product id every time you switch mode, i was using the wrong id). I'd still love to know how this is done in virt-manager.

Last edited by DelusionalLogic (2015-01-19 21:33:51)

Offline

#3928 2015-01-19 21:32:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

I was under the impression that you would have to reboot the VM to attach a new USB device to it. Is that wrong? If so, then how would i go about hotplugging passthrough devices into the vm. It's all on my local qemu:///system.

I tried virsh attach-device usb-dev.xml (with a correctly formatted xml file specifying my phone vendor and product id) sadly that had no effect on my vm.

I see no option in virt-manager to do this.

Hotplug USB should work w/o spice.

View -> Details

Add Hardware -> USB Host device -> [select] -> Finish


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3929 2015-01-19 21:39:53

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Hotplug USB should work w/o spice.

View -> Details

Add Hardware -> USB Host device -> [select] -> Finish

Ahh, the problem here is that virt-manager is as dumb as me. It isn't updating the product id of the device, so it's trying to add 0x6765 when the correct id is 0x6764. In static applications (such as a controller) that shouldn't be an issue. thanks.

Offline

#3930 2015-01-19 21:49:23

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

With i915 is blacklisted I got host system hanged when run two Guests

Offline

#3931 2015-01-19 23:51:03

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw, you mentioned audio and spice. Is that the only way to get audio back to the host and out through the same speakers?

I'm trying to do it right now, and the ICH6 driver is working (i can see it in windows) but nothing is playing from the host, and pulse isn't detecting a source. I would really prefer not to configure pulse as a system wide service.

Last edited by DelusionalLogic (2015-01-20 00:29:02)

Offline

#3932 2015-01-20 12:29:34

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Help
I am trying to install Win 7 via ovmf / uefi. Win 8 works nicely via ovmf in Libvirt / virt-manager but Win 7 does not
The machine is running Ubuntu 14.04 Trusty LTS, patched to support KVM VMs with VGA passthrough - which has been working well for about a year. And a Win8 uefi VM was easy to install. But I want Win 7 smile
I found NBH reference to success at https://bbs.archlinux.org/viewtopic.php … 0#p1483550 and I tried several variations of that method (starting with a Win 7 image which is known to work for BIOS install)
ie. create a new image including Virtio and QXL drivers using wintoolkit, create a Fat32 partition on a GPT disk, copy the image files to the new partition and then start the VM for install

Somehow I always wind up with BCD issues .. or worse eg.

File: \EFI\Microsoft\Boot\BCD Status: 0x0000178 Info: An error occurred while attempting to read the boot configuration data

I've looked at the BCD using bcd-edit and cannot see anything obviously wrong ... but then I'm an absolute neophyte so who knows

The EFI shell is available ..... but I cannot find any useful commands and executing \efi\boot\bootx64.efi just takes me back to the same error - as you might expect

Can anyone advise a way forward ?

Offline

#3933 2015-01-20 18:42:03

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just noticed something, and i can't be the only one.

I'm getting really screwy screen updates on the host. If i switch between pages in my PDF reader the display will be one input behind (if i pgup and then pgdown it will go up when i click down, but then after a couple of seconds it goes down). In vim i'm getting 2 lines highlighted at once, and in firefox sometimes my keypresses does the same as the PDF reader (so i write hello, and the h appears when i press e, then the e when i press l and so on).

It seems to have something to do with the VM running, because it doesn't really happen when it's off. I really wish i could turn off the card (at least the fan, the card does seem to go into D3) when i don't need it.

Offline

#3934 2015-01-20 18:57:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

I just noticed something, and i can't be the only one.

I'm getting really screwy screen updates on the host.[snip]

Are you using synergy?  I had some strange mouse behavior in chrome that was resolved by using the upstream (paid) version of synergy rather than the distro version.  Otherwise I don't notice anything like you're describing with a VM running.  I also don't try to share mouse/keyboard except through synergy.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3935 2015-01-20 19:20:54

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

Tried to switch GPUs on my mobo:

so, now we have

HD 6850, x16, blue
R9 270, x4, black

after reboot can't load to my kernel, it just stuck on
grub complaining about diskfilter writes (system's on lvm, just wait or press any key -> it goes away, and system boots as expected)

It doesn't matter if I removed pci-ids or not.

But I managed to boot in ARCH-kernel, and now I get in dmesg:

walkindude@Desktop> sudo dmesg | grep -i AMD-Vi                                                                                ~
[sudo] password for walkindude:
[    0.107871] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
[    0.107872] AMD-Vi:        mmio-addr: 00000000feb20000
[    0.107883] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:00.0 flags: 00
[    0.107885] AMD-Vi:   DEV_RANGE_END         devid: 00:00.2
[    0.107886] AMD-Vi:   DEV_SELECT             devid: 00:02.0 flags: 00
[    0.107887] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 01:00.0 flags: 00
[    0.107888] AMD-Vi:   DEV_RANGE_END         devid: 01:00.1
[    0.107889] AMD-Vi:   DEV_SELECT             devid: 00:11.0 flags: 00
[    0.107890] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:12.0 flags: 00
[    0.107891] AMD-Vi:   DEV_RANGE_END         devid: 00:12.2
[    0.107892] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:13.0 flags: 00
[    0.107893] AMD-Vi:   DEV_RANGE_END         devid: 00:13.2
[    0.107894] AMD-Vi:   DEV_SELECT             devid: 00:14.0 flags: d7
[    0.107895] AMD-Vi:   DEV_SELECT             devid: 00:14.3 flags: 00
[    0.107896] AMD-Vi:   DEV_SELECT             devid: 00:14.4 flags: 00
[    0.107898] AMD-Vi:   DEV_ALIAS_RANGE         devid: 02:00.0 flags: 00 devid_to: 00:14.4
[    0.107898] AMD-Vi:   DEV_RANGE_END         devid: 02:1f.7
[    0.107906] AMD-Vi:   DEV_SELECT             devid: 00:14.5 flags: 00
[    0.107907] AMD-Vi:   DEV_SELECT             devid: 00:15.0 flags: 00
[    0.107908] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 03:00.0 flags: 00
[    0.107909] AMD-Vi:   DEV_RANGE_END         devid: 03:00.1
[    0.107910] AMD-Vi:   DEV_SELECT_RANGE_START     devid: 00:16.0 flags: 00
[    0.107911] AMD-Vi:   DEV_RANGE_END         devid: 00:16.2
[    0.107912] AMD-Vi:   DEV_SPECIAL(IOAPIC[0])        devid: 00:14.0
[    0.107914] AMD-Vi:   DEV_SPECIAL(HPET[0])        devid: 00:14.0
[    0.107915] AMD-Vi:   DEV_SPECIAL(IOAPIC[255])        devid: 00:00.1
[    2.908990] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    2.908992] AMD-Vi: Interrupt remapping enabled
[    2.909114] AMD-Vi: Initialized for Passthrough Mode
[    3.045906] AMD-Vi: Completion-Wait loop timed out
[    3.156248] AMD-Vi: Completion-Wait loop timed out
[    3.266509] AMD-Vi: Completion-Wait loop timed out
[    3.376798] AMD-Vi: Completion-Wait loop timed out
[    3.487084] AMD-Vi: Completion-Wait loop timed out
[    3.597349] AMD-Vi: Completion-Wait loop timed out
[    3.707628] AMD-Vi: Completion-Wait loop timed out
[    3.817917] AMD-Vi: Completion-Wait loop timed out
[    3.928191] AMD-Vi: Completion-Wait loop timed out
[    4.017795] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e88e0]
[    4.017805] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e8900]
[    4.017810] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e8930]

If I switch GPUs back,my kernel boots, but during this process I'm seing something strange (can't find it later in logs, so I shot it by my phone camera)
http://rghost.net/private/60392304/15d7 … e5413.view
These two line were present on all my attempts (on this and previous pages of current topic)

In dmesg after

AMD-Vi: Initialized for Passthrough Mode

nothing, no lines about

Completion-Wait loop timed out or

Event logged [IOTLB_INV_TIMEOUT device=03:00.0 address=0x00000002331e8900]

UEFI bios on my mobo (M5A97 PRO) is the latest available: 1604. I'd flashed it a long time ago, during the installation of FX CPU.

What is wrong here? Should I buy the second revision or completely another AM3+ motherboard? or smth wrong in my settings?

So no Windows with Seabios on this configuration?

walkindude@Desktop> sudo lspci -v -s 3:00.0                                                                                     ~
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] (prog-if 00 [VGA controller])
    Subsystem: PC Partner Limited / Sapphire Technology Device e174
    Flags: fast devsel, IRQ 16
    Memory at c0000000 (64-bit, prefetchable) [size=256M]
    Memory at fe820000 (64-bit, non-prefetchable) [size=128K]
    I/O ports at c000 [size=256]
    Expansion ROM at fe800000 [disabled] [size=128K]
    Capabilities: [50] Power Management version 3
    Capabilities: [58] Express Legacy Endpoint, MSI 00
    Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
    Capabilities: [150] Advanced Error Reporting
    Kernel driver in use: vfio-pci
    Kernel modules: radeon

meaning no MSI support currently enabled, am I right? Could I enable it?

Also tried these patches
http://vfio.blogspot.co.uk/2014/08/fixe … uests.html
the same,
AFTER choosing device to boot - black screen

Last edited by walkindude (2015-01-20 23:16:43)

Offline

#3936 2015-01-20 19:35:20

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Okay then. I guess i jumped the gun a bit. I noticed that it wasn't happening earlier, when the VM was down for a reboot, but was now. After turning off the VM and playing around a bit it's actually vims fault.

When i do some copy and paste operations in vim (or whatever it feels like it sometimes) it will start to not refresh a line on the screen. It seems that when this happens i have the issues in Zathura as well (my pdf reader). Quitting Vim while the issue is ongoing actually completely fixes the issue in zathura. Very strange, not the VM's fault though. Even stranger is that i've never had this problem before switching my graphics cards (the 660 was my main before), so either the VM does have some effect or the video card is acting up.

EDIT: Tried it again, and the issue happened again. It seems to be linked to the usage of the virtual machine as well... What's the current status on CPU pinning and hugepages? are either of those known to cause problems if on/off?

Last edited by DelusionalLogic (2015-01-20 20:10:39)

Offline

#3937 2015-01-20 21:18:08

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw
Could you please specify what log files, actions should be performed in order to collect necessary information - why virtual machine is not booted (screen is not turned on) if Nvidia GT640 is set as primary VGA device (via vfio x-vga=on).

Thank you in advance!

Offline

#3938 2015-01-21 08:03:14

MrTUX-10
Member
Registered: 2015-01-06
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you all nbhs, and everyone else for your contributions to this thread! It has been so helpful getting everything setup. I've had my VM running for a while now, everything runs pretty damn well. I'm using an Intel 4820k, EVGA-X79 Motherboard, GTX,970 with the latest drivers. I was wondering if anyone could provide some insight on vcpu pinning, i have cset and cgroups setup, and the performance on the machine is actually pretty awesome, but with that, making it even faster is always better, I have looked around a bit from googling, and everthing seems to point to an xml file, however I didn't create it using libvirt, I launch it from as script. Does anyone have any idea how you would pin the vpcu's? That would be so helpful!

Thanks in advance!

Offline

#3939 2015-01-21 10:52:20

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried one last solution (why I didn't do it earlier?). Booted in ARCH kernel (with no additional patching at all), and NO black screen: Windows DVD boots as expected!!
So I guess something wrong either in -ck patches or uksm.

No, nothing wrong with them:
just removed custom dsdt table from kernel and now no black screen

Last edited by walkindude (2015-01-21 16:55:28)

Offline

#3940 2015-01-21 11:41:33

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Catalyst installs without problems, I have 1920x1080 on guest.
Windows 10 Build 9888
Seabios for now, because desired card hasn't UEFI bios (6850) or maybe I don't know how to get one. Had seen on youtube how they managed to do it with 7000 series videocard, but for me it didn't work.

Offline

#3941 2015-01-21 21:57:07

redger
Member
Registered: 2014-02-26
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@walkindude
for UEFI bios on older AMD cards, try here http://www.overclock.net/t/1474306/rade … ios-thread ... worked for me

Offline

#3942 2015-01-21 22:35:44

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@redger
Yep, saw it, but not very thorough.. didn't find Sapphire 6850 1 gb, and moved to seabios then

tried it, now getting

walkindude@Desktop> ~/rom-parser/rom-parser uefi.rom                                                                                                                                                                                                                  ~/Dropbox
Valid ROM signature found @0h, PCIR offset 1ech
    PCIR: type 0, vendor: 1002, device: 6739, class: 030000
    PCIR: revision 0, vendor revision: d0a
Valid ROM signature found @fa00h, PCIR offset 1ch
    PCIR: type 3, vendor: 1002, device: 6779, class: 030000
    PCIR: revision 0, vendor revision: 0
        EFI: Signature Valid
    Last image

but isn't 1002:6779 a 6450? so I just lowered my card?)

Besides for now system is quite stable and I've already install a lot of things and don't really want to make any more experiments and reinstall everything again.

Thanks anyway

----

One question remains:

Arch:

Screen 1 -- Screen 2 <-- hdmi1
Synergy server

Windows:

Screen 2 <-- hdmi2
Synergy client

Synergys on both system work, I can use my host mouse / keyboard in guest, but

- If I place hosts horizontally I can't move to arch at all.

- If I place windows above, then at start I can move mouse in arch, but after it goes up in windows - it'll never return.

Offline

#3943 2015-01-22 11:51:52

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

one question; these days I am enjoying my successful VM smile Although I don't have to much time. Exams in exactly one month! Therefore I don't want to break something that works. However having archlinux as a host, I am kinda relying on system updates

So: How safe is it to update lnux without breaking the VM?

Offline

#3944 2015-01-22 13:09:08

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@4kGamer
Arch doesn't force you to update and will live one month and more happily without them.

As for me:
built new versions of kernel, qemu, libvirt several times, also upgraded other packages as well - all mistakes with VM were my own, not some buggy update

Also there is /var/cache/pacman/pkg is something goes terribly wrong

Last edited by walkindude (2015-01-22 13:12:02)

Offline

#3945 2015-01-22 18:55:46

KKF
Member
Registered: 2015-01-21
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

I am new to all this and have wanted to try it out for a while as a learning experience for myself.  I have the budget to build a PC just for this.  For starters I would like to have one Windows 8 VM and Elementary OS as host; if possible.  Eventually I would like to have an OSX VM and a Windows VM, though I understand how difficult that can be.

With anything of this magnitude I want to make sure I buy the right components for the job.  The only components I own at this point are a 660 ti gpu and a R4 case.  I will probably buy a 970 gpu at some point in the future but no rush at this point.  Any advise on what to buy for MOBO, CPU, and Ram?  Should I buy two separate SSD for each VM?  I have a server that hosts all my documents and files so don't need to much space on those SSD's. 

The OSX VM would mainly be used for Photoshop as my girlfriend prefers the layout of Photoshop in OSX to Windows. The Windows VM would be used for gaming.  I play Heroes of the Storm and SC2.  So nothing to intensive.

While I really do not have a budget for this I don't want to spend 2000 dollars either as it really is for learning and the challenge of putting this together.  So what would be my best options for a MOBO, CPU, and Ram? 

After I buy everything I plan to do an extensive how to on cost and time it took me, a novice, to do this.

Thanks!

Last edited by KKF (2015-01-22 18:57:21)

Offline

#3946 2015-01-22 20:08:21

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dude wink Thank you. I tried and everything is still working. Phew smile

@KKF: Make sure you get the motherboard right. Everything else depends on you.

I use this one: ASROCK Z97 Extreme6

It has been recommended to me by the user Archer here in this thread and I couldn't be happier

Last edited by 4kGamer (2015-01-22 20:15:38)

Offline

#3947 2015-01-22 20:26:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

KKF wrote:

Hey guys,

I am new to all this and have wanted to try it out for a while as a learning experience for myself.  I have the budget to build a PC just for this.  For starters I would like to have one Windows 8 VM and Elementary OS as host; if possible.  Eventually I would like to have an OSX VM and a Windows VM, though I understand how difficult that can be.

With anything of this magnitude I want to make sure I buy the right components for the job.  The only components I own at this point are a 660 ti gpu and a R4 case.  I will probably buy a 970 gpu at some point in the future but no rush at this point.  Any advise on what to buy for MOBO, CPU, and Ram?  Should I buy two separate SSD for each VM?  I have a server that hosts all my documents and files so don't need to much space on those SSD's. 

The OSX VM would mainly be used for Photoshop as my girlfriend prefers the layout of Photoshop in OSX to Windows. The Windows VM would be used for gaming.  I play Heroes of the Storm and SC2.  So nothing to intensive.

While I really do not have a budget for this I don't want to spend 2000 dollars either as it really is for learning and the challenge of putting this together.  So what would be my best options for a MOBO, CPU, and Ram? 

After I buy everything I plan to do an extensive how to on cost and time it took me, a novice, to do this.

Thanks!

Use a Xeon E5 or better and non-Intel graphics for the host if you don't want to deal with patching your kernel for ACS or VGA arbitration (BTW, X99 PCH root port patch just posted).  Gigabyte motherboards seem to be the most configurable when it comes selecting a primary graphics device, which may be important if you end up with 3 graphics cards.  If you blow all your money on an over-spec'd GPU for your needs and skimp a "client" processor, you'll pay for it patching your kernel forever.  Also, make sure you scale your core count and memory for the intended usage.  If you want to game at the same time as your gf is photoshop'ing, then you need to be able to dedicate some cores to each VM.  Also, OSX w/ GPU assignment is not something a lot of people are doing, so be sure to do your research on whether it's really an option.  There are parts of getting OSX to work that cannot be discussed on forums like this.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3948 2015-01-22 20:42:33

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Started recieving these after I upgraded to 3.19-rc1 and still reciving them on 3.19-rc5 :

[Thu Jan 22 23:13:54 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:13:54 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:55 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:13:55 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:55 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:13:55 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:56 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:13:56 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:56 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:13:56 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:57 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:13:57 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:58 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:13:58 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:58 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:13:58 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:59 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:13:59 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:13:59 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:13:59 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:00 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:00 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:00 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:00 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:01 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:01 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:01 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:01 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:02 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:02 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:03 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:03 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:03 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:03 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:04 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:04 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:04 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:04 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:05 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:05 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:05 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:05 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:06 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:06 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:07 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:07 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:07 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:07 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:08 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:08 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:08 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:08 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:09 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:09 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:09 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:09 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:10 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:10 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:10 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:10 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:11 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:11 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:12 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:12 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:12 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:12 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:13 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:13 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:13 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:13 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:14 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:14 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:14 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:14 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:15 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:15 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:16 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:16 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:16 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:16 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:17 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:17 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:17 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:17 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:18 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:18 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:18 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:18 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:19 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:19 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:19 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:19 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:20 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:20 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:21 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:21 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:21 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:21 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:22 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:22 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:22 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:22 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:23 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:23 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:23 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:23 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:24 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:24 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:25 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:25 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:25 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:25 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:26 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:26 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:26 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:26 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:27 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:27 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:27 2015] dmar: DRHD: handling fault status reg 302 [Thu Jan 22 23:14:27 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:28 2015] dmar: DRHD: handling fault status reg 402 [Thu Jan 22 23:14:28 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:28 2015] dmar: DRHD: handling fault status reg 502 [Thu Jan 22 23:14:28 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:29 2015] dmar: DRHD: handling fault status reg 602 [Thu Jan 22 23:14:29 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:30 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:30 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf11f000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf169000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf201000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf299000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf331000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf128000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf1c5000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf260000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf2fa000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf466000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf11f000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf18b000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf22a000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf2cc000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf370000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DRHD: handling fault status reg 700 [Thu Jan 22 23:14:46 2015] dmar: DRHD: handling fault status reg 702 [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf456000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf45f000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:46 2015] dmar: DRHD: handling fault status reg 102 [Thu Jan 22 23:14:46 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf45c000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:49 2015] dmar: DRHD: handling fault status reg 202 [Thu Jan 22 23:14:49 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bef2e000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:49 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bef7d000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:49 2015] dmar: DMAR:[DMA Read] Request device [02:00.0] fault addr bf01c000 DMAR:[fault reason 06] PTE Read access is not set [Thu Jan 22 23:14:49 2015] dmar: DMAR:[DMA Read] Request device [00:1a.0] fault addr ffffd000 DMAR:[fault reason 06] PTE Read access is not set

02:00.0 --> Nvidia GTX 770
00:1a.0 --> Intel X99 USB controller

Sometimes , the VM that has the GTX 770 throws code 43 and refuse to initialize the GPU when the above errors appear . Sometimes , it works perfectly even with the aforementioned errors .

Offline

#3949 2015-01-22 21:17:27

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, I've been trying Arch instead of Slackware and have noticed a few odd things.  My qemu script is

QEMU_SDL_SAMPLES=2048 \ qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none\ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio \ -device piix4-ide,bus=pcie.0 \ -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk \ -net vde -net nic,vlan=0,model=virtio \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -drive file=virtio-win-0.1-74.iso,id=isocd -device ide-cd,bus=ide.2,drive=isocd \ -rtc base=localtime

Which works fine under Slackware -current with kernel 3.16.2... but doesn't under Arch.  There are two issues.  First, qemu doesn't execute, it terminates with an error about how "ide-hd" yes "h" not "c" previously being in use. Which is odd considering there is no ide-hd.

The line -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk  is responsible, remove it and the problem goes away.  I can't find any permutation of -cdrom,  -device or -drive commands that allow sharing of the host cdrom drive.  Did I miss something new in qemu?  The version under slackware is 2.0.0.

The second problem is that Windows blue screens and can't boot, once I remove the offending cdrom line.  If I run the qemu script without passthrough (using vga-std) and without the cdrom line, then it boots fine.  Did something else change?

I also note that when I do boot successfully there are some console error messages about pulseaudio, presumably something I'm supposed to use under Arch and have not set up. (Slackware doesn't do pulseaudio, it's against our/their religion)

The hardware is an Nvidia Quadro K2000, the CPU intel 4960K, no intel graphics, vfio bind seems to have worked and there is the usual /dev/vfio/vfio and other files present.

Last edited by mostlyharmless (2015-01-22 21:18:02)

Offline

#3950 2015-01-22 23:32:57

nythrix
Member
Registered: 2015-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to the awesome people helping out in this thread I've been able to successfully pass through a 9600GT to a Windows 7 guest.

Unfortunately, linux guests don't work. I tried my oldish Debian VM, which has been running on "-vga std" since forever but I loose graphics output around the time the console switches fonts during boot (not sure how to describe it). Everything happens very quickly so the last messages I'm able to make out are stuff about intel RAPL and something about BARs ..? I wiped out the VM and reinstalled Debian but the problem persisted. The installation itself was flawless though (with passthrough). Twice.

Then I tried installing Fedora 21 Workstation. Couldn't get much further than some sort of greeting menu. It seems the signal is again lost during attempts to fiddle with the graphics/framebuffer.

Is this some sort of device reset problems I'm observing?

PS: I'm running 3.17 with the i915 arbiter patch. Qemu 2.1.2.

Offline

#3951 2015-01-23 01:00:37

SEPIPES
Member
Registered: 2015-01-23
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

setup:

Distro: Ubuntu, 14.10 kernel: 3.18.0, patched ACS+VGA qemu: 2.1.0 libvirt: 1.2.8 Platform: Laptop, clevo p370SM proc: i7-4810MQ (VTX, VTD) GPU1 (host): NVIDIA 870m (geforce) GPU2 (guest): NVIDIA 980m (geforce) - aftermarket from clevo GuestOSE: Win 8.1|Server 2012R2, Fresh installs, can reinstall if necessary. QEMU startup params of test VM currently loaded with 8.1: qemu-system-x86_64 -enable-kvm -name Windows81DevWorkstation -S -machine pc-i440fx-utopic,accel=kvm,usb=off -m 8192 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 074a3dbe-51d7-4a55-85f9-9330516fba1b -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows81DevWorkstation.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/CG4REPO/VirtualMachines/WindowsWork_New.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/var/CG4REPO/ISO/Microsoft/win81.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=24,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:73:07:3c,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -chardev spicevmc,id=charredir2,name=usbredir -device usb-redir,chardev=charredir2,id=redir2 -chardev spicevmc,id=charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -device vfio-pci,host=02:00.0,x-vga=on -cpu Haswell,kvm=off -msg timestamp=on (virsh generated, will post if necessary)

-side note-
Been reading this post for awhile going through the various issues in my setup (been fun getting this to work without direct access to a head on the secondary GPU wink. Thanks for the awesome tutorials / troubleshooting, it's gotten me almost to the end! and I do apologize if this was covered in the past, I've been reading / searching and haven't found a direct reference to this yet. My brain is pretty squishy right now, though. While this is an arch forum [and I do apologize for posting regarding ubuntu], this is the most active thread, so I was hoping someone could help me out smile.

Issue: Incorrect device ID on host passed to guest prevents secondary GPU from installing.

WITIR:
The device is passed through, however, the device hardware ID inside the guest is short compared to the current 980Ms [If you compare this to what a quadro reports as in their drivers it is more along those lines in terms of pc hardware IDs], I believe this stems from the host [host hardware ID: 10de:13d7] so when attempting driver installation it fails.

hardware ID: PCI\VEN_10DE&DEV_13D7&SUBSYS_110B10DE&REV_A1 PCI\VEN_10DE&DEV_13D7&SUBSYS_110B10DE PCI\VEN_10DE&DEV_13D7&CC_030000 PCI\VEN_10DE&DEV_13D7&CC_0300 Device Instance Path: PCI\VEN_10DE&DEV_13D7&SUBSYS_110B10DE&REV_A1\3&13C0B0C5&0&10 Would look like: %NVIDIA_DEV.13D7% = Section232, PCI\VEN_10DE&DEV_13D7&SUBSYS_110B10DE compared to NVIDIA INF output (1 example of a 980M): %NVIDIA_DEV.13D7.19FD.1043% = Section228, PCI\VEN_10DE&DEV_13D7&SUBSYS_19FD1043 

If you mod the INF for the driver with the line first line mentioned above you can get the installer to work [and the driver to load], however, with 8.1 there is no permanent driver signing disable - and modding an INF (I used nvami.inf) breaks the hash check on a normal boot and fails out the driver. Without direct access to the head on the GPU, this can be... problematic wink. With a stripped down version of the libvirt xml to qemu args (only adding disk, network adapter) the same issue occurs.

So I guess the question is, does anyone have any suggestions on a quick workaround for this [a way to manipulate the hardware ID when passed from host to guest]? Or is there something obvious I'm missing in the setup that is causing this issue to begin with? I imagine it's the latter, hopefully smile.

Any thoughts or suggestions would be welcome - if you need more data just let me know what you'd like. I am probably overthinking this way too much at this point.

Offline

#3952 2015-01-23 01:19:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SEPIPES wrote:

So I guess the question is, does anyone have any suggestions on a quick workaround for this [a way to manipulate the hardware ID when passed from host to guest]? Or is there something obvious I'm missing in the setup that is causing this issue to begin with? I imagine it's the latter, hopefully smile.

https://bbs.archlinux.org/viewtopic.php … 7#p1474497


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3953 2015-01-23 05:23:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

Started recieving these after I upgraded to 3.19-rc1 and still reciving them on 3.19-rc5 :

Running 3.19-rc5 here, no issues.  There were relatively few VT-d changes between 3.18 and 3.19 and effectively no vfio changes.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3954 2015-01-23 12:08:06

lordleto
Member
Registered: 2014-11-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
KKF wrote:

Hey guys,

I am new to all this and have wanted to try it out for a while as a learning experience for myself.  I have the budget to build a PC just for this.  For starters I would like to have one Windows 8 VM and Elementary OS as host; if possible.  Eventually I would like to have an OSX VM and a Windows VM, though I understand how difficult that can be.

With anything of this magnitude I want to make sure I buy the right components for the job.  The only components I own at this point are a 660 ti gpu and a R4 case.  I will probably buy a 970 gpu at some point in the future but no rush at this point.  Any advise on what to buy for MOBO, CPU, and Ram?  Should I buy two separate SSD for each VM?  I have a server that hosts all my documents and files so don't need to much space on those SSD's. 

The OSX VM would mainly be used for Photoshop as my girlfriend prefers the layout of Photoshop in OSX to Windows. The Windows VM would be used for gaming.  I play Heroes of the Storm and SC2.  So nothing to intensive.

While I really do not have a budget for this I don't want to spend 2000 dollars either as it really is for learning and the challenge of putting this together.  So what would be my best options for a MOBO, CPU, and Ram? 

After I buy everything I plan to do an extensive how to on cost and time it took me, a novice, to do this.

Thanks!

Use a Xeon E5 or better and non-Intel graphics for the host if you don't want to deal with patching your kernel for ACS or VGA arbitration (BTW, X99 PCH root port patch just posted).  Gigabyte motherboards seem to be the most configurable when it comes selecting a primary graphics device, which may be important if you end up with 3 graphics cards.  If you blow all your money on an over-spec'd GPU for your needs and skimp a "client" processor, you'll pay for it patching your kernel forever.  Also, make sure you scale your core count and memory for the intended usage.  If you want to game at the same time as your gf is photoshop'ing, then you need to be able to dedicate some cores to each VM.  Also, OSX w/ GPU assignment is not something a lot of people are doing, so be sure to do your research on whether it's really an option.  There are parts of getting OSX to work that cannot be discussed on forums like this.


Hi just wanted to give a short overview over my system that i am really content with. Essentially I started with an typical workstation build and done dualboot over 2 years. But when building the ws i was already looking at functional vt-d and enough pcie slots. So after a lot of success stories in this thread i finally bought a used 2 gpu and tried kvm and was more then pleased with the result.

My build is as follows

MB  ASUS P9X79WS
CPU Xeon E5 1650 (Xeon E5 16xx are a much better choice then similar i7, for same money you get ECC support and v1 and v2 still have an open multi, so totally easy to OC. Essentially this is the only Xeon line that you can OC, just make sure your MB supports it. Dont know whether v3 (Haswell) is still open multi, but i dont plan to update soon. My CPU is running on 4GHz (didnt bother to go higher, stability is king) which is still plenty.)
32GB ECC Ram
Geforce GTX 750 Ti Primary
Geforce GTX Titan Passthrough
2 SSD (1 Boot, 1 VM)
Mirrored HD for Data (ZFS, with actually a third ssd for l2arc)
Usb audiocard
Aten switch

All is running on Ubuntu 12.04 with Mainline 3.17.1 Kernel and no patches.
Selfcompiled qemu 2.1.2
libvirt 1.2 from ubuntu cloud ppa
I know ubuntu is not latest, but i am to lazy to update and important stuff (kernel, driver, qemu) is uptodate

pci-stub is done by module, this way i can disable blacklisting the titan during boot if i want. (key is to add softdep drm pre: pci-stub somewehre in modprobe.d)
i patched nvidia dkms module with the patch linked here somewhere

Windows is running on 4 HT Core pinned to them. Rest of the system is moved to remaining 2 cores when vm is spun up by cset, called from a script in libvirt hooks folder. Windows gets 12gb in hugepages.
For cpu i have kvm=off so i can run 340 driver in vm. All is working, even game streaming to raspberrypi through geforce experience
Performance is near bare metal (Firestrike 9100 and Firestrike Extreme 4800)

Windows VM gets also a passed through nic (MB has 2) and the asmedia USB 3.0 onboard controller
Audio is done by usb sound card hooked to the passed through usb 3.0 controller.
Windows VM is actually the image of the bare metal installation, didnt want to reinstall smile

Input and Output is switched by aten switch (got it used)

At the moment the VM is running on q35 machine and seabios. I tried also 440fx (with qemu wrapper script), it worked but for some reason some things in geforce experience were disabled, didnt look to closely into that.
Only issue is 1 out of 20 times the whole system freezes on VM start, and really on start only, even before any output on second card is to be seen. Cant really point to why. But its so seldom and only occuring during vm boot that i didnt know how to debug this.
Now i am sorting out what gives me best io performace and whether i can get trim working for sparse raw files on zfs datasets. (Virtio blk now, trim not working)
Also Windows 7 virtio net seems to be bottlenecked at 500Mbyte/s for incoming tcp connections. Outgoing and bothways with linux vm is at 2GByte/s and more. That is important metric for internal shares on ssd, but even 500Mbyte/s is not bad.

Endresult:
I dont need to double boot to play games or use lightroom with color calibration on monitor.
All important stuff (Dokuments, Pictures) is finally on zfs storage (zfs send recieve, checksums, snapshots, clones, way better than ntfs or ext4 for important data)

I can go into configuration detail (xmls, scripts) if someone is interested.
I want to say many thanks to all who write here for invaluable information and hard programming work.
And finally i can only vouch for the Xeon route.

Last edited by lordleto (2015-01-23 13:09:34)

Offline

#3955 2015-01-23 15:05:36

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Running 3.19-rc5 here, no issues.  There were relatively few VT-d changes between 3.18 and 3.19 and effectively no vfio changes.

That's weird . I just downgraded to 3.18.2 and I no longer have any issues , the VM works correctly . No code 43 and no DMAR errors in host's dmesg output .

It surely is related to 3.19 + X99 .

Offline

#3956 2015-01-23 15:27:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

Running 3.19-rc5 here, no issues.  There were relatively few VT-d changes between 3.18 and 3.19 and effectively no vfio changes.

That's weird . I just downgraded to 3.18.2 and I no longer have any issues , the VM works correctly . No code 43 and no DMAR errors in host's dmesg output .

It surely is related to 3.19 + X99 .

$ git log --oneline --no-merges v3.18..v3.19-rc1 drivers/iommu | grep -v omap | grep -v amd | grep -v smmu | grep -v rockchip | grep -v msm | grep -v vmsa | grep -v exynos 91411da iommu/vt-d: Use helpers to access irq_cfg data structure associated with IRQ b71a3b2 x86: irq_remapping: Use helpers to access irq_cfg data structure associated with IRQ a42a7a1 iommu: store DT-probed IOMMU data privately 8918465 memory: Add NVIDIA Tegra memory controller support 18f2340 iommu: Decouple iommu_map_sg from CPU page size cc4f14a iommu/vt-d: Fix an off-by-one bug in __domain_mapping() 461bfb3f iommu: fix initialization without 'add_device' callback 7eba1d5 iommu: provide helper function to configure an IOMMU for an of master 1cd076bf iommu: provide early initialisation hook for IOMMU drivers 63a7b17 PCI/MSI: Simplify PCI MSI code by initializing msi_desc.nvec_used earlier ffebeb4 iommu/vt-d: Enhance intel-iommu driver to support DMAR unit hotplug 51acce3 iommu/vt-d: Enhance error recovery in function intel_enable_irq_remapping() a7a3dad iommu/vt-d: Enhance intel_irq_remapping driver to support DMAR unit hotplug d35165a iommu/vt-d: Search for ACPI _DSM method for DMAR hotplug 6b19724 iommu/vt-d: Implement DMAR unit hotplug framework 78d8e70 iommu/vt-d: Dynamically allocate and free seq_id for DMAR units c2a0b53 iommu/vt-d: Introduce helper function dmar_walk_resources() 1a2262f x86/vt-d: Fix incorrect bit operations in setting values d7da6bd iommu: Improve error handling when setting bus iommu 38ec010 iommu: Do more input validation in iommu_map_sg() 315786e iommu: Add iommu_map_sg() function 98b773c iommu: drop owner assignment from platform_drivers $ git log --oneline --no-merges v3.18..v3.19-rc1 drivers/vfio 83a1891 PCI/MSI: Rename write_msi_msg() to pci_write_msi_msg() 5e9f36c drivers/vfio: allow type-1 IOMMU instantiation on top of an ARM SMMU 1d53a3a vfio: make vfio run on s390

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3957 2015-01-23 17:19:10

SEPIPES
Member
Registered: 2015-01-23
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SEPIPES wrote:

So I guess the question is, does anyone have any suggestions on a quick workaround for this [a way to manipulate the hardware ID when passed from host to guest]? Or is there something obvious I'm missing in the setup that is causing this issue to begin with? I imagine it's the latter, hopefully smile.

https://bbs.archlinux.org/viewtopic.php … 7#p1474497

Thanks for the link.
It reminded me that I could get a signed driver by manipulating the compat list rather than the hardware dev_ID itself [inside the guest]. I was kind of surprised that worked (well, worked enough that I could get past the hardware ID check). This gave me the latest driver that matches against a win booted host but gets me the notorious code 43 from the 980m post driver install.

Ah well, if it was easy it wouldn't be fun. Time to go reread some posts, thanks again smile.

Offline

#3958 2015-01-23 17:26:08

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

I got it that changes are only happening in the IOMMU driver for 3.19 and not VFIO . What I'm certain of , is that these issues do not exist with 3.18.2 . They only happen with 3.19-rc1~rc5 .

Thank you for posting these commit diffs , I can see that some changes list DMAR . Those errors I had were DMAR related .

Offline

#3959 2015-01-24 07:17:38

AdamM
Member
Registered: 2015-01-24
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

Started recieving these after I upgraded to 3.19-rc1 and still reciving them on 3.19-rc5 :

[Thu Jan 22 23:13:54 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:13:54 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set ...

I've been having the same issues after the update.

The errors first appeared when booting, complaining about the integrated graphics card. I tried intel_iommu=on,igfx_off which resolved the boot time errors, but caused qemu to crash.

I should point out that I've been running both Window 7 and Mac OS X Mavericks for months without a single problem on previous kernels.

Specs:

Motherboard: Asus Maximus VII Ranger
CPU: Intel i7-4790K
GPU: Radeon R9 270X

Offline

#3960 2015-01-24 14:37:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AdamM wrote:
Denso wrote:

@aw :

Started recieving these after I upgraded to 3.19-rc1 and still reciving them on 3.19-rc5 :

[Thu Jan 22 23:13:54 2015] dmar: DRHD: handling fault status reg 2 [Thu Jan 22 23:13:54 2015] dmar: DMAR:[DMA Read] Request device [02:00.1] fault addr 10af25000 DMAR:[fault reason 06] PTE Read access is not set ...

I've been having the same issues after the update.

Does anything change adding sp_off to the intel_iommu options?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3961 2015-01-25 09:48:13

AdamM
Member
Registered: 2015-01-24
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does anything change adding sp_off to the intel_iommu options?

I've tried all the available options (igfx_off, forcedac, strict, and sp_off) but the results are the same.

I've been religiously checking the log for the past week hoping for a magical fix to appear. No such luck unfortunately.

Offline

#3962 2015-01-25 16:58:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AdamM wrote:
aw wrote:

Does anything change adding sp_off to the intel_iommu options?

I've tried all the available options (igfx_off, forcedac, strict, and sp_off) but the results are the same.

I've been religiously checking the log for the past week hoping for a magical fix to appear. No such luck unfortunately.

It would probably be more useful to hunt down the change that broke it rather than blindly hope for a fix.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3963 2015-01-25 19:30:45

AdamM
Member
Registered: 2015-01-24
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It would probably be more useful to hunt down the change that broke it rather than blindly hope for a fix.

I have been tempted to compile the kernel for each commit, but at the moment my system needs to stay on, so I'd have no way of testing them. Hence the waiting for a magical fix to appear.

I should also point out that unlike Denso, my problems started when I upgraded from 3.17.6 to 3.18.2. I've looked at all the changes to the iommu code between those releases but nothing stands out.

Offline

#3964 2015-01-26 09:52:50

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried a new workstation card, AMD W7100 (Tonga family) and it's causing me problems, although it's also possible that my setup (see below) or, less likely, kernel upgrade from 3.17.7 to 3.17.8 could be also to blame. Everything works fine until I shutdown VM machine win171 & and start it again. Then I receive following dmesg on the console (it fills the screen very very quickly - below was retyped by me from screen photo, I am yet to attach serial console cable):

dmar: DRHD: handling fault status reg 40 dmar: DRHD: handling fault status reg 40 dmar: DRHD: handling fault status reg 40 . . . 

which is next followed by

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [qemu:win171:4568] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [qemu:win249:4259] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [qemu:win171:4568] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [qemu:win249:4259] INFO: rcu_preempt_detected stalls on CPUs/tasks: { 2} (detected by 9, t=18002 jiffies, g=56726, c=56725, q=11821) mpt2sas0: mpt2sas_scsih_issue_tm: timeout NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [qemu:win171:4568] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [qemu:win249:4259] . . . 

When this happens the computer becomes totally unresponsive and has to be powered down. Normal reset is not enough, I get dmar errors as soon as kernel is loaded again.

I'm running 2 different VMs, win171 and win249 . Both are Windows 7, each has its own GPU (and small number of PCI passed-thru devices) and both perform well and are stable - that is, until I restart win171 (which is attached to W7100).

Another factor which might be causing problems here is that card W7100 is used by system as primary VGA, i.e. first it shows BIOS startup screen, next SYSLINUX and then kernel messages on startup. Also, kernel is actually writing to it (as VGA framebuffer) after win171 has been started, as I see some low priority dmesg (warnings related to my USB devices setup) on the screen overwriting SEABIOS messages. One way I could try to prevent this is to disable ROM of this card in BIOS thus rendering my machine entirely headless, but this is a bit drastic.

Ideally I'd rather find the right combination of kernel parameters to prevent kernel from using VGA after after its been loaded to memory (while still allowing both BIOS and SYSLINUX to use it). I will try "console=ttyS0 nomodeset", any better ideas? The hypervisor is meant to run headless, but I'd rather see initial startup sequence until the kernel has loaded.

Last edited by Bronek (2015-01-26 10:01:40)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3965 2015-01-27 07:47:26

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

10Gbps Pipe :

Hi , I just got around setting an effective 10Gbps networking between the host and Windows 8.1 guest (For Samba shares) and I got a very good performance . Here is a SMB share on the host's RAM Disk result :

http://i.imgur.com/Ro6Jj5y.png

Execute this script BEFORE booting your VM to set up the host's interface :

ip tuntap add tap_s mode tap multi_queue ip link set tap_s mtu 65500 ip link set tap_s txqueuelen 5000 ip link set dev tap_s up ip addr add 195.165.1.2/24 broadcast 195.165.1.255 dev tap_s

Of course , change the IP address to whatever you like .

Then add this interface as a device to your QEMU command line :

-netdev type=tap,ifname=tap_s,id=net1,vhost=on,vhostforce=on,queues=4,script= \ -device virtio-net-pci,netdev=net1,mq=on,vectors=9 \

Now on your Windows VM , set the new interface , give it an IP address in the same space as the host's interface like : 195.165.1.3 with a netmask of 255.255.255.0 , and no gateway or DNS .

Done !

You can now ping your host and connect to your SMB shares at 10Gbps .

You can also raise the MTU value in the interface's configuration dialog in Windows Device Manager from 1500 to 65500 . I got only 5Gbps with 1500 MTU , so you should change it to 65500 .

Please note that this is going to be from host to guest and vice versa only , you still need another interface for internet / bridge networking .

Hope this helps smile

Last edited by Denso (2015-01-27 08:01:05)

Offline

#3966 2015-01-27 11:59:00

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello all,

My system is running perfect normally, thanks to all of you but, sometimes, after shutting down my virtual guest and turn it on again, I had a segmentation fault:

Jan 27 07:26:26 mycolonialone kernel: general protection fault: 0000 [#1] PREEMPT SMP Jan 27 07:26:26 mycolonialone kernel: Modules linked in: vfio_pci vfio_iommu_type1 vfio ecb joydev mousedev btusb uvcvideo videobuf2_vmalloc bluetooth videobuf2_memops s Jan 27 07:26:26 mycolonialone kernel: 8250_dw dw_dmac_core i2c_designware_platform i2c_designware_core gpio_lynxpoint spi_pxa2xx_platform acpi_pad nouveau mxm_wmi wmi t Jan 27 07:26:26 mycolonialone kernel: CPU: 2 PID: 2176 Comm: vfio-bind Not tainted 3.18.2-2-ARCH #1 Jan 27 07:26:26 mycolonialone kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme6, BIOS P1.60 12/09/2014 Jan 27 07:26:26 mycolonialone kernel: task: ffff88028c42da90 ti: ffff88002c97c000 task.ti: ffff88002c97c000 Jan 27 07:26:26 mycolonialone kernel: RIP: 0010:[<ffffffff813c3bf3>] [<ffffffff813c3bf3>] __rpm_callback+0x33/0x90 Jan 27 07:26:26 mycolonialone kernel: RSP: 0018:ffff88002c97fd38 EFLAGS: 00010286 Jan 27 07:26:26 mycolonialone kernel: RAX: 0000000000000008 RBX: ffff880458dad098 RCX: 0000000000000000 Jan 27 07:26:26 mycolonialone kernel: RDX: ffff8804541e8b88 RSI: ffff880458dad098 RDI: ffff880458dad098 Jan 27 07:26:26 mycolonialone kernel: RBP: ffff88002c97fd58 R08: 0000000000000246 R09: ffffea0001a8b800 Jan 27 07:26:26 mycolonialone kernel: R10: ffffffffa096a1c8 R11: ffffea001138da40 R12: ffff880458dad146 Jan 27 07:26:26 mycolonialone kernel: R13: 1b48313b30315b1b R14: 0000000000000000 R15: fffffffffffffff2 Jan 27 07:26:26 mycolonialone kernel: FS: 00007f2964cef700(0000) GS:ffff88046fa80000(0000) knlGS:0000000000000000 Jan 27 07:26:26 mycolonialone kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 27 07:26:26 mycolonialone kernel: CR2: 00007f2964d1b000 CR3: 00000000109ff000 CR4: 00000000001407e0 Jan 27 07:26:26 mycolonialone kernel: Stack: Jan 27 07:26:26 mycolonialone kernel: 00000000ffffffff ffff880458dad098 ffff880458dad098 0000000000000004 Jan 27 07:26:26 mycolonialone kernel: ffff88002c97fd88 ffffffff813c4a89 ffff880458dad098 0000000000000004 Jan 27 07:26:26 mycolonialone kernel: ffff880458dad146 0000000000000246 ffff88002c97fdb8 ffffffff813c4bc3 Jan 27 07:26:26 mycolonialone kernel: Call Trace: Jan 27 07:26:26 mycolonialone kernel: [<ffffffff813c4a89>] rpm_idle+0x259/0x340 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff813c4bc3>] __pm_runtime_idle+0x53/0x70 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff812f3208>] pci_device_remove+0x78/0xc0 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff813bab7f>] __device_release_driver+0x7f/0xf0 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff813bac13>] device_release_driver+0x23/0x30 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff813b98ed>] unbind_store+0xed/0x150 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff813b8b45>] drv_attr_store+0x25/0x40 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff8124747a>] sysfs_kf_write+0x3a/0x50 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff812469be>] kernfs_fop_write+0xee/0x180 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff811cf1d7>] vfs_write+0xb7/0x200 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff811cfd29>] SyS_write+0x59/0xd0 Jan 27 07:26:26 mycolonialone kernel: [<ffffffff81554ca9>] system_call_fastpath+0x12/0x17 Jan 27 07:26:26 mycolonialone kernel: Code: e5 41 55 41 54 53 4c 8d a6 ae 00 00 00 49 89 fd 48 89 f3 48 83 ec 08 f6 86 89 01 00 00 02 4c 89 e7 74 35 e8 50 06 19 00 48 89 Jan 27 07:26:26 mycolonialone kernel: RIP [<ffffffff813c3bf3>] __rpm_callback+0x33/0x90 Jan 27 07:26:26 mycolonialone kernel: RSP <ffff88002c97fd38> Jan 27 07:26:26 mycolonialone kernel: ---[ end trace 449fe5da686b88a6 ]---

Please, let me know all the files, logs or whatever you need and I will provide them with pleasure.

Is not anything urgent because only happens sometimes but is annoying smile

Thanks a lot in advance smile


Regards,


TheArcher

Offline

#3967 2015-01-27 21:38:20

KKF
Member
Registered: 2015-01-21
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
KKF wrote:

Hey guys,

I am new to all this and have wanted to try it out for a while as a learning experience for myself.  I have the budget to build a PC just for this.  For starters I would like to have one Windows 8 VM and Elementary OS as host; if possible.  Eventually I would like to have an OSX VM and a Windows VM, though I understand how difficult that can be.

With anything of this magnitude I want to make sure I buy the right components for the job.  The only components I own at this point are a 660 ti gpu and a R4 case.  I will probably buy a 970 gpu at some point in the future but no rush at this point.  Any advise on what to buy for MOBO, CPU, and Ram?  Should I buy two separate SSD for each VM?  I have a server that hosts all my documents and files so don't need to much space on those SSD's. 

The OSX VM would mainly be used for Photoshop as my girlfriend prefers the layout of Photoshop in OSX to Windows. The Windows VM would be used for gaming.  I play Heroes of the Storm and SC2.  So nothing to intensive.

While I really do not have a budget for this I don't want to spend 2000 dollars either as it really is for learning and the challenge of putting this together.  So what would be my best options for a MOBO, CPU, and Ram? 

After I buy everything I plan to do an extensive how to on cost and time it took me, a novice, to do this.

Thanks!

Use a Xeon E5 or better and non-Intel graphics for the host if you don't want to deal with patching your kernel for ACS or VGA arbitration (BTW, X99 PCH root port patch just posted).  Gigabyte motherboards seem to be the most configurable when it comes selecting a primary graphics device, which may be important if you end up with 3 graphics cards.  If you blow all your money on an over-spec'd GPU for your needs and skimp a "client" processor, you'll pay for it patching your kernel forever.  Also, make sure you scale your core count and memory for the intended usage.  If you want to game at the same time as your gf is photoshop'ing, then you need to be able to dedicate some cores to each VM.  Also, OSX w/ GPU assignment is not something a lot of people are doing, so be sure to do your research on whether it's really an option.  There are parts of getting OSX to work that cannot be discussed on forums like this.


So I take it that the cores are not dynamically shared as a hyper-v?  Wouldn't an AMD chip be a better fit then?  (More cores cheaper... AMD FX-9590?)  Thank you for your insight I am sure I will have more questions.  smile

Last edited by KKF (2015-01-27 21:51:55)

Offline

#3968 2015-01-27 21:50:44

KKF
Member
Registered: 2015-01-21
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lordleto wrote:
aw wrote:
KKF wrote:

Hey guys,

I am new to all this and have wanted to try it out for a while as a learning experience for myself.  I have the budget to build a PC just for this.  For starters I would like to have one Windows 8 VM and Elementary OS as host; if possible.  Eventually I would like to have an OSX VM and a Windows VM, though I understand how difficult that can be.

With anything of this magnitude I want to make sure I buy the right components for the job.  The only components I own at this point are a 660 ti gpu and a R4 case.  I will probably buy a 970 gpu at some point in the future but no rush at this point.  Any advise on what to buy for MOBO, CPU, and Ram?  Should I buy two separate SSD for each VM?  I have a server that hosts all my documents and files so don't need to much space on those SSD's. 

The OSX VM would mainly be used for Photoshop as my girlfriend prefers the layout of Photoshop in OSX to Windows. The Windows VM would be used for gaming.  I play Heroes of the Storm and SC2.  So nothing to intensive.

While I really do not have a budget for this I don't want to spend 2000 dollars either as it really is for learning and the challenge of putting this together.  So what would be my best options for a MOBO, CPU, and Ram? 

After I buy everything I plan to do an extensive how to on cost and time it took me, a novice, to do this.

Thanks!

Use a Xeon E5 or better and non-Intel graphics for the host if you don't want to deal with patching your kernel for ACS or VGA arbitration (BTW, X99 PCH root port patch just posted).  Gigabyte motherboards seem to be the most configurable when it comes selecting a primary graphics device, which may be important if you end up with 3 graphics cards.  If you blow all your money on an over-spec'd GPU for your needs and skimp a "client" processor, you'll pay for it patching your kernel forever.  Also, make sure you scale your core count and memory for the intended usage.  If you want to game at the same time as your gf is photoshop'ing, then you need to be able to dedicate some cores to each VM.  Also, OSX w/ GPU assignment is not something a lot of people are doing, so be sure to do your research on whether it's really an option.  There are parts of getting OSX to work that cannot be discussed on forums like this.


Hi just wanted to give a short overview over my system that i am really content with. Essentially I started with an typical workstation build and done dualboot over 2 years. But when building the ws i was already looking at functional vt-d and enough pcie slots. So after a lot of success stories in this thread i finally bought a used 2 gpu and tried kvm and was more then pleased with the result.

My build is as follows

MB  ASUS P9X79WS
CPU Xeon E5 1650 (Xeon E5 16xx are a much better choice then similar i7, for same money you get ECC support and v1 and v2 still have an open multi, so totally easy to OC. Essentially this is the only Xeon line that you can OC, just make sure your MB supports it. Dont know whether v3 (Haswell) is still open multi, but i dont plan to update soon. My CPU is running on 4GHz (didnt bother to go higher, stability is king) which is still plenty.)
32GB ECC Ram
Geforce GTX 750 Ti Primary
Geforce GTX Titan Passthrough
2 SSD (1 Boot, 1 VM)
Mirrored HD for Data (ZFS, with actually a third ssd for l2arc)
Usb audiocard
Aten switch

All is running on Ubuntu 12.04 with Mainline 3.17.1 Kernel and no patches.
Selfcompiled qemu 2.1.2
libvirt 1.2 from ubuntu cloud ppa
I know ubuntu is not latest, but i am to lazy to update and important stuff (kernel, driver, qemu) is uptodate

pci-stub is done by module, this way i can disable blacklisting the titan during boot if i want. (key is to add softdep drm pre: pci-stub somewehre in modprobe.d)
i patched nvidia dkms module with the patch linked here somewhere

Windows is running on 4 HT Core pinned to them. Rest of the system is moved to remaining 2 cores when vm is spun up by cset, called from a script in libvirt hooks folder. Windows gets 12gb in hugepages.
For cpu i have kvm=off so i can run 340 driver in vm. All is working, even game streaming to raspberrypi through geforce experience
Performance is near bare metal (Firestrike 9100 and Firestrike Extreme 4800)

Windows VM gets also a passed through nic (MB has 2) and the asmedia USB 3.0 onboard controller
Audio is done by usb sound card hooked to the passed through usb 3.0 controller.
Windows VM is actually the image of the bare metal installation, didnt want to reinstall smile

Input and Output is switched by aten switch (got it used)

At the moment the VM is running on q35 machine and seabios. I tried also 440fx (with qemu wrapper script), it worked but for some reason some things in geforce experience were disabled, didnt look to closely into that.
Only issue is 1 out of 20 times the whole system freezes on VM start, and really on start only, even before any output on second card is to be seen. Cant really point to why. But its so seldom and only occuring during vm boot that i didnt know how to debug this.
Now i am sorting out what gives me best io performace and whether i can get trim working for sparse raw files on zfs datasets. (Virtio blk now, trim not working)
Also Windows 7 virtio net seems to be bottlenecked at 500Mbyte/s for incoming tcp connections. Outgoing and bothways with linux vm is at 2GByte/s and more. That is important metric for internal shares on ssd, but even 500Mbyte/s is not bad.

Endresult:
I dont need to double boot to play games or use lightroom with color calibration on monitor.
All important stuff (Dokuments, Pictures) is finally on zfs storage (zfs send recieve, checksums, snapshots, clones, way better than ntfs or ext4 for important data)

I can go into configuration detail (xmls, scripts) if someone is interested.
I want to say many thanks to all who write here for invaluable information and hard programming work.
And finally i can only vouch for the Xeon route.

Wow, thanks this was very helpful!  I will post more questions as time progresses.

Offline

#3969 2015-01-27 21:51:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

KKF wrote:

So I take it that the cores are not dynamically shared as a hyper-v?  Wouldn't an AMD chip be a better fit then?  (More cores cheaper...)  Thank you for your insight I am sure I will have more questions.  smile

Cores are share-able, what I'm suggesting and what many people here have confirmed is that if you intend to use one of the VMs for gaming, then you don't want the variable performance associated with sharing cores.  Do you really want to get fragged in your game because your girlfriend applied a complex transform to an image and your frame rate bottomed out?  If you were only running desktop applications in each VM, then by all means oversubscribe the cores between VMs.  As for AMD, I just don't see them being all that competitive and they certainly don't have the enterprise related testing that Intel does.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3970 2015-01-28 01:03:16

nikitoss
Member
Registered: 2015-01-28
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear friends,
After one of the kernel update, as i think, I have a strange problem.
Everything works fine, until the drivers are installed (latest version I tried was Catalyst Beta 14.11.2 Beta) after installing the display driver i has reboot loop.

Windows 8.1 Enterprice
Radeon 7970

supermicro ~ # uname -a
Linux supermicro 3.19.0-1-mainline #1 SMP PREEMPT Mon Jan 26 00:18:49 MSK 2015 x86_64 GNU/Linux

supermicro ~ # lspci |grep -i vga
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
82:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 620] (rev a1)

supermicro ~ # cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=89c5cccc-bc84-4e48-aa7a-4c2da2c3c0ff rw rootflags=subvol=root_subvolume intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0

supermicro ~ # cat /etc/modprobe.d/kvm.conf 
options kvm ignore_msrs=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

supermicro ~ # zcat /proc/config | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_KVM_VFIO=y

supermicro ~ # lsmod | grep vfio
vfio_iommu_type1       17118  1
vfio_pci               35525  2
vfio                   18477  6 vfio_iommu_type1,vfio_pci

supermicro ~ # cat /home/nikitos/vm_sources/w8q35/w8.sh
/usr/sbin/qemu-system-x86_64 \
--enable-kvm -M q35 -cpu host -balloon none \
-monitor none -display none -vga none -nographic \
-m 4096 -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin \
-boot menu=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=MSI.HD7970.3072.130104.rom \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=09:00.0,bus=pcie.0 \
-device vfio-pci,host=06:00.1,bus=pcie.0 \
-drive file=/images/vdi/w8system0.qcow2,if=virtio,format=qcow2 \
-rtc base=localtime

P.S. Sory for my English

Offline

#3971 2015-01-28 02:20:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've seen this more than once now, rombar=0,romfile=/path/to/rom is a bad idea.  What this does is to disable the PCI ROM BAR, but specify via the device a ROM to load.  Sound strange?  It is.  QEMU will then place the ROM into the "genroms" section, which disassociates it from the device.  The ROM then has to go find the device itself, which may be a pretty common thing for it to do, but is really just a pointless exercise imposed by the user.  The only time to use rombar=0 with vfio-pci is to disable the device from having a ROM BAR at all.  If you simply pass romfile=/path/to/rom it will override the physical ROM.  There is absolutely no need to both disable the ROM BAR and still provide a ROM.  </psa>


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3972 2015-01-28 02:37:29

rman
Member
From: United States
Registered: 2013-08-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to report an OVMF success story. For awhile, I had been running with a standard VGA passthrough setup, but this weekend I decided to experiment with OVMF following the instructions.

Works perfectly. Windows 8.1 runs flawlessly in EFI mode with TianoCore. I can now use the stock Arch kernel and I have DRI on the host iGPU. smile

Kernel: 3.18.2-2-ARCH
CPU: Intel Core i5-4690
Host GPU: Intel HD Graphics 4600
Guest GPU: AMD Radeon R7 265 (rebranded HD 7850)

I will say that this bit,

http://vfio.blogspot.com/2014/08/primary-graphics-assignment-without-vga.html wrote:

If you remove the "Graphics" (ie. VNC/Spice) and "Video" (ie. VGA/QXL/Cirrus) devices from the VM, the assigned GPU will be the primary display.

didn't work for me; with emulated graphics disabled, the VM ran headless and did not use the attached GPU.


Laptop: Lenovo L440, Intel Core i3-4000M, HD Graphics 4600
Desktop: Intel Core i5-4690, HD Graphics 4600, AMD Radeon R7 265 (KVM VGA passthrough)

Offline

#3973 2015-01-28 02:41:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rman wrote:

I will say that this bit,

http://vfio.blogspot.com/2014/08/primary-graphics-assignment-without-vga.html wrote:

If you remove the "Graphics" (ie. VNC/Spice) and "Video" (ie. VGA/QXL/Cirrus) devices from the VM, the assigned GPU will be the primary display.

didn't work for me; with emulated graphics disabled, the VM ran headless and did not use the attached GPU.

How sure are you that your GPU has a UEFI ROM?  AIUI, AMD GPUs often work as a secondary device regardless of VGA support.  If it's not running as a primary, it's possible you might simply have been able to drop x-vga=off and let the driver initialize it w/o VGA.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3974 2015-01-28 02:55:24

rman
Member
From: United States
Registered: 2013-08-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

How sure are you that your GPU has a UEFI ROM?

I just ran your rom-parser program and indeed it seems to lack an UEFI ROM, despite being a relatively recent card.

Valid ROM signature found @0h, PCIR offset 22ch PCIR: type 0, vendor: 1002, device: 6819, class: 030000 PCIR: revision 0, vendor revision: f2c Last image

So I'm guessing that the setup works because the card doesn't need VGA initialization.

EDIT: The Radeon card is definitely being used as a secondary graphics card.

Last edited by rman (2015-01-28 03:06:57)


Laptop: Lenovo L440, Intel Core i3-4000M, HD Graphics 4600
Desktop: Intel Core i5-4690, HD Graphics 4600, AMD Radeon R7 265 (KVM VGA passthrough)

Offline

#3975 2015-01-28 15:01:15

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

It seems that DMAR issues made it into 3.18.3 too .

3.18.3 contains 2 IOMMU commits :

iommu/vt-d: Fix dmar_domain leak in iommu_attach_device
iommu/vt-d: Fix an off-by-one bug in __domain_mapping()

Which one do you think is to blame ?

Offline

#3976 2015-01-28 15:40:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

It seems that DMAR issues made it into 3.18.3 too .

3.18.3 contains 2 IOMMU commits :

iommu/vt-d: Fix dmar_domain leak in iommu_attach_device
iommu/vt-d: Fix an off-by-one bug in __domain_mapping()

Which one do you think is to blame ?

By that, do you mean that it is not present in 3.18.2?  I'm familiar with the first patch, I doubt it's to blame.  If you can revert one of those patches and toggle the problem let me know and I'll investigate further.  Easiest way to do a quick revert test is git show <commit ID> | patch -p1 -R  It's also possible that the problem is a side-effect of other code changes.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3977 2015-01-28 19:59:06

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, here's an update, in case anyone is interested.  The cd line that allows host access is now

-drive file=/dev/sr0,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0

Why qemu now requires that instead of the previous

 -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk 

is beyond me. I presume that is a recent qemu development not an arch vs Slackware thing.

Speaking of which, of course the pulseaudio warning was eliminated by my installing pulseaudio.. heh heh

However, the bluescreen with error 0x0000007b continues.  Works without passthrough under arch... works under qemu 2.0.0 under Slackware, but not passthrough under arch.  Clearly the screen gets passed through, as I get to see all the futile Windows recovery options.  Perhaps if I uninstall the nvidia driver from Windows while not passing through, and then, assuming I can boot Windows passedthrough with the default Windows VGA driver I can reinstall the NVIDIA driver.  Supposedly 0x0000007b is a drive problem though.

Any other insight into the problem would be welcome.

Last edited by mostlyharmless (2015-01-28 19:59:50)

Offline

#3978 2015-01-28 20:08:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Hmm, here's an update, in case anyone is interested.  The cd line that allows host access is now

-drive file=/dev/sr0,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0

Why qemu now requires that instead of the previous

 -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk 

is beyond me. I presume that is a recent qemu development not an arch vs Slackware thing.

The difference is if=none.  Without that QEMU assumes an interface for the device, then when you attempt to use the same interface later it's already in use.

Speaking of which, of course the pulseaudio warning was eliminated by my installing pulseaudio.. heh heh

However, the bluescreen with error 0x0000007b continues.  Works without passthrough under arch... works under qemu 2.0.0 under Slackware, but not passthrough under arch.  Clearly the screen gets passed through, as I get to see all the futile Windows recovery options.  Perhaps if I uninstall the nvidia driver from Windows while not passing through, and then, assuming I can boot Windows passedthrough with the default Windows VGA driver I can reinstall the NVIDIA driver.  Supposedly 0x0000007b is a drive problem though.

Any other insight into the problem would be welcome.

If you only get the problem when you attempt to pass through the host drive like this, then apparently it's related to the drive...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3979 2015-01-28 20:57:11

dhiru1602
Member
Registered: 2015-01-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

I am in need of desperate help. I have followed all of the posts on this topic and I am stuck with no solution to my problem. I was able to passthrough my GPU in Linux guest but on Windows guests I get Code 43.

Current Setup:
Core I7 920
Intel Desktop Board DX58SO (No Integrated Graphics)
Host GPU: Nvidia 9600 GT
Guest GPU: Nvidia 9600 GT

The Intel Processor page doesn't show support for VT-d, but instead mentions that the processor supports VT-x. I have passed "intel_iommu=on"  in the commandline to check if IOMMU worked and it did. This is when I have decided to give this method a try.

$ dmesg | grep "IOMMU" [ 0.000000] Intel-IOMMU: enabled [ 0.022507] dmar: IOMMU 0: reg_base_addr fe711000 ver 1:0 cap c9008010e60262 ecap f0207a [ 0.022512] dmar: IOMMU 1: reg_base_addr fe710000 ver 1:0 cap c90780106f0462 ecap f020fa [ 0.468207] IOMMU 0 0xfe711000: using Queued invalidation [ 0.468210] IOMMU 1 0xfe710000: using Queued invalidation [ 0.468214] IOMMU: Setting RMRR: [ 0.468222] IOMMU: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeefff] [ 0.468245] IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4a1000 - 0xbf4a1fff] [ 0.468264] IOMMU: Setting identity map for device 0000:00:1a.2 [0xbf4ac000 - 0xbf4acfff] [ 0.468282] IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4aa000 - 0xbf4aafff] [ 0.468300] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a7000 - 0xbf4a7fff] [ 0.468318] IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4a5000 - 0xbf4a5fff] [ 0.468335] IOMMU: Setting identity map for device 0000:00:1d.2 [0xbf4b0000 - 0xbf4b0fff] [ 0.468353] IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4af000 - 0xbf4affff] [ 0.468364] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4ad000 - 0xbf4adfff] [ 0.468373] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.468379] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

I have downloaded the kernel from the OP, built and installed it. I am using the following commandline:

intel_iommu=on pcie_acs_override=downstream

and confirmed that ACS override was enabled

[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

Since I had 2 identical cards, they had identical vendor and device ID's. If I used the pci-stub, it would end up stubbing both the graphic cards which will render my host useless.

$ lspci -nn | grep VGA 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation G94 [GeForce 9600 GT] [10de:0622] (rev a1) 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation G94 [GeForce 9600 GT] [10de:0622] (rev a1)

Thanks to @cjn! I was using his workaround to unbind the primary card after the pci-stub module has loaded.

$ cat /etc/modprobe.d/pci-stub.conf options pci-stub ids=10de:0622 install pci-stub /sbin/modprobe --ignore-install pci-stub $CMDLINE_OPTS; /usr/local/sbin/pci-stub-unbind 0000:02:00.0 softdep drm pre: pci-stub
$ cat /usr/local/sbin/pci-stub-unbind #!/bin/sh for dev in "$@" do echo $dev > /sys/bus/pci/drivers/pci-stub/unbind done

The first GPU is now used by the Nvidia 340 driver on the host and the GPU is assigned to pci-stub. I have then assigned the GPU to the vfio-pci driver and to confirm the same

03:00.0 VGA compatible controller: NVIDIA Corporation G94 [GeForce 9600 GT] (rev a1) (prog-if 00 [VGA controller]) Subsystem: XFX Pine Group Inc. Device 234b Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at c2000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at c0000000 (64-bit, non-prefetchable) [size=32M] Region 5: I/O ports at 2000 [size=128] Expansion ROM at <ignored> [disabled] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia

I have set the following module options.

options kvm ignore_msrs=1 options vfio_iommu_type1 allow_unsafe_interrupts=1 #options kvm_intel emulate_invalid_guest_state=0 [With this set, I had no VGA output even in Virt-Manager console]
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows7</name> <uuid>e9a9c56a-0708-4539-9d77-130283cbb354</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Nehalem</model> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/VMs/qemu/Windows7.qcow2'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='direct'> <mac address='52:54:00:91:20:1c'/> <source dev='enp0s25' mode='bridge'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0,multifunction=on'/> <qemu:arg value='-vga'/> <qemu:arg value='none'/> <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/> </qemu:commandline> </domain>

With all the above changes, I was able to boot into Ubuntu and install the Nvidia graphic drivers and everything worked fine. I was able to get the display out of the 2nd monitor. However, with Windows, I always get Code 43 error.

I have done extensive reading and I made sure that I was using <hidden state='on'/> and kvm=off to make sure that the newer Nvidia driver started properly. I have even tried using an older 304 Nvidia driver in the guest and it didn't change a thing.

While looking at the Host Kernel log, I was able to make a comparision.

Kernel log for Windows booting

<6>[ 3916.375945] vfio-pci 0000:03:00.0: enabling device (0400 -> 0403) <6>[ 3922.353501] kvm: zapping shadow pages for mmio generation wraparound

Kernel log for Linux booting

<6>[ 5821.639931] vfio-pci 0000:03:00.0: enabling device (0400 -> 0403) <6>[ 5827.818355] kvm: zapping shadow pages for mmio generation wraparound <3>[ 5839.699364] kvm [30427]: vcpu0 ignored rdmsr: 0x1c9 <3>[ 5839.699368] kvm [30427]: vcpu0 ignored wrmsr: 0x1c9 data 3 <3>[ 5839.699370] kvm [30427]: vcpu0 ignored rdmsr: 0x1c9 <3>[ 5839.699372] kvm [30427]: vcpu0 ignored rdmsr: 0x1a6 <3>[ 5839.699373] kvm [30427]: vcpu0 ignored wrmsr: 0x1a6 data 1ff <3>[ 5839.699375] kvm [30427]: vcpu0 ignored rdmsr: 0x1a6 <3>[ 5839.699378] kvm [30427]: vcpu0 ignored rdmsr: 0x3f6 <3>[ 5839.699380] kvm [30427]: vcpu0 ignored wrmsr: 0x3f6 data 1ff <3>[ 5839.699381] kvm [30427]: vcpu0 ignored rdmsr: 0x3f6 <7>[ 5844.410667] vfio-pci 0000:03:00.0: irq 36 for MSI/MSI-X

"vfio-pci 0000:03:00.0: irq 36 for MSI/MSI-X" doesn't occur with windows. Probably the interrupt's are not working?

Please advice.

Offline

#3980 2015-01-28 21:41:35

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw Interesting about the if=none; apparently a new requirement?

As to the other, I'm not passing through a drive, just my NVIDIA card. It passes through great under Slackware and the older qemu. The same VM with the video pass through stripped out boots fine under Arch and Slackware with their respective qemus.  The CD drive line has nothing to do with that either, as the same problem exists regardless of whether I include the CD host drive line in the qemu arguments.

@dhiru1602

Interesting, the intel ark http://ark.intel.com/search/advanced?s= … s&VTD=true

doesn't list the i7-920 as having Vt-d support, yet you apparently had success with a linux guest.

[Edit] I see the 920XM on this list http://ark.intel.com/search/advanced?s=t&VTD=true

Last edited by mostlyharmless (2015-01-28 21:56:22)

Offline

#3981 2015-01-28 21:57:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

By that, do you mean that it is not present in 3.18.2?  I'm familiar with the first patch, I doubt it's to blame.  If you can revert one of those patches and toggle the problem let me know and I'll investigate further.  Easiest way to do a quick revert test is git show <commit ID> | patch -p1 -R  It's also possible that the problem is a side-effect of other code changes.

Yes , those errors didn't happen with 3.18.2 . I noticed that a system upgrade that happened today pulled 3.18.3 . Again , DMAR issues + Code 43 in one Windows VM .

I didn't know that I can revert a patch from an already compiled \ running kernel ?! Is that what  git show <commit ID> | patch -p1 -R does ?

EDIT : Sorry , I'm just stupid . I'll pull the source and revert those patches one by one .

Thanks !

Last edited by Denso (2015-01-28 21:59:26)

Offline

#3982 2015-01-28 21:59:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I didn't know that I can revert a patch from an already compiled \ running kernel ?! Is that what  git show <commit ID> | patch -p1 -R does ?

No, that reverts it from the source, you'd need to rebuild.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3983 2015-01-28 23:08:06

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, that reverts it from the source, you'd need to rebuild.

Yup . I realized how stupid that question was , maybe I was dreaming of runtime dynamic patching / reverting !

By the way , did I go blind or is git REALLY pulling a 3.9GB tree ?! lol

http://i.imgur.com/H8JPe2c.png

Offline

#3984 2015-01-28 23:35:51

nythrix
Member
Registered: 2015-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nythrix wrote:

Thanks to the awesome people helping out in this thread I've been able to successfully pass through a 9600GT to a Windows 7 guest.

Unfortunately, linux guests don't work. I tried my oldish Debian VM, which has been running on "-vga std" since forever but I loose graphics output around the time the console switches fonts during boot (not sure how to describe it). Everything happens very quickly so the last messages I'm able to make out are stuff about intel RAPL and something about BARs ..? I wiped out the VM and reinstalled Debian but the problem persisted. The installation itself was flawless though (with passthrough). Twice.

Then I tried installing Fedora 21 Workstation. Couldn't get much further than some sort of greeting menu. It seems the signal is again lost during attempts to fiddle with the graphics/framebuffer.

Is this some sort of device reset problems I'm observing?

PS: I'm running 3.17 with the i915 arbiter patch. Qemu 2.1.2.

For future reference:
Nvidia 9600GT works great with Windows 7 guests (w/o additional ROM file) but not with Linux guests. I plugged in a GTX 460 for those and it worked (requires ROM and guest restart is still kinda hairy).
My motherboard is a Gigabyte GA-Z97X-UD3H (rev. 1.0, bios F7). Offers two usable IOMMU groups (i.e. you can safely run 2 independently accelerated VMs w/o the ACS hack): CPU-based, PCIe v3.0 slots in x16 or x8+x8 configurations and PCH-based, PCIe v2.1 slots in x4 or x1+x1+x1 configurations. The primary graphics can be the iGFX or a dGPU plugged into any physical PCIe x16 slot or even the friggin legacy PCI slot (hanging of the PCH-based IOMMU group). Any RIVA TNT owners should be satisfied with this one smile
Everything is powered by an i5-4590S. I still think AMD is way superior when it comes to virtualization. I hate Intel's market segmentation and purposeful technology crippling. Were it not for the atrocious TDP, I would've rather gone for an FX-8xxx.
Kernel 3.17 + VGA patch.
Qemu 2.1.2.

My launcher script:

#!/bin/bash vfiobind() { pci_addr="$1" vendor=$(cat /sys/bus/pci/devices/$pci_addr/vendor) device=$(cat /sys/bus/pci/devices/$pci_addr/device) if [ -e /sys/bus/pci/devices/$pci_addr/driver ]; then echo $pci_addr > /sys/bus/pci/devices/$pci_addr/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci vfiobind 0000:01:00.0 # PCI Express x16 slot #vfiobind 0000:01:00.1 #vfiobind 0000:02:00.0 # PCI Express x8 slot #vfiobind 0000:05:00.0 # PCI Express x4 slot #vfiobind 0000:05:00.1 export QEMU_AUDIO_DRV=pa pulseaudio --start pacmd set-sink-mute 1 false pacmd set-sink-volume 1 8192 qemu_args="" qemu_args=$qemu_args" -machine type=q35,accel=kvm" qemu_args=$qemu_args" -cpu host" qemu_args=$qemu_args" -smp cpus=3,cores=1,threads=1" qemu_args=$qemu_args" -boot order=dc,menu=on" qemu_args=$qemu_args" -m size=10G" qemu_args=$qemu_args" -soundhw hda" qemu_args=$qemu_args" -device ahci,id=ahci-id" #qemu_args=$qemu_args" -device ide-cd,bus=ahci-id.0,drive=CD-id" qemu_args=$qemu_args" -device ide-hd,bus=ahci-id.1,drive=HD-id" qemu_args=$qemu_args" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=ioh3420-id" qemu_args=$qemu_args" -device vfio-pci,host=01:00.0,bus=ioh3420-id,addr=00.0,multifunction=on,x-vga=on" qemu_args=$qemu_args" -usb" qemu_args=$qemu_args" -nographic" #qemu_args=$qemu_args" -drive if=none,id=CD-id,file=/dev/cdrom" qemu_args=$qemu_args" -drive if=none,id=HD-id,file=./vm.img" #qemu_args=$qemu_args" -vga none" qemu_args=$qemu_args" -net nic" qemu_args=$qemu_args" -net user,restrict=on" qemu_args=$qemu_args" -enable-kvm" qemu_args=$qemu_args" -nodefaults" #qemu_args=$qemu_args" -runas user" qemu-system-x86_64 $qemu_args

I'm still fighting with access permissions here and there. "-runas user" doesn't really help.

Offline

#3985 2015-01-29 15:05:48

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, so removing the NVIDIA driver from within Windows didn't change the 0x0000007b error.  Still mystified. Here's more data:

qemu under Slackware is qemu 2.1.0, under Arch qemu 2.2.0 

Under qemu 2.2.0, removing the

-drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio 

line allows the system to boot from the install CD for Windows 7... 

Something has changed in qemu's hard disk handling, akin to the if=none for the CD, perhaps with virtio?

but not from a Windows XPSP2 install disk that fails with bluescreen error 0x000000A5 Bios not fully ACPI complaint. Pressing F7 to bypass ACPI leads to the bluescreen with 0x0000007b.

Now let's see what happens under Slackware.... nah same thing, obviously that's a red herring secondary to WinXPSP2.  Hmm, though it runs under libvirt through under both Slackware and Arch, maybe that's a -M q35 thing since libvirt defaults to 440.

Last edited by mostlyharmless (2015-01-29 15:13:55)

Offline

#3986 2015-01-29 17:53:53

vx
Member
Registered: 2015-01-29
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

Thanks for this thread. Been playing with this for about a week. I got passthrough working on the first day, but been having one big issue since then...

Windows installs fine. The first time I get to the desktop, it works fine. I can reboot multiple times and it brings me back to the desktop, until any one of the following happens:

1) I get near the the end of the catalyst driver installation
2) Windows power saving kicks in and turns off the monitor connected to the passthrough GPU
3) The host!!! monitor turns off because I haven't touched that keyboard in a while

When any of these happens, the passthrough monitor turns off, and can't be turned back on by touching the passthrough keyboard/mouse. If I kill the qemu process and restart it, it shows me SeaBIOS, the windows splash screen, and then the passthrough monitor goes off again. Can't recover from this point without reinstalling windows, and then I'm back to square one.

Apologies if this has been covered already. The closest I can find are a few posts by syllopsium and firewing1 way back on page 37. syllopsium eventually used the NoSnoop patch, which apparently has been deprecated since, so I'm not sure if to try it.

I've been trying a bunch of different stuff (detailed below), and literally every working combination that gets the VM to start ends up with the same issue. Any help or guidance on what to try next would be appreciated. Thanks in advance.

Setup:
I got here via this Puget Systems how-to: http://www.pugetsystems.com/labs/articl … 4-KVM-585/, so my configuration is somewhat based on their instructions, but I have since updated to a newer Ubuntu version & kernel, applied the i915 arbitration patch and am using qemu 2.1.0.

Problem was the same in previous setup (14.04 distro / 3.13 kernel / qemu 2.0). Only the arbitration patch made a small difference. My host monitor does not turn blue now when I start the VM. Host monitor just displays Ubuntu Server console, no X. Could not having X installed be a problem?

GPU ROM was dumped using instructions from the virtio blog. It doesn't have a UEFI bios, so OVMF is not an option. Whether the romfile is passed to qemu or not makes no difference.

I've tried with both the q35 and 440fx machines (see below for qemu params). Same problem on both of them.

I'm using the Windows 8.1 90 day eval to make sure it works before I buy it. Not sure if this will make a difference from the retail version. Or maybe I need to use an older version instead? Have not tried that.

I've not tried attaching a cirrus adapter and installing catalyst over VNC yet. Since it happens whether catalyst is installed or not, I'm guessing this is not the issue. Will try it later today anyway though.

qemu startup (q35):

sudo qemu-system-x86_64 \ -enable-kvm \ -daemonize \ -M q35 \ -m 4096 \ -cpu host \ -smp 2,sockets=1,cores=2 \ -vga none \ -display none \ -serial none -parallel none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/r9270.rom \ -device vfio-pci,host=0000:01:00.1,bus=root.1,addr=00.1 \ -usb -usbdevice host:046d:c52e -usbdevice host:1bcf:0007 \ -drive file=$HOME/kvm/gaming.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=$HOME/win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot order='dc' \

qemu startup (440fx):

sudo qemu-system-x86_64 \ -enable-kvm \ -daemonize \ -M pc \ -m 4096 \ -cpu host \ -smp 2,sockets=1,cores=2 \ -vga none \ -display none \ -serial none \ -parallel none \ -device vfio-pci,host=0000:01:00.0,multifunction=on,x-vga=on,romfile=$HOME/r9270.rom \ -device vfio-pci,host=0000:01:00.1 \ -usb -usbdevice host:046d:c52e -usbdevice host:1bcf:0007 \ -drive file=$HOME/kvm/gaming.img,media=disk,format=raw \ -drive file=$HOME/win8.iso,media=cdrom \ -boot order='dc' \

Hardware:
CPU:                 Intel i5 3470S - http://ark.intel.com/products/68315/Int … o-3_60-GHz
Motherboard:    MSI H77MA-G43 - http://www.msi.com/product/mb/H77MA-G43 … o-overview
GPU Host:        Intel HD 2500 from CPU
GPU VM:          Sapphire Radeon R9 270 - http://www.sapphiretech.com/presentatio … 2089&lid=1

Software:
Host OS:           Ubuntu 14.10 Server
Kernel:              3.16.0-29 (+i915 arbitration patch)
qemu version:   2.1.0
Guest OS:         Windows 8.1 (90 day eval)

Offline

#3987 2015-01-29 17:59:29

Wooots
Member
Registered: 2015-01-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there

I've read this article and some others and got my Nvidia working at the moment using the following command:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 6,sockets=1,cores=6,threads=2 -vga none -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 -drive file=/opt/VirtualMachines/Passthrough-test/win7.qcow2,id=disk,format=qcow2,if=none -device ide-hd,drive=disk -usb -device usb-host,vendorid=0x046d,productid=0xc531 -usb -device usb-host,vendorid=0x046d,productid=0xc24d

But I have one little question do you all start qemu as root? Isn't this a security problem?

Last edited by Wooots (2015-01-29 18:17:24)

Offline

#3988 2015-01-29 18:46:20

dhiru1602
Member
Registered: 2015-01-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was finally able to get the passthrough working on Windows but I had to use a dirty hack to start the VM everytime.

4sOjjB5s.png

As I have mentioned earlier, my GPU was working fine in passthrough on Linux but I got Code 43 error on Windows. This was because I don't get ""vfio-pci 0000:03:00.0: irq 36 for MSI/MSI-X" on Windows.

I created a Linux Virtual Machine and after booting it, whenever I got the irq message in the kernel log, I would immediately turn off the Linux VM and boot the Windows VM. The GPU gets detected and works perfectly on Windows without Code 43. I have even installed the latest 340 nvidia driver and games work great. However, once I shutdown or reboot the VM, I will have to start the Linux VM first and force stop it when the interrupt line is registered and then boot the windows virtual machine. This works like about 5 out of 10 times and it's really a pain.

Any thoughts on what might be preventing the interrupts from working on a Windows guest?

mostlyharmless wrote:

@dhiru1602

Interesting, the intel ark http://ark.intel.com/search/advanced?s= … s&VTD=true

doesn't list the i7-920 as having Vt-d support, yet you apparently had success with a linux guest.

[Edit] I see the 920XM on this list http://ark.intel.com/search/advanced?s=t&VTD=true

Yes. Interestingly enough, the passthrough works even without the patched ACS kernel.

Last edited by dhiru1602 (2015-01-29 18:49:53)

Offline

#3989 2015-01-29 21:26:00

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a good post where someone converted from a QEMU command to an XML file? I'm having issues getting my xml file created. I have no issues with multiple device passthrough, GTX760/USB/Ethernet/Creative Sounds Blaster Z, I just want to be able to manage using an XML and LibVirt/Virt-Manager

Offline

#3990 2015-01-29 21:53:12

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wooots;  I usually run as root or use sudo; I'm pretty sure somewhere in the last 160 pages there's some detail on how to fudge the permissions to avoid that...

@The_Moves on that note, see page 94

Offline

#3991 2015-01-30 00:10:20

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using Fedora 21 and Virt-Manager, I successfully installed a VM with UEFI/OVMF by editing this:

/etc/libvirt/qemu.conf
nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

Everything works flawlessly except my VM is NOT saving EFI Boot entry (Ubuntu/Fedora for now, later Windows). What could be the problem? I tried manually adding "by file" but on reboot it resets and forgets the entry. Also tried "sudo grub-install" in Ubuntu so Ubuntu makes the entry with no success either.

Please help!

Last edited by devianceluka (2015-01-30 00:12:21)

Offline

#3992 2015-01-30 12:01:44

maxexcloo
Member
From: Australia
Registered: 2009-10-14
Posts: 177
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it at all possible to use the dedicated card on my optimus enabled laptop as a passthrough? I'm thinking it could be possible (although probably not implemented) as the output from the card could then be copied to the normal output (as optimus usually works), providing an awesome experience.

(Just thinking about it, unless the card is different somehow or has some feature disabled why isn't it possible?)

Offline

#3993 2015-01-30 12:48:29

fatfingers
Member
Registered: 2015-01-30
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:

Tried to get it working for me too, but with no success. sad

My system:
Radeon 7950
i5 3470 (non-K) incl Intel GPU for Host
AsRock Z77 Extreme4
8GB Ram

First thing I was trying was the old way, with pci-assign. I was able to first boot with VNC, installing ATI drivers. After reboot the screen woke up at login prompt and the Performance was ok/good (I wasn't trying too much. Windows Performance Index at 7.5, so it should have worked wink). But at reboot of VM the host crashed....

So I was looking around the internet and found this thread. Tried many hours to get it working, but I don't get any picture to the screen. No Seabios Texts and nothing in Windows after installing ATI drivers with vnc (and removing VNC again) like it was working with pci-assign.

Have tried Qemu-git from AUR, Qemu-1.5-rc1, seabios from your link, stock kernel (3.9.2-1) and kernel 3.10-rc1.
Did not change anything wink.

The only thing I havn't tried till now is to dump the VGA bios and put it in. Will do this today, when I´m at home.

IOMMU is enabled. logfiles look good (for me big_smile). The only thing is "DMAR: No ATSR found". You have a Idea about this?
Full logfiles I can add later. 

Bind everything to vfio-pci seems to work. Also there isn't any error message at KVM-Start
I also tried all with pci-stub entries at grub and without, same with the interrupt remapping entry.

Do you have any Idea? big_smile

Thanks
apo

---------------------------------------------------------------------------------------------------------------------------------
I have the same mainboard except its the extreme4-M.

Brief setup ,kernel 3.18 mainline -recompiled,NO ACS PATCH ,went back to distro re-compiled kernel as issues with mainline.
ati 7950 and nvidia q570,xeon 1245v2
ATI is the boot card which then swaps to the nvidia.
ATI on SLOT1 and the Nvidia on slot3
No Xorg.conf
Nouveau on nvidia
Non UEFI / legacy kernel 
Using the cards extracted Rom ,and the Rom Bios option in the qemu config .


Some testing with the bridges as i did have a fully working pci passthrough till ubuntu went from
13.12 to 14.04 and xen to 4.4 i have had issues ,not sure if its xorg or the kernel,still testing
and havent had passthrough in xen since 4.3 and earlier kernels,hence the testing with
qemu again.xen does utilise the qemu platform.
xen 4.5 has no pci passthrough,and iam trying to get it working on qemu.

eg:

chmod 660 /sys/bus/pci/devices/0000\:00\:01.0/driver/new_id
chmod 660 /sys/bus/pci/devices/0000\:00\:01.0/driver/unbind

echo 8086 0151 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id


I also detach and re-attach in the qemu script

virsh nodedev-reattach pci_8086_1e31
virsh nodedev-detach pci_8086_1e31
virsh nodedev-detach   pci_8086_1e31 # bridge # Be careful ! this can cause major issues if the groups of the ati card and the bridge are not on their own.
virsh nodedev-reattach pci_8086_1e31 # bridge # Be careful ! this can cause major issues if the groups of the ati card and the bridge are not on their own.

# Grub Linux Default
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on,pt,igfx_off vfio_iommu_type1.allow_unsafe_interrupts=1  \
kvm_intel.emulate_invalid_guest_state=0 radeon.blacklist=1 radeon.modeset=0 rd.driver.blacklist=radeon rd.blacklist=radeon"

I also have a modified xorg.conf to ignore and remove fbdev and vesa from attaching itself via Xorg ( takes over).
Also removed xorg ati and xorg fglrx from the packages.

I know this isnt a xen forum but i have found that pciback does bind to the bridge although i dont pass it through.

Regards


I followed some of the intel vfio ddk ,fortunately the machine didnt fall over like i expected when unbinding this bus,intel vfio ddk -notes page 14

Last edited by fatfingers (2015-01-31 13:04:45)

Offline

#3994 2015-01-30 15:55:43

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

@Wooots;  I usually run as root or use sudo; I'm pretty sure somewhere in the last 160 pages there's some detail on how to fudge the permissions to avoid that...

@The_Moves on that note, see page 94


I've gone through page 94, however my goal is to follow Alex's lead and get rid of all 'qemu' calls. I'll have some dedicated time this weekend to sit and figure it out, but anything to reference will help.

Offline

#3995 2015-01-30 19:33:19

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all i have some troubles with my KVM host computer. I dont know what can happened. So sorry for my english.

I using KVM VGA passthrough for some time i think more than six mounths. All works fine but sometimes i see only some error msq with libusb in console where i run my qemu image. I think this libusb problems is not too much big. I have usb mouse, keyobard, webcam, flashdrive and usb soundcard, usb network card attach to my qemu virtual machine. In virtual machine runs Windows 7 and here is my setting for qemu kvm.

sudo vfio-bind 0000:01:00.0 0000:01:00.1 && sudo qemu-system-x86_64 -rtc clock=host -cpu host,hv-time -smp 4,sockets=1,cores=2,threads=2 -M q35 -enable-kvm -m 5120 -bios /usr/share/qemu/bios.bin -vga none -device ahci,bus=pcie.0,id=ahci -drive file=/home/back/KVM/windows.iso,id=isocd -device ide-cd,bus=ahci.0,drive=isocd -drive file=/home/back/KVM/virtio.iso,id=isocd2 -device ide-cd,bus=ahci.1,drive=isocd2 -drive file=disk-preallc.qcow2,if=virtio -drive file=/media/back/btrfs-test400/ntfsDisk.qcow2,if=virtio -drive file=/media/back/btrfs-test400/ntfsDisk-preallocated.qcow2,if=virtio -drive file=/media/back/btrfs-test400/ntfs-100-Disk-preallocated.qcow2,if=virtio -boot menu=on -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off,romfile=vbios2.rom,x-vga=on -device vfio-pci,host=01:00.1,bus=pcie.0 -usb -usbdevice host:13ba:0018 -usbdevice host:093a:2521 -usbdevice host:1a2c:0002 -usbdevice host:9710:7830 -usbdevice host:0d8c:000c -usbdevice host:041e:4064

Three day ago i shutdown PC becouse i want to clean up the room and relocate table to near window-I have dark room. I plug off all cables from back (Monitor, mouse, power..). With this opportunity i want to try my older VGA card if still working or not. For this i go to BIOS to change primary VGA output from intelGFX(on board) to PCIe card. SHUTDOWN and remove my GTX 560 from PCIe slot and istead put older VGA card and then. Putt the all cables back to PC and POWERON ... i saw POST output on LCD i open BIOS and change primary graphic to intelGFX and shutdown PC. Remove old but workin VGA card and insert back my GTX 560 card.

Boot computer ubuntu 14.04 start and then i open browser. Site not found? whats wrong. Ohh then i see my ethernet card ETH0 missing in ifconfig. I fast reboot PC just for test. And my card still not registred like eth0. I see only eth1-this is my USB network card for qemu machine. I run my virtual machine they boot verry slow. I also try next restart of PC but pc verry slow boot and hangs. I must more time reboot pc. Now PC boot work but somethimes freeze and network card still not work. Also not blink leds and also lspci dont see this network card-like no exist. I think integrated on board network card is now die- maybe my motherboard go die.

at this moment i run dmesg and see too much this lines..

[34913.005352] DMAR:[fault reason 06] PTE Read access is not set [35395.484547] dmar: DRHD: handling fault status reg 3 [35395.484553] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 [35395.484553] DMAR:[fault reason 06] PTE Read access is not set [35395.957843] dmar: DRHD: handling fault status reg 3 [35395.957851] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 [35395.957851] DMAR:[fault reason 06] PTE Read access is not set [36830.943853] dmar: DRHD: handling fault status reg 3 [36830.943858] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 [36830.943858] DMAR:[fault reason 06] PTE Read access is not set [37196.245627] dmar: DRHD: handling fault status reg 3 [37196.245633] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 

still increasing

Yesterday i backup all files in /var/log/ but i am not guru for this. In syslog.3 there is last information about module atl1c -this was network module for my integrated network card.
There are syslog files http://www.mediafire.com/download/57314 … yslogs.zip
I have also PCI NETWORK CARD which i use at this moment instead integrated.

This is my lspci output:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.5 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) 00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1) 03:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03) 04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10) 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

kernel:

Linux KVM-Ubuntu 3.15.1 #1 SMP Thu Jun 19 00:44:25 CEST 2014 x86_64 x86_64 x86_64 GNU/Linux

   
CPU: i7 3770
MB: Gigabyte Z77X-D3H ver. 1.0
VGA: Asus GTX 560 DCII
------------------------------------------
RAM: 12Gb=3*4Gb Crucial (i dont know which one)
SSD1: intel 530 256Gb  (Ubuntu 14.04)
SSD2: samsung evo 840 256Gb (using only for qemu images)
HDD3: junk
-------------------------------------------

I think this can be related with virtualization, bios or just my mainboard is broken.

Offline

#3996 2015-01-30 22:16:34

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[ 250.726291] ------------[ cut here ]------------ [ 250.730914] kernel BUG at drivers/pci/ats.c:138! [ 250.735535] invalid opcode: 0000 [#1] PREEMPT SMP [ 250.740379] Modules linked in: vfio_pci vfio_iommu_type1 vfio iTCO_wdt iTCO_vendor_support ext4 crc16 mbcache jbd2 mxm_wmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd pcspkr serio_raw sb_edac edac_core igb joydev mousedev evdev mac_hid lpc_ich i2c_i801 ptp pps_core hwmon snd_hda_intel i2c_algo_bit snd_hda_controller i2c_core snd_hda_codec mei_me mei snd_hwdep snd_pcm snd_timer snd soundcore tpm_tis ioatdma tpm dca wmi processor shpchp button sch_fq_codel zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common atkbd libps2 megaraid_sas ahci isci libahci libsas ehci_pci ehci_hcd xhci_hcd libata mpt2sas raid_class scsi_transport_sas usbcore scsi_mod usb_common i8042 serio bridge stp llc vhost_net tun vhost macvtap macvlan [ 250.824223] CPU: 0 PID: 1305 Comm: qemu:win171 Tainted: P O 3.17.8-2-ARCH #1 [ 250.832308] Hardware name: Supermicro X9DA7/E/X9DA7/E, BIOS 3.0a 07/02/2014 [ 250.839270] task: ffff882027203250 ti: ffff880c3facc000 task.ti: ffff880c3facc000 [ 250.846752] RIP: 0010:[<ffffffff812dd5ff>] [<ffffffff812dd5ff>] pci_restore_ats_state+0x6f/0x80 [ 250.855562] RSP: 0018:ffff880c3facfd80 EFLAGS: 00010246 [ 250.860885] RAX: 0000000000000000 RBX: ffff8820283cb000 RCX: 0000000000000ffc [ 250.868019] RDX: 00000000000000ff RSI: 0000000000000246 RDI: 0000000000000246 [ 250.875163] RBP: ffff880c3facfd88 R08: 0000000000000004 R09: ffff880c3facfd04 [ 250.882309] R10: 0000000000000000 R11: 0000000000000246 R12: ffff8820283cb000 [ 250.889443] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000 [ 250.896572] FS: 00007f7df0a78940(0000) GS:ffff88103fc00000(0000) knlGS:0000000000000000 [ 250.904675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 250.910417] CR2: 00007f79437fdc88 CR3: 0000002027240000 CR4: 00000000001427f0 [ 250.917562] Stack: [ 250.919579] 0000000000000000 ffff880c3facfda8 ffffffff812bde59 ffff8820283cb000 [ 250.927035] ffff88202844ac28 ffff880c3facfdc0 ffffffff812be0a5 ffff8820283cb000 [ 250.934510] ffff880c3facfde0 ffffffff812be0d8 ffff88202844ac00 ffff880c3facfe50 [ 250.941967] Call Trace: [ 250.944426] [<ffffffff812bde59>] pci_restore_state.part.31+0x29/0x210 [ 250.950951] [<ffffffff812be0a5>] pci_dev_restore+0x45/0x50 [ 250.956543] [<ffffffff812be0d8>] pci_bus_restore+0x28/0x50 [ 250.962120] [<ffffffff812c028e>] pci_try_reset_bus+0x3e/0x70 [ 250.967895] [<ffffffffa04333bc>] vfio_pci_ioctl+0x9ac/0xa00 [vfio_pci] [ 250.974508] [<ffffffffa04323b0>] ? vfio_pci_mmap+0x1f0/0x1f0 [vfio_pci] [ 250.981239] [<ffffffff811ea3d8>] ? fsnotify+0x228/0x2f0 [ 250.986552] [<ffffffffa0421283>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] [ 250.993708] [<ffffffff811c01d0>] do_vfs_ioctl+0x2e0/0x4c0 [ 250.999192] [<ffffffff811ca08e>] ? __fget+0x6e/0xb0 [ 251.004175] [<ffffffff811c0431>] SyS_ioctl+0x81/0xa0 [ 251.009239] [<ffffffff81502129>] system_call_fastpath+0x16/0x1b [ 251.015263] Code: 8b 73 38 48 8b 7b 10 83 c2 06 e8 4d a4 fd ff 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f b7 70 04 8d 4e f4 83 e1 1f 80 cd 80 eb d3 <0f> 0b 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 [ 251.035239] RIP [<ffffffff812dd5ff>] pci_restore_ats_state+0x6f/0x80 [ 251.041697] RSP <ffff880c3facfd80> [ 251.045286] ---[ end trace ff4dde6dedc315f3 ]---

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

Last edited by Bronek (2015-01-30 22:21:58)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#3997 2015-01-30 22:32:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[ 250.730914] kernel BUG at drivers/pci/ats.c:138!

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

That's saying that the device had ATS (Address Translation Services) enabled before reset, but when we went to restore the state of it after reset, we couldn't even find the capability.  Maybe the device never really recovers from reset?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#3998 2015-01-30 23:23:13

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am attempting to follow aw's blog and have been following this thread on and off for a long time.

When attempting to follow this part of his instructions:

VFIO tips and tricks > Primary Graphics Assignment Without VGA... wrote:

Make the first line of the XML look like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

The xmlns specification is the part that needs to be added.  Next we need to add OVMF, just before the </domain> close at the end of the file, add something like this:

 <qemu:commandline> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/> </qemu:commandline>

Adjust the path as necessary if you're using a different installation.

I cannot find a file called OVMF-pure-efi.fd anywhere on my system. I have tried:

sudo find / -name 'OVMF*' /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_CODE.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_VARS.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_CODE.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_VARS.fd

Have I missed something in the installation of the ovmf-svn package? Or which of my installed files do I need to point it to?
P.S I am new to this game if my basic question didn't give it away.

Last edited by cdrjameson (2015-01-30 23:25:53)

Offline

#3999 2015-01-30 23:31:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cdrjameson wrote:

I am attempting to follow aw's blog and have been following this thread on and off for a long time.

When attempting to follow this part of his instructions:

VFIO tips and tricks > Primary Graphics Assignment Without VGA... wrote:

Make the first line of the XML look like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

The xmlns specification is the part that needs to be added.  Next we need to add OVMF, just before the </domain> close at the end of the file, add something like this:

 <qemu:commandline> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/> </qemu:commandline>

Adjust the path as necessary if you're using a different installation.

I cannot find a file called OVMF-pure-efi.fd anywhere on my system. I have tried:

sudo find / -name 'OVMF*' /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_CODE.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_VARS.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_CODE.fd /home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_VARS.fd

Have I missed something in the installation of the ovmf-svn package? Or which of my installed files do I need to point it to?


I'd try the latter set, but you're missing an update to those instructions here:

http://vfio.blogspot.com/2014/09/libvir … -ovmf.html

My slides from KVM Forum include a further updated setup (hit space a couple times):

http://awilliam.github.io/presentations … 2014/#/4/2

The "pure" name indicates that it doesn't include the CSM (Compatibility Support Module), which would provide the legacy support that we're trying to avoid.  Upstream and your distro provided packages may name it differently.  There are really very, very few reasons that anyone here should be using <qemu:arg> options any longer.  If you are, you're probably doing something wrong.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4000 2015-01-30 23:51:15

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There are really very, very few reasons that anyone here should be using <qemu:arg> options any longer.  If you are, you're probably doing something wrong.

Correct me if I'm wrong, but Win 7 guests and below who use libvirt will still use qemu:arg since apparently OVMF + EFI + Win 7 seems pretty fragile, at least from what I've read here.  So you're referring to Win 8+

Last edited by mostlyharmless (2015-01-30 23:51:29)

Offline

#4001 2015-01-30 23:52:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

There are really very, very few reasons that anyone here should be using <qemu:arg> options any longer.  If you are, you're probably doing something wrong.

Correct me if I'm wrong, but Win 7 guests and below who use libvirt will still use qemu:arg since apparently OVMF + EFI + Win 7 seems pretty fragile, at least from what I've read here.  So you're referring to Win 8+

No, you should be using a wrapper script around the qemu binary to add the x-vga=on option so that libvirt can manage the vfio device itself.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4002 2015-01-31 00:01:28

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm obviously I missed that somewhere, could you give a link to the page number? EDIT: never mind, found it post 3262... whoops too late thanks though

Last edited by mostlyharmless (2015-01-31 00:05:32)

Offline

#4003 2015-01-31 00:04:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Hmm obviously I missed that somewhere, could you give a link to the page number?

https://bbs.archlinux.org/viewtopic.php … 1#p1475541

EDIT: PS, letting libvirt manage the device is how you avoid needing to elevate permissions for the VM or run it as root.  You should not need to touch anything in /etc/libvirt/qemu.conf

Last edited by aw (2015-01-31 01:32:41)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4004 2015-01-31 03:25:11

dhiru1602
Member
Registered: 2015-01-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guys, GIVE THIS A READ.

Checking the Device Manager in my Windows guest, I have noticed that my GPU was not using MSI. I have followed the guide linked to Alex's blog post and I had made the registry changes in Windows and I was able to get around 10% improvement in benchmarks and the overall performance has definitely improved. I will link the benchmark results later.

Thanks "aw" for your hardwork!

Offline

#4005 2015-01-31 11:27:16

cdrjameson
Member
Registered: 2013-05-19
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

...you're probably doing something wrong.

You're absolutely right!

aw wrote:

I'd try the latter set, but you're missing an update to those instructions here:
http://vfio.blogspot.com/2014/09/libvir … -ovmf.html
My slides from KVM Forum include a further updated setup (hit space a couple times):
http://awilliam.github.io/presentations … 2014/#/4/2

Thanks for this direction Alex, I thought I had read most things but my comprehension of what it all means is somewhat lacking.
I have set up the VM using virt-manager and am trying to edit the domain xml. I have installed the ovmf-svn package from the AUR but I am confused to what I should edit these lines to:

<domain type='kvm'> ... <os> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/> ... </os> </domain>

I am confused because under /usr/share I do not have an edk2.git directory and:

 # find / -name edk2* 

returns nothing.

*Edit*

$ pacman -Ql ovmf-svn ovmf-svn /usr/ ovmf-svn /usr/share/ ovmf-svn /usr/share/ovmf/ ovmf-svn /usr/share/ovmf/ia32/ ovmf-svn /usr/share/ovmf/ia32/ovmf_code_ia32.bin ovmf-svn /usr/share/ovmf/ia32/ovmf_ia32.bin ovmf-svn /usr/share/ovmf/ia32/ovmf_vars_ia32.bin ovmf-svn /usr/share/ovmf/x64/ ovmf-svn /usr/share/ovmf/x64/ovmf_code_x64.bin ovmf-svn /usr/share/ovmf/x64/ovmf_vars_x64.bin ovmf-svn /usr/share/ovmf/x64/ovmf_x64.bin

Which of these, if any, do I need to point to in the domain xml list?

Last edited by cdrjameson (2015-01-31 11:44:16)

Offline

#4006 2015-01-31 14:40:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cdrjameson wrote:
$ pacman -Ql ovmf-svn ovmf-svn /usr/ ovmf-svn /usr/share/ ovmf-svn /usr/share/ovmf/ ovmf-svn /usr/share/ovmf/ia32/ ovmf-svn /usr/share/ovmf/ia32/ovmf_code_ia32.bin ovmf-svn /usr/share/ovmf/ia32/ovmf_ia32.bin ovmf-svn /usr/share/ovmf/ia32/ovmf_vars_ia32.bin ovmf-svn /usr/share/ovmf/x64/ ovmf-svn /usr/share/ovmf/x64/ovmf_code_x64.bin <--- CODE ovmf-svn /usr/share/ovmf/x64/ovmf_vars_x64.bin <--- VARS ovmf-svn /usr/share/ovmf/x64/ovmf_x64.bin

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4007 2015-01-31 20:46:54

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Bronek wrote:

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[ 250.730914] kernel BUG at drivers/pci/ats.c:138!

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

That's saying that the device had ATS (Address Translation Services) enabled before reset, but when we went to restore the state of it after reset, we couldn't even find the capability.  Maybe the device never really recovers from reset?

Got it. I suppose I should complain to AMD support ...


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4008 2015-02-01 03:13:38

spijet
Member
Registered: 2012-01-10
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Good morning everyone! First of all, thank you nbhs for your awesome KVM guide and giving me a hope of using my GPU with KVM! smile

However, I'm having a little problem here with my setup.

I've managed to 'unbind' the GPU from the host and do the vfio-bind stuff and it seems that QEMU picks it up and tries to use it, but I don't see anything on the screen: it's either "no signal" warning and then powersave mode, or just the black screen  (the monitor, however, acts as if there is something, no powersave).
When I tried to boot archiso in guest, it detected the GPU properly (when nouveau module had started). Everything else gives me a black screen: qemu-bios, OVMF, Windows, etc.
When I was playing with qemu options, I've got Windows to display its bootloader (but no bios) on the screen, but then it gave me a BSOD. big_smile After that, nothing again.

The only thing I think I'm doing wrong is that I'm trying to build a system with totally headless host - 1 GPU for guest only (I'm doing everything from my laptop over SSH).

Here's my config:

Motherboard: ASUS M5A97 (with fixed IVRS table -- see boot options) CPU: AMD FX-4100 RAM: 8GB DDR3, trying to pass 4-6GB to the VM GPU: Gigabyte NV-65TOC1GI (nVidia GeForce GTX 650Ti 1GB), VBIOS F21 Arch Linux x86_64, linux 3.18.4-1-ARCH Boot cmdline: dolvm root=/dev/mapper/roccatbase-baseroot rw ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 pci-stub.ids=10de:11c6,10de:0e0b video=efifb:off lspci: 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] [10de:11c6] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1) cat /etc/vfio-pci.cfg DEVICES="0000:01:00.0 0000:01:00.1" I use QEMU-git 2.2.r36653.ga46b3aa-1.

Here are some qemu scripts that I'm trying to boot.


Is there anything else I can do to make it work, or should I give up the idea of using only one GPU?


ADD: I've just boot Fedora 21 x86_64 live iso with my testvm5.sh script and it works flawlessly! I've added my USB keyboard and mouse to the script and now I can use it pretty much like if it was bare-metal installation. But still, I didn't see OVMF bios on my screen, instead it showed up in text console I was running testvm5.sh from. O_o

ADD2:I replaced 2 usb device entries in my script with one USB controller, which I'm passing through via VFIO-PCI. Now I can see the (possible?) cause of my problem.
In system logs, when the devices are initialised, I can see some vfio-pci messages about reserving IRQs for PCI devices. When I used Fedora ISO, I got this:

Feb 01 12:43:12 roccatbase kernel: vfio-pci 0000:03:00.0: enabling device (0400 -> 0402) Feb 01 12:43:15 roccatbase kernel: kvm: zapping shadow pages for mmio generation wraparound Feb 01 12:43:24 roccatbase kernel: kvm [807]: vcpu0 ignored rdmsr: 0xc0010112 Feb 01 12:43:24 roccatbase kernel: kvm [807]: vcpu0 unimplemented perfctr wrmsr: 0xc0010004 data 0xffff Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 34 for MSI/MSI-X Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 34 for MSI/MSI-X Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 35 for MSI/MSI-X Feb 01 12:43:26 roccatbase kernel: vfio-pci 0000:01:00.0: irq 36 for MSI/MSI-X

01:00.0 is the pci-id of my GPU and 03:00.0 is my USB controller. The GPU starts up shortly after I get the last message.

But when trying to boot Windows, I get:

Feb 01 13:22:39 roccatbase kernel: kvm: zapping shadow pages for mmio generation wraparound Feb 01 13:23:17 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X ... Feb 01 13:23:17 roccatbase kernel: vfio-pci 0000:03:00.0: irq 39 for MSI/MSI-X Feb 01 13:23:17 roccatbase kernel: vfio-pci 0000:03:00.0: irq 40 for MSI/MSI-X

And there's no pci-id of the GPU. The only thing I've changed was the cdrom entry.

Last edited by spijet (2015-02-01 05:30:10)

Offline

#4009 2015-02-01 09:24:35

nythrix
Member
Registered: 2015-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

3. Dude, check this out! A whole table with stats and graphics and stuff!
https://docs.google.com/spreadsheet/ccc … _web#gid=0

What the...? I am speechless! Why isn't this awesome list on the first page??? I spent months shopping for compatible hardware and never ever saw this! Damn you, gianormous thread! big_smile

On a side note: I fixed my audio cracking/popping issues by forcing the same format (16bit x 44100Hz) in both the guest and the host. Hooray!

Last edited by nythrix (2015-02-01 09:27:25)

Offline

#4010 2015-02-01 14:35:13

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.

Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?

Please help!

Offline

#4011 2015-02-01 14:36:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.

Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?

OVMF does not change anything about ACS.  ACS is a hardware isolation issue.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4012 2015-02-01 14:37:22

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:

I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.

Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?

OVMF does not change anything about ACS.  ACS is a hardware isolation issue.

I understand, but im 100% positive that I saw numerous times that with OVMF one does not need ACS override patch?

Offline

#4013 2015-02-01 14:38:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:
aw wrote:
devianceluka wrote:

I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.

Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?

OVMF does not change anything about ACS.  ACS is a hardware isolation issue.

I understand, but im 100% positive that I saw numerous times that with OVMF one does not need ACS override patch?

You are 100% wrong.  With OVMF you don't need the i915 VGA arbitration patch.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4014 2015-02-01 14:44:20

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
devianceluka wrote:
aw wrote:

OVMF does not change anything about ACS.  ACS is a hardware isolation issue.

I understand, but im 100% positive that I saw numerous times that with OVMF one does not need ACS override patch?

You are 100% wrong.  With OVMF you don't need the i915 VGA arbitration patch.

So your saying to passthrough bus 2 GPU and bus 3 GPU to 2 different VMs - it does not matter whether OVMF or legacy ?

Offline

#4015 2015-02-01 16:04:26

BlackDwarf
Member
Registered: 2015-02-01
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is possible to create vfio on Asrock Z68 Pro3, i5 3570 and G650TI?

Offline

#4016 2015-02-01 16:08:57

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BlackDwarf wrote:

Is possible to create vfio on Asrock Z68 Pro3, i5 3570 and G650TI?

Does your motherboard and CPU support IOMMU and VT-D?

http://ark.intel.com/products/65702/Int … o-3_80-GHz

It looks like your CPU does. Explore your BIOS to see if there is an option to enable VT-D for it.

Offline

#4017 2015-02-01 16:20:33

BlackDwarf
Member
Registered: 2015-02-01
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I have 2500k and will buy 3570.
Last bios and there is option VT-d Capability Unsupported. I think that's because my cpu (i5 2500k) is not supported.

Is possible to have problems?

Offline

#4018 2015-02-01 16:48:14

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, i've run into an issue. The ultimate goal of my system is to combine both my Gaming and FreeNAS systems into one. Over the last two days, with help from others here and a bunch of googling, i've been able to get the Gaming portion of my system to run. I have been able to successfully passthrough my USB Controller/Createive SB Z/Ethernet Controller and my  GTX 760 with no code 43. When I went to pass my  LSI RAID Card through, it complained about that my Ethertnet Adapter as already assigned. This makes sense as my Ethernet Adaoter,SBZ, and the LSI RAID Card are in the same IOMMU Group. I was hoping that multiple devices in a single IOMMU Group could be shared out to different VMs, but this doesn't appear to be the case - correct?

My system has3  IOMMU Groups which I can assign devices from. Group 1 is PCIe Slot 1 which contains an old single slot Quadro NVS graphics card for console access. Group 2 is PCIe Slot 2 which has my GTX760. As I don't need console access once I get everything setup, I figured I could free up Group 1 and place my LSI Card in instead - pulling it out of the group with my USB Contooler/Creative SBZ/Ethernet. Now with my console is being supplied by my GTX 760. I am able to boot my FreeNAS VM and have it see my LSI RAID Card. However, when I boot my Gaming VM, the 'virsh start gamingvm' will return my prompt, the monitor attached to my GTX 760 goes blank, then shortly after the system will hard lock. Is there a setting i'm missing to enable my host to run headless?

03:00.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS [1000:0056] (rev 02) 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

I defined my GTX 760 device-id in: /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=8086:3a37,8086:3a38,8086:3a39,8086:3a3c,10de:1187,10de:0e0a,1033:0194,8086:3a40,8086:3a42,8086:3a44,8086:3a48,8086:3a4a,11ab:4364,1102:12:00,11ab:4364,1000:0056 quiet"

As well as listed the slot which it resides in: /etc/vfio-pci.cfg

DEVICES="0000:00:1a.0 0000:00:1a.1 0000:00:1a.2 0000:00:1a.7 0000:04:00.0 0000:04:00.1 0000:02:00.0 0000:00:1c.0 0000:00:1c.1 0000:00:1c.2 0000:00:1c.4 0000:00:1c.5 0000:05:00.0 0000:06:00.0 0000:07:00.0 0000:03:00.0"

NOUVEAU is blacklisted:

[user@xenhost1 ~]$ cat /etc/modprobe.d/blacklist.conf blacklist nouveau

My GTX 760 does support EFI:

[user@xenhost1 ~]$ rom-parser/rom-parser GTX760EV.ROM Valid ROM signature found @600h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 1187, class: 030000 PCIR: revision 0, vendor revision: 1 Valid ROM signature found @fc00h, PCIR offset 1ch PCIR: type 3, vendor: 10de, device: 1187, class: 030000 PCIR: revision 3, vendor revision: 0 EFI: Signature Valid Last image

I'm at a loss as to how to get around this, if even possible...

Last edited by The_Moves (2015-02-01 16:49:21)

Offline

#4019 2015-02-01 16:55:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do not attach PCIe root ports to pci-stub or bind them to vfio-pci.  Leave them alone


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4020 2015-02-01 23:21:58

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does there exist ACS override patch for kernel 3.18.3 (Fedora 21) ? Trying the patch from OP fails...

EDIT:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)

Last edited by devianceluka (2015-02-01 23:24:27)

Offline

#4021 2015-02-02 02:35:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Does there exist ACS override patch for kernel 3.18.3 (Fedora 21) ? Trying the patch from OP fails...

EDIT:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)

It applies just fine to stock 3.18.5, so you probably have to patch it hand

Offline

#4022 2015-02-02 03:07:49

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
devianceluka wrote:

Does there exist ACS override patch for kernel 3.18.3 (Fedora 21) ? Trying the patch from OP fails...

EDIT:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)

It applies just fine to stock 3.18.5, so you probably have to patch it hand

I'm doing the exact same thing as on Fedora 20 (~3.17) where it was compiling flawlessly for months. Patch it by hand?

Offline

#4023 2015-02-02 03:38:20

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Do not attach PCIe root ports to pci-stub or bind them to vfio-pci.  Leave them alone

I went through an made sure my XML made sense after reading the below thread in which your a part of, as well as looked at a number of other XMLs:

https://www.redhat.com/archives/libvir- … 00881.html

But are you talking actual physical PCIe Root Ports? What is considered a root port? I apologize as I am still learning here.

Offline

#4024 2015-02-02 03:57:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
aw wrote:

Do not attach PCIe root ports to pci-stub or bind them to vfio-pci.  Leave them alone

I went through an made sure my XML made sense after reading the below thread in which your a part of, as well as looked at a number of other XMLs:

https://www.redhat.com/archives/libvir- … 00881.html

Way off track, that's guest configuration, vfio-pci and pci-stub are host drivers.

But are you talking actual physical PCIe Root Ports? What is considered a root port? I apologize as I am still learning here.

Those things that say Root Port when you run lspci | grep -i "root port".  You list some of them in both your kernel command line and your DEVICES.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4025 2015-02-02 04:48:52

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
The_Moves wrote:
aw wrote:

Do not attach PCIe root ports to pci-stub or bind them to vfio-pci.  Leave them alone

I went through an made sure my XML made sense after reading the below thread in which your a part of, as well as looked at a number of other XMLs:

https://www.redhat.com/archives/libvir- … 00881.html

Way off track, that's guest configuration, vfio-pci and pci-stub are host drivers.

But are you talking actual physical PCIe Root Ports? What is considered a root port? I apologize as I am still learning here.

Those things that say Root Port when you run lspci | grep -i "root port".  You list some of them in both your kernel command line and your DEVICES.

Wow, thank you do much! I can't believe I did that. i thought those were my other set of USB devices, no wonder why they weren't working as expected. I also had a typo for my Sound Card "1102:12:00". I'm surprised I was able to get it passed through and working.

What happens when I have the root ports assigned? Freezes? Panics?

Last edited by The_Moves (2015-02-02 04:53:56)

Offline

#4026 2015-02-02 06:01:19

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys, meet again~

In these days, I have a time to keep working on VFIO's work.
And I found that if I want to solve the issue that VM cannot reboot if guest OS is Linux, maybe OVMF is another options.

I'll try Ubuntu 14.04-1 as guest os and try it again.
But now my Kernel version is 3.17-rc6 with QEMU 2.1.0

Does OVMF included in QEMU 2.2.0?
If so, how can I change my script to use OVMF?

For example:

Original code is:

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

To this?

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none -nographic -localtime \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

Offline

#4027 2015-02-02 06:38:07

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Hello guys, meet again~

In these days, I have a time to keep working on VFIO's work.
And I found that if I want to solve the issue that VM cannot reboot if guest OS is Linux, maybe OVMF is another options.

I'll try Ubuntu 14.04-1 as guest os and try it again.
But now my Kernel version is 3.17-rc6 with QEMU 2.1.0

Does OVMF included in QEMU 2.2.0?
If so, how can I change my script to use OVMF?

For example:

Original code is:

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -localtime \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

To this?

#!/bin/bash qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none -nographic -localtime \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -device ahci,bus=pcie.0,id=ahci \ -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

OVMF isnt included with QEMU, but perhaps it ships with Ubuntu, you should drop Q35, and be prepared to reinstall windows.

Offline

#4028 2015-02-02 06:56:39

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

OVMF isnt included with QEMU, but perhaps it ships with Ubuntu, you should drop Q35, and be prepared to reinstall windows.

Nope, I didn't see any about OVMF in Ubuntu 14.04.

I checked wiki about OVMF, seems like need to build one by myself, I'll try it.

BTW, OVMF will use in Ubuntu guest OS, because Windows 7 is reboot normally when using legacy seabios.
And then drop Q35 and use 440FX if I use OVMF.
Any notice about change script from Q35 to 440FX?

Last edited by AKSN74 (2015-02-02 06:58:01)

Offline

#4029 2015-02-02 08:36:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
nbhs wrote:

OVMF isnt included with QEMU, but perhaps it ships with Ubuntu, you should drop Q35, and be prepared to reinstall windows.

Nope, I didn't see any about OVMF in Ubuntu 14.04.

I checked wiki about OVMF, seems like need to build one by myself, I'll try it.

BTW, OVMF will use in Ubuntu guest OS, because Windows 7 is reboot normally when using legacy seabios.
And then drop Q35 and use 440FX if I use OVMF.
Any notice about change script from Q35 to 440FX?

Should be like this:

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none -nographic -localtime \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

OVMF binaries

Last edited by nbhs (2015-02-02 08:36:36)

Offline

#4030 2015-02-02 18:57:52

lordleto
Member
Registered: 2014-11-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Quick heads up for those who want to run linux guest with nvidia.
Apparently you need kvm=off exactly like in Windows. Then it works like a charm.
Installed Ubuntu 14.04 with Nvidia 346 from CUDA 7.0RC with OMVF (see xml below)
Works like a charm, cuda runs perfectly. Even PCI3.0 with host-device transfers in 10GByte/s Range

Also switched Windows 7 to 440FX. It is truly better (GFE prolbems seemed to disappear).
I can even hotplug the whole USB 3.0 controller to a running the guest, thats awesome. Didnt expect Windows 7 to handle that.
Windows 7 runs on Seabios and qemu wrapper script.
To anyone having passing Asmedia USB 3.0 controller, you should get the latest firmware for them. That made alot of warning in dmesg disappear for me.
Its a bit of pain to find the latest one and beware there are apperently 2 versions of the chip, but it worth it.

I will try now to get OS X running. Any one got any luck starting Yosemite install? Mavericks install starts fine with chameleon. Yosemite with clover hangs on logo. need to research more. Would be nice to have a virtualized backup clone of wifes mac air.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Ubuntu1404</name> <memory unit='G'>12</memory> <currentMemory unit='G'>12</currentMemory> <memoryBacking> <hugepages/> <nosharepages/> <locked/> </memoryBacking> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu> <vcpu placement='static'>8</vcpu> <iothreads>4</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='1' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> <vcpupin vcpu='5' cpuset='10'/> <vcpupin vcpu='7' cpuset='11'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader readonly='yes' type='pflash'>..OVMF_CODE-pure-efi.fd</loader> <nvram template='..OVMF_VARS-pure-efi.fd'/> </os> <features> <pae/> <acpi/> <apic/> </features> <clock offset='utc'> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/bin/qemu-system-x86_64</emulator> <controller type='scsi' model='virtio-scsi' index='0'> <driver queues='8' /> </controller> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writethrough' io='threads' discard='unmap'/> <source file='/daten/vms/storage/Ubuntu1404/ubuntu1404.img'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/daten/images/Images/Linux/ubuntu-14.04-desktop-amd64.iso'/> <target dev='sdb' bus='scsi'/> <readonly/> <address type='drive' controller='0' bus='0' target='1' unit='0'/> </disk> <interface type='bridge'> <mac address='52:54:00:c4:c4:d0'/> <source bridge='virbr0'/> <model type='virtio'/> <driver name='vhost' queues='1'> <guest csum='off' tso4='off' tso6='off'/> </driver> </interface> <!-- TITAN --> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' bus='0x00' slot='0x09' function='0x0' multifunction='on'/> </hostdev> <!-- LAN --> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address bus='0x0c' slot='0x00' function='0x0'/> </source> <address type='pci' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> <!-- USB 3.0 --> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address bus='0x0a' slot='0x00' function='0x0'/> </source> <address type='pci' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> </devices> <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/> </qemu:commandline> </domain>

Offline

#4031 2015-02-02 19:08:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lordleto wrote:
 <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/> </qemu:commandline>

http://vfio.blogspot.com/2014/08/upstre … -2014.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4032 2015-02-02 19:42:08

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What does a person have to do to get pass their console/only graphics adapter passed through to a VM? Is it even possible? I would use a PCI video card except that my BIOS doesn't allow me to choose between PCI and PCIe for graphics.

Offline

#4033 2015-02-02 19:49:36

lordleto
Member
Registered: 2014-11-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
lordleto wrote:
 <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/> </qemu:commandline>

http://vfio.blogspot.com/2014/08/upstre … -2014.html

Yeah i know. My libvirt is too old to support the proper way. Should still have put it in to not make bad examples.

Offline

#4034 2015-02-02 21:29:22

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In the first post of this thread, NBHS references x-vga=on, is this the value I need to set in order to pass my Primary Graphics card through? (Essentially running the host headless) When i attempt to do this now, my system freezes hard require a system reset.

Qemu also mention adding x-vga support.

http://wiki.qemu.org/ChangeLog/1.5

If x-vga is the proper option I need to enable, what is the syntax I need to do to add it to my XML file? Or do I need to use a wrapper script?

I've see references to Primary and Secondary passthrough, but I'm confuzed as what they actually mean.

Offline

#4035 2015-02-02 21:56:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

In the first post of this thread, NBHS references x-vga=on, is this the value I need to set in order to pass my Primary Graphics card through? (Essentially running the host headless) When i attempt to do this now, my system freezes hard require a system reset.

Qemu also mention adding x-vga support.

http://wiki.qemu.org/ChangeLog/1.5

If x-vga is the proper option I need to enable, what is the syntax I need to do to add it to my XML file? Or do I need to use a wrapper script?

I've see references to Primary and Secondary passthrough, but I'm confuzed as what they actually mean.

Primary and secondary in this thread predominantly refers to how the graphics device is presented to the guest.  It's generally recommended that the graphics card for assignment is a secondary device on the host because it's often difficult to detach all the host drivers from the primary display.  Even if you use pci-stub.ids to prevent PCI drivers from attaching, the low level VGA/VESA drivers can still make use of it.  It's not necessarily impossible, but it's not a heavily used or documented path.

x-vga=on is a QEMU vfio-pci option that allows you to enable VGA resource access for the assigned graphics card.  This is necessary if the assigned graphics card is configured as the primary graphics for the guest in a non-OVMF configuration.  It's optional for graphics cards assigned as a secondary graphics device to the guest.  It should never be used with an OVMF guest configuration.  Whether x-vga=on is used for the guest is entirely independent of whether the graphics card is primary or secondary on the host.

If you need to use x-vga=on and you're using libvirt to manage your VM, the recommended way to add it is via a wrapper script.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4036 2015-02-03 03:12:35

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Should be like this:

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none -nographic -localtime \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

OVMF binaries

I got a first problem when I tried to install Win7.
When I boot VM, it gave me a message

Boot Failed EFI DVD/CDROM

And I google this message, found this solution.
Microsoft KB

Maybe I've try Win8 or 8.1

Offline

#4037 2015-02-03 03:25:09

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
nbhs wrote:

Should be like this:

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none -nographic -localtime \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=virtio \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

OVMF binaries

I got a first problem when I tried to install Win7.
When I boot VM, it gave me a message

Boot Failed EFI DVD/CDROM

And I google this message, found this solution.
Microsoft KB

Maybe I've try Win8 or 8.1

You have to create a fat32 formatted disk image containing the windows installation files

Offline

#4038 2015-02-03 03:31:59

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You have to create a fat32 formatted disk image containing the windows installation files

That means I have to use USB Stick to make a installation disk and then mount into VM, I thought this is a easiest way to install Win7.

Or any options?

EDIT: I tried make Win7 installation USB stick(FAT32 Format) and mount into VM by using -usb command
But after that, OVMF can't find out my USB stick to load. And it still can't find device to boot.

Last edited by AKSN74 (2015-02-03 04:09:50)

Offline

#4039 2015-02-03 05:10:24

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

That means I have to use USB Stick to make a installation disk and then mount into VM, I thought this is a easiest way to install Win7.

Or any options?

EDIT: I tried make Win7 installation USB stick(FAT32 Format) and mount into VM by using -usb command
But after that, OVMF can't find out my USB stick to load. And it still can't find device to boot.

I created a empty image file, then use mkfs.vfat to format image to FAT32.
After that, I mount my image and copy WIndows 7 installation file into image.

Edit my script to:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.1,drive=ins_cd

But OVMF still can't boot my image.

Last edited by AKSN74 (2015-02-03 05:10:43)

Offline

#4040 2015-02-03 05:16:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
AKSN74 wrote:

That means I have to use USB Stick to make a installation disk and then mount into VM, I thought this is a easiest way to install Win7.

Or any options?

EDIT: I tried make Win7 installation USB stick(FAT32 Format) and mount into VM by using -usb command
But after that, OVMF can't find out my USB stick to load. And it still can't find device to boot.

I created a empty image file, then use mkfs.vfat to format image to FAT32.
After that, I mount my image and copy WIndows 7 installation file into image.

Edit my script to:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.1,drive=ins_cd

But OVMF still can't boot my image.

Make sure this file exists:

/efi/boot/bootx64.efi

EDIT: btw since its a disk image and not a cdrom image you should change this:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.1,drive=ins_cd

to this:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-hd,bus=ide.1,drive=ins_cd

Last edited by nbhs (2015-02-03 05:20:20)

Offline

#4041 2015-02-03 05:42:21

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Make sure this file exists:

/efi/boot/bootx64.efi

EDIT: btw since its a disk image and not a cdrom image you should change this:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.1,drive=ins_cd

to this:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-hd,bus=ide.1,drive=ins_cd

Things not always be easy.

When I set install image from ide-cd to ide-hd, it gave me this:
CfZ1Edq.jpg

And if I set ide-hd back to ide-cd, it can enter without any problem, but stuck at "Starting Windows."

Maybe I can try put bootx64.efi into ISO file by using UltraISO or something.

And here is my script:

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime -boot menu=on \ -drive if=pflash,format=raw,readonly,file=/home/eddieak/OVMF/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/home/eddieak/OVMF/OVMF_VARS.fd \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 \ -device virtio-scsi-pci,id=scsi \ -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.0,drive=ins_cd \ -drive file=/home/eddieak/virtio.iso,id=virtocd,if=none -device ide-cd,bus=ide.1,drive=virtocd \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

Now I have no idea what problem about this.

Last edited by AKSN74 (2015-02-03 05:42:43)

Offline

#4042 2015-02-03 05:55:41

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
nbhs wrote:

Make sure this file exists:

/efi/boot/bootx64.efi

EDIT: btw since its a disk image and not a cdrom image you should change this:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.1,drive=ins_cd

to this:

-drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-hd,bus=ide.1,drive=ins_cd

Things not always be easy.

When I set install image from ide-cd to ide-hd, it gave me this:
http://i.imgur.com/CfZ1Edq.jpg

And if I set ide-hd back to ide-cd, it can enter without any problem, but stuck at "Starting Windows."

Maybe I can try put bootx64.efi into ISO file by using UltraISO or something.

And here is my script:

#!/bin/bash qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic -localtime -boot menu=on \ -drive if=pflash,format=raw,readonly,file=/home/eddieak/OVMF/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/home/eddieak/OVMF/OVMF_VARS.fd \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 \ -device virtio-scsi-pci,id=scsi \ -usb -usbdevice host:046d:c534 \ -drive file=/home/eddieak/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/home/eddieak/wintest.img,id=ins_cd,if=none -device ide-cd,bus=ide.0,drive=ins_cd \ -drive file=/home/eddieak/virtio.iso,id=virtocd,if=none -device ide-cd,bus=ide.1,drive=virtocd \ -net tap -net nic,macaddr=40-D8-2D-52-FB-43

Now I have no idea what problem about this.

The problem is windows 7 requires a CSM present in order to boot, thats why you're stuck on the windows logo, i posted a workarround a few pages back, i recommend you use windows 8 instead.

Offline

#4043 2015-02-03 06:16:54

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

The problem is windows 7 requires a CSM present in order to boot, thats why you're stuck on the windows logo, i posted a workarround a few pages back, i recommend you use windows 8 instead.

Mmm... Seems like it's only way to use Windows 8.1 instead of Win7.

ITOW, maybe it will not be stuck when rebooting if guest OS is Linux. From legacy bios to OVMF.

Offline

#4044 2015-02-03 08:55:04

nythrix
Member
Registered: 2015-01-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Quick question: Does anyone know why we can't pass through the intel igfx? Is it a technical limitation or a matter of not enough programmers/money thrown at it?

Offline

#4045 2015-02-03 08:59:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nythrix wrote:

Quick question: Does anyone know why we can't pass through the intel igfx? Is it a technical limitation or a matter of not enough programmers/money thrown at it?

https://01.org/kvm/blogs/albcamus/2014/ … st-release

Offline

#4046 2015-02-03 09:15:28

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Primary and secondary in this thread predominantly refers to how the graphics device is presented to the guest.  It's generally recommended that the graphics card for assignment is a secondary device on the host because it's often difficult to detach all the host drivers from the primary display.  Even if you use pci-stub.ids to prevent PCI drivers from attaching, the low level VGA/VESA drivers can still make use of it.  It's not necessarily impossible, but it's not a heavily used or documented path.


One way to do it is to avoid any use of VGA from host, e.g. redirect console to serial port (kernel command line parameter console=ttyS0 which effectively turns your host headless). I'd be interested to learn other ways.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4047 2015-02-03 12:52:09

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Supposedly the CSM requirement is only for Setup; making an unattended installation is thought to fix the Win 7 installation problem also, but I haven't verified this yet.

Offline

#4048 2015-02-03 14:59:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nythrix wrote:

Quick question: Does anyone know why we can't pass through the intel igfx? Is it a technical limitation or a matter of not enough programmers/money thrown at it?

The problem is that IGD is not a self-contained device like a discrete graphics card.  Enabling it in the guest requires not only passing the device at PCI address 0000:00:02.0, but also modifying the PCI device IDs of various chipset components and emulating or passing through I/O ranges of other devices obliquely associated with graphics.  IOW, instead of being isolated to a single device, it's smeared across parts of the whole chipset.  It's not impossible to support it, Xen has done it and there's very rough prototype code to do it in KVM.  It is pretty messy though due to the architecture (or perhaps more correctly, lack of architecture in IGD design).

Last I heard, we're waiting on Intel to see if they can redesign their driver to take a more self-contained approach to IGD.  It's possible that some of the chipset and miscellaneous resource requirements are no longer necessary.  However, that may also mean that a solution relying on that approach may only work for the very latest devices.  I haven't had the time or interest to put in to IGD assignment, but another KVM developer has recently started looking at it, so it's possible it could move forward.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4049 2015-02-03 15:00:39

lael
Member
Registered: 2015-02-03
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys congrats to this humongus thread!

I have been wanting to do such a setup for a long time, had some experience with XEN but i bought the wrong cpu for this with no vt-d and i postponed up until now that i am able to buy new hardware and would like to hear what parts to look out for and hopefully have less hassle setting everything up.

The doc with the listed Mainboards https://docs.google.com/spreadsheet/ccc … _web#gid=0 is a huge help but i fear that some info might be missing like info on Revision and the Motheboards i checked so far seem to be a quite a few years old but i am not done with the list yet.

Thanks for any suggestions.

Offline

#4050 2015-02-03 15:04:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lael wrote:

Hi guys congrats to this humongus thread!

I have been wanting to do such a setup for a long time, had some experience with XEN but i bought the wrong cpu for this with no vt-d and i postponed up until now that i am able to buy new hardware and would like to hear what parts to look out for and hopefully have less hassle setting everything up.

The doc with the listed Mainboards https://docs.google.com/spreadsheet/ccc … _web#gid=0 is a huge help but i fear that some info might be missing like info on Revision and the Motheboards i checked so far seem to be a quite a few years old but i am not done with the list yet.

Thanks for any suggestions.

https://bbs.archlinux.org/viewtopic.php … 6#p1495466


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4051 2015-02-03 17:58:49

Wooots
Member
Registered: 2015-01-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Installed Windows 10 TP on the weekend through virt-manager, edited xml (virsh-edit) to use ovmf. So now I can manage everything with libvirt that's really great.

One question, I had to deactivate the hv_* options (@aw thanks for your blog) to get Code 43 Error away from my Nvidia GTX 780 (Next time I'll consider buying an AMD GPU)
I've read that deactivating those extensions impose a performance penalty.

So has someone measured how big this penalty is?
Would be very interested in some numbers.

Last edited by Wooots (2015-02-03 18:01:33)

Offline

#4052 2015-02-03 18:13:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wooots wrote:

Installed Windows 10 TP on the weekend through virt-manager, edited xml (virsh-edit) to use ovmf. So now I can manage everything with libvirt that's really great.

One question, I had to deactivate the hv_* options (@aw thanks for your blog) to get Code 43 Error away from my Nvidia GTX 780 (Next time I'll consider buying an AMD GPU)
I've read that deactivating those extensions impose a performance penalty.

So has someone measured how big this penalty is?
Would be very interested in some numbers.

You can test this yourself, download a version of the nvidia driver prior to 337.88, re-enable all the hyperv extensions and run your benchmarks.  It depends on the benchmark what you'll see, but I'm sure folks here would appreciate seeing your results.  Note that you really want to test both hyperv enabled vs disabled on the same old driver, then for graphics, test again with the new driver and hyperv disabled.  I think you'll find that the new driver more than compensates for the missing hyperv extensions, but non-graphics related benchmarks may still suffer slightly.

Last edited by aw (2015-02-03 18:22:44)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4053 2015-02-03 18:29:40

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is this impossible to achieve on Core 2 Duo P9500? I was unable to find precise information but one site claimed that VT-d is available only from i7 onwards.


Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4054 2015-02-03 19:29:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lockheed wrote:

Is this impossible to achieve on Core 2 Duo P9500? I was unable to find precise information but one site claimed that VT-d is available only from i7 onwards.

Maybe.  For some reason Intel was shipping VT-d on laptop chipsets way before servers.  The Core 2 Duos were before Intel moved graphics onto the CPU.  With the graphics, VT-d also had to migrate to the CPU.  So in this generation, while the CPU doesn't show any signs of VT-d (http://ark.intel.com/products/35566), the chipset actually does (http://ark.intel.com/products/35135), assuming you're talking about the W500 in your sig and google told me the right chipset.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4055 2015-02-03 19:53:25

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, that's W500 with dual Intel+Ati graphics. Is there a way to check for sure whether it supports it?


Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4056 2015-02-03 19:55:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lockheed wrote:

Yes, that's W500 with dual Intel+Ati graphics. Is there a way to check for sure whether it supports it?

Boot with intel_iommu=on, run find /sys/kernel/iommu_groups  If you have groups and devices, then you have VT-d.  That of course doesn't guarantee that you'll be able to get VGA assignment working, hybrid graphic laptops can be tricky.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4057 2015-02-03 20:06:25

Wooots
Member
Registered: 2015-01-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok did a quick performance test

For the cpu-test I used the free geekbench and for gpu test I used 3D Mark Firestrike test.
My Hardware: i7-4930k, Geforce GTX-780

Results with hv options on:
  - Geekbench (Single): 3314
  - Geekbench (Multi): 20082
  - 3D Mark: 9424, Graphics Score 10441

Results with hv options off:
  - Geekbench (Single): 3203
  - Geekbench (Multi): 19410
  - 3D Mark: 9045, Graphics Score 10249

Cpufreq governor was set to ondemand, next time i'll use performance.
On the 3D Mark Score, the Graphics Score is more interesting, because I didn't install PhysX, and physics calculation has a high impact on 3D Mark Score (Correct me if I'm wrong).

To sum up:
CPU: -3.4% (96.6%)
GPU: -1.9% (98.1%)

It was just a quick test but for the first, hv options seems not a really big performance penalty. I can live with that.
I would have to do these performance tests multiple times to get a more accurate value. I guess the cpu value could be +-1% when tested multiple times.

Last edited by Wooots (2015-02-03 20:11:18)

Offline

#4058 2015-02-03 20:15:21

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Boot with intel_iommu=on, run find /sys/kernel/iommu_groups  If you have groups and devices, then you have VT-d.  That of course doesn't guarantee that you'll be able to get VGA assignment working, hybrid graphic laptops can be tricky.

The groups and devices are there. I guess I will have to try, then.


Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4059 2015-02-03 20:50:20

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, I didn't know that Intel had VT-D pre Core i7 processors. Super cool!

Offline

#4060 2015-02-03 20:51:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

Wow, I didn't know that Intel had VT-D pre Core i7 processors. Super cool!

It exists on many i5 processors as well.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4061 2015-02-03 21:33:53

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not clear on the vfio-bind part. You wrote you have two radeons, so have to  use pci-stub. But since I have intel+radeon, I can just blacklist radeon and skip entire 2nd point, right?
Then, running vfio-bind, I only need to run it with BusID of radeon card, and not the audio, right?

vfio-bind 0000:01:00.0

Because I only have one Audio device and it's Intel:

$ lspci | grep Audio 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03

Last edited by Lockheed (2015-02-03 21:34:37)


Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4062 2015-02-03 22:51:04

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lockheed wrote:

I am not clear on the vfio-bind part. You wrote you have two radeons, so have to  use pci-stub. But since I have intel+radeon, I can just blacklist radeon and skip entire 2nd point, right?
Then, running vfio-bind, I only need to run it with BusID of radeon card, and not the audio, right?

vfio-bind 0000:01:00.0

Because I only have one Audio device and it's Intel:

$ lspci | grep Audio 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03

Can you post your full 'lspci -nn' output?

Offline

#4063 2015-02-03 22:52:10

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

00:00.0 Host bridge [0600]: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub [8086:2a40] (rev 07) 00:01.0 PCI bridge [0604]: Intel Corporation Mobile 4 Series Chipset PCI Express Graphics Port [8086:2a41] (rev 07) 00:02.0 VGA compatible controller [0300]: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller [8086:2a42] (rev 07) 00:03.0 Communication controller [0780]: Intel Corporation Mobile 4 Series Chipset MEI Controller [8086:2a44] (rev 07) 00:03.3 Serial controller [0700]: Intel Corporation Mobile 4 Series Chipset AMT SOL Redirection [8086:2a47] (rev 07) 00:1a.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 03) 00:1a.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 03) 00:1a.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 03) 00:1a.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 03) 00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03) 00:1c.0 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 [8086:2940] (rev 03) 00:1c.1 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 2 [8086:2942] (rev 03) 00:1c.2 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 3 [8086:2944] (rev 03) 00:1c.3 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 [8086:2946] (rev 03) 00:1d.0 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 03) 00:1d.1 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 03) 00:1d.2 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 03) 00:1d.7 USB controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 03) 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 Mobile PCI Bridge [8086:2448] (rev 93) 00:1f.0 ISA bridge [0601]: Intel Corporation ICH9M-E LPC Interface Controller [8086:2917] (rev 03) 00:1f.2 SATA controller [0106]: Intel Corporation 82801IBM/IEM (ICH9M/ICH9M-E) 4 port SATA Controller [AHCI mode] [8086:2929] (rev 03) 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 03) 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] RV635/M86 [Mobility Radeon HD 3650] [1002:9591] 03:00.0 Network controller [0280]: Intel Corporation Ultimate N WiFi Link 5300 [8086:4236] 04:00.0 Memory controller [0580]: Intel Corporation Turbo Memory Controller [8086:444e] (rev 11) 05:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02) 15:00.0 CardBus bridge [0607]: Ricoh Co Ltd RL5c476 II [1180:0476] (rev ba) 15:00.2 SD Host controller [0805]: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter [1180:0822] (rev 21) 15:00.4 System peripheral [0880]: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter [1180:0592] (rev 11) 15:00.5 System peripheral [0880]: Ricoh Co Ltd xD-Picture Card Controller [1180:0852] (rev 11)

Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4064 2015-02-03 22:58:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lockheed wrote:

I am not clear on the vfio-bind part. You wrote you have two radeons, so have to  use pci-stub. But since I have intel+radeon, I can just blacklist radeon and skip entire 2nd point, right?
Then, running vfio-bind, I only need to run it with BusID of radeon card, and not the audio, right?

vfio-bind 0000:01:00.0

Because I only have one Audio device and it's Intel:

$ lspci | grep Audio 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03

The set of devices largely depends on the isolation of the devices, which is represented by the IOMMU groups (http://vfio.blogspot.com/2014/08/iommu- … d-out.html).  So we really need to see what your groups look like (that directory you ran find on before).  In an ideal case, you can just blacklist radeon or use pci-stub for just the GPU.  You may need to apply the ACS patch to break down groups into something usable (at your own risk).  Hybrid laptops like this often have the VGA BIOS for the discrete card buried in the system firmware/ACPI, so you might have fun trying to get it.  Since AMD has a fair success rate as a secondary GPU for the VM, that's the route I'd probably start with to avoid that problem as well as VGA arbitration.  The good news is that your system is old enough that you probably don't need the i915 VGA arbitration patch if you do end going the primary head/VGA route.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4065 2015-02-04 03:02:51

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

The problem is windows 7 requires a CSM present in order to boot, thats why you're stuck on the windows logo, i posted a workarround a few pages back, i recommend you use windows 8 instead.

Does Win8 and Win8.1 have EFI difference?

Now I'm trying to boot Win8.1 ISO, and trying to convert as img file /w FAT32 format and included bootx64.efi.
But both of them can't boot in OVMF, the EFI can't find any bootable devices.

EDIT:

Finally, I started install Windows 8 now.
I use my trump card: Mount USB 2.0 into VM and use USB Stick to install Windows 8, and it worked without any problem.

Last edited by AKSN74 (2015-02-04 04:01:13)

Offline

#4066 2015-02-04 05:07:50

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm successfully install WIn8 in VM with OVMF, but I found something strange.

My mouse and keyboard get 'laggy' when in Win8 desktop, and while I test 3DMark on VM, some scenes will get hangs about 0.3 to 0.5 second
It never happen when I still use legacy BIOS with WIn7. And I'm sure that its not caused by HDD.
How can I improve the performance and less delay?

BTW, now my GPU is GTX980.

Last edited by AKSN74 (2015-02-04 05:09:40)

Offline

#4067 2015-02-04 05:10:18

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Now I'm successfully install WIn8 in VM with OVMF, but I found something strange.

My mouse and keyboard get 'laggy' when in Win8 desktop, and while I test 3DMark on VM, some scenes will get hangs about 0.3 to 0.5 second
It never happen when I still use legacy BIOS with WIn7.
How can I improve the performance and less delay?

BTW, now my GPU is GTX980.

PIn vcpus, hugepages, hv-time, etc.

Last edited by nbhs (2015-02-04 05:10:26)

Offline

#4068 2015-02-04 07:07:06

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

PIn vcpus, hugepages, hv-time, etc.

Thanks for advice!

Now I added hv_time after -cpu host

-cpu host,hv_time,kvm=off

And the result of 3DMARK seems good

3DMARK result Webpage

gyI5P1P.png

Offline

#4069 2015-02-04 07:51:48

MrTUX-10
Member
Registered: 2015-01-06
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
AKSN74 wrote:

Now I'm successfully install WIn8 in VM with OVMF, but I found something strange.

My mouse and keyboard get 'laggy' when in Win8 desktop, and while I test 3DMark on VM, some scenes will get hangs about 0.3 to 0.5 second
It never happen when I still use legacy BIOS with WIn7.
How can I improve the performance and less delay?

BTW, now my GPU is GTX980.

PIn vcpus, hugepages, hv-time, etc.


How do you pin the vcpus while using a script instead of virsh? Thanks in advance!

Offline

#4070 2015-02-04 08:04:13

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I got another problem.

After the test, I reboot my VM, gets Win8 crashes (How sucks of Win8......)

So, I re-install Win8 and NVIDIA Driver (347.25)
But now my GTX980 gets "Warning" icon and can't use.

I thought the reason is perhaps the hv_time. Because when I turn it on, it got a same problem at first time. (But solved by re-install driver.)
This time. it's still happening even I re-install driver.

EDIT: Problem solved if delete hv_time and re-install driver.
Really strange, it's not happen before.

Last edited by AKSN74 (2015-02-04 08:12:24)

Offline

#4071 2015-02-04 08:41:58

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lael wrote:

Hi guys congrats to this humongus thread!

I have been wanting to do such a setup for a long time, had some experience with XEN but i bought the wrong cpu for this with no vt-d and i postponed up until now that i am able to buy new hardware and would like to hear what parts to look out for and hopefully have less hassle setting everything up.

The doc with the listed Mainboards https://docs.google.com/spreadsheet/ccc … _web#gid=0 is a huge help but i fear that some info might be missing like info on Revision and the Motheboards i checked so far seem to be a quite a few years old but i am not done with the list yet.

Thanks for any suggestions.

I have good experience with workstation class SuperMicro X9DA7  - no compatibility issues of any kind, can turn on and off ROM of any individual slot, all slots on separate IOMMUs and also builtin hardware (USB and eth), can even redirect BIOS console to serial port. I use it with Xeon E5-2630 v2 and ECC memory.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4072 2015-02-04 08:52:29

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

$ find /sys/kernel/iommu_groups /sys/kernel/iommu_groups /sys/kernel/iommu_groups/0 /sys/kernel/iommu_groups/0/devices /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1 /sys/kernel/iommu_groups/1/devices /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/2 /sys/kernel/iommu_groups/2/devices /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3 /sys/kernel/iommu_groups/3/devices /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.3 /sys/kernel/iommu_groups/4 /sys/kernel/iommu_groups/4/devices /sys/kernel/iommu_groups/4/devices/0000:00:1a.0 /sys/kernel/iommu_groups/4/devices/0000:00:1a.1 /sys/kernel/iommu_groups/4/devices/0000:00:1a.2 /sys/kernel/iommu_groups/4/devices/0000:00:1a.7 /sys/kernel/iommu_groups/5 /sys/kernel/iommu_groups/5/devices /sys/kernel/iommu_groups/5/devices/0000:00:1b.0 /sys/kernel/iommu_groups/6 /sys/kernel/iommu_groups/6/devices /sys/kernel/iommu_groups/6/devices/0000:00:1c.0 /sys/kernel/iommu_groups/6/devices/0000:00:1c.1 /sys/kernel/iommu_groups/6/devices/0000:00:1c.2 /sys/kernel/iommu_groups/6/devices/0000:00:1c.3 /sys/kernel/iommu_groups/6/devices/0000:03:00.0 /sys/kernel/iommu_groups/6/devices/0000:04:00.0 /sys/kernel/iommu_groups/6/devices/0000:05:00.0 /sys/kernel/iommu_groups/7 /sys/kernel/iommu_groups/7/devices /sys/kernel/iommu_groups/7/devices/0000:00:1d.0 /sys/kernel/iommu_groups/7/devices/0000:00:1d.1 /sys/kernel/iommu_groups/7/devices/0000:00:1d.2 /sys/kernel/iommu_groups/7/devices/0000:00:1d.7 /sys/kernel/iommu_groups/8 /sys/kernel/iommu_groups/8/devices /sys/kernel/iommu_groups/8/devices/0000:00:1e.0 /sys/kernel/iommu_groups/8/devices/0000:15:00.0 /sys/kernel/iommu_groups/8/devices/0000:15:00.2 /sys/kernel/iommu_groups/8/devices/0000:15:00.4 /sys/kernel/iommu_groups/8/devices/0000:15:00.5 /sys/kernel/iommu_groups/9 /sys/kernel/iommu_groups/9/devices /sys/kernel/iommu_groups/9/devices/0000:00:1f.0 /sys/kernel/iommu_groups/9/devices/0000:00:1f.2 /sys/kernel/iommu_groups/9/devices/0000:00:1f.3

What is better/easier - blacklisting, or using PCI stub? Do I understand it right that you suggest going the pci-stub route because of VGA BIOS being integrated into system firmware?

I plan to use Intel GPU for the host system and give Radeon entirely to qemu. Not sure if that's what you meant by

you probably don't need the i915 VGA arbitration patch if you do end going the primary head/VGA route.

?

Also, when I am not running qemu, is there a way to keep using radeon (occasionally) with DRI_PRIME?


EDIT:
I booted with that custom kernel form the OP, but it has negative influence on the radeon GPU temperature. It skyrockets to ~90 C even when not in use and the fan is running like crazy. Perhaps it disables power management for it?

Last edited by Lockheed (2015-02-04 10:41:32)


Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4073 2015-02-04 10:23:10

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
aw wrote:
Bronek wrote:

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[ 250.730914] kernel BUG at drivers/pci/ats.c:138!

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

That's saying that the device had ATS (Address Translation Services) enabled before reset, but when we went to restore the state of it after reset, we couldn't even find the capability.  Maybe the device never really recovers from reset?

Got it. I suppose I should complain to AMD support ...

so I asked Sapphire support (it's their card) but no reply yet. I'm tempted to experiment with a quirk for the device to disable ATS detection on boot. Any hints where to start?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4074 2015-02-04 13:49:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:

Now I got another problem.

After the test, I reboot my VM, gets Win8 crashes (How sucks of Win8......)

So, I re-install Win8 and NVIDIA Driver (347.25)
But now my GTX980 gets "Warning" icon and can't use.

I thought the reason is perhaps the hv_time. Because when I turn it on, it got a same problem at first time. (But solved by re-install driver.)
This time. it's still happening even I re-install driver.

EDIT: Problem solved if delete hv_time and re-install driver.
Really strange, it's not happen before.

As reported countless times in this thread, the latest Nvidia driver is incompatible with hyper-v extensions, including hv_time.  The driver doesn't seem to detect these extensions on every boot, which can lead to confusing issues like this, but hv_time and friends should never be recommended for Nvidia users.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4075 2015-02-04 14:13:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

I can confirm that I no longer have DMAR issues with 3.18.5 smile

Offline

#4076 2015-02-04 15:29:22

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all, It is possible Passthrough another PCIe device? PCIe USB controller, PCIe Sound Card? PCIe network card?

Offline

#4077 2015-02-04 15:53:55

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:

Hi all, It is possible Passthrough another PCIe device? PCIe USB controller, PCIe Sound Card? PCIe network card?

Yes . Have a look at my VM :

qemu-system-x86_64 -name main -nographic \ -enable-kvm -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \ -vga none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/VMs/ovmf_main.bin \ -device vfio-pci,host=04:00.0,multifunction=on \ -device vfio-pci,host=04:00.1 \ -device vfio-pci,host=07:00.0 \ -device vfio-pci,host=00:1b.0 \ -drive file=/VMs/Win_Main.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win7.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \ -localtime \ -net nic,model=virtio,macaddr=38:29:21:5F:C4:7D -net bridge,br=br0 \ -netdev type=tap,ifname=tap_s,id=net1,vhost=on,vhostforce=on,queues=4,script= \ -device virtio-net-pci,netdev=net1,mq=on,vectors=9 \ -monitor unix:/tmp/vm_main,server,nowait &

As you can see , I'm passing-through a total of 4 PCI devices . My GPU + HDMI Audio , a USB 3.0 PCI-E card and the onboard Intel HD Audio controller .

Offline

#4078 2015-02-04 15:56:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:

Hi all, It is possible Passthrough another PCIe device? PCIe USB controller, PCIe Sound Card? PCIe network card?

That's mostly the purpose of vfio, GPU assignment with vfio is just a fun hack.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4079 2015-02-04 16:28:30

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

THX Denso and aw

aw wrote:
flack wrote:

Hi all, It is possible Passthrough another PCIe device? PCIe USB controller, PCIe Sound Card? PCIe network card?

That's mostly the purpose of vfio, GPU assignment with vfio is just a fun hack.

I ask becouse using USB network card and USB soundcard in qemu. But when i play over internet on virtual machine then i got some latency lags.  I think this can be reduced when i use USB PCIe card. And then put all USB devices to this PCIe USB controller.

Offline

#4080 2015-02-04 18:28:32

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:

THX Denso and aw

aw wrote:
flack wrote:

Hi all, It is possible Passthrough another PCIe device? PCIe USB controller, PCIe Sound Card? PCIe network card?

That's mostly the purpose of vfio, GPU assignment with vfio is just a fun hack.

I ask becouse using USB network card and USB soundcard in qemu. But when i play over internet on virtual machine then i got some latency lags.  I think this can be reduced when i use USB PCIe card. And then put all USB devices to this PCIe USB controller.

Yeeeah, that's usually how people do when they're dealing with USB latency - they pass-through the whole USB controller as a PCIe device. As stated in Denso's example - it's done the same way as GPU pass-through, only much easier.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4081 2015-02-04 20:33:12

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:

THX Denso and aw

aw wrote:
flack wrote:

Hi all, It is possible Passthrough another PCIe device? PCIe USB controller, PCIe Sound Card? PCIe network card?

That's mostly the purpose of vfio, GPU assignment with vfio is just a fun hack.

I ask becouse using USB network card and USB soundcard in qemu. But when i play over internet on virtual machine then i got some latency lags.  I think this can be reduced when i use USB PCIe card. And then put all USB devices to this PCIe USB controller.

TBH virtio-net is probably much better (and faster) than your USB NIC.

Offline

#4082 2015-02-04 20:34:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

TBH virtio-net is probably much better (and faster) than your USB NIC.

Agreed


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4083 2015-02-05 03:57:25

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

As reported countless times in this thread, the latest Nvidia driver is incompatible with hyper-v extensions, including hv_time.  The driver doesn't seem to detect these extensions on every boot, which can lead to confusing issues like this, but hv_time and friends should never be recommended for Nvidia users.

Got it, looks like some boost function can't use on NVIDIA Driver.
Also, I found out the delay problem I got just Windows 8 problem. After update, everything got right.

(P.S  I got Windows Update from 4 p.m to 3 a.m, how 'wonderful' does Windows 8.)

Offline

#4084 2015-02-05 07:06:11

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I'm trying to install Ubuntu 14.04.1 on OVMF VM.
It can enter into GRUB menu, but I select install Ubuntu, only black screen.

Is my GTX980 too new so that Ubuntu's driver didn't support yet?
Maybe I should change back to GTX480, but I guess GTX480's BIOS didn't support UEFI model.

Offline

#4085 2015-02-05 15:23:30

lordleto
Member
Registered: 2014-11-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i got Yosemite in KVM with GPU passtrough working!

Essentially i created a pure mac install usb by

sudo /Applications/Install\ OS\ X\ Yosemite.app/Contents/Resources/createinstallmedia --volume /Volumes/Name-des-USB-Sticks/ --applicationpath /Applications/Install\ OS\ X\ Yosemite.app/ --nointeraction

Booted seabios and used chameleon 2510 as kernel. Installation worked quite nicely on q35 machine. But after install also 440fx works. So i use this.
One needs to play with slots for e1000-82545em to get it working. (Some strange irq issue apparently).
GPU passthrough works, without any kvm=off stuff. On mac i can even pass my titan as secondary card besides vga. Go figure. USB3 (ASM1042) works with driver from multibeast. Pass through of 82574L also works with driver from multibeast but performance is abysimal. Hope virtio-net is working on Yosemite soon, no need to pass a real card anymore. But e1000 is not bad for now.

One last problem is cpu. Now it works with model core2duo but this disables some sse instructions, and all aes and avx and other features my Xeon 1650 v1 (Think i7-3930k with ECC) has. Model Sandybridge kinda works. But OS X cant figure the right clock and think it runs on 133MHz and so the realtime clock runs 20 times faster. Anyone knows how to fix this? Passing cpu as model host ends in kernel panic on boot.

I also run some test on Mac VM, Ubuntu 14.04 VM and Windows7 VM and also on Host. All VM have 4 pinned HT Cores (8 Logical) Windows VM is additinally guared by cset, Ubuntu and Mac are not. Hugepages are on.
no hv enhancments on Windows VM due to latest driver.

Geekbench (first single then multi)
Baseline Host Ubuntu 12.04 3.17.1: http://browser.primatelabs.com/geekbench3/1848887 3218     17638
Windows 7 VM: http://browser.primatelabs.com/geekbench3/1848676 3221     12821
Ubuntu 14.04: http://browser.primatelabs.com/geekbench3/1848847 3230     13215
Mac 10.10.2: http://browser.primatelabs.com/geekbench3/1848531 2940     12055
lower result on mac almost fully attributable to lacking of aes on core2duo, so for some stuff somehow managing to get mac vm working with sandybridge or even host would be great.

Also done Unigine 4.0 benches

Winows 7  VM
FPS:    61.3 Score: 1544 Min FPS: 8.6 Max FPS:    141.3
Windows 7 (build 7601, Service Pack 1) 64bit CPU model:    Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz (3999MHz) x4 GPU model:     NVIDIA GeForce GTX TITAN 9.18.13.4725 (4095MB) x1
Settings Render:    Direct3D11 Mode:     1600x900 8xAA windowed Preset    Extreme

Ubuntu 14.04 VM
FPS:    63.8 Score:    1608 Min FPS:     9.3 Max FPS:    132.5
Linux 3.13.0-45-generic x86_64 CPU model:     Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz (3999MHz) x8 GPU model:     GeForce GTX TITAN PCI Express 346.35 (6144MB) x1
Settings Render:    OpenGL Mode:     1600x900 8xAA windowed Preset     Extreme

Mac OS X 10.10.2 (Stock Nvidia Driver)
FPS:    59.1 Score:    1490 Min FPS:     8.5 Max FPS:     122.1
Darwin 14.1.0 x86_64 CPU model:    Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz (3999MHz) x8 GPU model:     NVIDIA GeForce GTX TITAN (6144MB) x1
Settings Render:    OpenGL Mode:     1600x900 8xAA windowed Preset    Extreme

Id say not bad, differences are not that big.
Maybe i do also bare metal run on Heaven to compare but this needs a reboot and a new xorg.conf.
So appart from some difficulties in setting this up, there is no reason to not do it smile

Last edited by lordleto (2015-02-05 15:28:18)

Offline

#4086 2015-02-05 16:30:57

Lockheed
Member
Registered: 2010-03-16
Posts: 1,420

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lockheed wrote:
$ find /sys/kernel/iommu_groups /sys/kernel/iommu_groups /sys/kernel/iommu_groups/0 /sys/kernel/iommu_groups/0/devices /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1 /sys/kernel/iommu_groups/1/devices /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/2 /sys/kernel/iommu_groups/2/devices /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3 /sys/kernel/iommu_groups/3/devices /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.3 /sys/kernel/iommu_groups/4 /sys/kernel/iommu_groups/4/devices /sys/kernel/iommu_groups/4/devices/0000:00:1a.0 /sys/kernel/iommu_groups/4/devices/0000:00:1a.1 /sys/kernel/iommu_groups/4/devices/0000:00:1a.2 /sys/kernel/iommu_groups/4/devices/0000:00:1a.7 /sys/kernel/iommu_groups/5 /sys/kernel/iommu_groups/5/devices /sys/kernel/iommu_groups/5/devices/0000:00:1b.0 /sys/kernel/iommu_groups/6 /sys/kernel/iommu_groups/6/devices /sys/kernel/iommu_groups/6/devices/0000:00:1c.0 /sys/kernel/iommu_groups/6/devices/0000:00:1c.1 /sys/kernel/iommu_groups/6/devices/0000:00:1c.2 /sys/kernel/iommu_groups/6/devices/0000:00:1c.3 /sys/kernel/iommu_groups/6/devices/0000:03:00.0 /sys/kernel/iommu_groups/6/devices/0000:04:00.0 /sys/kernel/iommu_groups/6/devices/0000:05:00.0 /sys/kernel/iommu_groups/7 /sys/kernel/iommu_groups/7/devices /sys/kernel/iommu_groups/7/devices/0000:00:1d.0 /sys/kernel/iommu_groups/7/devices/0000:00:1d.1 /sys/kernel/iommu_groups/7/devices/0000:00:1d.2 /sys/kernel/iommu_groups/7/devices/0000:00:1d.7 /sys/kernel/iommu_groups/8 /sys/kernel/iommu_groups/8/devices /sys/kernel/iommu_groups/8/devices/0000:00:1e.0 /sys/kernel/iommu_groups/8/devices/0000:15:00.0 /sys/kernel/iommu_groups/8/devices/0000:15:00.2 /sys/kernel/iommu_groups/8/devices/0000:15:00.4 /sys/kernel/iommu_groups/8/devices/0000:15:00.5 /sys/kernel/iommu_groups/9 /sys/kernel/iommu_groups/9/devices /sys/kernel/iommu_groups/9/devices/0000:00:1f.0 /sys/kernel/iommu_groups/9/devices/0000:00:1f.2 /sys/kernel/iommu_groups/9/devices/0000:00:1f.3

What is better/easier - blacklisting, or using PCI stub? Do I understand it right that you suggest going the pci-stub route because of VGA BIOS being integrated into system firmware?

I plan to use Intel GPU for the host system and give Radeon entirely to qemu. Not sure if that's what you meant by

you probably don't need the i915 VGA arbitration patch if you do end going the primary head/VGA route.

?

Also, when I am not running qemu, is there a way to keep using radeon (occasionally) with DRI_PRIME?


EDIT:
I booted with that custom kernel form the OP, but it has negative influence on the radeon GPU temperature. It skyrockets to ~90 C even when not in use and the fan is running like crazy. Perhaps it disables power management for it?


@aw, could you please comment on my post?


Laptop: ThinkPad W500, C2D P9500, 8GB, Radeon RV635 (HD3650), Arch | Server/fw: Zotac AQ01, A4-5000 Kabini, 4GB, Arch/pfSense VM

Offline

#4087 2015-02-05 19:22:03

Utku1Kan
Member
Registered: 2014-02-04
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So guys can you tell me why are we compiling linux-mainline but not stock arch package? Is it possible that all things work when i compile linux package with patches applied?

Offline

#4088 2015-02-06 01:31:59

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Utku1Kan wrote:

So guys can you tell me why are we compiling linux-mainline but not stock arch package? Is it possible that all things work when i compile linux package with patches applied?

I guess because not all kernel included VFIO driver, like Ubuntu generic kernel not included yet. So we usually install kernel from kernel.org.

And I don't know what do you mean about 'all things work', the patches you need is depends on your hardware, like if you have Intel graphic on your computer, you need to install i915 patch.

Offline

#4089 2015-02-06 01:55:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lockheed wrote:

What is better/easier - blacklisting, or using PCI stub?

I tend to prefer pci-stub.ids, but neither is really better or easier.

Do I understand it right that you suggest going the pci-stub route because of VGA BIOS being integrated into system firmware?

No, that's unrelated

I plan to use Intel GPU for the host system and give Radeon entirely to qemu. Not sure if that's what you meant by

you probably don't need the i915 VGA arbitration patch if you do end going the primary head/VGA route.

?

That's really your only option. Theoretically the existing VGA arbitration in the i915 driver actually works for the generation of hardware you have.

Also, when I am not running qemu, is there a way to keep using radeon (occasionally) with DRI_PRIME?

Maybe.  Graphics drivers often don't work well with binding and unbinding the device.  GPUs aren't hotplugged nearly as often as NICs.

EDIT:
I booted with that custom kernel form the OP, but it has negative influence on the radeon GPU temperature. It skyrockets to ~90 C even when not in use and the fan is running like crazy. Perhaps it disables power management for it?

How are you reading the GPU temperature without a driver?  Is the exhaust hot?  It's possible that the system relies on the driver for reporting the temperature and freaks out when there isn't one.  These are just some of the fun problems you're going to have trying to make this work on a laptop.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4090 2015-02-06 02:32:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AKSN74 wrote:
Utku1Kan wrote:

So guys can you tell me why are we compiling linux-mainline but not stock arch package? Is it possible that all things work when i compile linux package with patches applied?

I guess because not all kernel included VFIO driver, like Ubuntu generic kernel not included yet.

Now you've gone and made me install an ubuntu VM just to check.  Not true, both 14.04 and 14.10 have vfio enabled in the kernel.  I run a stock Fedora kernel since I don't need either the i915 patch (using OVMF) or the ACS override.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4091 2015-02-06 02:40:11

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
AKSN74 wrote:
Utku1Kan wrote:

So guys can you tell me why are we compiling linux-mainline but not stock arch package? Is it possible that all things work when i compile linux package with patches applied?

I guess because not all kernel included VFIO driver, like Ubuntu generic kernel not included yet.

Now you've gone and made me install an ubuntu VM just to check.  Not true, both 14.04 and 14.10 have vfio enabled in the kernel.  I run a stock Fedora kernel since I don't need either the i915 patch (using OVMF) or the ACS override.

Sorry, my information still old sad , now I'm install 3.18.5 kernel.
After that, keep testing why Ubuntu 14.04 installation can't start with OVMF.

Offline

#4092 2015-02-06 05:10:26

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I had another problem while installing Ubuntu 14.04-1 with OVMF.
(Sorry about so many problems first.)

First installation, I use GTX980 to install, but only got blank screen (monitor power is on) after I select 'Install Ubuntu'.
Then I change GTX480 to install. In OVMF message, it shows:

No suitable video mode. Enter into blind mode.

But I can see installation wizard. So I installed Ubuntu then reboot, OVMF shows same message, but get blank screen with monitor power off.
Then I switch back to GTX980, same as beginning, get blank screen with monitor power is on.
Now it still blank, and I can reboot VM with Ctrl+Alt+Delete then Enter.

Before that, I'm sure that I can install and enter into Ubuntu in real computer with GTX980. But not install with UEFI mode.
Does anyone know how to solve it?

EDIT: Fixed with nomodeset, seems like Ubuntu got a problem with GTX980 while in UEFI mode.

Last edited by AKSN74 (2015-02-06 05:39:20)

Offline

#4093 2015-02-06 10:02:21

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
AKSN74 wrote:
Utku1Kan wrote:

So guys can you tell me why are we compiling linux-mainline but not stock arch package? Is it possible that all things work when i compile linux package with patches applied?

I guess because not all kernel included VFIO driver, like Ubuntu generic kernel not included yet.

Now you've gone and made me install an ubuntu VM just to check.  Not true, both 14.04 and 14.10 have vfio enabled in the kernel.  I run a stock Fedora kernel since I don't need either the i915 patch (using OVMF) or the ACS override.

So you're actually using Fedora 21 with stock 3.18.3? How can you compile ACS patch then?

Offline

#4094 2015-02-06 12:20:22

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

How can you compile ACS patch then?

He doesn't need it on his hardware .

Last edited by Denso (2015-02-06 12:20:56)

Offline

#4095 2015-02-06 12:26:55

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
devianceluka wrote:

How can you compile ACS patch then?

He doesn't need it on his hardware .

I understand he doesnt need it. I would like to know how can he compile one. Few pages back I posted that on my Fedora I cannot compile ACS patch and did not get an answer, while on previous Fedora I was compiling it flawlessly for more then 6 months. Looks like now I will not get an answer either.

Offline

#4096 2015-02-06 14:00:01

lordleto
Member
Registered: 2014-11-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have redone Unigine test.
First attempt i forgot to disable cuda double precision and also switching intput during run cause spikes prob due to usb disconnect.
So clean results are

Host Ubuntu 12.04 FPS: 67.7 Score: 1704 Min FPS: 39.3 Max FPS: 144.2 System Platform: Linux 3.17.1-031701-generic x86_64 CPU model: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz (3999MHz) x12 GPU model: GeForce GTX TITAN PCI Express 340.32 (6144MB) x1 Settings Render: OpenGL Mode: 1600x900 8xAA windowed Preset Extreme Windows 7 KVM FPS: 70.2 Score: 1768 Min FPS: 27.4 Max FPS: 153.6 System Platform: Windows 7 (build 7601, Service Pack 1) 64bit CPU model: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz (3999MHz) x4 GPU model: NVIDIA GeForce GTX TITAN 9.18.13.4725 (4095MB) x1 Settings Render: Direct3D11 Mode: 1600x900 8xAA windowed Preset Extreme Ubuntu 14.04 KVM FPS: 66.8 Score: 1684 Min FPS: 38.8 Max FPS: 138.6 System Platform: Linux 3.13.0-45-generic x86_64 CPU model: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz (3999MHz) x8 GPU model: GeForce GTX TITAN PCI Express 346.35 (6144MB) x1 Settings Render: OpenGL Mode: 1600x900 8xAA windowed Preset Extreme Mac OS X 10.10.2 KVM FPS: 61.3 Score: 1544 Min FPS: 37.2 Max FPS: 133.5 System Platform: Darwin 14.1.0 x86_64 CPU model: Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz (3999MHz) x8 GPU model: NVIDIA GeForce GTX TITAN (6144MB) x1 Settings Render: OpenGL Mode: 1600x900 8xAA windowed Preset Extreme

So i will try now to boot mac os though omvf and clover, maybe then i can use a proper cpu model (ie SandyBridge). Host prob will never work since there never was a mac with SandyBridge-E Xeons.

Offline

#4097 2015-02-06 16:43:06

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys,

I'm dealing with a problem I can't solve, I have successfully passed through an AMD R7 250 to my windows 8.1 guest but I'm getting a lot of audio hiccups everytime other guest starts an io intensive job.

The vms are stored at an zfs dataset.

My setup is the follow:

CPU: Intel Xeon E3-1245V3
MOBO: Supermicro  X10SAE-O
RAM: 16GB ECC
VIDEO: AMD R7 250

Any help would be welcome.

Thanks.

Offline

#4098 2015-02-06 16:48:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:

Hello guys,

I'm dealing with a problem I can't solve, I have successfully passed through an AMD R7 250 to my windows 8.1 guest but I'm getting a lot of audio hiccups everytime other guest starts an io intensive job.

The vms are stored at an zfs dataset.

My setup is the follow:

CPU: Intel Xeon E3-1245V3
MOBO: Supermicro  X10SAE-O
RAM: 16GB ECC
VIDEO: AMD R7 250

Any help would be welcome.

Thanks.

Pin vCPUs and don't oversubscribe physical CPUs if you expect the guest to handle latency sensitive tasks.  You may also need to move host device interrupts to other CPUs.  Use isolcpus if you really want to have vCPU isolation guarantees.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4099 2015-02-06 17:27:34

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bpbastos wrote:

Hello guys,

I'm dealing with a problem I can't solve, I have successfully passed through an AMD R7 250 to my windows 8.1 guest but I'm getting a lot of audio hiccups everytime other guest starts an io intensive job.

The vms are stored at an zfs dataset.

My setup is the follow:

CPU: Intel Xeon E3-1245V3
MOBO: Supermicro  X10SAE-O
RAM: 16GB ECC
VIDEO: AMD R7 250

Any help would be welcome.

Thanks.

Pin vCPUs and don't oversubscribe physical CPUs if you expect the guest to handle latency sensitive tasks.  You may also need to move host device interrupts to other CPUs.  Use isolcpus if you really want to have vCPU isolation guarantees.

Thank you aw.

I'm already using isolcpus=2-7 and pinning vcpus for my guest.
The only thing I'm not doing is moving my host device interrups. Do you have any script to do it?

Here my Windows 8.1 xml:

<domain type='kvm' id='2'> <name>htpc-sala-windows</name> <uuid>89abba7f-cbb4-4360-8dc2-a1187be73e1c</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu mode='host-passthrough'> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64.vgaon</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback' io='threads'/> <source file='/var/lib/libvirt/images/htpc-sala-windows.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:ed:4b:fb'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> <address bus='3' device='6'/> </source> <alias name='hostdev0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0471'/> <product id='0x060d'/> <address bus='3' device='5'/> </source> <alias name='hostdev3'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

cmdline:

BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=1ff9fe0c-8a92-42b2-97ac-0892e610d9ad rw quiet intel_iommu=on i915.enable_hd_vgaarb=1 isolcpus=2-7 pci-stub.ids=1002:6610,1002:aab0

Offline

#4100 2015-02-06 17:37:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:
aw wrote:

Pin vCPUs and don't oversubscribe physical CPUs if you expect the guest to handle latency sensitive tasks.  You may also need to move host device interrupts to other CPUs.  Use isolcpus if you really want to have vCPU isolation guarantees.

Thank you aw.

I'm already using isolcpus=2-7 and pinning vcpus for my guest.
The only thing I'm not doing is moving my host device interrups. Do you have any script to do it?

I don't have anything, but you want to manipulate /proc/irq/*/smp_affinity.  You probably want to be careful only to do this for device interrupts (ie. things with IO-APIC or PCI-MSI in the type from /proc/interrupts).  You'll also want to make sure irqbalance doesn't move interrutps back to your isolated CPUs, there's a IRQBALANCE_BANNED_CPUS environment variable that can be used to do that.

EDIT: I doubt an E3 v3 has it, but if /sys/module/kvm_intel/parameters/enable_apicv reports 'Y' then by making sure assigned device interrupts come to a CPU not running the guest, KVM can inject the interrupt into the guest with forcing a VM exit.  I also recall someone was using (I think) the nohz_full= boot option to stop timer ticks on the isolated CPUs.

Last edited by aw (2015-02-06 17:42:29)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4101 2015-02-06 19:01:42

venim1103
Member
Registered: 2015-02-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried using Oculus Rift (dk1 or 2) with any of the GPU passtrough setups? I'm not sure how that device works exactly as if it only uses HDMI for communication and all the processing happens inside the GPU, I guess it's possible?

I just wanted to know if anyone has any experience or knowledge about this subject as I'm really interested in all this vga passtrough stuff and virtual reality (but I don't have the equipment to test out yet).

Offline

#4102 2015-02-06 23:46:10

TcShadowWalker
Member
Registered: 2013-06-04
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey,

I've got VGA passthrough running with a windows 7 64 bit guest VM.
Apart from the PCI device, I'm passing through three drives using virtio:

-drive file=/home/janphilip/VMs/Win7.img,id=disk0,if=virtio,format=raw -drive file=/dev/disk/by-label/Drive1,id=disk1,if=virtio,format=raw -drive file=/dev/disk/by-label/Drive2_SSD,id=disk2,if=virtio,format=raw

Everything works; However Windows messes up the device order.
The Win7.img contains the windows system partition; this one is always named "C:".
Unfortunately, sometimes Drive1 ist called "E:" and Drive2_SSD "F:", and after a reboot it's the other way around.

Windows offers a feature to assign drive letters in the control panel.
However, even these are not persistent; After a reboot, the guest acts as if it had never seen that disk before.

Did anyone else encounter this problem of non-persistent drive letters or has any suggestions?
I tried googling, but couldn't find anyone having the same problem. sad

Offline

#4103 2015-02-07 10:47:35

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If anyone wants to try combining Windows 7 with OVMF again:
Latest qemu git seems to enable hv_time by default, so at least I had to disable it explicitly (and un-enable all other hyper-v options). Apart from that all I had to do is create a UEFI-bootable Win7 USB drive to install from.

Also, I can confirm that cpu pinning does really help in combination with the kernel boot options isolcpus=[cpus, eg. 2-7] and nohz_full=[cpus].
For the first time I could listen to music (via USB DAC connected to a passed-through USB-controller) continously in Windows 8, while both host and guest have quite a bit of cpu load (tested with prime95/mprime).
So thanks for that information!

Still, using http://www.resplendence.com/latencymon results in a frozen guest, everytime. I thought Windows 7 was less sensitive in that matter but the latest test proved otherwise. Gonna stay with 8 and 10 as guests.

Offline

#4104 2015-02-07 13:56:22

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:

If anyone wants to try combining Windows 7 with OVMF again:
Latest qemu git seems to enable hv_time by default, so at least I had to disable it explicitly (and un-enable all other hyper-v options). Apart from that all I had to do is create a UEFI-bootable Win7 USB drive to install from.

Also, I can confirm that cpu pinning does really help in combination with the kernel boot options isolcpus=[cpus, eg. 2-7] and nohz_full=[cpus].
For the first time I could listen to music (via USB DAC connected to a passed-through USB-controller) continously in Windows 8, while both host and guest have quite a bit of cpu load (tested with prime95/mprime).
So thanks for that information!

Still, using http://www.resplendence.com/latencymon results in a frozen guest, everytime. I thought Windows 7 was less sensitive in that matter but the latest test proved otherwise. Gonna stay with 8 and 10 as guests.


Wait, you got win7 booting in UEFI? And you have NVidia...
Something hints me that my GPUs have broken GOP for win7...
Time to dig the vbios again...

Last edited by Duelist (2015-02-07 13:58:23)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4105 2015-02-07 15:08:35

Lederstrumpf
Member
Registered: 2015-02-07
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I've been pushing to get GPU passthrough working for a while using the great info found in this forum and elsewhere, but have so far not been able to get past this:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -vga none -device vfio-pci,host=06:00.0,x-vga=on -device vfio-pci,host=06:00.1 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

I first suspected it might be openSuSE, but I subsequently installed Arch and compiled the kernel as provided by nbhs with all the following instructions, yielding the same result.

My system config:

Motherboard HP ProLiant SE316M1
CPU 2 x Xeon X5650
Memory 24GB ECC DDR3 RAM
Primary GPU Nvidia 8800GT
Secondary GPU ATI R9 290X
Storage A Crucial 240GB SSD
Storage B Seagate 500GB HDD

A lot of digging led to the discovery that the 5520 chipset revision 0x13 has issues with interrupt remapping and is therefore disabled by the kernel.
http://www.intel.co.uk/content/dam/doc/ … update.pdf (Errata 47 & 53)

[ 0.025776] This system BIOS has enabled interrupt remapping on a chipset that contains an erratum making that feature unstable. To maintain system stability interrupt remapping is being disabled. Please contact your BIOS vendor for an update 

Output of dmesg | grep -e DMAR -e IOMMU

[ 0.000000] ACPI: DMAR 0x00000000AF632E80 000136 (v01 HP ProLiant 00000001 \xffffffd2? 0000162E) [ 0.000000] Intel-IOMMU: enabled [ 0.025667] dmar: IOMMU 0: reg_base_addr b7ffe000 ver 1:0 cap c90780106f0462 ecap f0207e 

For testing, I subsequently commented lines 212-213 in arch/x86/kernel/early-quirks.c to bypass disabling interrupt remapping.
New output of dmesg | grep -e DMAR -e IOMMU

[ 0.000000] ACPI: DMAR 0x00000000AF632E80 000136 (v01 HP ProLiant 00000001 \xffffffd2? 0000162E) [ 0.000000] Intel-IOMMU: enabled [ 0.025667] dmar: IOMMU 0: reg_base_addr b7ffe000 ver 1:0 cap c90780106f0462 ecap f0207e [ 0.025772] IOAPIC id 8 under DRHD base 0xb7ffe000 IOMMU 0 [ 0.025774] IOAPIC id 0 under DRHD base 0xb7ffe000 IOMMU 0

However, my qemu output remains the same, and "find /sys/kernel/iommu_groups/ -type l" still returns nothing.

Full output of dmesg
Full output of lspci

Before I proceed further, I wanted to check whether anyone has gotten GPU passthrough to work with the 5520 chipset and said revision, or whether this has been ruled impossible/unusable.
From the snippets I gathered across forums, it seems that the interrupt remapping does work, but can lead to system lockups at times, which is why it is prudently disabled.
My servers will not be used for commercial purposes, so stability is not the greatest concern.

Offline

#4106 2015-02-07 15:37:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lederstrumpf wrote:

Hi everyone,

I've been pushing to get GPU passthrough working for a while using the great info found in this forum and elsewhere, but have so far not been able to get past this:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -vga none -device vfio-pci,host=06:00.0,x-vga=on -device vfio-pci,host=06:00.1 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

Interrupt remapping is not the problem, VT-d is broken on your system and not being enabled for DMA:

[ 0.960573] dmar: Device scope type does not match for 0000:00:14.0 [ 0.960576] dmar: Device scope type does not match for 0000:00:14.1 [ 0.960578] dmar: Device scope type does not match for 0000:00:14.2 [ 0.960621] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

Check for a BIOS update for the system


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4107 2015-02-07 22:14:49

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Wait, you got win7 booting in UEFI? And you have NVidia...
Something hints me that my GPUs have broken GOP for win7...
Time to dig the vbios again...

I'm sorry but it seems all I've accomplished was to find another reason why win7 + ovmf won't hamonize. As I struggled getting it to boot at all, I haven't had tested if the nvidia worked. I"ve catched up on that and nope, doesn't work for me either.
To summarize the installation procedure:
- prepare Win7 UEFI medium (google for USB UEFI Win7, use diskpart and copy a bootx64.efi to EFI/boot/)
- generate xml for libvirt by virt-manager (Windows 7)
- add uefi-bits
==> won't install (when boot animation should appear, it freezes with 1 cpu @ 100%)
- redhat bug report suggests that smp (>=2) + hv-time is responsible, but using only 1 core and disabling hypervclock timer in xml didn't help
- remove everything hyperv-related, activly disable hv-time via <qemu:arg value='-cpu'/><qemu:arg value='host,hv_time=off'/>
==> will install and afterwards boot sometimes (or switches off instead of boot animation without noticable dmesg errors - if you see the 4 dots appear instead, it should boot fine)
- add nvidia card, install drivers
- remove default qxl graphics
- if nvidia drivers installed: image freezes at the last part of the boot animation, system seems to act normal otherwise (sound etc.)
- without nvidia drivers: hangs during boot, in secure mode @classpnp.sys

All in all, win7 + ovmf works theoretically, in my experiences it's broken at the moment, because of timing problems (hv_time-related) and nvidia card doesn't work.
As in contradiction to my own memory, interrupt handling (usb audio, latency) is as bad as with newer windows, so I have no reason to try any further. I still have student win8 keys and the Win10-experience after one day (same xml as Win8) is quite good!

Offline

#4108 2015-02-07 23:37:50

Tim_J
Member
Registered: 2015-02-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I have a Debian machine with 3 NVIDIA cards: two GTX 580s (EVGA Hydro Copper 2 FTW) and a 210 GT. I'm trying to get VGA passthrough working with Windows 7 in QEMU.

The 580s are on 0000:01:00.0/0000:01:00.1 and 0000:03:00.0/0000:03:00.1, and the 210 is on 0000:04:00.0/0000:04:00.1. My ultimate goal is to passthrough both 580s while using the 210 for the host.

I can get VGA passthrough working fine with the 580 card on 03 or with the 210 on 04, but not the 580 on 01. When I attempt to do so, the QEMU machine never reaches SeaBIOS and it uses 100% CPU on one core (this is the same behavior as if the GPU ROM is invalid).

I think that this might have something to do with the host BIOS claiming the card in some way. With both 01 and 03 stubbed and 04 used by nouveau, there is interesting behavior on boot. The host BIOS appears on the monitor connected to the 01 580, and the text remains on the screen after the OS boots (with the initial dmesg output and a flashing cursor). Despite this, Debian shows that the card has been claimed by pci-stub and later by vfio. When I attempt to start QEMU with the card, the physical monitor connected to it goes blank and loses signal, never to return. The VM locks up as mentioned earlier. As I wrote before, if I do this exact same process with the 580 in port 03, all works well (it has no physical monitors attached). If I boot with no physical monitors attached to the 01 580, it still does not work.

My host BIOS has no way to change the primary adapter.

Does anyone have any thoughts about this issue? Sorry if this has been covered before -- this thread is getting a bit unwieldy tongue

Offline

#4109 2015-02-08 02:21:35

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tim_J wrote:

Does anyone have any thoughts about this issue?

The Mainboard BIOS/UEFI really musn't initialize any of the cards you want to pass through.
Try to swap the PCIe slots. Put the 210 in the first slot and the 580 in the lower ones. Downside might be that the 580 get less PCIe bandwidth, but if the mainboard doesn't offer any software setting to choose its primary card it might be the only way. There is no onboard GPU that could become primary? Or change your mainboard as a last resort, recommendations within this thread are Gigabyte boards, but I have no experiences myself.

Offline

#4110 2015-02-08 02:34:32

Tim_J
Member
Registered: 2015-02-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:

Try to swap the PCIe slots. Put the 210 in the first slot and the 580 in the lower ones.

Ok, I assumed that this would work but I wanted to know if there was a software fix before going this route. Moving the cards to other PCIe slots will require draining the system and then refilling, which is a process that I'd like to avoid if possible. neutral

The motherboard is an ASUS Rampage IV Extreme, which does not come with any onboard GPU (the rationale being that this board is meant to appeal to gamers who will have dedicated GPUs anyway). As you suspect, only the first PCIe slot is x16 while the others are x8, but I'm pretty sure these cards aren't using x16 right now in Windows anyway. I've checked the BIOS several times and I am quite certain there is no option for changing the primary adapter.

Sounds like it is time to go get the bucket and hose... sad

Offline

#4111 2015-02-08 06:51:49

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tim_J wrote:

Ok, I assumed that this would work but I wanted to know if there was a software fix before going this route. Moving the cards to other PCIe slots will require draining the system and then refilling, which is a process that I'd like to avoid if possible. neutral

The only other way is if there is an option in bios setup to change the primary gpu and set it to the slot where the 210 is.

Offline

#4112 2015-02-08 11:44:27

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:
Tim_J wrote:

Does anyone have any thoughts about this issue?

The Mainboard BIOS/UEFI really musn't initialize any of the cards you want to pass through.

This is not true (in my case both Intel and AMD are initialized; furthermore, UEFI settings are accessible only using AMD card, no output on Intel). You can't let their video driver to initialize on specific devices you want to pass.

Offline

#4113 2015-02-08 19:58:34

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tim_J wrote:

The 580s are on 0000:01:00.0/0000:01:00.1 and 0000:03:00.0/0000:03:00.1, and the 210 is on 0000:04:00.0/0000:04:00.1. My ultimate goal is to passthrough both 580s while using the 210 for the host.

I can get VGA passthrough working fine with the 580 card on 03 or with the 210 on 04, but not the 580 on 01. When I attempt to do so, the QEMU machine never reaches SeaBIOS and it uses 100% CPU on one core (this is the same behavior as if the GPU ROM is invalid).

I think that this might have something to do with the host BIOS claiming the card in some way. With both 01 and 03 stubbed and 04 used by nouveau, there is interesting behavior on boot. The host BIOS appears on the monitor connected to the 01 580, and the text remains on the screen after the OS boots (with the initial dmesg output and a flashing cursor). Despite this, Debian shows that the card has been claimed by pci-stub and later by vfio. When I attempt to start QEMU with the card, the physical monitor connected to it goes blank and loses signal, never to return. The VM locks up as mentioned earlier. As I wrote before, if I do this exact same process with the 580 in port 03, all works well (it has no physical monitors attached). If I boot with no physical monitors attached to the 01 580, it still does not work.

My host BIOS has no way to change the primary adapter.

Does anyone have any thoughts about this issue? Sorry if this has been covered before -- this thread is getting a bit unwieldy tongue

Well..
Let me tell you my story:

I have ASUS F2A55(so i can't really change my primary gpu) HD7750 in PCI-E x16 first slot(01:00.[0-1]), HD7750 in PCI-E x8 second slot(02:00.[0-1]), AND GT610 in PCI-E x1 zero(the top one) slot(it gets mapped as 04:00.[0-1] when both AMDs are plugged).
And i've got a signal out of every card connected that way:
01:00.0 - DVI 1st screen;
02:00.0 - D-SUB 1st screen;
04:00.0 - D-SUB 2nd screen.

So, what i get:
1. BIOS/UEFI setup menus and PCI Option ROMs get shown via 01:00.0 on the first screen AND i can't change it. That is normal.
2. Linux VTs are getting shown on 01:00.0 AND i can't change it. That is normal since it's using legacy VGA ports.
3. 01:00.0 and 02:00.0 are bound to pci-stub via kernel command line. And as the host boots - they get bound to vfio-pci by a simple script.
4. If there's another VT(like X) is activated - 1st will screen remain powered on, but empty. 02:00.0 is sleeping and not showing anything. It might not get cleared, AFAIR it depends on the distro. But if it isn't cleared - that's normal, VM boot up will clear it by resetting the device.
5. My distro(fedora) had plymouth installed, you have to check /proc/iomem for something like vesafb appearing on 01:00.0. Uninstalling plymouth and disabling KMS(kernel mode setting) via kernel command line helped, but i no longer have fancy console or a splash screen. Otherwise it won't work when booting the VM - check my very first posts for details.
6. I'm using proprietary NVIDIA driver, and it has a neat option in xorg.conf available(actually, it will crash without it on my system):

Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GT 610" BusID "PCI:04:00:00" EndSection

BusID. That option clearly specifies which device nvidia driver should poke.

And now about your problems:
1. Give us your dmesg as the VM boots. Maybe there's something useful there.
2. You want to use nouveau, that's fine, but i don't know if there's BusID option available.
3. You're using nvidia host+nvidia guest. There was an ISSUES section in the op-post, but nbhs deleted it for some reason. I've tried to remember why the hell and how i've patched host nvidia drivers, but failed. I remember that there was something related to vgaarb. To nbhs:can you please bring that section back?

P.S.
Oh, and BTW, you might have some issues with pci-e bandwidth, like i have 02:00.[0-1] running on pci-e 2.0 x4 (when 04:00.[0-1] is plugged in) while physically it's x8. That depends on the motherboard and pci-e controller design(i have athlon x4 750k and it works that weird way).
Since you're doing SLI(right?) - there shouldn't be any performance drawbacks(the cards are connected via an external bridge), but for me - this is an issue, my crossfire runs via XDMA.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4114 2015-02-08 20:22:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

2. You want to use nouveau, that's fine, but i don't know if there's BusID option available.

BusID is not nvidia specific, it should work for all drivers afaik.

3. You're using nvidia host+nvidia guest. There was an ISSUES section in the op-post, but nbhs deleted it for some reason. I've tried to remember why the hell and how i've patched host nvidia drivers, but failed. I remember that there was something related to vgaarb. To nbhs:can you please bring that section back?

The issue was that using the proprietary nvidia driver on the host would take the vga arbiter lock and never release it.  This would make the guest hang on its first VGA region access as it waits on a lock that will never be released.  Maybe nvidia has fixed this in their driver now?

Last edited by aw (2015-02-08 20:34:32)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4115 2015-02-08 20:33:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

3. You're using nvidia host+nvidia guest. There was an ISSUES section in the op-post, but nbhs deleted it for some reason. I've tried to remember why the hell and how i've patched host nvidia drivers, but failed. I remember that there was something related to vgaarb. To nbhs:can you please bring that section back?

Im using an nvidia gpu on my host, and i havent had that problem in a long time now, but yes why not.

Offline

#4116 2015-02-08 21:33:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

BusID is not nvidia specific, it should work for all drivers afaik.

If that's so - awesome. I recall seeing it in nvidia's docs somewhere.

aw wrote:

The issue was that using the proprietary nvidia driver on the host would take the vga arbiter lock and never release it.  This would make the guest hang on its first VGA region access as it waits on a lock that will never be released.  Maybe nvidia has fixed this in their driver now?

Well... yyeah, they've fixed it. I've had to patch 343.22, but it's

[ 3.573430] vgaarb: device changed decodes: PCI:0000:03:00.0,olddecodes=io+mem,decodes=none:owns=none

like that on 346.35.

Whatever, that's not the problem - i've just shared my experience with non-primary-VGA being used with VFIO. I hope he didn't drain his system yet.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4117 2015-02-08 21:40:09

Tim_J
Member
Registered: 2015-02-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

5. My distro(fedora) had plymouth installed, you have to check /proc/iomem for something like vesafb appearing on 01:00.0.

I don't see anything like that. Here's my /proc/iomem:

00000000-00000fff : reserved 00001000-0009e7ff : System RAM 0009e800-0009ffff : reserved 000a0000-000bffff : PCI Bus 0000:00 000c0000-000dffff : PCI Bus 0000:00 000c0000-000c7fff : Video ROM 000e0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-9beb8fff : System RAM 01000000-015124cf : Kernel code 015124d0-018eb8ff : Kernel data 01a1f000-01af0fff : Kernel bss 9beb9000-9c49ffff : reserved 9c4a0000-9c5adfff : ACPI Tables 9c5ae000-9c7d4fff : ACPI Non-volatile Storage 9c7d5000-9d74afff : reserved 9d74b000-9d74bfff : System RAM 9d74c000-9d7d1fff : ACPI Non-volatile Storage 9d7d2000-9dc11fff : System RAM 9dc12000-9dff3fff : reserved 9dff4000-9dffffff : System RAM 9e000000-9fffffff : RAM buffer a0000000-ffffffff : PCI Bus 0000:00 a0000000-b1ffffff : PCI Bus 0000:04 a0000000-afffffff : 0000:04:00.0 b0000000-b1ffffff : 0000:04:00.0 b8000000-c1ffffff : PCI Bus 0000:03 b8000000-bfffffff : 0000:03:00.0 c0000000-c1ffffff : 0000:03:00.0 c8000000-d1ffffff : PCI Bus 0000:01 c8000000-cfffffff : 0000:01:00.0 d0000000-d1ffffff : 0000:01:00.0 e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff] e0000000-efffffff : reserved f6000000-f70fffff : PCI Bus 0000:04 f6000000-f6ffffff : 0000:04:00.0 f7000000-f707ffff : 0000:04:00.0 f7080000-f7083fff : 0000:04:00.1 f7080000-f7083fff : ICH HD audio f7100000-f711ffff : 0000:00:19.0 f7100000-f711ffff : e1000e f7120000-f7123fff : 0000:00:1b.0 f7120000-f7123fff : ICH HD audio f7124000-f71240ff : 0000:00:1f.3 f7125000-f71257ff : 0000:00:1f.2 f7125000-f71257ff : ahci f7126000-f71263ff : 0000:00:1d.0 f7126000-f71263ff : ehci_hcd f7127000-f71273ff : 0000:00:1a.0 f7127000-f71273ff : ehci_hcd f7128000-f7128fff : 0000:00:19.0 f7128000-f7128fff : e1000e f7129000-f712900f : 0000:00:16.0 f7129000-f712900f : mei_me f712a000-f712afff : 0000:00:05.4 f8000000-f90fffff : PCI Bus 0000:03 f8000000-f8ffffff : 0000:03:00.0 f9000000-f907ffff : 0000:03:00.0 f9080000-f9083fff : 0000:03:00.1 fa000000-fb0fffff : PCI Bus 0000:01 fa000000-faffffff : 0000:01:00.0 fb000000-fb07ffff : 0000:01:00.0 fb080000-fb083fff : 0000:01:00.1 fb200000-fb2fffff : PCI Bus 0000:0c fb200000-fb2001ff : 0000:0c:00.0 fb200000-fb2001ff : ahci fb300000-fb3fffff : PCI Bus 0000:0b fb300000-fb3001ff : 0000:0b:00.0 fb300000-fb3001ff : ahci fb400000-fb4fffff : PCI Bus 0000:0a fb400000-fb407fff : 0000:0a:00.0 fb400000-fb407fff : xhci_hcd fb500000-fb5fffff : PCI Bus 0000:09 fb500000-fb507fff : 0000:09:00.0 fb500000-fb507fff : xhci_hcd fb600000-fb6fffff : PCI Bus 0000:08 fb600000-fb607fff : 0000:08:00.0 fb600000-fb607fff : xhci_hcd fb700000-fb7fffff : PCI Bus 0000:07 fb700000-fb707fff : 0000:07:00.0 fb700000-fb707fff : xhci_hcd fbffc000-fbffcfff : dmar0 fc000000-fcffffff : pnp 00:00 fd000000-fdffffff : pnp 00:00 fe000000-feafffff : pnp 00:00 feb00000-febfffff : pnp 00:00 fec00000-fec003ff : IOAPIC 0 fec01000-fec013ff : IOAPIC 1 fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed1c000-fed1ffff : reserved fed1c000-fed1ffff : pnp 00:05 fed1f410-fed1f414 : iTCO_wdt fed1f410-fed1f414 : iTCO_wdt fed45000-fedfffff : pnp 00:00 fee00000-feefffff : pnp 00:00 fee00000-fee00fff : Local APIC ff000000-ffffffff : reserved ff000000-ffffffff : pnp 00:05 100000000-65fffffff : System RAM
Duelist wrote:

1. Give us your dmesg as the VM boots. Maybe there's something useful there.

After booting, 01 and 03 are bound to pci-stub. I run a script to rebind them to vfio, and this is the only addition to dmesg:

VFIO - User Level meta-driver version: 0.3

When I attempt to start the VM, no new messages are written.

Here's an excerpt of my entire dmesg log, from boot to running the VM:

... [ 0.000000] Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.3 (Debian 4.8.3-16) ) #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) ... [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=/dev/mapper/vg1-lv.root ro quiet intel_iommu=on ... [ 0.000000] DMI: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 4901 05/14/2014 ... [ 0.000000] Intel-IOMMU: enabled ... [ 0.000000] Memory: 24661772K/25102988K available (5192K kernel code, 942K rwdata, 1828K rodata, 1200K init, 840K bss, 441216K reserved) ... [ 0.053422] Freeing SMP alternatives memory: 20K (ffffffff81a19000 - ffffffff81a1e000) [ 0.054163] ftrace: allocating 21561 entries in 85 pages [ 0.060881] dmar: Host address width 46 [ 0.060883] dmar: DRHD base: 0x000000fbffc000 flags: 0x1 [ 0.060890] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de [ 0.060891] dmar: RMRR base: 0x0000009c475000 end: 0x0000009c482fff [ 0.060892] dmar: ATSR flags: 0x0 [ 0.060893] dmar: RHSA base: 0x000000fbffc000 proximity domain: 0x0 [ 0.060982] IOAPIC id 0 under DRHD base 0xfbffc000 IOMMU 0 [ 0.060982] IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 0 [ 0.060983] HPET id 0 under DRHD base 0xfbffc000 [ 0.060984] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.061108] Enabled IRQ remapping in x2apic mode [ 0.061109] Enabling x2apic [ 0.061110] Enabled x2apic [ 0.061113] Switched APIC routing to cluster x2apic. [ 0.061610] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.101249] smpboot: CPU0: Intel(R) Core(TM) i7-4960X CPU @ 3.60GHz (fam: 06, model: 3e, stepping: 04) [ 0.101254] TSC deadline timer enabled [ 0.101270] Performance Events: PEBS fmt1+, 16-deep LBR, IvyBridge events, full-width counters, Intel PMU driver. [ 0.101284] ... version: 3 [ 0.101285] ... bit width: 48 [ 0.101285] ... generic registers: 4 [ 0.101286] ... value mask: 0000ffffffffffff [ 0.101286] ... max period: 0000ffffffffffff [ 0.101287] ... fixed-purpose events: 3 [ 0.101287] ... event mask: 000000070000000f [ 0.102318] x86: Booting SMP configuration: [ 0.102319] .... node #0, CPUs: #1 [ 0.117314] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.117394] #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 [ 0.255363] x86: Booted up 1 node, 12 CPUs [ 0.255366] smpboot: Total of 12 processors activated (86439.09 BogoMIPS) ... [ 0.270631] ACPI: bus type PCI registered [ 0.270632] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.270701] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.270702] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820 [ 0.270903] PCI: Using configuration type 1 for base access [ 0.282954] ACPI: Added _OSI(Module Device) [ 0.282956] ACPI: Added _OSI(Processor Device) [ 0.282957] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282957] ACPI: Added _OSI(Processor Aggregator Device) [ 0.291313] ACPI: Executed 1 blocks of module-level executable AML code [ 0.379221] ACPI: Interpreter enabled [ 0.379226] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20140424/hwxface-580) [ 0.379228] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20140424/hwxface-580) [ 0.379236] ACPI: (supports S0 S3 S4 S5) [ 0.379237] ACPI: Using IOAPIC for interrupt routing [ 0.379258] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.388311] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe]) [ 0.388315] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.388417] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug PME AER] [ 0.388510] acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability] [ 0.388747] PCI host bridge to bus 0000:00 [ 0.388749] pci_bus 0000:00: root bus resource [bus 00-fe] [ 0.388751] pci_bus 0000:00: root bus resource [io 0x0000-0x03af] [ 0.388752] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7] [ 0.388753] pci_bus 0000:00: root bus resource [io 0x03b0-0x03df] [ 0.388753] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.388755] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.388755] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000dffff] [ 0.388756] pci_bus 0000:00: root bus resource [mem 0xa0000000-0xffffffff] [ 0.388765] pci 0000:00:00.0: [8086:0e00] type 00 class 0x060000 [ 0.388820] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold [ 0.388882] pci 0000:00:01.0: [8086:0e02] type 01 class 0x060400 [ 0.388947] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 0.388984] pci 0000:00:01.0: System wakeup disabled by ACPI [ 0.389015] pci 0000:00:02.0: [8086:0e04] type 01 class 0x060400 [ 0.389080] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 0.389116] pci 0000:00:02.0: System wakeup disabled by ACPI [ 0.389147] pci 0000:00:03.0: [8086:0e08] type 01 class 0x060400 [ 0.389213] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 0.389248] pci 0000:00:03.0: System wakeup disabled by ACPI [ 0.389277] pci 0000:00:03.2: [8086:0e0a] type 01 class 0x060400 [ 0.389341] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold [ 0.389377] pci 0000:00:03.2: System wakeup disabled by ACPI [ 0.389403] pci 0000:00:05.0: [8086:0e28] type 00 class 0x088000 [ 0.389492] pci 0000:00:05.2: [8086:0e2a] type 00 class 0x088000 [ 0.389579] pci 0000:00:05.4: [8086:0e2c] type 00 class 0x080020 [ 0.389588] pci 0000:00:05.4: reg 0x10: [mem 0xf712a000-0xf712afff] [ 0.389691] pci 0000:00:11.0: [8086:1d3e] type 01 class 0x060400 [ 0.389776] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold [ 0.389847] pci 0000:00:16.0: [8086:1d3a] type 00 class 0x078000 [ 0.389867] pci 0000:00:16.0: reg 0x10: [mem 0xf7129000-0xf712900f 64bit] [ 0.389934] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 0.389995] pci 0000:00:19.0: [8086:1503] type 00 class 0x020000 [ 0.390009] pci 0000:00:19.0: reg 0x10: [mem 0xf7100000-0xf711ffff] [ 0.390016] pci 0000:00:19.0: reg 0x14: [mem 0xf7128000-0xf7128fff] [ 0.390023] pci 0000:00:19.0: reg 0x18: [io 0xf040-0xf05f] [ 0.390076] pci 0000:00:19.0: PME# supported from D0 D3hot D3cold [ 0.390105] pci 0000:00:19.0: System wakeup disabled by ACPI [ 0.390136] pci 0000:00:1a.0: [8086:1d2d] type 00 class 0x0c0320 [ 0.390154] pci 0000:00:1a.0: reg 0x10: [mem 0xf7127000-0xf71273ff] [ 0.390233] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 0.390264] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 0.390293] pci 0000:00:1b.0: [8086:1d20] type 00 class 0x040300 [ 0.390307] pci 0000:00:1b.0: reg 0x10: [mem 0xf7120000-0xf7123fff 64bit] [ 0.390371] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold [ 0.390426] pci 0000:00:1c.0: [8086:1d10] type 01 class 0x060400 [ 0.390501] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 0.390519] pci 0000:00:1c.0: Disabling UPDCR peer decodes [ 0.390523] pci 0000:00:1c.0: Enabling MPC IRBNCE [ 0.390525] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled [ 0.390548] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 0.390576] pci 0000:00:1c.1: [8086:1d12] type 01 class 0x060400 [ 0.390648] pci 0000:00:1c.1: PME# supported from D0 D3hot D3cold [ 0.390667] pci 0000:00:1c.1: Enabling MPC IRBNCE [ 0.390669] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled [ 0.390691] pci 0000:00:1c.1: System wakeup disabled by ACPI [ 0.390718] pci 0000:00:1c.2: [8086:1d14] type 01 class 0x060400 [ 0.390790] pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold [ 0.390808] pci 0000:00:1c.2: Enabling MPC IRBNCE [ 0.390810] pci 0000:00:1c.2: Intel PCH root port ACS workaround enabled [ 0.390831] pci 0000:00:1c.2: System wakeup disabled by ACPI [ 0.390860] pci 0000:00:1c.3: [8086:1d16] type 01 class 0x060400 [ 0.390932] pci 0000:00:1c.3: PME# supported from D0 D3hot D3cold [ 0.390950] pci 0000:00:1c.3: Enabling MPC IRBNCE [ 0.390951] pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled [ 0.390973] pci 0000:00:1c.3: System wakeup disabled by ACPI [ 0.391000] pci 0000:00:1c.4: [8086:1d18] type 01 class 0x060400 [ 0.391072] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold [ 0.391090] pci 0000:00:1c.4: Enabling MPC IRBNCE [ 0.391092] pci 0000:00:1c.4: Intel PCH root port ACS workaround enabled [ 0.391113] pci 0000:00:1c.4: System wakeup disabled by ACPI [ 0.391140] pci 0000:00:1c.5: [8086:1d1a] type 01 class 0x060400 [ 0.391212] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold [ 0.391231] pci 0000:00:1c.5: Enabling MPC IRBNCE [ 0.391233] pci 0000:00:1c.5: Intel PCH root port ACS workaround enabled [ 0.391254] pci 0000:00:1c.5: System wakeup disabled by ACPI [ 0.391283] pci 0000:00:1c.7: [8086:1d1e] type 01 class 0x060400 [ 0.391355] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 0.391373] pci 0000:00:1c.7: Enabling MPC IRBNCE [ 0.391375] pci 0000:00:1c.7: Intel PCH root port ACS workaround enabled [ 0.391396] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 0.391427] pci 0000:00:1d.0: [8086:1d26] type 00 class 0x0c0320 [ 0.391445] pci 0000:00:1d.0: reg 0x10: [mem 0xf7126000-0xf71263ff] [ 0.391524] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 0.391555] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 0.391581] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401 [ 0.391645] pci 0000:00:1e.0: System wakeup disabled by ACPI [ 0.391675] pci 0000:00:1f.0: [8086:1d41] type 00 class 0x060100 [ 0.391818] pci 0000:00:1f.2: [8086:1d02] type 00 class 0x010601 [ 0.391833] pci 0000:00:1f.2: reg 0x10: [io 0xf090-0xf097] [ 0.391839] pci 0000:00:1f.2: reg 0x14: [io 0xf080-0xf083] [ 0.391845] pci 0000:00:1f.2: reg 0x18: [io 0xf070-0xf077] [ 0.391851] pci 0000:00:1f.2: reg 0x1c: [io 0xf060-0xf063] [ 0.391857] pci 0000:00:1f.2: reg 0x20: [io 0xf020-0xf03f] [ 0.391863] pci 0000:00:1f.2: reg 0x24: [mem 0xf7125000-0xf71257ff] [ 0.391900] pci 0000:00:1f.2: PME# supported from D3hot [ 0.391951] pci 0000:00:1f.3: [8086:1d22] type 00 class 0x0c0500 [ 0.391963] pci 0000:00:1f.3: reg 0x10: [mem 0xf7124000-0xf71240ff 64bit] [ 0.391981] pci 0000:00:1f.3: reg 0x20: [io 0xf000-0xf01f] [ 0.392074] pci 0000:00:01.0: PCI bridge to [bus 02] [ 0.392123] pci 0000:01:00.0: [10de:1080] type 00 class 0x030000 [ 0.392131] pci 0000:01:00.0: reg 0x10: [mem 0xfa000000-0xfaffffff] [ 0.392138] pci 0000:01:00.0: reg 0x14: [mem 0xc8000000-0xcfffffff 64bit pref] [ 0.392144] pci 0000:01:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref] [ 0.392149] pci 0000:01:00.0: reg 0x24: [io 0xe000-0xe07f] [ 0.392154] pci 0000:01:00.0: reg 0x30: [mem 0xfb000000-0xfb07ffff pref] [ 0.392216] pci 0000:01:00.1: [10de:0e09] type 00 class 0x040300 [ 0.392223] pci 0000:01:00.1: reg 0x10: [mem 0xfb080000-0xfb083fff] [ 0.398491] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.398494] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.398496] pci 0000:00:02.0: bridge window [mem 0xfa000000-0xfb0fffff] [ 0.398500] pci 0000:00:02.0: bridge window [mem 0xc8000000-0xd1ffffff 64bit pref] [ 0.398546] pci 0000:03:00.0: [10de:1080] type 00 class 0x030000 [ 0.398553] pci 0000:03:00.0: reg 0x10: [mem 0xf8000000-0xf8ffffff] [ 0.398559] pci 0000:03:00.0: reg 0x14: [mem 0xb8000000-0xbfffffff 64bit pref] [ 0.398565] pci 0000:03:00.0: reg 0x1c: [mem 0xc0000000-0xc1ffffff 64bit pref] [ 0.398569] pci 0000:03:00.0: reg 0x24: [io 0xd000-0xd07f] [ 0.398573] pci 0000:03:00.0: reg 0x30: [mem 0xf9000000-0xf907ffff pref] [ 0.398640] pci 0000:03:00.1: [10de:0e09] type 00 class 0x040300 [ 0.398648] pci 0000:03:00.1: reg 0x10: [mem 0xf9080000-0xf9083fff] [ 0.406471] pci 0000:00:03.0: PCI bridge to [bus 03] [ 0.406477] pci 0000:00:03.0: bridge window [io 0xd000-0xdfff] [ 0.406482] pci 0000:00:03.0: bridge window [mem 0xf8000000-0xf90fffff] [ 0.406500] pci 0000:00:03.0: bridge window [mem 0xb8000000-0xc1ffffff 64bit pref] [ 0.406546] pci 0000:04:00.0: [10de:0a65] type 00 class 0x030000 [ 0.406553] pci 0000:04:00.0: reg 0x10: [mem 0xf6000000-0xf6ffffff] [ 0.406559] pci 0000:04:00.0: reg 0x14: [mem 0xa0000000-0xafffffff 64bit pref] [ 0.406565] pci 0000:04:00.0: reg 0x1c: [mem 0xb0000000-0xb1ffffff 64bit pref] [ 0.406569] pci 0000:04:00.0: reg 0x24: [io 0xc000-0xc07f] [ 0.406573] pci 0000:04:00.0: reg 0x30: [mem 0xf7000000-0xf707ffff pref] [ 0.406639] pci 0000:04:00.1: [10de:0be3] type 00 class 0x040300 [ 0.406647] pci 0000:04:00.1: reg 0x10: [mem 0xf7080000-0xf7083fff] [ 0.414468] pci 0000:00:03.2: PCI bridge to [bus 04] [ 0.414474] pci 0000:00:03.2: bridge window [io 0xc000-0xcfff] [ 0.414479] pci 0000:00:03.2: bridge window [mem 0xf6000000-0xf70fffff] [ 0.414498] pci 0000:00:03.2: bridge window [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.414548] pci 0000:00:11.0: PCI bridge to [bus 05] [ 0.414602] pci 0000:00:1c.0: PCI bridge to [bus 06] [ 0.414679] pci 0000:07:00.0: [1b21:1042] type 00 class 0x0c0330 [ 0.414709] pci 0000:07:00.0: reg 0x10: [mem 0xfb700000-0xfb707fff 64bit] [ 0.414858] pci 0000:07:00.0: PME# supported from D3hot D3cold [ 0.422467] pci 0000:00:1c.1: PCI bridge to [bus 07] [ 0.422475] pci 0000:00:1c.1: bridge window [mem 0xfb700000-0xfb7fffff] [ 0.422564] pci 0000:08:00.0: [1b21:1042] type 00 class 0x0c0330 [ 0.422594] pci 0000:08:00.0: reg 0x10: [mem 0xfb600000-0xfb607fff 64bit] [ 0.422742] pci 0000:08:00.0: PME# supported from D3hot D3cold [ 0.430463] pci 0000:00:1c.2: PCI bridge to [bus 08] [ 0.430471] pci 0000:00:1c.2: bridge window [mem 0xfb600000-0xfb6fffff] [ 0.430559] pci 0000:09:00.0: [1b21:1042] type 00 class 0x0c0330 [ 0.430589] pci 0000:09:00.0: reg 0x10: [mem 0xfb500000-0xfb507fff 64bit] [ 0.430738] pci 0000:09:00.0: PME# supported from D3hot D3cold [ 0.438460] pci 0000:00:1c.3: PCI bridge to [bus 09] [ 0.438469] pci 0000:00:1c.3: bridge window [mem 0xfb500000-0xfb5fffff] [ 0.438559] pci 0000:0a:00.0: [1b21:1042] type 00 class 0x0c0330 [ 0.438589] pci 0000:0a:00.0: reg 0x10: [mem 0xfb400000-0xfb407fff 64bit] [ 0.438737] pci 0000:0a:00.0: PME# supported from D3hot D3cold [ 0.446456] pci 0000:00:1c.4: PCI bridge to [bus 0a] [ 0.446465] pci 0000:00:1c.4: bridge window [mem 0xfb400000-0xfb4fffff] [ 0.446549] pci 0000:0b:00.0: [1b21:0612] type 00 class 0x010601 [ 0.446569] pci 0000:0b:00.0: reg 0x10: [io 0xb050-0xb057] [ 0.446581] pci 0000:0b:00.0: reg 0x14: [io 0xb040-0xb043] [ 0.446593] pci 0000:0b:00.0: reg 0x18: [io 0xb030-0xb037] [ 0.446605] pci 0000:0b:00.0: reg 0x1c: [io 0xb020-0xb023] [ 0.446617] pci 0000:0b:00.0: reg 0x20: [io 0xb000-0xb01f] [ 0.446629] pci 0000:0b:00.0: reg 0x24: [mem 0xfb300000-0xfb3001ff] [ 0.454453] pci 0000:00:1c.5: PCI bridge to [bus 0b] [ 0.454459] pci 0000:00:1c.5: bridge window [io 0xb000-0xbfff] [ 0.454464] pci 0000:00:1c.5: bridge window [mem 0xfb300000-0xfb3fffff] [ 0.454548] pci 0000:0c:00.0: [1b21:0612] type 00 class 0x010601 [ 0.454568] pci 0000:0c:00.0: reg 0x10: [io 0xa050-0xa057] [ 0.454580] pci 0000:0c:00.0: reg 0x14: [io 0xa040-0xa043] [ 0.454592] pci 0000:0c:00.0: reg 0x18: [io 0xa030-0xa037] [ 0.454604] pci 0000:0c:00.0: reg 0x1c: [io 0xa020-0xa023] [ 0.454616] pci 0000:0c:00.0: reg 0x20: [io 0xa000-0xa01f] [ 0.454628] pci 0000:0c:00.0: reg 0x24: [mem 0xfb200000-0xfb2001ff] [ 0.462450] pci 0000:00:1c.7: PCI bridge to [bus 0c] [ 0.462456] pci 0000:00:1c.7: bridge window [io 0xa000-0xafff] [ 0.462461] pci 0000:00:1c.7: bridge window [mem 0xfb200000-0xfb2fffff] [ 0.462536] pci 0000:00:1e.0: PCI bridge to [bus 0d] (subtractive decode) [ 0.462543] pci 0000:00:1e.0: bridge window [io 0x0000-0x03af] (subtractive decode) [ 0.462544] pci 0000:00:1e.0: bridge window [io 0x03e0-0x0cf7] (subtractive decode) [ 0.462545] pci 0000:00:1e.0: bridge window [io 0x03b0-0x03df] (subtractive decode) [ 0.462546] pci 0000:00:1e.0: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.462547] pci 0000:00:1e.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.462548] pci 0000:00:1e.0: bridge window [mem 0x000c0000-0x000dffff] (subtractive decode) [ 0.462549] pci 0000:00:1e.0: bridge window [mem 0xa0000000-0xffffffff] (subtractive decode) [ 0.462820] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus ff]) [ 0.462822] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.462836] acpi PNP0A03:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability] [ 0.462869] PCI host bridge to bus 0000:ff [ 0.462870] pci_bus 0000:ff: root bus resource [bus ff] [ 0.462876] pci 0000:ff:08.0: [8086:0e80] type 00 class 0x088000 [ 0.462914] pci 0000:ff:09.0: [8086:0e90] type 00 class 0x088000 [ 0.462950] pci 0000:ff:0a.0: [8086:0ec0] type 00 class 0x088000 [ 0.462981] pci 0000:ff:0a.1: [8086:0ec1] type 00 class 0x088000 [ 0.463012] pci 0000:ff:0a.2: [8086:0ec2] type 00 class 0x088000 [ 0.463042] pci 0000:ff:0a.3: [8086:0ec3] type 00 class 0x088000 [ 0.463073] pci 0000:ff:0b.0: [8086:0e1e] type 00 class 0x088000 [ 0.463102] pci 0000:ff:0b.3: [8086:0e1f] type 00 class 0x088000 [ 0.463131] pci 0000:ff:0c.0: [8086:0ee0] type 00 class 0x088000 [ 0.463159] pci 0000:ff:0c.1: [8086:0ee2] type 00 class 0x088000 [ 0.463187] pci 0000:ff:0c.2: [8086:0ee4] type 00 class 0x088000 [ 0.463217] pci 0000:ff:0d.0: [8086:0ee1] type 00 class 0x088000 [ 0.463247] pci 0000:ff:0d.1: [8086:0ee3] type 00 class 0x088000 [ 0.463275] pci 0000:ff:0d.2: [8086:0ee5] type 00 class 0x088000 [ 0.463306] pci 0000:ff:0e.0: [8086:0ea0] type 00 class 0x088000 [ 0.463336] pci 0000:ff:0e.1: [8086:0e30] type 00 class 0x110100 [ 0.463371] pci 0000:ff:0f.0: [8086:0ea8] type 00 class 0x088000 [ 0.463412] pci 0000:ff:0f.1: [8086:0e71] type 00 class 0x088000 [ 0.463453] pci 0000:ff:0f.2: [8086:0eaa] type 00 class 0x088000 [ 0.463494] pci 0000:ff:0f.3: [8086:0eab] type 00 class 0x088000 [ 0.463534] pci 0000:ff:0f.4: [8086:0eac] type 00 class 0x088000 [ 0.463575] pci 0000:ff:0f.5: [8086:0ead] type 00 class 0x088000 [ 0.463616] pci 0000:ff:10.0: [8086:0eb0] type 00 class 0x088000 [ 0.463658] pci 0000:ff:10.1: [8086:0eb1] type 00 class 0x088000 [ 0.463701] pci 0000:ff:10.2: [8086:0eb2] type 00 class 0x088000 [ 0.463742] pci 0000:ff:10.3: [8086:0eb3] type 00 class 0x088000 [ 0.463784] pci 0000:ff:10.4: [8086:0eb4] type 00 class 0x088000 [ 0.463826] pci 0000:ff:10.5: [8086:0eb5] type 00 class 0x088000 [ 0.463868] pci 0000:ff:10.6: [8086:0eb6] type 00 class 0x088000 [ 0.463909] pci 0000:ff:10.7: [8086:0eb7] type 00 class 0x088000 [ 0.463950] pci 0000:ff:13.0: [8086:0e1d] type 00 class 0x088000 [ 0.463979] pci 0000:ff:13.1: [8086:0e34] type 00 class 0x110100 [ 0.464009] pci 0000:ff:13.4: [8086:0e81] type 00 class 0x088000 [ 0.464038] pci 0000:ff:13.5: [8086:0e36] type 00 class 0x110100 [ 0.464069] pci 0000:ff:16.0: [8086:0ec8] type 00 class 0x088000 [ 0.464098] pci 0000:ff:16.1: [8086:0ec9] type 00 class 0x088000 [ 0.464127] pci 0000:ff:16.2: [8086:0eca] type 00 class 0x088000 [ 0.464218] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12 14 15) [ 0.464251] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *10 11 12 14 15) [ 0.464283] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 11 12 14 *15) [ 0.464314] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 *4 5 6 10 11 12 14 15) [ 0.464345] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 *14 15) [ 0.464376] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 *5 6 7 10 11 12 14 15) [ 0.464407] ACPI: PCI Interrupt Link [LNKG] (IRQs *3 4 5 6 7 10 11 12 14 15) [ 0.464438] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *7 10 11 12 14 15) [ 0.466723] ACPI: Enabled 3 GPEs in block 00 to 3F [ 0.466804] vgaarb: setting as boot device: PCI:0000:01:00.0 [ 0.466806] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.466808] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=none,locks=none [ 0.466810] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none [ 0.466814] vgaarb: loaded [ 0.466815] vgaarb: bridge control possible 0000:04:00.0 [ 0.466816] vgaarb: bridge control possible 0000:03:00.0 [ 0.466816] vgaarb: bridge control possible 0000:01:00.0 [ 0.466857] PCI: Using ACPI for IRQ routing [ 0.472217] PCI: pci_cache_line_size set to 64 bytes [ 0.472319] e820: reserve RAM buffer [mem 0x0009e800-0x0009ffff] [ 0.472320] e820: reserve RAM buffer [mem 0x9beb9000-0x9bffffff] [ 0.472321] e820: reserve RAM buffer [mem 0x9d74c000-0x9fffffff] [ 0.472322] e820: reserve RAM buffer [mem 0x9dc12000-0x9fffffff] [ 0.472323] e820: reserve RAM buffer [mem 0x9e000000-0x9fffffff] [ 0.472427] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 0.472430] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 0.474466] Switched to clocksource hpet [ 0.478098] pnp: PnP ACPI init [ 0.478106] ACPI: bus type PNP registered [ 0.478172] system 00:00: [mem 0xfc000000-0xfcffffff] has been reserved [ 0.478174] system 00:00: [mem 0xfd000000-0xfdffffff] has been reserved [ 0.478175] system 00:00: [mem 0xfe000000-0xfeafffff] has been reserved [ 0.478176] system 00:00: [mem 0xfeb00000-0xfebfffff] has been reserved [ 0.478177] system 00:00: [mem 0xfed00400-0xfed3ffff] could not be reserved [ 0.478179] system 00:00: [mem 0xfed45000-0xfedfffff] has been reserved [ 0.478180] system 00:00: [mem 0xfee00000-0xfeefffff] has been reserved [ 0.478182] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.478236] system 00:01: [mem 0xfbffc000-0xfbffdfff] could not be reserved [ 0.478237] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.478304] system 00:02: [io 0x0290-0x029f] has been reserved [ 0.478306] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.478329] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.478377] system 00:04: [io 0x04d0-0x04d1] has been reserved [ 0.478380] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.478538] system 00:05: [io 0x0400-0x0453] could not be reserved [ 0.478539] system 00:05: [io 0x0458-0x047f] has been reserved [ 0.478540] system 00:05: [io 0x1180-0x119f] has been reserved [ 0.478541] system 00:05: [io 0x0500-0x057f] has been reserved [ 0.478543] system 00:05: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 0.478544] system 00:05: [mem 0xfec00000-0xfecfffff] could not be reserved [ 0.478546] system 00:05: [mem 0xff000000-0xffffffff] has been reserved [ 0.478547] system 00:05: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.478594] system 00:06: [io 0x0454-0x0457] has been reserved [ 0.478596] system 00:06: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 0.478764] pnp: PnP ACPI: found 7 devices [ 0.478764] ACPI: bus type PNP unregistered [ 0.484701] pci 0000:00:01.0: PCI bridge to [bus 02] [ 0.484709] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.484710] pci 0000:00:02.0: bridge window [io 0xe000-0xefff] [ 0.484714] pci 0000:00:02.0: bridge window [mem 0xfa000000-0xfb0fffff] [ 0.484716] pci 0000:00:02.0: bridge window [mem 0xc8000000-0xd1ffffff 64bit pref] [ 0.484720] pci 0000:00:03.0: PCI bridge to [bus 03] [ 0.484721] pci 0000:00:03.0: bridge window [io 0xd000-0xdfff] [ 0.484725] pci 0000:00:03.0: bridge window [mem 0xf8000000-0xf90fffff] [ 0.484727] pci 0000:00:03.0: bridge window [mem 0xb8000000-0xc1ffffff 64bit pref] [ 0.484731] pci 0000:00:03.2: PCI bridge to [bus 04] [ 0.484732] pci 0000:00:03.2: bridge window [io 0xc000-0xcfff] [ 0.484735] pci 0000:00:03.2: bridge window [mem 0xf6000000-0xf70fffff] [ 0.484738] pci 0000:00:03.2: bridge window [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.484741] pci 0000:00:11.0: PCI bridge to [bus 05] [ 0.484752] pci 0000:00:1c.0: PCI bridge to [bus 06] [ 0.484761] pci 0000:00:1c.1: PCI bridge to [bus 07] [ 0.484764] pci 0000:00:1c.1: bridge window [mem 0xfb700000-0xfb7fffff] [ 0.484770] pci 0000:00:1c.2: PCI bridge to [bus 08] [ 0.484774] pci 0000:00:1c.2: bridge window [mem 0xfb600000-0xfb6fffff] [ 0.484780] pci 0000:00:1c.3: PCI bridge to [bus 09] [ 0.484784] pci 0000:00:1c.3: bridge window [mem 0xfb500000-0xfb5fffff] [ 0.484790] pci 0000:00:1c.4: PCI bridge to [bus 0a] [ 0.484794] pci 0000:00:1c.4: bridge window [mem 0xfb400000-0xfb4fffff] [ 0.484800] pci 0000:00:1c.5: PCI bridge to [bus 0b] [ 0.484802] pci 0000:00:1c.5: bridge window [io 0xb000-0xbfff] [ 0.484805] pci 0000:00:1c.5: bridge window [mem 0xfb300000-0xfb3fffff] [ 0.484812] pci 0000:00:1c.7: PCI bridge to [bus 0c] [ 0.484813] pci 0000:00:1c.7: bridge window [io 0xa000-0xafff] [ 0.484817] pci 0000:00:1c.7: bridge window [mem 0xfb200000-0xfb2fffff] [ 0.484823] pci 0000:00:1e.0: PCI bridge to [bus 0d] [ 0.484831] pci_bus 0000:00: resource 4 [io 0x0000-0x03af] [ 0.484832] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7] [ 0.484833] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df] [ 0.484834] pci_bus 0000:00: resource 7 [io 0x0d00-0xffff] [ 0.484835] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff] [ 0.484836] pci_bus 0000:00: resource 9 [mem 0x000c0000-0x000dffff] [ 0.484837] pci_bus 0000:00: resource 10 [mem 0xa0000000-0xffffffff] [ 0.484838] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.484839] pci_bus 0000:01: resource 1 [mem 0xfa000000-0xfb0fffff] [ 0.484840] pci_bus 0000:01: resource 2 [mem 0xc8000000-0xd1ffffff 64bit pref] [ 0.484841] pci_bus 0000:03: resource 0 [io 0xd000-0xdfff] [ 0.484842] pci_bus 0000:03: resource 1 [mem 0xf8000000-0xf90fffff] [ 0.484843] pci_bus 0000:03: resource 2 [mem 0xb8000000-0xc1ffffff 64bit pref] [ 0.484844] pci_bus 0000:04: resource 0 [io 0xc000-0xcfff] [ 0.484845] pci_bus 0000:04: resource 1 [mem 0xf6000000-0xf70fffff] [ 0.484846] pci_bus 0000:04: resource 2 [mem 0xa0000000-0xb1ffffff 64bit pref] [ 0.484847] pci_bus 0000:07: resource 1 [mem 0xfb700000-0xfb7fffff] [ 0.484848] pci_bus 0000:08: resource 1 [mem 0xfb600000-0xfb6fffff] [ 0.484849] pci_bus 0000:09: resource 1 [mem 0xfb500000-0xfb5fffff] [ 0.484850] pci_bus 0000:0a: resource 1 [mem 0xfb400000-0xfb4fffff] [ 0.484851] pci_bus 0000:0b: resource 0 [io 0xb000-0xbfff] [ 0.484852] pci_bus 0000:0b: resource 1 [mem 0xfb300000-0xfb3fffff] [ 0.484853] pci_bus 0000:0c: resource 0 [io 0xa000-0xafff] [ 0.484854] pci_bus 0000:0c: resource 1 [mem 0xfb200000-0xfb2fffff] [ 0.484855] pci_bus 0000:0d: resource 4 [io 0x0000-0x03af] [ 0.484856] pci_bus 0000:0d: resource 5 [io 0x03e0-0x0cf7] [ 0.484857] pci_bus 0000:0d: resource 6 [io 0x03b0-0x03df] [ 0.484858] pci_bus 0000:0d: resource 7 [io 0x0d00-0xffff] [ 0.484859] pci_bus 0000:0d: resource 8 [mem 0x000a0000-0x000bffff] [ 0.484860] pci_bus 0000:0d: resource 9 [mem 0x000c0000-0x000dffff] [ 0.484860] pci_bus 0000:0d: resource 10 [mem 0xa0000000-0xffffffff] ... [ 0.526512] pci 0000:01:00.0: Video device with shadowed ROM [ 0.526967] PCI: CLS 64 bytes, default 64 [ 0.526999] Unpacking initramfs... [ 0.715857] Freeing initrd memory: 16148K (ffff880036066000 - ffff88003702b000) [ 0.716025] IOMMU 0 0xfbffc000: using Queued invalidation [ 0.716028] IOMMU: Setting RMRR: [ 0.716038] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9c475000 - 0x9c482fff] [ 0.716063] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9c475000 - 0x9c482fff] [ 0.716077] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.716085] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.716098] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O ... [ 0.720581] ioapic: probe of 0000:00:05.4 failed with error -22 [ 0.720588] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.720598] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 ... [ 0.725407] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 0.725409] AMD IOMMUv2 functionality not available on this system ... [ 1.039938] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 1.039942] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 9 [ 1.039953] ehci-pci 0000:00:1a.0: debug port 2 [ 1.043862] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 1.043874] ehci-pci 0000:00:1a.0: irq 21, io mem 0xf7127000 [ 1.053896] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 ... [ 1.054233] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 1.054237] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 10 [ 1.054247] ehci-pci 0000:00:1d.0: debug port 2 [ 1.058160] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 1.058172] ehci-pci 0000:00:1d.0: irq 23, io mem 0xf7126000 [ 1.069906] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 ... [ 18.164918] pci-stub: add 10DE:1080 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 18.164926] pci-stub 0000:01:00.0: claimed by stub [ 18.164933] pci-stub 0000:03:00.0: claimed by stub [ 18.164938] pci-stub: add 10DE:0E09 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 18.164944] pci-stub 0000:01:00.1: claimed by stub [ 18.164950] pci-stub 0000:03:00.1: claimed by stub ... [ 18.235038] nouveau 0000:04:00.0: enabling device (0004 -> 0007) ... [ 18.235444] nouveau [ DEVICE][0000:04:00.0] BOOT0 : 0x0a8280b1 [ 18.235447] nouveau [ DEVICE][0000:04:00.0] Chipset: GT218 (NVA8) [ 18.235448] nouveau [ DEVICE][0000:04:00.0] Family : NV50 [ 18.235470] nouveau [ VBIOS][0000:04:00.0] checking PRAMIN for image... [ 18.235474] nouveau [ VBIOS][0000:04:00.0] ... signature not found [ 18.235475] nouveau [ VBIOS][0000:04:00.0] checking PROM for image... ... [ 18.348136] nouveau [ VBIOS][0000:04:00.0] ... appears to be valid [ 18.348138] nouveau [ VBIOS][0000:04:00.0] using image from PROM [ 18.348211] nouveau [ VBIOS][0000:04:00.0] BIT signature found [ 18.348212] nouveau [ VBIOS][0000:04:00.0] version 70.18.64.00.05 [ 18.348385] nouveau [ DEVINIT][0000:04:00.0] adaptor not initialised [ 18.348390] nouveau [ VBIOS][0000:04:00.0] running init tables ... [ 18.416916] nouveau 0000:04:00.0: irq 103 for MSI/MSI-X [ 18.416924] nouveau [ PMC][0000:04:00.0] MSI interrupts enabled [ 18.416947] nouveau [ PFB][0000:04:00.0] RAM type: DDR3 [ 18.416948] nouveau [ PFB][0000:04:00.0] RAM size: 1024 MiB [ 18.416949] nouveau [ PFB][0000:04:00.0] ZCOMP: 960 tags [ 18.418379] nouveau [ VOLT][0000:04:00.0] GPU voltage: 900000uv ... [ 18.936188] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.2/0000:04:00.1/sound/card1/input24 [ 18.936307] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.2/0000:04:00.1/sound/card1/input25 [ 18.937044] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.2/0000:04:00.1/sound/card1/input26 [ 18.937126] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:03.2/0000:04:00.1/sound/card1/input27 [ 19.695122] nouveau [ PTHERM][0000:04:00.0] FAN control: none / external [ 19.695129] nouveau [ PTHERM][0000:04:00.0] fan management: automatic [ 19.695131] nouveau [ PTHERM][0000:04:00.0] internal sensor: yes [ 19.695144] nouveau [ CLK][0000:04:00.0] 03: core 135 MHz shader 270 MHz memory 135 MHz [ 19.695146] nouveau [ CLK][0000:04:00.0] 07: core 405 MHz shader 810 MHz memory 405 MHz [ 19.695147] nouveau [ CLK][0000:04:00.0] 0f: core 589 MHz shader 1402 MHz memory 600 MHz [ 19.695161] nouveau [ CLK][0000:04:00.0] --: core 405 MHz shader 810 MHz memory 405 MHz [ 19.695327] [TTM] Zone kernel: Available graphics memory: 12373126 kiB [ 19.695328] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 19.695329] [TTM] Initializing pool allocator [ 19.695332] [TTM] Initializing DMA pool allocator [ 19.695338] nouveau [ DRM] VRAM: 1024 MiB [ 19.695339] nouveau [ DRM] GART: 1048576 MiB [ 19.695342] nouveau [ DRM] TMDS table version 2.0 [ 19.695343] nouveau [ DRM] DCB version 4.0 [ 19.695344] nouveau [ DRM] DCB outp 00: 01000302 00020030 [ 19.695345] nouveau [ DRM] DCB outp 01: 02000300 00000000 [ 19.695346] nouveau [ DRM] DCB outp 02: 02011362 00020010 [ 19.695347] nouveau [ DRM] DCB outp 03: 01022310 00000000 [ 19.695348] nouveau [ DRM] DCB conn 00: 00001030 [ 19.695349] nouveau [ DRM] DCB conn 01: 00002161 [ 19.695350] nouveau [ DRM] DCB conn 02: 00000200 [ 19.712678] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 19.712679] [drm] Driver supports precise vblank timestamp query. [ 19.731746] nouveau [ DRM] MM: using COPY for buffer copies [ 19.803827] nouveau [ DRM] allocated 1920x1080 fb: 0x70000, bo ffff880641f6b800 [ 19.896487] Console: switching to colour frame buffer device 240x67 [ 19.899643] nouveau 0000:04:00.0: fb0: nouveaufb frame buffer device [ 19.899644] nouveau 0000:04:00.0: registered panic notifier [ 19.918915] [drm] Initialized nouveau 1.1.2 20120801 for 0000:04:00.0 on minor 0 ... [ 20.231080] vgaarb: device changed decodes: PCI:0000:04:00.0,olddecodes=io+mem,decodes=none:owns=none ... [ 23.484645] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 262.576441] VFIO - User Level meta-driver version: 0.3

The only difference between 01 and 03 is that 01 appears to be selected by vgaarb as the boot device and something about "shadowed ROM". Here's a grep of the 01 card:

[ 0.392123] pci 0000:01:00.0: [10de:1080] type 00 class 0x030000 [ 0.392131] pci 0000:01:00.0: reg 0x10: [mem 0xfa000000-0xfaffffff] [ 0.392138] pci 0000:01:00.0: reg 0x14: [mem 0xc8000000-0xcfffffff 64bit pref] [ 0.392144] pci 0000:01:00.0: reg 0x1c: [mem 0xd0000000-0xd1ffffff 64bit pref] [ 0.392149] pci 0000:01:00.0: reg 0x24: [io 0xe000-0xe07f] [ 0.392154] pci 0000:01:00.0: reg 0x30: [mem 0xfb000000-0xfb07ffff pref] [ 0.392216] pci 0000:01:00.1: [10de:0e09] type 00 class 0x040300 [ 0.392223] pci 0000:01:00.1: reg 0x10: [mem 0xfb080000-0xfb083fff] [ 0.466804] vgaarb: setting as boot device: PCI:0000:01:00.0 [ 0.466806] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.466816] vgaarb: bridge control possible 0000:01:00.0 [ 0.526512] pci 0000:01:00.0: Video device with shadowed ROM [ 18.164926] pci-stub 0000:01:00.0: claimed by stub [ 18.164944] pci-stub 0000:01:00.1: claimed by stub

...vs a grep of the 03 card:

[ 0.398546] pci 0000:03:00.0: [10de:1080] type 00 class 0x030000 [ 0.398553] pci 0000:03:00.0: reg 0x10: [mem 0xf8000000-0xf8ffffff] [ 0.398559] pci 0000:03:00.0: reg 0x14: [mem 0xb8000000-0xbfffffff 64bit pref] [ 0.398565] pci 0000:03:00.0: reg 0x1c: [mem 0xc0000000-0xc1ffffff 64bit pref] [ 0.398569] pci 0000:03:00.0: reg 0x24: [io 0xd000-0xd07f] [ 0.398573] pci 0000:03:00.0: reg 0x30: [mem 0xf9000000-0xf907ffff pref] [ 0.398640] pci 0000:03:00.1: [10de:0e09] type 00 class 0x040300 [ 0.398648] pci 0000:03:00.1: reg 0x10: [mem 0xf9080000-0xf9083fff] [ 0.466808] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=none,locks=none [ 0.466816] vgaarb: bridge control possible 0000:03:00.0 [ 18.164933] pci-stub 0000:03:00.0: claimed by stub [ 18.164950] pci-stub 0000:03:00.1: claimed by stub

Offline

#4118 2015-02-08 22:15:48

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I updated the qemu package today to version 2.2.0-2 and my win7 vm refuses to start now. I browsed through the last couple of pages here, to find some hint for a change or problem..something like that - but there isn't.
I`m running kernel 3.18.5-1-mainline of the op and the vm was fine since my upgrade today.
Here is what the terminal says

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fb78cf646e0, 0x0, 0x80000000, 0x7fb65c000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fb78cf646e0, 0x100000000, 0x80000000, 0x7fb6dc000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Qemu complains about something related to vfio now. Can anyone give me some background about this and how to fix this?

Regards apex

Offline

#4119 2015-02-08 22:22:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi, I updated the qemu package today to version 2.2.0-2 and my win7 vm refuses to start now. I browsed through the last couple of pages here, to find some hint for a change or problem..something like that - but there isn't.
I`m running kernel 3.18.5-1-mainline of the op and the vm was fine since my upgrade today.
Here is what the terminal says

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fb78cf646e0, 0x0, 0x80000000, 0x7fb65c000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fb78cf646e0, 0x100000000, 0x80000000, 0x7fb6dc000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Qemu complains about something related to vfio now. Can anyone give me some background about this and how to fix this?

Does dmesg show RLIMIT_MEMLOCK errors?

Last edited by aw (2015-02-08 22:22:32)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4120 2015-02-08 22:30:30

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
apex8 wrote:

Hi, I updated the qemu package today to version 2.2.0-2 and my win7 vm refuses to start now. I browsed through the last couple of pages here, to find some hint for a change or problem..something like that - but there isn't.
I`m running kernel 3.18.5-1-mainline of the op and the vm was fine since my upgrade today.
Here is what the terminal says

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fb78cf646e0, 0x0, 0x80000000, 0x7fb65c000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fb78cf646e0, 0x100000000, 0x80000000, 0x7fb6dc000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Qemu complains about something related to vfio now. Can anyone give me some background about this and how to fix this?

Does dmesg show RLIMIT_MEMLOCK errors?

Yes, it does

[ 1052.293809] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded [ 1052.293817] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded [ 1052.303814] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded [ 1052.303821] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded

Offline

#4121 2015-02-08 22:36:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
aw wrote:

Does dmesg show RLIMIT_MEMLOCK errors?

Yes, it does

[ 1052.293809] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded [ 1052.293817] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded [ 1052.303814] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded [ 1052.303821] vfio_pin_pages: RLIMIT_MEMLOCK (40960000) exceeded

Nothing has changed here, the user always needs to be able to lock enough pages for the VM.  You're not going to get very far with a locked memory limit of 40MB.  If you use libvirt (and do not hide the vfio device in <qemu:arg>!) this happens automatically.  If you run qemu by hand, the easiest solution is to run as root or use sudo.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4122 2015-02-08 22:44:50

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Nothing has changed here, the user always needs to be able to lock enough pages for the VM.  You're not going to get very far with a locked memory limit of 40MB.  If you use libvirt (and do not hide the vfio device in <qemu:arg>!) this happens automatically.  If you run qemu by hand, the easiest solution is to run as root or use sudo.

Ah I remember that. I think I adjusted this via

cat /etc/security/limits.conf | grep @users @users soft memlock 4301000 @users hard memlock 4301000

This was fine to run qemu as non root until now..

Offline

#4123 2015-02-09 07:20:44

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone, i've been reading a lot of posts here, and i wanted to try the passthrough by myself, but seemed that i got some error here, and firstly according to the steps of the guide, i have been doing step by steps, and here's my build :

Processor : Xeon E3-1200 (VT-d Enabled in BIOS)
ubuntu display GPU : Radeon HD4870
Passthrough GPU : GeForce GTX 560 Ti Hawx

So steps i've been done so far :
1. Ubuntu Server 12.04, then do-release-upgrade to 14.04
2. Downloaded Kernel 3.18-5 from Linux homepage
3. Downloaded 3.18-5 linux-mainline from nhbs's first post
4. Patched ACS and i915_317 patch from linux-mainline from nbhs's post
5. Qemu 2.2.0, also Seabios standard and no patches
6. lspci | grep NVIDIA :
 

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)

7. edited /etc/default/grub :
 

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c"

    and our /proc/cmdline :
 

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7

8. also blacklisted radeon on /etc/modprobe.d/blacklist.conf
9. dmesg pci-stub :

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[    2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787587] pci-stub 0000:05:00.0: claimed by stub
[    2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787597] pci-stub 0000:05:00.1: claimed by stub

10. i checked lspci -vnn for the graphics cards in the device :

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6220000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at b000 [size=256]
        Expansion ROM at f6200000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
        Subsystem: ASUSTeK Computer Inc. Device [1043:aa30]
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at f6230000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: snd_hda_intel

04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441]
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6120000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at a000 [size=256]
        Expansion ROM at f6100000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: fast devsel, IRQ 11
        Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M]
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M]
        I/O ports at e000 [disabled] [size=128]
        Expansion ROM at f6000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: pci-stub

05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: bus master, fast devsel, latency 0, IRQ 10
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: pci-stub

11. after all went good, i create vfio-pci1.cfg in /etc/vfio-pci1.cfg :

0000:05:00.0
0000:05:00.1

12. Booting VM's Script :

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -vga none -M q35 -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-drive file=/home/sslab719/VMimages/VM.img,id=disk,format=qcow2 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-net user,hostfwd=tcp::10022-:22 -net nic
#-boot menu=on

exit 0

Here at this point, i am always having black QEMU Monitor with "compat_monitor0 console"
Qemu 2.2.0 monitor - type help for more information -- when i'm using -vga none right over here.

If i remove -vga none there, i can boot to the VMs normally, and i can see the lspci went in correctly, but it does seem like it doesn't have the real GPU power... the passthrough went there and it shows up in lspci but i am guessing that theres no GPU clock, shader, and memory power.. I tried to install heaven engine benchmark in linux, and i tried to run it on, and it fails, and here i am guessing that perhaps the problem might be the GPU doesn't really passthrough successfully, but instead its just a name showup on the VMs, and i can still see 00:01.0 VGA compatible Controller: Device 1234:1111, and the others are our passthrough'd GPU 01:00.0..

Here by reading i know that we should add -vga none in order to passthrough it normally, but it seemed that -vga none didn't bring a help for me, and instead its just gave me a weird black qemu monitor which i can type some commands of qemu itself.

i tried -vga std and others, but those didn't worked too..

so i checked my dmesg when turning my VM to see whether vfio works correctly, here are some important points at my dmesg:

[    0.210689] vgaarb: setting as boot device: PCI:0000:03:00.0
[    0.210691] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.210695] vgaarb: device added: PCI:0000:05:00.0,decodes=io+mem,owns=none,locks=none
[    0.210696] vgaarb: loaded
[    0.210697] vgaarb: bridge control possible 0000:05:00.0
[    0.210697] vgaarb: bridge control possible 0000:03:00.0
[    0.210846] SCSI subsystem initialized
[    0.210873] libata version 3.00 loaded.
[    0.210889] ACPI: bus type USB registered
[    0.210902] usbcore: registered new interface driver usbfs
[    0.210908] usbcore: registered new interface driver hub
[    0.210919] usbcore: registered new device driver usb
[    0.211004] PCI: Using ACPI for IRQ routing
[    0.212320] PCI: pci_cache_line_size set to 64 bytes
[    0.212378] e820: reserve RAM buffer [mem 0x0009ec00-0x0009ffff]
[    0.212380] e820: reserve RAM buffer [mem 0xbdf9c000-0xbfffffff]
[    0.212381] e820: reserve RAM buffer [mem 0xbdfa3000-0xbfffffff]
[    0.212382] e820: reserve RAM buffer [mem 0xbec81000-0xbfffffff]
[    0.212383] e820: reserve RAM buffer [mem 0xbf000000-0xbfffffff]
[    0.212384] e820: reserve RAM buffer [mem 0x83e000000-0x83fffffff]
[    0.266687] pci 0000:03:00.0: Video device with shadowed ROM
[    0.266730] PCI: CLS 64 bytes, default 64
[    0.266768] Trying to unpack rootfs image as initramfs...
[    2.132383] Freeing initrd memory: 156660K (ffff880024df6000 - ffff88002e6f3000)
[    2.132421] DMAR: No ATSR found
[    2.132438] IOMMU 0 0xfed90000: using Queued invalidation
[    2.132439] IOMMU: Setting RMRR:
[    2.132450] IOMMU: Setting identity map for device 0000:00:14.0 [0xbe51f000 - 0xbe53bfff]
[    2.132468] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbe51f000 - 0xbe53bfff]
[    2.132481] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbe51f000 - 0xbe53bfff]
[    2.132489] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    2.132494] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    2.132571] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787587] pci-stub 0000:05:00.0: claimed by stub
[    2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787597] pci-stub 0000:05:00.1: claimed by stub
[    3.495732] VFIO - User Level meta-driver version: 0.3
[    3.529871] FS-Cache: Loaded
[    3.692679] RPC: Registered named UNIX socket transport module.
[    3.692682] RPC: Registered udp transport module.
[    3.692683] RPC: Registered tcp transport module.
[    3.692684] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.722255] ppdev: user-space parallel port driver
[    3.731469] parport_pc 00:05: reported by Plug and Play ACPI
[    3.731516] parport0: PC-style at 0x378, irq 5 [PCSPP]
[    3.816825] lp0: using parport0 (interrupt-driven).
[    3.821640] wmi: Mapper loaded
[    3.866631] init: avahi-cups-reload main process (565) terminated with status 1
[    3.878995] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20140926/utaddress-258)
[    3.879000] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879003] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879005] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879006] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879007] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879008] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879010] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879010] lpc_ich: Resource conflict(s) found affecting gpio_ich
[    3.962690] device-mapper: multipath: version 1.7.0 loaded
[    3.962999] mei_me 0000:00:16.0: irq 34 for MSI/MSI-X
[    3.977288] AVX version of gcm_enc/dec engaged.
[    3.977290] AES CTR mode by8 optimization enabled
[    4.182963] snd_hda_intel 0000:00:1b.0: irq 35 for MSI/MSI-X
[    4.183050] snd_hda_intel 0000:03:00.1: Handle VGA-switcheroo audio client
[    4.183071] snd_hda_intel 0000:03:00.1: irq 36 for MSI/MSI-X
[    4.202764] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.1/sound/card1/input5
[    4.221910] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[    4.221914] sound hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    4.221915] sound hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    4.221916] sound hdaudioC0D0:    mono: mono_out=0x0
[    4.221917] sound hdaudioC0D0:    dig-out=0x11/0x0
[    4.221918] sound hdaudioC0D0:    inputs:
[    4.221919] sound hdaudioC0D0:      Front Mic=0x19
[    4.221921] sound hdaudioC0D0:      Rear Mic=0x18
[    4.221922] sound hdaudioC0D0:      Line=0x1a
[    4.234307] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
[    4.234879] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
[    4.234930] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[    4.235658] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[    4.236186] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[    4.237295] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    4.258359] init: Failed to obtain startpar-bridge instance: Unknown parameter: INSTANCE
[    4.291101] r8169 0000:06:00.0 eth0: link down
[    4.291120] r8169 0000:06:00.0 eth0: link down
[    4.291144] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.646476] r8169 0000:06:00.0 eth0: link up
[    6.646484] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.469643] init: failsafe main process (690) killed by TERM signal
[    9.511040] audit_printk_skb: 21 callbacks suppressed
[   10.109377] init: plymouth-upstart-bridge main process ended, respawning
[   10.278910] cgroup: systemd-logind (574) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[   10.278913] cgroup: "memory" requires setting use_hierarchy to 1 on the root
[   10.676638] random: nonblocking pool is initialized
[  162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  166.777854] kvm: zapping shadow pages for mmio generation wraparound

i saw some kvm zapping shadow pages here, and these dmesg show up when its on -vga none..

and when i removed -vga none, and i can enter the VM heres the dmesg :

[  162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  166.777854] kvm: zapping shadow pages for mmio generation wraparound
[  262.768505] kvm: zapping shadow pages for mmio generation wraparound
[  269.342192] kvm [2329]: vcpu0 ignored rdmsr: 0x345
[  269.342216] kvm [2329]: vcpu0 ignored wrmsr: 0x680 data 0
[  269.342218] kvm [2329]: vcpu0 ignored wrmsr: 0x681 data 0
[  269.342219] kvm [2329]: vcpu0 ignored wrmsr: 0x682 data 0
[  269.342220] kvm [2329]: vcpu0 ignored wrmsr: 0x683 data 0
[  269.342222] kvm [2329]: vcpu0 ignored wrmsr: 0x684 data 0
[  269.342223] kvm [2329]: vcpu0 ignored wrmsr: 0x685 data 0
[  269.342224] kvm [2329]: vcpu0 ignored wrmsr: 0x686 data 0
[  269.342225] kvm [2329]: vcpu0 ignored wrmsr: 0x687 data 0
[  269.342227] kvm [2329]: vcpu0 ignored wrmsr: 0x688 data 0
[  269.342228] kvm [2329]: vcpu0 ignored wrmsr: 0x689 data 0

Right over here i am stuck in order to try a GPU passthrough.. did i miss anything? or did my dmesg shows any incorrect steps that i've been through? And also, why would the -vga none goes black instead of going to the VMs directly..

So would someone help about this case? i've been stuck for two weeks and nothing came up with any good.. sad

Thanks before !

Offline

#4124 2015-02-09 08:47:34

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.

driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql

here is my qemu commandline:

#!/bin/sh

qemu-system-x86_64 -cpu host,kvm=off -smp 4,sockets=2,cores=2,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-monitor stdio

Offline

#4125 2015-02-09 11:38:27

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bpbastos wrote:
aw wrote:

Pin vCPUs and don't oversubscribe physical CPUs if you expect the guest to handle latency sensitive tasks.  You may also need to move host device interrupts to other CPUs.  Use isolcpus if you really want to have vCPU isolation guarantees.

Thank you aw.

I'm already using isolcpus=2-7 and pinning vcpus for my guest.
The only thing I'm not doing is moving my host device interrups. Do you have any script to do it?

I don't have anything, but you want to manipulate /proc/irq/*/smp_affinity.  You probably want to be careful only to do this for device interrupts (ie. things with IO-APIC or PCI-MSI in the type from /proc/interrupts).  You'll also want to make sure irqbalance doesn't move interrutps back to your isolated CPUs, there's a IRQBALANCE_BANNED_CPUS environment variable that can be used to do that.

EDIT: I doubt an E3 v3 has it, but if /sys/module/kvm_intel/parameters/enable_apicv reports 'Y' then by making sure assigned device interrupts come to a CPU not running the guest, KVM can inject the interrupt into the guest with forcing a VM exit.  I also recall someone was using (I think) the nohz_full= boot option to stop timer ticks on the isolated CPUs.

Thank you aw,

I did everything you suggested but in the end, disabling ASMedia SATA Controller did end my audio lag problems.

Now I have another little problem, I successfully installed openelec and passed through my GT 630, but shutdowns and restarts of the guest results in no video output.

The only solution I've found so far to this problem is to reboot the host.

Offline

#4126 2015-02-09 11:42:37

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.

driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql

here is my qemu commandline:

#!/bin/sh

qemu-system-x86_64 -cpu host,kvm=off -smp 4,sockets=2,cores=2,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-monitor stdio

According to this thread, no solution yet for drivers > 340.52. I'm stuck at that version too.(gtx 560Ti on Windows 7 machine)

Offline

#4127 2015-02-09 11:57:28

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need bios from GT 630 and use it with romfile= option, for working reboot.

Offline

#4128 2015-02-09 12:41:52

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

You need bios from GT 630 and use it with romfile= option, for working reboot.

is it a problem only with gt630?

I'm doing a openelec vm with a gt610...

My vm windows 7 with gtx 560 TI doesn't suffer this problem...

Offline

#4129 2015-02-09 12:42:37

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmm, for me it was must with geforce fermi

Offline

#4130 2015-02-09 12:50:01

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit double post

Last edited by evilsephiroth (2015-02-09 12:50:46)

Offline

#4131 2015-02-09 12:50:44

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

from that link

4 The GeForce GT 620 (OEM) card is a rebranded GeForce GT 520.
5 The GeForce GT 620 card is a rebranded GeForce GT 530.
6 This revision of GeForce GT 630 (DDR3) card is a rebranded GeForce GT 440 (DDR3).

Offline

#4132 2015-02-09 13:39:54

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:
slis wrote:

You need bios from GT 630 and use it with romfile= option, for working reboot.

is it a problem only with gt630?

I'm doing a openelec vm with a gt610...

My vm windows 7 with gtx 560 TI doesn't suffer this problem...

I'm already using the romfile option but seems like my bios is corrupt or something because I could boot and use the card, except shutdown and reboot, with seabios but if try ovmf I get a lot of artifacts.
Do you have the bios for ZOTAC GeForce GT 630 Synergy Edition 2GB? I haven't found any.

Offline

#4133 2015-02-09 13:41:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.

driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql

Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode.  Configure the device as secondary graphics for the VM and do not use the x-vga option.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4134 2015-02-09 13:43:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

According to this thread, no solution yet for drivers > 340.52. I'm stuck at that version too.(gtx 560Ti on Windows 7 machine)

What are you talking about?!  I'm running 347.25 here with GTX750.  KVM and Hyper-V extensions need to be hidden/disabled to avoid Code 43 on GeForce cards.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4135 2015-02-09 13:56:17

flack
Member
Registered: 2014-06-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:
evilsephiroth wrote:
slis wrote:

You need bios from GT 630 and use it with romfile= option, for working reboot.

is it a problem only with gt630?

I'm doing a openelec vm with a gt610...

My vm windows 7 with gtx 560 TI doesn't suffer this problem...

I'm already using the romfile option but seems like my bios is corrupt or something because I could boot and use the card, except shutdown and reboot, with seabios but if try ovmf I get a lot of artifacts.
Do you have the bios for ZOTAC GeForce GT 630 Synergy Edition 2GB? I haven't found any.


Are you try this one? At techpowerup site

Offline

#4136 2015-02-09 14:03:33

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:
bpbastos wrote:
evilsephiroth wrote:

is it a problem only with gt630?

I'm doing a openelec vm with a gt610...

My vm windows 7 with gtx 560 TI doesn't suffer this problem...

I'm already using the romfile option but seems like my bios is corrupt or something because I could boot and use the card, except shutdown and reboot, with seabios but if try ovmf I get a lot of artifacts.
Do you have the bios for ZOTAC GeForce GT 630 Synergy Edition 2GB? I haven't found any.


Are you try this one? At techpowerup site

Thank you @flack.

EDIT:
The bios from techpowerup doesn't support EFI.

[root@home-server rom-parser]# ./rom-parser Zotac.GT630.2048.120511.rom Valid ROM signature found @0h, PCIR offset 188h PCIR: type 0, vendor: 10de, device: 0f00, class: 030000 PCIR: revision 0, vendor revision: 1 Last image

Last edited by bpbastos (2015-02-09 14:11:08)

Offline

#4137 2015-02-09 14:56:58

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

Hi everyone, i've been reading a lot of posts here, and i wanted to try the passthrough by myself, but seemed that i got some error here, and firstly according to the steps of the guide, i have been doing step by steps, and here's my build :

Processor : Xeon E3-1200 (VT-d Enabled in BIOS)
ubuntu display GPU : Radeon HD4870
Passthrough GPU : GeForce GTX 560 Ti Hawx

So steps i've been done so far :
1. Ubuntu Server 12.04, then do-release-upgrade to 14.04
2. Downloaded Kernel 3.18-5 from Linux homepage
3. Downloaded 3.18-5 linux-mainline from nhbs's first post
4. Patched ACS and i915_317 patch from linux-mainline from nbhs's post
5. Qemu 2.2.0, also Seabios standard and no patches
6. lspci | grep NVIDIA :
 

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)

7. edited /etc/default/grub :
 

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c"

    and our /proc/cmdline :
 

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7

8. also blacklisted radeon on /etc/modprobe.d/blacklist.conf
9. dmesg pci-stub :

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[    2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787587] pci-stub 0000:05:00.0: claimed by stub
[    2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787597] pci-stub 0000:05:00.1: claimed by stub

10. i checked lspci -vnn for the graphics cards in the device :

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6220000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at b000 [size=256]
        Expansion ROM at f6200000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
        Subsystem: ASUSTeK Computer Inc. Device [1043:aa30]
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at f6230000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: snd_hda_intel

04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441]
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6120000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at a000 [size=256]
        Expansion ROM at f6100000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: fast devsel, IRQ 11
        Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M]
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M]
        I/O ports at e000 [disabled] [size=128]
        Expansion ROM at f6000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: pci-stub

05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: bus master, fast devsel, latency 0, IRQ 10
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: pci-stub

11. after all went good, i create vfio-pci1.cfg in /etc/vfio-pci1.cfg :

0000:05:00.0
0000:05:00.1

12. Booting VM's Script :

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -vga none -M q35 -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-drive file=/home/sslab719/VMimages/VM.img,id=disk,format=qcow2 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-net user,hostfwd=tcp::10022-:22 -net nic
#-boot menu=on

exit 0

Here at this point, i am always having black QEMU Monitor with "compat_monitor0 console"
Qemu 2.2.0 monitor - type help for more information -- when i'm using -vga none right over here.

If i remove -vga none there, i can boot to the VMs normally, and i can see the lspci went in correctly, but it does seem like it doesn't have the real GPU power... the passthrough went there and it shows up in lspci but i am guessing that theres no GPU clock, shader, and memory power.. I tried to install heaven engine benchmark in linux, and i tried to run it on, and it fails, and here i am guessing that perhaps the problem might be the GPU doesn't really passthrough successfully, but instead its just a name showup on the VMs, and i can still see 00:01.0 VGA compatible Controller: Device 1234:1111, and the others are our passthrough'd GPU 01:00.0..

Here by reading i know that we should add -vga none in order to passthrough it normally, but it seemed that -vga none didn't bring a help for me, and instead its just gave me a weird black qemu monitor which i can type some commands of qemu itself.

i tried -vga std and others, but those didn't worked too..

so i checked my dmesg when turning my VM to see whether vfio works correctly, here are some important points at my dmesg:

[    0.210689] vgaarb: setting as boot device: PCI:0000:03:00.0
[    0.210691] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.210695] vgaarb: device added: PCI:0000:05:00.0,decodes=io+mem,owns=none,locks=none
[    0.210696] vgaarb: loaded
[    0.210697] vgaarb: bridge control possible 0000:05:00.0
[    0.210697] vgaarb: bridge control possible 0000:03:00.0
[    0.210846] SCSI subsystem initialized
[    0.210873] libata version 3.00 loaded.
[    0.210889] ACPI: bus type USB registered
[    0.210902] usbcore: registered new interface driver usbfs
[    0.210908] usbcore: registered new interface driver hub
[    0.210919] usbcore: registered new device driver usb
[    0.211004] PCI: Using ACPI for IRQ routing
[    0.212320] PCI: pci_cache_line_size set to 64 bytes
[    0.212378] e820: reserve RAM buffer [mem 0x0009ec00-0x0009ffff]
[    0.212380] e820: reserve RAM buffer [mem 0xbdf9c000-0xbfffffff]
[    0.212381] e820: reserve RAM buffer [mem 0xbdfa3000-0xbfffffff]
[    0.212382] e820: reserve RAM buffer [mem 0xbec81000-0xbfffffff]
[    0.212383] e820: reserve RAM buffer [mem 0xbf000000-0xbfffffff]
[    0.212384] e820: reserve RAM buffer [mem 0x83e000000-0x83fffffff]
[    0.266687] pci 0000:03:00.0: Video device with shadowed ROM
[    0.266730] PCI: CLS 64 bytes, default 64
[    0.266768] Trying to unpack rootfs image as initramfs...
[    2.132383] Freeing initrd memory: 156660K (ffff880024df6000 - ffff88002e6f3000)
[    2.132421] DMAR: No ATSR found
[    2.132438] IOMMU 0 0xfed90000: using Queued invalidation
[    2.132439] IOMMU: Setting RMRR:
[    2.132450] IOMMU: Setting identity map for device 0000:00:14.0 [0xbe51f000 - 0xbe53bfff]
[    2.132468] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbe51f000 - 0xbe53bfff]
[    2.132481] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbe51f000 - 0xbe53bfff]
[    2.132489] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    2.132494] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    2.132571] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787587] pci-stub 0000:05:00.0: claimed by stub
[    2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787597] pci-stub 0000:05:00.1: claimed by stub
[    3.495732] VFIO - User Level meta-driver version: 0.3
[    3.529871] FS-Cache: Loaded
[    3.692679] RPC: Registered named UNIX socket transport module.
[    3.692682] RPC: Registered udp transport module.
[    3.692683] RPC: Registered tcp transport module.
[    3.692684] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.722255] ppdev: user-space parallel port driver
[    3.731469] parport_pc 00:05: reported by Plug and Play ACPI
[    3.731516] parport0: PC-style at 0x378, irq 5 [PCSPP]
[    3.816825] lp0: using parport0 (interrupt-driven).
[    3.821640] wmi: Mapper loaded
[    3.866631] init: avahi-cups-reload main process (565) terminated with status 1
[    3.878995] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20140926/utaddress-258)
[    3.879000] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879003] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879005] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879006] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879007] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879008] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879010] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879010] lpc_ich: Resource conflict(s) found affecting gpio_ich
[    3.962690] device-mapper: multipath: version 1.7.0 loaded
[    3.962999] mei_me 0000:00:16.0: irq 34 for MSI/MSI-X
[    3.977288] AVX version of gcm_enc/dec engaged.
[    3.977290] AES CTR mode by8 optimization enabled
[    4.182963] snd_hda_intel 0000:00:1b.0: irq 35 for MSI/MSI-X
[    4.183050] snd_hda_intel 0000:03:00.1: Handle VGA-switcheroo audio client
[    4.183071] snd_hda_intel 0000:03:00.1: irq 36 for MSI/MSI-X
[    4.202764] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.1/sound/card1/input5
[    4.221910] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[    4.221914] sound hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    4.221915] sound hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    4.221916] sound hdaudioC0D0:    mono: mono_out=0x0
[    4.221917] sound hdaudioC0D0:    dig-out=0x11/0x0
[    4.221918] sound hdaudioC0D0:    inputs:
[    4.221919] sound hdaudioC0D0:      Front Mic=0x19
[    4.221921] sound hdaudioC0D0:      Rear Mic=0x18
[    4.221922] sound hdaudioC0D0:      Line=0x1a
[    4.234307] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
[    4.234879] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
[    4.234930] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[    4.235658] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[    4.236186] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[    4.237295] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    4.258359] init: Failed to obtain startpar-bridge instance: Unknown parameter: INSTANCE
[    4.291101] r8169 0000:06:00.0 eth0: link down
[    4.291120] r8169 0000:06:00.0 eth0: link down
[    4.291144] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.646476] r8169 0000:06:00.0 eth0: link up
[    6.646484] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.469643] init: failsafe main process (690) killed by TERM signal
[    9.511040] audit_printk_skb: 21 callbacks suppressed
[   10.109377] init: plymouth-upstart-bridge main process ended, respawning
[   10.278910] cgroup: systemd-logind (574) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[   10.278913] cgroup: "memory" requires setting use_hierarchy to 1 on the root
[   10.676638] random: nonblocking pool is initialized
[  162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  166.777854] kvm: zapping shadow pages for mmio generation wraparound

i saw some kvm zapping shadow pages here, and these dmesg show up when its on -vga none..

and when i removed -vga none, and i can enter the VM heres the dmesg :

[  162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  166.777854] kvm: zapping shadow pages for mmio generation wraparound
[  262.768505] kvm: zapping shadow pages for mmio generation wraparound
[  269.342192] kvm [2329]: vcpu0 ignored rdmsr: 0x345
[  269.342216] kvm [2329]: vcpu0 ignored wrmsr: 0x680 data 0
[  269.342218] kvm [2329]: vcpu0 ignored wrmsr: 0x681 data 0
[  269.342219] kvm [2329]: vcpu0 ignored wrmsr: 0x682 data 0
[  269.342220] kvm [2329]: vcpu0 ignored wrmsr: 0x683 data 0
[  269.342222] kvm [2329]: vcpu0 ignored wrmsr: 0x684 data 0
[  269.342223] kvm [2329]: vcpu0 ignored wrmsr: 0x685 data 0
[  269.342224] kvm [2329]: vcpu0 ignored wrmsr: 0x686 data 0
[  269.342225] kvm [2329]: vcpu0 ignored wrmsr: 0x687 data 0
[  269.342227] kvm [2329]: vcpu0 ignored wrmsr: 0x688 data 0
[  269.342228] kvm [2329]: vcpu0 ignored wrmsr: 0x689 data 0

Right over here i am stuck in order to try a GPU passthrough.. did i miss anything? or did my dmesg shows any incorrect steps that i've been through? And also, why would the -vga none goes black instead of going to the VMs directly..

So would someone help about this case? i've been stuck for two weeks and nothing came up with any good.. sad

Thanks before !

Can someone help me on this please...

Offline

#4138 2015-02-09 15:37:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

Hi everyone, i've been reading a lot of posts here, and i wanted to try the passthrough by myself, but seemed that i got some error here, and firstly according to the steps of the guide, i have been doing step by steps, and here's my build :

Processor : Xeon E3-1200 (VT-d Enabled in BIOS)
ubuntu display GPU : Radeon HD4870
Passthrough GPU : GeForce GTX 560 Ti Hawx

Step 1, please learn how to use code boxes rather than quote boxes on the list.

So steps i've been done so far :
1. Ubuntu Server 12.04, then do-release-upgrade to 14.04
2. Downloaded Kernel 3.18-5 from Linux homepage
3. Downloaded 3.18-5 linux-mainline from nhbs's first post
4. Patched ACS and i915_317 patch from linux-mainline from nbhs's post
5. Qemu 2.2.0, also Seabios standard and no patches
6. lspci | grep NVIDIA :
 

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)

7. edited /etc/default/grub :
 

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c"

    and our /proc/cmdline :
 

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7

If you don't have Intel integrated graphics, you don't need the i915 patch.  You're not even using the ACS override patch, so you may be able to use a stock kernel.

8. also blacklisted radeon on /etc/modprobe.d/blacklist.conf

Why?  Radeon is your host device.  How are you going to have host graphics without it?

9. dmesg pci-stub :

[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7 [ 2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.787587] pci-stub 0000:05:00.0: claimed by stub [ 2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.787597] pci-stub 0000:05:00.1: claimed by stub

10. i checked lspci -vnn for the graphics cards in the device :

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device [1043:0284] Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at f6220000 (64-bit, non-prefetchable) [size=64K] I/O ports at b000 [size=256] Expansion ROM at f6200000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30] Subsystem: ASUSTeK Computer Inc. Device [1043:aa30] Flags: bus master, fast devsel, latency 0, IRQ 36 Memory at f6230000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: snd_hda_intel 04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] Subsystem: ASUSTeK Computer Inc. Device [1043:0284] Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at f6120000 (64-bit, non-prefetchable) [size=64K] I/O ports at a000 [size=256] Expansion ROM at f6100000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601] Flags: fast devsel, IRQ 11 Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M] Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M] Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M] I/O ports at e000 [disabled] [size=128] Expansion ROM at f6000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: pci-stub 05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601] Flags: bus master, fast devsel, latency 0, IRQ 10 Memory at f6080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Kernel driver in use: pci-stub

See how much nicer that is in a code box?

11. after all went good, i create vfio-pci1.cfg in /etc/vfio-pci1.cfg :

0000:05:00.0
0000:05:00.1

12. Booting VM's Script :

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -vga none -M q35 -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -drive file=/home/sslab719/VMimages/VM.img,id=disk,format=qcow2 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -net user,hostfwd=tcp::10022-:22 -net nic #-boot menu=on exit 0

Here at this point, i am always having black QEMU Monitor with "compat_monitor0 console"
Qemu 2.2.0 monitor - type help for more information -- when i'm using -vga none right over here.

If i remove -vga none there, i can boot to the VMs normally, and i can see the lspci went in correctly, but it does seem like it doesn't have the real GPU power... the passthrough went there and it shows up in lspci but i am guessing that theres no GPU clock, shader, and memory power.. I tried to install heaven engine benchmark in linux, and i tried to run it on, and it fails, and here i am guessing that perhaps the problem might be the GPU doesn't really passthrough successfully, but instead its just a name showup on the VMs, and i can still see 00:01.0 VGA compatible Controller: Device 1234:1111, and the others are our passthrough'd GPU 01:00.0..

Here by reading i know that we should add -vga none in order to passthrough it normally, but it seemed that -vga none didn't bring a help for me, and instead its just gave me a weird black qemu monitor which i can type some commands of qemu itself.

i tried -vga std and others, but those didn't worked too..

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

so i checked my dmesg when turning my VM to see whether vfio works correctly, here are some important points at my dmesg:

[ 0.210689] vgaarb: setting as boot device: PCI:0000:03:00.0 [ 0.210691] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.210695] vgaarb: device added: PCI:0000:05:00.0,decodes=io+mem,owns=none,locks=none [ 0.210696] vgaarb: loaded [ 0.210697] vgaarb: bridge control possible 0000:05:00.0 [ 0.210697] vgaarb: bridge control possible 0000:03:00.0 [ 0.210846] SCSI subsystem initialized [ 0.210873] libata version 3.00 loaded. [ 0.210889] ACPI: bus type USB registered [ 0.210902] usbcore: registered new interface driver usbfs [ 0.210908] usbcore: registered new interface driver hub [ 0.210919] usbcore: registered new device driver usb [ 0.211004] PCI: Using ACPI for IRQ routing [ 0.212320] PCI: pci_cache_line_size set to 64 bytes [ 0.212378] e820: reserve RAM buffer [mem 0x0009ec00-0x0009ffff] [ 0.212380] e820: reserve RAM buffer [mem 0xbdf9c000-0xbfffffff] [ 0.212381] e820: reserve RAM buffer [mem 0xbdfa3000-0xbfffffff] [ 0.212382] e820: reserve RAM buffer [mem 0xbec81000-0xbfffffff] [ 0.212383] e820: reserve RAM buffer [mem 0xbf000000-0xbfffffff] [ 0.212384] e820: reserve RAM buffer [mem 0x83e000000-0x83fffffff] [ 0.266687] pci 0000:03:00.0: Video device with shadowed ROM [ 0.266730] PCI: CLS 64 bytes, default 64 [ 0.266768] Trying to unpack rootfs image as initramfs... [ 2.132383] Freeing initrd memory: 156660K (ffff880024df6000 - ffff88002e6f3000) [ 2.132421] DMAR: No ATSR found [ 2.132438] IOMMU 0 0xfed90000: using Queued invalidation [ 2.132439] IOMMU: Setting RMRR: [ 2.132450] IOMMU: Setting identity map for device 0000:00:14.0 [0xbe51f000 - 0xbe53bfff] [ 2.132468] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbe51f000 - 0xbe53bfff] [ 2.132481] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbe51f000 - 0xbe53bfff] [ 2.132489] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 2.132494] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 2.132571] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.787587] pci-stub 0000:05:00.0: claimed by stub [ 2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.787597] pci-stub 0000:05:00.1: claimed by stub [ 3.495732] VFIO - User Level meta-driver version: 0.3 [ 3.529871] FS-Cache: Loaded [ 3.692679] RPC: Registered named UNIX socket transport module. [ 3.692682] RPC: Registered udp transport module. [ 3.692683] RPC: Registered tcp transport module. [ 3.692684] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.722255] ppdev: user-space parallel port driver [ 3.731469] parport_pc 00:05: reported by Plug and Play ACPI [ 3.731516] parport0: PC-style at 0x378, irq 5 [PCSPP] [ 3.816825] lp0: using parport0 (interrupt-driven). [ 3.821640] wmi: Mapper loaded [ 3.866631] init: avahi-cups-reload main process (565) terminated with status 1 [ 3.878995] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20140926/utaddress-258) [ 3.879000] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 3.879003] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258) [ 3.879005] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 3.879006] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258) [ 3.879007] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 3.879008] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258) [ 3.879010] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 3.879010] lpc_ich: Resource conflict(s) found affecting gpio_ich [ 3.962690] device-mapper: multipath: version 1.7.0 loaded [ 3.962999] mei_me 0000:00:16.0: irq 34 for MSI/MSI-X [ 3.977288] AVX version of gcm_enc/dec engaged. [ 3.977290] AES CTR mode by8 optimization enabled [ 4.182963] snd_hda_intel 0000:00:1b.0: irq 35 for MSI/MSI-X [ 4.183050] snd_hda_intel 0000:03:00.1: Handle VGA-switcheroo audio client [ 4.183071] snd_hda_intel 0000:03:00.1: irq 36 for MSI/MSI-X [ 4.202764] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.1/sound/card1/input5 [ 4.221910] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line [ 4.221914] sound hdaudioC0D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0) [ 4.221915] sound hdaudioC0D0: hp_outs=1 (0x1b/0x0/0x0/0x0/0x0) [ 4.221916] sound hdaudioC0D0: mono: mono_out=0x0 [ 4.221917] sound hdaudioC0D0: dig-out=0x11/0x0 [ 4.221918] sound hdaudioC0D0: inputs: [ 4.221919] sound hdaudioC0D0: Front Mic=0x19 [ 4.221921] sound hdaudioC0D0: Rear Mic=0x18 [ 4.221922] sound hdaudioC0D0: Line=0x1a [ 4.234307] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6 [ 4.234879] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7 [ 4.234930] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8 [ 4.235658] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9 [ 4.236186] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10 [ 4.237295] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 4.258359] init: Failed to obtain startpar-bridge instance: Unknown parameter: INSTANCE [ 4.291101] r8169 0000:06:00.0 eth0: link down [ 4.291120] r8169 0000:06:00.0 eth0: link down [ 4.291144] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 6.646476] r8169 0000:06:00.0 eth0: link up [ 6.646484] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 9.469643] init: failsafe main process (690) killed by TERM signal [ 9.511040] audit_printk_skb: 21 callbacks suppressed [ 10.109377] init: plymouth-upstart-bridge main process ended, respawning [ 10.278910] cgroup: systemd-logind (574) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future. [ 10.278913] cgroup: "memory" requires setting use_hierarchy to 1 on the root [ 10.676638] random: nonblocking pool is initialized [ 162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) [ 166.777854] kvm: zapping shadow pages for mmio generation wraparound

i saw some kvm zapping shadow pages here, and these dmesg show up when its on -vga none..

and when i removed -vga none, and i can enter the VM heres the dmesg :

[ 162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) [ 166.777854] kvm: zapping shadow pages for mmio generation wraparound [ 262.768505] kvm: zapping shadow pages for mmio generation wraparound [ 269.342192] kvm [2329]: vcpu0 ignored rdmsr: 0x345 [ 269.342216] kvm [2329]: vcpu0 ignored wrmsr: 0x680 data 0 [ 269.342218] kvm [2329]: vcpu0 ignored wrmsr: 0x681 data 0 [ 269.342219] kvm [2329]: vcpu0 ignored wrmsr: 0x682 data 0 [ 269.342220] kvm [2329]: vcpu0 ignored wrmsr: 0x683 data 0 [ 269.342222] kvm [2329]: vcpu0 ignored wrmsr: 0x684 data 0 [ 269.342223] kvm [2329]: vcpu0 ignored wrmsr: 0x685 data 0 [ 269.342224] kvm [2329]: vcpu0 ignored wrmsr: 0x686 data 0 [ 269.342225] kvm [2329]: vcpu0 ignored wrmsr: 0x687 data 0 [ 269.342227] kvm [2329]: vcpu0 ignored wrmsr: 0x688 data 0 [ 269.342228] kvm [2329]: vcpu0 ignored wrmsr: 0x689 data 0

Nope, nothing particularly important in either of those

Right over here i am stuck in order to try a GPU passthrough.. did i miss anything? or did my dmesg shows any incorrect steps that i've been through? And also, why would the -vga none goes black instead of going to the VMs directly..

So would someone help about this case? i've been stuck for two weeks and nothing came up with any good.. sad

Thanks before !

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4139 2015-02-09 16:02:01

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

What is the benefit of adding the "romfile/rom" attribute? Under what conditions should it be provided?

Offline

#4140 2015-02-09 16:16:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
aw wrote:

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

What is the benefit of adding the "romfile/rom" attribute? Under what conditions should it be provided?

The romfile= option makes QEMU read the ROM from a file rather than the device itself.  It should only be used if you have trouble reading the ROM, such as seeing the invalid ROM contents message in dmesg, or if you're replacing the ROM with one that has features the on-device ROM does not, ex. you've hacked on it yourself or you're providing an updated ROM, maybe with UEFI support.  The ROM is read from the device once and cached in QEMU, so there shouldn't be any issues with re-reading the physical ROM at VM reset.  Also, I feel like I shouldn't need to say this, but if you attempt to update any assigned device ROM from inside the VM, you do so at your own risk.

The rombar= option is used to hide the device ROM if you don't want to use it, for instance if you assign a NIC with a PXE ROM but you don't want to boot from the device.

We've seen some instance of people here using rombar=0,romfile=$FILE.  Unless you know exactly what you're doing, this is almost certainly wrong.  This says to hide the device ROM BAR, but still provide a ROM, which gets pushed into the legacy ROMs address space, disassociating it from the device.  Don't do this.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4141 2015-02-09 16:26:10

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The romfile= option makes QEMU read the ROM from a file rather than the device itself.

..a quick distraction..
Is it 100% "accurate"? So if i'll actually flash that ROM on the device, nothing will change for the VM?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4142 2015-02-09 16:32:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

The romfile= option makes QEMU read the ROM from a file rather than the device itself.

..a quick distraction..
Is it 100% "accurate"? So if i'll actually flash that ROM on the device, nothing will change for the VM?

So long as we can read the ROM from the device, that should be true.  QEMU also takes a lazy approach to reading the ROM, not attempting to do so until the guest tries to read it.  Theoretically this makes sure that the device has been reset and is in a proper state to access the ROM.  I rely on the on-card ROM for most of my devices.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4143 2015-02-09 19:50:23

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
aw wrote:

Nothing has changed here, the user always needs to be able to lock enough pages for the VM.  You're not going to get very far with a locked memory limit of 40MB.  If you use libvirt (and do not hide the vfio device in <qemu:arg>!) this happens automatically.  If you run qemu by hand, the easiest solution is to run as root or use sudo.

Ah I remember that. I think I adjusted this via

cat /etc/security/limits.conf | grep @users @users soft memlock 4301000 @users hard memlock 4301000

This was fine to run qemu as non root until now..

OK I found and fixed the issue...

cat /etc/security/limits.conf @users soft memlock 4301000 @users hard memlock 4301000 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html - rtprio 0 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_16.html topic_17.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html - nice 0 @audio - rtprio 65 @audio - nice -10 @audio - memlock 4301000

The memlock for @audio was set to 40MB. I haven't touched this file for month and obviously the last value which was set was taken into account. Still wondering why this broke suddenly...
Thanks for the hints!

Offline

#4144 2015-02-09 20:02:06

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've gotten it to work, but for some reason Visual studio wont install. Have anyone had any success installing it? It's some error about wpfgfx.dll, which makes me afraid it has something to do with the graphics card.

Offline

#4145 2015-02-09 20:06:51

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

I've gotten it to work, but for some reason Visual studio wont install. Have anyone had any success installing it? It's some error about wpfgfx.dll, which makes me afraid it has something to do with the graphics card.

Works for me.

Offline

#4146 2015-02-09 22:18:15

valnare
Member
Registered: 2015-02-09
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to get my AMD Radeon 290X to work with VGA passtrough. TRIM also works on the virtual disk controller for the SSD after changing the controller type to IDE and AHCI. I wanted to pass a real disk to the VM, so I don't loose the ability to natively boot into Windows.

For playback I am using the HDMI output of my GPU. Unfortunately the audio there stutters while playing sound. The same happens when I try to use the virtualized AC97 output. Did someone else also had problems with the HDMI sound output? Everything else is working perfectly as far as I can tell. Thank you for the great Guide!

Here is the script I am using to start the VM:

export QEMU_PA_SAMPLES=1024 export QEMU_PA_SERVER="127.0.0.1" export QEMU_AUDIO_DRV=pa export VMPATH="/home/felix" export DISK="/dev/disk/by-id/ata-SanDisk_SDSSDHII960G_144340400518" qemu-system-x86_64 -enable-kvm -m 8192 -cpu Haswell -smp 4,sockets=1,cores=4,threads=1 -vga none \ -mem-path /dev/hugepages -monitor stdio \ -device vfio-pci,host=01:00.0,x-vga=on \ -device vfio-pci,host=01:00.1 \ -net nic -net bridge,br=br0 \ -usb -usbdevice host:045e:0719 -usbdevice host:046d:c07d -usbdevice host:04d9:2013 -usbdevice host:1852:50d0 \ -soundhw ac97 \ -drive id=disk,file=$DISK,if=none,discard=on,detect-zeroes=on -device ahci,id=ahci -device ide-drive,drive=disk,bus=ahci.0 

Offline

#4147 2015-02-10 03:00:18

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks a lot.

aw wrote:
erganzi wrote:

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.

driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql

Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode.  Configure the device as secondary graphics for the VM and do not use the x-vga option.

when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.
like "This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system. (Code 12)"

but if I disable the first device, and reboot it reported code 12 too.

here is my qemu commandline:

#!/bin/sh

qemu-system-x86_64 -cpu host,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

Offline

#4148 2015-02-10 03:01:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

thanks a lot.

aw wrote:
erganzi wrote:

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.

driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql

Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode.  Configure the device as secondary graphics for the VM and do not use the x-vga option.

when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.

Last edited by aw (2015-02-10 03:12:46)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4149 2015-02-10 05:22:37

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi and thank you for your reply sir aw !

If you don't have Intel integrated graphics, you don't need the i915 patch.  You're not even using the ACS override patch, so you may be able to use a stock kernel.

yes, i checked on my lspci and i think that my graphics card doesn't have IGP on it, so i changed the boot parameter and in cmdline changes into this :

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7

next, i also cleared the blacklisted radeon on /etc/modprobe.d/blacklist.conf, and i added nvidia, for passthrough instead.

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.

i tried adding -nographic and putting back -vga none on the script, but the session is stuck there, and sometimes it made the whole machine stuck until i had to hard-reboot it.  i also tried adding -nographic without -vga none, and gives the same result too.. so i checked using top while having other stuck'd session, and qemu shows up that they own 99.8% of CPU usage.. so i tried to kill the process, and that stucked session went back, so that means actually that -nographic option still boots up the VM, but i can't see any output.. any idea on this?

also as for the 440fx version, i configured my script, and it shows no good, still only black monitor, and if i add -vga none, same black_compat0 monitor still show up like before.. here is my script :

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -vga none -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -device vfio-pci,host=05:00.0,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \ -device vfio-pci,host=05:00.1 \ -net user,hostfwd=tcp::10022-:22 -net nic #-boot menu=on exit 0

I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

yes i am downloading ones from the techpowerup.. i am not sure whether its going to match perfectly, since 560 ti has a lot of variants, im using MSI GTX560 Ti twin frozr 2 version, and i am searching for better accurate rom for the graphics card.
so is that possible that because of the ROM that it brokes the whole process? i dont know why but when the nographic and -vga none added it will crash the pc instead..

Additional :
i tried to keep using q35 version of my script, and i tried to download some benchmark, so i downloaded GpuTest, and once i open it, the whole PC crashes and i had to hard reboot it, is it because of CPU usage that made the PC overloads until it had crash, or is it a stuck because of GPU operation fails?

Last edited by hotfunction (2015-02-10 15:41:13)

Offline

#4150 2015-02-10 07:13:43

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
erganzi wrote:

thanks a lot.

aw wrote:

Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode.  Configure the device as secondary graphics for the VM and do not use the x-vga option.

when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.


when I changed to i440fx, the driver report code 43.

my qemu commandline:

qemu-system-x86_64 -cpu SandyBridge,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device vfio-pci,host=04:00.0,addr=05.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

Offline

#4151 2015-02-10 11:21:11

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

valnare wrote:

Unfortunately the audio there stutters while playing sound.

In my case all sound problems were latency related. If the problem gets worse when you pass more CPU cores and better when you pass less, then it's propably the same. Other phenomenons: tools like LatencyMon let the windows guest freeze really soon.
What helps: cpu pinning (for each guest and the host!), in case of HDMI: enable MSI in registry
Unconfirmed: hugepages

Offline

#4152 2015-02-10 12:54:26

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello smile

I've never tried the CPU pinning so I will make a newbie question... is something that I can only make with libvirt? My windows 8.1 guest is working perfectly and I'm running it from the command line directly... is therre any way to improve (even more) the performance? Or I need libvirt and be able to convert form my script to the XML format?

Thanks a in advance smile

Offline

#4153 2015-02-10 15:13:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:
erganzi wrote:

thanks a lot.



when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.


when I changed to i440fx, the driver report code 43.

my qemu commandline:

qemu-system-x86_64 -cpu SandyBridge,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device vfio-pci,host=04:00.0,addr=05.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

kvm=off is also not necessary with supported nvidia cards, but I don't see anything else off.  What kernel/qemu are you using?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4154 2015-02-10 16:34:58

gnoma_86
Member
Registered: 2015-02-10
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I'm new in KVM and I'm trying to do a VGA pass through.

I've done the vfio grouping and I attached my HD5770 to the virtual machine. Installation goes well. After install I see my video card in the device manager and when I install the driver they are both - VGA and HDMI visible with exact model. However it says that I have to reboot in order to get the device online. Tested and no picture coming from the physical VGA before reboot. After I install the driver and reboot I got BSOD on the spice during boot. I can hear during boot that the FAN of the HD5770 rotates faster, but when I connect a monitor it instantly goes in power saving. Tried with AMD driver, tried also with the driver from ASUS, I got the same result.

I removed the virtual video card and spice. What I noticed is that even before OS startup I cannot see the boot at all, the display goes to power saving instantly. During power on the VM it doesn't even blink or go online for a second, it stays constantly off.

From what I saw in youtube, I should be able to see my VM booting on the VGA passed through to the VM.

Did anybody face the same issue?
Please help.

Thank you.

Offline

#4155 2015-02-10 16:39:47

valnare
Member
Registered: 2015-02-09
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:

In my case all sound problems were latency related. If the problem gets worse when you pass more CPU cores and better when you pass less, then it's propably the same. Other phenomenons: tools like LatencyMon let the windows guest freeze really soon.
What helps: cpu pinning (for each guest and the host!), in case of HDMI: enable MSI in registry
Unconfirmed: hugepages

I tried playing audio with only a single core pinned on the host and the guest. Still the same result. The stuttering also does not change with more load on the CPU.
Latencymon worked while only using one cpu core, when I tried running it with more than one it frooze after 2 seconds. MSI was enabled, didn't change anything for the stuttering. I'll experiment a bit more with cpu pinning though.
Enabling or disabling hugepages didn't do anything for me.

Thank you for your suggestions smile

Offline

#4156 2015-02-10 22:32:44

gessert
Member
Registered: 2015-02-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Goog evening,

I've been getting strange results and need guidance.

Graphicscards:
AMD Radeon r5 230
GeForce GTX 970
GeForce GTX 970

I'm trying to get two different configurations to work:
Host: AMD Radeon r5 230
Guest1: Nvidia GeForce GTX 970
Guest2: Nvidia GeForce GTX 970

Host: AMD Radeon r5 230
Guest1: Nvidia GeForce GTX 970
Guest1: Nvidia GeForce GTX 970

Basically I want to use the Nvidia cards either for two different guests or as an SLI configuration in a single guest.

I have successfully passed the second geforce card to a guest, but the first one will not work. The first card happens to be the card used by the host's uefi, grub and terminal booting messages. I assume the problem stems from here. I can not change the cards' order, nor set a primary grapicsadapter in the host's uefi. The first geforce card's monitor stays active after host boot and shows a single underscore. When I run the guest with the first card configured, the monitor goes blank and I get a warning in the qemu-window:

-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow

Info:
Distribution: Debian Wheezy (running on wheezy-backports)
qemu/kvm 2.1
Seabios
3.16.0-0.bpo.4-amd64

lspci | grep NVIDIA

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c2] (rev a1) 03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1) 04:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c2] (rev a1) 04:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

GRUB_CMDLINE_LINUX_DEFAULT
in /etc/default/grub.cfg

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"

run via "update-grub"

/etc/modules

pci_stub vfio vfio_iommu_type1 vfio_pci kvm kvm_intel

/etc/initramfs-tools/modules

pci_stub ids=10de:13c2,10de:0fbb

run via "update-initramfs -u"

dmesg | pci-stub

[ 1.449636] pci-stub: add 10DE:13C2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.449654] pci-stub 0000:03:00.0: claimed by stub [ 1.449662] pci-stub 0000:04:00.0: claimed by stub [ 1.449664] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.449675] pci-stub 0000:03:00.1: claimed by stub [ 1.449682] pci-stub 0000:04:00.1: claimed by stub

Working:

/etc/vfio-pci1.cfg

0000:03:00.0 0000:03:00.1

/home/me/vm1

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 sudo qemu-system-x86_64 -enable-kvm -M pc -m 8192 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/seabios/bios.bin -vga none \ -device vfio-pci,host=03:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1 \ -usb -usbdevice host:04d9:a086 -usbdevice host:046a:0023 \ -drive file=/home/me/windows10.img,format=raw,media=disk \ -drive file=/home/me/win10.iso,media=cdrom \ -soundhw ac97 \ -boot menu=on exit 0

not working:

/etc/vfio-pci2.cfg

0000:04:00.0 0000:04:00.1

/home/me/vm2

#!/bin/bash configfile=/etc/vfio-pci2.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 sudo qemu-system-x86_64 -enable-kvm -M pc -m 8192 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/seabios/bios.bin -vga none \ -device vfio-pci,host=04:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=04:00.1 \ -usb -usbdevice host:04d9:a086 -usbdevice host:046a:0023 \ -drive file=/home/me/windows10.img,format=raw,media=disk \ -drive file=/home/me/win10.iso,media=cdrom \ -soundhw ac97 \ -boot menu=on exit 0

The "broken" guest's boot attempt creates additional messages:

dmesg | grep 0000:04

vfio-pci 0000:04:00.0: Invalid ROM contents

I dumped the graphicscard's rom and added a romfile parameter to the config, but I guess it's just a follow up problem.

-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on,romfile=/home/me/vbios.rom \

This does get rid of the Invalid ROM contents error, but still no output on the screen.


I have already dumped around 70 hours into this project and I'm afraid my linux-knowledge is rather limited. I'm part of the raspberry-pi generation of linux users. So help is vastly appreciated.

I would offer beer, if you were in throwing range.

Offline

#4157 2015-02-10 22:59:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gessert wrote:
-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow

As Duelist figured out, this means a host driver is attached to the device.  A common problem when using the boot graphics device.  Look in /proc/iomem, figure out what it is, and disable it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4158 2015-02-11 01:18:46

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
erganzi wrote:
aw wrote:

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.


when I changed to i440fx, the driver report code 43.

my qemu commandline:

qemu-system-x86_64 -cpu SandyBridge,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device vfio-pci,host=04:00.0,addr=05.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

kvm=off is also not necessary with supported nvidia cards, but I don't see anything else off.  What kernel/qemu are you using?

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Offline

#4159 2015-02-11 01:26:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4160 2015-02-11 02:17:26

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
erganzi wrote:

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.


OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

Offline

#4161 2015-02-11 02:26:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:
erganzi wrote:

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.


OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4162 2015-02-11 02:31:59

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
erganzi wrote:
aw wrote:

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.


OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

Thanks for those work.  I will try what you suggest and give a feedback.

Offline

#4163 2015-02-11 02:51:36

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi..

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

I added -nographics option and -vga none, mixed together with the script changed into 440fx, but none came up out of the window, the session is there and running, but the qemu operation is not cancelable with ctrl + c and i dont see any window output. i went into another session and checked it with "top" command, and the qemu is still there...

also, i tried to remove my nographics and vga none, and just normally run that VM without those options, but all i get is just a black monitor of qemu, not even booting into seabios and it stucks there forever...

here is my script for 440fx:

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -enable-kvm -M pc -m 2048 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/seabios/bios.bin \ -device vfio-pci,host=05:00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \ -device vfio-pci,host=05:00.1 \ -drive file=/home/sslab719/VMimages/VM.img,format=qcow2,media=disk \ #-drive file=/home/me/win10.iso,media=cdrom \ -boot menu=on exit 0

any ideas..?

Last edited by hotfunction (2015-02-11 03:45:21)

Offline

#4164 2015-02-11 04:18:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

any ideas..?

I already suggested that the ROM might be the problem and you don't seem to have much confidence that it's the correct one.  Why haven't you tried anything further on that path?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4165 2015-02-11 05:47:11

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:
erganzi wrote:

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

Thanks for those work.  I will try what you suggest and give a feedback.

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

kernel: 3.12.5

04:00.0 VGA compatible controller: nVidia Corporation Device 11bf (rev a1) (prog-if 00 [VGA controller])
    Subsystem: nVidia Corporation Device 100d
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 11
    Region 0: Memory at c9000000 (32-bit, non-prefetchable) [disabled] [size=16M]
    Region 1: Memory at b8000000 (64-bit, prefetchable) [disabled] [size=128M]
    Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M]
    Region 5: I/O ports at 9000 [disabled] [size=128]
    Expansion ROM at ca000000 [disabled] [size=512K]
    Capabilities: [60] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [78] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #8, Speed unknown, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
            ClockPM+ Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range AB, TimeoutDis+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
        LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB
    Capabilities: [b4] Vendor Specific Information <?>
    Capabilities: [100] Virtual Channel <?>
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [420] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Capabilities: [600] Vendor Specific Information <?>
    Capabilities: [900] #19
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau, nvidiafb

kernel: 3.18.6

04:00.0 VGA compatible controller: nVidia Corporation Device 11bf (rev a1) (prog-if 00 [VGA controller])
    Subsystem: nVidia Corporation Device 100d
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 27
    Region 0: Memory at c9000000 (32-bit, non-prefetchable) [disabled] [size=16M]
    Region 1: Memory at b8000000 (64-bit, prefetchable) [disabled] [size=128M]
    Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M]
    Region 5: I/O ports at 9000 [disabled] [size=128]
    Expansion ROM at ca000000 [disabled] [size=512K]
    Capabilities: [60] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [78] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #8, Speed unknown, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
            ClockPM+ Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range AB, TimeoutDis+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
        LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB
    Capabilities: [b4] Vendor Specific Information <?>
    Capabilities: [100] Virtual Channel <?>
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [420] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Capabilities: [600] Vendor Specific Information <?>
    Capabilities: [900] #19
    Kernel driver in use: vfio-pci

Offline

#4166 2015-02-11 06:07:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
erganzi wrote:
aw wrote:

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

Thanks for those work.  I will try what you suggest and give a feedback.

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4167 2015-02-11 06:48:22

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
erganzi wrote:
erganzi wrote:

Thanks for those work.  I will try what you suggest and give a feedback.

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.


OK, thanks for reply. I will use the version 3.18.6.

Offline

#4168 2015-02-11 08:03:34

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

valnare wrote:
mauorrizze wrote:

In my case all sound problems were latency related. If the problem gets worse when you pass more CPU cores and better when you pass less, then it's propably the same. Other phenomenons: tools like LatencyMon let the windows guest freeze really soon.
What helps: cpu pinning (for each guest and the host!), in case of HDMI: enable MSI in registry
Unconfirmed: hugepages

I tried playing audio with only a single core pinned on the host and the guest. Still the same result. The stuttering also does not change with more load on the CPU.
Latencymon worked while only using one cpu core, when I tried running it with more than one it frooze after 2 seconds. MSI was enabled, didn't change anything for the stuttering. I'll experiment a bit more with cpu pinning though.
Enabling or disabling hugepages didn't do anything for me.

Thank you for your suggestions smile

Have same problems with ASUS GTX 670.  With GA GTX770 sound seems ok but that card spins FAN at maximum speed when VM was turned off, so she does not fit me. With 670 i turning off when turning on vm, till i not achieve stable sound, but as long as i do so it seems that getting stable sound requires even more reboots. I am very interested in a way you solve this problem.

Offline

#4169 2015-02-11 12:47:27

mmm
Member
Registered: 2014-11-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I'm having trouble passing through the third and fourth graphics card to VMs while passing through the first and second one works without any issues.
The VMs just dont give a signal on the GPU's outputs and they dont seem to boot up since no network services are starting from inside the VM.
Also neither qemu itself nor dmesg seem to report any errors which leaves me wondering how I can even get information about what the problem is.
All cards are GTX 780Ti, the motherboard is a dual-socket board with both sockets equipped and all cards are apparently not in an iommu group with any other devices.

Does anyone have suggestions how to approach this?

Offline

#4170 2015-02-11 17:52:03

gessert
Member
Registered: 2015-02-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
gessert wrote:
-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow

As Duelist figured out, this means a host driver is attached to the device.  A common problem when using the boot graphics device.  Look in /proc/iomem, figure out what it is, and disable it.

excerpt from cat /proc/iomem

90000000-fbffbfff : PCI Bus 0000:00 a0000000-b1ffffff : PCI Bus 0000:04 a0000000-afffffff : 0000:04:00.0 b0000000-b1ffffff : 0000:04:00.0 b1000000-b12fffff : BOOTFB

From what I could gather with a google search: BOOTFB is the framebuffer that sends terminal messages during boot. Could I just "disable" it? How do I proceed? Please advise.

complete cat /proc/iomem

00000000-00000fff : reserved 00001000-0009ffff : System RAM 000a0000-000bffff : PCI Bus 0000:00 000c0000-000cebff : Video ROM 000f0000-000fffff : System ROM 00100000-790c7fff : System RAM 01000000-0154d5b2 : Kernel code 0154d5b3-018ea13f : Kernel data 01a1e000-01aeffff : Kernel bss 790c8000-79c36fff : reserved 79c37000-79ef3fff : System RAM 79ef4000-7aa7cfff : ACPI Non-volatile Storage 7aa7d000-7afbefff : reserved 7afbf000-7afe5fff : System RAM 7afe6000-7afe6fff : reserved 7afe7000-7afe9fff : System RAM 7afea000-7afeafff : reserved 7afeb000-7afebfff : System RAM 7afec000-7b071fff : reserved 7b072000-7b289fff : System RAM 7b28a000-7bff8fff : reserved 7bff9000-7bffffff : System RAM 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] 80000000-8fffffff : reserved 90000000-fbffbfff : PCI Bus 0000:00 a0000000-b1ffffff : PCI Bus 0000:04 a0000000-afffffff : 0000:04:00.0 b0000000-b1ffffff : 0000:04:00.0 b1000000-b12fffff : BOOTFB c0000000-d1ffffff : PCI Bus 0000:03 c0000000-cfffffff : 0000:03:00.0 d0000000-d1ffffff : 0000:03:00.0 e0000000-efffffff : PCI Bus 0000:02 e0000000-efffffff : 0000:02:00.0 f8000000-f90fffff : PCI Bus 0000:04 f8000000-f8ffffff : 0000:04:00.0 f9000000-f907ffff : 0000:04:00.0 f9080000-f9083fff : 0000:04:00.1 f9100000-f911ffff : 0000:00:19.0 f9100000-f911ffff : e1000e f9120000-f912ffff : 0000:00:14.0 f9120000-f912ffff : xhci_hcd f9130000-f9133fff : 0000:00:1b.0 f9130000-f9133fff : ICH HD audio f9135000-f91350ff : 0000:00:1f.3 f9136000-f91367ff : 0000:00:1f.2 f9136000-f91367ff : ahci f9137000-f91373ff : 0000:00:1d.0 f9137000-f91373ff : ehci_hcd f9138000-f91383ff : 0000:00:1a.0 f9138000-f91383ff : ehci_hcd f9139000-f9139fff : 0000:00:19.0 f9139000-f9139fff : e1000e f913c000-f913c00f : 0000:00:16.0 f913c000-f913c00f : mei_me f913d000-f913d7ff : 0000:00:11.4 f913d000-f913d7ff : ahci f913e000-f913efff : 0000:00:05.4 fa000000-fb0fffff : PCI Bus 0000:03 fa000000-faffffff : 0000:03:00.0 fb000000-fb07ffff : 0000:03:00.0 fb080000-fb083fff : 0000:03:00.1 fb200000-fb2fffff : PCI Bus 0000:02 fb200000-fb21ffff : 0000:02:00.0 fb220000-fb23ffff : 0000:02:00.0 fb240000-fb243fff : 0000:02:00.1 fb240000-fb243fff : ICH HD audio fbffc000-fbffcfff : dmar1 fbffd000-fbffdfff : dmar0 fec00000-fecfffff : PNP0003:00 fec00000-fec003ff : IOAPIC 0 fec01000-fec013ff : IOAPIC 1 fed00000-fed003ff : HPET 0 fed00000-fed003ff : PNP0103:00 fed12000-fed1200f : pnp 00:01 fed12010-fed1201f : pnp 00:01 fed1b000-fed1bfff : pnp 00:01 fed1c000-fed1ffff : reserved fed1f410-fed1f414 : iTCO_wdt fed45000-fed8bfff : pnp 00:01 fee00000-feefffff : pnp 00:01 fee00000-fee00fff : Local APIC ff000000-ffffffff : reserved ff000000-ffffffff : pnp 00:01 100000000-47fffffff : System RAM

Last edited by gessert (2015-02-11 17:53:00)

Offline

#4171 2015-02-11 20:09:56

evilsephiroth
Member
Registered: 2014-09-16
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
evilsephiroth wrote:

According to this thread, no solution yet for drivers > 340.52. I'm stuck at that version too.(gtx 560Ti on Windows 7 machine)

What are you talking about?!  I'm running 347.25 here with GTX750.  KVM and Hyper-V extensions need to be hidden/disabled to avoid Code 43 on GeForce cards.

Exactly...
It s a steam in home streaming vm so performance would decrease also...

Offline

#4172 2015-02-11 20:21:23

gessert
Member
Registered: 2015-02-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have changed the grub command line and added:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 nofb" GRUB_GFXPAYLOAD_LINUX=text

run via update-grub

cat /proc/iomem now looks clean.

During reboot I noticed the terminal booting messages will now stop earlier. While loading the ramdisk. It keeps displaying it's last screen.

The guest does not generate the "-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow" error anymore.

"dmesg | grep 0000:04" still shows "vfio-pci 0000:04:00.0: Invalid ROM contents". So I added the romfile again. This gets rid of this problem.

When I run the vm now, the screen turns off. I still do not have graphics output. Please advise.

Offline

#4173 2015-02-12 02:00:36

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When starting

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device ahci,id=ahci0 \ -drive file=/dev/sda,format=raw,if=none,id=sata0-0-0,media=disk \ -drive file=/home/len/win.iso,if=none,id=sata0-0-1,media=cdrom -device ide-drive,bus=ahci0.0,unit=0,drive=drive-sata0-0-0,id=drive-sata0-0-0 \ -device ide-cd,bus=ahci0.1,unit=0,drive=drive-sata0-0-1,id=drive-sata0-0-1 \

getting this error

[ 40.960654] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 40.960757] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258 [ 40.960763] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 43.955151] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt [ 43.955202] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

I'm using kernel provided in this thread (uname checked),

[root@holo ~]# cat /etc/modules-load.d/kvm_virt.conf pci_stub vfio vfio_iommu_type1 vfio_pci kvm kvm_intel
[root@holo ~]# l /sys/bus/pci/drivers/vfio-pci/ total 0 drwxr-xr-x 2 root root 0 Feb 12 03:47 . drwxr-xr-x 21 root root 0 Feb 12 03:40 .. lrwxrwxrwx 1 root root 0 Feb 12 03:46 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 Feb 12 03:46 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 --w------- 1 root root 4096 Feb 12 03:46 bind lrwxrwxrwx 1 root root 0 Feb 12 03:46 module -> ../../../../module/vfio_pci --w------- 1 root root 4096 Feb 12 03:40 new_id --w------- 1 root root 4096 Feb 12 03:46 remove_id --w------- 1 root root 4096 Feb 12 03:40 uevent --w------- 1 root root 4096 Feb 12 03:46 unbind
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb i915.enable_hd_vgaarb=1"

Any clues guys?

Last edited by Len (2015-02-12 02:48:33)

Offline

#4174 2015-02-12 02:53:09

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

Last edited by hotfunction (2015-02-12 05:33:08)

Offline

#4175 2015-02-12 08:22:59

valnare
Member
Registered: 2015-02-09
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kainet wrote:

Have same problems with ASUS GTX 670.  With GA GTX770 sound seems ok but that card spins FAN at maximum speed when VM was turned off, so she does not fit me. With 670 i turning off when turning on vm, till i not achieve stable sound, but as long as i do so it seems that getting stable sound requires even more reboots. I am very interested in a way you solve this problem.

I still had an USB sound card lying around that I passed through, where I do not get any suttering. This is only a workaround for me and it would be nice to use the HDMI ouput, but I have no idea how to solve the stuttering.

Offline

#4176 2015-02-12 09:03:14

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:
erganzi wrote:

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.


OK, thanks for reply. I will use the version 3.18.6.


Hi, aw:
I have another question that usb device(mouse and keyboard) cannot passthrough

I direct run: 
qemu-system-x86_64 -usb -usbdevice host:17ef:6018

the error is :
qemu: could not add USB device 'host:17ef:6018'

there is no error in /var/log/message


the secondary question is :
I can successful passthrough AMD FirePro R5000, and it worked well. but if I passthrough the r5000 onece, the integrated graphics on host had no responce and the display of screen is messed.

but the Nvidia GRID K2 in same host is OK.

every test the host just plug one card.

Last edited by erganzi (2015-02-12 09:12:25)

Offline

#4177 2015-02-12 20:41:02

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that Q35 is working just fine with ovmf-svn compiled 4 hours ago . Gaming at 3840 x 2160 resolution is excellent inside the VM .

Offline

#4178 2015-02-13 03:08:40

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there anyone who can help me (preferred life-session on IRC) with configuration this on:
Motherboard: GA-Z97X-Gaming 5
CPU: i5-4690K
Host: Intel HD4600 (Arch)
Quest: Gigabyte GTX 970 (Windows)

I spend on this too many hours, still can't figure out why it's not working (old post), still new, can't handle all things on arch. After getting this working, I'll create post for others...

Last edited by Len (2015-02-13 03:09:19)

Offline

#4179 2015-02-13 03:31:27

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
erganzi wrote:
aw wrote:

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.


OK, thanks for reply. I will use the version 3.18.6.


Hi, aw:
I have another question that usb device(mouse and keyboard) cannot passthrough

I direct run: 
qemu-system-x86_64 -usb -usbdevice host:17ef:6018

the error is :
qemu: could not add USB device 'host:17ef:6018'

there is no error in /var/log/message


the secondary question is :
I can successful passthrough AMD FirePro R5000, and it worked well. but if I passthrough the r5000 onece, the integrated graphics on host had no responce and the display of screen is messed.

but the Nvidia GRID K2 in same host is OK.

every test the host just plug one card.

sorry, the usb device can not passthrough is my fault.
the libusb library is too old, when I recompiled the new one.
it can passthrough well.

Offline

#4180 2015-02-13 04:19:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

Is there anyone who can help me (preferred life-session on IRC) with configuration this on:
Motherboard: GA-Z97X-Gaming 5
CPU: i5-4690K
Host: Intel HD4600 (Arch)
Quest: Gigabyte GTX 970 (Windows)

I spend on this too many hours, still can't figure out why it's not working (old post), still new, can't handle all things on arch. After getting this working, I'll create post for others...

You're missing the i915 vga arbiter patch, if you're using arch and the kernel i provided make sure you're actually booting it and not arch's one, alternatively you could use OVMF and skip that problem completely

Last edited by nbhs (2015-02-13 04:21:47)

Offline

#4181 2015-02-13 09:12:49

SchaduwBlink
Member
Registered: 2007-01-07
Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

It seems that Q35 is working just fine with ovmf-svn compiled 4 hours ago . Gaming at 3840 x 2160 resolution is excellent inside the VM .

Where did you get the latest ovmf? It's no longer on Jenkins repo for some reason. Or did you compile it by yourself and if so, could you direct me on how to do that and make it split vars?

Thanks.

Offline

#4182 2015-02-13 10:08:17

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SchaduwBlink wrote:

Where did you get the latest ovmf? It's no longer on Jenkins repo for some reason. Or did you compile it by yourself and if so, could you direct me on how to do that and make it split vars?

Thanks.

There is a package in AUR if you're using Arch :

yaourt -S ovmf-svn

You must have "yaourt" to use the above command .

This will pull the latest code , compile it and then ask you for a confirmation to install it .

It will be installed into :

/usr/share/ovmf/x64/ovmf_code_x64.bin

This is the main binary

AND :

/usr/share/ovmf/x64/ovmf_vars_x64.bin

This is the vars file .

Good luck !

Last edited by Denso (2015-02-13 10:28:05)

Offline

#4183 2015-02-13 12:39:09

SchaduwBlink
Member
Registered: 2007-01-07
Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
SchaduwBlink wrote:

Where did you get the latest ovmf? It's no longer on Jenkins repo for some reason. Or did you compile it by yourself and if so, could you direct me on how to do that and make it split vars?

Thanks.

There is a package in AUR if you're using Arch :

yaourt -S ovmf-svn

You must have "yaourt" to use the above command .

This will pull the latest code , compile it and then ask you for a confirmation to install it .

It will be installed into :

/usr/share/ovmf/x64/ovmf_code_x64.bin

This is the main binary

AND :

/usr/share/ovmf/x64/ovmf_vars_x64.bin

This is the vars file .

Good luck !

Cheers for that. Forgot to look there. I've always been pulling the pre-compiled one from Jenkins RPM Repo and extracting it.

Offline

#4184 2015-02-13 14:30:41

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SchaduwBlink wrote:

Cheers for that. Forgot to look there. I've always been pulling the pre-compiled one from Jenkins RPM Repo and extracting it.

AUR is the main reason why I'm using Arch .

Best of luck to you on your virtualization journey !

Offline

#4185 2015-02-14 03:02:45

Tim_J
Member
Registered: 2015-02-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gessert wrote:

When I run the vm now, the screen turns off. I still do not have graphics output. Please advise.

It seems that you have the exact same problem that I do, which I posted a few pages ago. My system behaves the same as yours. In my case, I have one GT 210 and two GTX 580s. When the VM is launched and the screen turns off, it uses maximum CPU on a single core but never starts.

I've tried all sorts of things: unplugging the primary monitor at boot, putting GRUB in text-only mode, changing the boot TTY, quiet mode, and all manner of boot options to both GRUB and the kernel. So far, nothing has helped. It seems that the only recourse for us is to reassign the primary adapter (unsupported in both of our cases) or physically move cards around. sad Since this requires a system drain and cutting new tubes in my case, I'm postponing this for a while to see if anybody can come up with any ideas.

Offline

#4186 2015-02-14 03:55:28

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Len wrote:

Is there anyone who can help me (preferred life-session on IRC) with configuration this on:
Motherboard: GA-Z97X-Gaming 5
CPU: i5-4690K
Host: Intel HD4600 (Arch)
Quest: Gigabyte GTX 970 (Windows)

I spend on this too many hours, still can't figure out why it's not working (old post), still new, can't handle all things on arch. After getting this working, I'll create post for others...

You're missing the i915 vga arbiter patch, if you're using arch and the kernel i provided make sure you're actually booting it and not arch's one, alternatively you could use OVMF and skip that problem completely

Thank you for OVMF hint, it's now displaying to my second graphic card!
But I still can't get it to work on normal user,

vfio: error opening /dev/vfio/1: Permission denied.

I'm in libvirt and kvm groups, this isn't helping, don't know why...

>libvirt/qemu.conf user = "root" group ="kvm" clear_emultaor_capabilities = 0
>libvirt/libvirtd.conf unix_sock_group = "libvirt" unix_sock_rw_perms = "0770"

Also is there needed other way to mount raw disk for VM?
right now in config I'm using

-drive file=/dev/sdc,id=disk,format=raw,if=none -device ide-hd,drive=disk

But OVMF bios is not finding my disk, also can't find working way to support AHCI.

Thanks for help

Offline

#4187 2015-02-14 04:03:50

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, I'd given up on running Windows on a VM, thanks to that Sea Islands bus reset bug. 
I just had an interesting thought:

For me, the biggest annoyance about the bus-reset failure is not just the inability to reboot the guest.
What's worse is that, if I don't stop it in time, the failed boot will make Windows try to automatically fix things.
The automatic fixing has sometimes even undone the very Windows Updates that triggered the reboot.

Is it possible to force Qemu (via libvirt) to shut down the guest instead of restarting it, when the guest requests a reboot?
Even better, is it possible to make that "restart" (but not shutdown) automatically trigger a host suspend, or a host reboot?

Offline

#4188 2015-02-14 05:35:25

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

Is it possible to force Qemu (via libvirt) to shut down the guest instead of restarting it, when the guest requests a reboot?
Even better, is it possible to make that "restart" (but not shutdown) automatically trigger a host suspend, or a host reboot?

Assuming you're using libvirt, I think

<on_reboot>destroy</on_reboot>

might be what you're after.

I believe -no-reboot is the Qemu command line option, although I will admit, I haven't tried this.

Offline

#4189 2015-02-14 13:09:10

gessert
Member
Registered: 2015-02-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tim_J wrote:
gessert wrote:

When I run the vm now, the screen turns off. I still do not have graphics output. Please advise.

It seems that you have the exact same problem that I do, which I posted a few pages ago. My system behaves the same as yours. In my case, I have one GT 210 and two GTX 580s. When the VM is launched and the screen turns off, it uses maximum CPU on a single core but never starts.

I've tried all sorts of things: unplugging the primary monitor at boot, putting GRUB in text-only mode, changing the boot TTY, quiet mode, and all manner of boot options to both GRUB and the kernel. So far, nothing has helped. It seems that the only recourse for us is to reassign the primary adapter (unsupported in both of our cases) or physically move cards around. sad Since this requires a system drain and cutting new tubes in my case, I'm postponing this for a while to see if anybody can come up with any ideas.

Yes, I noticed we had basically the same problem. I have tried a similar approach. I'm currently sending my grub and terminal booting messages through my serial port into nirvana.

/etc/default/grub

GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 console=ttyS0,38400n8 nofb" GRUB_TERMINAL=serial GRUB_SERIAL_COMMAND="serial --speed=38400 --unit=0 --word=8 --parity=no" GRUB_CMDLINE_LINUX=""

run via "update-grub"

When booting, I do not see a grub, or any boot-messages anymore. The system seems to just magically boot into X *grin*. This is an ugly "solution", but untill we figure out something proper it's going to work.

Anyway this didn't solve the issue yet. There is one more thing running on the primary card, which I'm currently looking into changing or disabling. When I switch the runlevel - CTRL + ATL + F1 and so on - the primary graphicsadapter is being used. I'm currently trying to figure out how to configure the gettys.

And I'm in way over my head. Any help would be vastly appreciated.

/proc/iomem (excerpt)

90000000-fbffbfff : PCI Bus 0000:00 a0000000-b1ffffff : PCI Bus 0000:04 a0000000-afffffff : 0000:04:00.0 b0000000-b1ffffff : 0000:04:00.0 b1000000-b12fffff : BOOTFB

Last edited by gessert (2015-02-14 13:14:45)

Offline

#4190 2015-02-14 13:41:13

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,
Since my GTX 660 is successfully running as a Quadro K4000 for some time now, I want to check if it actually yields any performance improvements. Specifically I want to compare the following configurations:
Native Windows 8.1 with GTX 660
Virtual Windows 8.1 with GTX 660 (without hyperv extensions)
Virtual Windows 8.1 with Quadro K4000 (with hyperv extensions)
running on my Intel DH87RL Board with Core i7-4771 and latest nvidia game-ready drivers. My libvirt configuration is: https://gist.github.com/flyser/2c52d9b1dfca1f19e2f9

Which benchmarks would you recommend for such a comparison? Anything you would like to see tested?

Offline

#4191 2015-02-14 14:09:12

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can u see if u can enable HW OVERLAY in nvidia control panel 3d settings with quadro driver? (i am to lazy to bloat my windows install big_smile)

Offline

#4192 2015-02-14 15:22:28

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Can u see if u can enable HW OVERLAY in nvidia control panel 3d settings with quadro driver? (i am to lazy to bloat my windows install big_smile)

I doubt that would work, since I am using the nvidia game ready driver and not the quadro specific one.

Offline

#4193 2015-02-14 16:09:01

tuxGaming
Member
Registered: 2015-02-14
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I searced for a long time but did not find an answer to my problem:

i am starting with this:
sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/dev/vg0/windows1-clone,id=osdisk,format=raw,if=none -device ide-hd,bus=ide.0,drive=osdisk -drive file=/home/tuxgaming/images/Windows7x64pro.iso,id=isocd,if=none -device ide-cd,bus=ide.1,drive=isocd -drive file=/home/tuxgaming/images/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.2,drive=virtiocd -boot menu=on

And i am getting:
This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system. (Code 12)
I think it has a confict with the Standard VGA.
adding -vga none results in a black screen "QEMU 2.1.2 monitor -type 'help' for more information (qemu)" and a blinking cursor. The monitor attacht to the NVIDIA 660ti stays black.

http://de.tinypic.com/view.php?pic=2lbm … N9yIXWc3MU

Thank you for helping out

Offline

#4194 2015-02-14 17:58:47

gessert
Member
Registered: 2015-02-09
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuxGaming wrote:

I searced for a long time but did not find an answer to my problem:

i am starting with this:
sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/dev/vg0/windows1-clone,id=osdisk,format=raw,if=none -device ide-hd,bus=ide.0,drive=osdisk -drive file=/home/tuxgaming/images/Windows7x64pro.iso,id=isocd,if=none -device ide-cd,bus=ide.1,drive=isocd -drive file=/home/tuxgaming/images/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.2,drive=virtiocd -boot menu=on

And i am getting:
This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system. (Code 12)
I think it has a confict with the Standard VGA.
adding -vga none results in a black screen "QEMU 2.1.2 monitor -type 'help' for more information (qemu)" and a blinking cursor. The monitor attacht to the NVIDIA 660ti stays black.

http://de.tinypic.com/view.php?pic=2lbm … N9yIXWc3MU

Thank you for helping out

Iirc the problem stems from the "ioh3420" device. It limits your options. I remember getting rid of it by not using q35. Remove the addresses from the devices as they use the "ioh3420" device.

It should probably look something like this:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 \ -bios /usr/share/seabios/bios.bin -vga none \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive file=/dev/vg0/windows1-clone,format=raw,media=disk \ -drive file=/home/tuxgaming/images/Windows7x64pro.iso,media=cdrom \ -drive file=/home/tuxgaming/images/virtio-win-0.1-100.iso,media=cdrom \ -boot menu=on

I'm not sure if you need the seabios option. Give it a try.

Offline

#4195 2015-02-15 13:36:30

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Firstly, I do apologize if these issues have been addressed previously, this thread is massive to read through. I am currently in the process of attempting to utilize PCI/VGA passthrough to a VM guest, but have some concerns and issues I am already encountering. Currently using 3.19 kernel, and the hardware I am using includes:

Intel Core-i7 4790K cpu
Gigabyte Z97X-UD5H motherboard (Z97 "wildcat" pch chipset)
Nvidia GeForce GTX 650 + HDMI audio PCIe device (host gpu)
AMD Radeon HD7870 + HDMI audio PCIe device (pass to guest)
Intel Lan on motherboard (use for host)
Qualcomm Lan on motherboard (pass to guest instead of using virtio)


However, I guess as to be expected for Core-i7 cpus, the PCIe root ports for the x16/x8 slots are provided by the processor itself (as can be seen in Gigabyte's diagram as well):
hM97txY.png

As a result, I believe any PCIe cards installed in these slots, including the two GPUs I am attempting to use, will appear in the same IOMMU group, and this setup becomes impossible. I think my only options are then to:
1) Use integrated Intel HD4600 graphics for host instead of PCIe card (and hope they are in separate IOMMU groups?)
2) Use potentially unsafe ACS patch to 'force' isolation

Does this seem correct? Are there other alternatives at this point in time? I really don't want to deal with patches with each kernel update, and I definitely would like to be able to use the dedicated GPU I specifically got for this.

The Qualcomm ethernet device is also in the same IOMMU group as a Marvell SATA controller (not the main Z97 chipset SATA controller), some other device (PCIe <--> PCI bridge possibly IIRC), as well as some other PCIe root ports (probably the x1 slots, and of course the ethernet/SATA devices themselves). Interestingly, based on the diagram, the Intel ethernet device appears in its own group and not in the Marvell SATA/bridge/qualcomm IOMMU group. Everything else appears to be in their own IOMMU groups. I'm not sure if this is an expected result based on the Gigabyte diagram and the PCH ACS quirks in more recent kernels.

Also, is it possible that the ACS quirks implemented for the PCH chipset would some time in the future also be done for processors themselves? Such that the particular use case I am encountering will become possible without any kernel patches or resorting to restricting all processor provided PCIe slots to the VM and none for host? As I understand it (please do correct me if wrong), even though the chipset itself does not expose/support any ACS capabilities, the implemented quirks do some of its own checks to reveal at least some information, such as isolation, that ACS would have provided? Then, can these checks also be done for processor provided root ports?

Thanks for any information!

Last edited by mutiny (2015-02-15 13:55:35)

Offline

#4196 2015-02-16 05:58:36

rantoie
Member
Registered: 2015-02-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I'm trying to get things to work with my GeForce 8800, but I keep getting the dreaded Code 43 in Windows. I must be missing something obvious, but I've been combing this thread all day trying various things.

My setup:

Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz
05:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce 8800 GT] (rev a2)
MB: ASRock H97m Pro 4

# dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x00000000BE62B7F8 000080 (v01 INTEL BDW 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.023011] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da [ 0.025279] IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0 [ 0.591229] DMAR: No ATSR found [ 0.591290] IOMMU 0 0xfed90000: using Queued invalidation [ 0.591341] IOMMU: Setting RMRR: [ 0.591395] IOMMU: Setting identity map for device 0000:00:14.0 [0xbeea2000 - 0xbeeb0fff] [ 0.591472] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbeea2000 - 0xbeeb0fff] [ 0.591544] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbeea2000 - 0xbeeb0fff] [ 0.591611] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.591666] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.623526] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 0.623605] AMD IOMMUv2 functionality not available on this system

I notice that the IOMMUv2 functionality is not available, in fact there is also this odd thing in the dmesg:

[ 0.605969] DMAR: No ATSR found [ 0.606030] IOMMU 0 0xfed90000: using Queued invalidation [ 0.606082] IOMMU: Setting RMRR: [ 0.606139] IOMMU: Setting identity map for device 0000:00:14.0 [0xbeea2000 - 0xbeeb0fff] [ 0.606216] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbeea2000 - 0xbeeb0fff] [ 0.606287] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbeea2000 - 0xbeeb0fff] [ 0.606355] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.606410] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.606476] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.606591] ------------[ cut here ]------------ [ 0.606643] WARNING: CPU: 2 PID: 1 at /build/linux-y7bjb0/linux-3.16.7-ckt4/drivers/pci/search.c:131 pci_find_upstream_pcie_bridge+0x73/0x80() [ 0.606702] Modules linked in: [ 0.606793] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt4-3 [ 0.606850] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H97M Pro4, BIOS P1.20 07/29/2014 [ 0.606908] 0000000000000009 ffffffff815096a7 0000000000000000 ffffffff810676f7 [ 0.607130] ffff88042c9bd098 ffff88042c9bd000 ffff88042c9bd098 0000000000000000 [ 0.607352] 0000000000000000 ffffffff812e4103 ffffffff813f382b 0003000000000000 [ 0.607573] Call Trace: [ 0.607623] [<ffffffff815096a7>] ? dump_stack+0x41/0x51 [ 0.607674] [<ffffffff810676f7>] ? warn_slowpath_common+0x77/0x90 [ 0.607725] [<ffffffff812e4103>] ? pci_find_upstream_pcie_bridge+0x73/0x80 [ 0.607777] [<ffffffff813f382b>] ? intel_iommu_add_device+0x3b/0x210 [ 0.607828] [<ffffffff813e8900>] ? bus_set_iommu+0x50/0x50 [ 0.607879] [<ffffffff813e8925>] ? add_iommu_group+0x25/0x50 [ 0.607931] [<ffffffff8139fa5b>] ? bus_for_each_dev+0x5b/0x90 [ 0.607981] [<ffffffff813e88f3>] ? bus_set_iommu+0x43/0x50 [ 0.608033] [<ffffffff8194a7e3>] ? intel_iommu_init+0x53f/0x586 [ 0.608084] [<ffffffff8190aade>] ? memblock_find_dma_reserve+0x157/0x157 [ 0.608135] [<ffffffff8190aaec>] ? pci_iommu_init+0xe/0x37 [ 0.608186] [<ffffffff8100213c>] ? do_one_initcall+0xcc/0x200 [ 0.608238] [<ffffffff81085fb8>] ? parse_args+0x238/0x470 [ 0.608289] [<ffffffff819021db>] ? kernel_init_freeable+0x15f/0x1e1 [ 0.608339] [<ffffffff8190195e>] ? initcall_blacklist+0xb2/0xb2 [ 0.608391] [<ffffffff814fcc10>] ? rest_init+0x80/0x80 [ 0.608441] [<ffffffff814fcc1a>] ? kernel_init+0xa/0xf0 [ 0.608492] [<ffffffff8150f6bc>] ? ret_from_fork+0x7c/0xb0 [ 0.608542] [<ffffffff814fcc10>] ? rest_init+0x80/0x80 [ 0.608595] ---[ end trace f1670364591019ce ]---

See below for the full dmesg.

05:00.0 0300: 10de:0611 (rev a2) (prog-if 00 [VGA controller]) Subsystem: 10de:054f Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at ee000000 (32-bit, non-prefetchable) [size=16M] Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at ec000000 (64-bit, non-prefetchable) [size=32M] I/O ports at d000 [size=128] [virtual] Expansion ROM at ef000000 [disabled] [size=128K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci

BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=UUID=0fabcc0e-9578-4e2e-8967-fd22994e8189 ro intel_iommu=on pcie_acs_override=downstream

<domain type='kvm'> <name>win72</name> <uuid>969e8881-64d6-4381-a1d6-699a41cf1be4</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu> <topology sockets='1' cores='3' threads='2'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/windows.qcow2'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:df:34:1a'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

/etc/modprobe.d/nouveau.conf has:
blacklist nouveau

/etc/modprobe.d/kvm.conf has:
options kvm ignore_msrs=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

Last edited by rantoie (2015-02-16 06:19:43)

Offline

#4197 2015-02-16 06:52:43

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Everything looks good except

<video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

You don't need virtual gpu, it could cause error 43 cause your 8800gt becomes secondary adapter and that is unsupported with geforce cards.

Offline

#4198 2015-02-16 08:46:39

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And if you are using a driver newer than 340.52 I think

<timer name='hypervclock' present='yes'/>

should be set to no.

Offline

#4199 2015-02-16 11:35:54

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

just to chime in. I haven't been able to find where I got the DEVID patch but there were changes in the latest qemu source and I have modified the patch so it can be applied. So for anyone who is interested in virtual "Quadrification":

http://pastebin.com/APTu6r4f

usage as previously mentioned:
Target virtual card (K4000)
Device Id:     10DE 11FA
Subsystem Id: 10DE 097C

vfio-pci,host=06:00.0,x-vga=on,x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/whereever/NVIDIA.QuadroK4000.3072.120813.rom

It has some problems with my 660TI 3GB, probably because there isn't a quadro card with exact same hardware configuration. But generally works as K4000 - same RAM config and also as K5000 - same chip.


Regarding some problems with direct host pulseaudio connection I havent found the exact cause of this, but active QXL graphics diver in guest drasticaly reduces performance and causes sound crackling in widows guest (8.1). The device can be assigned to guest so you can see the boot process, but has to be disabled in windows or not installed at all (not activating the display is not enough). Hope some of this helps.

Last edited by JohnyPea (2015-02-16 12:14:21)

Offline

#4200 2015-02-16 14:39:54

rantoie
Member
Registered: 2015-02-16
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Everything looks good except

<video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

You don't need virtual gpu, it could cause error 43 cause your 8800gt becomes secondary adapter and that is unsupported with geforce cards.

Ok, I tried to remove that with virsh edit, and then saved the xml, it comes back automatically as:

 <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video>

Reading the libvirt documentation: "For backwards compatibility, if no video is set but there is a graphics in domain xml, then libvirt will add a default video according to the guest type. "

So I tried to remove the graphics section, and the video block. This time it didn't add it back, but when I start the guest, the video is now blank yikes

mauorrizze wrote:

And if you are using a driver newer than 340.52 I think

<timer name='hypervclock' present='yes'/>

should be set to no.

Thanks, I've set that to 'no'.

Last edited by rantoie (2015-02-16 14:50:05)

Offline

#4201 2015-02-16 14:47:39

avengerf12
Member
Registered: 2014-05-30
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, first of all thanks to everyone who contributed to this great forum post, it has been immensely useful to configure my setup.

Second, I'm having a bit of a problem with the CPU configuration in a Windows 8.1 VM.
I'm currently assigning 6 vcpus to my real cpu using libvirt and when i start the VM for the first time everything works flawlessly, but if I shutdown the VM and start it again without rebooting the computer it freezes the whole host. Searching through the thread about this issue only results in a few posts suggesting to use the "isolcpus" argument in the kernel commandline to avoid any conflict. Unfortunately it didn't work and so I was hoping if someone here had any idea of what the problem could be.

This is my current config

<domain type='kvm'> <name>win8.1</name> <uuid>f3aa271e-8184-4a56-9bf5-ffbacc2f1511</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static' cpuset='6-11'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='6'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> <vcpupin vcpu='4' cpuset='10'/> <vcpupin vcpu='5' cpuset='11'/> <emulatorpin cpuset='6-11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader type='pflash'>/usr/share/ovmf/ovmf_x64.bin</loader> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> <feature policy='require' name='pbe'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ss'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='monitor'/> <feature policy='require' name='ht'/> <feature policy='require' name='dca'/> <feature policy='require' name='pcid'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> <feature policy='require' name='invtsc'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Linux-Drive/VM/Windows/virtio.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/media/vm-drives/WBoot.img'/> <target dev='sda' bus='scsi'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda1'/> <target dev='sdb' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:fb:c5:e7'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </memballoon> </devices> </domain>

I'm using a Gigabyte X79 UD3 with an i7 3930k, a r7 270x on an Arch Linux host and a gtx 680 on a Windows 8.1 VM.

Offline

#4202 2015-02-16 15:23:52

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

avengerf12 wrote:

Hello everyone, first of all thanks to everyone who contributed to this great forum post, it has been immensely useful to configure my setup.

Second, I'm having a bit of a problem with the CPU configuration in a Windows 8.1 VM.
I'm currently assigning 6 vcpus to my real cpu using libvirt and when i start the VM for the first time everything works flawlessly, but if I shutdown the VM and start it again without rebooting the computer it freezes the whole host. Searching through the thread about this issue only results in a few posts suggesting to use the "isolcpus" argument in the kernel commandline to avoid any conflict. Unfortunately it didn't work and so I was hoping if someone here had any idea of what the problem could be.

This is my current config

<domain type='kvm'> <name>win8.1</name> <uuid>f3aa271e-8184-4a56-9bf5-ffbacc2f1511</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static' cpuset='6-11'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='6'/> <vcpupin vcpu='1' cpuset='7'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> <vcpupin vcpu='4' cpuset='10'/> <vcpupin vcpu='5' cpuset='11'/> <emulatorpin cpuset='6-11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader type='pflash'>/usr/share/ovmf/ovmf_x64.bin</loader> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> <feature policy='require' name='pbe'/> <feature policy='require' name='tm2'/> <feature policy='require' name='est'/> <feature policy='require' name='vmx'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ss'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='monitor'/> <feature policy='require' name='ht'/> <feature policy='require' name='dca'/> <feature policy='require' name='pcid'/> <feature policy='require' name='tm'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='acpi'/> <feature policy='require' name='invtsc'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Linux-Drive/VM/Windows/virtio.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/media/vm-drives/WBoot.img'/> <target dev='sda' bus='scsi'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda1'/> <target dev='sdb' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:fb:c5:e7'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </memballoon> </devices> </domain>

I'm using a Gigabyte X79 UD3 with an i7 3930k, a r7 270x on an Arch Linux host and a gtx 680 on a Windows 8.1 VM.


You probably do not need "<emulatorpin cpuset='6-11'/>", that will keep all IO for guest to the same CPUs which are used as well. It's probably better (for guest performance) to allow this IO to spread over all available CPUs - just remove the line. However, I do not expect this to fix your immediate problem since it seems something deeper than just performance issue. It is possible that selected CPU type is causing problems , see if replacing "  <cpu mode='host-passthrough'> ... </cpu>" with specific model helps. FWIW I had to downgrade my guests CPU to Penryn for stable Windows 7 guest, and it still occasionally drops to "paused" state when booting it up (no problem shutting down, though). My host CPUs are IvyBridge E5-2630v2 .


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4203 2015-02-16 16:14:32

avengerf12
Member
Registered: 2014-05-30
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

You probably do not need "<emulatorpin cpuset='6-11'/>", that will keep all IO for guest to the same CPUs which are used as well. It's probably better (for guest performance) to allow this IO to spread over all available CPUs - just remove the line. However, I do not expect this to fix your immediate problem since it seems something deeper than just performance issue. It is possible that selected CPU type is causing problems , see if replacing "  <cpu mode='host-passthrough'> ... </cpu>" with specific model helps. FWIW I had to downgrade my guests CPU to Penryn for stable Windows 7 guest, and it still occasionally drops to "paused" state when booting it up (no problem shutting down, though). My host CPUs are IvyBridge E5-2630v2 .

Thanks for your reply. I removed the emulatorpin line (even though I thought it was only an override for the cpuset variable) and changed my cpu mode to <cpu mode='custom' match='exact'>. It still hasn't crashed and has a bonus it even improved performance as far as I can tell so your suggestions were definitely helpful, thanks again.
Now I'll just to wait and see, even though I think it was "cpu-passthrough"'s fault.

Edit: In the exact moment I inserted again cpu-passthrough the whole machine froze. I think my problem is solved.

Edit2: Can someone tell me if the isolcpus option is necessary in my case?

Last edited by avengerf12 (2015-02-16 16:59:04)

Offline

#4204 2015-02-16 22:30:08

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Greetings, all. Is it currently possible to get VGA passthrough working with a haswell onboard Intel HD host GPU and a discrete nvidia 900-series guest GPU or is this a unicorn at the moment?

I've pieced together various bits of info from posts in this thread and other places online but no matter what I try I always run into the code 43 error after installing the nvidia drivers in the windows guest. For reference, the hardware is:

ASRock H97m-Pro4
Intel i7-4790K (onboard GPU is Intel HD 4600)
MSI GeForce GTX 970

I'm running a CRUX host with kernel 3.18.7 (also tried 3.14.x) and qemu 2.2.0 (also tried straight from git). The guest is Windows 8.1 Enterprise 64-bit. I'm using OVMF to avoid the VGA arbitration issues with Intel onboard graphics.

I'm using qemu on the command line rather than libvirt and disabling the kvm CPU flag. I'm able to successfully install and boot the windows VM as well as see its display on the monitor connected directly to the discrete nvidia GPU, but I still can't get past the code 43 error. I see quite often that kvm and various hyper-v flags should be disabled, and as mentioned I disable the kvm flag already. When using qemu's command line rather than libvirt is it necessary to explicitly hide/disable the hyper-v flags? Is it documented somewhere how to do that? Plenty of examples out there for libvirt but none that I've seen for command line. If I've simply missed it and someone has a link, please do tell.

Thanks in advance, hope someone's got a similar setup working so I can compare.

edit: Forgot to add that the iommu groups containing the onboard and discrete GPUs do NOT overlap.

Last edited by jaeger (2015-02-16 22:37:40)

Offline

#4205 2015-02-16 23:26:26

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jaeger wrote:

Greetings, all. Is it currently possible to get VGA passthrough working with a haswell onboard Intel HD host GPU and a discrete nvidia 900-series guest GPU or is this a unicorn at the moment?

I've pieced together various bits of info from posts in this thread and other places online but no matter what I try I always run into the code 43 error after installing the nvidia drivers in the windows guest. For reference, the hardware is:

ASRock H97m-Pro4
Intel i7-4790K (onboard GPU is Intel HD 4600)
MSI GeForce GTX 970

I'm running a CRUX host with kernel 3.18.7 (also tried 3.14.x) and qemu 2.2.0 (also tried straight from git). The guest is Windows 8.1 Enterprise 64-bit. I'm using OVMF to avoid the VGA arbitration issues with Intel onboard graphics.

I'm using qemu on the command line rather than libvirt and disabling the kvm CPU flag. I'm able to successfully install and boot the windows VM as well as see its display on the monitor connected directly to the discrete nvidia GPU, but I still can't get past the code 43 error. I see quite often that kvm and various hyper-v flags should be disabled, and as mentioned I disable the kvm flag already. When using qemu's command line rather than libvirt is it necessary to explicitly hide/disable the hyper-v flags? Is it documented somewhere how to do that? Plenty of examples out there for libvirt but none that I've seen for command line. If I've simply missed it and someone has a link, please do tell.

Thanks in advance, hope someone's got a similar setup working so I can compare.

edit: Forgot to add that the iommu groups containing the onboard and discrete GPUs do NOT overlap.

I believe the HyperV part is particular to libvirt and KVM, not qemu. What emulator are you using? Please paste your qemu command.

Offline

#4206 2015-02-17 00:07:29

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jaeger wrote:

Greetings, all. Is it currently possible to get VGA passthrough working with a haswell onboard Intel HD host GPU and a discrete nvidia 900-series guest GPU or is this a unicorn at the moment?

I've pieced together various bits of info from posts in this thread and other places online but no matter what I try I always run into the code 43 error after installing the nvidia drivers in the windows guest. For reference, the hardware is:

ASRock H97m-Pro4
Intel i7-4790K (onboard GPU is Intel HD 4600)
MSI GeForce GTX 970

I'm running a CRUX host with kernel 3.18.7 (also tried 3.14.x) and qemu 2.2.0 (also tried straight from git). The guest is Windows 8.1 Enterprise 64-bit. I'm using OVMF to avoid the VGA arbitration issues with Intel onboard graphics.

I'm using qemu on the command line rather than libvirt and disabling the kvm CPU flag. I'm able to successfully install and boot the windows VM as well as see its display on the monitor connected directly to the discrete nvidia GPU, but I still can't get past the code 43 error. I see quite often that kvm and various hyper-v flags should be disabled, and as mentioned I disable the kvm flag already. When using qemu's command line rather than libvirt is it necessary to explicitly hide/disable the hyper-v flags? Is it documented somewhere how to do that? Plenty of examples out there for libvirt but none that I've seen for command line. If I've simply missed it and someone has a link, please do tell.

Thanks in advance, hope someone's got a similar setup working so I can compare.

edit: Forgot to add that the iommu groups containing the onboard and discrete GPUs do NOT overlap.

They shouldn't be enabled by default, so problem is somewhere else. Still, you can try and disable/remove hv_time, hv_relaxed, hv_vapic and hv_spinlocks on your -cpu section.

Last edited by dwe11er (2015-02-17 01:04:48)

Offline

#4207 2015-02-17 01:11:25

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@jaeger:
There must be no video device defined (vga/qxl etc.) to ensure that the nvidia is the primary display device.
In -cpu kvm=off should be explecitly set and the hv_ items should be missing/removed as already said. If this doesn't help you can try to enforce hv_time=off.
That's all I can think of.

And yes, please paste your existing qemu command.

Offline

#4208 2015-02-17 01:11:34

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

I believe the HyperV part is particular to libvirt and KVM, not qemu. What emulator are you using? Please paste your qemu command.

Not sure what you mean by "what emulator", can you clarify?

The qemu command is this:

sudo /usr/bin/qemu-system-x86_64 \ -name "Windows 8.1" \ -nographic \ -enable-kvm \ -cpu host,kvm=off \ -m 8192 \ -smp cores=4,threads=1,sockets=1 \ -vga none -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/home/jaeger/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/home/jaeger/ovmf-x64/OVMF_VARS-pure-efi.fd \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -drive file="/home/jaeger/VMs/win81/win81.img",format=raw,if=virtio,id=disk0 \ -drive file="/home/jaeger/VMs/win81/win81ent.iso",format=raw,id=cd0,if=none -device ide-cd,bus=ide.1,drive=cd0 \ -netdev bridge,id=net0 \ -device virtio-net-pci,netdev=net0 \ -usbdevice host:04d9:0183 \ -usbdevice host:1038:1369

01:00.0 and 01:00.1 are:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

dwe11er wrote:

They shouldn't be enabled by default, so problem is somewhere else. Still, you can try and disable/remove hv_time, hv_relaxed, hv_vapic and hv_spinlocks on your -cpu section.

OK, I sorta wondered if that were the case. When using the command line and not libvirt/virt-manager, though, how do you disable them? Adding hv_time=off to the cpu line seems ok but the others it complains about no matter what I pass.

edit: I was a bit premature in saying it complained "no matter what I pass", sorry for that. Passing hv_time=off,hv_relaxed=off causes no problems but doesn't fix the issue. Passing hv_spinlocks wants a numeric value so I wasn't sure what to try there, and hv_apic is unrecognized.

edit 2: I looked in the qemu source for the hv* options, I was mistyping hv-vapic as hv-apic. Still have a code 43, though.

Last edited by jaeger (2015-02-17 03:16:03)

Offline

#4209 2015-02-17 01:57:17

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Playing with the various hv_options on the command line hasn't fixed my code 43 yet but I did get some new output in the qemu window when attempting to install the nvidia drivers. Lots of this sort of thing:

qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0xa60, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0xa64, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0xa68, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x488, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x144, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x144, 0xffffff30, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x658, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x658, 0xfffffff2, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x624, 0x1c94e307, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x62c, 0x4ce8f40f, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x634, 0x35b4f7ca, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x63c, 0xa33431b5, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x624, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x62c, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x634, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x63c, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x484, 0x0, 0x4) failed: Bad address

Have I missed something obvious here?

Offline

#4210 2015-02-17 02:26:42

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to get some things working, thanks guys!
But right now I'm facing other problems:

This is my running script

qemu-system-x86_64 \ -enable-kvm -vga none -display none \ -cpu host -m 2048 -smp 4,cores=4,threads=1,sockets=1 \ -drive if=pflash,format=raw,readonly,file=/home/len/win/bios/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/home/len/win/bios/ovmf_vars_x64.bin \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive if=none,file="/dev/sdc",format=raw,id=disk0 -device ide-hd,bus=ide.0,drive=disk0 \ -drive if=none,file="/home/len/win/win7.iso",format=raw,id=cd0 -device ide-cd,bus=ide.1,drive=cd0 \ -usbdevice host:046d:c317

It works, well in some way.. OVMF starts, then I can boot from CD, but installer freeze on: "Starting Windows" (~15min still nothing).
I wanted mount it using AHCI (or I should?), but OVMF is not finding it, dunno why.

Second problem is, how can I make possible mounting /dev/sdc for normal user?
I managed to allow non-root user, take care of vfio-pci devices using chown (-__-) and some limits.conf edits.

Btw, is that normal that sometimes after running qemu, host graphic gots glithes? (fixed by switching tty).

Offline

#4211 2015-02-17 06:16:07

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there still (as of 2015-02-17) a problem with the i915 patch disabling DRI in XORG?

Is the i915 patch still needed for Intel iGPU as host gpu, or are have the patches made it upstream?

Will things still work without any patches if you use AMD/Radeon GPU's only (host and guest)?

Thanks, JonasCJ

jonascj wrote:

Thank you, aw, for that very insightful reply.

I have a couple of follow-up questions and/or need a few clarifications.

aw wrote:

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI. The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

Regarding what is marked in red:

So applying the the first patch (which makes i915 work correctly with VGA arbitration) kills overall performance (because it makes Xorg  disable DRI which forces everyone to make indirect rendering) and because of that that patch was removed / reverted upstream?

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

Regarding the text marked in blue:
Solutions to which problem? Solutions to the i915 arbiter problem - i.e. an alternative to the current patch?

aw wrote:
jonascj wrote:
aw wrote:

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Sorry for being so thick, but this means that everyone using the i915 module/driver suffer from disabled DRI while they use kernels patched with this patch? I.e. everyone using Intel IGPU for their host while having a dedicated gfx passed through to a KVM guest have DRI disabled on their host.

Yes

Offline

#4212 2015-02-17 06:30:02

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Len: are you trying to install Windows 7 or Windows 8+?

@jonascj: AFAIK the i915 arbitration patch can't be improved without breaking things in Xorg or elsewhere. So there'd be no way that this became mainstream.
On the other hand if you install Windows 8 or newer, use OVMF, no need for VGA arbitration. AMD drivers don't need any patches, work with both VGA and OVMF.

Offline

#4213 2015-02-17 06:38:36

jonascj
Member
Registered: 2014-02-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:

@jonascj: AFAIK the i915 arbitration patch can't be improved without breaking things in Xorg or elsewhere. So there'd be no way that this became mainstream.
On the other hand if you install Windows 8 or newer, use OVMF, no need for VGA arbitration. AMD drivers don't need any patches, work with both VGA and OVMF.

So with AMD R9 270X as the intended guest card, and an Intel iGPU (from an i7-whatever) I would go with a unpatched kernel if I go the OVMF route?

I found this by rereading post #1: http://vfio.blogspot.dk/2014/08/primary … t-vga.html

Do you know of any other good starting points for diving into this.

Thanks, a lot!

Offline

#4214 2015-02-17 09:04:25

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

I managed to get some things working, thanks guys!
But right now I'm facing other problems:

This is my running script

qemu-system-x86_64 \ -enable-kvm -vga none -display none \ -cpu host -m 2048 -smp 4,cores=4,threads=1,sockets=1 \ -drive if=pflash,format=raw,readonly,file=/home/len/win/bios/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/home/len/win/bios/ovmf_vars_x64.bin \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive if=none,file="/dev/sdc",format=raw,id=disk0 -device ide-hd,bus=ide.0,drive=disk0 \ -drive if=none,file="/home/len/win/win7.iso",format=raw,id=cd0 -device ide-cd,bus=ide.1,drive=cd0 \ -usbdevice host:046d:c317

It works, well in some way.. OVMF starts, then I can boot from CD, but installer freeze on: "Starting Windows" (~15min still nothing).
I wanted mount it using AHCI (or I should?), but OVMF is not finding it, dunno why.

both AHCI and IDE are incredibly slow, I found that the best way to install Windows on kvm is to attach second CD with iso downloaded from http://www.linux-kvm.org/page/WindowsGu … ad_Drivers, define your C drive as virtio one. During system installation you will select virtio drivers from the attached 2nd CDROM (i.e. iso you downloaded from above URL) and then carry on with installation which will be much faster


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4215 2015-02-17 11:30:58

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Len wrote:

I managed to get some things working, thanks guys!
But right now I'm facing other problems:

This is my running script

qemu-system-x86_64 \ -enable-kvm -vga none -display none \ -cpu host -m 2048 -smp 4,cores=4,threads=1,sockets=1 \ -drive if=pflash,format=raw,readonly,file=/home/len/win/bios/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/home/len/win/bios/ovmf_vars_x64.bin \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive if=none,file="/dev/sdc",format=raw,id=disk0 -device ide-hd,bus=ide.0,drive=disk0 \ -drive if=none,file="/home/len/win/win7.iso",format=raw,id=cd0 -device ide-cd,bus=ide.1,drive=cd0 \ -usbdevice host:046d:c317

It works, well in some way.. OVMF starts, then I can boot from CD, but installer freeze on: "Starting Windows" (~15min still nothing).
I wanted mount it using AHCI (or I should?), but OVMF is not finding it, dunno why.

both AHCI and IDE are incredibly slow, I found that the best way to install Windows on kvm is to attach second CD with iso downloaded from http://www.linux-kvm.org/page/WindowsGu … ad_Drivers, define your C drive as virtio one. During system installation you will select virtio drivers from the attached 2nd CDROM (i.e. iso you downloaded from above URL) and then carry on with installation which will be much faster

Thanks for reply, I'll try it right now.

So I'm assuming that after installation, I should still use 'virtio' drivers for my drives?
I'm looking for ones with best performance for SSD drive, (have both SSD, for linux and windows).

@mauorrizze Windows 7

Offline

#4216 2015-02-17 11:59:52

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

@mauorrizze Windows 7

It won't work with OVMF without some hacking , you can search for some posts about installing it from FAT32 USB drive .

Also , I think its better to use SCSI rather than AHCI . My drives defined as follows :

-drive file=/VMs/Win_Main.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \

Good luck !

Last edited by Denso (2015-02-17 12:00:37)

Offline

#4217 2015-02-17 15:33:23

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

Thanks for reply, I'll try it right now.

So I'm assuming that after installation, I should still use 'virtio' drivers for my drives?

that's right


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4218 2015-02-17 18:03:27

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Len wrote:

@mauorrizze Windows 7

It won't work with OVMF without some hacking , you can search for some posts about installing it from FAT32 USB drive .

Also , I think its better to use SCSI rather than AHCI . My drives defined as follows :

-drive file=/VMs/Win_Main.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \

Good luck !

No luck even with scsi device, still stuck at installing stage,
I was trying to boot in OVMF with usb device too (pendrive created using dd), but no luck yet.

Last edited by Len (2015-02-17 18:04:06)

Offline

#4219 2015-02-17 18:46:35

sl44x
Member
Registered: 2015-02-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everbody. Just leaving my personal tests with my desktop hardware, from the last couple of days. This maybe can save some time for someone with similar hardware:

Gigabyte GA-H61M-D2-B3
Intel i5 2500 @ 3300 Mhz
Intel IGP
NVidia GeForce 9600GT (to passthrough)

I tryed with default Arch Kernel (latest) with no success and then try with custom patches from topic author, also, with no success.
In last, tryed with some other kernel parameters posted here like ACS override and vfio_iommu_type1.allow_unsafe_interrupts, but no success.

Working:

- Only can enable the Intel IOMMU on Kernel and get only this one message on dmesg: ([0.000000] Intel-IOMMU: enabled)

Issues:

- No mapping DMAR or others IOMMU related messages on dmesg. Just the above.
- The "/sys/kernel/iommu_groups" is always empty.
- The device to passthrough dont have a iommu_group link in "/sys/bus/pci/devices/$dev"
- In case to ignore all these "issues" and continue with vfio-bind command and open the qemu receive a already waited "vfio: error no iommu_group for device".

P.S:

- All tests done with NVidia driver on blacklist at "/etc/modprobe.d/blacklist.conf"
- The MOBA has no IOMMU/VT-d related configs on BIOS, but a generic "Enable Virtualization Features" (obviously made all tests with this enabled).
- I have seen some people reporting successuly VGA passthrough with H61 chipset from Intel, but with other MOBOs with same chipset (all reports I read are from non-Gigabyte). But with this from Gigabyte, I can't.

Final consideration:

- I assume that this MOBO (Gigabyte GA-H61M-D2-B3) don't support the VT-d (IOMMU) feature.

Someone else have any suggestion to try something more? Thanks.

Last edited by sl44x (2015-02-17 18:50:27)

Offline

#4220 2015-02-17 19:26:27

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sl44x wrote:

Final consideration:

- I assume that this MOBO (Gigabyte GA-H61M-D2-B3) don't support the VT-d (IOMMU) feature.

Someone else have any suggestion to try something more? Thanks.

Check for BIOS updates, my ASUS mobo didn't support IOMMU from two revisions ago, and there was a release with broken IOMMU support.

Last edited by Duelist (2015-02-17 19:26:37)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4221 2015-02-17 20:14:29

sl44x
Member
Registered: 2015-02-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
sl44x wrote:

Final consideration:

- I assume that this MOBO (Gigabyte GA-H61M-D2-B3) don't support the VT-d (IOMMU) feature.

Someone else have any suggestion to try something more? Thanks.

Check for BIOS updates, my ASUS mobo didn't support IOMMU from two revisions ago, and there was a release with broken IOMMU support.

Yes, sadly forgot to say that update the BIOS was one of the first things that I tryed. My BIOS already is in the latest version available. Thanks anyways.

Offline

#4222 2015-02-17 21:38:43

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Len: you can try to drop -smp and use only one core and/or force -cpu hv_time=off
According to https://bugzilla.redhat.com/show_bug.cgi?id=1185253 it's a interference bug between ovmf, smp and hyperv-timer but in my experience it's sadly a more general problem with ovmf and window 7 at the moment. Maybe try different versions of qemu and ovmf.

@All:
If both graphic cards are in the same IOMMU group, then I can only pass one or both of them to a common guest VM, not each to different guests, is that correct?

Offline

#4223 2015-02-17 21:49:01

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Recently I bought a sound card because I was fed up with laggy sound in VMs. The card works. I can pass it through... but only by soft-removing my main usb2.0 controller. All because the controller and the card share the same IRQ. I can live without that controller (already swapped my keyboard, mouse, etc to available usb3.0 ports), but it still sucks when you can't use 4 out of 6 available ports on a motherboard. I don't mind losing the controller while the VM is online. Therefore I've been using soft-remove (echo 1 > /sys/bus/pci/devices/***/remove) . But the biggest issue is that I have to reboot every time to get the controller back online. I also tried assigning both the controller and the sound card to the VM, but it doesn't work. What else I can do?

This is the error in dmesg when I try to assign the sound card alone or both the controller and the card:

genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:00:1a.0)) vs. 00000000 (vfio-intx(0000:08:04.0))

And here is the relevant lspci output (the sound card is behind a bridge, which I don't assign):

00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI]) Subsystem: Gigabyte Technology Co., Ltd 7 Series/C210 Series Chipset Family USB Enhanced Host Controller Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 16 Region 0: Memory at f7a18000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features AFCap: TP+ FLR+ AFCtrl: FLR- AFStatus: TP- Kernel driver in use: ehci-pci 07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 16 Bus: primary=07, secondary=08, subordinate=08, sec-latency=64 I/O behind bridge: 0000c000-0000cfff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Power Management version 3 Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [80] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [c0] Subsystem: Device 0000:0000 Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- 08:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] Subsystem: ASUSTeK Computer Inc. CMI8788 [Oxygen HD Audio] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: I/O ports at c000 [disabled] [size=256] Capabilities: [c0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

Offline

#4224 2015-02-17 22:00:42

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

P.S. if anyone needs UEFI-combatible VBIOS file for ASUS HD7750(silent, 800mhz one, but i think it's possible to edit it for any card) - send me an e-mail, because i seem to be unable to get it officially from support anymore.

Sorry, can I request that vbios file here (i.e. without sending you an email)? Also, sorry if you've already answered such a question. I haven't read all the posts after that message.

Offline

#4225 2015-02-17 22:06:13

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, something is very broken here.

sudo qemu-system-x86_64 \ -d out_asm,in_asm,op,op_opt,int,exec,cpu,pcall,cpu_reset,ioport,unimp,guest_errors \ -D /mnt/hdd/qemu/log-`date +%d-%m-%y-%T`.log \ -enable-kvm \ -M pc \ -m 4096 \ -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -smp 4,sockets=1,cores=4,threads=1 \ -vga none \ -usb \ -boot menu=on \ -soundhw hda \ -device vfio-pci,host=01:00.0,x-vga=off,multifunction=on \ -device vfio-pci,host=01:00.1 \ ...various stuff... -device qxl \ -drive if=pflash,format=raw,readonly,file=/mnt/hdd/qemu/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/mnt/hdd/qemu/OVMF_VARS.fd

And...
http://pastebin.com/MVaMNp7K
100% cpu load, nothing on any output.
Pure nothing!
WTF?

When booting bare metal, i can see the setup menu and even attempt to boot into windows7. But windows7 fails to output video signal after the starting logo.

UPD:
Alright, i've connected to qemu via gdb(had to set the mode manually), and..
I'm no low-level specialist, but something hints me that this won't work:

(gdb) info reg rax 0x0 0 rbx 0x810248 8454728 rcx 0x402 1026 rdx 0x402 1026 rsi 0x10 16 rdi 0xbfed1060 3219984480 rbp 0xbff62ad0 0xbff62ad0 rsp 0xbff62ac0 0xbff62ac0 r8 0x1 1 r9 0x0 0 r10 0x36 54 r11 0xd7 215 r12 0x0 0 r13 0x0 0 r14 0x0 0 r15 0x0 0 rip 0xbed9d9df 0xbed9d9df eflags 0x246 [ PF ZF IF ] cs 0x28 40 ss 0x8 8 ds 0x8 8 es 0x8 8 fs 0x8 8 gs 0x8 8 (gdb) x/10i $rip => 0xbed9d9df: mov -0x8(%rbp),%rax 0xbed9d9e3: test %rax,%rax 0xbed9d9e6: je 0xbed9d9df 0xbed9d9e8: leaveq 0xbed9d9e9: retq 0xbed9d9ea: push %rbp 0xbed9d9eb: mov %rsp,%rbp 0xbed9d9ee: sti 0xbed9d9ef: pop %rbp 0xbed9d9f0: retq

I don't recall mov being able to do stuff with MINUS, but let it be.
So it seems like we're trying to set RBP(Stack Base pointer register Holds the base address of the stack) to zero(RAX) MINUS 0x8.

I'm not familiar with GDB at all, but this looks strange as hell.
I guess i'll have to start qemu with -S and run the VM step-by-step.
Something hints me that the ROM on the device is broken, since this kind of stuff happens only after i add the ROM(currently it's flashed on the hardware).

Damn, forgot that it's AT&T syntax, so we're writing RBP-0x8 to RAX, and previously(at 0xbed9d9d7, not shown in the post) we did " movl $0x0,-0x8(%rbp)" and we have our zero in RAX. Why did it stop..
UPD2:
topkek. It moves zero to RAX, compares RAX with RAX, sets ZF to 1 and does JE(if ZF=1 then jump) back to that MOV. Alright, now i know that something is deeply borked here. Now i must determine where did this come from..

Last edited by Duelist (2015-02-18 00:25:16)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4226 2015-02-17 23:43:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sl44x wrote:

Intel i5 2500 @ 3300 Mhz

This is not a processor version, it's a family and a speed, so we have no idea if your CPU supports VT-d.  Look it up on ark.intel.com


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4227 2015-02-18 00:57:45

sl44x
Member
Registered: 2015-02-17
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
sl44x wrote:

Intel i5 2500 @ 3300 Mhz

This is not a processor version, it's a family and a speed, so we have no idea if your CPU supports VT-d.  Look it up on ark.intel.com

Sorry. Here the full description of this processor:

http://ark.intel.com/products/52209/Int … o-3_70-GHz

Intel® Virtualization Technology for Directed I/O (VT-d) ‡ Yes

The processor has support VT-d. I belive that is the bios or chipset of the mobo that not.

Last edited by sl44x (2015-02-18 01:06:32)

Offline

#4228 2015-02-18 05:16:06

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is the error in dmesg when I try to assign the sound card alone or both the controller and the card:

genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:00:1a.0)) vs. 00000000 (vfio-intx(0000:08:04.0))

And here is the relevant lspci output (the sound card is behind a bridge, which I don't assign):

00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI]) Subsystem: Gigabyte Technology Co., Ltd 7 Series/C210 Series Chipset Family USB Enhanced Host Controller 07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04) (prog-if 00 [Normal decode]) 08:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] Subsystem: ASUSTeK Computer Inc. CMI8788 [Oxygen HD Audio]

I had similar issues with my Xonar DX, and found two ways to possibly fix the issue: either move the card to another slot, or go into the BIOS and disable the conflicting EHCI controller (while leaving the other enabled). 
I can imagine some BIOSes having EHCI #1 and #2 swapped, so you should double-check it once you reboot.

Note that the Xonar DX is just a Xonar D1 behind a PLX PEX8112, and StarTech offers a standalone PEX8112 board for use with any arbitrary PCI card.

Offline

#4229 2015-02-18 08:55:14

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Something OVMF/UEFI-Rom related:
I just investigated why I don't see the tiancore bootscreen anymore after switching from my 750Ti to an R9 280X. Turns out it has only a legacy rom, but apart from the inability to see the UEFI output of the VM at boot it works fine!? Could this work for all cards or only AMD because the card isn't initialized as the primary one?

Edit: If this works with more cards, than we could use older cards with OVMF by installing Windows with a virtual graphics adapter and later switching to the real one, booting, letting Windows install drivers and ready is the legacy-ovmf-hybrid.
Regarding my 280X, I'm gonna flash an other BIOS because there's one rather similar to mine including UEFI support. Thanks again for the rom-parser!

Last edited by mauorrizze (2015-02-18 09:13:01)

Offline

#4230 2015-02-18 10:19:23

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

I had similar issues with my Xonar DX, and found two ways to possibly fix the issue: either move the card to another slot,

Not possible. This motherboard just refuses to boot when I move the sound card to another slot.

or go into the BIOS and disable the conflicting EHCI controller (while leaving the other enabled).

Also not possible. There is only one option in UEFI to disable both EHCI controllers. If I disable them, even xHCI stops working pre-boot, so no keyboard/mouse input possible until Linux. In any case, this method isn't better than soft-remove. With soft-remove I can at least use the controller until I choose to disable it.

Note that the Xonar DX is just a Xonar D1 behind a PLX PEX8112

I know there are certain Xonar DX with PLX chip, but mine uses ASMedia as seen in the lspci output.

Offline

#4231 2015-02-18 14:11:08

Maloute
Member
Registered: 2015-02-18
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello guys,

I am also hijacking this thread, thanks to the tutorial in first page, my vga passthrough is not far from working but I'm stuck at the "code 43" phase.

My config:
Cpu: Core I7 860 (first gen with no IGP, but with VT-D)
Motherboard: Asus P7P55D
Graphic card #1 in the first PCI-E 16X port: ATI/AMD Radeon HD4350 fanless for host display
Graphic card #2 in the 2nd PCI-E 16X port: Gainward Golden Sample Geforce 560ti 1Gb for vga passthrough
Host Software config: Archlinux, multiple kernel config, 3.18.5 patch + vga arbiter patch + acs override patch, or 3.18.6 not patched, stock QEMU from Archlinux
Guest soft: Windows 8.1 x64

The VGA passthrough is working fine, when I am launching QEMU, I do see the bios output on the 2nd screen and I can also just install and use Win 8.1 just fine. The only problem I have is that  guest reboot is not working and I can only launch QEMU once, if I need to halt the guest, I have to reboot the host to reuse the guest again.

So, after installing Geforce Drivers in the guest, the card is recognized by Windows as GeForce 560ti but I have an "code 43" error like a lot people...

What I tried to solve the problem:
- installing 331 and 335 Geforce drivers version -> same problem
- using microsoft whql drivers -> same problem
- using -cpu host,kvm=off option -> same problem
- Tried with the 3.18.5 patched kernel (i915 + acs override) -> same problem
- fresh 3.18.6 kernel -> same problem
- using 440FX chipset instead of Q35 with the Qemu config: -> same problem
- tried with romfile corresponding to my model -> same problem

Here is my QEMU command:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off -vga none \ -smp 6,sockets=1,cores=6,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \ -drive file=windows.img,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ -usb -device usb-host,hostbus=1,hostaddr=4 \ -usb -device usb-host,hostbus=1,hostaddr=3

I'm stuck and I don't see what is wrong... Ideas or advices would be appreciated :-)
Thank you !

Offline

#4232 2015-02-18 16:12:20

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maloute wrote:

The only problem I have is that  guest reboot is not working and I can only launch QEMU once, if I need to halt the guest, I have to reboot the host to reuse the guest again.

I had the same problem with a card which turned out not to support GPU pass through.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4233 2015-02-18 16:19:44

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As a followup to my earlier posts, I've been playing with the various hv* options but unfortunately still haven't had any luck. I always get the "code 43" error in the device manager for the nvidia GPU and the vfio_pci_read_config and vfio_pci_write_config "bad address" errors.

I've tried every version of the nvidia drivers that supports the GTX 970 with no luck, 344.11 being the oldest.

My current qemu invocation is this:

#!/bin/bash VFIO_DEVS="01:00.0 01:00.1" sudo /opt/bin/vfio-bind ${VFIO_DEVS} NAME="Windows 8.1 Enterprise" MEM=8192 CORES=4 DRIVE=/home/jaeger/VMs/win81/win81.img CDROM=/home/jaeger/VMs/win81/win81entx64.iso VIRTIO=/home/jaeger/VMs/win81/virtio-win-0.1-81.iso OVMF=/home/jaeger/ovmf-x64 sudo QEMU_AUDIO_DRV=pa \ /usr/bin/qemu-system-x86_64 \ -name "${NAME}" \ -nographic \ -enable-kvm \ -cpu host,kvm=off,hv-time=off,hv-vapic=off,hv-relaxed=off \ -m ${MEM} \ -smp cores=${CORES},threads=1,sockets=1 \ -vga none \ -nodefconfig \ -drive if=pflash,format=raw,readonly,file=${OVMF}/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=${OVMF}/OVMF_VARS-pure-efi.fd \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -drive file="${DRIVE}",format=raw,id=disk0,if=virtio \ -drive file="${CDROM}",format=raw,id=cd0,if=none \ -device ide-cd,drive=cd0 \ -drive file="${VIRTIO}",format=raw,id=cd1,if=none \ -device ide-cd,drive=cd1 \ -netdev bridge,id=net0 \ -device virtio-net-pci,netdev=net0 \ -soundhw hda \ -usbdevice host:046d:c52b \ -usbdevice host:03f0:0024

Has anyone else seen the vfio_pci_read_config and vfio_pci_write_config errors?

Offline

#4234 2015-02-18 16:35:23

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

thanks to this thread and the great tutorial my setup is also up and works like expected.

Setup:
Asrock Extreme6
I7-4790K
HD4850

Just the power consumption of the second card after shutting down the guest is a problem.

41W without HD4850, linux idle
61W with installed HD4850, linux idle
83W Guest Windows7 idle, linux idle
83W after Guest Windows7 is shut down, linux idle.

After digging around in sysfs and ASPM and googling around, i have absolutly no idea how set the second card in back in idle state.

Maybe someone with more experience have a solution to this problem.

Thank you in advance.

Last edited by Schlunze (2015-02-18 16:37:43)

Offline

#4235 2015-02-18 17:25:47

Jcrash
Member
From: Toronto
Registered: 2012-11-12
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maloute wrote:

Hello guys,

I am also hijacking this thread, thanks to the tutorial in first page, my vga passthrough is not far from working but I'm stuck at the "code 43" phase.

[...]

Here is my QEMU command:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off -vga none \ -smp 6,sockets=1,cores=6,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \ -drive file=windows.img,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ -usb -device usb-host,hostbus=1,hostaddr=4 \ -usb -device usb-host,hostbus=1,hostaddr=3

I'm stuck and I don't see what is wrong... Ideas or advices would be appreciated :-)
Thank you !

Try loading your card from a ROM file (see bottom of post #1). I got mine from

http://www.techpowerup.com/vgabios/

I had a lot of trouble with my 560ti within the VM regarding drivers, sleep functions, resets...right now I have it assigned to the host. I use a 6800 series ATI for the VM and it behaves, updates clean.

Offline

#4236 2015-02-18 17:39:06

Mysingen
Member
Registered: 2015-02-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone. Thank you all for this guide. I have been able to pass through an Nvidia 970 card to Windows 8.1 and had it working. Briefly... Then the driver detected the virtualization and threw up Code 43.

The problem seems to be that I am not booting with OVMF even though I try to. When I point directly to the TianoCore firmware, the ISO is not detected as bootable:

-drive if=pflash,format=raw,readonly,file=/mnt/vault/qemu/ovmf/20150218/OVMF_CODE-pure-efi.fd -drive if=pflash,format=raw,file=/mnt/vault/qemu/ovmf/20150218/OVMF_VARS-pure-efi.fd

I first suspected that I was missing other component such as ipxe.git, etc, due to dangling symlinks in the 20150218 directory and installed those as well. But it doesn't help. When I point to the entire 20150218 directory (which contains a symlink bios.bin -> OVMF-pure-efi.fd), QEMU boots with SeaBios instead:

-L /mnt/vault/qemu/ovmf/20150218

Any idea what I might be doing wrong? There are guides on the net for converting USB sticks from MBR to GPT so that they can be used as installation media, but I found nothing similar for ISO's. My suspicion is that the ISO somehow does not support UEFI but I don't know how to check this theory. All I see is that TianoCore detects the virtual DVD drive, but not the ISO in it.

Current config:

ROM=/mnt/vault/qemu/rom/Asus.GTX970.4096.140917.rom DISK=/mnt/vault/qemu/img/win81-ovmf.qcow2 ISO1=/mnt/vault/qemu/iso/windows81pro64.iso BIOS=/mnt/vault/qemu/ovmf/20150218 CODE=/mnt/vault/qemu/ovmf/20150218/OVMF_CODE-pure-efi.fd VARS=/mnt/vault/qemu/ovmf/20150218/OVMF_VARS-pure-efi-windows81.fd qemu-system-x86_64 \ -nodefconfig \ -drive if=pflash,format=raw,readonly,file=$CODE \ -drive if=pflash,format=raw,file=$VARS \ -enable-kvm \ -M pc-i440fx-2.1 \ -m 16384 \ -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -rtc base=localtime \ -soundhw hda \ -device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=$ROM \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=$ISO1,media=cdrom \ -drive file=$DISK,media=disk \ -monitor stdio \ -vga cirrus

/M

Last edited by Mysingen (2015-02-18 18:17:38)

Offline

#4237 2015-02-18 18:19:26

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

100% cpu load, nothing on any output.
Pure nothing!
WTF?

When booting bare metal, i can see the setup menu and even attempt to boot into windows7. But windows7 fails to output video signal after the starting logo.

UPD:
Alright, i've connected to qemu via gdb(had to set the mode manually), and..
I'm no low-level specialist, but something hints me that this won't work:

(gdb) info reg rax 0x0 0 rbx 0x810248 8454728 rcx 0x402 1026 rdx 0x402 1026 rsi 0x10 16 rdi 0xbfed1060 3219984480 rbp 0xbff62ad0 0xbff62ad0 rsp 0xbff62ac0 0xbff62ac0 r8 0x1 1 r9 0x0 0 r10 0x36 54 r11 0xd7 215 r12 0x0 0 r13 0x0 0 r14 0x0 0 r15 0x0 0 rip 0xbed9d9df 0xbed9d9df eflags 0x246 [ PF ZF IF ] cs 0x28 40 ss 0x8 8 ds 0x8 8 es 0x8 8 fs 0x8 8 gs 0x8 8 (gdb) x/10i $rip => 0xbed9d9df: mov -0x8(%rbp),%rax 0xbed9d9e3: test %rax,%rax 0xbed9d9e6: je 0xbed9d9df 0xbed9d9e8: leaveq 0xbed9d9e9: retq 0xbed9d9ea: push %rbp 0xbed9d9eb: mov %rsp,%rbp 0xbed9d9ee: sti 0xbed9d9ef: pop %rbp 0xbed9d9f0: retq

Well, I have good news and bad news.
Good news:
I did this:

(gdb) x/10b 0xbed9d9df 0xbed9d9df: 0x48 0x8b 0x45 0xf8 0x48 0x85 0xc0 0x74 0xbed9d9e7: 0xf7 0xc9

Got some bytes from here, like: 0x48 0x8b ... 0x74 0xf7 0xc9.
So i've used bgrep to search for this sequence of bytes in all my binaries being loaded to QEMU... naaah, just kidding, i'm an idiot, so i did just

hexdump -C /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd | grep "74 f7 c9" 001cdaf0 90 48 8b 45 f8 48 85 c0 74 f7 c9 c3 55 48 89 e5 |.H.E.H..t...UH..|

and

hexdump -C ~/rom-parser/image_uefi.rom | grep "74 f7 c9"

.
Good news are - IT IS OVMF THAT IS BROKEN AND PREVENTING MY VM TO BOOT WITH MY HARDWARE
Bad news are - while I know who broke it, i don't know what section of the actual code does this.

Can someone skilled enough help me to advance further on fixing this bug?

P.S.
I'm tempted to just manually break that cycle by inserting instructions via gdb live, but i don't know how to do that.

EDIT:
Piece of code that wasn't included in previous post:

 0xbed9d9d6: movq $0x0,-0x8(%rbp) 0xbed9d9de: nop 0xbed9d9df: mov -0x8(%rbp),%rax => 0xbed9d9e3: test %rax,%rax 0xbed9d9e6: je 0xbed9d9df

UPD:
Okay, I did something(actually, i've updated OVMF to the freshest git build) and now it just outputs "invalid ROM contents" to dmesg when trying to boot. Trying to pass the romfile option. That worked. Looks like i've accidentally flashed some piece of ROM somehow. Spooky.

UPD2:
Stopping qemu with GDB, changing that JE to JNE(really, one byte)... worked. When QEMU process continued, the GPU successfully fired-up and showed me graphikz. Le success. Well, now i have to write a little essay to EDK2-devel mailing list regarding that. Maybe someone more clever will fix it properly.

Last edited by Duelist (2015-02-18 20:03:30)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4238 2015-02-18 22:36:39

Maloute
Member
Registered: 2015-02-18
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

I had the same problem with a card which turned out not to support GPU pass through.

Jcrash wrote:

Try loading your card from a ROM file (see bottom of post #1). I got mine from

http://www.techpowerup.com/vgabios/

I had a lot of trouble with my 560ti within the VM regarding drivers, sleep functions, resets...right now I have it assigned to the host. I use a 6800 series ATI for the VM and it behaves, updates clean.

Thanks for your replies !
I tried the ROM file corresponding to my graphic card (Gainward Golden Sample 560 ti 1Gb) but there was no improvement...  It seems that this graphic card is not very well suited for VGA passthrough...

Offline

#4239 2015-02-18 22:41:51

fingerboxes
Member
Registered: 2015-02-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I've tried sorting through the last 25 pages or so of this thread and I think I'm sort of on the right path. It seems as though the OP is wildly out of date, though; can anyone direct me toward a document which contain more up to date information (In particular, on setting up OVMF instead of using vga arbitration, or any information from someone who has had success using a physical disk instead of an image for qemu)

Offline

#4240 2015-02-18 23:43:28

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jaeger wrote:

As a followup to my earlier posts, I've been playing with the various hv* options but unfortunately still haven't had any luck. I always get the "code 43" error in the device manager for the nvidia GPU and the vfio_pci_read_config and vfio_pci_write_config "bad address" errors.

I've tried every version of the nvidia drivers that supports the GTX 970 with no luck, 344.11 being the oldest.

My current qemu invocation is this:

#!/bin/bash VFIO_DEVS="01:00.0 01:00.1" sudo /opt/bin/vfio-bind ${VFIO_DEVS} NAME="Windows 8.1 Enterprise" MEM=8192 CORES=4 DRIVE=/home/jaeger/VMs/win81/win81.img CDROM=/home/jaeger/VMs/win81/win81entx64.iso VIRTIO=/home/jaeger/VMs/win81/virtio-win-0.1-81.iso OVMF=/home/jaeger/ovmf-x64 sudo QEMU_AUDIO_DRV=pa \ /usr/bin/qemu-system-x86_64 \ -name "${NAME}" \ -nographic \ -enable-kvm \ -cpu host,kvm=off,hv-time=off,hv-vapic=off,hv-relaxed=off \ -m ${MEM} \ -smp cores=${CORES},threads=1,sockets=1 \ -vga none \ -nodefconfig \ -drive if=pflash,format=raw,readonly,file=${OVMF}/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=${OVMF}/OVMF_VARS-pure-efi.fd \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -drive file="${DRIVE}",format=raw,id=disk0,if=virtio \ -drive file="${CDROM}",format=raw,id=cd0,if=none \ -device ide-cd,drive=cd0 \ -drive file="${VIRTIO}",format=raw,id=cd1,if=none \ -device ide-cd,drive=cd1 \ -netdev bridge,id=net0 \ -device virtio-net-pci,netdev=net0 \ -soundhw hda \ -usbdevice host:046d:c52b \ -usbdevice host:03f0:0024

Has anyone else seen the vfio_pci_read_config and vfio_pci_write_config errors?

For science, today I installed Fedora 21 on an extra hard drive and tested my setup using libvirt and virt-manager (vs. qemu command line under CRUX). Under Fedora 21 it works! So I know now that my hardware isn't to blame for the failures I've been experiencing, it's got to be somewhere in my qemu or kernel configuration, I guess.

Unfortunately I don't know what exactly to chase regarding the vfio_pci_{read,write}_config errors but at least I know that what I want to do is possible with this particular hardware config.

I dumped the xml from my libvirt VM for comparison but I don't see anything between it and my qemu command line that screams broken. That makes me think perhaps there's a kernel issue. Fedora includes a lot of things I'm not building into my own kernel like cgroups, etc., so I'll poke around and see if I can find something there. Maybe I'll try the Fedora kernel config on my CRUX machine.

Offline

#4241 2015-02-19 03:38:27

Jcrash
Member
From: Toronto
Registered: 2012-11-12
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maloute wrote:

[...]
I tried the ROM file corresponding to my graphic card (Gainward Golden Sample 560 ti 1Gb) but there was no improvement...  [...]

I've used roms from other manufacturers (equal hardware specs; gpu/#core's/clock)

My process:
1. new win7 VM (base, sytem updates) for every rom file
2. shutdown; make copy of VM on host
3. boot copy; install video drivers (start with most recent, work back)
4. working? rename and file

I threw in the towel because I couldn't take advantage of any power saving features; VM's wouldn't recover from sleep or monitor wake. Force close through host, reboot to a corrupted VM image (every time, unrecoverable), and rollback to step 3.

Careful though: I have a wireless hdmi 460ti that lost it's wireless hdmi capability through this type of tinkering (i think). Can't find the card-specific rom for it so I gambled

Haven't tried with win8

Offline

#4242 2015-02-19 14:43:39

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jaeger wrote:
#!/bin/bash VFIO_DEVS="01:00.0 01:00.1" sudo /opt/bin/vfio-bind ${VFIO_DEVS} NAME="Windows 8.1 Enterprise" MEM=8192 CORES=4 DRIVE=/home/jaeger/VMs/win81/win81.img CDROM=/home/jaeger/VMs/win81/win81entx64.iso VIRTIO=/home/jaeger/VMs/win81/virtio-win-0.1-81.iso OVMF=/home/jaeger/ovmf-x64 sudo QEMU_AUDIO_DRV=pa \ /usr/bin/qemu-system-x86_64 \ -name "${NAME}" \ -nographic \ -enable-kvm \ -cpu host,kvm=off,hv-time=off,hv-vapic=off,hv-relaxed=off \ -m ${MEM} \ -smp cores=${CORES},threads=1,sockets=1 \ -vga none \ -nodefconfig \ -drive if=pflash,format=raw,readonly,file=${OVMF}/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=${OVMF}/OVMF_VARS-pure-efi.fd \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -drive file="${DRIVE}",format=raw,id=disk0,if=virtio \ -drive file="${CDROM}",format=raw,id=cd0,if=none \ -device ide-cd,drive=cd0 \ -drive file="${VIRTIO}",format=raw,id=cd1,if=none \ -device ide-cd,drive=cd1 \ -netdev bridge,id=net0 \ -device virtio-net-pci,netdev=net0 \ -soundhw hda \ -usbdevice host:046d:c52b \ -usbdevice host:03f0:0024

By emulator I meant '/usr/bin/qemu-system-x86_64'. I don't have my system running to test now, but are your variables getting passed through correctly? Have you tried removing all other devices besides the video card and image of your os? Maybe try passing through one of your USB controllers through, that is what I did instead of passing through single USB devices.

Offline

#4243 2015-02-19 16:19:10

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

By emulator I meant '/usr/bin/qemu-system-x86_64'. I don't have my system running to test now, but are your variables getting passed through correctly? Have you tried removing all other devices besides the video card and image of your os? Maybe try passing through one of your USB controllers through, that is what I did instead of passing through single USB devices.

Ah, ok, thanks for the clarification. As far as I can tell the variables are all passed correctly. VirtIO drivers work fine, windows works fine, it's just the GPU that's giving me trouble. The USB devices work fine as well, for what that's worth. At least now that I know the setup works in Fedora 21 with libvirt I can try to narrow down the differences between the two installs.

Offline

#4244 2015-02-19 16:31:48

SchaduwBlink
Member
Registered: 2007-01-07
Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My OVMF q35 booting is going strait to the UEFI command line. It doesn't see my drives. This is part of the libvirt xml:

 <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/mordor/EmmaPC'/> <target dev='sda' bus='sata'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Storage/OS/en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso'/> <target dev='sdb' bus='scsi'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk>

This is with me trying the most basic of settings for the drive. I'm using the latest git source for virt-manager and libvirt.

Hope you can help because I really want to get this set up with q35 because of the better handling of AMD cards.

Thanks.

Denso wrote:
Len wrote:

@mauorrizze Windows 7

It won't work with OVMF without some hacking , you can search for some posts about installing it from FAT32 USB drive .

Also , I think its better to use SCSI rather than AHCI . My drives defined as follows :

-drive file=/VMs/Win_Main.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/VMs/Win8.iso,id=iso_install,if=none \ -device scsi-cd,drive=iso_install \ -cdrom /VMs/virtio.iso \

Good luck !

Offline

#4245 2015-02-19 16:41:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

100% cpu load, nothing on any output.
..deep OVMF problems..

In case of VERY STRANGE behaviour with OVMF and qemu in general, you can turn on the debug console via adding following line to qemu startup options:

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Actually, the dead loop appeared to be a ... CPU dead loop function, so it was intentional, heh. Turns out there was an error message in the debug console:

 ASSERT /var/lib/jenkins/jobs/edk2/workspace/rpmbuild/rpm/BUILD/edk2.git/MdeModulePkg/Core/Dxe/Mem/Pool.c(475): CR has Bad Signature

I'll keep posting some updates on my problem, just in case it might get useful for someone later.

UPD:
Seems like GPU's UEFI oprom is broken. Deep debugging to be done... or not. Maybe i'll just sack the cards and buy something MORE working. Or try poking ASUS. Or something else. But seems like i'm damned with those cards and shall continue digging.

Last edited by Duelist (2015-02-19 17:46:34)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4246 2015-02-19 17:10:10

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SchaduwBlink wrote:

My OVMF q35 booting is going strait to the UEFI command line. It doesn't see my drives. This is part of the libvirt xml:

 <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/mordor/EmmaPC'/> <target dev='sda' bus='sata'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Storage/OS/en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso'/> <target dev='sdb' bus='scsi'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk>

You should be using 'file'  as the disk type for '/dev/mordor/EmmaPC'. This does not look like a valid block device, like /dev/sda2. Also you will want to change the target bus from SATA to IDE, and SCSI to IDE. Here is mine:

 <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/user/virtual_machines/disks/GamingMachineOS.img'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/user/Disk_Images/Windows10_TechnicalPreview_x64_EN-US_9926.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk>

Offline

#4247 2015-02-19 19:33:31

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
anickname wrote:

I was researching that too, if anyone can confirm that the HD7790 has the reset problem, it might be affordable enough that I could pick one up for testing.

Yes, the HD7790 has the reset problem.

Ok, I'll try to keep an eye out for a cheap one on ebay

Was the HD7790 reset problem ever addressed, if so, can somebody link to the resolution?

Offline

#4248 2015-02-19 23:54:00

Mysingen
Member
Registered: 2015-02-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mysingen wrote:

Hello everyone. Thank you all for this guide. I have been able to pass through an Nvidia 970 card to Windows 8.1 and had it working. Briefly... Then the driver detected the virtualization and threw up Code 43.

The problem seems to be that I am not booting with OVMF even though I try to. When I point directly to the TianoCore firmware, the ISO is not detected as bootable:

-drive if=pflash,format=raw,readonly,file=/mnt/vault/qemu/ovmf/20150218/OVMF_CODE-pure-efi.fd -drive if=pflash,format=raw,file=/mnt/vault/qemu/ovmf/20150218/OVMF_VARS-pure-efi.fd

I first suspected that I was missing other component such as ipxe.git, etc, due to dangling symlinks in the 20150218 directory and installed those as well. But it doesn't help. When I point to the entire 20150218 directory (which contains a symlink bios.bin -> OVMF-pure-efi.fd), QEMU boots with SeaBios instead:

-L /mnt/vault/qemu/ovmf/20150218

Any idea what I might be doing wrong? There are guides on the net for converting USB sticks from MBR to GPT so that they can be used as installation media, but I found nothing similar for ISO's. My suspicion is that the ISO somehow does not support UEFI but I don't know how to check this theory. All I see is that TianoCore detects the virtual DVD drive, but not the ISO in it.
/M

Problem solved, partly.

It turns out that there (reasonably) must be a bug in OVMF. I could install by first formatting a USB stick as GPT+FAT32, copy all files from the Win8.1 ISO over to it and then pass the physical USB stick on to OVMF which then detected its file system and let me boot the installation from it.

After that, I was able to install Nvidia 344.11 drivers and it actually worked really well. I restarted the VM a great number of times and played Mass Effect for a few hours. However...

At some point Mass Effect started to crash on startup and I tried to solve the problem by installing newer drivers. Since then I always get Error 43. What baffles me is that I now get Error 43 even after completely starting over with the original working formula, including a fresh copy of the OVMF VARS file. I cannot for my life understand how the driver can see *this* time that it is running under virtualization, while it didn't the first time. Absolutely astonishing.

Offline

#4249 2015-02-20 00:26:55

Mysingen
Member
Registered: 2015-02-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

At some point Mass Effect started to crash on startup and I tried to solve the problem by installing newer drivers. Since then I always get Code 43. What baffles me is that I now get Code 43 even after completely starting over with the original working formula, including a fresh copy of the OVMF VARS file. I cannot for my life understand how the driver can see *this* time that it is running under virtualization, while it didn't the first time. Absolutely astonishing.

What is even weirder is that now the damn thing works again! All I did was to restart the host a couple of times. Posts from normal Windows users on the net give me the idea that Code 43 is a generic cannot-initialize-the-card-for-whatever-reason kind of thing. And that it happens on windows *hosts* under a very wide range of conditions. I haven't actually used Windows at home since 1997. Now I'm rather extra happy to not let it near the bare metal and especially not my host BIOS. Quirky stuff.

Last edited by Mysingen (2015-02-20 02:09:11)

Offline

#4250 2015-02-20 04:30:53

arch742
Member
Registered: 2015-02-20
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am running into a very strange issue. On Intel I 'm passing through  a GTX 460 and using internal graphics for the host, which works perfectly except for when the guest is shut down and started again (asks for rom and throws irqs, then freezes up the host completely). This in itself is not a problem as I can use the suspend/resume trick.

The weird part is if I try to use virtio-net instead of e1000, windows will get to the spinning logo and then dmesg will show irq 34 or 35 for the gtx460, at which point the host will freeze. Also, if the virtio driver is not installed but the device is still emulated, it does not crash.

Has anyone else had a similar issue with virtio-net?

Offline

#4251 2015-02-20 08:57:56

SchaduwBlink
Member
Registered: 2007-01-07
Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You should be using 'file'  as the disk type for '/dev/mordor/EmmaPC'. This does not look like a valid block device, like /dev/sda2. Also you will want to change the target bus from SATA to IDE, and SCSI to IDE. Here is mine:

 <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/user/virtual_machines/disks/GamingMachineOS.img'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/user/Disk_Images/Windows10_TechnicalPreview_x64_EN-US_9926.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk>

It's a ZFS ZVol, so I'm installing to a real hard drive partition, with the extras that come along with ZFS. wink The drives are mapped to /dez/zd*, but I don't know what numbers and I was afraid they'd change on reboot, but I guess they don't.

I'll make the other changes you said and see if that works.

But I heard you can use virtio disks to install and OVMF was supposed to pick them up.

Thanks.

Offline

#4252 2015-02-20 12:23:37

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jcrash wrote:

I've used roms from other manufacturers (equal hardware specs; gpu/#core's/clock)

My process:
1. new win7 VM (base, sytem updates) for every rom file
2. shutdown; make copy of VM on host
3. boot copy; install video drivers (start with most recent, work back)
4. working? rename and file

I threw in the towel because I couldn't take advantage of any power saving features; VM's wouldn't recover from sleep or monitor wake. Force close through host, reboot to a corrupted VM image (every time, unrecoverable), and rollback to step 3.

Careful though: I have a wireless hdmi 460ti that lost it's wireless hdmi capability through this type of tinkering (i think). Can't find the card-specific rom for it so I gambled

Haven't tried with win8


You have been playing with fire. If you have to use romfile, use the one which is for your specific card, even better which you have read yourself from your card. But most of the time you do not need romfile anyway, if the card can actually work with GPU passthrough.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4253 2015-02-20 12:26:41

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SchaduwBlink wrote:

My OVMF q35 booting is going strait to the UEFI command line. It doesn't see my drives. This is part of the libvirt xml:

 <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/mordor/EmmaPC'/> <target dev='sda' bus='sata'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/media/Storage/OS/en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso'/> <target dev='sdb' bus='scsi'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk>

This is with me trying the most basic of settings for the drive. I'm using the latest git source for virt-manager and libvirt.

Hope you can help because I really want to get this set up with q35 because of the better handling of AMD cards.

whut??? AMD cards have no problem with FX440 at all, unless you are unlucky to own a model which is not suited for GPU passthrough at all. In which case Q35 is not going to help you anyway.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4254 2015-02-20 13:14:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Duelist wrote:

100% cpu load, nothing on any output.
..deep OVMF problems..

Seems like GPU's UEFI oprom is broken. Deep debugging to be done... or not. Maybe i'll just sack the cards and buy something MORE working. Or try poking ASUS. Or something else. But seems like i'm damned with those cards and shall continue digging.

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Last edited by Duelist (2015-02-20 15:56:48)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4255 2015-02-20 14:00:32

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

Any help on this anyone? sad

Offline

#4256 2015-02-20 16:57:09

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Duelist wrote:
Duelist wrote:

100% cpu load, nothing on any output.
..deep OVMF problems..

Seems like GPU's UEFI oprom is broken. Deep debugging to be done... or not. Maybe i'll just sack the cards and buy something MORE working. Or try poking ASUS. Or something else. But seems like i'm damned with those cards and shall continue digging.

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Congratulations. You can call your GPU "Frankenstein"


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4257 2015-02-20 17:58:46

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

Congratulations. You can call your GPU "Frankenstein"

I can call my whole system a "Frankenstein" now, my host gpu is tied to the case in front of the CPU cooler, connected via riser card. There's 1-1.5cm of space between steel brace and 1800rpm 120mm fan.

BTW, check out my virtual disk subsystem, it's awesome.
yep, that's GIGABYTES PER SECOND
It's a raw image residing on now-six-years-old-750gb-wd-black-louder-than-a-CNC-mill connected to the VM via virtio-blk-pci.

Astonishing read results. I assume there's some hardcore caching involved.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4258 2015-02-21 00:20:58

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested some more today with Fedora 21 and everything works fine there. When I boot back into the CRUX installation I still can't get things working. I even tried dumping the exact qemu command line from the working libvirt and I see the same results, these same errors:

jaeger wrote:
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x624, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x62c, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x634, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x63c, 0x0, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x484, 0x0, 0x4) failed: Bad address

At this point I'm thinking it has to be a kernel configuration issue but I haven't been able to pinpoint it so far. Searching online hasn't revealed the answer, there aren't many hits for these particular errors. Does anyone at all have an idea where I should be looking? It's getting pretty frustrating at this point, not making any progress.

For reference, my kernel config already contains these:

CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_KVM_VFIO=y

pci-stub.ids and intel_iommu=on are passed on the grub2 command line.

-----

EDIT: After this bit of whining I decided to try Fedora 21's kernel config on my CRUX install. Lo and behold, the GTX 970 works properly in the VM after that! So now I have to figure out exactly which option I'm missing, since the Fedora kernel includes a LOT more stuff than mine.

Now that I know I can make it work in CRUX I feel a lot better, sorry for the downer tone in the post.

Last edited by jaeger (2015-02-21 00:59:10)

Offline

#4259 2015-02-21 02:15:22

Mysingen
Member
Registered: 2015-02-18
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jaeger wrote:

EDIT: After this bit of whining I decided to try Fedora 21's kernel config on my CRUX install. Lo and behold, the GTX 970 works properly in the VM after that! So now I have to figure out exactly which option I'm missing, since the Fedora kernel includes a LOT more stuff than mine.

Now that I know I can make it work in CRUX I feel a lot better, sorry for the downer tone in the post.

Have you diffed the .config files for both kernels? If that doesn't reveal anything of interest, you should check for distribution specific kernel patches. At least you shouldn't have to look anywhere else in the systems since the "adopted" Fedora kernel worked for you in CRUX.

Offline

#4260 2015-02-21 03:41:44

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mysingen wrote:

Have you diffed the .config files for both kernels? If that doesn't reveal anything of interest, you should check for distribution specific kernel patches. At least you shouldn't have to look anywhere else in the systems since the "adopted" Fedora kernel worked for you in CRUX.

That's what I'll be doing next, got distracted by trying to pass a usb gamepad into the VM, which is failing. For some reason I can't get it to play nicely with either my Logitech F710 or Sony DualShock 3. One thing at a time, I guess. :) As far as the kernel config goes, I didn't apply any of the Fedora patches, just used the .config with vanilla 3.18.7 and things work. So yeah, I'll be combing through that to figure out which option is the magic one.

Offline

#4261 2015-02-21 18:31:37

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Thanks a lot for this message! I've managed to make my own rom. Since I liked the latest vbios for my 7750, which is a bit overclocked compared to the original, I decided to replace only the buggy GOP part. And it even works! I can boot systemrescuecd in pure EFI mode. Using DRI freezes the VM, though (not that I care at this moment).

The biggest issue now is that I can't run windows install. OVMF doesn't support UDF filesystem that is used by the install image. It refuses to detect an actual USB pendrive created by rufus. My last resort is a fat32 partition. OVMF detects it, and even starts (I get, "Press any key to boot from CD/DVD..."), just to die the moment I press a key. So I've turned on the debug console. It looks like the problem is here:

Booting EFI USB Device BlockSize : 512 LastBlock : 7FFFFF Valid efi partition table header Valid efi partition table header Valid primary and Valid backup partition table Partition entries read block success Number of partition entries: 128 start check partition entries End check partition entries Index : 0 Start LBA : 800 End LBA : 7FFFDE Partition size: 7FF7DF Start : 100000 End : FFFFBC00 Prepare to Free Pool InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7F307A40 Loading driver at 0x00010000000 EntryPoint=0x000100061C0 cdboot.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7EFD8A18 Error: Image at 00010000000 start failed: No mapping Image Return Status = No mapping PlatformBdsBootFail

Wish I knew what "no mapping" means. Can somebody help me?

P.S. Installing Windows on UEFI is such a pain. :x

Last edited by deniv (2015-02-21 18:40:48)

Offline

#4262 2015-02-21 19:00:45

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm can someone help me, with running my windows 8 rootless, using virsh (libvirt)?
I'm in libvirt group, still it can't attach my vfio-pci devices.
Also I don't know how can I handle mounting /dev/sda /dev/sda1 /dev/sdb /dev/sdb2 /dev/sdb5 /dev/sdc /dev/sdc1 /dev/sdd /dev/sdd1 disks on normal user.

Btw, there is fully working XML (on root, Windows 8 + GTX 970 + SSD as primary + 2 x HDD + mouse, keyboard + some other devices),
But maybe someone have any hints, to tune performance? My CPU is lacking on guest, don't have HT on i5-4690K, and 3 cores on host is not good enough for me : /

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Win81</name> <uuid>df602579-a4f2-41b9-bb2f-6e23d65a24bd</uuid> <title>Windows 8.1</title> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader readonly='yes' type='pflash'>/home/len/win/bios/ovmf_code_x64.bin</loader> <nvram template='/home/len/win/bios/ovmf_vars_x64.bin'>/var/lib/libvirt/qemu/nvram/Win81_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> <pmu state='on'/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>Haswell</model> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <on_lockfailure>poweroff</on_lockfailure> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdb'/> <target dev='vda' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' io='native'/> <source dev='/dev/sdc'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' io='native'/> <source dev='/dev/sdd'/> <target dev='vdc' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='scsi' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <interface type='bridge'> <mac address='33:45:1c:fc:6d:cd'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </interface> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc317'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc068'/> </source> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/random</backend> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </rng> </devices> <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='Haswell,kvm=off'/> </qemu:commandline> </domain>

And some minor case, I have Sound Card, but I can't attach it using vfio-pci, don't know why...

05:00.0 Multimedia audio controller [0401]: Creative Labs CA0106 Soundblaster [1102:0007] Subsystem: Creative Labs SB0570 [SB Audigy SE] [1102:100a] Flags: bus master, medium devsel, latency 32, IRQ 19 I/O ports at c000 [size=32] Capabilities: [dc] Power Management version 2 Kernel driver in use: snd_ca0106 Kernel modules: snd_ca0106

Any clues?




If there is someone whos trying to install Windows 8, there are working qemu args for this XML

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='virtio-scsi-pci,id=scsi'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/home/len/win/win81.iso,id=cd0,if=none'/> <qemu:arg value='-device'/> <qemu:arg value='scsi-cd,drive=cd0'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/home/len/win/virtio-win-0.1-100.iso,id=virtiocd,if=none'/> <qemu:arg value='-device'/> <qemu:arg value='ide-cd,bus=ide.1,drive=virtiocd'/> </qemu:commandline>

Also weird case, I have new versions of qemu and libvirt, but

 <kvm> <hidden state='on'/> </kvm>

is not working for me, so I have use

 <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='Haswell,kvm=off'/> </qemu:commandline>

Oh, and thanks for help!

Last edited by Len (2015-02-21 19:03:48)

Offline

#4263 2015-02-21 22:26:31

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

But maybe someone have any hints, to tune performance? My CPU is lacking on guest, don't have HT on i5-4690K, and 3 cores on host is not good enough for me : /

Hugepages, vcpu pinning, iothreads for virtio-blk, and virtio for network. AFAIK, io='native' is good if you want the best latency, otherwise cache='writeback' is better because it uses the host RAM for caching.
Another option is to install Windows on baremetal and use Linux in a VM. Seriously, you want all cores and 3 full disks in your Windows VM. Again, three full disks! It looks like you're planning to use the Windows VM as your primary machine.

Len wrote:
 <rng model='virtio'> <backend model='random'>/dev/random</backend> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </rng>

AFAIK, there are no virtio-rng drivers for Windows. That device is not used.

Len wrote:

And some minor case, I have Sound Card, but I can't attach it using vfio-pci, don't know why...

05:00.0 Multimedia audio controller [0401]: Creative Labs CA0106 Soundblaster [1102:0007] Subsystem: Creative Labs SB0570 [SB Audigy SE] [1102:100a] Flags: bus master, medium devsel, latency 32, IRQ 19 I/O ports at c000 [size=32] Capabilities: [dc] Power Management version 2 Kernel driver in use: snd_ca0106 Kernel modules: snd_ca0106

Any clues?

Post logs and how you attach it.

Len wrote:

If there is someone whos trying to install Windows 8, there are working qemu args for this XML

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='virtio-scsi-pci,id=scsi'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/home/len/win/win81.iso,id=cd0,if=none'/> <qemu:arg value='-device'/> <qemu:arg value='scsi-cd,drive=cd0'/> <qemu:arg value='-drive'/> <qemu:arg value='file=/home/len/win/virtio-win-0.1-100.iso,id=virtiocd,if=none'/> <qemu:arg value='-device'/> <qemu:arg value='ide-cd,bus=ide.1,drive=virtiocd'/> </qemu:commandline>

One question. Why? Why do you use qemu:arg to add disks? It's so much better to use libvirt defines if you're using libvirt in the first place.

Offline

#4264 2015-02-21 22:51:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For your testing consideration: https://github.com/awilliam/linux-vfio.git tags/vfio-misc-test-20150221

A few vfio-pci things I've been playing with over the last couple days:

- Add an ids module option like pci-stub, hopefully self explanatory
- Add a vgaoptout=1 module option, also enabled by disabling CONFIG_VFIO_PCI_VGA.  This is useful if you're using OVMF because it disables VGA region access to devices bound to vfio-pci and marks them as not participating in VGA arbitration.  If you have a config like me (i915 + nvidia + amd) then i915 opts-out of VGA arbitration on the host (but not really), but DRI2 still gets disabled because there are 2 other VGA arbitration participants.  With this and OVMF, there are none and DRI2 works in the host (seems to fix a screensaver bug in the host failing to render the unlock dialog too).
- Put unused devices bound to vfio-pci in D3 power state.  This seems to save 5-10 watts on my system.  If you have a kill-a-watt or other means to measure power, please provide before/after figures.  Anecdotal fan noise might be interesting too.

This is for testing only, I intend to try to get these upstream, so please don't ask for refreshes of this series for newer/older kernels.  Thanks.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4265 2015-02-21 23:23:37

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for reply so...

deniv wrote:

Hugepages, vcpu pinning, iothreads for virtio-blk, and virtio for network. AFAIK, io='native' is good if you want the best latency, otherwise cache='writeback' is better because it uses the host RAM for caching.

Right now, I don't have much RAM (only 8GB), so I'll stay with native io.
Should I use hugepages, or it's not needed right now?
Disks performance is very good in my opinion, so I'll stay with this configuration.
I have switched my driver for network, thanks for hint!
Thanks for info about rnd.

deniv wrote:

Another option is to install Windows on baremetal and use Linux in a VM. Seriously, you want all cores and 3 full disks in your Windows VM. Again, three full disks! It looks like you're planning to use the Windows VM as your primary machine.

This option is not for me, don't get me wrong.. I'm trying to switch FROM Windows TO Linux+Windows. I was using very long time windows, and linux for servers (only). So those 3 disks are still using NTFS partitions, but it gonna change later.

deniv wrote:

Post logs and how you attach it.

Group 8 is added to /etc/libvirt/qemu.conf.
Stubbed '1102:0007', binded 0000:05:00.0 to vfio-pci and this is log

2015-02-22 00:10:14.074+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name Win81 -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on -drive file=/home/len/win/bios/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/Win81_VARS.fd,if=pflash,format=raw,unit=1 -m 6144 -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 -uuid df602579-a4f2-41b9-bb2f-6e23d65a24bd -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Win81.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x2 -drive file=/dev/sdb,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sdc,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/sdd,if=none,id=drive-virtio-disk2,format=raw,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fc:a7:cd,bus=pci.0,addr=0xb -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,multifunction=on,addr=0x4 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x4.0x1 -device vfio-pci,host=00:1b.0,id=hostdev2,bus=pci.0,addr=0x5 -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7 -device usb-host,hostbus=3,hostaddr=3,id=hostdev4 -device usb-host,hostbus=3,hostaddr=2,id=hostdev5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -cpu Haswell,kvm=off -device virtio-scsi-pci,id=scsi -drive file=/home/len/win/win81.iso,id=cd0,if=none -device scsi-cd,drive=cd0 -drive file=/home/len/win/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd -msg timestamp=on Domain id=3 is tainted: high-privileges Domain id=3 is tainted: custom-argv 2015-02-22T00:10:14.975176Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. 2015-02-22T00:10:14.975208Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: vfio: failed to get group 8 2015-02-22T00:10:14.975219Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: Device initialization failed. 2015-02-22T00:10:14.975232Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized 2015-02-22 00:10:15.096+0000: shutting down
deniv wrote:

One question. Why? Why do you use qemu:arg to add disks? It's so much better to use libvirt defines if you're using libvirt in the first place.

Got many problems witch 'translating' this commands to libvirt XML format, so I just leaved it like that.
And if you guess about -cpu host,kvm=off... I have to, otherwise that hidden=on from libvirt, is not working for me

Oh, and what should I do to run it as normal user, any hints? Right now I have...

virsh start Win81u error: Failed to start domain Win81u error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

Offline

#4266 2015-02-21 23:45:02

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len, what version of libvirt are you using?  You should not need to touch /etc/libvirt/qemu.conf and <qemu:arg> options are nothing but trouble.  They're often a prime cause of not being able to run a VM w/o screwing with permissions.  Requiring <qemu:arg> for kvm=off is a good indication that something is wrong there.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4267 2015-02-21 23:50:58

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Len, what version of libvirt are you using?  You should not need to touch /etc/libvirt/qemu.conf and <qemu:arg> options are nothing but trouble.  They're often a prime cause of not being able to run a VM w/o screwing with permissions.  Requiring <qemu:arg> for kvm=off is a good indication that something is wrong there.

Hello aw,

libvirtd (libvirt) 1.2.12 QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

Ops, I was going through this: https://bbs.archlinux.org/viewtopic.php … 6#p1438076
How should I fix it?

Offline

#4268 2015-02-21 23:56:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:

Len, what version of libvirt are you using?  You should not need to touch /etc/libvirt/qemu.conf and <qemu:arg> options are nothing but trouble.  They're often a prime cause of not being able to run a VM w/o screwing with permissions.  Requiring <qemu:arg> for kvm=off is a good indication that something is wrong there.

Hello aw,

libvirtd (libvirt) 1.2.12 QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

Ops, I was going through this: https://bbs.archlinux.org/viewtopic.php … 6#p1438076
How should I fix it?

Your libvirt version looks ok.  How are you determining that the hidden='true' option isn't working?  You should just be able to look at ps aux | grep qemu.  All of the modifications to qemu.conf in that link are wrong.  Practically all uses of <qemu:args> is also wrong.  Maybe you want to reset to a default qemu.conf, you might have broken something trying to follow those instructions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4269 2015-02-22 00:11:01

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Your libvirt version looks ok.  How are you determining that the hidden='true' option isn't working?  You should just be able to look at ps aux | grep qemu.  All of the modifications to qemu.conf in that link are wrong.  Practically all uses of <qemu:args> is also wrong.  Maybe you want to reset to a default qemu.conf, you might have broken something trying to follow those instructions.

I have removed all qemu args from xml, except kvm=off.

I have run my vm without qemu:arg kvm=off, and I'm getting Error Code 43.
After checking logs for qemu command, one difference is https://www.diffchecker.com/an4xdivl this, and yet it's declared before... still don't know why it's not working.

My old qemu.conf: http://wklej.org/id/1642973/
and libvirtd.conf: http://wklej.org/id/1642974/

Offline

#4270 2015-02-22 00:15:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:

Your libvirt version looks ok.  How are you determining that the hidden='true' option isn't working?  You should just be able to look at ps aux | grep qemu.  All of the modifications to qemu.conf in that link are wrong.  Practically all uses of <qemu:args> is also wrong.  Maybe you want to reset to a default qemu.conf, you might have broken something trying to follow those instructions.

I have removed all qemu args from xml, except kvm=off.

I have run my vm without qemu:arg kvm=off, and I'm getting Error Code 43.
After checking logs for qemu command, one difference is https://www.diffchecker.com/an4xdivl this, and yet it's declared before... still don't know why it's not working.

My old qemu.conf: http://wklej.org/id/1642973/
and libvirtd.conf: http://wklej.org/id/1642974/

From your diff:

-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on ... -cpu Haswell,kvm=off

Tell me again how hidden='on' isn't working...  Of course you're still getting code 43, you still have hyperv extensions enabled.  This is covered in the faq


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4271 2015-02-22 00:23:01

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Tell me again how hidden='on' isn't working...  Of course you're still getting code 43, you still have hyperv extensions enabled.  This is covered in the faq

I guess, my first cpu declaration

-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on

was ignored due to second

-cpu Haswell,kvm=off

Okay, it's working with hidden=on, sorry for problems.


About running libvirt on normal user, I'm doing something wrong?

[len@holo ~]$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

libvirtd.conf: http://wklej.org/id/1642998/
qemu.conf: http://wklej.org/id/1643001/
xml: http://wklej.org/id/1643004/

Last edited by Len (2015-02-22 01:32:58)

Offline

#4272 2015-02-22 02:47:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:

Tell me again how hidden='on' isn't working...  Of course you're still getting code 43, you still have hyperv extensions enabled.  This is covered in the faq

I guess, my first cpu declaration

-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on

was ignored due to second

-cpu Haswell,kvm=off

Okay, it's working with hidden=on, sorry for problems.


About running libvirt on normal user, I'm doing something wrong?

[len@holo ~]$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

libvirtd.conf: http://wklej.org/id/1642998/
qemu.conf: http://wklej.org/id/1643001/
xml: http://wklej.org/id/1643004/

You can find a default qemu.conf here: http://libvirt.org/git/?p=libvirt.git;a … e3;hb=HEAD

Reboot to make sure it takes effect.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4273 2015-02-22 02:57:45

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You can find a default qemu.conf here: http://libvirt.org/git/?p=libvirt.git;a … e3;hb=HEAD

Reboot to make sure it takes effect.

Okay got it, so safely what should I change?

My old qemu.conf

security_driver = "none" security_default_confined = 0 user = "+0" group = "+0" dynamic_ownership = 1 cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1", "/dev/vfio/7", "/dev/vfio/8" ] hugetlbfs_mount = "/dev/hugepages" clear_emulator_capabilities = 0 relaxed_acs_check = 1

Offline

#4274 2015-02-22 03:02:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You should not need to touch /etc/libvirt/qemu.conf


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4275 2015-02-22 03:05:16

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:

You should not need to touch /etc/libvirt/qemu.conf

Without touching user, group:

$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable # virsh start Win81 error: Failed to start domain Win81 error: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied failed to initialize KVM: Permission denied

After changing user and group to root, I can use my vm as root, on user still same error.

Btw, forgot to ask... I have 'The drivers for this device are not installed.' on SCSI Controller in Windows 8, all the time...
PCI\VEN_1000&DEV_0012&SUBSYS_10000000&REV_00
Where can I get this drivers?

Can I catch you somewhere on IRC, aw?

Last edited by Len (2015-02-22 03:32:34)

Offline

#4276 2015-02-22 03:07:45

instantepiphany
Member
Registered: 2013-09-04
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not using arch currently, but this is the best resource I could find with regards to vga passthrough on QEMU.
I have gotten everything working so far, except actual passthrough. When I run my vm, I hear the graphics card that I am passing through spin up like it does on boot, but when I switch to the input from that card, there is no display. What could be causing this?

Edit: And, in the original guide (which is a few years old now) it says you need seabios - but I cant see any commands that link to it or ask QEMU to use it in the OP, does qemu automatically detect that seabios is installed and use it ?

Last edited by instantepiphany (2015-02-22 03:09:03)

Offline

#4277 2015-02-22 03:59:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:
aw wrote:

You should not need to touch /etc/libvirt/qemu.conf

Without touching user, group:

Does that mean you only reverted user/group or does it mean that you're using the referenced qemu.conf as is like I've suggested a couple times now?

$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable # virsh start Win81 error: Failed to start domain Win81 error: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied failed to initialize KVM: Permission denied

After changing user and group to root, I can use my vm as root, on user still same error.

Do you have any errors in /var/log/libvirt/qemu/Win81.log?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4278 2015-02-22 04:02:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

instantepiphany wrote:

I am not using arch currently, but this is the best resource I could find with regards to vga passthrough on QEMU.
I have gotten everything working so far, except actual passthrough. When I run my vm, I hear the graphics card that I am passing through spin up like it does on boot, but when I switch to the input from that card, there is no display. What could be causing this?

FAQ #3/4

Edit: And, in the original guide (which is a few years old now) it says you need seabios - but I cant see any commands that link to it or ask QEMU to use it in the OP, does qemu automatically detect that seabios is installed and use it ?

Don't bother with a -bios option, those instructions are outdated.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4279 2015-02-22 04:10:25

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does that mean you only reverted user/group or does it mean that you're using the referenced qemu.conf as is like I've suggested a couple times now?

I have used qemu.conf from you, and changed user, group (only!) to get it working (at least on root).
Without set user,group I'm getting error which you can see below.

$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable # virsh start Win81 error: Failed to start domain Win81 error: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied failed to initialize KVM: Permission denied
aw wrote:

Do you have any errors in /var/log/libvirt/qemu/Win81.log?


After setting user, group to "root" in qemu.conf >
Starting virsh start Win81 on...
* normal user: (which is giving me PCI device 0000:01:00.0 is not assignable) is not producing anything in /var/log/libvirt/qemu/Win81.log.
* root user: it's working, and no errors in log

2015-02-22 05:14:40.522+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name Win81 -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu host,kvm=off,pmu=on -drive file=/home/len/win/bios/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/Win81_VARS.fd,if=pflash,format=raw,unit=1 -m 6144 -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 -uuid df602579-a4f2-41b9-bb2f-6e23d65a24bd -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Win81.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x3 -drive file=/dev/sdb,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sdc,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/sdd,if=none,id=drive-virtio-disk2,format=raw,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fc:a7:cd,bus=pci.0,addr=0x2 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,multifunction=on,addr=0x7 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x7.0x1 -device vfio-pci,host=00:1b.0,id=hostdev2,bus=pci.0,addr=0x9 -device usb-host,hostbus=3,hostaddr=3,id=hostdev3 -device usb-host,hostbus=3,hostaddr=2,id=hostdev4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on Domain id=3 is tainted: high-privileges Domain id=3 is tainted: host-cpu 2015-02-22T05:14:47.464676Z qemu-system-x86_64: libusb_release_interface: -4 [NO_DEVICE] 2015-02-22T05:14:47.464697Z qemu-system-x86_64: libusb_release_interface: -4 [NO_DEVICE]

Only mouse is not working (sometimes)









____________________________________________

There are some performance test (compared bare and kvm), for people who care:
PerformanceTest 7.0
Info
Summary - 84,8%
CPU - 90,9%
2D - 79,3%
3D - 84,4%
Mem - 82,3%
Disk - 97,3%

3DMark11
Test 1 - 85.5%
Test 2 - 97.7%
Test 3 - 99.5%

My XML

Last edited by Len (2015-02-23 14:55:16)

Offline

#4280 2015-02-22 04:20:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:

Does that mean you only reverted user/group or does it mean that you're using the referenced qemu.conf as is like I've suggested a couple times now?

I have used qemu.conf from you, and changed user, group (only!) to get it working (at least on root).
Without set user,group I'm getting error which you can see below.

$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable # virsh start Win81 error: Failed to start domain Win81 error: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied failed to initialize KVM: Permission denied
aw wrote:

Do you have any errors in /var/log/libvirt/qemu/Win81.log?

Starting on normal user: virsh start Win81 (which is giving me PCI device 0000:01:00.0 is not assignable) is not producing anything in /var/log/libvirt/qemu/Win81.log.

The only thing I can find that prints that "not assignable" error in libvirt deals with PCIe ACS testing.  I'm surprised both that libvirt is doing this test when the hostdev driver is set to vfio and also that it's failing since libvirt's version of ACS testing is far inferior to vfio in the kernel.  You can try setting relaxed_acs_check=1 in qemu.conf, but I also don't see why running as root would make a difference for this test.  Perhaps libvirt can't even scan the PCI capabilities of the devices when not running as root.  What distro are you using?  Something still seems very broken with libvirt.  Have you tried disabling selinux/apparmor?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4281 2015-02-22 04:30:18

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The only thing I can find that prints that "not assignable" error in libvirt deals with PCIe ACS testing.  I'm surprised both that libvirt is doing this test when the hostdev driver is set to vfio and also that it's failing since libvirt's version of ACS testing is far inferior to vfio in the kernel.  You can try setting relaxed_acs_check=1 in qemu.conf, but I also don't see why running as root would make a difference for this test.  Perhaps libvirt can't even scan the PCI capabilities of the devices when not running as root.  What distro are you using?  Something still seems very broken with libvirt.  Have you tried disabling selinux/apparmor?

I'm using archlinux, with nothing more then kdebase, qemu, libvirt and some other stuff for 'how can I get it working'-system.
Didn't installed selinux nor apparmor

Checked relaxed_acs_check=1, didn't helped on normal-user, still same. On root, it's pointless I think, cause vm is working on root, without problems (sometimes mouse, hehe).

Maybe something wrong with kernel? .config

Okay, thanks for your time, gotta take a sleep.. it's already 6am here hmm

Last edited by Len (2015-02-22 16:08:01)

Offline

#4282 2015-02-22 19:05:09

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guys,

I know the main topic here is vga passthrough but this topic is the best resource over the internet when it comes to device passthrough.

I'm trying to make a tv tuner card (TBS6285) works under guest system without success. I can passthrough it successfully, install the drivers and use it for some time, but after some time it just stop get signal.

Neither the host or the guest gives me any hint about the problem looking at dmesg, the only thing I have noticed is the interrupt count at /proc/interrupts stops as soon as the problem occurs.

My setup is the follow:

Hardware:

CPU: Intel Xeon E3-1245V3
MOBO: Supermicro  X10SAE-O
RAM: 16GB ECC
VIDEO1: AMD R7 250
VIDEO2: NVIDIA Geforce GT 610
TV TUNER: TBS 6285

Software:

S.O: Arch linux - 3.18.6-1-ARCH (ACS + VGA Arbiter)
Qemu 2.2.50
Libvirt 1.2.13

Any guidance will be welcome.

Thanks

Offline

#4283 2015-02-22 19:10:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:

Guys,

I know the main topic here is vga passthrough but this topic is the best resource over the internet when it comes to device passthrough.

I'm trying to make a tv tuner card (TBS6285) works under guest system without success. I can passthrough it successfully, install the drivers and use it for some time, but after some time it just stop get signal.

Neither the host or the guest gives me any hint about the problem looking at dmesg, the only thing I have noticed is the interrupt count at /proc/interrupts stops as soon as the problem occurs.

Probably broken INTx masking on the tuner card.  You can try using the vfio-pci module option nointxmask=1 to mask at the APIC instead of the device, but it makes life very difficult since you then require an exclusive interrupt line for the device.  You may try adding a quirk for the device to drivers/pci/quirks.c to isolate it to just that device (see fixups using quirk_broken_intx_masking).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4284 2015-02-22 19:28:25

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bpbastos wrote:

Guys,

I know the main topic here is vga passthrough but this topic is the best resource over the internet when it comes to device passthrough.

I'm trying to make a tv tuner card (TBS6285) works under guest system without success. I can passthrough it successfully, install the drivers and use it for some time, but after some time it just stop get signal.

Neither the host or the guest gives me any hint about the problem looking at dmesg, the only thing I have noticed is the interrupt count at /proc/interrupts stops as soon as the problem occurs.

Probably broken INTx masking on the tuner card.  You can try using the vfio-pci module option nointxmask=1 to mask at the APIC instead of the device, but it makes life very difficult since you then require an exclusive interrupt line for the device.  You may try adding a quirk for the device to drivers/pci/quirks.c to isolate it to just that device (see fixups using quirk_broken_intx_masking).

This tv tuner card supports msi-x interrupt type and this should give an exclusive interrupt line. Could I use what you have suggest with msi interrupts?

Offline

#4285 2015-02-22 19:36:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:
aw wrote:
bpbastos wrote:

Guys,

I know the main topic here is vga passthrough but this topic is the best resource over the internet when it comes to device passthrough.

I'm trying to make a tv tuner card (TBS6285) works under guest system without success. I can passthrough it successfully, install the drivers and use it for some time, but after some time it just stop get signal.

Neither the host or the guest gives me any hint about the problem looking at dmesg, the only thing I have noticed is the interrupt count at /proc/interrupts stops as soon as the problem occurs.

Probably broken INTx masking on the tuner card.  You can try using the vfio-pci module option nointxmask=1 to mask at the APIC instead of the device, but it makes life very difficult since you then require an exclusive interrupt line for the device.  You may try adding a quirk for the device to drivers/pci/quirks.c to isolate it to just that device (see fixups using quirk_broken_intx_masking).

This tv tuner card supports msi-x interrupt type and this should give an exclusive interrupt line. Could I use what you have suggest with msi interrupts?

The nointxmask won't have any effect if MSI-X is actually being used.  Is MSI-X being used by the device when it's assigned?  (host: grep vfio /proc/interrupts)  If not, the better option would be to try to make it use MSI-X.  If it's a Windows guest, see http://vfio.blogspot.com/2014/09/vfio-i … ndows.html  If it is using MSI-X already, then I'm not sure what to suggest.  We've seen cards like Nvidia that actually require a PCI config space write to re-enable MSI and needs a quirk to do so because it writes to a read-only offset.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4286 2015-02-22 19:57:32

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bpbastos wrote:
aw wrote:

Probably broken INTx masking on the tuner card.  You can try using the vfio-pci module option nointxmask=1 to mask at the APIC instead of the device, but it makes life very difficult since you then require an exclusive interrupt line for the device.  You may try adding a quirk for the device to drivers/pci/quirks.c to isolate it to just that device (see fixups using quirk_broken_intx_masking).

This tv tuner card supports msi-x interrupt type and this should give an exclusive interrupt line. Could I use what you have suggest with msi interrupts?

The nointxmask won't have any effect if MSI-X is actually being used.  Is MSI-X being used by the device when it's assigned?  (host: grep vfio /proc/interrupts)  If not, the better option would be to try to make it use MSI-X.  If it's a Windows guest, see http://vfio.blogspot.com/2014/09/vfio-i … ndows.html  If it is using MSI-X already, then I'm not sure what to suggest.  We've seen cards like Nvidia that actually require a PCI config space write to re-enable MSI and needs a quirk to do so because it writes to a read-only offset.

It's a Linux guest (Ubuntu 14.10), this card seems to have a lot of problems with systems where iommu is enabled, see https://github.com/ljalves/linux_media/issues/66

My card is a TBS6285 v21M like the one on the link above and it has the pericom pci bridge which seems to be the root cause of the problems.

I've already tried to passthrough the pericom pci bridge but i think its not possible to passthrough pci bridges.


When the guest starts, it starts with intx:

[root@home-server ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 19 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 7: 10 0 0 0 0 0 0 0 IR-IO-APIC-edge 8: 0 0 0 0 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 2 0 0 0 IR-IO-APIC-fasteoi acpi 12: 2 0 1 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 13 7 0 0 2 6 1 0 IR-IO-APIC 16-fasteoi ehci_hcd:usb5 17: 288441 31873 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi vfio-intx(0000:07:00.0) 18: 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi i801_smbus 23: 28 2 0 0 4 1 0 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb6 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 32: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 40 0 2 1 12 2 2 2 IR-PCI-MSI-edge xhci_hcd 36: 31 2 3 2 13 3 0 1 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 42: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 44: 80885 118083 83 67 1826 908 382 88 IR-PCI-MSI-edge 0000:00:1f.2 45: 1 0 1 2 4 5 8 3 IR-PCI-MSI-edge mei_me 46: 16896 89344 8 12 263 163 10 19 IR-PCI-MSI-edge eno1 52: 4 0 172 0 20 1 154 3 IR-PCI-MSI-edge snd_hda_intel 53: 1 0 2 0 5 0 0 0 IR-PCI-MSI-edge i915 54: 0 0 39 1 15 0 1 4 IR-PCI-MSI-edge snd_hda_intel NMI: 1 1 0 0 10 3 0 0 Non-maskable interrupts LOC: 25166 45620 3162 3118 77738 45167 1567 1477 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 1 1 0 0 10 3 0 0 Performance monitoring interrupts IWI: 2 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 1660 2067 219 179 71690 20139 91 124 Rescheduling interrupts CAL: 988 994 1648 1680 1692 1463 1664 1664 Function call interrupts TLB: 112 85 53 39 658 312 16 33 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 6 6 6 6 6 6 6 6 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 10 MIS: 0

Then it changes to msi-x, I'm forcing msi-x interrupts with saa716x_budget int_type=1 in the guest

[root@home-server ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 19 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 7: 10 0 0 0 0 0 0 0 IR-IO-APIC-edge 8: 0 0 0 0 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 2 0 0 0 IR-IO-APIC-fasteoi acpi 12: 2 0 1 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 13 7 0 0 2 6 1 0 IR-IO-APIC 16-fasteoi ehci_hcd:usb5 17: 288441 31873 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi 18: 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi i801_smbus 23: 28 2 0 0 4 1 0 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb6 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 32: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 40 0 2 1 12 2 2 2 IR-PCI-MSI-edge xhci_hcd 36: 31 2 3 2 13 3 0 1 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 42: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 44: 80885 112767 83 67 1826 908 382 88 IR-PCI-MSI-edge 0000:00:1f.2 45: 1 0 1 2 4 5 8 3 IR-PCI-MSI-edge mei_me 46: 16896 88670 8 12 263 163 10 19 IR-PCI-MSI-edge eno1 52: 4 0 172 0 20 1 154 3 IR-PCI-MSI-edge snd_hda_intel 53: 1 0 2 0 5 0 0 0 IR-PCI-MSI-edge i915 54: 0 0 39 1 15 0 1 4 IR-PCI-MSI-edge snd_hda_intel 55: 556 253 0 0 68 25 0 1 IR-PCI-MSI-edge vfio-msi[0](0000:07:00.0) NMI: 1 1 0 0 9 3 0 0 Non-maskable interrupts LOC: 23680 43632 2955 2927 71642 41880 1478 1388 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 1 1 0 0 9 3 0 0 Performance monitoring interrupts IWI: 2 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 1647 2052 209 172 68764 19165 88 118 Rescheduling interrupts CAL: 964 968 1622 1654 1666 1461 1640 1640 Function call interrupts TLB: 107 82 46 37 622 305 15 30 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 5 5 5 5 5 5 5 5 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 10 MIS: 0

Offline

#4287 2015-02-22 20:08:05

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw, any more ideas about my problem? I'm kinda stuck.

Offline

#4288 2015-02-22 20:46:51

jaeger
Member
Registered: 2015-02-16
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On the subject of passthrough, I've got everything working pretty well with my windows gaming VM now except for one thing: I'm unable to pass my DualShock 3 controller's USB connection properly to the VM. In the device manager I see the device but it's failed with a note saying "Device descriptor request failed".

Is anyone passing one of these to a VM? My intent is to use it via bluetooth but need to have it connected via USB first to get the XInputWrapper service working properly.

Last edited by jaeger (2015-02-22 20:47:10)

Offline

#4289 2015-02-22 21:28:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

@aw, any more ideas about my problem? I'm kinda stuck.

Sorry, I don't use arch, I have no idea what's wrong with your libvirt installation.  Maybe others using arch+libvirt have some ideas.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4290 2015-02-22 22:18:31

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Len wrote:

@aw, any more ideas about my problem? I'm kinda stuck.

Sorry, I don't use arch, I have no idea what's wrong with your libvirt installation.  Maybe others using arch+libvirt have some ideas.

No problem, at least we are trying, btw what dist are you using?

[len@holo ~]$ /usr/sbin/libvirtd 2015-02-22 23:16:51.294+0000: 6911: info : libvirt version: 1.2.13 2015-02-22 23:16:51.294+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:00.0/config': Permission denied 2015-02-22 23:16:51.294+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:01.0/config': Permission denied 2015-02-22 23:16:51.294+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:02.0/config': Permission denied 2015-02-22 23:16:51.294+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:03.0/config': Permission denied 2015-02-22 23:16:51.294+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:14.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:16.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1a.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1b.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1c.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1c.2/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1c.3/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1d.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1f.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1f.2/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:00:1f.3/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:01:00.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:01:00.1/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:03:00.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:04:00.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: warning : virPCIDeviceConfigOpen:307 : Failed to open config space file '/sys/bus/pci/devices/0000:05:00.0/config': Permission denied 2015-02-22 23:16:51.295+0000: 6911: error : virPCIDeviceIsBehindSwitchLackingACS:2299 : internal error: Failed to find parent device for 0000:01:00.0 2015-02-22 23:16:51.300+0000: 6911: warning : virHostdevReAttachUSBDevices:1431 : Unable to find device 000.000 in list of active USB devices 2015-02-22 23:16:51.300+0000: 6911: warning : virHostdevReAttachUSBDevices:1431 : Unable to find device 000.000 in list of active USB devices

Managed to get this log, while running

virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

as normal user

Offline

#4291 2015-02-23 07:51:00

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:
aw wrote:
bpbastos wrote:

This tv tuner card supports msi-x interrupt type and this should give an exclusive interrupt line. Could I use what you have suggest with msi interrupts?

The nointxmask won't have any effect if MSI-X is actually being used.  Is MSI-X being used by the device when it's assigned?  (host: grep vfio /proc/interrupts)  If not, the better option would be to try to make it use MSI-X.  If it's a Windows guest, see http://vfio.blogspot.com/2014/09/vfio-i … ndows.html  If it is using MSI-X already, then I'm not sure what to suggest.  We've seen cards like Nvidia that actually require a PCI config space write to re-enable MSI and needs a quirk to do so because it writes to a read-only offset.

It's a Linux guest (Ubuntu 14.10), this card seems to have a lot of problems with systems where iommu is enabled, see https://github.com/ljalves/linux_media/issues/66

My card is a TBS6285 v21M like the one on the link above and it has the pericom pci bridge which seems to be the root cause of the problems.

I've already tried to passthrough the pericom pci bridge but i think its not possible to passthrough pci bridges.


When the guest starts, it starts with intx:

[root@home-server ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 19 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 7: 10 0 0 0 0 0 0 0 IR-IO-APIC-edge 8: 0 0 0 0 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 2 0 0 0 IR-IO-APIC-fasteoi acpi 12: 2 0 1 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 13 7 0 0 2 6 1 0 IR-IO-APIC 16-fasteoi ehci_hcd:usb5 17: 288441 31873 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi vfio-intx(0000:07:00.0) 18: 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi i801_smbus 23: 28 2 0 0 4 1 0 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb6 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 32: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 40 0 2 1 12 2 2 2 IR-PCI-MSI-edge xhci_hcd 36: 31 2 3 2 13 3 0 1 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 42: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 44: 80885 118083 83 67 1826 908 382 88 IR-PCI-MSI-edge 0000:00:1f.2 45: 1 0 1 2 4 5 8 3 IR-PCI-MSI-edge mei_me 46: 16896 89344 8 12 263 163 10 19 IR-PCI-MSI-edge eno1 52: 4 0 172 0 20 1 154 3 IR-PCI-MSI-edge snd_hda_intel 53: 1 0 2 0 5 0 0 0 IR-PCI-MSI-edge i915 54: 0 0 39 1 15 0 1 4 IR-PCI-MSI-edge snd_hda_intel NMI: 1 1 0 0 10 3 0 0 Non-maskable interrupts LOC: 25166 45620 3162 3118 77738 45167 1567 1477 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 1 1 0 0 10 3 0 0 Performance monitoring interrupts IWI: 2 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 1660 2067 219 179 71690 20139 91 124 Rescheduling interrupts CAL: 988 994 1648 1680 1692 1463 1664 1664 Function call interrupts TLB: 112 85 53 39 658 312 16 33 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 6 6 6 6 6 6 6 6 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 10 MIS: 0

Then it changes to msi-x, I'm forcing msi-x interrupts with saa716x_budget int_type=1 in the guest

[root@home-server ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 19 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 7: 10 0 0 0 0 0 0 0 IR-IO-APIC-edge 8: 0 0 0 0 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 2 0 0 0 IR-IO-APIC-fasteoi acpi 12: 2 0 1 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 13 7 0 0 2 6 1 0 IR-IO-APIC 16-fasteoi ehci_hcd:usb5 17: 288441 31873 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi 18: 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi i801_smbus 23: 28 2 0 0 4 1 0 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb6 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 32: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 40 0 2 1 12 2 2 2 IR-PCI-MSI-edge xhci_hcd 36: 31 2 3 2 13 3 0 1 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 42: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 44: 80885 112767 83 67 1826 908 382 88 IR-PCI-MSI-edge 0000:00:1f.2 45: 1 0 1 2 4 5 8 3 IR-PCI-MSI-edge mei_me 46: 16896 88670 8 12 263 163 10 19 IR-PCI-MSI-edge eno1 52: 4 0 172 0 20 1 154 3 IR-PCI-MSI-edge snd_hda_intel 53: 1 0 2 0 5 0 0 0 IR-PCI-MSI-edge i915 54: 0 0 39 1 15 0 1 4 IR-PCI-MSI-edge snd_hda_intel 55: 556 253 0 0 68 25 0 1 IR-PCI-MSI-edge vfio-msi[0](0000:07:00.0) NMI: 1 1 0 0 9 3 0 0 Non-maskable interrupts LOC: 23680 43632 2955 2927 71642 41880 1478 1388 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 1 1 0 0 9 3 0 0 Performance monitoring interrupts IWI: 2 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 1647 2052 209 172 68764 19165 88 118 Rescheduling interrupts CAL: 964 968 1622 1654 1666 1461 1640 1640 Function call interrupts TLB: 107 82 46 37 622 305 15 30 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 5 5 5 5 5 5 5 5 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 10 MIS: 0

Hi,

I have a TBS 6985 (Quad Satellite card) and after quite a lot of messing about got it running within a VM using the open source driver. I would get heavy pixellation until I disabled all power management in the bios (Disabling within linux didn't work as Turbo modes were still used). Driver info and interrupts below:

modinfo saa716x_budget
filename:       /lib/modules/3.2.0-4-amd64/kernel/drivers/media/pci/saa716x/saa716x_budget.ko
license:        GPL
author:         Manu Abraham
description:    SAA716x Budget driver
alias:          pci:v00001131d00007160sv000013C2sd00003010bc*sc*i*
alias:          pci:v00001131d00007160sv00006985sd00000002bc*sc*i*
alias:          pci:v00001131d00007160sv00006985sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00006984sd00000013bc*sc*i*
alias:          pci:v00001131d00007160sv00006982sd00000002bc*sc*i*
alias:          pci:v00001131d00007160sv00006982sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00006925sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00006922sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00006220sd00000002bc*sc*i*
alias:          pci:v00001131d00007160sv00006285sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00006281sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00006280sd00000011bc*sc*i*
alias:          pci:v00001131d00007160sv00006284sd00000001bc*sc*i*
alias:          pci:v00001131d00007160sv00001AE4sd00000700bc*sc*i*
alias:          pci:v00001131d00007160sv00001894sd00000110bc*sc*i*
alias:          pci:v00001131d00007160sv00001822sd00000047bc*sc*i*
alias:          pci:v00001131d00007160sv00001822sd00000039bc*sc*i*
alias:          pci:v00001131d00007160sv00001822sd00000044bc*sc*i*
depends:        cx24117,saa716x_core,mb86a16,dvb-core,i2c-core,cxd2820r,tas2101
vermagic:       3.2.0-4-amd64 SMP mod_unload modversions
parm:           verbose:verbose startup messages, default is 1 (yes) (int)
parm:           int_type:force Interrupt Handler type: 0=INT-A, 1=MSI, 2=MSI-X. default INT-A mode (int)


cat /proc/interrupts
           CPU0
  0:         59   IO-APIC-edge      timer
  1:          9   IO-APIC-edge      i8042
  2:          0    XT-PIC-XT-PIC    cascade
  8:          0   IO-APIC-edge      rtc0
11:          0    XT-PIC-XT-PIC    virtio0
12:        142   IO-APIC-edge      i8042
40:          1   PCI-MSI-edge      pciehp
41:   27657437   PCI-MSI-edge      eth0-rx-0
42:   22435145   PCI-MSI-edge      eth0-tx-0
43:         12   PCI-MSI-edge      eth0:mbx
44:      43260   PCI-MSI-edge      ahci
45:  102414501   PCI-MSI-edge      SAA716x Core
NMI:          0   Non-maskable interrupts
LOC:   50181720   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:       6555   Machine check polls
ERR:          0
MIS:          0

VM Kernel:
uname -r
3.2.0-4-amd64

Host Kernel:
uname -r
3.18.5-1-ARCH

VM Config:
cat VM/TVHeadend.xml
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>TVHeadend</name>
  <uuid>f702c5eb-83d4-4b50-b909-8da878e08d37</uuid>
  <memory unit='KiB'>1036288</memory>
  <currentMemory unit='KiB'>1036288</currentMemory>
<vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/VM/TVHeadend.img'/>
      <target dev='sda' bus='sata'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/ISO/debian-7.7.0-amd64-netinst.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
<graphics type='vnc' port='5900' />
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=09:11.2'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,id=net10'/>
  </qemu:commandline>
</domain>

Last edited by Krobar (2015-02-23 07:52:27)

Offline

#4292 2015-02-23 10:47:24

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:
aw wrote:

You should not need to touch /etc/libvirt/qemu.conf

Without touching user, group:

$ virsh start Win81 error: Failed to start domain Win81 error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable # virsh start Win81 error: Failed to start domain Win81 error: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied failed to initialize KVM: Permission denied

After changing user and group to root, I can use my vm as root, on user still same error.

Btw, forgot to ask... I have 'The drivers for this device are not installed.' on SCSI Controller in Windows 8, all the time...
PCI\VEN_1000&DEV_0012&SUBSYS_10000000&REV_00
Where can I get this drivers?

Can I catch you somewhere on IRC, aw?


Check this https://bbs.archlinux.org/viewtopic.php?id=69454, it could be relevant to your permission errors


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4293 2015-02-23 10:51:59

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Krobar wrote:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=09:11.2'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,id=net10'/>
  </qemu:commandline>
</domain>

Two points:
1. every time someone forgets to use [code] BBCode, God kills a kitten
2. do not use "<qemu:arg ", you are only missing libvirt capability to manage devices for you


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4294 2015-02-23 12:46:31

bpbastos
Member
Registered: 2014-11-21
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Krobar wrote:
bpbastos wrote:
aw wrote:

The nointxmask won't have any effect if MSI-X is actually being used.  Is MSI-X being used by the device when it's assigned?  (host: grep vfio /proc/interrupts)  If not, the better option would be to try to make it use MSI-X.  If it's a Windows guest, see http://vfio.blogspot.com/2014/09/vfio-i … ndows.html  If it is using MSI-X already, then I'm not sure what to suggest.  We've seen cards like Nvidia that actually require a PCI config space write to re-enable MSI and needs a quirk to do so because it writes to a read-only offset.

It's a Linux guest (Ubuntu 14.10), this card seems to have a lot of problems with systems where iommu is enabled, see https://github.com/ljalves/linux_media/issues/66

My card is a TBS6285 v21M like the one on the link above and it has the pericom pci bridge which seems to be the root cause of the problems.

I've already tried to passthrough the pericom pci bridge but i think its not possible to passthrough pci bridges.


When the guest starts, it starts with intx:

[root@home-server ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 19 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 7: 10 0 0 0 0 0 0 0 IR-IO-APIC-edge 8: 0 0 0 0 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 2 0 0 0 IR-IO-APIC-fasteoi acpi 12: 2 0 1 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 13 7 0 0 2 6 1 0 IR-IO-APIC 16-fasteoi ehci_hcd:usb5 17: 288441 31873 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi vfio-intx(0000:07:00.0) 18: 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi i801_smbus 23: 28 2 0 0 4 1 0 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb6 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 32: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 40 0 2 1 12 2 2 2 IR-PCI-MSI-edge xhci_hcd 36: 31 2 3 2 13 3 0 1 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 42: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 44: 80885 118083 83 67 1826 908 382 88 IR-PCI-MSI-edge 0000:00:1f.2 45: 1 0 1 2 4 5 8 3 IR-PCI-MSI-edge mei_me 46: 16896 89344 8 12 263 163 10 19 IR-PCI-MSI-edge eno1 52: 4 0 172 0 20 1 154 3 IR-PCI-MSI-edge snd_hda_intel 53: 1 0 2 0 5 0 0 0 IR-PCI-MSI-edge i915 54: 0 0 39 1 15 0 1 4 IR-PCI-MSI-edge snd_hda_intel NMI: 1 1 0 0 10 3 0 0 Non-maskable interrupts LOC: 25166 45620 3162 3118 77738 45167 1567 1477 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 1 1 0 0 10 3 0 0 Performance monitoring interrupts IWI: 2 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 1660 2067 219 179 71690 20139 91 124 Rescheduling interrupts CAL: 988 994 1648 1680 1692 1463 1664 1664 Function call interrupts TLB: 112 85 53 39 658 312 16 33 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 6 6 6 6 6 6 6 6 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 10 MIS: 0

Then it changes to msi-x, I'm forcing msi-x interrupts with saa716x_budget int_type=1 in the guest

[root@home-server ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 19 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 3 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 7: 10 0 0 0 0 0 0 0 IR-IO-APIC-edge 8: 0 0 0 0 1 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 1 0 2 0 0 0 IR-IO-APIC-fasteoi acpi 12: 2 0 1 0 0 0 1 0 IR-IO-APIC-edge i8042 16: 13 7 0 0 2 6 1 0 IR-IO-APIC 16-fasteoi ehci_hcd:usb5 17: 288441 31873 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi 18: 0 0 0 0 0 0 0 0 IR-IO-APIC 18-fasteoi i801_smbus 23: 28 2 0 0 4 1 0 0 IR-IO-APIC 23-fasteoi ehci_hcd:usb6 24: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 25: 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 26: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 28: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 29: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 30: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 32: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 40 0 2 1 12 2 2 2 IR-PCI-MSI-edge xhci_hcd 36: 31 2 3 2 13 3 0 1 IR-PCI-MSI-edge xhci_hcd 37: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 38: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 39: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 40: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 41: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 42: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 44: 80885 112767 83 67 1826 908 382 88 IR-PCI-MSI-edge 0000:00:1f.2 45: 1 0 1 2 4 5 8 3 IR-PCI-MSI-edge mei_me 46: 16896 88670 8 12 263 163 10 19 IR-PCI-MSI-edge eno1 52: 4 0 172 0 20 1 154 3 IR-PCI-MSI-edge snd_hda_intel 53: 1 0 2 0 5 0 0 0 IR-PCI-MSI-edge i915 54: 0 0 39 1 15 0 1 4 IR-PCI-MSI-edge snd_hda_intel 55: 556 253 0 0 68 25 0 1 IR-PCI-MSI-edge vfio-msi[0](0000:07:00.0) NMI: 1 1 0 0 9 3 0 0 Non-maskable interrupts LOC: 23680 43632 2955 2927 71642 41880 1478 1388 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 1 1 0 0 9 3 0 0 Performance monitoring interrupts IWI: 2 0 0 0 0 0 0 0 IRQ work interrupts RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries RES: 1647 2052 209 172 68764 19165 88 118 Rescheduling interrupts CAL: 964 968 1622 1654 1666 1461 1640 1640 Function call interrupts TLB: 107 82 46 37 622 305 15 30 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 5 5 5 5 5 5 5 5 Machine check polls HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts ERR: 10 MIS: 0

Hi,

I have a TBS 6985 (Quad Satellite card) and after quite a lot of messing about got it running within a VM using the open source driver. I would get heavy pixellation until I disabled all power management in the bios (Disabling within linux didn't work as Turbo modes were still used). Driver info and interrupts below:

Thanks @Krobar.

I've tried to disable power management settings like you suggested but unfortunately it didn't solve my problem.

Offline

#4295 2015-02-23 14:57:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Thanks a lot for this message! I've managed to make my own rom. Since I liked the latest vbios for my 7750, which is a bit overclocked compared to the original, I decided to replace only the buggy GOP part. And it even works! I can boot systemrescuecd in pure EFI mode. Using DRI freezes the VM, though (not that I care at this moment).
P.S. Installing Windows on UEFI is such a pain. :x

Forget all that crap you have with filesystems.
Check out how it's really done:

-device ahci \ -device virtio-scsi-pci \ -device scsi-cd,drive=install \ -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/mnt/hdd/qemu/windows8.iso,id=install,if=none \ -drive file=/mnt/hdd/qemu/virtio.iso,id=virtiocd,if=none \

I don't recall which(ahci or virtio-scsi-pci) controller controls the scsi-cd, but since we don't need supa-dupa-high-performance and we need this to run the install cd only once - try adding both, qemu will sort it out for you.
Connecting disks that way works, but you must use windows 8 system, because win7 fails to work with GOP, halting in another dead loop somewhere. I might do some additional testing with gigabyte GT610 uefi-compliant card and windows7 much later, just for fun.
Please note that i am specifically using -M pc and not Q35.

Regarding the ROM stuff - the AMD GOP is almost universal. It's just some revision of it that is broken. As you can see, using the older part from HD6450 actually helps. So you may use ANY BIOS for your HD7XXX card, especially if you want to tinker with voltages, power limits and clocks and stuff.
But beware - to do any actual harm tweaking to your GPU you must use windows, since VBE7 is written only for windows, atiwinflash is only for windows and various other stuff is only for windows. I have windowns 7 system installed on bare metal somewhere on the hard drive just for my own comfort.

Right now i have windowns 8 installed and directx9-11 and opengl works fine showing almost zero performance drawbacks, HOWEVER seems like there's still some issues left, as i'm getting half the framerate in some old and buggy(it has bugs under windows too) games like rfactor.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4296 2015-02-23 18:04:08

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nbhs

Add info on first page, might help some people...

How can I run virtual machine on normal user using libvirt:
- start libvirtd process on root user (systemd)
- path to bios files have to be executable, readable
- /etc/libvirt/libvirtd.conf (this one is working for me)

unix_sock_group = "libvirt" unix_sock_ro_perms = "0777" unix_sock_rw_perms = "0770" unix_sock_dir = "/var/run/libvirt" auth_unix_ro = "none" auth_unix_rw = "none"

- /etc/libvirt/qemu.conf
in my case is blank.
- starting vm on normal user: virsh -c qemu:///system start ...

If it was somewhere in this thread, I must missed it.
Thanks aw & laine for help!

Last edited by Len (2015-02-23 18:06:25)

Offline

#4297 2015-02-24 01:17:08

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

[Forget all that crap you have with filesystems.
Check out how it's really done:

I was going to say I'd already tried that, but decided to give it another chance. Guess what? By pure luck I found a working combination: my fat32 partition mounted as scsi-disk AND my udf iso mounted as scsi-cdrom. Only when I connected both disks, installation started. Just to remind what goes wrong in other setups: the fat32 partition alone won't launch because of the "no mapping" error; the udf iso is simply not supported by ovmf (yet). There are patches that add UDF support on ovmf's ml.

Anyway, now I have Windows 8.1 and I'm a happy panda. I've just run the Unigine Valley benchmark. It works. Pure EFI. I haven't noticed any issues. On the contrary, this setup somehow feels snappier than my Win7 VM.

Meanwhile I noticed that the standard video card driver (i.e. the driver used by qxl) in Win8 supports fancy effects like transparency. It isn't the case in Win7. Also, I can have both qxl and the passed through GPU work together. Each device gets its own screen. Nice!

P.S. Don't know why, but a clean install of Win8 takes two times less space than a clean install of Win7 (with updates).

LV VG Attr LSize Pool Origin Data% win7-clean vg0 Vwi-a-tz-- 50.00g thin-pool1 33.44 win81-clean vg0 Vwi-a-tz-- 50.00g thin-pool1 win81 15.89

Offline

#4298 2015-02-24 05:27:55

instantepiphany
Member
Registered: 2013-09-04
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
instantepiphany wrote:

I am not using arch currently, but this is the best resource I could find with regards to vga passthrough on QEMU.
I have gotten everything working so far, except actual passthrough. When I run my vm, I hear the graphics card that I am passing through spin up like it does on boot, but when I switch to the input from that card, there is no display. What could be causing this?

FAQ #3/4

Edit: And, in the original guide (which is a few years old now) it says you need seabios - but I cant see any commands that link to it or ask QEMU to use it in the OP, does qemu automatically detect that seabios is installed and use it ?

Don't bother with a -bios option, those instructions are outdated.

Thanks, I thought it might be outdated.

Looks like the "other" vga arbiter patch is for me, as I am using 2 amd cards rather than intel IGP + dedicated card, and my host graphics don't get corrupted. I haven't applied a patch to the kernel before (though I have applied patches to other code), is there any thing I need be aware of in terms of kernel version? I am currently using 3.18.5, will the patch break anything with my kernel?

Offline

#4299 2015-02-24 06:56:16

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

on this case, i tried to extract the ROM of my own GTX560Ti GPU, and i used it to launch my VMs, as the same that using -nographic and -vga none gives no output, any help on this anyone? sad

also i tried to put the romfile= option in the q35 machine ( becuase 440fx didn't work for me ) and i put my extracted rom from my GPU to the path, it still give me a same result instead.. -vga none still give me qemu command line monitor, adding -vga none and -nographics shows no output, and also if i remove -vga none and qemu command in q35 script, i can boot inside the VM, but it looks like the GPU is not utilized properly.. it shows in lspci in the linux, but the GPU passthrough didn't work, as i test opening youtube and it lags, and also tried unigine heaven benchmark and it still wont work, glx errors.. and also when i boot into windows 8, and adding kvm=off after -cpu argument, it won't let me boot in, soon after i log in into my account, its having an error_exception which keep restarts...

can someone help me on this.. i've been stuck here for 2 months and i am hoping this to work sad

Offline

#4300 2015-02-24 08:33:14

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am having a terrible time getting audio working in a Windows 8.1 VM with qemu...
PCI passthrough is working perfectly with Radeon GPU, motherboard's 2nd ethernet port, and motherboard's 2nd SATA controller being passed to the VM. The last piece I need is working audio! smile

To try to enable sound (with pulseaudio used on host), I am doing qemu.sh:

export QEMU_AUDIO_DRV=pa exec qemu-system-x86_64 \ -enable-kvm \ ...... \ -soundhw hda

,

and the VM fails to launch, with dmesg showing:

[79307.989184] kvm [22292]: vcpu0 unhandled rdmsr: 0xce [79307.993410] kvm [22292]: vcpu0 unhandled rdmsr: 0x637 [79307.993418] kvm [22292]: vcpu0 unhandled rdmsr: 0xe8 [79307.993422] kvm [22292]: vcpu0 unhandled rdmsr: 0xe7 [79307.993426] kvm [22292]: vcpu0 unhandled rdmsr: 0x31 [79307.993430] kvm [22292]: vcpu0 unhandled rdmsr: 0x35 [79307.993434] kvm [22292]: vcpu0 unhandled rdmsr: 0x39 [79307.993448] kvm [22292]: vcpu0 unhandled rdmsr: 0xce [79307.993451] kvm [22292]: vcpu0 unhandled rdmsr: 0x194 [79307.993454] kvm [22292]: vcpu0 unhandled rdmsr: 0x19a [79313.196146] kvm_get_msr_common: 170 callbacks suppressed [79313.196149] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.196156] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.196158] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.196161] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.205095] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.205102] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.205105] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.205107] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.214116] kvm [22292]: vcpu1 unhandled rdmsr: 0x150 [79313.214123] kvm [22292]: vcpu1 unhandled rdmsr: 0x150 [79318.241747] kvm_get_msr_common: 70 callbacks suppressed ... ... ... [79950.317052] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3 [79950.317119] vfio_pci_disable: Failed to reset device 0000:02:00.1 (-22) [79950.328050] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3 [84325.855696] vfio-pci 0000:02:00.0: enabling device (0400 -> 0403) [84325.876928] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 [84325.876931] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0 [84325.878188] vfio-pci 0000:02:00.1: enabling device (0400 -> 0402) [84325.900537] vfio-pci 0000:04:00.0: enabling device (0400 -> 0403) [84326.906061] vfio-pci 0000:07:00.0: enabling device (0400 -> 0403) [84330.202221] kvm: zapping shadow pages for mmio generation wraparound [84338.019479] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (c1f400001000)

Any insight into what I am experience? Thanks!

Offline

#4301 2015-02-24 08:52:16

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:

I am having a terrible time getting audio working in a Windows 8.1 VM with qemu...
PCI passthrough is working perfectly with Radeon GPU, motherboard's 2nd ethernet port, and motherboard's 2nd SATA controller being passed to the VM. The last piece I need is working audio! smile

To try to enable sound (with pulseaudio used on host), I am doing qemu.sh:

export QEMU_AUDIO_DRV=pa exec qemu-system-x86_64 \ -enable-kvm \ ...... \ -soundhw hda

,

and the VM fails to launch, with dmesg showing:

[79307.989184] kvm [22292]: vcpu0 unhandled rdmsr: 0xce [79307.993410] kvm [22292]: vcpu0 unhandled rdmsr: 0x637 [79307.993418] kvm [22292]: vcpu0 unhandled rdmsr: 0xe8 [79307.993422] kvm [22292]: vcpu0 unhandled rdmsr: 0xe7 [79307.993426] kvm [22292]: vcpu0 unhandled rdmsr: 0x31 [79307.993430] kvm [22292]: vcpu0 unhandled rdmsr: 0x35 [79307.993434] kvm [22292]: vcpu0 unhandled rdmsr: 0x39 [79307.993448] kvm [22292]: vcpu0 unhandled rdmsr: 0xce [79307.993451] kvm [22292]: vcpu0 unhandled rdmsr: 0x194 [79307.993454] kvm [22292]: vcpu0 unhandled rdmsr: 0x19a [79313.196146] kvm_get_msr_common: 170 callbacks suppressed [79313.196149] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.196156] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.196158] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.196161] kvm [22292]: vcpu0 unhandled rdmsr: 0x150 [79313.205095] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.205102] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.205105] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.205107] kvm [22292]: vcpu3 unhandled rdmsr: 0x150 [79313.214116] kvm [22292]: vcpu1 unhandled rdmsr: 0x150 [79313.214123] kvm [22292]: vcpu1 unhandled rdmsr: 0x150 [79318.241747] kvm_get_msr_common: 70 callbacks suppressed ... ... ... [79950.317052] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3 [79950.317119] vfio_pci_disable: Failed to reset device 0000:02:00.1 (-22) [79950.328050] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3 [84325.855696] vfio-pci 0000:02:00.0: enabling device (0400 -> 0403) [84325.876928] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 [84325.876931] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0 [84325.878188] vfio-pci 0000:02:00.1: enabling device (0400 -> 0402) [84325.900537] vfio-pci 0000:04:00.0: enabling device (0400 -> 0403) [84326.906061] vfio-pci 0000:07:00.0: enabling device (0400 -> 0403) [84330.202221] kvm: zapping shadow pages for mmio generation wraparound [84338.019479] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (c1f400001000)

Any insight into what I am experience? Thanks!

In my case I'm using the audio from the graphic card instead of the emulated one. And the performance is really good not like the emulated that is quite choppy and delayed (at least in my case).

Offline

#4302 2015-02-24 09:45:56

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The monitor I am using doesn't have any speakers.

Also, the first set of messages (kvm [22292]: vcpu2 unhandled rdmsr: 0x150 etc) may not necessarily be related to the second set of messages (iommu width (39) is not sufficient for the mapped address) and non working audio/VM not working with -soundhw hda option, since I only got the 2nd set of messages after trying again.

EDIT:

Apparently it is an issue with passing through the HDMI audio of the video card that is causing the issue. It seems to work fine if I don't pass the HDMI audio on 02:00.1 and only pass the GPU itself to VM on 02:00.0. I guess it doesn't matter since I won't be using HDMI audio through the video card anyways.

I guess it makes sense from what dmesg says

vfio_pci_disable: Failed to reset device 0000:02:00.1 (-22) vfio-pci 0000:02:00.1: Refused to change power state, currently in D3

It seems to cause an issue in trying to start the guest a 2nd time after shutting down if I don't pass the HDMI along with the GPU, but maybe it's another problem going on...
How would one resolve this, just in case they did want to be able to have the option? Is it a matter of ONLY having the option of passed through audio device or emulated audio through qemu and never both?

Last edited by mutiny (2015-02-24 10:07:55)

Offline

#4303 2015-02-24 10:29:02

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

on this case, i tried to extract the ROM of my own GTX560Ti GPU, and i used it to launch my VMs, as the same that using -nographic and -vga none gives no output, any help on this anyone? sad

also i tried to put the romfile= option in the q35 machine ( becuase 440fx didn't work for me ) and i put my extracted rom from my GPU to the path, it still give me a same result instead.. -vga none still give me qemu command line monitor, adding -vga none and -nographics shows no output, and also if i remove -vga none and qemu command in q35 script, i can boot inside the VM, but it looks like the GPU is not utilized properly.. it shows in lspci in the linux, but the GPU passthrough didn't work, as i test opening youtube and it lags, and also tried unigine heaven benchmark and it still wont work, glx errors.. and also when i boot into windows 8, and adding kvm=off after -cpu argument, it won't let me boot in, soon after i log in into my account, its having an error_exception which keep restarts...

also i rechecked every steps and tried every possible thing on the nhbs's tutorial and went over some user's configuration here on this thread..
after i do some dmesg checking i found this on my dmesg:

[ 0.029557] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 40001000 [ 0.029557] DMAR:[fault reason 06] PTE Read access is not set

here's my lspci :

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4) 00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04) 00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04) 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ab) 02:04.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ab) 02:08.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ab) 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] 03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] 04:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] 05:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) 05:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1) 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) 07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

cat /proc/cmdline/

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLVFIO--vg-root ro quiet i915.enable_hd_vgaarb=1 pcie_acs_override=downstream intel_iommu=on

qemu 2.2.0
linux 3.18-5, i patched linux-mainline 3.18-5 package from this thread

lspci -vnn to graphics cards:

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device [1043:0284] Flags: bus master, fast devsel, latency 0, IRQ 37 Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at f6220000 (64-bit, non-prefetchable) [size=64K] I/O ports at b000 [size=256] Expansion ROM at f6200000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: radeon 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30] Subsystem: ASUSTeK Computer Inc. Device [1043:aa30] Flags: bus master, fast devsel, latency 0, IRQ 36 Memory at f6230000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: snd_hda_intel 04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] Subsystem: ASUSTeK Computer Inc. Device [1043:0284] Flags: bus master, fast devsel, latency 0, IRQ 38 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at f6120000 (64-bit, non-prefetchable) [size=64K] I/O ports at a000 [size=256] Expansion ROM at f6100000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Kernel driver in use: radeon 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601] Flags: fast devsel, IRQ 16 Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M] Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M] Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M] I/O ports at e000 [size=128] Expansion ROM at f6000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100] Virtual Channel Capabilities: [128] Power Budgeting <?> Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci 05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601] Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at f6080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Kernel driver in use: vfio-pci

- i think all steps were good.. suppose to be working.. but im always having black qemu command line on the result after adding -vga none..
- adding -vga none and -nographic together didn't show any result, no display at all..
- extracted ROM from my own NVIDIA card, and putting it romfile= into my scripts.. still not working at all
- tried using q35 and 440fx.. resulting q35 boots the VM, but 440fx only gave me black screen without any output.. ( both of them were not using -vga none and -nographic, of course, since those 2 options wont let me go in )
- booting on windows 8 without installing NVIDIA driver gives me error 43 , after installing, error 43 still shows up. ( q35, since 440fx gives no work for me )
- for windows 8 : added -cpu host,kvm=off, it wont let me boot, had an system_exceptional error and it bootloops. ( q35, since 440fx gives no work for me )

here are my scripts :
q35

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -M q35 -vga std -hda /home/sslab719/VMimages/w8.img -enable-kvm -m 2048 -cpu host \ -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -drive file=/home/sslab719/VMimages/w8.img,id=disk,format=qcow2 \ -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/vbios1.rom \ -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \ -net user,hostfwd=tcp::10022-:22 -net nic #-boot menu=on exit 0

i440fx

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -hda /home/sslab719/VMimages/w8.img -enable-kvm -M pc -nographic -vga none -m 2048 -cpu host,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -device vfio-pci,host=05:00.0,x-vga=on,romfile=/home/sslab719/vbios1.rom \ -device vfio-pci,host=05:00.1 \ -drive file=/home/sslab719/VMimages/w8.img,format=qcow2,media=disk \ -boot menu=on exit 0

also i blacklisted nvidia from the host ( since i wanted to passthrough nvidia to the VMs ) it still doesn't make any change, and i tried removing the blacklist to standard configuration, not blacklisting any fglrx, nvidia, noveau, or radeon, but none worked for me..

additional : i tried testing running on linux ( q35, still no -vga none, no -nographic ), ubuntu 12.04 desktop version, and i had no glxinfo available, also on lspci shown standard graphics card 1234:1111, but i see my lspci showing my graphics card GTX 560 Ti, but it seems that my lspci is nothing, which doesn't give any good result, on additional drivers tab on settings it can automatically searches and installs nvidia drivers, but after install and reboot, it shows "this driver is installed but not activated"

also, i tried installing steam, and trying to type glxinfo, both of them saying that my GLX has errors, which mean its not really passthrough'in my GPU card...

i've been stuck here for 2 months and i am hoping this to work sad ..

EDIT 2 : I tried installing gnome on my host ubuntu server, and run a benchmark, installed Unigine Heaven Benchmark, the benchmark works correctly but it is showing Unknown GPU ( 256 MB ) -> ran on the host..

EDIT 3 : i finally tried using -vga none and -nographics menu, now the black qemu monitor shows up, then when i plug in my monitor to my secondary GPU ( the passthrough GPU ), finally i am able to see my VM using -vga none.. but this vm installs windows 8 with any installed NVIDIA drivers yet,  and also, this only works on the first boot of the VM, lets say i kill the VM and i change the script, then i start my VM again, the output won't show up again in the passthrough'd monitor, the only way to make it show up again is by rebooting the host.. and after it reboots i can only boot the VM again once... if i boot the VM twice, starting from that time it won't show the output again unless i reboot one more time..

any help on this please !! i think im almost there..

Last edited by hotfunction (2015-02-25 07:13:51)

Offline

#4304 2015-02-24 10:55:22

instantepiphany
Member
Registered: 2013-09-04
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
instantepiphany wrote:

I am not using arch currently, but this is the best resource I could find with regards to vga passthrough on QEMU.
I have gotten everything working so far, except actual passthrough. When I run my vm, I hear the graphics card that I am passing through spin up like it does on boot, but when I switch to the input from that card, there is no display. What could be causing this?

FAQ #3/4

Aw,

I am using 3.18.5, and it seems the "other" vga patch was included in 3.17 - so I don't think that is causing my problem. What then do you think is causing my no output issue?

In case it is relevant - this is the only output I get in dmesg when running qemu (2:00.0 is a usb hub)

[20165.012459] vfio-pci 0000:02:00.0: enabling device (0400 -> 0402) [20165.116200] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403) [20165.144582] vfio-pci 0000:01:00.1: enabling device (0400 -> 0402)

Last edited by instantepiphany (2015-02-24 10:58:33)

Offline

#4305 2015-02-24 10:56:21

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

Offline

#4306 2015-02-24 13:43:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Try adding this to see more logs from OVMF or seabios.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4307 2015-02-24 14:07:26

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
JohnyPea wrote:

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Try adding this to see more logs from OVMF or seabios.


Thank you, this trick is very usefull. Did a quick test remotely and the log is quite long, but I dont see anything. Last lines:

Support(): UNDI3.1 found on handle BF29E298 BlockSize : 512 LastBlock : 95FFF PartitionValidMbr: Bad MBR partition size EndingLBA(DEB56E8B) > LastLBA(95FFF) BlockSize : 512 LastBlock : 3FFFF BlockSize : 512 LastBlock : C6F6FFF PartitionValidMbr: Bad MBR partition size EndingLBA(DEB56E8B) > LastLBA(C6F6FFF) Select Item: 0x19 Select Item: 0x2B SetBootOrderFromQemu: setting BootOrder: success VideoFill: Past screen (Y) PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor Memory Previous Current Next Type Pages Pages Pages ====== ======== ======== ======== 0A 00000004 00000023 0000002B 09 00000008 00000006 00000008 00 00000004 00000000 00000004 06 00000024 000000FB 00000139 05 00000030 000000C9 000000FB 03 00000180 00000538 00000686 04 00000F00 00001226 000016AF Booting Windows Boot Manager InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF309E40 Loading driver at 0x00010000000 EntryPoint=0x000100061C0 bootmgfw.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BEF79918

I have noticed on this machine, which uses hugepages, that it probably allocated all of the memory. Therefore my initial thought was incorrect. I see that the boot process starts, on linux guest hangs on "loading initramfs" and on windows only displays logo, before animation. Will do more tests later with the simplest qemu VMs without periperals.

Offline

#4308 2015-02-24 17:55:57

Krobar
Member
Registered: 2014-04-12
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A quick note for those wondering about using the GT620 I also had trouble with this card:
1) Would only work on first boot of the VM (No Reboot and unable tp stop and start after initial start)
2) Kodi under Linux would occasionally stop and then fast-forward video (Strange as the sound over HDMI was fine).

Geforce 960 does not suffer either of these issues and works prefectly within the same VM and config (Although serious overkill for HTPC VM use)

Last edited by Krobar (2015-02-24 17:57:08)

Offline

#4309 2015-02-24 18:02:47

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:
Duelist wrote:
JohnyPea wrote:

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Try adding this to see more logs from OVMF or seabios.


Thank you, this trick is very usefull. Did a quick test remotely and the log is quite long, but I dont see anything. Last lines:
...

So it may be some change in cgroups system which libvirt doesnt hadle well.

2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpu, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply 2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuacct, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply 2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuset, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply

Qemu and kvm modules work. Tested with basic command line invocation and livecd.

Offline

#4310 2015-02-24 18:16:50

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:

So it may be some change in cgroups system which libvirt doesnt hadle well.

2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpu, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply 2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuacct, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply 2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuset, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply

Qemu and kvm modules work. Tested with basic command line invocation and livecd.

Well, sorry for this noise. Thats not it either. Happens exactly the same in 3.19 where the machine works flawlessly. Just keep an eye out for "lapic_timer_advance_ns" and "halt_poll_ns" in the future. I will just shut up now. :-)

Offline

#4311 2015-02-25 10:08:46

Frans-Willem
Member
Registered: 2015-02-25
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got everything booting up on my system with a q35 machine model and qemu, and would like to convert it to a libvirt domain for easier start/stop management. However, I can't seem to find a XML example for Q35 and a PCIe device attached. Has anyone got this working, and if so, could you please share your XML ?

Offline

#4312 2015-02-25 10:17:31

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Frans-Willem wrote:

I've got everything booting up on my system with a q35 machine model and qemu, and would like to convert it to a libvirt domain for easier start/stop management. However, I can't seem to find a XML example for Q35 and a PCIe device attached. Has anyone got this working, and if so, could you please share your XML ?

This is old and maybe not very good config:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>GTX660Ti</name> <uuid>00000000-0000-0000-0000-000000000001</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> </memoryBacking> <vcpu placement='static' cpuset='0-7'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='5'/> <vcpupin vcpu='6' cpuset='6'/> <vcpupin vcpu='7' cpuset='7'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> <hap/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='4096'/> </hyperv> <kvm> <hidden state='on'/> </kvm> <pvspinlock state='on'/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='8' threads='1'/> </cpu> <clock offset='utc'> <timer name='hypervclock' present='yes'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='unsafe' io='threads' discard='unmap'/> <source file='/mnt/home/dzon/VM/WinVfio.qcow2'/> <target dev='vda' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </disk> <disk type='file' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/dev/sda2'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='pci' index='3' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='ide' index='0'/> <interface type='network'> <mac address='52:54:00:12:34:01'/> <source network='virt'/> <model type='virtio'/> <driver name='vhost' txmode='iothread' ioeventfd='on' event_idx='on' queues='4'/> <rom bar='off'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=06:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk0.x-data-plane=off'/> <qemu:arg value='-set'/> <qemu:arg value='device.virtio-disk1.x-data-plane=on'/> <qemu:arg value='-set'/> <qemu:arg value='drive.drive-virtio-disk0.detect-zeroes=unmap'/> </qemu:commandline> </domain>

I have switched to pc-i440fx machine. Libvirt handles it very well and I have generally less problems with it.

There is also a virsh command, try looking into this:

virsh help domxml-from-native virsh domxml-from-native qemu-argv <config_file>

Last edited by JohnyPea (2015-02-25 10:33:02)

Offline

#4313 2015-02-25 10:31:58

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1?

4.0-RC1 here . No issue . It was a completely transparent upgrade .

Last edited by Denso (2015-02-25 10:32:16)

Offline

#4314 2015-02-25 10:36:34

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Did anybody try kernel 4.0-rc1?

4.0-RC1 here . No issue . It was a completely transparent upgrade .

There has to be some specific error in my kernel or libvirt configs. At least I know it should be working. Thanks.

Offline

#4315 2015-02-25 19:16:51

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

People with nvidia cards passthrough. Are you using it with the 340.52 driver and hv-time or using it with the latest driver with no hv-time?
Im unsure if I get better performance with the former or the latter. Anyone with some experience on this?

Offline

#4316 2015-02-25 20:12:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

People with nvidia cards passthrough. Are you using it with the 340.52 driver and hv-time or using it with the latest driver with no hv-time?
Im unsure if I get better performance with the former or the latter. Anyone with some experience on this?

Depends on the card.
If it's an old card like 4XX series - chances are there will be performance drawback just from newer drivers alone. I've witnessed that on a very old card, 250GTS(9800GTX+) - HDMI audio just broke, hardware GPU scaling got broken and there was a performance hit in GPU benchmarks.
If it's 9XX series card - it won't start with drivers-older-than-the-card because.. it's obvious why.

I'd suggest you googling driver benchmarks of some sort. As measured deeply back in time(there was a post from AW with screenshots from borderlands, somewhere last year maybe), performance impact of disabled hyper-v enlightenment and stuff is very little. I'd say, IOMMU and memory management(hugepages and such) issues chop off your performance more.

For example, my AMD system has some weird issues, getting 100%(compared to bare metal setup) GPU benchmarks(furmark, unigine heaven 4.0) BUT lagging in old games like NFS:Prostreet or rFactor. And i'm not using hugepages or cpu pinning or any other cool stuff this thread is suggesting.

P.S. word-parasite of the day: stuff.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4317 2015-02-26 01:34:05

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

People with nvidia cards passthrough. Are you using it with the 340.52 driver and hv-time or using it with the latest driver with no hv-time?
Im unsure if I get better performance with the former or the latter. Anyone with some experience on this?

I did some performance tests on my GTX970, latest drivers with kvm=off.
Check it here https://bbs.archlinux.org/viewtopic.php … 4#p1505174

Offline

#4318 2015-02-26 02:24:21

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

Offline

#4319 2015-02-26 05:53:28

instantepiphany
Member
Registered: 2013-09-04
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My problem with no graphics output was solved by passing my graphics card bios file to qemu.

Performance is almost what I get on my dualboot windows 8 setup - with the exception of guild wars 2. I get equal performance - except for when I turn the camera, which drops the FPS down horribly to about 2, until I stop turning the camera. My vcpus aren't maxed out when i turn the camera, and neither is my virt-ram. What could be causing this? I have tried enabling huge pages, it hasn't helped.

Offline

#4320 2015-02-26 09:20:52

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4321 2015-02-26 09:46:52

thearcherblog
Member
Registered: 2014-10-30
Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Tyrewt wrote:

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.


Mmmmm thanks a lot! I will follow your advice because sometimes when I passthrough my USB for keyboard and mouse... if I shutdown the virtual computer and turn it on again... it give me a segmentation fault. I hope can be solved with that.

Thanks a for that smile

Offline

#4322 2015-02-26 10:28:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

instantepiphany wrote:

My problem with no graphics output was solved by passing my graphics card bios file to qemu.

Performance is almost what I get on my dualboot windows 8 setup - with the exception of guild wars 2. I get equal performance - except for when I turn the camera, which drops the FPS down horribly to about 2, until I stop turning the camera. My vcpus aren't maxed out when i turn the camera, and neither is my virt-ram. What could be causing this? I have tried enabling huge pages, it hasn't helped.

Check your disk subsystem, maybe GW2 tries to load tons of small files when you turn around..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4323 2015-02-26 11:11:37

instantepiphany
Member
Registered: 2013-09-04
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
instantepiphany wrote:

My problem with no graphics output was solved by passing my graphics card bios file to qemu.

Performance is almost what I get on my dualboot windows 8 setup - with the exception of guild wars 2. I get equal performance - except for when I turn the camera, which drops the FPS down horribly to about 2, until I stop turning the camera. My vcpus aren't maxed out when i turn the camera, and neither is my virt-ram. What could be causing this? I have tried enabling huge pages, it hasn't helped.

Check your disk subsystem, maybe GW2 tries to load tons of small files when you turn around..

What should I check? The loading small files possibility is what I thought may be the case, and ended up in figuring out I should be using virtio driver rather than IDE for the disks I am passing through. But the problem still persists. I launch qemu with:

qemu-system-x86_64 -enable-kvm -m 4096 -smp 6,sockets=1,cores=6,threads=1 \ -cpu host \ -machine type=pc,accel=kvm \ -mem-path /libhugetlbfs \ -boot menu=on \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/instantepiphany/qemu/windows7/windows7.img,if=virtio \ -device vfio-pci,host=02:00.0 \ -vga none \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/instantepiphany/qemu/6950.rom \ -device vfio-pci,host=01:00.1 \ -debugcon file:debug.log -global isa-debugcon.iobase=0x402 \ -usb -usbdevice host:046d:c22a -usbdevice host:05e3:0607 \ -drive file=/dev/sdc,cache=none,if=virtio \ -drive file=/dev/sda,cache=none,if=virtio \

Offline

#4324 2015-02-26 11:55:51

bubbacub
Member
Registered: 2015-02-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,
Just hoping for a bit of advice.
I've got passthrough of a r9 270x on a nehalem based quadcore xeon and 3420gp server board with the following startup script on ubuntu 14.04:

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/dev/sdb,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=~/virtio-win-0.1-94.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on \ -usb -usbdevice host:045e:0745 \ -vnc :0 exit 0

Works great as a script and I have a nice host server with a multi TB zfs arrays running two guests, one is a headless linux install running plex server, transmission and filebot for TV and movies and the second is windows 8 instance for virtualised gaming using steam in home streaming over my home network.

My issue that I've been banging my head against a wall is getting the above script to run via libvirt so that I could use all the awesome VM management tools it comes with.

I've tried using domxml-from-native to create an xml file that sucessfully defines through virsh but doesn't let me start a vm.

my xml file based on the above qemu arguments is:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8qemu</name> <uuid>1009128f-983a-45e5-9d88-9295fcf0a692</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='i686' machine='q35'>hvm</type> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>host</model> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <controller type='sata' index='0'/> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='sdl'/> <video> <model type='cirrus' vram='9216' heads='1'/> </video> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> </source> </hostdev> <memballoon model='virtio'/> </devices> <qemu:commandline> <qemu:arg value='qemu-system-x86_64'/> <qemu:arg value='\ -smp'/> <qemu:arg value='4,sockets=1,cores=4,threads=1'/> <qemu:arg value='\ -bios'/> <qemu:arg value='/usr/share/qemu/bios.bin'/> <qemu:arg value='\ -device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='\ -device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='\ -device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='\ -drive'/> <qemu:arg value='file=/dev/sdb,id=disk,format=raw'/> <qemu:arg value='-device'/> <qemu:arg value='ide-hd,bus=ide.0,drive=disk'/> <qemu:arg value='\ -drive'/> <qemu:arg value='file=~/virtio-win-0.1-94.iso,id=isocd'/> <qemu:arg value='-device'/> <qemu:arg value='ide-cd,bus=ide.1,drive=isocd'/> <qemu:arg value='\ -boot'/> <qemu:arg value='menu=on'/> <qemu:arg value='\ -usb'/> <qemu:arg value='\ -vnc'/> <qemu:arg value=':0'/> </qemu:commandline> </domain>

I think this VM doesnt work because the bits of my vm script that sort out the pci passthrough:

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done

arn't been executed by libvirt. Any ideas for how I might get the above snippet of code running as a service continuously?


p.s. If I've broken the local ethos of the forum in any way by posting here, please accept my apologies, correct me and (hopefully help me!)

Offline

#4325 2015-02-26 16:37:11

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bubbacub wrote:

...
arn't been executed by libvirt. Any ideas for how I might get the above snippet of code running as a service continuously?


p.s. If I've broken the local ethos of the forum in any way by posting here, please accept my apologies, correct me and (hopefully help me!)

That domxml-from-native went horrible. I tried it to see why, because my previous attempts were better. I had to edit the result a lot in the end, but I think this is a good approximation to your original machine (I don't exactly know the logic that qemu uses for address assignment so those can differ and cause problems for your guest). Seeing you are using fedoras virtio image, I set the disk type to virtio:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8qemu</name> <uuid>6e68b59c-0977-49d9-82b7-325662a8ee13</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-q35'>hvm</type> <loader type='rom'>/usr/share/qemu/bios.bin</loader> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/dev/sdb'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/dzon/Images/virtio-win-0.1-94.iso'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> </source> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

Regarding the vfio binding, you have to do it only once (if you dont use the device for anything else than passthrough, which I suppouse you dont). So it can be run in startup script (the simplest way for ubuntu: /etc/rc.local), or if compiled in kernel, you can do the binding through kernel command line directly to vfio driver instead of pci-stub. Didnt test this last one, yet.

There is also the possibility to run it through /etc/libvirt/hooks/qemu, but there you need to make tests for what event is occuring (start/stop, which machine). I'm using it for setting host tasks affinity to non-guest cores and reservation/release of hugepages.

Also I would suggest to use pc-i440fx machine type to avoid "qemu:arg" mayhem as much as possibe. Then libvirt does a pretty good job with management. But you would probably need to reinstall the guest.

I am thinking of getting 290x, out of curiosity and also because the effort needed to circumvent nVidias "helpfullness". So please let us know how it works out.

Offline

#4326 2015-02-26 17:15:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:

I am thinking of getting 290x, out of curiosity and also because the effort needed to circumvent nVidias "helpfullness". So please let us know how it works out.

Check it out:
https://docs.google.com/spreadsheet/ccc … _web#gid=0
A whole spreadsheet with graphics and stuff!
There's Eight people with R9 290X.

Here's a COPY with R9 290(x) filtered. You can check these guys' posts by searching their nicknames.
https://docs.google.com/spreadsheet/ccc … zX1E#gid=0

Last edited by Duelist (2015-02-26 17:15:38)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4327 2015-02-26 17:34:50

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
JohnyPea wrote:

I am thinking of getting 290x, out of curiosity and also because the effort needed to circumvent nVidias "helpfullness". So please let us know how it works out.

Check it out:
https://docs.google.com/spreadsheet/ccc … _web#gid=0
A whole spreadsheet with graphics and stuff!
There's Eight people with R9 290X.

Here's a COPY with R9 290(x) filtered. You can check these guys' posts by searching their nicknames.
https://docs.google.com/spreadsheet/ccc … zX1E#gid=0

Nice! Did I realy miss THIS? I think I have seen something simmilar, but this list is excellent. Thank you.

Offline

#4328 2015-02-26 17:58:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:
Duelist wrote:
JohnyPea wrote:

I am thinking of getting 290x, out of curiosity and also because the effort needed to circumvent nVidias "helpfullness". So please let us know how it works out.

Check it out:
https://docs.google.com/spreadsheet/ccc … _web#gid=0
A whole spreadsheet with graphics and stuff!
There's Eight people with R9 290X.

Here's a COPY with R9 290(x) filtered. You can check these guys' posts by searching their nicknames.
https://docs.google.com/spreadsheet/ccc … zX1E#gid=0

Nice! Did I realy miss THIS? I think I have seen something simmilar, but this list is excellent. Thank you.

QEMU IS LOVE
QEMU IS LIFE
THZjGKp.png

Sorry for being silly, i just love VFIO a bit. I've actually red through this all. I should dedicate myself more..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4329 2015-02-26 18:05:36

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

QEMU IS LOVE
QEMU IS LIFE

Sorry for being silly, i just love VFIO a bit. I've actually red through this all. I should dedicate myself more..

LOL, I couldn't agree more!

Offline

#4330 2015-02-26 20:28:18

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs: Perhaps update the first post with this link https://wiki.archlinux.org/index.php/PC … h_via_OVMF and https://docs.google.com/spreadsheet/ccc … _web#gid=0 ?

Last edited by trimm (2015-02-26 20:28:37)

Offline

#4331 2015-02-26 22:01:51

bubbacub
Member
Registered: 2015-02-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi thanks for your help - very grateful.

Tried the xml file you posted (realised my win8 install doesnt have virtio drivers yet and went back to ide) and have been fiddling away all evening.

I put the vfio binding script into startup as you suggested and I can now boot my vm with the following command

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.$ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/dev/sdb,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -boot menu=on \ -usb -usbdevice host:045e:0745 \ -vnc :0

I was getting some apparmor permision denied errors which came from a bug concerning the bios.bin file - the file has to be in a non elevated user directory apparently

Apparmor issues resolved and the VM is now definable.

Also had to change the machine type to pc-q35-1.5 otherwise it refused to be defined.

Here is the final/latest xml file I've got


<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>6e68b59c-0977-49d9-82b7-325662a8ee13</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-q35-1.5'>hvm</type> <loader type='rom'>/home/terminator2/bios.bin</loader> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/sdb'/> <target dev='hda' bus='ide'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> </source> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

but it get this on starting the vm via virsh

error: Failed to start domain win8 error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/10: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 10 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I've edited my qemu.conf and added /dev/vfio/10 and /dev/vfio/vfio to cgroup_device_acl, i've turned clear_emulator_capabilities to 0 and set to run as root. (not sure if this does anything in ubuntu though?).
I've added /dev/vfio/10 and /dev/vfio/vfio to my user and chmod to 666.

bit stuck now :-(

Wondering if the solution is to find a way to get the vfio parts out of the qemu:arg bit of the xml file into something higher up in the xml file?

Last edited by bubbacub (2015-02-26 22:13:11)

Offline

#4332 2015-02-26 22:28:59

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Tyrewt wrote:

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.




So I followed your advice with no luck, maybe I should focus efforts on USB Device Pass-through as it seems get further with Ubuntu.

For the record, the three USB controllers were:

00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 04)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 04)

Added them to my /etc/vfio-pci.cfg

0000:00:14.0
0000:00:1a.0
0000:00:1d.0

Then added the following lines to my qemu config

-usb \
-device vfio-pci,host=00:14.0 \
-device vfio-pci,host=00:1a.0 \
-device vfio-pci,host=00:1d.0 \


Starting the Virtual Machine fails at the first controller with the following error.

qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio: error opening /dev/vfio/4: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio: failed to get group 4
qemu-system-x86_64: -device vfio-pci,host=00:14.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:14.0: Device 'vfio-pci' could not be initialized

Any suggestions?

Last edited by Tyrewt (2015-02-26 23:09:16)

Offline

#4333 2015-02-26 23:18:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great!, are you maintaining this database?

Offline

#4334 2015-02-27 00:08:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Great!, are you maintaining this database?

Ah, No!  Please don't put anything that talks about modifying qemu.conf or using <qemu:arg> options on the first post.  That's WRONG.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4335 2015-02-27 00:22:45

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
nbhs wrote:

Great!, are you maintaining this database?

Ah, No!  Please don't put anything that talks about modifying qemu.conf or using <qemu:arg> options on the first post.  That's WRONG.

And, I think that writing those

<hyperv> ... <relaxed state='off'/> <vapic state='off'/> <spinlocks state='off'/> ... </hyperv> <clock> ... <timer name='hypervclock' present='no'/> ... </clock>

is not needed, when \./ this is on

<kvm> ... <hidden state='on'/> ... </kvm>

Offline

#4336 2015-02-27 00:53:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:
aw wrote:
nbhs wrote:

Great!, are you maintaining this database?

Ah, No!  Please don't put anything that talks about modifying qemu.conf or using <qemu:arg> options on the first post.  That's WRONG.

And, I think that writing those

<hyperv> ... <relaxed state='off'/> <vapic state='off'/> <spinlocks state='off'/> ... </hyperv> <clock> ... <timer name='hypervclock' present='no'/> ... </clock>

is not needed, when \./ this is on

<kvm> ... <hidden state='on'/> ... </kvm>

Hiding KVM and hiding hyper-v extensions are two separate things.  Both are required for Nvidia (though it's usually sufficient to simply not specify hyper-v extensions unless some defaults have changed).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4337 2015-02-27 00:54:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
nbhs wrote:

Great!, are you maintaining this database?

Ah, No!  Please don't put anything that talks about modifying qemu.conf or using <qemu:arg> options on the first post.  That's WRONG.

Ok thanks for the tip aw

Offline

#4338 2015-02-27 07:59:52

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bubbacub wrote:
error: Failed to start domain win8 error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/10: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 10 qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I've edited my qemu.conf and added /dev/vfio/10 and /dev/vfio/vfio to cgroup_device_acl, i've turned clear_emulator_capabilities to 0 and set to run as root. (not sure if this does anything in ubuntu though?).
I've added /dev/vfio/10 and /dev/vfio/vfio to my user and chmod to 666.

bit stuck now :-(

Wondering if the solution is to find a way to get the vfio parts out of the qemu:arg bit of the xml file into something higher up in the xml file?

I went exaxtly the same way, and got it working in the end but got stuck on permissions when I tried to use host pulseaudio by guest directly. Therefore better (or mayby only) solution was OVMF with primary virtual graphics and machine type pc-i440fx.
from notes:

/etc/rc.local chown dzon:libvirtd /dev/vfio/16 /dev/vfio/17 /dev/vfio/7 /dev/vfio/vfio chmod 666 /dev/vfio/16 /dev/vfio/17 /dev/vfio/7 /dev/vfio/vfio chown dzon:libvirtd /dev/kvm /etc/libvirt/qemu.conf cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/23","/dev/vfio/18" ] clear_emulator_capabilities = 0 /etc/apparmor.d/usr.sbin.libvirtd /dev/hugepages rw, /dev/vfio/18 rw, /dev/vfio/23 rw,

So you have the qemu.conf part probably done.
At the begining I too needed to do some fiddling with

/etc/libvirt/qemu.conf user = "root" group = "root"

but the point was not to.
After all this, there was a problem with memory allocation limit. That should be normally managed by libvirt, but at this point it probably went crazy. I cant find the startup script I used for that, but it was messy anyway.
Yes, the solution IS getting the vfio qemu:arg parts out. That didnt work with q35 and passed through graphic as primary, though. I too thought that q35, being based on newer architecture, would be better. Maybe in virtual enviroment doesn't matter that much. 440 is working at least the same if not better and is more compatible.
If you need to have a machine of *1.5, then you have probably a quite old qemu. I am compiling mine, but there are a few good ppas for newer packages (including libvirt and virt-manager) even for older ubuntus. This maybe shouldnt be discussed on Arch forum? ;-)

Offline

#4339 2015-02-27 12:07:49

instantepiphany
Member
Registered: 2013-09-04
Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am unfortunately still having slowdown issues - but only in guild wars 2, and only when I turn the camera. The frames drop to about 10% of normal at best. I tried using the q35 chipset today, and that increase performance significantly in Shadows of Mordor, and even solved a similar issue in SoM. Someone suggested checking my "disk subsystems" but I don't really know what that means. My harddrives are all using the virtio interface, and are entire physical hard drives that I am passing through to the machine. Neither my ram nor cpu in the vm gets maxed out when I turn in GW2 - which makes me think it is possibly something to do with my disks but like I said I don't know how to troubleshoot this.

Offline

#4340 2015-02-27 15:45:51

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am unfortunately still having slowdown issues - but only in guild wars 2, and only when I turn the camera. The frames drop to about 10% of normal at best. I tried using the q35 chipset today, and that increase performance significantly in Shadows of Mordor, and even solved a similar issue in SoM. Someone suggested checking my "disk subsystems" but I don't really know what that means. My harddrives are all using the virtio interface, and are entire physical hard drives that I am passing through to the machine. Neither my ram nor cpu in the vm gets maxed out when I turn in GW2 - which makes me think it is possibly something to do with my disks but like I said I don't know how to troubleshoot this.

Hi .

Games usually don't hammer HDDs except for loading the area or level .

You would get better help if you'd paste your QEMU command line here .

Thanks .

Offline

#4341 2015-02-27 16:38:33

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Trying to get Windows 7 Guest with USB device passthough (A camera), more specifically the MTP Device Driver working. My camera gets detected oddly enough by both Host and Guest when connected, but the driver in Windows fails. Appaently it should be done using USB 2.0, but I am unsure how to connect USB devices to the ehci bus. Can anyone show me?

Currently I have:

-usb \
-usbdevice host:056a:00b9 \
-usbdevice host:2672:000d \
-usbdevice host:046d:c21d \

Would EHCI (USB 2.0) be like this?

-usb \
-usbdevice usb-host,bus=ehci.0,host:056a:00b9 \
-usbdevice usb-host,bus=ehci.0,host:2672:000d \
-usbdevice usb-host,bus=ehci.0,host:046d:c21d \

Last edited by Tyrewt (2015-02-27 18:45:16)

Offline

#4342 2015-02-27 21:52:36

Frans-Willem
Member
Registered: 2015-02-25
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

I've got almost everything working, but sound. Either way I add it, as ich6 (intel-hda), or usb, it all results in System Interrupts taking 30-40% CPU on the Windows 8.1 guest. I've already tried the QEMU command line stuff, but it doesn't seem to help at all.
I've also tried to pass through the integrated audio, but that just results in the driver failing on the windows guest, sadly.

Any ideas ?

Offline

#4343 2015-02-27 22:09:47

bubbacub
Member
Registered: 2015-02-26
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No matter!

Just rebooted and everything is working great under libvirt.

Thanks for all your help, looks like your advice and my flailing around like a blind man with a text editor worked.

this is my final xml file in case it helps anyone else:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>6e68b59c-0977-49d9-82b7-325662a8ee13</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-q35-1.5'>hvm</type> <loader type='rom'>/home/terminator2/bios.bin</loader> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/sdb'/> <target dev='hda' bus='ide'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> </source> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain>

Offline

#4344 2015-02-28 00:21:23

Jodaco
Member
Registered: 2014-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone else with an amd system done this with nbhs' suggested option of disabling npt?  I found that with it disabled it did improve my benchmark scores, doubled in some cases, and games hit higher framerates, but its not consistent.  With npt=0 set cpu load is all over the damn place and framerates in games vary anywhere from 5 fps to 150fps.

I have an AMD 8320, Asus M5A99X EVO PRO r2.0 with the latest bios, 32 GB of ram, Geforce 660 ti for host, and a Geforce 970 on the guest.

Offline

#4345 2015-02-28 03:41:17

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmmm~ two questions,

1. How can I tell my Windows 8.1 that one drive is SSD (for TRIM and other improvements), I'm using VirtIO driver.
2. How can I pass specific USB ports, using usb-devices I know, that I want pass to VM my:

Bus=01 Lev=01 Prnt=01 Port=10 Cnt=01 Bus=01 Lev=01 Prnt=01 Port=11 Cnt=01 Bus=01 Lev=01 Prnt=01 Port=00 Cnt=01

ports (10,11,00 -- front panel, on Z97X-Gaming 5)

Last edited by Len (2015-02-28 03:41:42)

Offline

#4346 2015-02-28 17:40:05

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try passing usb controller with vifio, for me 1a and 1d are front headers/ports on z77.

lspci | grep USB
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
04:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03)

Offline

#4347 2015-02-28 21:12:48

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Try passing usb controller with vifio, for me 1a and 1d are front headers/ports on z77.

lspci | grep USB
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
04:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03)

In my case, I have binded 00:1a.0 and 00:1d.0, but USB3.0 port is not binded. So no luck...

lspci -nn | grep USB 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]

Offline

#4348 2015-02-28 23:04:06

terusus
Member
Registered: 2015-02-27
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,
I've been trying to do VGA passthrough for the last week and I am a bit lost. First, I decided to go with XEN but I could not bind the VGA to a virtual machine. Then I found this guide and I gave KVM a try. The device is successfully passed to the VM but it appears to be dead.

My system:
Ubuntu 14.10 (kernel 3.19 + i915_317 patch + acs override patch + cpu optimizations patch +
  CONFIG_INTEL_IOMMU_DEFAULT_ON=y
  CONFIG_IRQ_FORCED_THREADING_DEFAULT=y
  CONFIG_PREEMPT=y
  CONFIG_VFIO=y
  CONFIG_VFIO_IOMMU_TYPE1=y
  CONFIG_VFIO_PCI=y)

Default QEMU for Ubuntu 14.10 is 2.1.0

i5 4670
ASRock H87 Pro4
nVidia 560Ti

I have added GRUB_CMDLINE_LINUX="i915.enable_hd_vgaarb=1". And I have blacklisted nouveau (from /etc/modprobe.d/blacklist).

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1) 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)

I successfully use the vfio-bind.sh script and prepare the VGA (01:00.0 and 01:00.1). (No errors or warnings.)

I start the vm like this (No errors or warnings.):

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,hv_time,kvm=off \ -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,bus=pcie.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/storage/vm/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/media/.../win7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/media/.../virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

However, with or without the nVidia drivers GPUZ shows a similar picture (in win7 and win8):
GPUZ

There is no signal on the nVidia DVI port.

I have read the VFIO FAQ. I have also tried Alex Williamson's OVMF guide with no luck - win8 freezes when trying to "setup devices". Right now I seriously envy the guy - it looks like everything he touches starts working.

So, please, help me out!

This is my dmesg.

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.19.0-6-kvm (root@UCPC) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #6 SMP PREEMPT Fri Feb 27 12:37:03 EAT 2015 (Ubuntu 3.19.0-6.6-kvm 3.19.0) [ 0.000000] Command line: BOOT_IMAGE=/@/boot/vmlinuz-3.19.0-6-kvm root=UUID=a3929651-e09b-4f1c-afa3-a5516b17eabb ro rootflags=subvol=@ i915.enable_hd_vgaarb=1 quiet splash crashkernel=384M-:128M vt.handoff=7 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a956ffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000a9570000-0x00000000a9576fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000a9577000-0x00000000a9e4afff] usable [ 0.000000] BIOS-e820: [mem 0x00000000a9e4b000-0x00000000aa0e4fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000aa0e5000-0x00000000bc1e3fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bc1e4000-0x00000000bc3e9fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bc3ea000-0x00000000bc426fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bc427000-0x00000000bc4cefff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bc4cf000-0x00000000bcffefff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bcfff000-0x00000000bcffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bf800000-0x00000000cf9fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000042e5fffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./H87 Pro4, BIOS P2.10 07/09/2014 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] AGP: No AGP bridge found [ 0.000000] e820: last_pfn = 0x42e600 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-CFFFF write-protect [ 0.000000] D0000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000000000 mask 7C00000000 write-back [ 0.000000] 1 base 0400000000 mask 7FE0000000 write-back [ 0.000000] 2 base 0420000000 mask 7FF8000000 write-back [ 0.000000] 3 base 0428000000 mask 7FFC000000 write-back [ 0.000000] 4 base 042C000000 mask 7FFE000000 write-back [ 0.000000] 5 base 042E000000 mask 7FFFC00000 write-back [ 0.000000] 6 base 042E400000 mask 7FFFE00000 write-back [ 0.000000] 7 base 00C0000000 mask 7FC0000000 uncachable [ 0.000000] 8 base 00BF800000 mask 7FFF800000 uncachable [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WC UC- UC [ 0.000000] original variable MTRRs [ 0.000000] reg 0, base: 0GB, range: 16GB, type WB [ 0.000000] reg 1, base: 16GB, range: 512MB, type WB [ 0.000000] reg 2, base: 16896MB, range: 128MB, type WB [ 0.000000] reg 3, base: 17024MB, range: 64MB, type WB [ 0.000000] reg 4, base: 17088MB, range: 32MB, type WB [ 0.000000] reg 5, base: 17120MB, range: 4MB, type WB [ 0.000000] reg 6, base: 17124MB, range: 2MB, type WB [ 0.000000] reg 7, base: 3GB, range: 1GB, type UC [ 0.000000] reg 8, base: 3064MB, range: 8MB, type UC [ 0.000000] total RAM covered: 16094M [ 0.000000] gran_size: 64K chunk_size: 64K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 128K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 256K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 512K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 1M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 2M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 64K chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 64K chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 64K chunk_size: 32M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 64K chunk_size: 64M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 64K chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 64K chunk_size: 256M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 64K chunk_size: 512M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 64K chunk_size: 1G num_reg: 10 lose cover RAM: 0G [ 0.000000] *BAD*gran_size: 64K chunk_size: 2G num_reg: 10 lose cover RAM: -1G [ 0.000000] gran_size: 128K chunk_size: 128K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 128K chunk_size: 256K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 128K chunk_size: 512K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 128K chunk_size: 1M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 128K chunk_size: 2M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 128K chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 128K chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 128K chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 128K chunk_size: 32M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 128K chunk_size: 64M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 128K chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 128K chunk_size: 256M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 128K chunk_size: 512M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 128K chunk_size: 1G num_reg: 10 lose cover RAM: 0G [ 0.000000] *BAD*gran_size: 128K chunk_size: 2G num_reg: 10 lose cover RAM: -1G [ 0.000000] gran_size: 256K chunk_size: 256K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 256K chunk_size: 512K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 256K chunk_size: 1M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 256K chunk_size: 2M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 256K chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 256K chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 256K chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 256K chunk_size: 32M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 256K chunk_size: 64M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 256K chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 256K chunk_size: 256M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 256K chunk_size: 512M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 256K chunk_size: 1G num_reg: 10 lose cover RAM: 0G [ 0.000000] *BAD*gran_size: 256K chunk_size: 2G num_reg: 10 lose cover RAM: -1G [ 0.000000] gran_size: 512K chunk_size: 512K num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 512K chunk_size: 1M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 512K chunk_size: 2M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 512K chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 512K chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 512K chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 512K chunk_size: 32M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 512K chunk_size: 64M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 512K chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 512K chunk_size: 256M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 512K chunk_size: 512M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 512K chunk_size: 1G num_reg: 10 lose cover RAM: 0G [ 0.000000] *BAD*gran_size: 512K chunk_size: 2G num_reg: 10 lose cover RAM: -1G [ 0.000000] gran_size: 1M chunk_size: 1M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 1M chunk_size: 2M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 1M chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 1M chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 1M chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 1M chunk_size: 32M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 1M chunk_size: 64M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 1M chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 1M chunk_size: 256M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 1M chunk_size: 512M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 1M chunk_size: 1G num_reg: 10 lose cover RAM: 0G [ 0.000000] *BAD*gran_size: 1M chunk_size: 2G num_reg: 10 lose cover RAM: -1G [ 0.000000] gran_size: 2M chunk_size: 2M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 2M chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 2M chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 2M chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 2M chunk_size: 32M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 2M chunk_size: 64M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 2M chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 2M chunk_size: 256M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 2M chunk_size: 512M num_reg: 10 lose cover RAM: 0G [ 0.000000] gran_size: 2M chunk_size: 1G num_reg: 10 lose cover RAM: 0G [ 0.000000] *BAD*gran_size: 2M chunk_size: 2G num_reg: 10 lose cover RAM: -1G [ 0.000000] gran_size: 4M chunk_size: 4M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 4M chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] *BAD*gran_size: 4M chunk_size: 16M num_reg: 10 lose cover RAM: -10M [ 0.000000] gran_size: 4M chunk_size: 32M num_reg: 10 lose cover RAM: 2M [ 0.000000] gran_size: 4M chunk_size: 64M num_reg: 10 lose cover RAM: 2M [ 0.000000] gran_size: 4M chunk_size: 128M num_reg: 10 lose cover RAM: 2M [ 0.000000] gran_size: 4M chunk_size: 256M num_reg: 10 lose cover RAM: 2M [ 0.000000] gran_size: 4M chunk_size: 512M num_reg: 10 lose cover RAM: 2M [ 0.000000] gran_size: 4M chunk_size: 1G num_reg: 10 lose cover RAM: 2M [ 0.000000] *BAD*gran_size: 4M chunk_size: 2G num_reg: 10 lose cover RAM: -1022M [ 0.000000] gran_size: 8M chunk_size: 8M num_reg: 10 lose cover RAM: 742M [ 0.000000] gran_size: 8M chunk_size: 16M num_reg: 9 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 32M num_reg: 9 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 64M num_reg: 8 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 128M num_reg: 8 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 256M num_reg: 8 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 512M num_reg: 8 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 1G num_reg: 8 lose cover RAM: 6M [ 0.000000] gran_size: 8M chunk_size: 2G num_reg: 9 lose cover RAM: 6M [ 0.000000] gran_size: 16M chunk_size: 16M num_reg: 10 lose cover RAM: 238M [ 0.000000] gran_size: 16M chunk_size: 32M num_reg: 9 lose cover RAM: 14M [ 0.000000] gran_size: 16M chunk_size: 64M num_reg: 8 lose cover RAM: 14M [ 0.000000] gran_size: 16M chunk_size: 128M num_reg: 8 lose cover RAM: 14M [ 0.000000] gran_size: 16M chunk_size: 256M num_reg: 8 lose cover RAM: 14M [ 0.000000] gran_size: 16M chunk_size: 512M num_reg: 8 lose cover RAM: 14M [ 0.000000] gran_size: 16M chunk_size: 1G num_reg: 8 lose cover RAM: 14M [ 0.000000] gran_size: 16M chunk_size: 2G num_reg: 9 lose cover RAM: 14M [ 0.000000] gran_size: 32M chunk_size: 32M num_reg: 10 lose cover RAM: 126M [ 0.000000] gran_size: 32M chunk_size: 64M num_reg: 8 lose cover RAM: 30M [ 0.000000] gran_size: 32M chunk_size: 128M num_reg: 8 lose cover RAM: 30M [ 0.000000] gran_size: 32M chunk_size: 256M num_reg: 8 lose cover RAM: 30M [ 0.000000] gran_size: 32M chunk_size: 512M num_reg: 8 lose cover RAM: 30M [ 0.000000] gran_size: 32M chunk_size: 1G num_reg: 8 lose cover RAM: 30M [ 0.000000] gran_size: 32M chunk_size: 2G num_reg: 9 lose cover RAM: 30M [ 0.000000] gran_size: 64M chunk_size: 64M num_reg: 10 lose cover RAM: 94M [ 0.000000] gran_size: 64M chunk_size: 128M num_reg: 8 lose cover RAM: 94M [ 0.000000] gran_size: 64M chunk_size: 256M num_reg: 8 lose cover RAM: 94M [ 0.000000] gran_size: 64M chunk_size: 512M num_reg: 8 lose cover RAM: 94M [ 0.000000] gran_size: 64M chunk_size: 1G num_reg: 8 lose cover RAM: 94M [ 0.000000] gran_size: 64M chunk_size: 2G num_reg: 9 lose cover RAM: 94M [ 0.000000] gran_size: 128M chunk_size: 128M num_reg: 8 lose cover RAM: 222M [ 0.000000] gran_size: 128M chunk_size: 256M num_reg: 8 lose cover RAM: 222M [ 0.000000] gran_size: 128M chunk_size: 512M num_reg: 8 lose cover RAM: 222M [ 0.000000] gran_size: 128M chunk_size: 1G num_reg: 8 lose cover RAM: 222M [ 0.000000] gran_size: 128M chunk_size: 2G num_reg: 9 lose cover RAM: 222M [ 0.000000] gran_size: 256M chunk_size: 256M num_reg: 6 lose cover RAM: 478M [ 0.000000] gran_size: 256M chunk_size: 512M num_reg: 6 lose cover RAM: 478M [ 0.000000] gran_size: 256M chunk_size: 1G num_reg: 7 lose cover RAM: 478M [ 0.000000] gran_size: 256M chunk_size: 2G num_reg: 8 lose cover RAM: 478M [ 0.000000] gran_size: 512M chunk_size: 512M num_reg: 5 lose cover RAM: 734M [ 0.000000] gran_size: 512M chunk_size: 1G num_reg: 7 lose cover RAM: 734M [ 0.000000] gran_size: 512M chunk_size: 2G num_reg: 8 lose cover RAM: 734M [ 0.000000] gran_size: 1G chunk_size: 1G num_reg: 3 lose cover RAM: 1758M [ 0.000000] gran_size: 1G chunk_size: 2G num_reg: 3 lose cover RAM: 1758M [ 0.000000] gran_size: 2G chunk_size: 2G num_reg: 3 lose cover RAM: 1758M [ 0.000000] mtrr_cleanup: can not find optimal value [ 0.000000] please specify mtrr_gran_size/mtrr_chunk_size [ 0.000000] e820: update [mem 0xbf800000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0xbd000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd7f0-0x000fd7ff] mapped at [ffff8800000fd7f0] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01fd8000, 0x01fd8fff] PGTABLE [ 0.000000] BRK [0x01fd9000, 0x01fd9fff] PGTABLE [ 0.000000] BRK [0x01fda000, 0x01fdafff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x42e400000-0x42e5fffff] [ 0.000000] [mem 0x42e400000-0x42e5fffff] page 2M [ 0.000000] BRK [0x01fdb000, 0x01fdbfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x420000000-0x42e3fffff] [ 0.000000] [mem 0x420000000-0x42e3fffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x400000000-0x41fffffff] [ 0.000000] [mem 0x400000000-0x41fffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0xa956ffff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0xa93fffff] page 2M [ 0.000000] [mem 0xa9400000-0xa956ffff] page 4k [ 0.000000] init_memory_mapping: [mem 0xa9577000-0xa9e4afff] [ 0.000000] [mem 0xa9577000-0xa95fffff] page 4k [ 0.000000] [mem 0xa9600000-0xa9dfffff] page 2M [ 0.000000] [mem 0xa9e00000-0xa9e4afff] page 4k [ 0.000000] BRK [0x01fdc000, 0x01fdcfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0xaa0e5000-0xbc1e3fff] [ 0.000000] [mem 0xaa0e5000-0xaa1fffff] page 4k [ 0.000000] [mem 0xaa200000-0xbbffffff] page 2M [ 0.000000] [mem 0xbc000000-0xbc1e3fff] page 4k [ 0.000000] BRK [0x01fdd000, 0x01fddfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0xbc3ea000-0xbc426fff] [ 0.000000] [mem 0xbc3ea000-0xbc426fff] page 4k [ 0.000000] init_memory_mapping: [mem 0xbcfff000-0xbcffffff] [ 0.000000] [mem 0xbcfff000-0xbcffffff] page 4k [ 0.000000] init_memory_mapping: [mem 0x100000000-0x3ffffffff] [ 0.000000] [mem 0x100000000-0x3ffffffff] page 1G [ 0.000000] RAMDISK: [mem 0x357ee000-0x36beefff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F0490 000024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 0x00000000BC4AB088 00008C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 0x00000000BC4B57E8 00010C (v05 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: DSDT 0x00000000BC4AB1A8 00A63B (v02 ALASKA A M I 00000210 INTL 20091112) [ 0.000000] ACPI: FACS 0x00000000BC4CD080 000040 [ 0.000000] ACPI: APIC 0x00000000BC4B58F8 000072 (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 0x00000000BC4B5970 000044 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: SSDT 0x00000000BC4B59B8 000539 (v01 PmRef Cpu0Ist 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 0x00000000BC4B5EF8 000AD8 (v01 PmRef CpuPm 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 0x00000000BC4B69D0 0001C7 (v01 PmRef LakeTiny 00003000 INTL 20051117) [ 0.000000] ACPI: MCFG 0x00000000BC4B6B98 00003C (v01 ALASKA A M I 01072009 MSFT 00000097) [ 0.000000] ACPI: HPET 0x00000000BC4B6BD8 000038 (v01 ALASKA A M I 01072009 AMI. 00000005) [ 0.000000] ACPI: SSDT 0x00000000BC4B6C10 00036D (v01 SataRe SataTabl 00001000 INTL 20091112) [ 0.000000] ACPI: SSDT 0x00000000BC4B6F80 003493 (v01 SaSsdt SaSsdt 00003000 INTL 20091112) [ 0.000000] ACPI: AAFT 0x00000000BC4BA418 0002BB (v01 ALASKA OEMAAFT 01072009 MSFT 00000097) [ 0.000000] ACPI: ASF! 0x00000000BC4BA6D8 0000A5 (v32 INTEL HCG 00000001 TFSM 000F4240) [ 0.000000] ACPI: DMAR 0x00000000BC4BA780 0000B8 (v01 INTEL HSW 00000001 INTL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000042e5fffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x42e5f5000-0x42e5f9fff] [ 0.000000] Reserving 128MB of memory at 720MB for crashkernel (System RAM: 16037MB) [ 0.000000] [ffffea0000000000-ffffea0010bfffff] PMD -> [ffff88041e000000-ffff88042dbfffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x42e5fffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0xa956ffff] [ 0.000000] node 0: [mem 0xa9577000-0xa9e4afff] [ 0.000000] node 0: [mem 0xaa0e5000-0xbc1e3fff] [ 0.000000] node 0: [mem 0xbc3ea000-0xbc426fff] [ 0.000000] node 0: [mem 0xbcfff000-0xbcffffff] [ 0.000000] node 0: [mem 0x100000000-0x42e5fffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x42e5fffff] [ 0.000000] On node 0 totalpages: 4105501 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 11967 pages used for memmap [ 0.000000] DMA32 zone: 765825 pages, LIFO batch:31 [ 0.000000] Normal zone: 52120 pages used for memmap [ 0.000000] Normal zone: 3335680 pages, LIFO batch:31 [ 0.000000] Reserving Intel graphics stolen memory at 0xbfa00000-0xcf9fffff [ 0.000000] ACPI: PM-Timer IO Port: 0x1808 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xa9570000-0xa9576fff] [ 0.000000] PM: Registered nosave memory: [mem 0xa9e4b000-0xaa0e4fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbc1e4000-0xbc3e9fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbc427000-0xbc4cefff] [ 0.000000] PM: Registered nosave memory: [mem 0xbc4cf000-0xbcffefff] [ 0.000000] PM: Registered nosave memory: [mem 0xbd000000-0xbf7fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbf800000-0xcf9fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xcfa00000-0xf7ffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfecfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed00000-0xfed03fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed04000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff] [ 0.000000] e820: [mem 0xcfa00000-0xf7ffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 31 pages/cpu @ffff88042e200000 s87616 r8192 d31168 u524288 [ 0.000000] pcpu-alloc: s87616 r8192 d31168 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 4041329 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/@/boot/vmlinuz-3.19.0-6-kvm root=UUID=a3929651-e09b-4f1c-afa3-a5516b17eabb ro rootflags=subvol=@ i915.enable_hd_vgaarb=1 quiet splash crashkernel=384M-:128M vt.handoff=7 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] Calgary: detecting Calgary via BIOS EBDA area [ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing! [ 0.000000] Memory: 15929224K/16422004K available (8010K kernel code, 1219K rwdata, 3752K rodata, 1376K init, 1300K bss, 492780K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS:16640 nr_irqs:456 16 [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] vt handoff: transparent VT on vt#7 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 3399.180 MHz processor [ 0.000020] Calibrating delay loop (skipped), value calculated using timer frequency.. 6798.36 BogoMIPS (lpj=3399180) [ 0.000022] pid_max: default: 32768 minimum: 301 [ 0.000025] ACPI: Core revision 20141107 [ 0.006731] ACPI: All ACPI Tables successfully acquired [ 0.014007] Security Framework initialized [ 0.014017] AppArmor: AppArmor initialized [ 0.014018] Yama: becoming mindful. [ 0.014664] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.016565] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.017396] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.017407] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.017573] Initializing cgroup subsys memory [ 0.017576] Initializing cgroup subsys devices [ 0.017578] Initializing cgroup subsys freezer [ 0.017579] Initializing cgroup subsys net_cls [ 0.017581] Initializing cgroup subsys blkio [ 0.017582] Initializing cgroup subsys perf_event [ 0.017583] Initializing cgroup subsys net_prio [ 0.017585] Initializing cgroup subsys hugetlb [ 0.017601] CPU: Physical Processor ID: 0 [ 0.017601] CPU: Processor Core ID: 0 [ 0.018351] mce: CPU supports 9 MCE banks [ 0.018361] CPU0: Thermal monitoring enabled (TM1) [ 0.018370] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024 Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4 [ 0.018443] Freeing SMP alternatives memory: 28K (ffffffff81e8a000 - ffffffff81e91000) [ 0.018922] ftrace: allocating 30017 entries in 118 pages [ 0.027558] dmar: Host address width 39 [ 0.027559] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.027566] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.027567] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.027569] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.027570] dmar: RMRR base: 0x000000bc371000 end: 0x000000bc37dfff [ 0.027571] dmar: RMRR base: 0x000000bf800000 end: 0x000000cf9fffff [ 0.027635] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.027636] HPET id 0 under DRHD base 0xfed91000 [ 0.027637] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.027743] Enabled IRQ remapping in x2apic mode [ 0.027744] Enabling x2apic [ 0.027744] Enabled x2apic [ 0.027748] Switched APIC routing to cluster x2apic. [ 0.028163] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.038156] smpboot: CPU0: Intel(R) Core(TM) i5-4670 CPU @ 3.40GHz (fam: 06, model: 3c, stepping: 03) [ 0.038161] TSC deadline timer enabled [ 0.038177] Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. [ 0.038190] ... version: 3 [ 0.038191] ... bit width: 48 [ 0.038192] ... generic registers: 8 [ 0.038192] ... value mask: 0000ffffffffffff [ 0.038193] ... max period: 0000ffffffffffff [ 0.038193] ... fixed-purpose events: 3 [ 0.038194] ... event mask: 00000007000000ff [ 0.046264] x86: Booting SMP configuration: [ 0.046266] .... node #0, CPUs: #1 [ 0.060109] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.062145] #2 #3 [ 0.092026] x86: Booted up 1 node, 4 CPUs [ 0.092028] smpboot: Total of 4 processors activated (27193.44 BogoMIPS) [ 0.094493] devtmpfs: initialized [ 0.096390] evm: security.selinux [ 0.096391] evm: security.SMACK64 [ 0.096391] evm: security.SMACK64EXEC [ 0.096392] evm: security.SMACK64TRANSMUTE [ 0.096392] evm: security.SMACK64MMAP [ 0.096393] evm: security.ima [ 0.096393] evm: security.capability [ 0.096434] PM: Registering ACPI NVS region [mem 0xa9570000-0xa9576fff] (28672 bytes) [ 0.096435] PM: Registering ACPI NVS region [mem 0xbc427000-0xbc4cefff] (688128 bytes) [ 0.096534] pinctrl core: initialized pinctrl subsystem [ 0.096604] RTC time: 19:46:49, date: 02/28/15 [ 0.096673] NET: Registered protocol family 16 [ 0.101393] cpuidle: using governor ladder [ 0.104397] cpuidle: using governor menu [ 0.104440] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.104441] ACPI: bus type PCI registered [ 0.104442] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.104489] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000) [ 0.104490] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820 [ 0.104543] PCI: Using configuration type 1 for base access [ 0.108644] ACPI: Added _OSI(Module Device) [ 0.108645] ACPI: Added _OSI(Processor Device) [ 0.108646] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.108647] ACPI: Added _OSI(Processor Aggregator Device) [ 0.110935] ACPI: Executed 1 blocks of module-level executable AML code [ 0.112366] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored [ 0.112665] ACPI: Dynamic OEM Table Load: [ 0.112669] ACPI: SSDT 0xFFFF88041BFE4800 0003D3 (v01 PmRef Cpu0Cst 00003001 INTL 20051117) [ 0.113089] ACPI: Dynamic OEM Table Load: [ 0.113092] ACPI: SSDT 0xFFFF88041BF4A800 0005AA (v01 PmRef ApIst 00003000 INTL 20051117) [ 0.113543] ACPI: Dynamic OEM Table Load: [ 0.113546] ACPI: SSDT 0xFFFF88041BFDCC00 000119 (v01 PmRef ApCst 00003000 INTL 20051117) [ 0.114191] ACPI: Interpreter enabled [ 0.114196] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20141107/hwxface-580) [ 0.114199] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20141107/hwxface-580) [ 0.114210] ACPI: (supports S0 S3 S4 S5) [ 0.114211] ACPI: Using IOAPIC for interrupt routing [ 0.114245] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.119480] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e]) [ 0.119484] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.119613] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug PME] [ 0.119693] acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability] [ 0.120027] PCI host bridge to bus 0000:00 [ 0.120028] pci_bus 0000:00: root bus resource [bus 00-3e] [ 0.120029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] [ 0.120030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.120031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.120032] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff] [ 0.120033] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] [ 0.120034] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] [ 0.120035] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff] [ 0.120036] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff] [ 0.120037] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff] [ 0.120038] pci_bus 0000:00: root bus resource [mem 0xcfa00000-0xfeafffff] [ 0.120043] pci 0000:00:00.0: [8086:0c00] type 00 class 0x060000 [ 0.120100] pci 0000:00:01.0: [8086:0c01] type 01 class 0x060400 [ 0.120126] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 0.120174] pci 0000:00:02.0: [8086:0412] type 00 class 0x030000 [ 0.120182] pci 0000:00:02.0: reg 0x10: [mem 0xee400000-0xee7fffff 64bit] [ 0.120186] pci 0000:00:02.0: reg 0x18: [mem 0xd0000000-0xdfffffff 64bit pref] [ 0.120189] pci 0000:00:02.0: reg 0x20: [io 0xf000-0xf03f] [ 0.120241] pci 0000:00:03.0: [8086:0c0c] type 00 class 0x040300 [ 0.120247] pci 0000:00:03.0: reg 0x10: [mem 0xee834000-0xee837fff 64bit] [ 0.120322] pci 0000:00:14.0: [8086:8c31] type 00 class 0x0c0330 [ 0.120337] pci 0000:00:14.0: reg 0x10: [mem 0xee820000-0xee82ffff 64bit] [ 0.120388] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 0.120418] pci 0000:00:14.0: System wakeup disabled by ACPI [ 0.120442] pci 0000:00:16.0: [8086:8c3a] type 00 class 0x078000 [ 0.120458] pci 0000:00:16.0: reg 0x10: [mem 0xee840000-0xee84000f 64bit] [ 0.120513] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 0.120566] pci 0000:00:19.0: [8086:153b] type 00 class 0x020000 [ 0.120580] pci 0000:00:19.0: reg 0x10: [mem 0xee800000-0xee81ffff] [ 0.120586] pci 0000:00:19.0: reg 0x14: [mem 0xee83d000-0xee83dfff] [ 0.120593] pci 0000:00:19.0: reg 0x18: [io 0xf080-0xf09f] [ 0.120644] pci 0000:00:19.0: PME# supported from D0 D3hot D3cold [ 0.120670] pci 0000:00:19.0: System wakeup disabled by ACPI [ 0.120697] pci 0000:00:1a.0: [8086:8c2d] type 00 class 0x0c0320 [ 0.120713] pci 0000:00:1a.0: reg 0x10: [mem 0xee83c000-0xee83c3ff] [ 0.120789] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 0.120826] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 0.120851] pci 0000:00:1b.0: [8086:8c20] type 00 class 0x040300 [ 0.120864] pci 0000:00:1b.0: reg 0x10: [mem 0xee830000-0xee833fff 64bit] [ 0.120923] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold [ 0.120950] pci 0000:00:1b.0: System wakeup disabled by ACPI [ 0.120972] pci 0000:00:1c.0: [8086:8c10] type 01 class 0x060400 [ 0.121029] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 0.121045] pci 0000:00:1c.0: Enabling MPC IRBNCE [ 0.121047] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled [ 0.121066] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 0.121092] pci 0000:00:1c.6: [8086:244e] type 01 class 0x060401 [ 0.121149] pci 0000:00:1c.6: PME# supported from D0 D3hot D3cold [ 0.121178] pci 0000:00:1c.6: System wakeup disabled by ACPI [ 0.121206] pci 0000:00:1d.0: [8086:8c26] type 00 class 0x0c0320 [ 0.121223] pci 0000:00:1d.0: reg 0x10: [mem 0xee83b000-0xee83b3ff] [ 0.121300] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 0.121335] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 0.121360] pci 0000:00:1f.0: [8086:8c4a] type 00 class 0x060100 [ 0.121496] pci 0000:00:1f.2: [8086:8c02] type 00 class 0x010601 [ 0.121509] pci 0000:00:1f.2: reg 0x10: [io 0xf0d0-0xf0d7] [ 0.121515] pci 0000:00:1f.2: reg 0x14: [io 0xf0c0-0xf0c3] [ 0.121520] pci 0000:00:1f.2: reg 0x18: [io 0xf0b0-0xf0b7] [ 0.121526] pci 0000:00:1f.2: reg 0x1c: [io 0xf0a0-0xf0a3] [ 0.121532] pci 0000:00:1f.2: reg 0x20: [io 0xf060-0xf07f] [ 0.121538] pci 0000:00:1f.2: reg 0x24: [mem 0xee83a000-0xee83a7ff] [ 0.121569] pci 0000:00:1f.2: PME# supported from D3hot [ 0.121611] pci 0000:00:1f.3: [8086:8c22] type 00 class 0x0c0500 [ 0.121623] pci 0000:00:1f.3: reg 0x10: [mem 0xee839000-0xee8390ff 64bit] [ 0.121639] pci 0000:00:1f.3: reg 0x20: [io 0xf040-0xf05f] [ 0.121722] pci 0000:01:00.0: [10de:1200] type 00 class 0x030000 [ 0.121730] pci 0000:01:00.0: reg 0x10: [mem 0xec000000-0xedffffff] [ 0.121737] pci 0000:01:00.0: reg 0x14: [mem 0xe0000000-0xe7ffffff 64bit pref] [ 0.121744] pci 0000:01:00.0: reg 0x1c: [mem 0xe8000000-0xebffffff 64bit pref] [ 0.121748] pci 0000:01:00.0: reg 0x24: [io 0xe000-0xe07f] [ 0.121752] pci 0000:01:00.0: reg 0x30: [mem 0xee000000-0xee07ffff pref] [ 0.121817] pci 0000:01:00.1: [10de:0e0c] type 00 class 0x040300 [ 0.121826] pci 0000:01:00.1: reg 0x10: [mem 0xee080000-0xee083fff] [ 0.123417] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.123420] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.123422] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xee0fffff] [ 0.123471] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.123532] pci 0000:03:00.0: [1b21:1080] type 01 class 0x060401 [ 0.123631] pci 0000:03:00.0: System wakeup disabled by ACPI [ 0.123651] pci 0000:00:1c.6: PCI bridge to [bus 03-04] (subtractive decode) [ 0.123659] pci 0000:00:1c.6: bridge window [io 0x0000-0x0cf7] (subtractive decode) [ 0.123659] pci 0000:00:1c.6: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.123660] pci 0000:00:1c.6: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.123661] pci 0000:00:1c.6: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) [ 0.123662] pci 0000:00:1c.6: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 0.123663] pci 0000:00:1c.6: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 0.123664] pci 0000:00:1c.6: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 0.123665] pci 0000:00:1c.6: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 0.123666] pci 0000:00:1c.6: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 0.123667] pci 0000:00:1c.6: bridge window [mem 0xcfa00000-0xfeafffff] (subtractive decode) [ 0.123772] pci 0000:03:00.0: PCI bridge to [bus 04] (subtractive decode) [ 0.123788] pci 0000:03:00.0: bridge window [io 0x0000-0x0cf7] (subtractive decode) [ 0.123789] pci 0000:03:00.0: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.123790] pci 0000:03:00.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.123791] pci 0000:03:00.0: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) [ 0.123792] pci 0000:03:00.0: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 0.123793] pci 0000:03:00.0: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 0.123793] pci 0000:03:00.0: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 0.123794] pci 0000:03:00.0: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 0.123795] pci 0000:03:00.0: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 0.123796] pci 0000:03:00.0: bridge window [mem 0xcfa00000-0xfeafffff] (subtractive decode) [ 0.123818] acpi PNP0A08:00: Disabling ASPM (FADT indicates it is unsupported) [ 0.124305] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.124334] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.124361] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.124393] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 10 11 12 14 15) [ 0.124420] ACPI: PCI Interrupt Link [LNKE] (IRQs *3 4 5 6 10 11 12 14 15) [ 0.124446] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 0.124473] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.124499] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *5 6 10 11 12 14 15) [ 0.124621] ACPI: Enabled 4 GPEs in block 00 to 3F [ 0.124683] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 0.124685] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.124688] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.124688] vgaarb: loaded [ 0.124689] vgaarb: bridge control possible 0000:01:00.0 [ 0.124690] vgaarb: no bridge control possible 0000:00:02.0 [ 0.124815] SCSI subsystem initialized [ 0.124843] libata version 3.00 loaded. [ 0.124867] ACPI: bus type USB registered [ 0.124877] usbcore: registered new interface driver usbfs [ 0.124883] usbcore: registered new interface driver hub [ 0.124893] usbcore: registered new device driver usb [ 0.124976] PCI: Using ACPI for IRQ routing [ 0.126163] PCI: pci_cache_line_size set to 64 bytes [ 0.126202] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff] [ 0.126203] e820: reserve RAM buffer [mem 0xa9570000-0xabffffff] [ 0.126204] e820: reserve RAM buffer [mem 0xa9e4b000-0xabffffff] [ 0.126205] e820: reserve RAM buffer [mem 0xbc1e4000-0xbfffffff] [ 0.126206] e820: reserve RAM buffer [mem 0xbc427000-0xbfffffff] [ 0.126207] e820: reserve RAM buffer [mem 0xbd000000-0xbfffffff] [ 0.126207] e820: reserve RAM buffer [mem 0x42e600000-0x42fffffff] [ 0.126275] NetLabel: Initializing [ 0.126276] NetLabel: domain hash size = 128 [ 0.126276] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.126283] NetLabel: unlabeled traffic allowed by default [ 0.126331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 0.126335] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 0.128361] Switched to clocksource hpet [ 0.132246] AppArmor: AppArmor Filesystem Enabled [ 0.132279] pnp: PnP ACPI init [ 0.132325] system 00:00: [mem 0xfed40000-0xfed44fff] has been reserved [ 0.132327] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.132424] system 00:01: [io 0x0680-0x069f] has been reserved [ 0.132426] system 00:01: [io 0xffff] has been reserved [ 0.132427] system 00:01: [io 0xffff] has been reserved [ 0.132428] system 00:01: [io 0xffff] has been reserved [ 0.132429] system 00:01: [io 0x1c00-0x1cfe] has been reserved [ 0.132430] system 00:01: [io 0x1d00-0x1dfe] has been reserved [ 0.132431] system 00:01: [io 0x1e00-0x1efe] has been reserved [ 0.132432] system 00:01: [io 0x1f00-0x1ffe] has been reserved [ 0.132433] system 00:01: [io 0x1800-0x18fe] could not be reserved [ 0.132434] system 00:01: [io 0x164e-0x164f] has been reserved [ 0.132435] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.132452] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.132480] system 00:03: [io 0x1854-0x1857] has been reserved [ 0.132481] system 00:03: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 0.132524] system 00:04: [io 0x0290-0x029f] has been reserved [ 0.132526] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.132643] system 00:05: [io 0x04d0-0x04d1] has been reserved [ 0.132645] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.132734] pnp 00:06: [dma 0 disabled] [ 0.132757] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.132999] system 00:07: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 0.133000] system 00:07: [mem 0xfed10000-0xfed17fff] has been reserved [ 0.133001] system 00:07: [mem 0xfed18000-0xfed18fff] has been reserved [ 0.133002] system 00:07: [mem 0xfed19000-0xfed19fff] has been reserved [ 0.133003] system 00:07: [mem 0xf8000000-0xfbffffff] has been reserved [ 0.133004] system 00:07: [mem 0xfed20000-0xfed3ffff] has been reserved [ 0.133005] system 00:07: [mem 0xfed90000-0xfed93fff] could not be reserved [ 0.133007] system 00:07: [mem 0xfed45000-0xfed8ffff] has been reserved [ 0.133008] system 00:07: [mem 0xff000000-0xffffffff] has been reserved [ 0.133009] system 00:07: [mem 0xfee00000-0xfeefffff] could not be reserved [ 0.133010] system 00:07: [mem 0xf7fef000-0xf7feffff] has been reserved [ 0.133011] system 00:07: [mem 0xf7ff0000-0xf7ff0fff] has been reserved [ 0.133012] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.133155] pnp: PnP ACPI: found 8 devices [ 0.138903] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.138905] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.138908] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xee0fffff] [ 0.138911] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.138919] pci 0000:03:00.0: PCI bridge to [bus 04] [ 0.138934] pci 0000:00:1c.6: PCI bridge to [bus 03-04] [ 0.138943] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7] [ 0.138944] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff] [ 0.138945] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff] [ 0.138945] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.138946] pci_bus 0000:00: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.138947] pci_bus 0000:00: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.138948] pci_bus 0000:00: resource 10 [mem 0x000dc000-0x000dffff] [ 0.138949] pci_bus 0000:00: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.138950] pci_bus 0000:00: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.138951] pci_bus 0000:00: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.138952] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.138953] pci_bus 0000:01: resource 1 [mem 0xe0000000-0xee0fffff] [ 0.138954] pci_bus 0000:03: resource 4 [io 0x0000-0x0cf7] [ 0.138955] pci_bus 0000:03: resource 5 [io 0x0d00-0xffff] [ 0.138956] pci_bus 0000:03: resource 6 [mem 0x000a0000-0x000bffff] [ 0.138957] pci_bus 0000:03: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.138957] pci_bus 0000:03: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.138958] pci_bus 0000:03: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.138959] pci_bus 0000:03: resource 10 [mem 0x000dc000-0x000dffff] [ 0.138960] pci_bus 0000:03: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.138961] pci_bus 0000:03: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.138962] pci_bus 0000:03: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.138963] pci_bus 0000:04: resource 4 [io 0x0000-0x0cf7] [ 0.138964] pci_bus 0000:04: resource 5 [io 0x0d00-0xffff] [ 0.138965] pci_bus 0000:04: resource 6 [mem 0x000a0000-0x000bffff] [ 0.138966] pci_bus 0000:04: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.138966] pci_bus 0000:04: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.138967] pci_bus 0000:04: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.138968] pci_bus 0000:04: resource 10 [mem 0x000dc000-0x000dffff] [ 0.138969] pci_bus 0000:04: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.138970] pci_bus 0000:04: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.138971] pci_bus 0000:04: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.138986] NET: Registered protocol family 2 [ 0.139110] TCP established hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.139246] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.139342] TCP: Hash tables configured (established 131072 bind 65536) [ 0.139359] TCP: reno registered [ 0.139371] UDP hash table entries: 8192 (order: 6, 262144 bytes) [ 0.139406] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes) [ 0.139454] NET: Registered protocol family 1 [ 0.139464] pci 0000:00:02.0: Video device with shadowed ROM [ 0.171419] PCI: CLS 64 bytes, default 64 [ 0.171450] Trying to unpack rootfs image as initramfs... [ 0.372381] Freeing initrd memory: 20484K (ffff8800357ee000 - ffff880036bef000) [ 0.372397] DMAR: No ATSR found [ 0.372467] IOMMU: dmar0 using Queued invalidation [ 0.372468] IOMMU: dmar1 using Queued invalidation [ 0.372469] IOMMU: Setting RMRR: [ 0.372477] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.373656] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc371000 - 0xbc37dfff] [ 0.373676] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc371000 - 0xbc37dfff] [ 0.373694] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc371000 - 0xbc37dfff] [ 0.373706] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.373711] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.373819] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.375262] RAPL PMU detected, hw unit 2^-14 Joules, API unit is 2^-32 Joules, 4 fixed counters 655360 ms ovfl timer [ 0.375312] microcode: CPU0 sig=0x306c3, pf=0x2, revision=0x9 [ 0.375317] microcode: CPU1 sig=0x306c3, pf=0x2, revision=0x9 [ 0.375321] microcode: CPU2 sig=0x306c3, pf=0x2, revision=0x9 [ 0.375326] microcode: CPU3 sig=0x306c3, pf=0x2, revision=0x9 [ 0.375364] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 0.375380] Scanning for low memory corruption every 60 seconds [ 0.375570] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 0.375583] Initialise system trusted keyring [ 0.375597] audit: initializing netlink subsys (disabled) [ 0.375606] audit: type=2000 audit(1425152809.357:1): initialized [ 0.375815] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.376631] zpool: loaded [ 0.376632] zbud: loaded [ 0.376737] VFS: Disk quotas dquot_6.5.2 [ 0.376757] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.377020] fuse init (API version 7.23) [ 0.377108] Key type big_key registered [ 0.377413] Key type asymmetric registered [ 0.377414] Asymmetric key parser 'x509' registered [ 0.377434] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.377461] io scheduler noop registered [ 0.377463] io scheduler deadline registered (default) [ 0.377479] io scheduler cfq registered [ 0.377694] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.377705] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.377724] vesafb: mode is 1280x1024x32, linelength=5120, pages=0 [ 0.377724] vesafb: scrolling: redraw [ 0.377726] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0 [ 0.377736] vesafb: framebuffer at 0xd0000000, mapped to 0xffffc90005c00000, using 5120k, total 5120k [ 0.377788] Console: switching to colour frame buffer device 160x64 [ 0.377799] fb0: VESA VGA frame buffer device [ 0.377810] intel_idle: MWAIT substates: 0x42120 [ 0.377811] intel_idle: v0.4 model 0x3C [ 0.377811] intel_idle: lapic_timer_reliable_states 0xffffffff [ 0.377950] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0 [ 0.377952] ACPI: Power Button [PWRB] [ 0.377973] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 0.377974] ACPI: Power Button [PWRF] [ 0.378087] GHES: HEST is not enabled! [ 0.378152] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 0.398652] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 0.399762] Linux agpgart interface v0.103 [ 0.400530] brd: module loaded [ 0.400882] loop: module loaded [ 0.400992] libphy: Fixed MDIO Bus: probed [ 0.400994] tun: Universal TUN/TAP device driver, 1.6 [ 0.400995] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 0.401020] PPP generic driver version 2.4.2 [ 0.401065] VFIO - User Level meta-driver version: 0.3 [ 0.401372] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 0.401377] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1 [ 0.401470] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported [ 0.401555] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.401557] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.401558] usb usb1: Product: xHCI Host Controller [ 0.401559] usb usb1: Manufacturer: Linux 3.19.0-6-kvm xhci-hcd [ 0.401560] usb usb1: SerialNumber: 0000:00:14.0 [ 0.401650] hub 1-0:1.0: USB hub found [ 0.401666] hub 1-0:1.0: 14 ports detected [ 0.403110] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 0.403113] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ 0.403144] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 [ 0.403145] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.403146] usb usb2: Product: xHCI Host Controller [ 0.403147] usb usb2: Manufacturer: Linux 3.19.0-6-kvm xhci-hcd [ 0.403148] usb usb2: SerialNumber: 0000:00:14.0 [ 0.403221] hub 2-0:1.0: USB hub found [ 0.403231] hub 2-0:1.0: 6 ports detected [ 0.403856] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.403863] ehci-pci: EHCI PCI platform driver [ 0.403922] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 0.403925] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 3 [ 0.403934] ehci-pci 0000:00:1a.0: debug port 2 [ 0.407820] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 0.407874] ehci-pci 0000:00:1a.0: irq 16, io mem 0xee83c000 [ 0.413145] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 0.413183] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.413185] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.413186] usb usb3: Product: EHCI Host Controller [ 0.413187] usb usb3: Manufacturer: Linux 3.19.0-6-kvm ehci_hcd [ 0.413187] usb usb3: SerialNumber: 0000:00:1a.0 [ 0.413326] hub 3-0:1.0: USB hub found [ 0.413330] hub 3-0:1.0: 2 ports detected [ 0.413453] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 0.413456] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 4 [ 0.413465] ehci-pci 0000:00:1d.0: debug port 2 [ 0.417363] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 0.417502] ehci-pci 0000:00:1d.0: irq 23, io mem 0xee83b000 [ 0.423154] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 0.423179] usb usb4: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.423181] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.423182] usb usb4: Product: EHCI Host Controller [ 0.423183] usb usb4: Manufacturer: Linux 3.19.0-6-kvm ehci_hcd [ 0.423184] usb usb4: SerialNumber: 0000:00:1d.0 [ 0.423293] hub 4-0:1.0: USB hub found [ 0.423296] hub 4-0:1.0: 2 ports detected [ 0.423368] ehci-platform: EHCI generic platform driver [ 0.423374] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.423377] ohci-pci: OHCI PCI platform driver [ 0.423384] ohci-platform: OHCI generic platform driver [ 0.423388] uhci_hcd: USB Universal Host Controller Interface driver [ 0.423416] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 0.426196] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.426216] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.426409] mousedev: PS/2 mouse device common for all mice [ 0.426684] rtc_cmos 00:02: RTC can wake from S4 [ 0.426782] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0 [ 0.426894] rtc_cmos 00:02: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 0.426909] i2c /dev entries driver [ 0.426952] device-mapper: uevent: version 1.0.3 [ 0.427009] device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com [ 0.427020] Intel P-state driver initializing. [ 0.427145] Consider also installing thermald for improved thermal control. [ 0.427148] ledtrig-cpu: registered to indicate activity on CPUs [ 0.427386] PCCT header not found. [ 0.427388] ACPI PCC probe failed. [ 0.427441] TCP: cubic registered [ 0.427498] NET: Registered protocol family 10 [ 0.427656] NET: Registered protocol family 17 [ 0.427663] Key type dns_resolver registered [ 0.427925] Loading compiled-in X.509 certificates [ 0.428429] Loaded X.509 cert 'Magrathea: Glacier signing key: e053bb5652cbcfaa2aef7163e2bf6c27749be754' [ 0.428436] registered taskstats version 1 [ 0.429619] Key type trusted registered [ 0.432446] Key type encrypted registered [ 0.432449] AppArmor: AppArmor sha1 policy hashing enabled [ 0.432452] ima: No TPM chip found, activating TPM-bypass! [ 0.432465] evm: HMAC attrs: 0x1 [ 0.432862] Magic number: 3:332:800 [ 0.432973] rtc_cmos 00:02: setting system clock to 2015-02-28 19:46:49 UTC (1425152809) [ 0.433106] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 0.433107] EDD information not available. [ 0.433164] PM: Hibernation image not present or could not be loaded. [ 0.433424] Freeing unused kernel memory: 1376K (ffffffff81d32000 - ffffffff81e8a000) [ 0.433425] Write protecting the kernel read-only data: 12288k [ 0.433603] Freeing unused kernel memory: 168K (ffff8800017d6000 - ffff880001800000) [ 0.433660] Freeing unused kernel memory: 344K (ffff880001baa000 - ffff880001c00000) [ 0.439365] systemd-udevd[135]: starting version 208 [ 0.439641] random: systemd-udevd urandom read with 1 bits of entropy available [ 0.449592] pps_core: LinuxPPS API ver. 1 registered [ 0.449594] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 0.450433] PTP clock support registered [ 0.451210] ahci 0000:00:1f.2: version 3.0 [ 0.451338] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0xa impl SATA mode [ 0.451339] ahci 0000:00:1f.2: flags: 64bit ncq led clo pio slum part ems apst [ 0.453024] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k [ 0.453025] e1000e: Copyright(c) 1999 - 2014 Intel Corporation. [ 0.453933] scsi host0: ahci [ 0.454069] scsi host1: ahci [ 0.454184] scsi host2: ahci [ 0.454264] scsi host3: ahci [ 0.454389] scsi host4: ahci [ 0.454471] scsi host5: ahci [ 0.454523] ata1: DUMMY [ 0.454525] ata2: SATA max UDMA/133 abar m2048@0xee83a000 port 0xee83a180 irq 28 [ 0.454526] ata3: DUMMY [ 0.454527] ata4: SATA max UDMA/133 abar m2048@0xee83a000 port 0xee83a280 irq 28 [ 0.454527] ata5: DUMMY [ 0.454528] ata6: DUMMY [ 0.454637] e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 0.619330] e1000e 0000:00:19.0 eth0: registered PHC clock [ 0.619333] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) d0:50:99:15:70:12 [ 0.619334] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection [ 0.619370] e1000e 0000:00:19.0 eth0: MAC: 11, PHY: 12, PBA No: FFFFFF-0FF [ 0.714887] usb 3-1: new high-speed USB device number 2 using ehci-pci [ 0.724878] usb 4-1: new high-speed USB device number 2 using ehci-pci [ 0.754850] usb 1-4: new high-speed USB device number 2 using xhci_hcd [ 0.759886] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.759910] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.760196] ata2.00: supports DRM functions and may not be fully accessible [ 0.760427] ata2.00: failed to get NCQ Send/Recv Log Emask 0x1 [ 0.760429] ata2.00: ATA-9: Samsung SSD 850 EVO 500GB, EMT01B6Q, max UDMA/133 [ 0.760430] ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA [ 0.760915] ata2.00: supports DRM functions and may not be fully accessible [ 0.760958] ata4.00: ATA-8: WDC WD10EZRX-00A8LB0, 01.01A01, max UDMA/133 [ 0.760970] ata4.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 0.761078] ata2.00: failed to get NCQ Send/Recv Log Emask 0x1 [ 0.761219] ata2.00: configured for UDMA/133 [ 0.761384] scsi 1:0:0:0: Direct-Access ATA Samsung SSD 850 1B6Q PQ: 0 ANSI: 5 [ 0.761618] sd 1:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 0.761626] sd 1:0:0:0: Attached scsi generic sg0 type 0 [ 0.761817] sd 1:0:0:0: [sda] Write Protect is off [ 0.761819] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 0.761874] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.762044] ata4.00: configured for UDMA/133 [ 0.762099] scsi 3:0:0:0: Direct-Access ATA WDC WD10EZRX-00A 1A01 PQ: 0 ANSI: 5 [ 0.762220] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 0.762222] sd 3:0:0:0: [sdb] 4096-byte physical blocks [ 0.762235] sd 3:0:0:0: Attached scsi generic sg1 type 0 [ 0.762274] sd 3:0:0:0: [sdb] Write Protect is off [ 0.762276] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 0.762300] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.762445] sda: sda1 sda2 < sda5 > [ 0.762657] sd 1:0:0:0: [sda] Attached SCSI disk [ 0.779140] sdb: sdb1 sdb2 sdb3 [ 0.779762] sd 3:0:0:0: [sdb] Attached SCSI disk [ 0.830770] raid6: sse2x1 11769 MB/s [ 0.843201] usb 3-1: New USB device found, idVendor=8087, idProduct=8008 [ 0.843202] usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 0.843227] usb 4-1: New USB device found, idVendor=8087, idProduct=8000 [ 0.843228] usb 4-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 0.843456] hub 3-1:1.0: USB hub found [ 0.843479] hub 4-1:1.0: USB hub found [ 0.843560] hub 3-1:1.0: 6 ports detected [ 0.843566] hub 4-1:1.0: 8 ports detected [ 0.847757] raid6: sse2x2 14488 MB/s [ 0.864735] raid6: sse2x4 17296 MB/s [ 0.881719] raid6: avx2x1 23332 MB/s [ 0.898721] raid6: avx2x2 26101 MB/s [ 0.915687] raid6: avx2x4 31621 MB/s [ 0.915688] raid6: using algorithm avx2x4 (31621 MB/s) [ 0.915689] raid6: using avx2x2 recovery algorithm [ 0.916474] xor: automatically using best checksumming function: [ 0.925705] avx : 36740.000 MB/sec [ 0.932605] Btrfs loaded [ 1.139033] usb 1-4: New USB device found, idVendor=046d, idProduct=0825 [ 1.139035] usb 1-4: New USB device strings: Mfr=0, Product=0, SerialNumber=2 [ 1.139036] usb 1-4: SerialNumber: 5713F440 [ 1.189196] BTRFS: device fsid a3929651-e09b-4f1c-afa3-a5516b17eabb devid 1 transid 8536 /dev/sda1 [ 1.192878] BTRFS info (device sda1): disk space caching is enabled [ 1.207324] BTRFS: detected SSD devices, enabling SSD mode [ 1.292394] usb 1-13: new high-speed USB device number 3 using xhci_hcd [ 1.324761] init: plymouth-upstart-bridge main process (236) terminated with status 1 [ 1.324770] init: plymouth-upstart-bridge main process ended, respawning [ 1.325793] init: plymouth-upstart-bridge main process (252) terminated with status 1 [ 1.325799] init: plymouth-upstart-bridge main process ended, respawning [ 1.327177] init: plymouth-upstart-bridge main process (254) terminated with status 1 [ 1.327184] init: plymouth-upstart-bridge main process ended, respawning [ 1.328042] init: plymouth-upstart-bridge main process (255) terminated with status 1 [ 1.328048] init: plymouth-upstart-bridge main process ended, respawning [ 1.375276] tsc: Refined TSC clocksource calibration: 3399.013 MHz [ 1.456468] usb 1-13: New USB device found, idVendor=1a40, idProduct=0101 [ 1.456470] usb 1-13: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 1.456471] usb 1-13: Product: USB 2.0 Hub [MTT] [ 1.456758] hub 1-13:1.0: USB hub found [ 1.456780] hub 1-13:1.0: 4 ports detected [ 1.610081] usb 1-14: new low-speed USB device number 4 using xhci_hcd [ 1.751783] random: nonblocking pool is initialized [ 1.779040] usb 1-14: New USB device found, idVendor=046d, idProduct=c01e [ 1.779043] usb 1-14: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.779044] usb 1-14: Product: USB-PS/2 Optical Mouse [ 1.779045] usb 1-14: Manufacturer: Logitech [ 1.779109] usb 1-14: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.836010] usb 1-13.3: new low-speed USB device number 5 using xhci_hcd [ 1.866773] Adding 16002044k swap on /dev/sda5. Priority:-1 extents:1 across:16002044k SSFS [ 1.870613] BTRFS info (device sda1): turning on discard [ 1.870617] BTRFS info (device sda1): enabling auto defrag [ 1.870619] BTRFS info (device sda1): disk space caching is enabled [ 1.910440] systemd-udevd[411]: starting version 208 [ 1.922543] nct6775: Found NCT6776D/F or compatible chip at 0x2e:0x290 [ 1.927797] lp: driver loaded but no devices found [ 1.929990] ppdev: user-space parallel port driver [ 1.935139] usb 1-13.3: New USB device found, idVendor=09da, idProduct=0260 [ 1.935141] usb 1-13.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.935143] usb 1-13.3: Product: USB Keyboard [ 1.935144] usb 1-13.3: Manufacturer: [ 1.936263] usb 1-13.3: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.936265] usb 1-13.3: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.990765] audit: type=1400 audit(1425142011.058:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/cups/backend/cups-pdf" pid=515 comm="apparmor_parser" [ 1.990769] audit: type=1400 audit(1425142011.058:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/cupsd" pid=515 comm="apparmor_parser" [ 1.990771] audit: type=1400 audit(1425142011.058:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="third_party" pid=515 comm="apparmor_parser" [ 1.992199] Bluetooth: Core ver 2.20 [ 1.992208] NET: Registered protocol family 31 [ 1.992208] Bluetooth: HCI device and connection manager initialized [ 1.992211] Bluetooth: HCI socket layer initialized [ 1.992213] Bluetooth: L2CAP socket layer initialized [ 1.992216] Bluetooth: SCO socket layer initialized [ 2.005060] Bluetooth: RFCOMM TTY layer initialized [ 2.005065] Bluetooth: RFCOMM socket layer initialized [ 2.005068] Bluetooth: RFCOMM ver 1.11 [ 2.005149] init: cups main process (517) killed by HUP signal [ 2.005154] init: cups main process ended, respawning [ 2.005703] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 [ 2.005705] Bluetooth: BNEP filters: protocol multicast [ 2.005707] Bluetooth: BNEP socket layer initialized [ 2.025992] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.028714] [drm] Initialized drm 1.1.0 20060810 [ 2.040797] audit: type=1400 audit(1425142011.108:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=581 comm="apparmor_parser" [ 2.040802] audit: type=1400 audit(1425142011.108:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=581 comm="apparmor_parser" [ 2.040804] audit: type=1400 audit(1425142011.108:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=581 comm="apparmor_parser" [ 2.045713] [drm] Memory usable by graphics device = 2048M [ 2.045715] [drm] VT-d active for gfx access [ 2.045717] checking generic (d0000000 500000) vs hw (d0000000 10000000) [ 2.045718] fb: switching to inteldrmfb from VESA VGA [ 2.045730] Console: switching to colour dummy device 80x25 [ 2.045762] [drm] Replacing VGA console driver [ 2.051843] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 2.051844] [drm] Driver supports precise vblank timestamp query. [ 2.051868] [drm] DMAR active, disabling use of stolen memory [ 2.060582] AVX2 version of gcm_enc/dec engaged. [ 2.060583] AES CTR mode by8 optimization enabled [ 2.082972] hidraw: raw HID events driver (C) Jiri Kosina [ 2.094078] media: Linux media interface: v0.10 [ 2.096038] Linux video capture interface: v2.00 [ 2.098455] intel_rapl: Found RAPL domain package [ 2.098457] intel_rapl: Found RAPL domain core [ 2.098459] intel_rapl: Found RAPL domain uncore [ 2.098460] intel_rapl: Found RAPL domain dram [ 2.100352] uvcvideo: Found UVC 1.00 device <unnamed> (046d:0825) [ 2.100366] device-mapper: multipath: version 1.7.0 loaded [ 2.116011] input: UVC Camera (046d:0825) as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.0/input/input5 [ 2.116162] usbcore: registered new interface driver uvcvideo [ 2.116164] USB Video Class driver (1.1.1) [ 2.119831] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem [ 2.120764] usbcore: registered new interface driver usbhid [ 2.120765] usbhid: USB HID core driver [ 2.121120] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 2.121403] acpi device:5d: registered as cooling_device5 [ 2.121449] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input6 [ 2.121608] [drm] Initialized i915 1.6.0 20141121 for 0000:00:02.0 on minor 0 [ 2.122015] snd_hda_intel 0000:01:00.1: Disabling MSI [ 2.122049] snd_hda_intel 0000:01:00.1: Handle VGA-switcheroo audio client [ 2.137140] input: HDA Intel HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.0/sound/card0/input7 [ 2.137179] input: HDA Intel HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.0/sound/card0/input8 [ 2.137209] input: HDA Intel HDMI HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.0/sound/card0/input9 [ 2.143337] sound hdaudioC2D0: autoconfig: line_outs=4 (0x14/0x15/0x16/0x17/0x0) type:line [ 2.143339] sound hdaudioC2D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0) [ 2.143340] sound hdaudioC2D0: hp_outs=1 (0x1b/0x0/0x0/0x0/0x0) [ 2.143341] sound hdaudioC2D0: mono: mono_out=0x0 [ 2.143342] sound hdaudioC2D0: dig-out=0x1e/0x0 [ 2.143343] sound hdaudioC2D0: inputs: [ 2.143344] sound hdaudioC2D0: Front Mic=0x19 [ 2.143345] sound hdaudioC2D0: Rear Mic=0x18 [ 2.143346] sound hdaudioC2D0: Line=0x1a [ 2.149206] fbcon: inteldrmfb (fb0) is primary device [ 2.149268] Console: switching to colour frame buffer device 160x64 [ 2.149281] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device [ 2.149282] i915 0000:00:02.0: registered panic notifier [ 2.158330] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card2/input10 [ 2.158389] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card2/input11 [ 2.158443] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card2/input12 [ 2.158496] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card2/input13 [ 2.158546] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card2/input14 [ 2.158656] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card2/input15 [ 2.158758] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card2/input16 [ 2.158836] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card2/input17 [ 2.159068] input: Logitech USB-PS/2 Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-14/1-14:1.0/0003:046D:C01E.0001/input/input18 [ 2.159132] hid-generic 0003:046D:C01E.0001: input,hidraw0: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:14.0-14/input0 [ 2.159213] input: USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-13/1-13.3/1-13.3:1.0/0003:09DA:0260.0002/input/input19 [ 2.181454] init: failsafe main process (785) killed by TERM signal [ 2.209644] hid-generic 0003:09DA:0260.0002: input,hidraw1: USB HID v1.10 Keyboard [ USB Keyboard] on usb-0000:00:14.0-13.3/input0 [ 2.212064] input: USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-13/1-13.3/1-13.3:1.1/0003:09DA:0260.0003/input/input20 [ 2.235618] init: samba-ad-dc main process (883) terminated with status 1 [ 2.255222] audit: type=1400 audit(1425142011.322:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session" pid=984 comm="apparmor_parser" [ 2.255226] audit: type=1400 audit(1425142011.322:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="chromium" pid=984 comm="apparmor_parser" [ 2.256198] audit: type=1400 audit(1425142011.323:10): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/sbin/dhclient" pid=984 comm="apparmor_parser" [ 2.262601] hid-generic 0003:09DA:0260.0003: input,hidraw2: USB HID v1.10 Device [ USB Keyboard] on usb-0000:00:14.0-13.3/input1 [ 2.294949] systemd-logind[963]: New seat seat0. [ 2.295143] systemd-logind[963]: Watching system buttons on /dev/input/event1 (Power Button) [ 2.295175] systemd-logind[963]: Watching system buttons on /dev/input/event3 (Video Bus) [ 2.295202] systemd-logind[963]: Watching system buttons on /dev/input/event0 (Power Button) [ 2.374501] Switched to clocksource tsc [ 2.419305] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input21 [ 2.419369] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input22 [ 2.420313] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input23 [ 2.421452] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input24 [ 2.521472] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.529105] vboxdrv: module verification failed: signature and/or required key missing - tainting kernel [ 2.531134] vboxdrv: Found 4 processor cores. [ 2.531464] vboxdrv: fAsync=0 offMin=0x1df offMax=0xe1d [ 2.531510] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'. [ 2.531511] vboxdrv: Successfully loaded version 4.3.18_Ubuntu (interface 0x001a0008). [ 2.539538] vboxpci: IOMMU found [ 2.692404] ------------[ cut here ]------------ [ 2.692446] WARNING: CPU: 3 PID: 599 at /home/terusus/ubuntu-vivid/source/drivers/gpu/drm/i915/intel_display.c:9705 intel_check_page_flip+0xa2/0xf0 [i915]() [ 2.692447] WARN_ON(!in_irq()) [ 2.692448] Modules linked in: [ 2.692449] pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) hid_generic snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_usb_audio(+) snd_usbmidi_lib uvcvideo dm_multipath videobuf2_vmalloc scsi_dh intel_rapl videobuf2_memops videobuf2_core v4l2_common iosf_mbi videodev x86_pkg_temp_thermal media usbhid intel_powerclamp snd_hda_intel kvm_intel snd_hda_controller hid snd_hda_codec snd_hwdep kvm snd_pcm crct10dif_pclmul crc32_pclmul snd_seq_midi ghash_clmulni_intel snd_seq_midi_event aesni_intel snd_rawmidi aes_x86_64 lrw gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_timer serio_raw i915 snd drm_kms_helper drm lpc_ich mei_me shpchp soundcore mei i2c_algo_bit 8250_fintek nuvoton_cir rc_core bnep rfcomm video soc_button_array mac_hid bluetooth binfmt_misc [ 2.692469] parport_pc ppdev lp parport nct6775 hwmon_vid coretemp btrfs xor raid6_pq e1000e ahci psmouse ptp libahci pps_core [ 2.692476] CPU: 3 PID: 599 Comm: irq/31-i915 Tainted: G OE 3.19.0-6-kvm #6 [ 2.692476] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H87 Pro4, BIOS P2.10 07/09/2014 [ 2.692477] ffffffffc07661b8 ffff880419b93c98 ffffffff817c6613 0000000000000007 [ 2.692479] ffff880419b93ce8 ffff880419b93cd8 ffffffff8107732a ffff880419b93cd8 [ 2.692480] ffff8804186e3000 ffff880419b88800 ffff880419b88800 0000000000000000 [ 2.692481] Call Trace: [ 2.692496] [<ffffffff817c6613>] dump_stack+0x4c/0x6e [ 2.692499] [<ffffffff8107732a>] warn_slowpath_common+0x8a/0xc0 [ 2.692501] [<ffffffff810773a6>] warn_slowpath_fmt+0x46/0x50 [ 2.692510] [<ffffffffc07111a2>] intel_check_page_flip+0xa2/0xf0 [i915] [ 2.692518] [<ffffffffc06de277>] ironlake_irq_handler+0x417/0x1000 [i915] [ 2.692521] [<ffffffff8109bc4d>] ? finish_task_switch+0x5d/0x100 [ 2.692523] [<ffffffff810d0340>] ? irq_thread_fn+0x50/0x50 [ 2.692525] [<ffffffff810d036d>] irq_forced_thread_fn+0x2d/0x70 [ 2.692526] [<ffffffff810d08b7>] irq_thread+0x137/0x160 [ 2.692527] [<ffffffff810d03e0>] ? wake_threads_waitq+0x30/0x30 [ 2.692529] [<ffffffff810d0780>] ? irq_thread_check_affinity+0x90/0x90 [ 2.692531] [<ffffffff810961eb>] kthread+0xdb/0x100 [ 2.692541] [<ffffffff81096110>] ? kthread_create_on_node+0x1c0/0x1c0 [ 2.692543] [<ffffffff817cd5bc>] ret_from_fork+0x7c/0xb0 [ 2.692545] [<ffffffff81096110>] ? kthread_create_on_node+0x1c0/0x1c0 [ 2.692546] ---[ end trace 3bc57ae31fe4d29b ]--- [ 2.714351] init: plymouth-upstart-bridge main process ended, respawning [ 2.715692] init: plymouth-upstart-bridge main process (1679) killed by TERM signal [ 2.749461] systemd-logind[963]: Failed to start unit user@112.service: Unknown unit: user@112.service [ 2.749464] systemd-logind[963]: Failed to start user service: Unknown unit: user@112.service [ 2.751775] systemd-logind[963]: New session c1 of user lightdm. [ 2.751786] systemd-logind[963]: Linked /tmp/.X11-unix/X0 to /run/user/112/X11-display. [ 2.947777] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this. [ 3.001213] ip_tables: (C) 2000-2006 Netfilter Core Team [ 3.026918] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 3.279516] usb 1-4: set resolution quirk: cval->res = 384 [ 3.279678] usbcore: registered new interface driver snd-usb-audio [ 4.040540] e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx [ 4.040544] e1000e 0000:00:19.0 eth0: 10/100 speed: disabling TSO [ 4.040583] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 10.029597] systemd-logind[963]: Failed to start unit user@1000.service: Unknown unit: user@1000.service [ 10.029602] systemd-logind[963]: Failed to start user service: Unknown unit: user@1000.service [ 10.032050] systemd-logind[963]: New session c2 of user terusus. [ 10.032061] systemd-logind[963]: Linked /tmp/.X11-unix/X0 to /run/user/1000/X11-display.

Offline

#4349 2015-03-01 00:24:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is CONFIG_VFIO_PCI_VGA also set?

hv_time is incompatible with nvidia

Missing -vga none

Don't use q35


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4350 2015-03-01 11:24:29

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, I finally managed to passthrough my GPU. I've tried before and had lots of errors and eventually gave up. Later on I found my motherboard just didn't support IOMMU -.-
Anyways i've bought a new motherboard that does support IOMMU so now I have the following setup:
MOBO: Asrock Extreme 6/ac
CPU: i7 4790k @ 4Ghz
GPU Host: GTX 980
GPU Guest: GTX 770 (I can't turn these around sadly, it will not give any video output otherwise)
Linux Mainline 3.18.5 kernel with ACS patch enabled (from the first post).

Using the following commandline I managed to set up a Windows 8 VM running on my SSD's, and having the GTX 770 as passthrough:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -M q35 -m 8096 -cpu host -soundhw hda \ -smp 6,sockets=1,cores=6,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=ssddisk,format=raw,if=none -device scsi-hd,drive=ssddisk \ -drive file=/dev/sdd,id=hdddisk,format=raw,if=none -device scsi-hd,drive=hdddisk

Problem 1:
Everything appeared to be succesful, Windows runs without problems (just the mouse is a bit itchy), and the NVIDIA driver installer was able to detect the GPU and install the driver.
Rebooted VM, installed GPU-Z and it shows my GTX 770 (with 347.52 driver). However it also shows the GPU Clock is 0 Mhz, just as memory and default clock.
I also can't open the NVIDIA settings screen, first it gave me some runtime errors, now it just doesn't show up at all.
Whenever I run Geforce Experience it seems to not find any installed driver, as it automatically starts downloading the driver I already have installed.

Problem 2:
Sound laggs badly. I can hear it, but if it would be measured in frames per second, it would've been around 3. As you can see in my commandline I try to use PulseAudio.
What am I doing wrong here?

Offline

#4351 2015-03-01 11:37:09

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Hey guys, I finally managed to passthrough my GPU. I've tried before and had lots of errors and eventually gave up. Later on I found my motherboard just didn't support IOMMU -.-
Anyways i've bought a new motherboard that does support IOMMU so now I have the following setup:
MOBO: Asrock Extreme 6/ac
CPU: i7 4790k @ 4Ghz
GPU Host: GTX 980
GPU Guest: GTX 770 (I can't turn these around sadly, it will not give any video output otherwise)
Linux Mainline 3.18.5 kernel with ACS patch enabled (from the first post).

Using the following commandline I managed to set up a Windows 8 VM running on my SSD's, and having the GTX 770 as passthrough:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -M q35 -m 8096 -cpu host -soundhw hda \ -smp 6,sockets=1,cores=6,threads=1 \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=ssddisk,format=raw,if=none -device scsi-hd,drive=ssddisk \ -drive file=/dev/sdd,id=hdddisk,format=raw,if=none -device scsi-hd,drive=hdddisk

Problem 1:
Everything appeared to be succesful, Windows runs without problems (just the mouse is a bit itchy), and the NVIDIA driver installer was able to detect the GPU and install the driver.
Rebooted VM, installed GPU-Z and it shows my GTX 770 (with 347.52 driver). However it also shows the GPU Clock is 0 Mhz, just as memory and default clock.
I also can't open the NVIDIA settings screen, first it gave me some runtime errors, now it just doesn't show up at all.
Whenever I run Geforce Experience it seems to not find any installed driver, as it automatically starts downloading the driver I already have installed.

Problem 2:
Sound laggs badly. I can hear it, but if it would be measured in frames per second, it would've been around 3. As you can see in my commandline I try to use PulseAudio.
What am I doing wrong here?


Well..

aw wrote:

...
Missing -vga none
Don't use q35

For the sound you could try using alsa, and/or playing with the parameters:

qemu-system-x86_64 -audio-help

to see all available drivers and params, also i experienced some crackling and lag until i started pinning the vcpus

Offline

#4352 2015-03-01 12:36:43

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Well..

aw wrote:

...
Missing -vga none
Don't use q35

Well -vga none explains something. However if I change my command line to use -vga none, it only gives me a qemu command line:

compat_monitor0 console QEMU 2.2.0 monitor - type 'help' for more information (qemu)

This happens with both Q35 and i440fx.


nbhs wrote:

For the sound you could try using alsa, and/or playing with the parameters:

qemu-system-x86_64 -audio-help

to see all available drivers and params, also i experienced some crackling and lag until i started pinning the vcpus

I'll try some things and report back.

Last edited by PureTryOut (2015-03-01 12:39:09)

Offline

#4353 2015-03-01 12:43:57

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Well -vga none explains something. However if I change my command line to use -vga none, it only gives me a qemu command line:

compat_monitor0 console QEMU 2.2.0 monitor - type 'help' for more information (qemu)

Yeah that's whats supposed to happen you'll get output on your gpu

Offline

#4354 2015-03-01 13:03:47

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry I didn't make myself clear I guess.

Without -vga none it boots up Windows, although with non working GPU.
With -vga none it only shows that. It doesn't boot up the VM.

Offline

#4355 2015-03-01 14:14:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Sorry I didn't make myself clear I guess.

Without -vga none it boots up Windows, although with non working GPU.
With -vga none it only shows that. It doesn't boot up the VM.

You also need:

-cpu host,kvm=off

Offline

#4356 2015-03-01 15:00:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Sorry I didn't make myself clear I guess.

Without -vga none it boots up Windows, although with non working GPU.
With -vga none it only shows that. It doesn't boot up the VM.

Then either VGA arbitration is not working or QEMU can't read the ROM on the GPU.  VGA arbitration not working was a problem we used to see using the proprietary nvidia driver in the host (driver would lock and never release the VGA arbitration lock).  Nvidia might have fixed this, dunno.  The latter ROM issue would produce an invalid ROM contents message in dmesg.  GeForce cards do not work as secondary GPUs in the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4357 2015-03-01 15:38:03

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You also need:

-cpu host,kvm=off

Sadly that did not work either. Still showing the console...
Current commandline:

sudo qemu-system-x86_64 -enable-kvm -m 8096 -cpu host,kvm=off \ -smp 6,sockets=1,cores=6,threads=1 \ -vga none -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=ssddisk,format=raw,if=none -device scsi-hd,drive=ssddisk \ -drive file=/dev/sdd,id=hdddisk,format=raw,if=none -device scsi-hd,drive=hdddisk
aw wrote:

Then either VGA arbitration is not working or QEMU can't read the ROM on the GPU.  VGA arbitration not working was a problem we used to see using the proprietary nvidia driver in the host (driver would lock and never release the VGA arbitration lock).  Nvidia might have fixed this, dunno.  The latter ROM issue would produce an invalid ROM contents message in dmesg.  GeForce cards do not work as secondary GPUs in the guest.

So is there any workaround or any way I can fix this?

Offline

#4358 2015-03-01 18:52:07

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jodaco wrote:

Has anyone else with an amd system done this with nbhs' suggested option of disabling npt?  I found that with it disabled it did improve my benchmark scores, doubled in some cases, and games hit higher framerates, but its not consistent.  With npt=0 set cpu load is all over the damn place and framerates in games vary anywhere from 5 fps to 150fps.

I have an AMD 8320, Asus M5A99X EVO PRO r2.0 with the latest bios, 32 GB of ram, Geforce 660 ti for host, and a Geforce 970 on the guest.

I am using npt=0. Have used npt=1  for an extended period and performance was worse. The effect was that in some games the framerate was lower, but GPU and CPU loads were lower too and a bit more kernel time on host. npt=0 is generally much better. In both cases the machines are stable.

Offline

#4359 2015-03-01 19:35:29

terusus
Member
Registered: 2015-02-27
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Is CONFIG_VFIO_PCI_VGA also set?

hv_time is incompatible with nvidia

Missing -vga none

Don't use q35

Thanks for the fast replay.

1. CONFIG_VFIO_PCI_VGA=y is set by default. I double checked with my /boot/config-...
2. Removed hv_time.
3. Changed q35 to pc (as I understand this is the default).
4. I had to remove "bus=pcie.0" from ioh3420 line because of an error.
5. I had to use "rombar=0" or "romfile=..." when assigning my VGA.

So, my QEMU start line became:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 \ -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/media/.../GF114.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi \ -drive file=/storage/vm/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/media/.../win7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/media/.../virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -vga none

The results were the same for rombar/romfile:
1) VGA appears dead WITHOUT "-vga none".
2) No output on the nVidia DVI ports WITH "-vga none"

PS: Tried a few restarts. Also reinstalled the win7 vm.

Last edited by terusus (2015-03-01 21:20:30)

Offline

#4360 2015-03-01 21:08:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JohnyPea wrote:
Jodaco wrote:

Has anyone else with an amd system done this with nbhs' suggested option of disabling npt?  I found that with it disabled it did improve my benchmark scores, doubled in some cases, and games hit higher framerates, but its not consistent.  With npt=0 set cpu load is all over the damn place and framerates in games vary anywhere from 5 fps to 150fps.

I have an AMD 8320, Asus M5A99X EVO PRO r2.0 with the latest bios, 32 GB of ram, Geforce 660 ti for host, and a Geforce 970 on the guest.

I am using npt=0. Have used npt=1  for an extended period and performance was worse. The effect was that in some games the framerate was lower, but GPU and CPU loads were lower too and a bit more kernel time on host. npt=0 is generally much better. In both cases the machines are stable.

http://magazine.redhat.com/2007/11/20/r … ed-guests/
That's ridiculous. Simply crazy. I don't get it.
How BROKEN NPT should be that it slows down the VM? And I've been worrying about too big errata list on my CPU family.

EDIT:
http://www.cse.iitd.ernet.in/~sbansal/c … nal-TM.pdf
Oooh, nice document to read. I'll try reading this somewhere in future time.

Last edited by Duelist (2015-03-01 21:24:13)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4361 2015-03-01 21:30:03

hurenkam
Member
Registered: 2012-12-10
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It's generally recommended that the graphics card for assignment is a secondary device on the host because it's often difficult to detach all the host drivers from the primary display.  Even if you use pci-stub.ids to prevent PCI drivers from attaching, the low level VGA/VESA drivers can still make use of it.  It's not necessarily impossible, but it's not a heavily used or documented path.

Passing a secondary device (from host perspective) is indeed easier.
However, after some fiddling, i managed to passthrough the primary card on my system, but it does involve some extra work.
Apart from the usual driver blacklisting (radeon in my case) i had to do the following as well:
- Switch grub to text-only mode by adding "GRUB_GFXPAYLOAD_LINUX=text" to /etc/defaults/grub
- Switch linux boot to text-only mode by adding "nomodeset nofb" to GRUB_CMDLINE_LINUX_DEFAULT=... in /etc/defaults/grub

After adding these parameters, I was able to passthrough the host primary graphics to qemu guest in the same way as i passthrough my secondary graphics.

Note that you will completely lose your host console when you do this, so make sure you can ssh in, and perhaps have a serial terminal ready as well.

Offline

#4362 2015-03-01 22:01:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hurenkam wrote:

Note that you will completely lose your host console when you do this, so make sure you can ssh in, and perhaps have a serial terminal ready as well.

You've got a headless host?.. So you're doing GPU passthrough with only one GPU? Cool.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4363 2015-03-01 22:48:41

hurenkam
Member
Registered: 2012-12-10
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
hurenkam wrote:

Note that you will completely lose your host console when you do this, so make sure you can ssh in, and perhaps have a serial terminal ready as well.

You've got a headless host?.. So you're doing GPU passthrough with only one GPU? Cool.

Actually, I'm doing GPU passthrough with 3 GPU's, but indeed leaving the host headless:
- GPU1: Radeon HD6450; Passthrough to Windows 8 Guest
- GPU2: Radeon HD4350; Passthrough to OSX Mavericks Guest
- GPU3: Radeon HD4350; Passthrough to Arch Linux Guest

All this is on rather dated hardware (but since all is working fine, why bother to upgrade?):
- ASUS P7P55D EVO Mainboard
- Intel i7 860 CPU

This machine has been running Xen for several years, I've only recently switched to qemu because i wanted to see how stable it runs with pci & vga passthrough.
So far I have no regrets.

Last edited by hurenkam (2015-03-01 23:05:57)

Offline

#4364 2015-03-02 00:04:46

Jodaco
Member
Registered: 2014-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
JohnyPea wrote:
Jodaco wrote:

Has anyone else with an amd system done this with nbhs' suggested option of disabling npt?  I found that with it disabled it did improve my benchmark scores, doubled in some cases, and games hit higher framerates, but its not consistent.  With npt=0 set cpu load is all over the damn place and framerates in games vary anywhere from 5 fps to 150fps.

I have an AMD 8320, Asus M5A99X EVO PRO r2.0 with the latest bios, 32 GB of ram, Geforce 660 ti for host, and a Geforce 970 on the guest.

I am using npt=0. Have used npt=1  for an extended period and performance was worse. The effect was that in some games the framerate was lower, but GPU and CPU loads were lower too and a bit more kernel time on host. npt=0 is generally much better. In both cases the machines are stable.

http://magazine.redhat.com/2007/11/20/r … ed-guests/
That's ridiculous. Simply crazy. I don't get it.
How BROKEN NPT should be that it slows down the VM? And I've been worrying about too big errata list on my CPU family.

EDIT:
http://www.cse.iitd.ernet.in/~sbansal/c … nal-TM.pdf
Oooh, nice document to read. I'll try reading this somewhere in future time.

I gave up and I built a new intel machine, pretty costly decision but in the end it is well worth it. I have not installed a bare-metal version of windows on it to test, but performance in my games seems to work atleast as well as the AMD 8320 did on bare metal.  I had an issue with this stock intel cooler running a little warm with Turbo Boost turned on in the UEFI and it started throttling the cpu back which destroyed vm performance.  Once I turned that option off everything seems to run great. Using synergy to switch mouse and keyboard between host and guest and it works really well, I have played several hours of Battlefield 4.

Cpu: Core i7-4790k
Mobo: Gigabyte Z97X-U3H
Memory: Corsair Vengeance 32gb DDR3-1600
GPU: EVGA Geforce 970 and using intel graphics for host

Booting on Arch Linux in an uefi setup and also using ovmf from the extra repository in arch and it worked perfectly out of the box, just using qemu and passing command line options.

Offline

#4365 2015-03-02 06:24:51

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

terusus wrote:
sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 \

PC is not 440fx... in my xml i got "pc-i440fx-2.1", not sure if it's same for qemu command line, but pretty sure it's not PC.

Offline

#4366 2015-03-02 07:35:35

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
terusus wrote:
sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 \

PC is not 440fx... in my xml i got "pc-i440fx-2.1", not sure if it's same for qemu command line, but pretty sure it's not PC.

Pc is i440fx

[nbhs@virtbox ~]$ qemu-system-x86_64 -M ? Supported machines are: pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-2.2) pc-i440fx-2.2 Standard PC (i440FX + PIIX, 1996) (default) pc-i440fx-2.1 Standard PC (i440FX + PIIX, 1996) pc-i440fx-2.0 Standard PC (i440FX + PIIX, 1996) pc-i440fx-1.7 Standard PC (i440FX + PIIX, 1996) pc-i440fx-1.6 Standard PC (i440FX + PIIX, 1996) pc-i440fx-1.5 Standard PC (i440FX + PIIX, 1996) pc-i440fx-1.4 Standard PC (i440FX + PIIX, 1996) pc-1.3 Standard PC (i440FX + PIIX, 1996) pc-1.2 Standard PC (i440FX + PIIX, 1996) pc-1.1 Standard PC (i440FX + PIIX, 1996) pc-1.0 Standard PC (i440FX + PIIX, 1996) pc-0.15 Standard PC (i440FX + PIIX, 1996) pc-0.14 Standard PC (i440FX + PIIX, 1996) pc-0.13 Standard PC (i440FX + PIIX, 1996) pc-0.12 Standard PC (i440FX + PIIX, 1996) pc-0.11 Standard PC (i440FX + PIIX, 1996) pc-0.10 Standard PC (i440FX + PIIX, 1996) q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.2) pc-q35-2.2 Standard PC (Q35 + ICH9, 2009) pc-q35-2.1 Standard PC (Q35 + ICH9, 2009) pc-q35-2.0 Standard PC (Q35 + ICH9, 2009) pc-q35-1.7 Standard PC (Q35 + ICH9, 2009) pc-q35-1.6 Standard PC (Q35 + ICH9, 2009) pc-q35-1.5 Standard PC (Q35 + ICH9, 2009) pc-q35-1.4 Standard PC (Q35 + ICH9, 2009) isapc ISA-only PC none empty machine

Offline

#4367 2015-03-02 07:39:11

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

terusus wrote:
aw wrote:

Is CONFIG_VFIO_PCI_VGA also set?

hv_time is incompatible with nvidia

Missing -vga none

Don't use q35

Thanks for the fast replay.

1. CONFIG_VFIO_PCI_VGA=y is set by default. I double checked with my /boot/config-...
2. Removed hv_time.
3. Changed q35 to pc (as I understand this is the default).
4. I had to remove "bus=pcie.0" from ioh3420 line because of an error.
5. I had to use "rombar=0" or "romfile=..." when assigning my VGA.

So, my QEMU start line became:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 \ -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/media/.../GF114.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi \ -drive file=/storage/vm/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/media/.../win7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/media/.../virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -vga none

The results were the same for rombar/romfile:
1) VGA appears dead WITHOUT "-vga none".
2) No output on the nVidia DVI ports WITH "-vga none"

PS: Tried a few restarts. Also reinstalled the win7 vm.

Why are you using -device ioh3420 with -M pc???

Offline

#4368 2015-03-02 08:58:48

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

terusus wrote:
aw wrote:

Is CONFIG_VFIO_PCI_VGA also set?

hv_time is incompatible with nvidia

Missing -vga none

Don't use q35

Thanks for the fast replay.

1. CONFIG_VFIO_PCI_VGA=y is set by default. I double checked with my /boot/config-...
2. Removed hv_time.
3. Changed q35 to pc (as I understand this is the default).
4. I had to remove "bus=pcie.0" from ioh3420 line because of an error.
5. I had to use "rombar=0" or "romfile=..." when assigning my VGA.

So, my QEMU start line became:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 \ -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/media/.../GF114.rom \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi \ -drive file=/storage/vm/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/media/.../win7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/media/.../virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -vga none

The results were the same for rombar/romfile:
1) VGA appears dead WITHOUT "-vga none".
2) No output on the nVidia DVI ports WITH "-vga none"

PS: Tried a few restarts. Also reinstalled the win7 vm.


You are supposed to use "-vga none", it's in the first post. That's the whole point of this thread, so you can use real GPU hardware as primary (and quite possibly, the only) VGA output in the guest VM. Also, if you do not use OVMF then you need to use x-vga=on parameter, so vfio knows to passthorugh VGA correctly. This "x-vga=on" is not needed if you do use OVMF because this standard (only possible for UEFI guests) has discarded the whole notion of "primary VGA output". But in any case you do need "-vga none".

If you start qemu without "-vga none" then emulated Cirrus becomes your primary VGA in guest and hardware GPU becomes secondary VGA, which you could do a long time ago in Xen and which is not what this thread is about.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4369 2015-03-02 09:09:24

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
Bronek wrote:
Tyrewt wrote:

Trying to do some USB passthough in my config with:



The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.




So I followed your advice with no luck, maybe I should focus efforts on USB Device Pass-through as it seems get further with Ubuntu.

For the record, the three USB controllers were:

00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 04)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 04)

Added them to my /etc/vfio-pci.cfg

0000:00:14.0
0000:00:1a.0
0000:00:1d.0

Then added the following lines to my qemu config

-usb \
-device vfio-pci,host=00:14.0 \
-device vfio-pci,host=00:1a.0 \
-device vfio-pci,host=00:1d.0 \


Starting the Virtual Machine fails at the first controller with the following error.

qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio: error opening /dev/vfio/4: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio: failed to get group 4
qemu-system-x86_64: -device vfio-pci,host=00:14.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:14.0: Device 'vfio-pci' could not be initialized

Any suggestions?

ok so you are trying to add three USB controller devices (1x USB 3.0 at 00:14.0 and 2x USD 2.0 at 00:1a.0,00:1d.0) . Apparently adding USB 3.0 controller fails, although we do not see anything about the other two USB 2.0 controllers. Did you ensure you are passing whole IOMMU group? Did you remember to add vendorid:deviceid to pci-stub.ids=... in kernel command line? Did you try passing only USB 2.0 devices? I assume your host was restarted to allow for vfio to claim these USB controllers?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4370 2015-03-02 10:53:37

nalare
Member
Registered: 2015-03-02
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

this topic has been one of the guides I've followed to build my new rigg:

core i7 4790
asrock z97 extreme 4
32 gb ddr3 RAM 1600 kingston hyper x savage
crucial mx 256gb SSD
NIC Intel Gigabit CT Desktop Adapter PCI-e (EXPI9301CT 1000)
PSU corsair gold CS650M

+ 4tb iSCSI target ->ZFS raid Z1 storage in a nas4free dedicated box (which I dont know if I should virtualize)

I've also got an old asus geforce GTS250 and an older ati radeon x1500. I will use one of them for the host if I choose the XEN or KVM option.
I'm also looking for a cheap hba SATA card for storage passthrough
and, of course, a new GPU. I will choose the gpu (probably an nvidia GPU, I don't like amd GPUs) when I decide which hypervisor is the best for my needs.

I'm still trying to decide between vmware sphere, xen or KVM

My idea is to use a hypervisor for something like a Multi-headed Gaming Setup, plus a few more VMs:

-linux "main" VM -> (office, video, websurfing, development...)
-windows server 2012 R2 VM -> active directory, (perhaps vCenter Server if I choose the vmware way)
-windows 8.1 gaming VM
-NAS4free file server (a BSD distro used for file sharing)-> I already use it as iSCSI target for VMware (or other hypervisor) in a dedicated old box, I'd like to implement it as a VM as well in the same rigg
-windows 7 VM (for websurfing, software testing...).

obviously not all of them will be used at the same time, only the host, windows server 2012 r2, and the bsd NAS appliance (if I decide to virtualize my NAS dedicated box) will be plugged 24/7
Only the windows 8.1 gaming VM (one GPU) and perhaps the NAS appliance, if I decide to use it, will need PCI passthrough.

I've read that the performance of them three is, nowadays, quite good in VGA passthrough, but I still don't know which of them (vmware sphere, xen or KVM) will fit my need, or is the best for my hardware/network.

I've got a GBE network at home (controlled by an asus DSL-N16U modem-router), but many VMs will be used through wifi n300 or even n150. The most demanding VMs will be used through wired gigabite network (the main linux VM and the windows 8.1 gaming VM).


KVM pros: I can use geforce cards for VGA passthrough, I can use a linux local host as my main as and as the hypervisor, all in one
KVM contras: I've read it has the worst VGA passthrough support of the 3 hypervisors. It seems the most dificult to admin (no good GUIs). Constant changes of kernel, drivers, configs...

XEN pros: it seems it has a good performance in VGA passthrough, I can use a linux local host as my main as and as the hypervisor, all in one
XEN contras: no VGA passthrough support for geforce cards, it seems easier to config & admin than KVM (but more difficult than vmware sphere for me)

VMware Sphere pros: people report it has a very good performance in VGA passthrough, and it's the hypervisor I've used the most.
VMware Sphere contras: no support for geforce GPUs on passthough, only AMD or nvidia quadro cards (or modified geforces). And all the operating systems act as VMs, no "usable" local host like KVM or ZEN (dom0), -> so one more VM that needs to be used through the home network from another computer.


Any suggestion about which hypervisor should I use?

Any suggestion about a good+cheap nvidia GPU for passthrough?
or, if I choose vmware sphere or XEN, I will not spend my money in a multiOS quadro card, so perhaps I will choose an AMD gpu ..any suggestion? should be a cheap one, passthrogh capable, and not powerhungry (my PSU is "only" 650w, and if I have to choose an amd card for the windows 8.1 gaming VM, that card will have to be in the same motherboard with my power-hungry old asus nvidia geforce GTS250 used by the linux host).

Any suggestion for a good+cheap RAID sata card (i don't need SAS, and I don't need more than 4 SATA3 ports) passthrough capable to use it with a BSD based NAS distro?

thanks for any help

PD: sorry for my poor english

Offline

#4371 2015-03-02 11:31:22

JohnyPea
Member
Registered: 2014-07-17
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

http://magazine.redhat.com/2007/11/20/r … ed-guests/
That's ridiculous. Simply crazy. I don't get it.
How BROKEN NPT should be that it slows down the VM? And I've been worrying about too big errata list on my CPU family.

EDIT:
http://www.cse.iitd.ernet.in/~sbansal/c … nal-TM.pdf
Oooh, nice document to read. I'll try reading this somewhere in future time.

At first I thought that it should lower host virtualization overhead, therefore I tried using it. Something isn't probably working as expected? Thanks for the document, will take a look later.

Offline

#4372 2015-03-02 14:47:31

terusus
Member
Registered: 2015-02-27
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

You are supposed to use "-vga none", it's in the first post. That's the whole point of this thread, so you can use real GPU hardware as primary (and quite possibly, the only) VGA output in the guest VM. Also, if you do not use OVMF then you need to use x-vga=on parameter, so vfio knows to passthorugh VGA correctly. This "x-vga=on" is not needed if you do use OVMF because this standard (only possible for UEFI guests) has discarded the whole notion of "primary VGA output". But in any case you do need "-vga none".

If you start qemu without "-vga none" then emulated Cirrus becomes your primary VGA in guest and hardware GPU becomes secondary VGA, which you could do a long time ago in Xen and which is not what this thread is about.

I understood that. But, the fact is that I do not get any graphical output from the VM with "-vga none". Nothing on the DVI ports or when connecting with VNC.

(I am not using OVMF - I'm testing with a win7 machine. I did run an OVMF win8 machine and did have "x-vga=on" + "-vga none" but the results were the same.)

And this is the strange thing for me:
- I get no errors when starting the VM.
- With standard VGA (without -vga none) I see my nVidia card in the VM. I even see the option to unplug the hardware from windows. Standard drivers run their checks and get installed successfully. And still nothing.

And obviously I got something wrong. However, I am really determined to get this working and I really appreciate your input. So, as I understand, if everything worked well I should have video output from the VM on nVidia DVI port and a monitor connected there should work normally? Do see any obvious mistakes that I have made?

Offline

#4373 2015-03-02 15:57:50

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

terusus wrote:

I understood that. But, the fact is that I do not get any graphical output from the VM with "-vga none". Nothing on the DVI ports or when connecting with VNC.

(I am not using OVMF - I'm testing with a win7 machine. I did run an OVMF win8 machine and did have "x-vga=on" + "-vga none" but the results were the same.)

And this is the strange thing for me:
- I get no errors when starting the VM.
- With standard VGA (without -vga none) I see my nVidia card in the VM. I even see the option to unplug the hardware from windows. Standard drivers run their checks and get installed successfully. And still nothing.

And obviously I got something wrong. However, I am really determined to get this working and I really appreciate your input. So, as I understand, if everything worked well I should have video output from the VM on nVidia DVI port and a monitor connected there should work normally? Do see any obvious mistakes that I have made?

are you able to verify that this card works at all, for bare metal configuration, when placed in the slot you try to use it in?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4374 2015-03-02 16:13:15

terusus
Member
Registered: 2015-02-27
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

Yes. When I installed Ubuntu I was using my nVidia card as a primary VGA. It was working with nouveau and with the proprietary nVidia drivers.

It worked fine on win7 too.

PS: I had to set the Intel IGP as a primary VGA in the bios in order to blacklist nouveau.

Last edited by terusus (2015-03-02 16:14:41)

Offline

#4375 2015-03-02 19:43:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

terusus wrote:
[ 2.529105] vboxdrv: module verification failed: signature and/or required key missing - tainting kernel [ 2.531134] vboxdrv: Found 4 processor cores. [ 2.531464] vboxdrv: fAsync=0 offMin=0x1df offMax=0xe1d [ 2.531510] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'. [ 2.531511] vboxdrv: Successfully loaded version 4.3.18_Ubuntu (interface 0x001a0008). [ 2.539538] vboxpci: IOMMU found [ 2.692404] ------------[ cut here ]------------ [ 2.692446] WARNING: CPU: 3 PID: 599 at /home/terusus/ubuntu-vivid/source/drivers/gpu/drm/i915/intel_display.c:9705 intel_check_page_flip+0xa2/0xf0 [i915]() [ 2.692447] WARN_ON(!in_irq()) [ 2.692448] Modules linked in: [ 2.692449] pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) hid_generic snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_usb_audio(+) snd_usbmidi_lib uvcvideo dm_multipath videobuf2_vmalloc scsi_dh intel_rapl videobuf2_memops videobuf2_core v4l2_common iosf_mbi videodev x86_pkg_temp_thermal media usbhid intel_powerclamp snd_hda_intel kvm_intel snd_hda_controller hid snd_hda_codec snd_hwdep kvm snd_pcm crct10dif_pclmul crc32_pclmul snd_seq_midi ghash_clmulni_intel snd_seq_midi_event aesni_intel snd_rawmidi aes_x86_64 lrw gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_timer serio_raw i915 snd drm_kms_helper drm lpc_ich mei_me shpchp soundcore mei i2c_algo_bit 8250_fintek nuvoton_cir rc_core bnep rfcomm video soc_button_array mac_hid bluetooth binfmt_misc [ 2.692469] parport_pc ppdev lp parport nct6775 hwmon_vid coretemp btrfs xor raid6_pq e1000e ahci psmouse ptp libahci pps_core [ 2.692476] CPU: 3 PID: 599 Comm: irq/31-i915 Tainted: G OE 3.19.0-6-kvm #6 [ 2.692476] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H87 Pro4, BIOS P2.10 07/09/2014 [ 2.692477] ffffffffc07661b8 ffff880419b93c98 ffffffff817c6613 0000000000000007 [ 2.692479] ffff880419b93ce8 ffff880419b93cd8 ffffffff8107732a ffff880419b93cd8 [ 2.692480] ffff8804186e3000 ffff880419b88800 ffff880419b88800 0000000000000000 [ 2.692481] Call Trace: [ 2.692496] [<ffffffff817c6613>] dump_stack+0x4c/0x6e [ 2.692499] [<ffffffff8107732a>] warn_slowpath_common+0x8a/0xc0 [ 2.692501] [<ffffffff810773a6>] warn_slowpath_fmt+0x46/0x50 [ 2.692510] [<ffffffffc07111a2>] intel_check_page_flip+0xa2/0xf0 [i915] [ 2.692518] [<ffffffffc06de277>] ironlake_irq_handler+0x417/0x1000 [i915] [ 2.692521] [<ffffffff8109bc4d>] ? finish_task_switch+0x5d/0x100 [ 2.692523] [<ffffffff810d0340>] ? irq_thread_fn+0x50/0x50 [ 2.692525] [<ffffffff810d036d>] irq_forced_thread_fn+0x2d/0x70 [ 2.692526] [<ffffffff810d08b7>] irq_thread+0x137/0x160 [ 2.692527] [<ffffffff810d03e0>] ? wake_threads_waitq+0x30/0x30 [ 2.692529] [<ffffffff810d0780>] ? irq_thread_check_affinity+0x90/0x90 [ 2.692531] [<ffffffff810961eb>] kthread+0xdb/0x100 [ 2.692541] [<ffffffff81096110>] ? kthread_create_on_node+0x1c0/0x1c0 [ 2.692543] [<ffffffff817cd5bc>] ret_from_fork+0x7c/0xb0 [ 2.692545] [<ffffffff81096110>] ? kthread_create_on_node+0x1c0/0x1c0 [ 2.692546] ---[ end trace 3bc57ae31fe4d29b ]---

This doesn't look very healthy.  Get rid of those vbox drivers, we have no idea what they're doing and competing hypervisors often don't get along so well with each other.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4376 2015-03-02 19:47:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
aw wrote:

Then either VGA arbitration is not working or QEMU can't read the ROM on the GPU.  VGA arbitration not working was a problem we used to see using the proprietary nvidia driver in the host (driver would lock and never release the VGA arbitration lock).  Nvidia might have fixed this, dunno.  The latter ROM issue would produce an invalid ROM contents message in dmesg.  GeForce cards do not work as secondary GPUs in the guest.

So is there any workaround or any way I can fix this?

If you read the FAQ from the link below, you might find Question 9 interesting.  If you look at the comments, you'll find a link to that patch.  It looks like even the latest version of the Linux nvidia driver still has this problem.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4377 2015-03-02 19:54:44

noctavian
Member
Registered: 2013-07-11
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Great!, are you maintaining this database?

That would be me.

Offline

#4378 2015-03-02 20:30:03

terusus
Member
Registered: 2015-02-27
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
terusus wrote:
dmesg...

This doesn't look very healthy.  Get rid of those vbox drivers, we have no idea what they're doing and competing hypervisors often don't get along so well with each other.

OK. After apt-get purge virtualbox* I did a restart and tried again - no change. In the meantime I read about MTRR and added some boot parameters. Here is my new dmesg:

[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.19.0-6-kvm (root@UCPC) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #6 SMP PREEMPT Fri Feb 27 12:37:03 EAT 2015 (Ubuntu 3.19.0-6.6-kvm 3.19.0) [ 0.000000] Command line: BOOT_IMAGE=/@/boot/vmlinuz-3.19.0-6-kvm root=UUID=a3929651-e09b-4f1c-afa3-a5516b17eabb ro rootflags=subvol=@ i915.enable_hd_vgaarb=1 pcie_acs_override=multifunction intel_iommu=on enable_mtrr_cleanup mtrr_spare_reg_nr=1 mtrr_gran_size=2M mtrr_chunk_size=128M quiet splash crashkernel=384M-:128M vt.handoff=7 [ 0.000000] KERNEL supported cpus: [ 0.000000] Intel GenuineIntel [ 0.000000] AMD AuthenticAMD [ 0.000000] Centaur CentaurHauls [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d7ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009d800-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a956ffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000a9570000-0x00000000a9576fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000a9577000-0x00000000a9e4afff] usable [ 0.000000] BIOS-e820: [mem 0x00000000a9e4b000-0x00000000aa0e4fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000aa0e5000-0x00000000bc1e3fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bc1e4000-0x00000000bc3e9fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bc3ea000-0x00000000bc426fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bc427000-0x00000000bc4cefff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bc4cf000-0x00000000bcffefff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bcfff000-0x00000000bcffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bf800000-0x00000000cf9fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000042e5fffff] usable [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.7 present. [ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./H87 Pro4, BIOS P2.10 07/09/2014 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] AGP: No AGP bridge found [ 0.000000] e820: last_pfn = 0x42e600 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-CFFFF write-protect [ 0.000000] D0000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000000000 mask 7C00000000 write-back [ 0.000000] 1 base 0400000000 mask 7FE0000000 write-back [ 0.000000] 2 base 0420000000 mask 7FF8000000 write-back [ 0.000000] 3 base 0428000000 mask 7FFC000000 write-back [ 0.000000] 4 base 042C000000 mask 7FFE000000 write-back [ 0.000000] 5 base 042E000000 mask 7FFFC00000 write-back [ 0.000000] 6 base 042E400000 mask 7FFFE00000 write-back [ 0.000000] 7 base 00C0000000 mask 7FC0000000 uncachable [ 0.000000] 8 base 00BF800000 mask 7FFF800000 uncachable [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WC UC- UC [ 0.000000] original variable MTRRs [ 0.000000] reg 0, base: 0GB, range: 16GB, type WB [ 0.000000] reg 1, base: 16GB, range: 512MB, type WB [ 0.000000] reg 2, base: 16896MB, range: 128MB, type WB [ 0.000000] reg 3, base: 17024MB, range: 64MB, type WB [ 0.000000] reg 4, base: 17088MB, range: 32MB, type WB [ 0.000000] reg 5, base: 17120MB, range: 4MB, type WB [ 0.000000] reg 6, base: 17124MB, range: 2MB, type WB [ 0.000000] reg 7, base: 3GB, range: 1GB, type UC [ 0.000000] reg 8, base: 3064MB, range: 8MB, type UC [ 0.000000] total RAM covered: 16094M [ 0.000000] gran_size: 2M chunk_size: 128M num_reg: 10 lose cover RAM: 0G [ 0.000000] New variable MTRRs [ 0.000000] reg 0, base: 0GB, range: 2GB, type WB [ 0.000000] reg 1, base: 2GB, range: 1GB, type WB [ 0.000000] reg 2, base: 3064MB, range: 8MB, type UC [ 0.000000] reg 3, base: 4GB, range: 4GB, type WB [ 0.000000] reg 4, base: 8GB, range: 8GB, type WB [ 0.000000] reg 5, base: 16GB, range: 512MB, type WB [ 0.000000] reg 6, base: 16896MB, range: 256MB, type WB [ 0.000000] reg 7, base: 17126MB, range: 2MB, type UC [ 0.000000] reg 8, base: 17128MB, range: 8MB, type UC [ 0.000000] reg 9, base: 17136MB, range: 16MB, type UC [ 0.000000] e820: update [mem 0xbf800000-0xffffffff] usable ==> reserved [ 0.000000] e820: last_pfn = 0xbd000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fd7f0-0x000fd7ff] mapped at [ffff8800000fd7f0] [ 0.000000] Scanning 1 areas for low memory corruption [ 0.000000] Base memory trampoline at [ffff880000097000] 97000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x01fd8000, 0x01fd8fff] PGTABLE [ 0.000000] BRK [0x01fd9000, 0x01fd9fff] PGTABLE [ 0.000000] BRK [0x01fda000, 0x01fdafff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x42e400000-0x42e5fffff] [ 0.000000] [mem 0x42e400000-0x42e5fffff] page 2M [ 0.000000] BRK [0x01fdb000, 0x01fdbfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x420000000-0x42e3fffff] [ 0.000000] [mem 0x420000000-0x42e3fffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x400000000-0x41fffffff] [ 0.000000] [mem 0x400000000-0x41fffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0xa956ffff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0x3fffffff] page 2M [ 0.000000] [mem 0x40000000-0x7fffffff] page 1G [ 0.000000] [mem 0x80000000-0xa93fffff] page 2M [ 0.000000] [mem 0xa9400000-0xa956ffff] page 4k [ 0.000000] init_memory_mapping: [mem 0xa9577000-0xa9e4afff] [ 0.000000] [mem 0xa9577000-0xa95fffff] page 4k [ 0.000000] [mem 0xa9600000-0xa9dfffff] page 2M [ 0.000000] [mem 0xa9e00000-0xa9e4afff] page 4k [ 0.000000] BRK [0x01fdc000, 0x01fdcfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0xaa0e5000-0xbc1e3fff] [ 0.000000] [mem 0xaa0e5000-0xaa1fffff] page 4k [ 0.000000] [mem 0xaa200000-0xbbffffff] page 2M [ 0.000000] [mem 0xbc000000-0xbc1e3fff] page 4k [ 0.000000] BRK [0x01fdd000, 0x01fddfff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0xbc3ea000-0xbc426fff] [ 0.000000] [mem 0xbc3ea000-0xbc426fff] page 4k [ 0.000000] init_memory_mapping: [mem 0xbcfff000-0xbcffffff] [ 0.000000] [mem 0xbcfff000-0xbcffffff] page 4k [ 0.000000] init_memory_mapping: [mem 0x100000000-0x3ffffffff] [ 0.000000] [mem 0x100000000-0x3ffffffff] page 1G [ 0.000000] RAMDISK: [mem 0x357ee000-0x36beefff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F0490 000024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 0x00000000BC4AB088 00008C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 0x00000000BC4B57E8 00010C (v05 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: DSDT 0x00000000BC4AB1A8 00A63B (v02 ALASKA A M I 00000210 INTL 20091112) [ 0.000000] ACPI: FACS 0x00000000BC4CD080 000040 [ 0.000000] ACPI: APIC 0x00000000BC4B58F8 000072 (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 0x00000000BC4B5970 000044 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: SSDT 0x00000000BC4B59B8 000539 (v01 PmRef Cpu0Ist 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 0x00000000BC4B5EF8 000AD8 (v01 PmRef CpuPm 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 0x00000000BC4B69D0 0001C7 (v01 PmRef LakeTiny 00003000 INTL 20051117) [ 0.000000] ACPI: MCFG 0x00000000BC4B6B98 00003C (v01 ALASKA A M I 01072009 MSFT 00000097) [ 0.000000] ACPI: HPET 0x00000000BC4B6BD8 000038 (v01 ALASKA A M I 01072009 AMI. 00000005) [ 0.000000] ACPI: SSDT 0x00000000BC4B6C10 00036D (v01 SataRe SataTabl 00001000 INTL 20091112) [ 0.000000] ACPI: SSDT 0x00000000BC4B6F80 003493 (v01 SaSsdt SaSsdt 00003000 INTL 20091112) [ 0.000000] ACPI: AAFT 0x00000000BC4BA418 0002BB (v01 ALASKA OEMAAFT 01072009 MSFT 00000097) [ 0.000000] ACPI: ASF! 0x00000000BC4BA6D8 0000A5 (v32 INTEL HCG 00000001 TFSM 000F4240) [ 0.000000] ACPI: DMAR 0x00000000BC4BA780 0000B8 (v01 INTEL HSW 00000001 INTL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000042e5fffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x42e5f5000-0x42e5f9fff] [ 0.000000] Reserving 128MB of memory at 720MB for crashkernel (System RAM: 16037MB) [ 0.000000] [ffffea0000000000-ffffea0010bfffff] PMD -> [ffff88041e000000-ffff88042dbfffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x42e5fffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009cfff] [ 0.000000] node 0: [mem 0x00100000-0xa956ffff] [ 0.000000] node 0: [mem 0xa9577000-0xa9e4afff] [ 0.000000] node 0: [mem 0xaa0e5000-0xbc1e3fff] [ 0.000000] node 0: [mem 0xbc3ea000-0xbc426fff] [ 0.000000] node 0: [mem 0xbcfff000-0xbcffffff] [ 0.000000] node 0: [mem 0x100000000-0x42e5fffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x42e5fffff] [ 0.000000] On node 0 totalpages: 4105501 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3996 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 11967 pages used for memmap [ 0.000000] DMA32 zone: 765825 pages, LIFO batch:31 [ 0.000000] Normal zone: 52120 pages used for memmap [ 0.000000] Normal zone: 3335680 pages, LIFO batch:31 [ 0.000000] Reserving Intel graphics stolen memory at 0xbfa00000-0xcf9fffff [ 0.000000] ACPI: PM-Timer IO Port: 0x1808 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009d000-0x0009dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xa9570000-0xa9576fff] [ 0.000000] PM: Registered nosave memory: [mem 0xa9e4b000-0xaa0e4fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbc1e4000-0xbc3e9fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbc427000-0xbc4cefff] [ 0.000000] PM: Registered nosave memory: [mem 0xbc4cf000-0xbcffefff] [ 0.000000] PM: Registered nosave memory: [mem 0xbd000000-0xbf7fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbf800000-0xcf9fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xcfa00000-0xf7ffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xf8000000-0xfbffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfecfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed00000-0xfed03fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed04000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff] [ 0.000000] e820: [mem 0xcfa00000-0xf7ffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 31 pages/cpu @ffff88042e200000 s87616 r8192 d31168 u524288 [ 0.000000] pcpu-alloc: s87616 r8192 d31168 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 4041329 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/@/boot/vmlinuz-3.19.0-6-kvm root=UUID=a3929651-e09b-4f1c-afa3-a5516b17eabb ro rootflags=subvol=@ i915.enable_hd_vgaarb=1 pcie_acs_override=multifunction intel_iommu=on enable_mtrr_cleanup mtrr_spare_reg_nr=1 mtrr_gran_size=2M mtrr_chunk_size=128M quiet splash crashkernel=384M-:128M vt.handoff=7 [ 0.000000] Intel-IOMMU: enabled [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] Calgary: detecting Calgary via BIOS EBDA area [ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing! [ 0.000000] Memory: 15929224K/16422004K available (8010K kernel code, 1219K rwdata, 3752K rodata, 1376K init, 1300K bss, 492780K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS:16640 nr_irqs:456 16 [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] vt handoff: transparent VT on vt#7 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 3399.297 MHz processor [ 0.000020] Calibrating delay loop (skipped), value calculated using timer frequency.. 6798.59 BogoMIPS (lpj=3399297) [ 0.000022] pid_max: default: 32768 minimum: 301 [ 0.000025] ACPI: Core revision 20141107 [ 0.006733] ACPI: All ACPI Tables successfully acquired [ 0.014052] Security Framework initialized [ 0.014062] AppArmor: AppArmor initialized [ 0.014062] Yama: becoming mindful. [ 0.014711] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.016608] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.017436] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.017447] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.017612] Initializing cgroup subsys memory [ 0.017616] Initializing cgroup subsys devices [ 0.017617] Initializing cgroup subsys freezer [ 0.017619] Initializing cgroup subsys net_cls [ 0.017620] Initializing cgroup subsys blkio [ 0.017621] Initializing cgroup subsys perf_event [ 0.017623] Initializing cgroup subsys net_prio [ 0.017624] Initializing cgroup subsys hugetlb [ 0.017640] CPU: Physical Processor ID: 0 [ 0.017641] CPU: Processor Core ID: 0 [ 0.018391] mce: CPU supports 9 MCE banks [ 0.018401] CPU0: Thermal monitoring enabled (TM1) [ 0.018409] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024 Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4 [ 0.018483] Freeing SMP alternatives memory: 28K (ffffffff81e8a000 - ffffffff81e91000) [ 0.018970] ftrace: allocating 30017 entries in 118 pages [ 0.027605] dmar: Host address width 39 [ 0.027606] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.027613] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.027614] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.027617] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.027617] dmar: RMRR base: 0x000000bc371000 end: 0x000000bc37dfff [ 0.027618] dmar: RMRR base: 0x000000bf800000 end: 0x000000cf9fffff [ 0.027682] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.027683] HPET id 0 under DRHD base 0xfed91000 [ 0.027684] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.027791] Enabled IRQ remapping in x2apic mode [ 0.027791] Enabling x2apic [ 0.027792] Enabled x2apic [ 0.027796] Switched APIC routing to cluster x2apic. [ 0.028210] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.038204] smpboot: CPU0: Intel(R) Core(TM) i5-4670 CPU @ 3.40GHz (fam: 06, model: 3c, stepping: 03) [ 0.038209] TSC deadline timer enabled [ 0.038226] Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. [ 0.038239] ... version: 3 [ 0.038239] ... bit width: 48 [ 0.038240] ... generic registers: 8 [ 0.038241] ... value mask: 0000ffffffffffff [ 0.038241] ... max period: 0000ffffffffffff [ 0.038242] ... fixed-purpose events: 3 [ 0.038242] ... event mask: 00000007000000ff [ 0.046313] x86: Booting SMP configuration: [ 0.046314] .... node #0, CPUs: #1 [ 0.060158] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.062182] #2 #3 [ 0.092063] x86: Booted up 1 node, 4 CPUs [ 0.092065] smpboot: Total of 4 processors activated (27194.37 BogoMIPS) [ 0.094761] devtmpfs: initialized [ 0.096674] evm: security.selinux [ 0.096675] evm: security.SMACK64 [ 0.096676] evm: security.SMACK64EXEC [ 0.096676] evm: security.SMACK64TRANSMUTE [ 0.096677] evm: security.SMACK64MMAP [ 0.096677] evm: security.ima [ 0.096678] evm: security.capability [ 0.096731] PM: Registering ACPI NVS region [mem 0xa9570000-0xa9576fff] (28672 bytes) [ 0.096732] PM: Registering ACPI NVS region [mem 0xbc427000-0xbc4cefff] (688128 bytes) [ 0.096830] pinctrl core: initialized pinctrl subsystem [ 0.096899] RTC time: 23:15:12, date: 03/02/15 [ 0.096969] NET: Registered protocol family 16 [ 0.101662] cpuidle: using governor ladder [ 0.104667] cpuidle: using governor menu [ 0.104721] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.104722] ACPI: bus type PCI registered [ 0.104723] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.104770] PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000) [ 0.104771] PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820 [ 0.104823] PCI: Using configuration type 1 for base access [ 0.108913] ACPI: Added _OSI(Module Device) [ 0.108914] ACPI: Added _OSI(Processor Device) [ 0.108915] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.108915] ACPI: Added _OSI(Processor Aggregator Device) [ 0.111207] ACPI: Executed 1 blocks of module-level executable AML code [ 0.112634] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored [ 0.112933] ACPI: Dynamic OEM Table Load: [ 0.112937] ACPI: SSDT 0xFFFF88041BFE4800 0003D3 (v01 PmRef Cpu0Cst 00003001 INTL 20051117) [ 0.113357] ACPI: Dynamic OEM Table Load: [ 0.113360] ACPI: SSDT 0xFFFF88041BF4A800 0005AA (v01 PmRef ApIst 00003000 INTL 20051117) [ 0.113812] ACPI: Dynamic OEM Table Load: [ 0.113814] ACPI: SSDT 0xFFFF88041BFDCC00 000119 (v01 PmRef ApCst 00003000 INTL 20051117) [ 0.114459] ACPI: Interpreter enabled [ 0.114464] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20141107/hwxface-580) [ 0.114468] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20141107/hwxface-580) [ 0.114479] ACPI: (supports S0 S3 S4 S5) [ 0.114480] ACPI: Using IOAPIC for interrupt routing [ 0.114514] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.119745] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3e]) [ 0.119749] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.119877] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug PME] [ 0.119957] acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability] [ 0.120291] PCI host bridge to bus 0000:00 [ 0.120293] pci_bus 0000:00: root bus resource [bus 00-3e] [ 0.120294] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] [ 0.120295] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.120296] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.120297] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff] [ 0.120298] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff] [ 0.120298] pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff] [ 0.120299] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff] [ 0.120300] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff] [ 0.120301] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff] [ 0.120302] pci_bus 0000:00: root bus resource [mem 0xcfa00000-0xfeafffff] [ 0.120307] pci 0000:00:00.0: [8086:0c00] type 00 class 0x060000 [ 0.120365] pci 0000:00:01.0: [8086:0c01] type 01 class 0x060400 [ 0.120391] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 0.120439] pci 0000:00:02.0: [8086:0412] type 00 class 0x030000 [ 0.120446] pci 0000:00:02.0: reg 0x10: [mem 0xee400000-0xee7fffff 64bit] [ 0.120450] pci 0000:00:02.0: reg 0x18: [mem 0xd0000000-0xdfffffff 64bit pref] [ 0.120453] pci 0000:00:02.0: reg 0x20: [io 0xf000-0xf03f] [ 0.120505] pci 0000:00:03.0: [8086:0c0c] type 00 class 0x040300 [ 0.120511] pci 0000:00:03.0: reg 0x10: [mem 0xee834000-0xee837fff 64bit] [ 0.120586] pci 0000:00:14.0: [8086:8c31] type 00 class 0x0c0330 [ 0.120601] pci 0000:00:14.0: reg 0x10: [mem 0xee820000-0xee82ffff 64bit] [ 0.120652] pci 0000:00:14.0: PME# supported from D3hot D3cold [ 0.120682] pci 0000:00:14.0: System wakeup disabled by ACPI [ 0.120707] pci 0000:00:16.0: [8086:8c3a] type 00 class 0x078000 [ 0.120723] pci 0000:00:16.0: reg 0x10: [mem 0xee840000-0xee84000f 64bit] [ 0.120777] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 0.120830] pci 0000:00:19.0: [8086:153b] type 00 class 0x020000 [ 0.120844] pci 0000:00:19.0: reg 0x10: [mem 0xee800000-0xee81ffff] [ 0.120850] pci 0000:00:19.0: reg 0x14: [mem 0xee83d000-0xee83dfff] [ 0.120857] pci 0000:00:19.0: reg 0x18: [io 0xf080-0xf09f] [ 0.120908] pci 0000:00:19.0: PME# supported from D0 D3hot D3cold [ 0.120935] pci 0000:00:19.0: System wakeup disabled by ACPI [ 0.120961] pci 0000:00:1a.0: [8086:8c2d] type 00 class 0x0c0320 [ 0.120978] pci 0000:00:1a.0: reg 0x10: [mem 0xee83c000-0xee83c3ff] [ 0.121056] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 0.121093] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 0.121118] pci 0000:00:1b.0: [8086:8c20] type 00 class 0x040300 [ 0.121130] pci 0000:00:1b.0: reg 0x10: [mem 0xee830000-0xee833fff 64bit] [ 0.121189] pci 0000:00:1b.0: PME# supported from D0 D3hot D3cold [ 0.121216] pci 0000:00:1b.0: System wakeup disabled by ACPI [ 0.121238] pci 0000:00:1c.0: [8086:8c10] type 01 class 0x060400 [ 0.121295] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 0.121311] pci 0000:00:1c.0: Enabling MPC IRBNCE [ 0.121313] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled [ 0.121333] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 0.121358] pci 0000:00:1c.6: [8086:244e] type 01 class 0x060401 [ 0.121415] pci 0000:00:1c.6: PME# supported from D0 D3hot D3cold [ 0.121444] pci 0000:00:1c.6: System wakeup disabled by ACPI [ 0.121472] pci 0000:00:1d.0: [8086:8c26] type 00 class 0x0c0320 [ 0.121489] pci 0000:00:1d.0: reg 0x10: [mem 0xee83b000-0xee83b3ff] [ 0.121565] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 0.121600] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 0.121626] pci 0000:00:1f.0: [8086:8c4a] type 00 class 0x060100 [ 0.121763] pci 0000:00:1f.2: [8086:8c02] type 00 class 0x010601 [ 0.121775] pci 0000:00:1f.2: reg 0x10: [io 0xf0d0-0xf0d7] [ 0.121781] pci 0000:00:1f.2: reg 0x14: [io 0xf0c0-0xf0c3] [ 0.121786] pci 0000:00:1f.2: reg 0x18: [io 0xf0b0-0xf0b7] [ 0.121792] pci 0000:00:1f.2: reg 0x1c: [io 0xf0a0-0xf0a3] [ 0.121798] pci 0000:00:1f.2: reg 0x20: [io 0xf060-0xf07f] [ 0.121804] pci 0000:00:1f.2: reg 0x24: [mem 0xee83a000-0xee83a7ff] [ 0.121835] pci 0000:00:1f.2: PME# supported from D3hot [ 0.121877] pci 0000:00:1f.3: [8086:8c22] type 00 class 0x0c0500 [ 0.121888] pci 0000:00:1f.3: reg 0x10: [mem 0xee839000-0xee8390ff 64bit] [ 0.121905] pci 0000:00:1f.3: reg 0x20: [io 0xf040-0xf05f] [ 0.121988] pci 0000:01:00.0: [10de:1200] type 00 class 0x030000 [ 0.121995] pci 0000:01:00.0: reg 0x10: [mem 0xec000000-0xedffffff] [ 0.122002] pci 0000:01:00.0: reg 0x14: [mem 0xe0000000-0xe7ffffff 64bit pref] [ 0.122009] pci 0000:01:00.0: reg 0x1c: [mem 0xe8000000-0xebffffff 64bit pref] [ 0.122013] pci 0000:01:00.0: reg 0x24: [io 0xe000-0xe07f] [ 0.122017] pci 0000:01:00.0: reg 0x30: [mem 0xee000000-0xee07ffff pref] [ 0.122083] pci 0000:01:00.1: [10de:0e0c] type 00 class 0x040300 [ 0.122091] pci 0000:01:00.1: reg 0x10: [mem 0xee080000-0xee083fff] [ 0.123688] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.123691] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.123693] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xee0fffff] [ 0.123742] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.123803] pci 0000:03:00.0: [1b21:1080] type 01 class 0x060401 [ 0.123901] pci 0000:03:00.0: System wakeup disabled by ACPI [ 0.123922] pci 0000:00:1c.6: PCI bridge to [bus 03-04] (subtractive decode) [ 0.123929] pci 0000:00:1c.6: bridge window [io 0x0000-0x0cf7] (subtractive decode) [ 0.123930] pci 0000:00:1c.6: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.123931] pci 0000:00:1c.6: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.123932] pci 0000:00:1c.6: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) [ 0.123933] pci 0000:00:1c.6: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 0.123934] pci 0000:00:1c.6: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 0.123935] pci 0000:00:1c.6: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 0.123935] pci 0000:00:1c.6: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 0.123936] pci 0000:00:1c.6: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 0.123937] pci 0000:00:1c.6: bridge window [mem 0xcfa00000-0xfeafffff] (subtractive decode) [ 0.124040] pci 0000:03:00.0: PCI bridge to [bus 04] (subtractive decode) [ 0.124056] pci 0000:03:00.0: bridge window [io 0x0000-0x0cf7] (subtractive decode) [ 0.124057] pci 0000:03:00.0: bridge window [io 0x0d00-0xffff] (subtractive decode) [ 0.124057] pci 0000:03:00.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) [ 0.124058] pci 0000:03:00.0: bridge window [mem 0x000d0000-0x000d3fff] (subtractive decode) [ 0.124059] pci 0000:03:00.0: bridge window [mem 0x000d4000-0x000d7fff] (subtractive decode) [ 0.124060] pci 0000:03:00.0: bridge window [mem 0x000d8000-0x000dbfff] (subtractive decode) [ 0.124061] pci 0000:03:00.0: bridge window [mem 0x000dc000-0x000dffff] (subtractive decode) [ 0.124062] pci 0000:03:00.0: bridge window [mem 0x000e0000-0x000e3fff] (subtractive decode) [ 0.124063] pci 0000:03:00.0: bridge window [mem 0x000e4000-0x000e7fff] (subtractive decode) [ 0.124064] pci 0000:03:00.0: bridge window [mem 0xcfa00000-0xfeafffff] (subtractive decode) [ 0.124085] acpi PNP0A08:00: Disabling ASPM (FADT indicates it is unsupported) [ 0.124573] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.124602] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.124629] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 *10 11 12 14 15) [ 0.124661] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 *5 6 10 11 12 14 15) [ 0.124688] ACPI: PCI Interrupt Link [LNKE] (IRQs *3 4 5 6 10 11 12 14 15) [ 0.124715] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 0.124741] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 *11 12 14 15) [ 0.124768] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *5 6 10 11 12 14 15) [ 0.124889] ACPI: Enabled 4 GPEs in block 00 to 3F [ 0.124951] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 0.124953] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.124956] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.124957] vgaarb: loaded [ 0.124957] vgaarb: bridge control possible 0000:01:00.0 [ 0.124958] vgaarb: no bridge control possible 0000:00:02.0 [ 0.125082] SCSI subsystem initialized [ 0.125111] libata version 3.00 loaded. [ 0.125134] ACPI: bus type USB registered [ 0.125145] usbcore: registered new interface driver usbfs [ 0.125150] usbcore: registered new interface driver hub [ 0.125160] usbcore: registered new device driver usb [ 0.125243] PCI: Using ACPI for IRQ routing [ 0.126428] PCI: pci_cache_line_size set to 64 bytes [ 0.126468] e820: reserve RAM buffer [mem 0x0009d800-0x0009ffff] [ 0.126469] e820: reserve RAM buffer [mem 0xa9570000-0xabffffff] [ 0.126470] e820: reserve RAM buffer [mem 0xa9e4b000-0xabffffff] [ 0.126470] e820: reserve RAM buffer [mem 0xbc1e4000-0xbfffffff] [ 0.126471] e820: reserve RAM buffer [mem 0xbc427000-0xbfffffff] [ 0.126472] e820: reserve RAM buffer [mem 0xbd000000-0xbfffffff] [ 0.126473] e820: reserve RAM buffer [mem 0x42e600000-0x42fffffff] [ 0.126541] NetLabel: Initializing [ 0.126542] NetLabel: domain hash size = 128 [ 0.126542] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.126549] NetLabel: unlabeled traffic allowed by default [ 0.126609] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 0.126613] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 0.128639] Switched to clocksource hpet [ 0.132518] AppArmor: AppArmor Filesystem Enabled [ 0.132551] pnp: PnP ACPI init [ 0.132596] system 00:00: [mem 0xfed40000-0xfed44fff] has been reserved [ 0.132598] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 0.132695] system 00:01: [io 0x0680-0x069f] has been reserved [ 0.132696] system 00:01: [io 0xffff] has been reserved [ 0.132698] system 00:01: [io 0xffff] has been reserved [ 0.132698] system 00:01: [io 0xffff] has been reserved [ 0.132699] system 00:01: [io 0x1c00-0x1cfe] has been reserved [ 0.132700] system 00:01: [io 0x1d00-0x1dfe] has been reserved [ 0.132701] system 00:01: [io 0x1e00-0x1efe] has been reserved [ 0.132702] system 00:01: [io 0x1f00-0x1ffe] has been reserved [ 0.132703] system 00:01: [io 0x1800-0x18fe] could not be reserved [ 0.132704] system 00:01: [io 0x164e-0x164f] has been reserved [ 0.132706] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.132723] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.132751] system 00:03: [io 0x1854-0x1857] has been reserved [ 0.132752] system 00:03: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 0.132795] system 00:04: [io 0x0290-0x029f] has been reserved [ 0.132796] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.132914] system 00:05: [io 0x04d0-0x04d1] has been reserved [ 0.132915] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.133004] pnp 00:06: [dma 0 disabled] [ 0.133027] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.133270] system 00:07: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 0.133271] system 00:07: [mem 0xfed10000-0xfed17fff] has been reserved [ 0.133272] system 00:07: [mem 0xfed18000-0xfed18fff] has been reserved [ 0.133273] system 00:07: [mem 0xfed19000-0xfed19fff] has been reserved [ 0.133274] system 00:07: [mem 0xf8000000-0xfbffffff] has been reserved [ 0.133275] system 00:07: [mem 0xfed20000-0xfed3ffff] has been reserved [ 0.133276] system 00:07: [mem 0xfed90000-0xfed93fff] could not be reserved [ 0.133277] system 00:07: [mem 0xfed45000-0xfed8ffff] has been reserved [ 0.133278] system 00:07: [mem 0xff000000-0xffffffff] has been reserved [ 0.133280] system 00:07: [mem 0xfee00000-0xfeefffff] could not be reserved [ 0.133281] system 00:07: [mem 0xf7fef000-0xf7feffff] has been reserved [ 0.133282] system 00:07: [mem 0xf7ff0000-0xf7ff0fff] has been reserved [ 0.133283] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active) [ 0.133426] pnp: PnP ACPI: found 8 devices [ 0.139164] pci 0000:00:01.0: PCI bridge to [bus 01] [ 0.139165] pci 0000:00:01.0: bridge window [io 0xe000-0xefff] [ 0.139168] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xee0fffff] [ 0.139171] pci 0000:00:1c.0: PCI bridge to [bus 02] [ 0.139179] pci 0000:03:00.0: PCI bridge to [bus 04] [ 0.139194] pci 0000:00:1c.6: PCI bridge to [bus 03-04] [ 0.139202] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7] [ 0.139204] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff] [ 0.139204] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff] [ 0.139205] pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.139206] pci_bus 0000:00: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.139207] pci_bus 0000:00: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.139208] pci_bus 0000:00: resource 10 [mem 0x000dc000-0x000dffff] [ 0.139209] pci_bus 0000:00: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.139210] pci_bus 0000:00: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.139211] pci_bus 0000:00: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.139212] pci_bus 0000:01: resource 0 [io 0xe000-0xefff] [ 0.139213] pci_bus 0000:01: resource 1 [mem 0xe0000000-0xee0fffff] [ 0.139214] pci_bus 0000:03: resource 4 [io 0x0000-0x0cf7] [ 0.139215] pci_bus 0000:03: resource 5 [io 0x0d00-0xffff] [ 0.139216] pci_bus 0000:03: resource 6 [mem 0x000a0000-0x000bffff] [ 0.139217] pci_bus 0000:03: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.139218] pci_bus 0000:03: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.139218] pci_bus 0000:03: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.139219] pci_bus 0000:03: resource 10 [mem 0x000dc000-0x000dffff] [ 0.139220] pci_bus 0000:03: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.139221] pci_bus 0000:03: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.139222] pci_bus 0000:03: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.139223] pci_bus 0000:04: resource 4 [io 0x0000-0x0cf7] [ 0.139224] pci_bus 0000:04: resource 5 [io 0x0d00-0xffff] [ 0.139225] pci_bus 0000:04: resource 6 [mem 0x000a0000-0x000bffff] [ 0.139226] pci_bus 0000:04: resource 7 [mem 0x000d0000-0x000d3fff] [ 0.139227] pci_bus 0000:04: resource 8 [mem 0x000d4000-0x000d7fff] [ 0.139227] pci_bus 0000:04: resource 9 [mem 0x000d8000-0x000dbfff] [ 0.139228] pci_bus 0000:04: resource 10 [mem 0x000dc000-0x000dffff] [ 0.139229] pci_bus 0000:04: resource 11 [mem 0x000e0000-0x000e3fff] [ 0.139230] pci_bus 0000:04: resource 12 [mem 0x000e4000-0x000e7fff] [ 0.139231] pci_bus 0000:04: resource 13 [mem 0xcfa00000-0xfeafffff] [ 0.139247] NET: Registered protocol family 2 [ 0.139369] TCP established hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.139504] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 0.139600] TCP: Hash tables configured (established 131072 bind 65536) [ 0.139610] TCP: reno registered [ 0.139622] UDP hash table entries: 8192 (order: 6, 262144 bytes) [ 0.139664] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes) [ 0.139712] NET: Registered protocol family 1 [ 0.139723] pci 0000:00:02.0: Video device with shadowed ROM [ 0.169698] PCI: CLS 64 bytes, default 64 [ 0.169730] Trying to unpack rootfs image as initramfs... [ 0.370675] Freeing initrd memory: 20484K (ffff8800357ee000 - ffff880036bef000) [ 0.370691] DMAR: No ATSR found [ 0.370760] IOMMU: dmar0 using Queued invalidation [ 0.370761] IOMMU: dmar1 using Queued invalidation [ 0.370762] IOMMU: Setting RMRR: [ 0.370770] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff] [ 0.371950] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc371000 - 0xbc37dfff] [ 0.371970] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc371000 - 0xbc37dfff] [ 0.371988] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc371000 - 0xbc37dfff] [ 0.372000] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.372005] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.372113] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 0.373572] RAPL PMU detected, hw unit 2^-14 Joules, API unit is 2^-32 Joules, 4 fixed counters 655360 ms ovfl timer [ 0.373622] microcode: CPU0 sig=0x306c3, pf=0x2, revision=0x9 [ 0.373626] microcode: CPU1 sig=0x306c3, pf=0x2, revision=0x9 [ 0.373631] microcode: CPU2 sig=0x306c3, pf=0x2, revision=0x9 [ 0.373636] microcode: CPU3 sig=0x306c3, pf=0x2, revision=0x9 [ 0.373673] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 0.373689] Scanning for low memory corruption every 60 seconds [ 0.373878] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 0.373892] Initialise system trusted keyring [ 0.373906] audit: initializing netlink subsys (disabled) [ 0.373916] audit: type=2000 audit(1425338112.355:1): initialized [ 0.374125] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.374942] zpool: loaded [ 0.374943] zbud: loaded [ 0.375049] VFS: Disk quotas dquot_6.5.2 [ 0.375069] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.375330] fuse init (API version 7.23) [ 0.375419] Key type big_key registered [ 0.375726] Key type asymmetric registered [ 0.375728] Asymmetric key parser 'x509' registered [ 0.375748] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.375775] io scheduler noop registered [ 0.375777] io scheduler deadline registered (default) [ 0.375793] io scheduler cfq registered [ 0.376008] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.376018] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.376038] vesafb: mode is 1280x1024x32, linelength=5120, pages=0 [ 0.376038] vesafb: scrolling: redraw [ 0.376039] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0 [ 0.376050] vesafb: framebuffer at 0xd0000000, mapped to 0xffffc90005c00000, using 5120k, total 5120k [ 0.376102] Console: switching to colour frame buffer device 160x64 [ 0.376112] fb0: VESA VGA frame buffer device [ 0.376123] intel_idle: MWAIT substates: 0x42120 [ 0.376124] intel_idle: v0.4 model 0x3C [ 0.376125] intel_idle: lapic_timer_reliable_states 0xffffffff [ 0.376263] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0 [ 0.376266] ACPI: Power Button [PWRB] [ 0.376286] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 [ 0.376287] ACPI: Power Button [PWRF] [ 0.376400] GHES: HEST is not enabled! [ 0.376466] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 0.396967] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 0.398034] Linux agpgart interface v0.103 [ 0.398771] brd: module loaded [ 0.399147] loop: module loaded [ 0.399258] libphy: Fixed MDIO Bus: probed [ 0.399260] tun: Universal TUN/TAP device driver, 1.6 [ 0.399261] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 0.399287] PPP generic driver version 2.4.2 [ 0.399331] VFIO - User Level meta-driver version: 0.3 [ 0.399668] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 0.399672] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1 [ 0.399805] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported [ 0.399917] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.399919] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.399920] usb usb1: Product: xHCI Host Controller [ 0.399921] usb usb1: Manufacturer: Linux 3.19.0-6-kvm xhci-hcd [ 0.399922] usb usb1: SerialNumber: 0000:00:14.0 [ 0.400005] hub 1-0:1.0: USB hub found [ 0.400019] hub 1-0:1.0: 14 ports detected [ 0.401461] xhci_hcd 0000:00:14.0: xHCI Host Controller [ 0.401464] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ 0.401487] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 [ 0.401488] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.401489] usb usb2: Product: xHCI Host Controller [ 0.401490] usb usb2: Manufacturer: Linux 3.19.0-6-kvm xhci-hcd [ 0.401490] usb usb2: SerialNumber: 0000:00:14.0 [ 0.401584] hub 2-0:1.0: USB hub found [ 0.401595] hub 2-0:1.0: 6 ports detected [ 0.402241] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.402245] ehci-pci: EHCI PCI platform driver [ 0.402303] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 0.402306] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 3 [ 0.402315] ehci-pci 0000:00:1a.0: debug port 2 [ 0.406229] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 0.406341] ehci-pci 0000:00:1a.0: irq 16, io mem 0xee83c000 [ 0.411453] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 0.411492] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.411493] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.411494] usb usb3: Product: EHCI Host Controller [ 0.411495] usb usb3: Manufacturer: Linux 3.19.0-6-kvm ehci_hcd [ 0.411496] usb usb3: SerialNumber: 0000:00:1a.0 [ 0.411634] hub 3-0:1.0: USB hub found [ 0.411638] hub 3-0:1.0: 2 ports detected [ 0.411773] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 0.411776] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 4 [ 0.411785] ehci-pci 0000:00:1d.0: debug port 2 [ 0.415682] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 0.415819] ehci-pci 0000:00:1d.0: irq 23, io mem 0xee83b000 [ 0.421443] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 0.421472] usb usb4: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.421473] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.421474] usb usb4: Product: EHCI Host Controller [ 0.421475] usb usb4: Manufacturer: Linux 3.19.0-6-kvm ehci_hcd [ 0.421476] usb usb4: SerialNumber: 0000:00:1d.0 [ 0.421620] hub 4-0:1.0: USB hub found [ 0.421623] hub 4-0:1.0: 2 ports detected [ 0.421696] ehci-platform: EHCI generic platform driver [ 0.421702] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.421706] ohci-pci: OHCI PCI platform driver [ 0.421712] ohci-platform: OHCI generic platform driver [ 0.421717] uhci_hcd: USB Universal Host Controller Interface driver [ 0.421746] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 0.424532] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.424551] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.424745] mousedev: PS/2 mouse device common for all mice [ 0.424958] rtc_cmos 00:02: RTC can wake from S4 [ 0.425068] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0 [ 0.425127] rtc_cmos 00:02: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 0.425137] i2c /dev entries driver [ 0.425176] device-mapper: uevent: version 1.0.3 [ 0.425231] device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com [ 0.425243] Intel P-state driver initializing. [ 0.425355] Consider also installing thermald for improved thermal control. [ 0.425358] ledtrig-cpu: registered to indicate activity on CPUs [ 0.425571] PCCT header not found. [ 0.425571] ACPI PCC probe failed. [ 0.425621] TCP: cubic registered [ 0.425673] NET: Registered protocol family 10 [ 0.425854] NET: Registered protocol family 17 [ 0.425862] Key type dns_resolver registered [ 0.426103] Loading compiled-in X.509 certificates [ 0.426610] Loaded X.509 cert 'Magrathea: Glacier signing key: e053bb5652cbcfaa2aef7163e2bf6c27749be754' [ 0.426616] registered taskstats version 1 [ 0.427870] Key type trusted registered [ 0.430763] Key type encrypted registered [ 0.430767] AppArmor: AppArmor sha1 policy hashing enabled [ 0.430778] ima: No TPM chip found, activating TPM-bypass! [ 0.430788] evm: HMAC attrs: 0x1 [ 0.431181] Magic number: 7:291:301 [ 0.431203] tty tty30: hash matches [ 0.431279] rtc_cmos 00:02: setting system clock to 2015-03-02 23:15:12 UTC (1425338112) [ 0.431406] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 0.431406] EDD information not available. [ 0.431452] PM: Hibernation image not present or could not be loaded. [ 0.431718] Freeing unused kernel memory: 1376K (ffffffff81d32000 - ffffffff81e8a000) [ 0.431719] Write protecting the kernel read-only data: 12288k [ 0.431896] Freeing unused kernel memory: 168K (ffff8800017d6000 - ffff880001800000) [ 0.431952] Freeing unused kernel memory: 344K (ffff880001baa000 - ffff880001c00000) [ 0.437662] systemd-udevd[135]: starting version 208 [ 0.437916] random: systemd-udevd urandom read with 1 bits of entropy available [ 0.449249] pps_core: LinuxPPS API ver. 1 registered [ 0.449251] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 0.450235] ahci 0000:00:1f.2: version 3.0 [ 0.450442] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0xa impl SATA mode [ 0.450444] ahci 0000:00:1f.2: flags: 64bit ncq led clo pio slum part ems apst [ 0.450572] PTP clock support registered [ 0.452771] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k [ 0.452772] e1000e: Copyright(c) 1999 - 2014 Intel Corporation. [ 0.453367] scsi host0: ahci [ 0.453459] scsi host1: ahci [ 0.453619] scsi host2: ahci [ 0.453706] scsi host3: ahci [ 0.453895] scsi host4: ahci [ 0.453975] scsi host5: ahci [ 0.454004] ata1: DUMMY [ 0.454006] ata2: SATA max UDMA/133 abar m2048@0xee83a000 port 0xee83a180 irq 28 [ 0.454007] ata3: DUMMY [ 0.454008] ata4: SATA max UDMA/133 abar m2048@0xee83a000 port 0xee83a280 irq 28 [ 0.454009] ata5: DUMMY [ 0.454010] ata6: DUMMY [ 0.454125] e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode [ 0.621563] e1000e 0000:00:19.0 eth0: registered PHC clock [ 0.621575] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) d0:50:99:15:70:12 [ 0.621576] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection [ 0.621612] e1000e 0000:00:19.0 eth0: MAC: 11, PHY: 12, PBA No: FFFFFF-0FF [ 0.713188] usb 3-1: new high-speed USB device number 2 using ehci-pci [ 0.723174] usb 4-1: new high-speed USB device number 2 using ehci-pci [ 0.754132] usb 1-4: new high-speed USB device number 2 using xhci_hcd [ 0.759167] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.760741] ata2.00: supports DRM functions and may not be fully accessible [ 0.760989] ata2.00: failed to get NCQ Send/Recv Log Emask 0x1 [ 0.760990] ata2.00: ATA-9: Samsung SSD 850 EVO 500GB, EMT01B6Q, max UDMA/133 [ 0.760991] ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA [ 0.761145] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.761499] ata2.00: supports DRM functions and may not be fully accessible [ 0.761704] ata2.00: failed to get NCQ Send/Recv Log Emask 0x1 [ 0.761851] ata2.00: configured for UDMA/133 [ 0.762002] scsi 1:0:0:0: Direct-Access ATA Samsung SSD 850 1B6Q PQ: 0 ANSI: 5 [ 0.762199] ata4.00: ATA-8: WDC WD10EZRX-00A8LB0, 01.01A01, max UDMA/133 [ 0.762200] ata4.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA [ 0.762279] sd 1:0:0:0: Attached scsi generic sg0 type 0 [ 0.762289] sd 1:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 0.762632] sd 1:0:0:0: [sda] Write Protect is off [ 0.762634] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 0.762703] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.763249] ata4.00: configured for UDMA/133 [ 0.763349] scsi 3:0:0:0: Direct-Access ATA WDC WD10EZRX-00A 1A01 PQ: 0 ANSI: 5 [ 0.763434] sda: sda1 sda2 < sda5 > [ 0.763477] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 0.763479] sd 3:0:0:0: [sdb] 4096-byte physical blocks [ 0.763505] sd 3:0:0:0: Attached scsi generic sg1 type 0 [ 0.763524] sd 3:0:0:0: [sdb] Write Protect is off [ 0.763526] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 0.763542] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.763681] sd 1:0:0:0: [sda] Attached SCSI disk [ 0.778718] sdb: sdb1 sdb2 sdb3 [ 0.779444] sd 3:0:0:0: [sdb] Attached SCSI disk [ 0.829073] raid6: sse2x1 11921 MB/s [ 0.846051] raid6: sse2x2 14863 MB/s [ 0.846281] usb 4-1: New USB device found, idVendor=8087, idProduct=8000 [ 0.846282] usb 4-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 0.846571] hub 4-1:1.0: USB hub found [ 0.846668] hub 4-1:1.0: 8 ports detected [ 0.857428] usb 3-1: New USB device found, idVendor=8087, idProduct=8008 [ 0.857429] usb 3-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 0.857663] hub 3-1:1.0: USB hub found [ 0.857803] hub 3-1:1.0: 6 ports detected [ 0.863016] raid6: sse2x4 16441 MB/s [ 0.879996] raid6: avx2x1 23273 MB/s [ 0.896992] raid6: avx2x2 26261 MB/s [ 0.913985] raid6: avx2x4 31335 MB/s [ 0.913987] raid6: using algorithm avx2x4 (31335 MB/s) [ 0.913988] raid6: using avx2x2 recovery algorithm [ 0.914725] xor: automatically using best checksumming function: [ 0.923953] avx : 36684.000 MB/sec [ 0.930765] Btrfs loaded [ 1.137168] usb 1-4: New USB device found, idVendor=046d, idProduct=0825 [ 1.137170] usb 1-4: New USB device strings: Mfr=0, Product=0, SerialNumber=2 [ 1.137171] usb 1-4: SerialNumber: 5713F440 [ 1.290663] usb 1-13: new high-speed USB device number 3 using xhci_hcd [ 1.375577] tsc: Refined TSC clocksource calibration: 3399.013 MHz [ 1.454848] usb 1-13: New USB device found, idVendor=1a40, idProduct=0101 [ 1.454850] usb 1-13: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 1.454851] usb 1-13: Product: USB 2.0 Hub [MTT] [ 1.455168] hub 1-13:1.0: USB hub found [ 1.455191] hub 1-13:1.0: 4 ports detected [ 1.466108] random: nonblocking pool is initialized [ 1.538057] BTRFS: device fsid a3929651-e09b-4f1c-afa3-a5516b17eabb devid 1 transid 9975 /dev/sda1 [ 1.541465] BTRFS info (device sda1): disk space caching is enabled [ 1.556231] BTRFS: detected SSD devices, enabling SSD mode [ 1.608398] usb 1-14: new low-speed USB device number 4 using xhci_hcd [ 1.693877] init: plymouth-upstart-bridge main process (236) terminated with status 1 [ 1.693883] init: plymouth-upstart-bridge main process ended, respawning [ 1.694825] init: plymouth-upstart-bridge main process (252) terminated with status 1 [ 1.694831] init: plymouth-upstart-bridge main process ended, respawning [ 1.695732] init: plymouth-upstart-bridge main process (254) terminated with status 1 [ 1.695739] init: plymouth-upstart-bridge main process ended, respawning [ 1.696821] init: plymouth-upstart-bridge main process (256) terminated with status 1 [ 1.696830] init: plymouth-upstart-bridge main process ended, respawning [ 1.777105] usb 1-14: New USB device found, idVendor=046d, idProduct=c01e [ 1.777107] usb 1-14: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.777108] usb 1-14: Product: USB-PS/2 Optical Mouse [ 1.777109] usb 1-14: Manufacturer: Logitech [ 1.777213] usb 1-14: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.834196] usb 1-13.3: new low-speed USB device number 5 using xhci_hcd [ 1.932341] usb 1-13.3: New USB device found, idVendor=09da, idProduct=0260 [ 1.932344] usb 1-13.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.932345] usb 1-13.3: Product: USB Keyboard [ 1.932346] usb 1-13.3: Manufacturer: [ 1.932478] usb 1-13.3: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [ 1.932481] usb 1-13.3: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [ 2.262218] Adding 16002044k swap on /dev/sda5. Priority:-1 extents:1 across:16002044k SSFS [ 2.264717] BTRFS info (device sda1): turning on discard [ 2.264721] BTRFS info (device sda1): enabling auto defrag [ 2.264723] BTRFS info (device sda1): disk space caching is enabled [ 2.323626] systemd-udevd[414]: starting version 208 [ 2.341739] nct6775: Found NCT6776D/F or compatible chip at 0x2e:0x290 [ 2.346091] lp: driver loaded but no devices found [ 2.347918] ppdev: user-space parallel port driver [ 2.375121] Switched to clocksource tsc [ 2.400316] Bluetooth: Core ver 2.20 [ 2.400327] NET: Registered protocol family 31 [ 2.400327] Bluetooth: HCI device and connection manager initialized [ 2.400330] Bluetooth: HCI socket layer initialized [ 2.400332] Bluetooth: L2CAP socket layer initialized [ 2.400335] Bluetooth: SCO socket layer initialized [ 2.401801] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 [ 2.401803] Bluetooth: BNEP filters: protocol multicast [ 2.401805] Bluetooth: BNEP socket layer initialized [ 2.402621] Bluetooth: RFCOMM TTY layer initialized [ 2.402625] Bluetooth: RFCOMM socket layer initialized [ 2.402628] Bluetooth: RFCOMM ver 1.11 [ 2.408200] init: avahi-cups-reload main process (526) terminated with status 1 [ 2.410099] audit: type=1400 audit(1425327314.479:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/cups/backend/cups-pdf" pid=532 comm="apparmor_parser" [ 2.410103] audit: type=1400 audit(1425327314.479:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/cupsd" pid=532 comm="apparmor_parser" [ 2.410105] audit: type=1400 audit(1425327314.479:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="third_party" pid=532 comm="apparmor_parser" [ 2.429219] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.453577] audit: type=1400 audit(1425327314.522:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=582 comm="apparmor_parser" [ 2.453582] audit: type=1400 audit(1425327314.522:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=582 comm="apparmor_parser" [ 2.453584] audit: type=1400 audit(1425327314.522:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=582 comm="apparmor_parser" [ 2.454234] [drm] Initialized drm 1.1.0 20060810 [ 2.454903] hidraw: raw HID events driver (C) Jiri Kosina [ 2.455036] media: Linux media interface: v0.10 [ 2.459964] Linux video capture interface: v2.00 [ 2.468530] uvcvideo: Found UVC 1.00 device <unnamed> (046d:0825) [ 2.478368] [drm] Memory usable by graphics device = 2048M [ 2.478370] [drm] VT-d active for gfx access [ 2.478371] checking generic (d0000000 500000) vs hw (d0000000 10000000) [ 2.478372] fb: switching to inteldrmfb from VESA VGA [ 2.478385] Console: switching to colour dummy device 80x25 [ 2.478413] [drm] Replacing VGA console driver [ 2.482145] AVX2 version of gcm_enc/dec engaged. [ 2.482147] AES CTR mode by8 optimization enabled [ 2.483043] input: UVC Camera (046d:0825) as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4:1.0/input/input5 [ 2.483110] usbcore: registered new interface driver uvcvideo [ 2.483112] USB Video Class driver (1.1.1) [ 2.483846] usbcore: registered new interface driver usbhid [ 2.483847] usbhid: USB HID core driver [ 2.492611] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 2.492613] [drm] Driver supports precise vblank timestamp query. [ 2.492625] [drm] DMAR active, disabling use of stolen memory [ 2.500282] intel_rapl: Found RAPL domain package [ 2.500284] intel_rapl: Found RAPL domain core [ 2.500286] intel_rapl: Found RAPL domain uncore [ 2.500288] intel_rapl: Found RAPL domain dram [ 2.533494] input: Logitech USB-PS/2 Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-14/1-14:1.0/0003:046D:C01E.0001/input/input6 [ 2.534931] hid-generic 0003:046D:C01E.0001: input,hidraw0: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:14.0-14/input0 [ 2.535040] input: USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-13/1-13.3/1-13.3:1.0/0003:09DA:0260.0002/input/input7 [ 2.564718] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem [ 2.566116] ACPI: Video Device [GFX0] (multi-head: yes rom: no post: no) [ 2.566439] acpi device:5d: registered as cooling_device5 [ 2.566485] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input8 [ 2.573293] init: failsafe main process (694) killed by TERM signal [ 2.585568] hid-generic 0003:09DA:0260.0002: input,hidraw1: USB HID v1.10 Keyboard [ USB Keyboard] on usb-0000:00:14.0-13.3/input0 [ 2.585659] [drm] Initialized i915 1.6.0 20141121 for 0000:00:02.0 on minor 0 [ 2.585841] snd_hda_intel 0000:01:00.1: Disabling MSI [ 2.585846] snd_hda_intel 0000:01:00.1: Handle VGA-switcheroo audio client [ 2.588045] input: USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-13/1-13.3/1-13.3:1.1/0003:09DA:0260.0003/input/input9 [ 2.591135] fbcon: inteldrmfb (fb0) is primary device [ 2.591176] Console: switching to colour frame buffer device 160x64 [ 2.591189] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device [ 2.591190] i915 0000:00:02.0: registered panic notifier [ 2.606138] input: HDA Intel HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.0/sound/card0/input10 [ 2.606800] sound hdaudioC2D0: autoconfig: line_outs=4 (0x14/0x15/0x16/0x17/0x0) type:line [ 2.606802] sound hdaudioC2D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0) [ 2.606803] sound hdaudioC2D0: hp_outs=1 (0x1b/0x0/0x0/0x0/0x0) [ 2.606804] sound hdaudioC2D0: mono: mono_out=0x0 [ 2.606805] sound hdaudioC2D0: dig-out=0x1e/0x0 [ 2.606806] sound hdaudioC2D0: inputs: [ 2.606807] sound hdaudioC2D0: Front Mic=0x19 [ 2.606808] sound hdaudioC2D0: Rear Mic=0x18 [ 2.606809] sound hdaudioC2D0: Line=0x1a [ 2.626288] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card2/input13 [ 2.633581] init: samba-ad-dc main process (771) terminated with status 1 [ 2.635292] audit: type=1400 audit(1425327314.704:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session" pid=881 comm="apparmor_parser" [ 2.635296] audit: type=1400 audit(1425327314.704:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="chromium" pid=881 comm="apparmor_parser" [ 2.636189] audit: type=1400 audit(1425327314.705:10): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/sbin/dhclient" pid=881 comm="apparmor_parser" [ 2.638556] hid-generic 0003:09DA:0260.0003: input,hidraw2: USB HID v1.10 Device [ USB Keyboard] on usb-0000:00:14.0-13.3/input1 [ 2.641521] input: HDA Intel HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.0/sound/card0/input11 [ 2.641879] input: HDA Intel HDMI HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.0/sound/card0/input12 [ 2.642009] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card2/input14 [ 2.642042] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card2/input15 [ 2.642078] input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card2/input16 [ 2.642111] input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card2/input17 [ 2.642144] input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card2/input18 [ 2.642382] input: HDA Intel PCH Line Out Side as /devices/pci0000:00/0000:00:1b.0/sound/card2/input19 [ 2.642418] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card2/input20 [ 2.680434] systemd-logind[852]: New seat seat0. [ 2.680634] systemd-logind[852]: Watching system buttons on /dev/input/event1 (Power Button) [ 2.680666] systemd-logind[852]: Watching system buttons on /dev/input/event5 (Video Bus) [ 2.680692] systemd-logind[852]: Watching system buttons on /dev/input/event0 (Power Button) [ 2.811625] init: Failed to spawn atd main process: unable to execute: No such file or directory [ 2.894911] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input21 [ 2.894954] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input22 [ 2.895015] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input23 [ 2.895058] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card3/input24 [ 2.906336] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.997641] ------------[ cut here ]------------ [ 2.997671] WARNING: CPU: 0 PID: 646 at /home/terusus/ubuntu-vivid/source/drivers/gpu/drm/i915/intel_display.c:9705 intel_check_page_flip+0xa2/0xf0 [i915]() [ 2.997672] WARN_ON(!in_irq()) [ 2.997673] Modules linked in: [ 2.997674] snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic hid_generic intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul snd_hda_intel snd_hda_controller ghash_clmulni_intel snd_hda_codec snd_usb_audio(+) aesni_intel snd_seq_midi snd_usbmidi_lib snd_seq_midi_event snd_hwdep snd_rawmidi aes_x86_64 lrw gf128mul snd_seq glue_helper i915 ablk_helper cryptd uvcvideo snd_pcm videobuf2_vmalloc videobuf2_memops videobuf2_core serio_raw v4l2_common drm_kms_helper videodev usbhid media hid drm mei_me lpc_ich snd_seq_device snd_timer mei snd i2c_algo_bit soundcore shpchp 8250_fintek nuvoton_cir rc_core video soc_button_array mac_hid rfcomm bnep bluetooth binfmt_misc parport_pc ppdev lp parport nct6775 hwmon_vid coretemp btrfs xor raid6_pq [ 2.997707] e1000e ptp ahci psmouse libahci pps_core [ 2.997711] CPU: 0 PID: 646 Comm: irq/31-i915 Not tainted 3.19.0-6-kvm #6 [ 2.997712] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H87 Pro4, BIOS P2.10 07/09/2014 [ 2.997713] ffffffffc08a51b8 ffff88041958bc98 ffffffff817c6613 0000000000000007 [ 2.997715] ffff88041958bce8 ffff88041958bcd8 ffffffff8107732a ffff88041958bcd8 [ 2.997716] ffff88041adfd000 ffff880419b7c000 ffff880419b7c000 0000000000000000 [ 2.997717] Call Trace: [ 2.997723] [<ffffffff817c6613>] dump_stack+0x4c/0x6e [ 2.997726] [<ffffffff8107732a>] warn_slowpath_common+0x8a/0xc0 [ 2.997728] [<ffffffff810773a6>] warn_slowpath_fmt+0x46/0x50 [ 2.997740] [<ffffffffc08501a2>] intel_check_page_flip+0xa2/0xf0 [i915] [ 2.997749] [<ffffffffc081d277>] ironlake_irq_handler+0x417/0x1000 [i915] [ 2.997751] [<ffffffff8109bc4d>] ? finish_task_switch+0x5d/0x100 [ 2.997753] [<ffffffff810d0340>] ? irq_thread_fn+0x50/0x50 [ 2.997755] [<ffffffff810d036d>] irq_forced_thread_fn+0x2d/0x70 [ 2.997756] [<ffffffff810d08b7>] irq_thread+0x137/0x160 [ 2.997758] [<ffffffff810d03e0>] ? wake_threads_waitq+0x30/0x30 [ 2.997759] [<ffffffff810d0780>] ? irq_thread_check_affinity+0x90/0x90 [ 2.997761] [<ffffffff810961eb>] kthread+0xdb/0x100 [ 2.997763] [<ffffffff81096110>] ? kthread_create_on_node+0x1c0/0x1c0 [ 2.997765] [<ffffffff817cd5bc>] ret_from_fork+0x7c/0xb0 [ 2.997766] [<ffffffff81096110>] ? kthread_create_on_node+0x1c0/0x1c0 [ 2.997767] ---[ end trace d19b5eb4f7000469 ]--- [ 3.083940] init: plymouth-upstart-bridge main process ended, respawning [ 3.085485] init: plymouth-upstart-bridge main process (1549) killed by TERM signal [ 3.120383] systemd-logind[852]: Failed to start unit user@112.service: Unknown unit: user@112.service [ 3.120387] systemd-logind[852]: Failed to start user service: Unknown unit: user@112.service [ 3.122355] systemd-logind[852]: New session c1 of user lightdm. [ 3.122374] systemd-logind[852]: Linked /tmp/.X11-unix/X0 to /run/user/112/X11-display. [ 3.298912] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this. [ 3.350660] ip_tables: (C) 2000-2006 Netfilter Core Team [ 3.372830] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 3.646690] usb 1-4: set resolution quirk: cval->res = 384 [ 3.646962] usbcore: registered new interface driver snd-usb-audio [ 4.423463] e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx [ 4.423466] e1000e 0000:00:19.0 eth0: 10/100 speed: disabling TSO [ 4.423494] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 8.942751] systemd-logind[852]: Failed to start unit user@1000.service: Unknown unit: user@1000.service [ 8.942754] systemd-logind[852]: Failed to start user service: Unknown unit: user@1000.service [ 8.945072] systemd-logind[852]: New session c2 of user terusus. [ 8.945093] systemd-logind[852]: Linked /tmp/.X11-unix/X0 to /run/user/1000/X11-display. [ 61.360511] systemd-hostnamed[2997]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname! [ 69.112449] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 73.738806] kvm: zapping shadow pages for mmio generation wraparound [ 84.697045] [drm:drm_edid_block_valid [drm]] *ERROR* EDID checksum is invalid, remainder is 190 [ 84.697048] Raw EDID: [ 84.697049] 00 ff ff ff ff ff ff 00 26 cd d7 46 8b 18 00 00 [ 84.697049] 1f 0f 01 03 80 22 1b 78 2f 5e 50 a6 54 4c 99 26 [ 84.697050] 14 50 54 bf ef 00 81 b9 7f ff ff ff ff ff ff ff [ 84.697051] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697051] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697052] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697053] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697053] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 167.715926] kvm: zapping shadow pages for mmio generation wraparound [ 256.890599] kvm: zapping shadow pages for mmio generation wraparound

I did start the VM 3 times and I believe the message kvm: zapping shadow pages for mmio generation wraparound corresponds to those. This at the end looks a bit strange. What does this "drm error" mean:

[ 61.360511] systemd-hostnamed[2997]: Warning: nss-myhostname is not installed. Changing the local hostname might make it unresolveable. Please install nss-myhostname! [ 69.112449] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 73.738806] kvm: zapping shadow pages for mmio generation wraparound [ 84.697045] [drm:drm_edid_block_valid [drm]] *ERROR* EDID checksum is invalid, remainder is 190 [ 84.697048] Raw EDID: [ 84.697049] 00 ff ff ff ff ff ff 00 26 cd d7 46 8b 18 00 00 [ 84.697049] 1f 0f 01 03 80 22 1b 78 2f 5e 50 a6 54 4c 99 26 [ 84.697050] 14 50 54 bf ef 00 81 b9 7f ff ff ff ff ff ff ff [ 84.697051] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697051] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697052] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697053] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 84.697053] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff [ 167.715926] kvm: zapping shadow pages for mmio generation wraparound [ 256.890599] kvm: zapping shadow pages for mmio generation wraparound

PS: My QEMU start code is:

qemu-system-x86_64 -enable-kvm -M pc -m 4096 \ -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -device ioh3420,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi \ -drive file=/storage/vm/win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/media/.../win7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/media/.../virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -vga none

PPS: drm error does not appear if I boot with "nomodeset".

PPPS: I read aw's replay regarding kvm zapping. How to do pre-boot?

Last edited by terusus (2015-03-02 22:08:52)

Offline

#4379 2015-03-02 22:12:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

terusus wrote:

PPPS: I read aw's replay regarding kvm zapping. How to do pre-boot?

That post has absolutely nothing to do with the zapping message.  If you don't have a Quadro or GRID, real or fake, ignore everything about that post.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4380 2015-03-02 22:44:21

prussian
Member
Registered: 2015-01-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone know how to fix this?

alsa: Requested buffer size 256 was rejected, using 2048

if the buffer is too large, the audio quality in Windows VMs seems to suffer massively and it seems like alsa isn't letting me set a low buffer.

edit:

never mind, I forgot to use sudo -E and now it's using the buffer size I set in my .asoundrc

Last edited by prussian (2015-03-02 22:56:47)

Offline

#4381 2015-03-03 02:57:00

Len
Member
Registered: 2015-01-06
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, any idea how can I share CPU between two guest, assuming that CPU load is equal to 100% on both guests and host:
Max allowed CPU usage for guest#1 - 40%
-- for guest#2 - 40%
-- for host - 20%

when guest#1 load is low, guest#2 is getting up to 80% of CPU usage and vice versa, host - always 20%.

Is this possible?
Also same thing, but with RAM.

Last edited by Len (2015-03-03 02:58:49)

Offline

#4382 2015-03-03 08:35:40

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got information that NVIDIA Game Ready drivers turn off CUDA. Is this true? Can anyone confirm this?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4383 2015-03-03 20:49:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

Hmm, any idea how can I share CPU between two guest, assuming that CPU load is equal to 100% on both guests and host:
Max allowed CPU usage for guest#1 - 40%
-- for guest#2 - 40%
-- for host - 20%

when guest#1 load is low, guest#2 is getting up to 80% of CPU usage and vice versa, host - always 20%.

Is this possible?
Also same thing, but with RAM.

google: cgroups

When device assignment is in play, VMs use fixed amounts of RAM, all guest memory is pinned, so maybe the only thing cgroups could control would be caching of block device in the host if they're setup to use host caching.  cgroups for setting CPU limits should be possible.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4384 2015-03-03 21:01:14

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
PureTryOut wrote:
aw wrote:

Then either VGA arbitration is not working or QEMU can't read the ROM on the GPU.  VGA arbitration not working was a problem we used to see using the proprietary nvidia driver in the host (driver would lock and never release the VGA arbitration lock).  Nvidia might have fixed this, dunno.  The latter ROM issue would produce an invalid ROM contents message in dmesg.  GeForce cards do not work as secondary GPUs in the guest.

So is there any workaround or any way I can fix this?

If you read the FAQ from the link below, you might find Question 9 interesting.  If you look at the comments, you'll find a link to that patch.  It looks like even the latest version of the Linux nvidia driver still has this problem.

I'm sorry, what link are you referring too? The one in your signature? I can't find a Question 9 on there?

Edit: Woops nvm I found it, i'll check it out because it seems to be my problem.

Edit 2: Well that went wrong... I've patched the host NVIDIA driver, installed it, and then it wouldn't boot anymore. Not even with nomodeset...
I had to switch back to the stock Arch kernel, add nomodeset, then reinstall the patched kernel + stock Arch drivers.
I can boot my system again but still without a patched NVIDIA driver.
Although i'm not sure Question 9 applies to me. They are talking about "nothing happening", while Qemu in fact launches for me, just not the VM itself.

Last edited by PureTryOut (2015-03-03 21:50:14)

Offline

#4385 2015-03-03 21:30:13

hurenkam
Member
Registered: 2012-12-10
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noctavian wrote:

That would be me.

Ah, I see you added me there as well, so let me provide some additional info:
- Host Distro: Arch
- Host Kernel: 3.17.6-1-ARCH
- Qemu version: 2.1.2
- Patches: None, just using standard Arch packages.
- Other remarks: Working guests: Mavericks / Windows 8 / Arch / Ubuntu 14.10

Offline

#4386 2015-03-03 23:05:06

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ACS override patch from OP doesnt work on Fedora 21 with 3.18.7 kernel.

The error:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.uVOn5W (%prep)

Can anyone help me?

Offline

#4387 2015-03-03 23:08:50

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Tyrewt wrote:
Bronek wrote:

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.




So I followed your advice with no luck, maybe I should focus efforts on USB Device Pass-through as it seems get further with Ubuntu.

For the record, the three USB controllers were:

00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 04)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 04)

Added them to my /etc/vfio-pci.cfg

0000:00:14.0
0000:00:1a.0
0000:00:1d.0

Then added the following lines to my qemu config

-usb \
-device vfio-pci,host=00:14.0 \
-device vfio-pci,host=00:1a.0 \
-device vfio-pci,host=00:1d.0 \


Starting the Virtual Machine fails at the first controller with the following error.

qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio: error opening /dev/vfio/4: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=00:14.0: vfio: failed to get group 4
qemu-system-x86_64: -device vfio-pci,host=00:14.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:14.0: Device 'vfio-pci' could not be initialized

Any suggestions?

ok so you are trying to add three USB controller devices (1x USB 3.0 at 00:14.0 and 2x USD 2.0 at 00:1a.0,00:1d.0) . Apparently adding USB 3.0 controller fails, although we do not see anything about the other two USB 2.0 controllers. Did you ensure you are passing whole IOMMU group? Did you remember to add vendorid:deviceid to pci-stub.ids=... in kernel command line? Did you try passing only USB 2.0 devices? I assume your host was restarted to allow for vfio to claim these USB controllers?



I was able to sucessfully pass USB devices by explicitly listing them by vendorid and productid. No kernel command is present, my issue was with proper syntax in my QEMU configuration.
I've noticed both the Host and Guest take control of the device at the same time once connected. Not too sure if this is normal, but once disconnected, the following error is displayed on the Host console:
libusb_release_interface: -4 [NO_DEVICE]
libusbx: error [_open_sysfs_attr] open /sys/bus/usb/devices/3-14/bConfigurationValue failed ret=-1 errno=2

Offline

#4388 2015-03-05 05:29:34

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW, what did you do in regards to SELinux and Fedora 21? I spent the last 3 nights trying to get my working Arch XML to boot under Fedora 21, only to figure out/remember SELinux was enabled...

Also, to everyone, what did you do to get Windows 7 with EFI booted to run with more than one CPU? Are there more HyperV parameters that need to be set in my XML?

Offline

#4389 2015-03-05 13:53:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

AW, what did you do in regards to SELinux and Fedora 21? I spent the last 3 nights trying to get my working Arch XML to boot under Fedora 21, only to figure out/remember SELinux was enabled...

Also, to everyone, what did you do to get Windows 7 with EFI booted to run with more than one CPU? Are there more HyperV parameters that need to be set in my XML?

You'd need to actually tell me what selinux is complaining about to know how to fix it, not that I'm an selinux expert.  However, if it's complaining due to a wrapper script being used to add the x-vga option, you can find instructions here to fix it: https://access.redhat.com/documentation … /virt.html  If it has something to do with <qemu:arg> options, you're on your own.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4390 2015-03-05 15:23:38

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You'd need to actually tell me what selinux is complaining about to know how to fix it, not that I'm an selinux expert.  However, if it's complaining due to a wrapper script being used to add the x-vga option, you can find instructions here to fix it: https://access.redhat.com/documentation … /virt.html  If it has something to do with <qemu:arg> options, you're on your own.

I ended up disabling SELinux... But here are the two situations I ran into:
1) I had the OVMF files CODE and VARs in my home directory -  these were from my working ARCH build. I went to start my VM and it said I had permission denied on the files, even with temporary chmod 777.
2) When I linked to the OVMF files in their default location, /usr/share/edk2.git/..., I wouldn't get the permissions denied, but the monitor attached to my passed-through wouldn't iniialize. I would hear the GPU spin up, the VM would say it's running, but no output. When I wasn't passing through my GPU, or any card, I would see the Tianocore efi screen on the virtual console.

Once I disabled SELinux, I ceased to have any of these issues. I should have thought of SELinux when I got the permission denied with the OVMF files set to 777.

In a post from 2013, you mentioned you had a Windows 7 VM running with GPU Passthrough. My issue I have is I cannot allocate more than one CPU to the VM. When I increase the Core and Thread count, it will get past the Tianocore screen, the Windows 7 logo will appear, then the screen will go black. I can even take my working Windows 10 config, swap my Windows7.img in, and the same issues will occur.

Offline

#4391 2015-03-05 15:43:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
aw wrote:

You'd need to actually tell me what selinux is complaining about to know how to fix it, not that I'm an selinux expert.  However, if it's complaining due to a wrapper script being used to add the x-vga option, you can find instructions here to fix it: https://access.redhat.com/documentation … /virt.html  If it has something to do with <qemu:arg> options, you're on your own.

I ended up disabling SELinux... But here are the two situations I ran into:
1) I had the OVMF files CODE and VARs in my home directory -  these were from my working ARCH build. I went to start my VM and it said I had permission denied on the files, even with temporary chmod 777.

pebkac... why would libvirt have permission to files in your home directory?  selinux rightfully prevents this

2) When I linked to the OVMF files in their default location, /usr/share/edk2.git/..., I wouldn't get the permissions denied, but the monitor attached to my passed-through wouldn't iniialize. I would hear the GPU spin up, the VM would say it's running, but no output. When I wasn't passing through my GPU, or any card, I would see the Tianocore efi screen on the virtual console.

The OVMF CODE file can be linked to the default location, but the VARS file needs to live somewhere libvirt has access to write.  If you use the template mechanism discussed in my KVM forum slides, libvirt will make the writable VARS file in /var/lib/libvirt/qemu/nvram/

Once I disabled SELinux, I ceased to have any of these issues. I should have thought of SELinux when I got the permission denied with the OVMF files set to 777.

In a post from 2013, you mentioned you had a Windows 7 VM running with GPU Passthrough. My issue I have is I cannot allocate more than one CPU to the VM. When I increase the Core and Thread count, it will get past the Tianocore screen, the Windows 7 logo will appear, then the screen will go black. I can even take my working Windows 10 config, swap my Windows7.img in, and the same issues will occur.

I've only used Windows 7 with x-vga, it's only here that I learned win7 can even run on OVMF, but it seems like more hassle than I'm interested in for a game VM.  IME, windows doesn't like hardware changes and uses a different internal machine type for a uniprocessor system versus a multi-processor system.  You may need to boot w/ a single cpu, go into device manager and change the model before you can boot with multiple cpus.  Sorry, that's as specific as I can be, it's been a long time since I've had to do that and I always install windows guests with at least 2 cpus to avoid it since.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4392 2015-03-05 16:39:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As far as i understand it - windows 7 doesn't work properly with any OVMF config at all. It tries to boot and fails to output any graphics.
I'm not sure, but eventually i might test this behaviour (again) on real UEFI platform.
Debug console isn't saying anything suspicious when it hangs. I might fiddle with it using GDB, but i highly doubt that it'll help.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4393 2015-03-05 17:03:27

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

ACS override patch from OP doesnt work on Fedora 21 with 3.18.7 kernel.

The error:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.uVOn5W (%prep)

Can anyone help me?

I have been able to patch 3.18.7 on Fedora for the ACS capabilities. I've taken Alex's patch and had to change it around a little - I reverse engineered it and applied it to the new quirks.c file. I posted the patch on this forum, search for pkim posts. I also posted a detailed experience with my setup as well - I almost got everything working stock, but I needed the ACS patch.

- Peter

Offline

#4394 2015-03-05 17:34:17

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wonder if it would be worth while to have an IRC chat going for this thread alone? ;-)

Offline

#4395 2015-03-05 21:40:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

I wonder if it would be worth while to have an IRC chat going for this thread alone? ;-)

I've been struggling to gather a fresh FAQ section for a week, and you suggest increasing the information trashing speed.
I've been trying to find debug console snippet of config for two days, and failed.
And the normal #kvm channel is so full of idlers, that i remembered the days of IRC Idle-RPG bots and stuff.
Of course, there was a quote like "do people talk about something except vga passthrough ion this channel anymore?" or something, but, you know..

We really need to gather a FAQ or wiki page with all the different problems and solutions. And archwiki... requires you to be registered on arch wiki.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4396 2015-03-05 21:43:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
The_Moves wrote:

I wonder if it would be worth while to have an IRC chat going for this thread alone? ;-)

I've been struggling to gather a fresh FAQ section for a week, and you suggest increasing the information trashing speed.
I've been trying to find debug console snippet of config for two days, and failed.
And the normal #kvm channel is so full of idlers, that i remembered the days of IRC Idle-RPG bots and stuff.
Of course, there was a quote like "do people talk about something except vga passthrough ion this channel anymore?" or something, but, you know..

We really need to gather a FAQ or wiki page with all the different problems and solutions. And archwiki... requires you to be registered on arch wiki.

I can update my FAQ, what's it missing?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4397 2015-03-06 09:15:55

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
The_Moves wrote:

I wonder if it would be worth while to have an IRC chat going for this thread alone? ;-)

I've been struggling to gather a fresh FAQ section for a week, and you suggest increasing the information trashing speed.
I've been trying to find debug console snippet of config for two days, and failed.
And the normal #kvm channel is so full of idlers, that i remembered the days of IRC Idle-RPG bots and stuff.
Of course, there was a quote like "do people talk about something except vga passthrough ion this channel anymore?" or something, but, you know..

We really need to gather a FAQ or wiki page with all the different problems and solutions. And archwiki... requires you to be registered on arch wiki.

I can update my FAQ, what's it missing?

I have been searching abit in this forum and on aw's vfio blog faq. When going through the talking points you had aw i got the impression that if i switch to using OVMF and UEFI on my windows8.1 guest, the vga arbiter issue with Nvidia proprietary host drivers is a non issue (your faq question 9) and i can install and use the nvidia driver on the host without patching?

Sorry if this question has been answered before i just had a hard time finding the answer that looked obvious from your talking points "VFIO, OVMF, GPU, and You ".

Offline

#4398 2015-03-06 14:10:33

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've seen this too (no need to patch host drivers if you're using OVMF). Sadly I have no clue how to make it work though.
From what I've read you should launch Qemu with

qemu-system-x86_64 -L .

This while being in the directory where the UEFI bios.bin is in.
Well tried that, it did nothing, not even give me an UEFI shell.

Is there a proper guide to OVMF in Qemu out there that we can use?

Offline

#4399 2015-03-06 14:19:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

I've seen this too (no need to patch host drivers if you're using OVMF). Sadly I have no clue how to make it work though.
From what I've read you should launch Qemu with

qemu-system-x86_64 -L .

This while being in the directory where the UEFI bios.bin is in.
Well tried that, it did nothing, not even give me an UEFI shell.

Is there a proper guide to OVMF in Qemu out there that we can use?

Assuming you're on arch:

Get OVMF-svn, build an install it.

Then launch qemu like this:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1 \ -vga none

Last edited by nbhs (2015-03-06 14:20:56)

Offline

#4400 2015-03-06 14:51:21

efeu
Member
Registered: 2015-03-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I havent that good CPU Performance in Win8.1 using -smp 8,sockets=1,cores=4,threads=2 with my 4970k@4,6GHz so I saw Pin vcpu to pcpu in first post but this site seems to be do offline.
Can someone help me with that?

Offline

#4401 2015-03-06 15:01:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbirkis wrote:

I have been searching abit in this forum and on aw's vfio blog faq. When going through the talking points you had aw i got the impression that if i switch to using OVMF and UEFI on my windows8.1 guest, the vga arbiter issue with Nvidia proprietary host drivers is a non issue (your faq question 9) and i can install and use the nvidia driver on the host without patching?

Sorry if this question has been answered before i just had a hard time finding the answer that looked obvious from your talking points "VFIO, OVMF, GPU, and You ".

It's my understanding that this is true.  The problem with the nvidia proprietary driver on the host is that it takes the VGA arbiter lock and never releases it.  The expected usage model for the VGA arbiter is that the lock is held only while accesses to the legacy VGA space are being used.  The lock is then released allowing other users of the arbiter a chance to acquire the lock and perform their access.  This is how pseudo-concurrent VGA access works with multiple VGA assignment VMs.  Therefore, so long as our VM is not performing VGA accesses, which can be enforced by disabling CONFIG_VFIO_PCI_VGA (or a vfio-pci module option I intend to add), there is no need to patch the host nvidia driver to prevent VGA locking.

Note however that while OVMF generally implies legacy-free, there is a legacy compatibility module, the CSM, available in some OVMF builds.  The CSM will potentially do legacy VGA accesses and put you back in the position of needing to patch host drivers for VGA arbitration.  You therefore want to make use of the "pure" version of OVMF to avoid this.  On Fedora, the OVMF file to use is /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd (avoid the "with-csm" versions).  Your builds may name them differently.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4402 2015-03-06 15:06:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

efeu wrote:

I havent that good CPU Performance in Win8.1 using -smp 8,sockets=1,cores=4,threads=2 with my 4970k@4,6GHz so I saw Pin vcpu to pcpu in first post but this site seems to be do offline.
Can someone help me with that?

Specifying a CPU topology with threads is generally only recommended in combination with pinning.  Using 100% of the host CPU resources for a guest is also likely to cause glitches when the host needs some CPU resources.  There's plenty of information online for vCPU pinning.  libvirt can do this for you and is detailed on their site.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4403 2015-03-06 15:33:38

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
PureTryOut wrote:

I've seen this too (no need to patch host drivers if you're using OVMF). Sadly I have no clue how to make it work though.
From what I've read you should launch Qemu with

qemu-system-x86_64 -L .

This while being in the directory where the UEFI bios.bin is in.
Well tried that, it did nothing, not even give me an UEFI shell.

Is there a proper guide to OVMF in Qemu out there that we can use?

Assuming you're on arch:

Get OVMF-svn, build an install it.

Then launch qemu like this:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1 \ -vga none

Yes i'm on Arch wink

Well without -"vga none", it at least drops me in the EFI shell now which it didn't before, so that's progress.
Sadly now with -"vga none" added again it only shows me the "compat_monitor0 console" again...
I seriously don't understand what i'm missing here...

Offline

#4404 2015-03-06 15:49:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:
PureTryOut wrote:

I've seen this too (no need to patch host drivers if you're using OVMF). Sadly I have no clue how to make it work though.
From what I've read you should launch Qemu with

qemu-system-x86_64 -L .

This while being in the directory where the UEFI bios.bin is in.
Well tried that, it did nothing, not even give me an UEFI shell.

Is there a proper guide to OVMF in Qemu out there that we can use?

Assuming you're on arch:

Get OVMF-svn, build an install it.

Then launch qemu like this:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 2,sockets=1,cores=2,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1 \ -vga none

Yes i'm on Arch wink

Well without -"vga none", it at least drops me in the EFI shell now which it didn't before, so that's progress.
Sadly now with -"vga none" added again it only shows me the "compat_monitor0 console" again...
I seriously don't understand what i'm missing here...

You're supposed to get output on the monitor connected to your secondary gpu, that black console doesnt matter, make sure your hdmi/dvi cables are ok

Offline

#4405 2015-03-06 16:17:29

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len wrote:

Hmm, any idea how can I share CPU between two guest, assuming that CPU load is equal to 100% on both guests and host:
Max allowed CPU usage for guest#1 - 40%
-- for guest#2 - 40%
-- for host - 20%

when guest#1 load is low, guest#2 is getting up to 80% of CPU usage and vice versa, host - always 20%.

Is this possible?
Also same thing, but with RAM.

efeu wrote:

I havent that good CPU Performance in Win8.1 using -smp 8,sockets=1,cores=4,threads=2 with my 4970k@4,6GHz so I saw Pin vcpu to pcpu in first post but this site seems to be do offline.
Can someone help me with that?

First get cpuset, (Arch pkg: cpuset.tar.gz) and set up huge pages as described here, also read http://www-01.ibm.com/support/knowledge … roctop.htm and http://wiki.qemu.org/QMP


i have an 8 core amd cpu, 16GB of ram and i want to achieve something like this:

  • VM1 running on cores 5-7 @ 6144MB

  • VM2 running on cores 2-4 @ 6144MB

  • I want my system task running on unused system cores @ 4GB

Create a "system" cpuset running on all cores [0-7]

cset set -c 0-7 -s system
cset: --> created cpuset "system"

Move all processes from the root cpuset to the new system cpuset

cset proc -m -f root -t system
cset: moving all tasks from root to /system cset: moving 289 userspace tasks to /system [==================================================]% cset: done

We want to launch VM1 on cores [5-7] (5,6,7)

So we modify the "system" cpuset and reduce our cores dedicated to systems tasks from [0-7] to [0-4]

cset set -c 0-4 -s system
cset: --> modified cpuset "system"

Create a new cpuset called "VM1" for cores [5-7]

cset set -c 5-7 -s VM1
cset: --> created cpuset "VM1"

Launch qemu, as described on http://wiki.qemu.org/QMP, on cpuset "VM1"

cset proc -s VM1 -e -- qemu-system-x86_64 -enable-kvm -cpu host -smp 3,sockets=1,cores=3,threads=1 -m 6144 -mem-path /dev/hugepages/ (...) -qmp unix:./qmp-vm1,server,nowait
cset: --> last message, executed args into cpuset "/VM1", new pid is: 1950

Query the vcpu thread ids

echo "{ \"execute\": \"qmp_capabilities\" }" "{ \"execute\": \"query-cpus\" }" | socat UNIX-CONNECT:./qmp-vm1 STDIO | awk 'NR>1{print $1}' FS=} RS='"thread_id": '
2039 2040 2041

Use taskset to pin vcpu-pcpu

taskset -pc 5 2039 taskset -pc 6 2040 taskset -pc 7 2041

Once we're done with our vm we destroy our VM1 cpuset

cset set -d VM1
cset: --> processing cpuset "VM1", moving 0 tasks to parent "/"... cset: --> deleting cpuset "/VM1" cset: done

Modify the "system" cpuset back to [0-7] affinity

cset set -c 0-7 -s system
cset: --> modified cpuset "system"

Do the same for VM2

Comments? Advice?

Last edited by nbhs (2015-03-08 02:16:15)

Offline

#4406 2015-03-06 16:32:58

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You're supposed to get output on the monitor connected to your secondary gpu, that black console doesnt matter, make sure your hdmi/dvi cables are ok

See that explains things. I had no clue that I had to had a display connected to my GPU. I thought it would just boot in the window that popped up. I guess i'll have to get an extra monitor then...
I've plugged in an left-over screen for now in the HDMI port of my GTX 770 (which is want I want to use for the passthrough).

Now it gives me the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0: vfio: Error: Failed to setup INTx fd: Device or resource busy

Offline

#4407 2015-03-06 16:41:31

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:

You're supposed to get output on the monitor connected to your secondary gpu, that black console doesnt matter, make sure your hdmi/dvi cables are ok

See that explains things. I had no clue that I had to had a display connected to my GPU. I thought it would just boot in the window that popped up. I guess i'll have to get an extra monitor then...
I've plugged in an left-over screen for now in the HDMI port of my GTX 770 (which is want I want to use for the passthrough).

Now it gives me the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0: vfio: Error: Failed to setup INTx fd: Device or resource busy

https://www.youtube.com/watch?v=p85xwZ_OLX0

Offline

#4408 2015-03-06 16:55:11

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:

You're supposed to get output on the monitor connected to your secondary gpu, that black console doesnt matter, make sure your hdmi/dvi cables are ok

See that explains things. I had no clue that I had to had a display connected to my GPU. I thought it would just boot in the window that popped up. I guess i'll have to get an extra monitor then...
I've plugged in an left-over screen for now in the HDMI port of my GTX 770 (which is want I want to use for the passthrough).

Now it gives me the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0: vfio: Error: Failed to setup INTx fd: Device or resource busy

You cannot assign individual video output ports on your graphics cards, it's all or none, the whole card or not. Also make sure your VM isn't actually running already.

Offline

#4409 2015-03-06 16:56:38

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lol as much as I would love that to work, it didn't...
After rebooting, Arch wouldn't detect my second GPU at all anymore.
Not with PCI (dmesg | grep pci-stub), and not with lspci.
Unplugged the display from the secondary GPU and rebooted, it booted fine.

Offline

#4410 2015-03-06 18:28:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Lol as much as I would love that to work, it didn't...
After rebooting, Arch wouldn't detect my second GPU at all anymore.
Not with PCI (dmesg | grep pci-stub), and not with lspci.
Unplugged the display from the secondary GPU and rebooted, it booted fine.

Do you really expect us to debug your hardware issues?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4411 2015-03-07 00:14:53

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Do you really expect us to debug your hardware issues?

What? No, of course not! Look you guys seem to get irritated with me,  and I can understand that.
However that is not what I want to do.

If this is a hardware issue then sorry, I had no clue. I have a fair bit of Linux and CLI knowledge but this is completely new to me. I'm have been trying to make this work for a while now, and even replaced my CPU and motherboard so they both would support VT-d... I get irritated with it not working myself too. Even though i'm faced with new errors each time I have a bit of progress, i'm not giving up as it is something I want to learn and have working to get rid of the dual-boot.

If you guys are not interested in helping me anymore, then please say so, I will search for solutions myself. I'm really sorry for bothering you guys with all of this, but like I said before, I really really want this to work.

Offline

#4412 2015-03-07 00:37:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
aw wrote:

Do you really expect us to debug your hardware issues?

What? No, of course not! Look you guys seem to get irritated with me,  and I can understand that.
However that is not what I want to do.

If this is a hardware issue then sorry, I had no clue. I have a fair bit of Linux and CLI knowledge but this is completely new to me. I'm have been trying to make this work for a while now, and even replaced my CPU and motherboard so they both would support VT-d... I get irritated with it not working myself too. Even though i'm faced with new errors each time I have a bit of progress, i'm not giving up as it is something I want to learn and have working to get rid of the dual-boot.

If you guys are not interested in helping me anymore, then please say so, I will search for solutions myself. I'm really sorry for bothering you guys with all of this, but like I said before, I really really want this to work.

Sorry, I kind of blew a gasket at seeing that you weren't even expecting output from the graphics card itself, which is covered in the first post here and should be evident from any of the videos you can find of this working.  I suggest you actually try to understand the setup rather than blindly following your interpretation of the instructions and reporting the latest failures with minimal information.  If you ask informed and direct questions, I'll be happy to try to help.  If you don't understand something about the setup, ask rather than pushing forward with error reports.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4413 2015-03-07 13:42:42

terusus
Member
Registered: 2015-02-27
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

I've seen this too (no need to patch host drivers if you're using OVMF). Sadly I have no clue how to make it work though.
From what I've read you should launch Qemu with

qemu-system-x86_64 -L .

This while being in the directory where the UEFI bios.bin is in.
Well tried that, it did nothing, not even give me an UEFI shell.

Is there a proper guide to OVMF in Qemu out there that we can use?

You should first check if your card supports EFI: http://vfio.blogspot.com/2014/08/does-m … t-efi.html
And assuming you have read this tread you will understand this guide: http://vfio.blogspot.com/2014/08/primar … t-vga.html

Offline

#4414 2015-03-07 16:53:21

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just in case anyone has the same issue, I was running Win 7 under Slackware with qemu 2.1.0 using the command line and migrated to Arch.  For some reason, the VM bluescreened on a HDD error only when passing through the graphics output.  Finally got it working with a change in the script. Old version:

-enable-kvm -M q35 -m 8192 \ -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -smp 6,sockets=1,cores=6,thre$ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/dev/cdrom,id=cddisk -device ide-cd,bus=piix4-ide.1,drive=cddisk \ -net vde,sock=/var/run/kvm0.ctl,vlan=0 -net nic,vlan=0,macaddr=XX:XX:XX:XX:00:01,model=virtio \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -rtc base=localtime \ -monitor unix:/home/user/monitor,server,nowait

(Note that Slackware needs the sock=/var/run/kvm0.ctl,vlan=0, omitted for Arch)

-enable-kvm -rtc base=localtime \ -m 8192 -mem-path /dev/hugepages -smp 6,sockets=1,cores=6,threads=1 \ -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \ -vga none -nographic \ -monitor stdio -serial none -parallel none \ -nodefconfig \ -device intel-hda -device hda-output \ -drive file=/dev/mapper/cryptvg2-win7,cache=none,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device vfio-pci,host=0000:03:00.0,multifunction=on,addr=9.0,x-vga=on \ -device vfio-pci,host=0000:03:00.1,addr=9.1 -drive file=/dev/sr0,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 \ -net vde -net nic,vlan=0,macaddr=XX:XX:XX:XX:00:01,model=virtio \ -monitor unix:/home/user/monitor,server,nowait \

The extra parameters on the lines
-drive file=/dev/mapper/cryptvg2-win7,cache=none,if=none,id=drive0,aio=threads \
-drive file=/dev/sr0,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 \
were absolutely necessary in order to get the CD/DVD drive and the HDD working.  Oddly enough the change to 440fx required a reactivation but caused no other problems... but I lost my desktop background for some reason.

Last edited by mostlyharmless (2015-03-07 19:10:04)

Offline

#4415 2015-03-07 19:35:39

giantegg
Member
Registered: 2015-03-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is this the right place to ask questions? I cant really get it to run.
no errors up until I tried to run it the first time.

qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

The binding process seems to work or at least, no errors.

dmesg output:

[ 0.000000] Command line: initrd=\initramfs-linux-missingacs-i915arbiter.img cryptdevice=UUID=f68cbc3f-37b1-46a1-8214-9d38bdd55453:root root=UUID=d4a843e2-ee3b-4c3e-8369-10e96f2f8b30 rootflags=subvol=BTRFSROOT/root quiet ro rw i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6798,1002:aaa0 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 [ 0.000000] Kernel command line: initrd=\initramfs-linux-missingacs-i915arbiter.img cryptdevice=UUID=f68cbc3f-37b1-46a1-8214-9d38bdd55453:root root=UUID=d4a843e2-ee3b-4c3e-8369-10e96f2f8b30 rootflags=subvol=BTRFSROOT/root quiet ro rw i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6798,1002:aaa0 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream [ 0.456140] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.456154] pci-stub 0000:01:00.0: claimed by stub [ 0.456162] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.456170] pci-stub 0000:01:00.1: claimed by stub

Hardware is:
i5 3570k
gigabyte z77 ud3h vt-d enabled
sapphire 7970 ghz edition

Has anyone got this working with the same or similar hardware and/or can help?

Offline

#4416 2015-03-07 19:41:02

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Your cpu = Intel® Virtualization Technology for Directed I/O (VT-d) = NO

Offline

#4417 2015-03-07 19:45:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Your cpu = Intel® Virtualization Technology for Directed I/O (VT-d) = NO

Yup - http://ark.intel.com/products/65520/Int … o-3_80-GHz


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4418 2015-03-07 19:55:28

giantegg
Member
Registered: 2015-03-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmmh thanks. I thought it supported it since it was in the bios options

EDIT: can someone please tell me why the 3570 supports vt-d but the 3570k doesnt ? How does this make any sense ?

Last edited by giantegg (2015-03-07 20:02:49)

Offline

#4419 2015-03-07 22:53:25

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Sorry, I kind of blew a gasket at seeing that you weren't even expecting output from the graphics card itself, which is covered in the first post here and should be evident from any of the videos you can find of this working.

Well I can understand that, however I have not read this anywhere in this thread. Looking back I can still not find where it says that, but i'm probably just blind. Doesn't matter anyway, I know now.

aw wrote:

I suggest you actually try to understand the setup rather than blindly following your interpretation of the instructions and reporting the latest failures with minimal information.

The thing is, I am trying to understand it, and i'm actually understanding it more and more.
This is my way of learning, I can't help it...

aw wrote:

If you ask informed and direct questions, I'll be happy to try to help.  If you don't understand something about the setup, ask rather than pushing forward with error reports.

I actually thought I was doing this, but clearly not. Sorry for that...


Anyways you'll be happy to know I finally got it working. Not sure why but it now detected the GPU properly with the monitor plugged in. In the end I didn't need to use OVMF and/or the NVIDIA driver patch.
So for the one that makes that spreadsheet, I run the following:
Host: Arch Linux 3.18.5 kernel with the ACS patch (from the first post), NVIDIA GTX 980
Guest: Windows 8.1, NVIDIA GTX 770
Motherboard: Asrock Extreme 6/ac
CPU: i7 4790k


One last question: is there any way to share the keyboard and mouse between the guest and host? I now just passthrough the specific usb devices, and although this works, I can't do anything in my host anymore until I turn off the VM.

Offline

#4420 2015-03-07 23:13:23

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
aw wrote:

Sorry, I kind of blew a gasket at seeing that you weren't even expecting output from the graphics card itself, which is covered in the first post here and should be evident from any of the videos you can find of this working.

Well I can understand that, however I have not read this anywhere in this thread. Looking back I can still not find where it says that, but i'm probably just blind. Doesn't matter anyway, I know now.

aw wrote:

I suggest you actually try to understand the setup rather than blindly following your interpretation of the instructions and reporting the latest failures with minimal information.

The thing is, I am trying to understand it, and i'm actually understanding it more and more.
This is my way of learning, I can't help it...

aw wrote:

If you ask informed and direct questions, I'll be happy to try to help.  If you don't understand something about the setup, ask rather than pushing forward with error reports.

I actually thought I was doing this, but clearly not. Sorry for that...


Anyways you'll be happy to know I finally got it working. Not sure why but it now detected the GPU properly with the monitor plugged in. In the end I didn't need to use OVMF and/or the NVIDIA driver patch.
So for the one that makes that spreadsheet, I run the following:
Host: Arch Linux 3.18.5 kernel with the ACS patch (from the first post), NVIDIA GTX 980
Guest: Windows 8.1, NVIDIA GTX 770
Motherboard: Asrock Extreme 6/ac
CPU: i7 4790k


One last question: is there any way to share the keyboard and mouse between the guest and host? I now just passthrough the specific usb devices, and although this works, I can't do anything in my host anymore until I turn off the VM.


Try Synergy, alternatively you could passthrough one of your usb controllers,  my Case has 2 frontal usb3 ports, and 2 usb2 ports, so i passthrough the usb3 controller to windows and i switch my KB/M by changing the ports they're connected to

Last edited by nbhs (2015-03-07 23:13:46)

Offline

#4421 2015-03-07 23:49:56

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Try Synergy, alternatively you could passthrough one of your usb controllers,  my Case has 2 frontal usb3 ports, and 2 usb2 ports, so i passthrough the usb3 controller to windows and i switch my KB/M by changing the ports they're connected to

Well i'm not going to pay for Synergy as good as it may be.
But your other solution is actually pretty smart, i'm going to use that. Thanks for all your help guys!

Offline

#4422 2015-03-07 23:53:20

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
nbhs wrote:

Try Synergy, alternatively you could passthrough one of your usb controllers,  my Case has 2 frontal usb3 ports, and 2 usb2 ports, so i passthrough the usb3 controller to windows and i switch my KB/M by changing the ports they're connected to

Well i'm not going to pay for Synergy as good as it may be.
But your other solution is actually pretty smart, i'm going to use that. Thanks for all your help guys!

Synergy is free and libre afaik

Offline

#4423 2015-03-07 23:55:53

blu3bird
Member
Registered: 2014-08-20
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Specifying a CPU topology with threads is generally only recommended in combination with pinning.  Using 100% of the host CPU resources for a guest is also likely to cause glitches when the host needs some CPU resources.  There's plenty of information online for vCPU pinning.  libvirt can do this for you and is detailed on their site.

Could you elaborate on that? I always assumed the point of cpu pinning was to reserve/restrict cpu usage, regardless of topology. For example to preventing a test vm from stealing resources from a production vm.

The reason I'm asking is that I'm also using threads without pinning. When I first tried kvm with vfio I did some benchmarks(cinebench) and threads=8,cores=1,sockets=1 gave me the best results for a quad-core cpu with ht.

Offline

#4424 2015-03-07 23:58:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
PureTryOut wrote:
nbhs wrote:

Try Synergy, alternatively you could passthrough one of your usb controllers,  my Case has 2 frontal usb3 ports, and 2 usb2 ports, so i passthrough the usb3 controller to windows and i switch my KB/M by changing the ports they're connected to

Well i'm not going to pay for Synergy as good as it may be.
But your other solution is actually pretty smart, i'm going to use that. Thanks for all your help guys!

Synergy is free and libre afaik

It changed recently, you can still get the source and build it yourself, or $10 for lifetime access to binaries.  Worth it, IMHO.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4425 2015-03-08 00:15:17

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blu3bird wrote:
aw wrote:

Specifying a CPU topology with threads is generally only recommended in combination with pinning.  Using 100% of the host CPU resources for a guest is also likely to cause glitches when the host needs some CPU resources.  There's plenty of information online for vCPU pinning.  libvirt can do this for you and is detailed on their site.

Could you elaborate on that? I always assumed the point of cpu pinning was to reserve/restrict cpu usage, regardless of topology. For example to preventing a test vm from stealing resources from a production vm.

The reason I'm asking is that I'm also using threads without pinning. When I first tried kvm with vfio I did some benchmarks(cinebench) and threads=8,cores=1,sockets=1 gave me the best results for a quad-core cpu with ht.

Here's an article i found very interesting http://www-01.ibm.com/support/knowledge … nprocs.htm, to restrict or reserve cpu usage i believe you need something like cgroups, see my previous post

Offline

#4426 2015-03-08 02:40:33

stl
Member
Registered: 2015-03-02
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the great guide here.  I've been trying to set this up lately and have had some success.

Hardware:
Asus Z97-A motherboard
Intel Core i7-4790K cpu
AMD Radeon HD 7850

Host system:
Arch Linux x86_64
default kernel and qemu, unpatched

Windows 7 test, with default BIOS:
Command line: QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_SIZE_IN_USEC=100 QEMU_ALSA_DAC_DEV=dac qemu-system-x86_64 -m 4G -enable-kvm -machine type=pc,accel=kvm -cpu host -smp cores=4,threads=2 -drive file=w7.img,media=disk,cache=none,format=raw,discard=unmap,detect-zeroes=unmap,if=virtio -vga none -nographic -net nic,model=virtio -net user -soundhw hda -vga none -nographic -device vfio-pci,host=01:00.0,x-vga=on -usb -device usb-host,hostbus=3,hostport=13 -device usb-host,hostbus=3,hostport=14 -monitor telnet::4444,server,nowait
Results:  Mostly flawless.  GPU is recognized fully by its drivers, games run well.  3Dmark crashes but I'm not concerned about that really.  The only issue is an occasional crackle/skip in the audio.  It doesn't happen frequently enough to really bother me, but I'd still like to get rid of it if possible.

Windows 8 test, with OVMF (using https://aur.archlinux.org/packages/ovmf-bin/ ):
Command line: QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_SIZE_IN_USEC=100 QEMU_ALSA_DAC_DEV=dac qemu-system-x86_64 -m 4G -enable-kvm -machine type=pc,accel=kvm -cpu host -smp cores=4,threads=2 -drive file=w8.img,media=disk,cache=none,format=raw,discard=unmap,detect-zeroes=unmap,if=virtio -vga none -nographic -net nic,model=virtio -net user -soundhw hda -vga none -nographic -device vfio-pci,host=01:00.0,x-vga=on -usb -device usb-host,hostbus=3,hostport=13 -device usb-host,hostbus=3,hostport=14 -monitor telnet::4444,server,nowait
Results: No video from the AMD GPU when using gpu passthrough with vfio.  Just a black screen.  No errors from QEMU or in dmesg.  VM boots fine without gpu passthrough, with -vga std.  Leaving out x-vga=on didn't change anything.

Audio:
I get an occasional crackle/skip in the audio.  It isn't a huge bother.  But what I'd really like to do is to avoid going through alsa and instead pass my USB DAC to the VM using USB passthrough.  That doesn't seem to work though.  QEMU gives me this:

qemu-system-x86_64: libusb_claim_interface: -5 [NOT_FOUND] qemu-system-x86_64: libusb_set_configuration: -6 [BUSY] qemu-system-x86_64: libusb_set_configuration: -6 [BUSY] qemu-system-x86_64: libusb_set_configuration: -6 [BUSY]

And in dmesg I get lots of this:

[ +0.201511] usb 3-4: reset full-speed USB device number 2 using xhci_hcd [ +0.211482] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff8804396d53c0 [ +0.169216] usb 3-4: usbfs: interface 0 claimed by usbfs while 'qemu-system-x86' sets config #1 [ +0.158978] usb 3-4: reset full-speed USB device number 2 using xhci_hcd [ +0.211597] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff8804396d53c0 [ +0.194835] usb 3-4: reset full-speed USB device number 2 using xhci_hcd [ +0.211542] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff8804396d53c0

Any ideas on this?  The DAC shouldn't be in use from anything after I've removed the snd-usb-audio module.  Inside the VM, I get "This device cannot start (Code 10)."

Offline

#4427 2015-03-08 11:20:25

gmk
Member
Registered: 2015-03-08
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

after avoiding archlinux and testing with debian, ubuntu and proxmox, out of a fear of having to do too much fine tuning to make things work, I was pleasantly surprised. In the other mentioned distributions I couldn't get passthrough to work or ran into issues when it worked.

@nbhs thank you for the guide, with the kernel you provide and the settings you suggest everything works more or less "out of the box"
I am not savvy on using a custom kernel though, how likely is it that updating the host will break the setup?


I have tested two windows 7 VMs (one for each graphics card). Both work fine, benchmark results are great, not far off from the ones I got with vmware esxi 5.5
my hardware:
i7-4790k
Asrock z97 Pro4
Radeon R9 280, msi twin frozr
Radeon 4850, asus

The only issue: as soon as hdtune (in windows) is started, the vm locks up and the host reports a segfault. Can anyone point me in the direction where to best report this behaviour/bug or even how to fix it?

Last edited by gmk (2015-03-08 11:21:25)

Offline

#4428 2015-03-08 11:41:04

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gmk wrote:

after avoiding archlinux and testing with debian, ubuntu and proxmox, out of a fear of having to do too much fine tuning to make things work, I was pleasantly surprised. In the other mentioned distributions I couldn't get passthrough to work or ran into issues when it worked.

@nbhs thank you for the guide, with the kernel you provide and the settings you suggest everything works more or less "out of the box"
I am not savvy on using a custom kernel though, how likely is it that updating the host will break the setup?


I have tested two windows 7 VMs (one for each graphics card). Both work fine, benchmark results are great, not far off from the ones I got with vmware esxi 5.5
my hardware:
i7-4790k
Asrock z97 Pro4
Radeon R9 280, msi twin frozr
Radeon 4850, asus


You can have as many kernels installed on arch as you want, upgrading is not an issue, it wont break anything

gmk wrote:

The only issue: as soon as hdtune (in windows) is started, the vm locks up and the host reports a segfault. Can anyone point me in the direction where to best report this behaviour/bug or even how to fix it?

Post your logs, i assume you've done this already

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Offline

#4429 2015-03-08 16:53:06

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hurenkam wrote:

Passing a secondary device (from host perspective) is indeed easier.
However, after some fiddling, i managed to passthrough the primary card on my system, but it does involve some extra work.
Apart from the usual driver blacklisting (radeon in my case) i had to do the following as well:
- Switch grub to text-only mode by adding "GRUB_GFXPAYLOAD_LINUX=text" to /etc/defaults/grub
- Switch linux boot to text-only mode by adding "nomodeset nofb" to GRUB_CMDLINE_LINUX_DEFAULT=... in /etc/defaults/grub

After adding these parameters, I was able to passthrough the host primary graphics to qemu guest in the same way as i passthrough my secondary graphics.

Note that you will completely lose your host console when you do this, so make sure you can ssh in, and perhaps have a serial terminal ready as well.

Hi . I am also interested at running a headless server .

I tried your method but the whole machine hangs as soon as I start the VM that uses the host's primary GPU .

I blacklisted nouveau and added "nomodeset nofb" to my kernel options . Still no go sad

I'm using UEFI host with Gummiboot boot manager if that would matter .

EDIT : Setting rombar=0 solves the hanging issue !!! The VM boots fine and I can even use VNC to access it m but no HDMI output .

I must obtain a ROM file and test ASAP smile

EDIT2 : Still no luck :

vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xd8000000-0xd9ffffff 64bit pref]

Last edited by Denso (2015-03-08 17:35:26)

Offline

#4430 2015-03-08 17:41:40

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
hurenkam wrote:

Passing a secondary device (from host perspective) is indeed easier.
However, after some fiddling, i managed to passthrough the primary card on my system, but it does involve some extra work.
Apart from the usual driver blacklisting (radeon in my case) i had to do the following as well:
- Switch grub to text-only mode by adding "GRUB_GFXPAYLOAD_LINUX=text" to /etc/defaults/grub
- Switch linux boot to text-only mode by adding "nomodeset nofb" to GRUB_CMDLINE_LINUX_DEFAULT=... in /etc/defaults/grub

After adding these parameters, I was able to passthrough the host primary graphics to qemu guest in the same way as i passthrough my secondary graphics.

Note that you will completely lose your host console when you do this, so make sure you can ssh in, and perhaps have a serial terminal ready as well.

Hi . I am also interested at running a headless server .

I tried your method but the whole machine hangs as soon as I start the VM that uses the host's primary GPU .

I blacklisted nouveau and added "nomodeset nofb" to my kernel options . Still no go sad

I'm using UEFI host with Gummiboot boot manager if that would matter .

EDIT : Setting rombar=0 solves the hanging issue !!! The VM boots fine and I can even use VNC to access it m but no HDMI output .

I must obtain a ROM file and test ASAP smile

EDIT2 : Still no luck :

vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xd8000000-0xd9ffffff 64bit pref]

Boot with video=efifb:off, i think i explained somewhere in the previous 100 pages or so how i did it.

Edit: here https://bbs.archlinux.org/viewtopic.php … 2#p1427202

Last edited by nbhs (2015-03-08 17:44:06)

Offline

#4431 2015-03-08 18:13:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blu3bird wrote:
aw wrote:

Specifying a CPU topology with threads is generally only recommended in combination with pinning.  Using 100% of the host CPU resources for a guest is also likely to cause glitches when the host needs some CPU resources.  There's plenty of information online for vCPU pinning.  libvirt can do this for you and is detailed on their site.

Could you elaborate on that? I always assumed the point of cpu pinning was to reserve/restrict cpu usage, regardless of topology. For example to preventing a test vm from stealing resources from a production vm.

The reason I'm asking is that I'm also using threads without pinning. When I first tried kvm with vfio I did some benchmarks(cinebench) and threads=8,cores=1,sockets=1 gave me the best results for a quad-core cpu with ht.

It's a complicated issue, probably worthy of research papers, but the thing to keep in mind is that physically, threads on the same core share execution units.  A process that can run at 100% on an otherwise idle core may only run at 75% performance if the other thread of that core is under load.  We can theoretically run two instances of that process simultaneously on a single core with hyperthreading, but our aggregate throughput would be only 150% whereas running those two instances on separate cores would result in 200% throughput.  This is where we rely on the scheduler knowing the difference between threads on the same core and separate cores.

Moving that to a virtual machine, the host doesn't implicitly know how any given vCPU is exposed in the guest, they're all equal processes for host scheduling.  By exposing threads to the guest without pinning, the guest is making scheduling decisions that don't necessarily make sense.  Maybe these work out, maybe they don't.

If we pin vCPUs in the host and expose the topology in the guest, then the topology we expose to the guest actually holds true to the vCPU processes on the host.  The guest also has more visibility than the host to running process threads from the same application on vCPU threads and cores that share cache.  The host can't know that vCPUs are running related code and therefore may make poor decisions when migrating a vCPU task to another core or thread.

In your case, you've told the guest that it has a single core with 8 threads, which I suspect the guest scheduler would handle the same as a single 8-core processor or the same as 8 single-core processors.  IOW, it doesn't really matter that they're threads vs cores vs sockets because they're all equal.  In your case, maybe it is worthwhile to ignore the locality of threads to cores since the guest can't make any useful scheduling decisions between cores with un-pinned vCPUs anyway.  Exposing 8-cores or 8-sockets to the guest also implies to the guest that each execution unit is independent of others, which is not actually true of the physical hardware.  The 8-core or 8-socket model would effectively be over-committing the host resources whereas 8-threads seems closer to accurate.  Maybe you're on to something, but I'd still expect pinning with an accurate physical topology exposed to the guest to provide a slight advantage.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4432 2015-03-08 18:37:46

gmk
Member
Registered: 2015-03-08
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
gmk wrote:

The only issue: as soon as hdtune (in windows) is started, the vm locks up and the host reports a segfault. Can anyone point me in the direction where to best report this behaviour/bug or even how to fix it?

Post your logs, i assume you've done this already

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

yes, "ignore_msrs=1" is in use as it was required for passmark to work, otherwise it would cause a bsod (but no segfault/crash of qemu)


the relevant logs I found:

Mar 08 20:07:57 archy kernel: qemu-system-x86[9157]: segfault at 7ffffcb17000 ip 00007f8643f8c5be sp 00007ffffcb15c88 error 6 in libc-2.21.so[7f8643e65000+199000] Mar 08 20:08:05 archy systemd-coredump[9190]: Coredump of 9157 (qemu-system-x86) is larger than configured processing limit, refusing. Mar 08 20:08:05 archy systemd-coredump[9190]: Process 9157 (qemu-system-x86) of user 0 dumped core.

the vm is using a raw-imagefile that resides on an ext4 partition (dd'ing that image to a lvm-partition yielded the same result)

Offline

#4433 2015-03-08 20:27:46

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Boot with video=efifb:off, i think i explained somewhere in the previous 100 pages or so how i did it.

Edit: here https://bbs.archlinux.org/viewtopic.php … 2#p1427202

Hi . Thanks for sharing this !

Indeed "video=efifb:off" solves the hanging issue without editing the VM script , but now I'm faced with with :

vfio-pci 0000:01:00.0: Invalid ROM contents vfio-pci 0000:01:00.0: Invalid ROM contents

I used a ROM file dumped from GPU-Z as well as a downloaded one from TPU but without any luck .

Trying to dump the ROM from host while GPU isn't initialized produces 0 byte file .

EDIT :  This is a GTX 770 by the way .

Last edited by Denso (2015-03-08 20:28:24)

Offline

#4434 2015-03-08 20:36:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
nbhs wrote:

Boot with video=efifb:off, i think i explained somewhere in the previous 100 pages or so how i did it.

Edit: here https://bbs.archlinux.org/viewtopic.php … 2#p1427202

Hi . Thanks for sharing this !

Indeed "video=efifb:off" solves the hanging issue without editing the VM script , but now I'm faced with with :

vfio-pci 0000:01:00.0: Invalid ROM contents vfio-pci 0000:01:00.0: Invalid ROM contents

I used a ROM file dumped from GPU-Z as well as a downloaded one from TPU but without any luck .

Trying to dump the ROM from host while GPU isn't initialized produces 0 byte file .

EDIT :  This is a GTX 770 by the way .

I had this problem before with my 470gtx, reboot your pc, start your vm, the first vm boot should start ok, dump your rom with gpu-z, transfer it to your host, then add romfile=... to your script

Last edited by nbhs (2015-03-08 20:38:54)

Offline

#4435 2015-03-08 21:01:26

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I had this problem before with my 470gtx, reboot your pc, start your vm, the first vm boot should start ok, dump your rom with gpu-z, transfer it to your host, then add romfile=... to your script

GPU-Z won't produce a full "pre-POST" ROM . the only way to do so is to install the card as a secondary card on host , then dump the image from there .

Please feel free to correct me if I'm wrong .

Offline

#4436 2015-03-08 22:13:59

n0rv
Member
Registered: 2012-02-27
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi.
Anyone can confirm that ASRock Z97 Extreme3 motherboard supports vt-d?

I know that Pro4 supports it (and Extreme4 / 6 probably too), but I would like to buy Extreme3 because of better layout.

Last edited by n0rv (2015-03-08 22:14:29)

Offline

#4437 2015-03-09 02:44:16

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

n0rv wrote:

Hi.
Anyone can confirm that ASRock Z97 Extreme3 motherboard supports vt-d?

I know that Pro4 supports it (and Extreme4 / 6 probably too), but I would like to buy Extreme3 because of better layout.

A user named noctavian put a comprehensive list of success/failure cases here -

https://docs.google.com/spreadsheet/ccc … _web#gid=0

Check for your motherboard and GPU on that list.

- Peter

Offline

#4438 2015-03-09 02:50:14

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Audio:
I get an occasional crackle/skip in the audio.  It isn't a huge bother.  But what I'd really like to do is to avoid going through alsa and instead pass my USB DAC to the VM using USB passthrough.

I used to think that I need to get a dedicated sound because of a stutter issue with ICH6 and ICH9 - it turns out many people are having issues with the ICH6/9 emulation - something to do with small buffer size. I switched over to AC97 and it works fine. In Win7, I had to install the Realtek AC97 driver. Win8.1 barfs at the Realtek AC97 driver because it is unsigned, but you can disable that protection and it will install fine. Once I switched to AC97, sound works flawlessly.

- Peter

Last edited by pkim (2015-03-09 04:37:10)

Offline

#4439 2015-03-09 02:59:03

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

giantegg wrote:

hmmh thanks. I thought it supported it since it was in the bios options

EDIT: can someone please tell me why the 3570 supports vt-d but the 3570k doesnt ? How does this make any sense ?

There is really no good reason other than that Intel is trying to be strategic. They think the Vt-d is a "corporate feature" along with vpro, etc. And since the "K" processors are targeted at the over-clock happy gamers, the VM features don't go together with those "gamer" processors - so the logic goes. I kind of got lucky - I really wanted the VM features, but the non-K processors were much lower on the bang-per-buck ratio. I bought a i7-3790K anyway (not knowing how good/bad the VM support woudl be), but it turns out Vt-d is one of the VM features that it supports - for the first time for a K processor. I don't know why Intel decided to start including Vt-d on the K processor. Maybe they thought VM features are starting to become more mainstream? I really don't know why Vt-d is part of the new K-type processor feature set but I'm happy with my 3790K.

- Peter

Offline

#4440 2015-03-09 04:35:36

pkim
Member
Registered: 2014-12-27
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

EDIT 3 : i finally tried using -vga none and -nographics menu, now the black qemu monitor shows up, then when i plug in my monitor to my secondary GPU ( the passthrough GPU ), finally i am able to see my VM using -vga none.. but this vm installs windows 8 with any installed NVIDIA drivers yet,  and also, this only works on the first boot of the VM, lets say i kill the VM and i change the script, then i start my VM again, the output won't show up again in the passthrough'd monitor, the only way to make it show up again is by rebooting the host.. and after it reboots i can only boot the VM again once... if i boot the VM twice, starting from that time it won't show the output again unless i reboot one more time..

any help on this please !! i think im almost there..

I had the same exact problem with my 560Ti card - it would work in the guest once but after shutting down the guest, it wouldn't work again - probably a failure to cleanly reset the hardware. Also I couldn't get it to work on Win8.1 with UEFI, the firmware wasn't compatible with UEFI. I ended up throwing a GTX 970 at the guest - and it solved both problems: win8.1 UEFI compatibility and also I can shutdown and restart the guest an infinite number of times. I sold the 560Ti card and some old computer hardware on ebay and bought a GTX 960 for the host so I now have a very nice setup - with the Vt-d processor and IOMMU motherboard, I really have two machines in one. And the new GTX 960 is also more power efficient than the 560Ti (about 20W - I estimate about $36 per year in energy savings) - so more money savings there - just a quick note to those of you who is overclocking your CPU thinking you are saving money on the CPU price and getting more computation power, think again: the power ramp up is exponential with the clock frequency/supply power and the money you saved on the CPU will be quickly eaten up by the electricity bill.

If you're sticking with your 560Ti card, please know that I was able to get mine to reset with the host pm-suspend - still a bit inconvenient, but beats rebooting.

- Peter

Offline

#4441 2015-03-09 17:56:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yikes.
So i've had my windows 7 system installed on bare metal into a partition on my hard drive.
Then i've made windows 8.1 virtual machine, and connected that partition to the virtual machine via virtio-blk-pci, just for data transferring needs.
That way, following cases are possible:
1. Linux host is offline since windows 7 is booted, accessing the partition.
2. Linux host is online, and the partition is mounted in host system for file transfers.
3. Linux host is online, partition is unmounted and VM boots, grabbing access to that partition. Linux host can not mount that partition, since mount is preventing me to shoot my leg.

Common sense hints me that EVERYTHING should be fine as long as no two systems use same partition concurrently.

But windows 8.1 defies common sense. Seems like it frigging starts defragging that darn partition if the user is idle(like, when i'm doing something on the host). I'll omit the heavy i/o issues generated by that, but..
I've shut down the guest via regular and fully legit way. I thought it would finish whatever it was doing with the disk and leave it free.

NOPE
I've started my windows 7 system, and a wild CHKDSK message appears. Let it go, i thought, and... CHKDSK uses filesystem checking! It's not very effective.. 40 gigabytes of data is now corrupted.
YAY FOR WINDOWNS!

Interesting, if i'll have two PCs and connect a physical hard drive in a similar way(like, came to work, plugged in my hard drive into windows 8 system, shutted it down, came home, plugged the hard disk back into windows 7 machine), will it go nuts as well?

Morale:
You shouldn't let VM touch your dual boot partition, especially if it is windows 8 vm.

Last edited by Duelist (2015-03-09 18:03:08)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4442 2015-03-09 18:01:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pkim wrote:

a quick note to those of you who is overclocking your CPU thinking you are saving money on the CPU price and getting more computation power, think again: the power ramp up is exponential with the clock frequency/supply power and the money you saved on the CPU will be quickly eaten up by the electricity bill.

But what if my CPU was free when i got it? wink
What if the electricity at my area costs less than 0.02$ per kw-h? wink


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4443 2015-03-09 21:50:17

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

I had this problem before with my 470gtx, reboot your pc, start your vm, the first vm boot should start ok, dump your rom with gpu-z, transfer it to your host, then add romfile=... to your script

GPU-Z won't produce a full "pre-POST" ROM . the only way to do so is to install the card as a secondary card on host , then dump the image from there .

Please feel free to correct me if I'm wrong .

Indeed, but it can still work anyway. At least worked for my NVIDIA cards.

Offline

#4444 2015-03-10 05:26:15

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am having the most bizarre problem with vfio and NIC device. My motherboard has 2 network interfaces, Intel e1000e as well as an Atheros AR8161, so I planned on passing the Atheros device to guest along with GPU. However, when the guest is not running, the Atheros NIC is held by pci-stub/vfio-pci driver, and brings my entire home network down when ethernet cable is plugged in. What could possibly be going on here? Anyone experience anything similar or have an idea what could be happening? Thanks!

Also, aw, on your vfio.blogspot.com FAQ, the use of vfio-pci.ids= is mentioned. Is this option no longer valid (it seems I could not get it to work the way pci-stub.ids= works)?

Last edited by mutiny (2015-03-10 05:40:36)

Offline

#4445 2015-03-10 08:05:29

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Indeed, but it can still work anyway. At least worked for my NVIDIA cards.

It doesn't work when trying to passthrough the host's primary GPU unfortunately : (

Offline

#4446 2015-03-10 11:18:41

efeu
Member
Registered: 2015-03-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

n0rv wrote:

Hi.
Anyone can confirm that ASRock Z97 Extreme3 motherboard supports vt-d?

I know that Pro4 supports it (and Extreme4 / 6 probably too), but I would like to buy Extreme3 because of better layout.

I am running Ubuntu 14.10 as host on ASRock Z97 Extreme3 with an i7 4790K and Win8.1 as VM. The Intel HD as primary and a nVidia GTX970 as passthrough adapter.
Everything is running well out of the box with a fresh build of OVMF UEFI instead of sea bios. So I am able to use stock kernel without any patches.

Offline

#4447 2015-03-10 13:22:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:

I am having the most bizarre problem with vfio and NIC device. My motherboard has 2 network interfaces, Intel e1000e as well as an Atheros AR8161, so I planned on passing the Atheros device to guest along with GPU. However, when the guest is not running, the Atheros NIC is held by pci-stub/vfio-pci driver, and brings my entire home network down when ethernet cable is plugged in. What could possibly be going on here? Anyone experience anything similar or have an idea what could be happening? Thanks!

Also, aw, on your vfio.blogspot.com FAQ, the use of vfio-pci.ids= is mentioned. Is this option no longer valid (it seems I could not get it to work the way pci-stub.ids= works)?

Sorry, vfio-pci.ids was complete mis-information, but I'm working to make it true.  I've never heard of a NIC affecting the network when simply bound to pci-stub or vfio-pci.  The e1000e may be a better target for device assignment.  I can't say i'm surprised that an atheros NIC has issues.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4448 2015-03-10 17:39:32

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wonder if anyone had been using asus A88x-Pro with kvm. I had it setup with xen 4.4.3 and I was able to pass R9 270 and couple of other  ati cards. I was using 1201 bios version. I had flashed latest bios 1701. I can boot windows 8.1 without ati drivers and with vga none my R 270 displays out put but the moment I had installed ati drivers the system only boots to blue screen it get stuck and does not show desktop.  I had to use this options vfio_iommu_type1 disable_hugepages=1. to work around bios bug.
Any tips on geting that motherboard working with kvm ? How I can get uefi emulation working under kvm. So far I had it load uefi but i dont see any options to boot from my hard drive,

Offline

#4449 2015-03-10 20:15:19

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry if this is a hijack, but I felt it was in paralell to OP.

tl;dr: Does anyone have a way of asking Gerd Hoffman to build OVMF with http://sourceforge.net/p/edk2/mailman/message/30377799/ this patch?

After a ton of digging to what seemed like the bottom of the internet, I found out Windows 7 on OVMF (uefi) won't boot to a vfio bound GPU due to it needing something called an int 10h.  I think: windows 7 will work on UEFI outside of vfio OVMF because CSM enables the int 10h legacy or something to move windows along.  Someone suggested making it possible for OVMF to allow this without(?) CSM in the link above.

My problem is I totally suck at building EDK2 and after a couple days of failing I'd like to see if anyone knows how to contact the owner of that repo to try a special build.

Offline

#4450 2015-03-10 20:25:49

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:

Sorry if this is a hijack, but I felt it was in paralell to OP.

tl;dr: Does anyone have a way of asking Gerd Hoffman to build OVMF with http://sourceforge.net/p/edk2/mailman/message/30377799/ this patch?

After a ton of digging to what seemed like the bottom of the internet, I found out Windows 7 on OVMF (uefi) won't boot to a vfio bound GPU due to it needing something called an int 10h.  I think: windows 7 will work on UEFI outside of vfio OVMF because CSM enables the int 10h legacy or something to move windows along.  Someone suggested making it possible for OVMF to allow this without(?) CSM in the link above.

My problem is I totally suck at building EDK2 and after a couple days of failing I'd like to see if anyone knows how to contact the owner of that repo to try a special build.

That's where i begin to feel really, REALLY old. Int 10h is, basically, call "bios function". You set up some registers, then do int 10h and the machine does stuff.
Kek point is - there's certain things that UEFI really wouldn't like, in my opinion. Like, using int 10h you could output text(say hi to VGA maybe?) or change the video mode, if i recall correctly. I remember correctly: http://en.wikipedia.org/wiki/INT_10H http://en.wikipedia.org/wiki/BIOS_interrupt_call
I guess you can't have int 10h working with GOP.

I can't comprehend the patch' insides, but this is very, very interesting.

Also, i've recently tried to install windows 7 on hardware UEFI platform, and failed just like i failed on the VM: stuck when windowns tries to output something on screen.

I guess if you could've feed it with some video drivers(there's some ways to do this, but i ain't no windows magician) before the install - it might work.

Seems like so far my guess is proving true - windows 7 doesn't work with GOP, it can't output graphics that way. If you fetch it drivers which do their own magic way(you can't change resolution on windowns 8 until you install drivers) outputting video - it should work.

Last edited by Duelist (2015-03-10 20:47:34)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4451 2015-03-10 20:31:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I wonder if anyone had been using asus A88x-Pro with kvm. I had it setup with xen 4.4.3 and I was able to pass R9 270 and couple of other  ati cards. I was using 1201 bios version. I had flashed latest bios 1701. I can boot windows 8.1 without ati drivers and with vga none my R 270 displays out put but the moment I had installed ati drivers the system only boots to blue screen it get stuck and does not show desktop.  I had to use this options vfio_iommu_type1 disable_hugepages=1. to work around bios bug.
Any tips on geting that motherboard working with kvm ? How I can get uefi emulation working under kvm. So far I had it load uefi but i dont see any options to boot from my hard drive,

https://docs.google.com/spreadsheet/ccc … _web#gid=0
Check there for A88X. I remember someone did this on A88X.

BTW, which CPU do you have? Since the Unified North Bridge is hidden inside the CPU, it doesn't depends much on the motherboard.

My experience is... flawful, if you can say it like that. I have Athlon x4 750k and asus F2A55, and i experience TONS OF GLITCHES, PERFORMANCE HITS AND STUFF, but at least it works.
So far:
1. x-vga is semi-working. You might get it working, but eventually i needed the mlock=on force switch(it's on by default!) and it sometimes crashed resulting in host shutdown.
2. i couldn't get hugepages working using hugetlbfs.
3. some guys report performance speed ups when using npt=0, which is counter-intuitive.
4. i've got myself a fully UEFI-compatible guest with no problems, except for broken GPU rom.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4452 2015-03-10 20:43:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, int10h is a gateway to the legacy VGA BIOS and would imply that you have VGA access to the device.  It's effectively taking the legacy part of the CSM that we're trying to avoid and using it anyway.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4453 2015-03-10 20:54:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yep, int10h is a gateway to the legacy VGA BIOS and would imply that you have VGA access to the device.  It's effectively taking the legacy part of the CSM that we're trying to avoid and using it anyway.

Okay...
But how does windows7 insanertaller outputs video? It works on pure-efi OVMF, but suddenly the real system breaks when trying to output graphics.
So, the installer supports GOP, and the system doesn't? That's weird as hell, but since it windowns - i could believe that. Maybe there's a special driver loaded somewhere.. Wait a second, it said "windows can't continue installation in safe mode" somewhere.. I've forgot all the testing i've done, dang it.

BTW, my windowns 8 VM refuses to read my real win7 partition, saying it's filesystem is "RAW", while i'm able to boot from that "RAW" filesystem. Heh, broken windows.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4454 2015-03-10 21:02:50

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yep, int10h is a gateway to the legacy VGA BIOS and would imply that you have VGA access to the device.  It's effectively taking the legacy part of the CSM that we're trying to avoid and using it anyway.

With all due respect sir, does that mean we will not have the capability (support in tianocore) of a win7 VM using a GPU that has been passed through via OVMF/VFIO?

That's a real shame, I don't have a win8 product key, and I certainly was not going to go buy one. sad

Edit: I also have not found anything relating as to why the CSM is bad, or why the CSM won't boot win7. Would you shed some light on that?

Last edited by Naruni (2015-03-10 21:03:57)

Offline

#4455 2015-03-10 21:10:55

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
aw wrote:

Yep, int10h is a gateway to the legacy VGA BIOS and would imply that you have VGA access to the device.  It's effectively taking the legacy part of the CSM that we're trying to avoid and using it anyway.

With all due respect sir, does that mean we will not have the capability (support in tianocore) of a win7 VM using a GPU that has been passed through via OVMF/VFIO?

That's a real shame, I don't have a win8 product key, and I certainly was not going to go buy one. sad

Edit: I also have not found anything relating as to why the CSM is bad, or why the CSM won't boot win7. Would you shed some light on that?

Try my theory - fetching the drivers for GPU before the install. Some OEMs do that, so there's definitely a way to do this. But it seems like windowns 7 can not into UEFI at all.

Regarding your edit - CSM includes VGA. VGA is bad. CSM will boot win7, but you won't have the advantages of UEFI.

Last edited by Duelist (2015-03-10 21:12:36)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4456 2015-03-10 21:10:55

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
aw wrote:

Yep, int10h is a gateway to the legacy VGA BIOS and would imply that you have VGA access to the device.  It's effectively taking the legacy part of the CSM that we're trying to avoid and using it anyway.

With all due respect sir, does that mean we will not have the capability (support in tianocore) of a win7 VM using a GPU that has been passed through via OVMF/VFIO?

That's a real shame, I don't have a win8 product key, and I certainly was not going to go buy one. sad

The benefit of OVMF for GPU assignment is when you can use it to avoid the need for VGA access to the device.  That requires a legacy-free ROM on the device, a legacy-free firmware for the VM and a legacy-free guest operating system.  Here you have a non-legacy-free operating system imposing legacy attributes on the VM firmware.  To support that, you need to re-enable VGA arbitration and all the headaches that come with it and you've lost the benefit that OVMF was meant to achieve for this use case.  If you don't have a legacy-free guest OS, don't go down this path.  Simple as that...

EDIT: imposing legacy attributes, not legacy-free attributes

Last edited by aw (2015-03-10 21:23:06)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4457 2015-03-10 21:23:32

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Try my theory - fetching the drivers for GPU before the install. Some OEMs do that, so there's definitely a way to do this. But it seems like windowns 7 can not into UEFI at all.

Regarding your edit - CSM includes VGA. VGA is bad. CSM will boot win7, but you won't have the advantages of UEFI.

Thinking about that, the first approach I would try is figuring out if a driver is making that call during the install step, then replacing that driver with something that didn't make that call. On one of my notes I remember CLASSPNP.SYS causing lots of hangs when trying safe modes and such... I'll dig in a bit more.

aw wrote:

The benefit of OVMF for GPU assignment is when you can use it to avoid the need for VGA access to the device.  That requires a legacy-free ROM on the device, a legacy-free firmware for the VM and a legacy-free guest operating system.  Here you have a non-legacy-free operating system imposing legacy free attributes on the VM firmware.  To support that, you need to re-enable VGA arbitration and all the headaches that come with it and you've lost the benefit that OVMF was meant to achieve for this use case.  If you don't have a legacy-free guest OS, don't go down this path.  Simple as that...

I had completely forgotten about arbitration and that OVMF removes VGA and all it's junk. It just wasnt clicking in my head that needing to do something about the 10h call is going to bring back VGA. Thanks.

Offline

#4458 2015-03-11 07:08:16

efeu
Member
Registered: 2015-03-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#!/bin/bash
vfio-bind 0000:01:00.0 0000:01:00.1
qemu-system-x86_64 -enable-kvm -m 13312 -device virtio-balloon -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 \
-pflash /some/directory/OVMF.fd \
-rtc base=localtime,clock=host \
-vga none \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/some/directory/win81.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \
-drive file=/some/directory/windows81.raw,id=disk,format=raw,if=none,discard=on -device scsi-hd,drive=disk \
-cdrom /some/directory/virtio-win-0.1-100.iso \
-usb -usbdevice host:046d:c52b \
-usb -usbdevice host:413c:2005 \
-usb -usbdevice host:046d:c051 \
-soundhw hda \
-net tap -net nic,model=virtio,macaddr=52:54:00:12:34:57 \
-boot d

exit 0

I am using the above script to start my VM. I'd like to use virt-manager so can anyone explain to me how i come from the commands to the xml config file?

Last edited by efeu (2015-03-11 07:24:50)

Offline

#4459 2015-03-11 08:54:51

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://libvirt.org/drvqemu.html#xmlimport

Try this, or check my posts for XML and change accordingly, but i use qemu-args that you should not do.

Offline

#4460 2015-03-11 10:49:01

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Will these provide a full "pre-POST" ROM file ?

# cd /sys/bus/pci/devices/0000:01:00.0/ # echo 1 > rom # cat rom > /tmp/image.rom # echo 0 > rom

I found them of AW's blog , yet they result in a 0 byte file . dmesg afterwards :

[Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:36 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents [Wed Mar 11 13:46:58 2015] vfio-pci 0000:01:00.0: Invalid ROM contents

Any thoughts ?

Offline

#4461 2015-03-11 15:01:47

efeu
Member
Registered: 2015-03-06
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

With my NVidia 520 and 970 I used GPU-Z to get the romfile.

Offline

#4462 2015-03-11 15:19:17

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

efeu wrote:

With my NVidia 520 and 970 I used GPU-Z to get the romfile.

It would not generate a "pre-POST" ROM as I mentioned earlier .

Offline

#4463 2015-03-11 15:29:59

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
tritron4 wrote:

I wonder if anyone had been using asus A88x-Pro with kvm. I had it setup with xen 4.4.3 and I was able to pass R9 270 and couple of other  ati cards. I was using 1201 bios version. I had flashed latest bios 1701. I can boot windows 8.1 without ati drivers and with vga none my R 270 displays out put but the moment I had installed ati drivers the system only boots to blue screen it get stuck and does not show desktop.  I had to use this options vfio_iommu_type1 disable_hugepages=1. to work around bios bug.
Any tips on geting that motherboard working with kvm ? How I can get uefi emulation working under kvm. So far I had it load uefi but i dont see any options to boot from my hard drive,

https://docs.google.com/spreadsheet/ccc … _web#gid=0
Check there for A88X. I remember someone did this on A88X.

BTW, which CPU do you have? Since the Unified North Bridge is hidden inside the CPU, it doesn't depends much on the motherboard.

My experience is... flawful, if you can say it like that. I have Athlon x4 750k and asus F2A55, and i experience TONS OF GLITCHES, PERFORMANCE HITS AND STUFF, but at least it works.
So far:
1. x-vga is semi-working. You might get it working, but eventually i needed the mlock=on force switch(it's on by default!) and it sometimes crashed resulting in host shutdown.
2. i couldn't get hugepages working using hugetlbfs.
3. some guys report performance speed ups when using npt=0, which is counter-intuitive.
4. i've got myself a fully UEFI-compatible guest with no problems, except for broken GPU rom.

I have AMD A8-6600K APU  I downgraded bios to 1401 and I can install win 8.1
I get tons of  Event logged [IO_PAGE_FAULT device=04:00.0 domain=0x0022
I tried to use ovmf but it just boots to interface and I dont see any way to boot installer

Offline

#4464 2015-03-11 17:27:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:
Duelist wrote:
tritron4 wrote:

I wonder if anyone had been using asus A88x-Pro with kvm. I had it setup with xen 4.4.3 and I was able to pass R9 270 and couple of other  ati cards. I was using 1201 bios version. I had flashed latest bios 1701. I can boot windows 8.1 without ati drivers and with vga none my R 270 displays out put but the moment I had installed ati drivers the system only boots to blue screen it get stuck and does not show desktop.  I had to use this options vfio_iommu_type1 disable_hugepages=1. to work around bios bug.
Any tips on geting that motherboard working with kvm ? How I can get uefi emulation working under kvm. So far I had it load uefi but i dont see any options to boot from my hard drive,

https://docs.google.com/spreadsheet/ccc … _web#gid=0
Check there for A88X. I remember someone did this on A88X.

BTW, which CPU do you have? Since the Unified North Bridge is hidden inside the CPU, it doesn't depends much on the motherboard.

My experience is... flawful, if you can say it like that. I have Athlon x4 750k and asus F2A55, and i experience TONS OF GLITCHES, PERFORMANCE HITS AND STUFF, but at least it works.
So far:
1. x-vga is semi-working. You might get it working, but eventually i needed the mlock=on force switch(it's on by default!) and it sometimes crashed resulting in host shutdown.
2. i couldn't get hugepages working using hugetlbfs.
3. some guys report performance speed ups when using npt=0, which is counter-intuitive.
4. i've got myself a fully UEFI-compatible guest with no problems, except for broken GPU rom.

I have AMD A8-6600K APU  I downgraded bios to 1401 and I can install win 8.1
I get tons of  Event logged [IO_PAGE_FAULT device=04:00.0 domain=0x0022
I tried to use ovmf but it just boots to interface and I dont see any way to boot installer

Precisely. AMD APU(and APU-based) products with Trinity and Richland microarchitecture seem to have broken IOMMU.
It gets IO_PAGE_FAULT everywhere when using x-vga. However, on rare occasions, when the weather on mars is partly cloudy with mild winds, it might work. Just as i stated before.
Check out huge errata list on those CPUs:
http://amd-dev.wpengine.netdna-cdn.com/ … _Guide.pdf

When using OVMF everything is much more stable.
If you can't find your disks inside OVMF, you're just doing something wrong.
Check out how i attached my disks:

-device ahci \ -device virtio-scsi-pci \ -device scsi-cd,drive=install \ -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/mnt/hdd/qemu/windows8.iso,id=install,if=none \ -drive file=/mnt/hdd/qemu/virtio.iso,id=virtiocd,if=none \

Again, i don't remember which scsi controller works, but since it's temporary - just add both. You can pin-point that to one, if you want.
The main disk is connected that way:

-device virtio-blk-pci,drive=disk1 \ -drive file=/mnt/hdd/qemu/win8.img,id=disk1,format=raw,if=none \

Please note that i am using -M pc and also -boot menu=on(though i'm not sure this option is relevant with OVMF, you can just drop it out after the initial setup).

You MUST have UEFI-bootable iso image. Otherwise it won't work, obviously. Use ovmf-pure-efi, not -with-csm one.

Downgrading bios doesn't matter much - i've tried every bios for my motherboard with IOMMU support, observing no difference at all, so, i guess, latest is best.

Last edited by Duelist (2015-03-11 17:30:38)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4465 2015-03-12 00:24:31

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well I got kvm working windows 8.1 options vfio_iommu_type1 disable_hugepages=1 I wonder if there is a way to pass

Offline

#4466 2015-03-12 18:59:59

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://msdn.microsoft.com/en-us/library/hh824938.aspx
Handy link for those who migrate from disk interface to disk interface often.
How-to avoid BSOD 7B after changing VM's configuration.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4467 2015-03-12 19:02:43

mbirkis
Member
Registered: 2013-12-12
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mbirkis wrote:

I have been searching abit in this forum and on aw's vfio blog faq. When going through the talking points you had aw i got the impression that if i switch to using OVMF and UEFI on my windows8.1 guest, the vga arbiter issue with Nvidia proprietary host drivers is a non issue (your faq question 9) and i can install and use the nvidia driver on the host without patching?

Sorry if this question has been answered before i just had a hard time finding the answer that looked obvious from your talking points "VFIO, OVMF, GPU, and You ".

It's my understanding that this is true.  The problem with the nvidia proprietary driver on the host is that it takes the VGA arbiter lock and never releases it.  The expected usage model for the VGA arbiter is that the lock is held only while accesses to the legacy VGA space are being used.  The lock is then released allowing other users of the arbiter a chance to acquire the lock and perform their access.  This is how pseudo-concurrent VGA access works with multiple VGA assignment VMs.  Therefore, so long as our VM is not performing VGA accesses, which can be enforced by disabling CONFIG_VFIO_PCI_VGA (or a vfio-pci module option I intend to add), there is no need to patch the host nvidia driver to prevent VGA locking.

Note however that while OVMF generally implies legacy-free, there is a legacy compatibility module, the CSM, available in some OVMF builds.  The CSM will potentially do legacy VGA accesses and put you back in the position of needing to patch host drivers for VGA arbitration.  You therefore want to make use of the "pure" version of OVMF to avoid this.  On Fedora, the OVMF file to use is /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd (avoid the "with-csm" versions).  Your builds may name them differently.

Just got around to testing this on my setup and so far it works without issues. I am now running standard Arch kernel, libvirt, ovmf etc. With the proprietary Nvidia driver for Host.

Thank you so much for all the awesome work and guideance you have given on this subject.

Offline

#4468 2015-03-14 18:50:23

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A GLORIOUS UPDATE :

I used "nvagetbios" to get a full "pre-POST" ROM dump for BOTH of my NVIDIA cards (GT610 + GTX770) .

I had to make the cards I want to dump the ROMs from secondary in the host's perspective for this to succeed .

nvagetbios -c 1 -s prom > /VMs/GT610.rom AND nvagetbios -c 2 -s prom > /VMs/GTX770.rom

-c 1 & -c 2 are the cards numbers which we want to dump . 0 being the host's primary obviously .

Each card threw a 1MB ROM file , as opposed to 164K files dumped from GPU-Z .

I used these in my VMs , and got these results :

1 - The GT610 no longer have rebooting issues , I can now reboot from within Windows without a problem . (It used to give a black screen) .

2 - I can now passthrough the GTX770 while it is the PRIMARY GPU of the host ! No issues at all (headless server) .

nvagetbios is part of envytools-git package (AUR) .

Hope this helps ! smile

Last edited by Denso (2015-03-14 18:54:24)

Offline

#4469 2015-03-14 20:33:35

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So is there virt manager package that supports ovmf? I am testing my hp server with kvm ovmf and I had installed server 2012 R2 as a test  and it boots and with cirrus it freezes.
What I am doing wrong. Is there a special vga bios for radeon R9 270 for uefi ?

#/bin/sh! /usr/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \ -smp 4,sockets=2,cores=2,threads=1 -vga cirrus -spice port=5910,addr=0.0.0.0,disable-ticketing \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/ovmf_x64.bin -drive if=pflash,format=raw,file=/usr/share/ovmf/ovmf_x64.bin \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device virtio-scsi-pci,id=scsi \ -net nic,model=e1000,macaddr=64:C5:63:4D:A4:44 -net bridge,br=xenbr0 \ -drive file=/data/kvm/win.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/home/kvm/iso/en_windows_server_2012_r2.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/home/kvm/iso/virtio-win-0.1-100.iso,id=isocd2,if=none -device ide-cd,drive=isocd2

Offline

#4470 2015-03-15 09:20:56

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
aw wrote:

Yep, int10h is a gateway to the legacy VGA BIOS and would imply that you have VGA access to the device.  It's effectively taking the legacy part of the CSM that we're trying to avoid and using it anyway.

With all due respect sir, does that mean we will not have the capability (support in tianocore) of a win7 VM using a GPU that has been passed through via OVMF/VFIO?

That's a real shame, I don't have a win8 product key, and I certainly was not going to go buy one. :(

Edit: I also have not found anything relating as to why the CSM is bad, or why the CSM won't boot win7. Would you shed some light on that?

Upstream OVMF co-maintainer here. Finally decided to register an account and chime in. Alex has been holding the front alone for way too long. :) I've been lurking around as r/o, but today I'll comment too.

So, I continue to see a lot of confusion both in the questions here and in the guides in the archlinux wiki (which is regrettable, because the archlinux wiki is usually one of the best sources of user-oriented technical info on the tubes). I'm happy to announce here too that the OVMF whitepaper has been released recently, and it should answer most, if not all, your questions (and probably even questions you have not had thus far) regarding OVMF. You can find the whitepaper (plaintext edition) in the KVM wiki or in my personal webspace.

Specifically, as to the question whether Win7 / Windows 2008 R2 can be used with OVMF (without a CSM): yes, it can; and you should be able to do GPU passthrough as well.

First, just for installing & then booting a Win7 or Windows 2008 R2 guest, please refer to the following sections in the whitepaper:

  • Supported guest operating systems | Notes about Windows Server 2008 R2
  • Compatibility Support Module (CSM)

    -- strictly as background material only, you do not need to do anything with this

  • Select features | Video driver

Windows 2008 R2 is a UEFI system "in general", it's just its stock (default) video driver that has this inconvenient dependency on Int10h. So, OVMF (the "pure" build, no CSM) fakes an Int10h handler for just this OS, which is pretty raw, but enough for the user to install the guest, and then to update / install "real" video drivers that do not have this Int10h dependency (and then other resolutions and S3 etc become available.)

In short,

  1. You should create a virtual machine with two video adapters. The first should be QXL, the second should be your passthru GPU. Use the "pure UEFI" build of OVMF from Gerd's package.

  2. QXL will allow you to install Win7 / Win2008R2 in 1024x768x32.

  3. Update the guest's video driver for the QXL card from the stock windows driver (which OVMF fools, with regard to Int10h, see the material above) to the real QXL XDDM driver. Look at http://www.spice-space.org/download.html , under Guest | Windows binaries.

  4. Install the driver for your secondary (passthru) GPU.

  5. Potentially mark the secondary (passthru) GPU as primary desktop provider.

Admittedly, I did not test the "GPU passthru" part with Windows 2008 R2 (only with Windows Server 2012 R2), but the OS installation I definitely tested, and I don't see any reason why GPU passthrough (and the dual display) should not work.

Finally, an unrelated, but nonetheless important point -- I recommend that everyone use libvirt. It supports OVMF very nicely. Please refer to the following section in the whitepaper:

Installation of OVMF guests with virt-manager and virt-install

.

Offline

#4471 2015-03-15 15:17:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

YAY! Lazlo has finally joined us!

lersek wrote:

Windows 2008 R2 is a UEFI system "in general", it's just its stock (default) video driver that has this inconvenient dependency on Int10h. So, OVMF (the "pure" build, no CSM) fakes an Int10h handler for just this OS, which is pretty raw, but enough for the user to install the guest, and then to update / install "real" video drivers that do not have this Int10h dependency (and then other resolutions and S3 etc become available.)

Damn, i've developed a special feeling for deep problems. My guess is finally proven.

lersek wrote:

In short,

  1. You should create a virtual machine with two video adapters. The first should be QXL, the second should be your passthru GPU. Use the "pure UEFI" build of OVMF from Gerd's package.

  2. QXL will allow you to install Win7 / Win2008R2 in 1024x768x32.

  3. Update the guest's video driver for the QXL card from the stock windows driver (which OVMF fools, with regard to Int10h, see the material above) to the real QXL XDDM driver. Look at http://www.spice-space.org/download.html , under Guest | Windows binaries.

  4. Install the driver for your secondary (passthru) GPU.

  5. Potentially mark the secondary (passthru) GPU as primary desktop provider.

Let me ask an important question:
-device qxl or -device qxl-vga?

Nevermind, this is covered in OVMF whitepaper.

# A Spice QXL GPU is recommended as the primary VGA-compatible display
  # device. It is a full-featured virtual video card, with great operating
  # system driver support. OVMF supports it too.
  OPTS="$OPTS -device qxl-vga"

They are both qxl.
I usually add -device qxl for... input purposes - that way qemu creates an SDL window where it can grab keyboard and mouse. But it didn't help.
Anyway, i'll try doing your way ASAP.

By the way, i guess, we get two screens connected to the machine, one to QXL and one to GPU? That's a little bit inconvenient, i guess it's OK to drop QXL device after installing GPU's drivers.

Last edited by Duelist (2015-03-15 15:27:22)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4472 2015-03-15 15:43:00

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

Upstream OVMF co-maintainer here. Finally decided to register an account and chime in. ...
So, I continue to see a lot of confusion both in the questions here and in the guides in the archlinux wiki (which is regrettable, because the archlinux wiki is usually one of the best sources of user-oriented technical info on the tubes). ...
I recommend that everyone use libvirt. It supports OVMF very nicely. Please refer to the following section in the whitepaper:

Installation of OVMF guests with virt-manager and virt-install

.

Every now and again a first post comes along that blows me away.  It is an honor to have you join us.
Please, don't be bashful about improving the wiki if you would like to do so.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#4473 2015-03-15 16:06:29

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Anyway, i'll try doing your way ASAP.

Right, I've become curious enough to try it myself as well :)

Duelist wrote:

By the way, i guess, we get two screens connected to the machine, one to QXL and one to GPU? That's a little bit inconvenient, i guess it's OK to drop QXL device after installing GPU's drivers.

Yes, two displays. I didn't find it inconvenient with Windows Server 2012 R2. I simply would not open the QXL display with virt-manager (in fact I wouldn't even start virt-manager, just start the guest with "virsh start"). So that would cause me to lose a tiny bit of boot progress info was sent to the "main" (QXL) display, but eg. the windows login screen is already displayed on the passthru GPU (once that GPU is configured as primary desktop). I wanted to keep the QXL display just in case (troubleshooting etc).

But, yes, I guess once the GPU works okay, you might be able to drop QXL completely.

Offline

#4474 2015-03-15 16:16:55

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ewaller wrote:

Please, don't be bashful about improving the wiki if you would like to do so.

Okay, I fixed two mistakes in https://wiki.archlinux.org/index.php/PC … h_via_OVMF

Offline

#4475 2015-03-15 19:32:25

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
  1. You should create a virtual machine with two video adapters. The first should be QXL, the second should be your passthru GPU. Use the "pure UEFI" build of OVMF from Gerd's package.

  2. QXL will allow you to install Win7 / Win2008R2 in 1024x768x32.

  3. Update the guest's video driver for the QXL card from the stock windows driver (which OVMF fools, with regard to Int10h, see the material above) to the real QXL XDDM driver. Look at http://www.spice-space.org/download.html , under Guest | Windows binaries.

  4. Install the driver for your secondary (passthru) GPU.

  5. Potentially mark the secondary (passthru) GPU as primary desktop provider.

Admittedly, I did not test the "GPU passthru" part with Windows 2008 R2 (only with Windows Server 2012 R2), but the OS installation I definitely tested, and I don't see any reason why GPU passthrough (and the dual display) should not work.

I tested it now, with a brand new Windows 2008 R2 guest. It works. (Checked it with Unigine Valley Benchmark.)

There's one small hiccup in step 5: as soon as the pass-through GPU's driver was installed and the guest was rebooted, the QXL display stopped working (exclamation mark in device manager, code 43). However, at that point the passthrough display worked fine, so I didn't bother trying to figure out why I could not have *dual* display. I was satisfied with the one passthrough display. IOW, step 5 became mandatory / automatic, effectively.

Offline

#4476 2015-03-15 19:48:27

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have windows 8.1 working under kvm using bios. I had passed R9 270 and it is working fine. I was able to pass my video card using ovfm and I can install win 8.1 bu it freezes on first boot. Does anyone know any work around ?
I have ati 6540 that i am trying to pass to second guest and the sound portion is not in the same group as video. I have compiled custom kernel with acs overwrite. How I can force my audio part to be in same group with video?
I am geting this /sys/bus/pci/devices/000:02:00.1/vendor: No such file or device for my hdmi portion of video card both are in group 15 I am using script and servie to bind devices
Rebidding solved the issue I wonder why it does not work at boot.

I can use ovmf with default chipset no q35

Last edited by tritron4 (2015-03-15 21:22:10)

Offline

#4477 2015-03-15 21:43:13

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

There's one small hiccup in step 5: as soon as the pass-through GPU's driver was installed and the guest was rebooted, the QXL display stopped working (exclamation mark in device manager, code 43). However, at that point the passthrough display worked fine, so I didn't bother trying to figure out why I could not have *dual* display. I was satisfied with the one passthrough display. IOW, step 5 became mandatory / automatic, effectively.

I guess some conflict arises. Like, interrupts or memory ranges. You know better. As i said, we can ditch that temporary stuff later.

I guess, if QXL now(btw since which version or date?) works with int 10h, we don't even need to install drivers for it, just do the setup and install GPU's drivers.
We need to install QXL's drivers only to get rid of that int 10h wizardry. That's the right way, but we don't always need it.

Is there any way to make QEMU "emulate" or make work int 10h for the physical GPU temporarily? Something hints me that won't work for some reason..
Like, we have "-win2k-hack     use it when installing Windows 2000 to avoid a disk full bug", maybe there's possibility to make a similar switch -srv2k8uefi-hack or something?

P.S.
Hail something, as it always hints me stuff.

Last edited by Duelist (2015-03-15 21:54:20)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4478 2015-03-15 21:52:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I have windows 8.1 working under kvm using bios. I had passed R9 270 and it is working fine. I was able to pass my video card using ovfm and I can install win 8.1 bu it freezes on first boot. Does anyone know any work around ?
I have ati 6540 that i am trying to pass to second guest and the sound portion is not in the same group as video. I have compiled custom kernel with acs overwrite. How I can force my audio part to be in same group with video?
I am geting this /sys/bus/pci/devices/000:02:00.1/vendor: No such file or device for my hdmi portion of video card both are in group 15 I am using script and servie to bind devices
Rebidding solved the issue I wonder why it does not work at boot.

I can use ovmf with default chipset no q35

Errr... So your windows8.1 VM with R9 270 works or freezes at the first boot?

Maybe you have a broken GPU ROM(vbios). Try adding

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

into your qemu startup script and show us the end of debug.log

Regarding your HD6450 card... I think if there's no other devices in the same group as it's HDMI audio device - it will work fine.
ACS override patch will 100% work without errors only on some selected motherboards, where there is granted that there is device isolation. Provide us motherboard make/model.

Last edited by Duelist (2015-03-15 22:05:36)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4479 2015-03-16 01:07:26

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ovmf works fine without q35 chip set. I wonder if I can use same ovmf files for second guest config or do I need setup custom location ?

Offline

#4480 2015-03-16 02:23:11

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
tritron4 wrote:

I have windows 8.1 working under kvm using bios. I had passed R9 270 and it is working fine. I was able to pass my video card using ovfm and I can install win 8.1 bu it freezes on first boot. Does anyone know any work around ?
I have ati 6540 that i am trying to pass to second guest and the sound portion is not in the same group as video. I have compiled custom kernel with acs overwrite. How I can force my audio part to be in same group with video?
I am geting this /sys/bus/pci/devices/000:02:00.1/vendor: No such file or device for my hdmi portion of video card both are in group 15 I am using script and servie to bind devices
Rebidding solved the issue I wonder why it does not work at boot.

I can use ovmf with default chipset no q35

Errr... So your windows8.1 VM with R9 270 works or freezes at the first boot?

Maybe you have a broken GPU ROM(vbios). Try adding

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

into your qemu startup script and show us the end of debug.log

Regarding your HD6450 card... I think if there's no other devices in the same group as it's HDMI audio device - it will work fine.
ACS override patch will 100% work without errors only on some selected motherboards, where there is granted that there is device isolation. Provide us motherboard make/model.

I have to rebind both pci addresses for it to work. I have assus A88X pro I am using  scripts from first page

Offline

#4481 2015-03-16 09:23:40

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I guess, if QXL now(btw since which version or date?)

https://github.com/tianocore/edk2/commit/90803342

Duelist wrote:

works with int 10h, we don't even need to install drivers for it, just do the setup and install GPU's drivers.

Yes, that seems to make sense.

Duelist wrote:

We need to install QXL's drivers only to get rid of that int 10h wizardry. That's the right way, but we don't always need it.

Well, the genuine QXL drivers (ie. those that are independent of Int10h) are necessary because they allow the user to take advantage of QXL's actual capabilities. The Int10h shim that OVMF installs is just a minimal service that isn't supposed to do anything beyond getting the user through the guest installation. After that you'll probably want something more than just 1 resolution, 1024x768x32. Normally that "more" means the QXL XDDM driver, but in this case, the passthru GPU and its drivers are just as fine, for "more".

IOW, the Int10h shim is not "harmful" in any way, for which reason you'd want to get rid of it; it simply has minimal functionality (== fill the gap for as long as the Win2k8r2 guest is being installed, and first booted).

Duelist wrote:

Is there any way to make QEMU "emulate" or make work int 10h for the physical GPU temporarily?

No. First, the Int10h shim is not provided by QEMU, it is provided by OVMF. Second, as you can read in the whitepaper and in the commit message I linked above, it is a valid implementation of a very small set of VBE services, as far as it comes to the underlying video card (ie. QXL). See the subroutine-like labels under the "Handler:" label in the source. Importantly, the code under "SetMode:" and "KnownMode2:" actually implement a graphics mode switch on the QXL card.

(The reason for providing the Int10h shim only on stdvga and QXL is exactly because they are simple (and identical, on this level) to program. And of those two -- I'll digress a bit --, we recommend QXL over stdvga only because QXL has full-featured guest drivers. But, if you're going to install the passthru GPU driver anyway, and not install the QXL guest driver, then you can just as well use

-device VGA

(ie. stdvga) with the Int10h shim.)

Back to your question: in order to provide an Int10h shim for any other card than stdvga & QXL, that shim whould have to program that card for real.

So, just to reiterate: the fake thing about the Int10h shim is not the video card programming; that's real. The fake thing is that

  • it provides just a minimal set of services, just enough for the win7 family installation to succeed, and

  • said VBE services and related descriptor structures are not placed in the C segment, and similarly the Int10h vector is not pointed to the handler in the C segment, by way of posting an actual legacy PCI oprom (which the CSM would do). Instead, they are put into place by OVMF just because we figured out that the legacy vga driver of win2k8r2 will look for them there as data, and it will emulate the VBE code (ie. not directly execute it on the CPU) in an internal real-mode (16-bit) emulator.

So, the "external interface" of the shim is fake (because it doesn't have to work in any other environment than with win2k8r2's real-mode emulator), but its internal (hardware facing) code is valid (because hardware accesses like ioport writes encountered in the internal emulator are replayed for real), and those would have to be implemented for any other card.

Offline

#4482 2015-03-16 09:35:26

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I wonder if I can use same ovmf files for second guest config or do I need setup custom location ?

The OVMF_CODE.fd file is mapped read-only and it is supposed to be shared by all guests. This allows central (host-wide) firmware binary upgrades that affect all guests at once, similarly to SeaBIOS upgrades.

The OVMF_VARS.fd file is not mapped by any guest; it is a varstore template. Whenever you create a new guest, it must start out with a private copy of this pristine (empty) varstore template. Then the guest will keep its non-volatile UEFI variables in this private variable store. That's why varstore files are VM-private.

When you use libvirt, libvirtd takes care of instantiating a private varstore from OVMF_VARS.fd for each new VM you create. (Look under /var/lib/libvirt/qemu/nvram.) In fact libvirt can accommodate several use cases (the domain XML is really flexible), but this is the main use case that virt-manager and virt-install generate a domain XML for.

However, if you use the qemu command line manually, you must copy OVMF_VARS.fd first, and pass the copy to the VM as the second pflash chip. See here (I fixed the wiki yesterday):

https://wiki.archlinux.org/index.php/PC … _OVMF#Test

Offline

#4483 2015-03-16 10:43:32

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So , GT610 reboot issue remains . It happens after hours though , meaning if I rebooted the VM when the host is freshly booted it reboots fine . Checking dmesg , the only thing that happens after hours of host being up are these type of messages :

perf samples too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 50100

Using :

echo 0 > /proc/sys/kernel/perf_cpu_time_max_percent

as soon as the host boots up disables that mechanism , and the GT610's VM is now rebooting fine after 10 hours of operation .

I don't know why they are related though .

Edit : Correcting the time periods . lol

Last edited by Denso (2015-03-16 11:10:44)

Offline

#4484 2015-03-16 14:08:12

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
tritron4 wrote:

I wonder if I can use same ovmf files for second guest config or do I need setup custom location ?

The OVMF_CODE.fd file is mapped read-only and it is supposed to be shared by all guests. This allows central (host-wide) firmware binary upgrades that affect all guests at once, similarly to SeaBIOS upgrades.

The OVMF_VARS.fd file is not mapped by any guest; it is a varstore template. Whenever you create a new guest, it must start out with a private copy of this pristine (empty) varstore template. Then the guest will keep its non-volatile UEFI variables in this private variable store. That's why varstore files are VM-private.

When you use libvirt, libvirtd takes care of instantiating a private varstore from OVMF_VARS.fd for each new VM you create. (Look under /var/lib/libvirt/qemu/nvram.) In fact libvirt can accommodate several use cases (the domain XML is really flexible), but this is the main use case that virt-manager and virt-install generate a domain XML for.

However, if you use the qemu command line manually, you must copy OVMF_VARS.fd first, and pass the copy to the VM as the second pflash chip. See here (I fixed the wiki yesterday):

https://wiki.archlinux.org/index.php/PC … _OVMF#Test

I created new folders for OVMF_VARS.fd. I wonder if there is a version of virt manager that will allow me to turnoff vga ?

Offline

#4485 2015-03-16 15:19:24

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I wonder if there is a version of virt manager that will allow me to turnoff vga ?

virt-manager does not provide access to low-level knobs, on purpose. (Unless of course I'm misunderstanding you. If you simply mean "removing the VGA device from the VM config", that's certainly doable, but I don't think you would ask about that, because it's obvious.) So virt-install / virt-manager should be used to install an OVMF-based guest the easy way (covering even the PCI and USB device assignments), then you can shut down the guest, and use "virsh edit" to customize the finer bits for GPU passthrough (vcpu pinning, cpuid massaging etc).

Offline

#4486 2015-03-16 16:19:07

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I had compiled virt-manger git aur and installed it but it cant find uefi ? Does anyone know the location and name of files virt-manager is looking.
Well I edited /etc/libvirt/qemu.conf end enabled nvram

Last edited by tritron4 (2015-03-16 16:29:11)

Offline

#4487 2015-03-16 16:33:37

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I had compiled virt-manger git aur and installed it but it cant find uefi ? Does anyone know the location and name of files virt-manager is looking.
Well I edited /etc/libvirt/qemu.conf end enabled nvram

Assuming you are using recent enough libvirtd and virt-manager / virt-install, see "Installation of OVMF guests with virt-manager and virt-install" in http://people.redhat.com/~lersek/ovmf-w … 770f8c.txt , especially point (5) there.

Offline

#4488 2015-03-16 16:52:51

VOT Productions
Member
Registered: 2011-10-22
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I am confused, and I'm not sure where I went wrong.

My system setup:
GPU #1: AMD Radeon HD 7750 (VM)
GPU #2: Terrible Intel Graphics (Host)
CPU: Intel i5 4460
Motherboard: Asus H81M-C

As you can see, my setup isn't the most complicated in the world, yet it confused me much frustation. I managed to get pci-stub and vfio-bind working, however I am having massive issues with running QEMU.

My first question is, should I be running Intel as bootup graphics in the BIOS? Or, should it be AMD that boots first?
Secondly, on both occasions in QEMU my GPU fan spins up, but this yields two different results. When running with AMD as boot-up, I have a lot of issues, such as not being to access X without nomodeset and doing all kinds of hacks to get in (no acceleration either), and while it does display an image with QEMU, it is artifacted and crashes when I mount an ISO or access any of the TTYs. When I use Intel as boot-up, I can actually get 3D acceleration, but QEMU shows nothing.

Thirdly, does this display in a small window, or does it show on a different monitor? On AMD, it seemed to have taken over the screen, however I've seen screenshots showing that it should be windowed.

I am very confused but I've almost got it hopefully working big_smile

Offline

#4489 2015-03-16 17:02:30

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I had compiled virt-manger git aur and installed it but it cant find uefi ? Does anyone know the location and name of files virt-manager is looking.
Well I edited /etc/libvirt/qemu.conf end enabled nvram

get ovmf-svn from aur

Offline

#4490 2015-03-16 17:04:42

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you are assigning ati video card I would boot intel first there is option in bios to select build in video card as first. This works for me you can setup vga to cirrus to get output in spice

#/bin/sh! QEMU_AUDIO_DRV=alsa \ /usr/bin/qemu-system-x86_64 -enable-kvm -m 8196 -cpu host \ -smp 4,sockets=2,cores=2,threads=1 -vga none -spice port=5910,addr=0.0.0.0,disable-ticketing \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE-pure-efi.fd -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/OVMF_VARS-pure-efi.fd \ -device virtio-scsi-pci,id=scsi \ -net nic,model=e1000,macaddr=64:C5:63:4D:A4:44 -net bridge,br=xenbr0 \ -usb -device usb-host,hostbus=9,hostaddr=2 \ -soundhw hda \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -device vfio-pci,host=04:00.0 \ -drive file=/data/kvm/win.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/home/kvm/iso/en_windows_8_1.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/home/kvm/iso/virtio-win-0.1-100.iso,id=isocd2,if=none -device ide-cd,drive=isocd2

Offline

#4491 2015-03-16 17:04:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VOT Productions wrote:

Hello, I am confused, and I'm not sure where I went wrong.

My system setup:
GPU #1: AMD Radeon HD 7750 (VM)
GPU #2: Terrible Intel Graphics (Host)
CPU: Intel i5 4460
Motherboard: Asus H81M-C

As you can see, my setup isn't the most complicated in the world, yet it confused me much frustation. I managed to get pci-stub and vfio-bind working, however I am having massive issues with running QEMU.

My first question is, should I be running Intel as bootup graphics in the BIOS? Or, should it be AMD that boots first?
Secondly, on both occasions in QEMU my GPU fan spins up, but this yields two different results. When running with AMD as boot-up, I have a lot of issues, such as not being to access X without nomodeset and doing all kinds of hacks to get in (no acceleration either), and while it does display an image with QEMU, it is artifacted and crashes when I mount an ISO or access any of the TTYs. When I use Intel as boot-up, I can actually get 3D acceleration, but QEMU shows nothing.

Thirdly, does this display in a small window, or does it show on a different monitor? On AMD, it seemed to have taken over the screen, however I've seen screenshots showing that it should be windowed.

I am very confused but I've almost got it hopefully working big_smile

Re-read the first post carefully

Offline

#4492 2015-03-16 17:17:44

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am running into issue where efi-e1000.rom is not found. Ho to tell libvirt proper location ? I have the files in /usr/share/qemu

Offline

#4493 2015-03-16 17:29:40

VOT Productions
Member
Registered: 2011-10-22
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm a knob, I forgot to enable the i915 patch. Also, is there a way of not losing DRI and making it run? (using i440fx instead of OVMF as apparently that doesn't work with non-UEFI graphic cards)

Offline

#4494 2015-03-16 17:39:49

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VOT Productions wrote:

I'm a knob, I forgot to enable the i915 patch. Also, is there a way of not losing DRI and making it run? (using i440fx instead of OVMF as apparently that doesn't work with non-UEFI graphic cards)

Yes, using OVMF, there's also a post by duelist here on how create an uefi rom for you amd card.

Offline

#4495 2015-03-16 19:30:23

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

After that you'll probably want something more than just 1 resolution, 1024x768x32. Normally that "more" means the QXL XDDM driver, but in this case, the passthru GPU and its drivers are just as fine, for "more".

HA! I've tried setting up windows 8 on a hardware UEFI platform, and i burst out laughing when..

1.It boots into 1024x768x32.
2.I try to download drivers(or, at least, a browser), starting Internet exploder..
3.And it says "Screen resolution is too small, internet will not be exploded for today"
4.No drivers - no resolution change.
5.No resolution change - no drivers.

So you've got to have some CD with drivers when you try installing windows 8 or some other UEFI-compatible windowns. Or you can't download drivers.


BTW, i've got my second GPU back, but i have my NTFS partition with a hybrid ROM in it trashed by windows 8 VM(apparently, you have to do shutdown -t 0 -h everytime you want to save your filesystem from trashing).


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4496 2015-03-16 19:32:15

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have setup windows 8.1 and windows 10 using ovmf libvirt. How I can use virsh to remove emulated vga cards ? How I can get my sound to work with alsa and my windows guest. Both windows guest have functioning emulated sound cards and I can hear sound in Linux.
Is there a way to hotswap vga cards between guests ?

Offline

#4497 2015-03-16 19:40:49

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

Is there a way to hotswap vga cards between guests ?

There is kind of way.
Basically, all you need to do is to eject the card from windows. Then it becomes bound to vfio-pci and not used by anyone. And you can attach it to some other VM via various methods, including starting a new VM for example.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4498 2015-03-17 18:24:36

VOT Productions
Member
Registered: 2011-10-22
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you so much for your help guys big_smile

I have hopefully the final two questions:
1. Windows hangs on "Loading Windows", any ideas why that might be caused?
2. Is there a way of making it on the Intel screen (hopefully windowed), or/and sharing the mouse/keyboard? I don't really want to plug in two keyboards and mice to operate 2 OS big_smile

Thanks again,
VOT.

Offline

#4499 2015-03-17 18:33:37

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can assign usb receiver for mouse/keyboard to guest or you can assign whole usb controller. I assign whole usb controller to windows 8 guest, with scripts you can return usb controller back to linux. If you want to use guest in windows why not to use spice client ? You can assign usb device to it and you can share mouse keyboard from client.
Did anyone install osx on amd cpu under ovmf kv ?

Last edited by tritron4 (2015-03-17 18:39:26)

Offline

#4500 2015-03-17 18:40:53

tholin
Member
Registered: 2015-03-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been struggling with VGA passthrough for a few days and finally got something that works but unfortunately not in an optimal way. My plan was to use a radeon juniper card in guest and a radeon cedar card on host. The problem is that the hardware manufacturer for my cpu decided in their wisdom to not support ACS on the pcie root port.
Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01]
Both graphic card therefore show up in the same iommu group. What options do I have for working around that problem?

Option 1: Use the ACS override patch and hope for the best.
Not a good idea in my opinion. I'm pretty sure the root port do per-to-per without any way of disabling that. Using the override patch is asking for problems.

Options 2: Rearrange the card to break apart the iommu group.
Unfortunately all 16X lanes go to the cpu through that root port. The 1X ports go through the chipset but I don't have any 1X graphics card. I could buy one but the selection is VERY limited. The only one I found only supports 2 monitors. I could do surgery on the 1X mobo slot or my 16X cedar graphics card to force it in but that not something I want to do.
http://i.imgur.com/NxzaK6L.png block diagram of my mobo. (Let's see if I'm allowed to post a link in my first post to this forum)

Option 3: Buy a new mobo
please no...

Option 4: Use the intel IGP for host graphics and non-legacy guest boot for working around the problems in IGP
My juniper card does not support EFI boot. I've seen some people do Macgyver style mods involving copy paste in hexeditors with other card's vbios. That is something I would like to avoid if possible.

Option 5: Use the intel IGP with the vga arbiter patch
That is what I'm using right now for testing but it's not a longterm solution. It's too inconvenient to apply a patch at every kernel update.

Strangely DRI is not disabled for me even with the vga arbiter patch. I assume I screwed up somewhere but I don't know where. The old patch submitted to the LKML doesn't apply on new kernels anymore. There is another patch named i915_317.patch that do apply to 3.19.1 with some fuss. I boot with the i915.enable_hd_vgaarb=1 argument and verify that argument is on with dmesg but I still get DRI.

# glxinfo |grep render direct rendering: Yes ... OpenGL renderer string: Mesa DRI Intel(R) Haswell Desktop # dmesg |grep vgaarb [ +0.000000] Command line: BOOT_IMAGE=/vmlinuz intel_iommu=on memtest=1 i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68ba,1002:aa58,1002:68f9,1002:aa68 [ +0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz intel_iommu=on memtest=1 i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68ba,1002:aa58,1002:68f9,1002:aa68 [ +0.000131] vgaarb: setting as boot device: PCI:0000:00:02.0 [ +0.000045] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ +0.000051] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=none,locks=none [ +0.000050] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none [ +0.000048] vgaarb: loaded [ +0.000040] vgaarb: bridge control possible 0000:04:00.0 [ +0.000044] vgaarb: bridge control possible 0000:03:00.0 [ +0.000043] vgaarb: no bridge control possible 0000:00:02.0 [ +0.073126] [drm:i915_disable_vga_mem] *ERROR* enable_hd_vgaarb is on at i915_disable_vga_mem [ +0.000002] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

The error is a DRM_ERROR() call I added in i915_disable_vga_mem() just to make sure the patch loaded.

The forwarding appears to work. I can run the unigine benchmarks with good speed and I don't notice any other problems on the host.

Offline

#4501 2015-03-17 18:48:59

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think you have few options. You could use intel gpu for host and bind both cards and use them with guest. You can also use xen i guess xen 4.5 package will be out in aur soon. xen does not care about acs as much as kvm does.

Offline

#4502 2015-03-17 18:53:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tholin wrote:

I've been struggling with VGA passthrough for a few days and finally got something that works but unfortunately not in an optimal way. My plan was to use a radeon juniper card in guest and a radeon cedar card on host. The problem is that the hardware manufacturer for my cpu decided in their wisdom to not support ACS on the pcie root port.
Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01]
Both graphic card therefore show up in the same iommu group. What options do I have for working around that problem?

Option 1: Use the ACS override patch and hope for the best.
Not a good idea in my opinion. I'm pretty sure the root port do per-to-per without any way of disabling that. Using the override patch is asking for problems.

Options 2: Rearrange the card to break apart the iommu group.
Unfortunately all 16X lanes go to the cpu through that root port. The 1X ports go through the chipset but I don't have any 1X graphics card. I could buy one but the selection is VERY limited. The only one I found only supports 2 monitors. I could do surgery on the 1X mobo slot or my 16X cedar graphics card to force it in but that not something I want to do.
http://i.imgur.com/NxzaK6L.png block diagram of my mobo. (Let's see if I'm allowed to post a link in my first post to this forum)

Option 3: Buy a new mobo
please no...

Option 4: Use the intel IGP for host graphics and non-legacy guest boot for working around the problems in IGP
My juniper card does not support EFI boot. I've seen some people do Macgyver style mods involving copy paste in hexeditors with other card's vbios. That is something I would like to avoid if possible.

Option 5: Use the intel IGP with the vga arbiter patch
That is what I'm using right now for testing but it's not a longterm solution. It's too inconvenient to apply a patch at every kernel update.

Strangely DRI is not disabled for me even with the vga arbiter patch. I assume I screwed up somewhere but I don't know where. The old patch submitted to the LKML doesn't apply on new kernels anymore. There is another patch named i915_317.patch that do apply to 3.19.1 with some fuss. I boot with the i915.enable_hd_vgaarb=1 argument and verify that argument is on with dmesg but I still get DRI.

You seem to have a good grasp of the options.  Personally I'd opt for using IGD for the host and replacing your guest card with one that supports UEFI.  You seem to be averse to the VGA BIOS hacking route, but that's actually one that's likely to continue working once you figure it out, with no future i915 kernel patching.  Remember you don't need to flash the card with the new ROM, you can use an external file as the ROM for the VM.  If you're willing to settle for a x1 graphics card, you can find x1-to-x16 adapters that will allow you to plug any low-profile x16 card into the x1 slot.  Double check your Xorg.0.log for DRI, DRI3 may still be enabled, but DRI2 has arbiter issues... don't ask me what the difference is, I don't know.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4503 2015-03-17 20:33:26

VOT Productions
Member
Registered: 2011-10-22
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, I'm still stuck with the Starting Windows screen when loading the installer, is my QEMU command right? (the paths are correct)

env QEMU_PA_SAMPLES=128 env QEMU_AUDIO_DRV=pa sudo qemu-system-x86_64 \ -enable-kvm -m 4096 -cpu host,kvm=off -smp cores=4,threads=1 \ -pflash /usr/share/ovmf/x64/ovmf_x64.bin \ -vga none \ -soundhw hda \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/1TB_LinuxMedia/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/mnt/1TB_LinuxMedia/windows.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/mnt/1TB_LinuxMedia/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -usb -usbdevice host:04d9:a055 -usbdevice host:04d9:a04a \ -boot menu=on

Last edited by VOT Productions (2015-03-17 20:49:47)

Offline

#4504 2015-03-17 21:40:26

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I found out that my quests run better under libvirt. I had my system freeze when left unattended. No I can leave it unattended  for days without single lockup.
I think options help 
<hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G5</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>

Offline

#4505 2015-03-17 21:41:09

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How do I ssetup secure boot for windows 8.1 ?

Offline

#4506 2015-03-17 23:30:42

Draizer
Member
Registered: 2015-03-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone.

I followed the first part of this (excellent) guide thoroughly up to the part where I have to test output in bios, unfortunately it yielded no results. Repo QEMU shows a console, qemu-git only shows a menubar and nothing else. The vga arbiter is activated (before I activated it, I had a blank QEMU and glitchy graphics on the host once I started QEMU, after I activated it ... QEMU was still blank but at least my host didn't have glitches anymore), as is ACS override (tried both with and without). I tried rebuilding the kernel multiple times, with different Linux versions (3.9, 3.7, neither worked, I'm currently back to the kernel made by the OP). Triple checked if VT-d was activated on the motherboard, it is. Bindings seems correct as well. I uninstalled the radeon drivers, but still kept them blacklisted and bound the devices to stub just to be sure. I tried both with the i440fx and Q35 chipset. I tried to use the manipulation suggested if nothing happened. I browsed this thread and others on the web through and through, and the only mentions of troubles that remotely looked like mine were because the vga arbiter wasn't activated, which isn't my case. I'm basically at my wits ends, which is why I'm finally posting here.

My hardware :
-Intel Core i7 3770
-ASrock Z77 Extreme6
-Host GPU : Intel IGD
-Guest GPU : ATI Radeon 7970


I'm not sure if it is related, but dmesg brings the following error messages:

[drm:ivybridge_set_fifo_underrun_reporting] *ERROR* uncleared fifo underrun on pipe B [drm:ivb_err_int_handler] *ERROR* Pipe B FIFO underrun [drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder A [drm:cpt_serr_int_handler] *ERROR* PCH transcoder A FIFO underrun [drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder B [drm:cpt_serr_int_handler] *ERROR* PCH transcoder B FIFO underrun

Didn't find anything helpful on this, either.

Offline

#4507 2015-03-18 04:44:44

Evonat
Member
Registered: 2013-07-24
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One year and a half after I abandoned my attempt I'm back at it again, and of course it still doesn't fully work!
Or actually, everything seems to work fine once, then if the VM reboots/closes and opens again, the screen stays black.
In other words it looks like that old VFIO VGA reset problem for which there used to be some patches, now gone because it was supposed to be solved(?).

I'm using a linux-3.18 kernel patched with override_for_missing_acs_capabilities.patch (pci-stub.ids=1002:94b3,1002:aa38 pcie_acs_override=downstream) and QEMU 2.2.1.
I get no errors with QEMU or dmesg. The thing simply works the first time and the first time only, then I have to restart the host to get it to work again.

Offline

#4508 2015-03-18 07:22:38

ceri
Member
Registered: 2013-10-12
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just found out there's now a wiki page for this, however it's incomplete and requires more contributors (if you have time):

https://wiki.archlinux.org/index.php/PC … h_via_OVMF

Offline

#4509 2015-03-18 10:08:01

kainet
Member
Registered: 2014-12-22
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Again experiencing sound stuttering when gpu is pretty loaded sad MSI GTX 760


Some times waiting for some times helps to get back normal sound but i usually disable and enable audio device in Windows guest. Anybody knows which part of software is responsible for that ?

Offline

#4510 2015-03-18 17:43:42

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How I can get sound using libvirt. I can hear sound if I use spice client but there is no sound from guest to alsa.

I see option like this:
The VM definition xml contains the following
<qemu:commandline>
<qemu:env name='QEMU_ALSA_DAC_BUFFER_SIZE' value='512'/>
<qemu:env name='QEMU_ALSA_DAC_PERIOD_SIZE' value='170'/>
<qemu:env name='QEMU_AUDIO_DRV' value='alsa'/>
</qemu:commandline>

Last edited by tritron4 (2015-03-18 18:23:57)

Offline

#4511 2015-03-18 19:04:37

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
Duelist wrote:

Try my theory - fetching the drivers for GPU before the install. Some OEMs do that, so there's definitely a way to do this. But it seems like windowns 7 can not into UEFI at all.

Thinking about that, the first approach I would try is figuring out if a driver is making that call during the install step, then replacing that driver with something that didn't make that call. On one of my notes I remember CLASSPNP.SYS causing lots of hangs when trying safe modes and such... I'll dig in a bit more.

Nothing came up as far as patching win7's graphics system, maybe next I could try ripping win8's graphics system and force it into win7?

On a side note, I have a working win8 now. It hangs at shutdown, I have to issue 'quit' in the qemu terminal, but windows doesn't complain about getting a bad shutdown.  I've tried disabling the power service in windows but that didn't do anything. Anyone else have experience with that?

Also, it was recommended I post this here: https://bbs.archlinux.org/viewtopic.php?id=194060

Offline

#4512 2015-03-18 19:31:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

How do I ssetup secure boot for windows 8.1 ?

http://www.linux-kvm.org/downloads/lers … 770f8c.txt

lersek wrote:

(6) Additionally, the "edk2.git-ovmf-x64" package seeks to simplify the
    enablement of Secure Boot in a virtual machine (strictly for development
    and testing purposes)

    - Boot the virtual machine off the CD-ROM image called
      "/usr/share/edk2.git/ovmf-x64/UefiShell.iso"; before or after installing
      the main guest operating system.

    - When the UEFI shell appears, issue the following commands:

      EnrollDefaultKeys.efi
      reset -s

Last edited by Duelist (2015-03-18 19:31:57)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4513 2015-03-18 19:42:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
Naruni wrote:
Duelist wrote:

Try my theory - fetching the drivers for GPU before the install. Some OEMs do that, so there's definitely a way to do this. But it seems like windowns 7 can not into UEFI at all.

Thinking about that, the first approach I would try is figuring out if a driver is making that call during the install step, then replacing that driver with something that didn't make that call. On one of my notes I remember CLASSPNP.SYS causing lots of hangs when trying safe modes and such... I'll dig in a bit more.

Nothing came up as far as patching win7's graphics system, maybe next I could try ripping win8's graphics system and force it into win7?

Also, it was recommended I post this here: https://bbs.archlinux.org/viewtopic.php?id=194060

Seems like you've missed some recent news about windows7:
https://bbs.archlinux.org/viewtopic.php … 0#p1511290
lersek has joined us and said that there's a FIX for win7 VM.
There's a whitepaper published here:
http://www.linux-kvm.org/downloads/lers … 770f8c.txt

And he also edited the wiki page of yours.

BTW: originally i've meant some way of tearing and ripping of win8's video drivers and shove them up win7. Yes, that sounds awful. That's the idea.

Last edited by Duelist (2015-03-18 22:39:54)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4514 2015-03-19 05:14:50

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any possible way to "inject" UEFI support inside non-UEFI ROMs ?

I have a GT630 , which has a non-UEFI ROM , yet I found some ROMs from other vendors that have UEFI ROMs (techpowerup) which don't work directly on my GPU .

It is possible to insert the UEFI partition for some Radeons using a hex editor . How about NVIDIA ?

Offline

#4515 2015-03-19 10:00:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

How about NVIDIA ?

Weird. I have gigabyte GT610 which supports UEFI perfectly..
Anyway, try searching for NVidia's GOP technical presentations and etc - i've found one for AMD dating back to 2010, i think, saying "there will be an universal GOP for all cards". That presentation just proved the idea of copying GOP from one's GPU to another.

Also, try writing a support ticket to your hardware's vendor - in my ASUS case it just cleared up some things. But actually, they have an official update tool(which requires windowns installed..).

Also, maybe you just haven't got enough luck, since some nvidia cards may be converted to quadro just by firmware change.

You can also dump all UEFI-compatible ROMs from DB and try ripping uncompressed EFI driver same way we did it for AMD, it may suddenly be universal for all vendors and containing only GOP without any important bits. Supplying it as romfile= option will be a safe way of testing it out. The resulting file from firmware should be .efi image, you glue it with your normal vbios same way as AMD, using EfiRom(a tool from EDK2 btw). I don't recall properly, but i think EfiRom utility could decompress that efi image from ROM..

P.S. it wasn't a hex editor, it's an Elegant Weapon for a More Civilized Age of UEFI.

Last edited by Duelist (2015-03-19 10:02:48)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4516 2015-03-19 11:58:08

tholin
Member
Registered: 2015-03-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You seem to have a good grasp of the options.  Personally I'd opt for using IGD for the host and replacing your guest card with one that supports UEFI.  You seem to be averse to the VGA BIOS hacking route, but that's actually one that's likely to continue working once you figure it out, with no future i915 kernel patching.  Remember you don't need to flash the card with the new ROM, you can use an external file as the ROM for the VM.  If you're willing to settle for a x1 graphics card, you can find x1-to-x16 adapters that will allow you to plug any low-profile x16 card into the x1 slot.  Double check your Xorg.0.log for DRI, DRI3 may still be enabled, but DRI2 has arbiter issues... don't ask me what the difference is, I don't know.

I've been using the IGD for a few days and realized the drivers are too buggy for daily use. Maybe it's related to the i915 patch but many other people have the same problems. To be fair the open radeon drivers also got a lot of problems but I'm used to them and know how to work around the problems. I ordered that 1X radeon card instead.

It looks like dri2 is enabled. Here are some parts of Xorg.log

[ 188.603] (II) xfree86: Adding drm device (/dev/dri/card0) [ 188.604] (--) PCI:*(0:0:2:0) 8086:0412:1458:d000 rev 6, Mem @ 0xf7400000/4194304, 0xc0000000/268435456, I/O @ 0x0000f000/64 [ 188.604] (--) PCI: (0:3:0:0) 1002:68f9:174b:e153 rev 0, Mem @ 0xe0000000/268435456, 0xf7920000/131072, I/O @ 0x0000a000/256, BIOS @ 0x????????/131072 [ 188.604] (--) PCI: (0:4:0:0) 1002:68ba:1458:220c rev 0, Mem @ 0xd0000000/268435456, 0xf7820000/131072, I/O @ 0x00009000/256, BIOS @ 0x????????/131072 ... [ 188.618] (II) LoadModule: "intel" [ 188.618] (II) Loading /usr/lib64/xorg/modules/drivers/intel_drv.so [ 188.620] (II) Module intel: vendor="X.Org Foundation" [ 188.620] compiled for 1.16.4, module version = 2.99.917 [ 188.620] Module class: X.Org Video Driver [ 188.620] ABI class: X.Org Video Driver, version 18.0 [ 188.620] (II) intel: Driver for Intel(R) Integrated Graphics Chipsets: i810, i810-dc100, i810e, i815, i830M, 845G, 854, 852GM/855GM, 865G, 915G, E7221 (i915), 915GM, 945G, 945GM, 945GME, Pineview GM, Pineview G, 965G, G35, 965Q, 946GZ, 965GM, 965GME/GLE, G33, Q35, Q33, GM45, 4 Series, G45/G43, Q45/Q43, G41, B43 [ 188.620] (II) intel: Driver for Intel(R) HD Graphics: 2000-6000 [ 188.620] (II) intel: Driver for Intel(R) Iris(TM) Graphics: 5100, 6100 [ 188.620] (II) intel: Driver for Intel(R) Iris(TM) Pro Graphics: 5200, 6200, P6300 [ 188.620] (++) using VT number 7 [ 188.626] (II) intel(0): Using Kernel Mode Setting driver: i915, version 1.6.0 20141121 [ 188.627] (--) intel(0): Integrated Graphics Chipset: Intel(R) HD Graphics 4600 [ 188.627] (--) intel(0): CPU: x86-64, sse2, sse3, ssse3, sse4.1, sse4.2, avx, avx2 [ 188.627] (II) intel(0): Creating default Display subsection in Screen section "Default Screen Section" for depth/fbbpp 24/32 [ 188.628] (==) intel(0): Depth 24, (--) framebuffer bpp 32 [ 188.628] (==) intel(0): RGB weight 888 [ 188.628] (==) intel(0): Default visual is TrueColor [ 188.628] (**) intel(0): Option "TearFree" "true" [ 188.628] (II) intel(0): Output VGA1 has no monitor section [ 188.628] (II) intel(0): Enabled output VGA1 [ 188.628] (II) intel(0): Output HDMI1 has no monitor section [ 188.628] (II) intel(0): Enabled output HDMI1 [ 188.628] (II) intel(0): Output HDMI2 has no monitor section [ 188.628] (II) intel(0): Enabled output HDMI2 [ 188.628] (II) intel(0): Output DP1 has no monitor section [ 188.628] (II) intel(0): Enabled output DP1 [ 188.628] (II) intel(0): Output HDMI3 has no monitor section [ 188.628] (II) intel(0): Enabled output HDMI3 [ 188.628] (--) intel(0): Using a maximum size of 256x256 for hardware cursors [ 188.628] (II) intel(0): Output VIRTUAL1 has no monitor section [ 188.628] (II) intel(0): Enabled output VIRTUAL1 [ 188.628] (--) intel(0): Output HDMI2 using initial mode 1920x1200 on pipe 0 [ 188.628] (--) intel(0): Output DP1 using initial mode 1920x1200 on pipe 1 [ 188.628] (**) intel(0): TearFree enabled [ 188.628] (==) intel(0): DPI set to (96, 96) [ 188.628] (II) Loading sub module "dri2" [ 188.628] (II) LoadModule: "dri2" [ 188.628] (II) Module "dri2" already built-in [ 188.628] (II) Loading sub module "present" [ 188.628] (II) LoadModule: "present" [ 188.628] (II) Module "present" already built-in [ 188.628] (==) Depth 24 pixmap format is 32 bpp [ 188.630] (II) intel(0): SNA initialized with Haswell (gen7.5, gt2) backend [ 188.630] (==) intel(0): Backing store enabled [ 188.630] (==) intel(0): Silken mouse enabled [ 188.631] (II) intel(0): HW Cursor enabled [ 188.631] (II) intel(0): RandR 1.2 enabled, ignore the following RandR disabled message. [ 188.631] (==) intel(0): DPMS enabled [ 188.631] (==) intel(0): display hotplug detection enabled [ 188.631] (II) intel(0): [DRI2] Setup complete [ 188.631] (II) intel(0): [DRI2] DRI driver: i965 [ 188.631] (II) intel(0): [DRI2] VDPAU driver: i965 [ 188.631] (II) intel(0): direct rendering: DRI2 enabled [ 188.631] (II) intel(0): hardware support for Present enabled [ 188.631] (--) RandR disabled [ 188.631] (II) Found 3 VGA devices: arbiter wrapping enabled [ 188.651] (II) AIGLX: enabled GLX_MESA_copy_sub_buffer [ 188.651] (II) AIGLX: enabled GLX_ARB_create_context [ 188.651] (II) AIGLX: enabled GLX_ARB_create_context_profile [ 188.651] (II) AIGLX: enabled GLX_EXT_create_context_es2_profile [ 188.651] (II) AIGLX: enabled GLX_INTEL_swap_event [ 188.651] (II) AIGLX: enabled GLX_SGI_swap_control and GLX_MESA_swap_control [ 188.651] (II) AIGLX: enabled GLX_EXT_framebuffer_sRGB [ 188.651] (II) AIGLX: enabled GLX_ARB_fbconfig_float [ 188.651] (II) AIGLX: GLX_EXT_texture_from_pixmap backed by buffer objects [ 188.651] (II) AIGLX: enabled GLX_ARB_create_context_robustness [ 188.651] (II) AIGLX: Loaded and initialized i965 [ 188.651] (II) GLX: Initialized DRI2 GL provider for screen 0 [ 188.653] (II) intel(0): switch to mode 1920x1200@60.0 on HDMI2 using pipe 0, position (0, 0), rotation normal, reflection none [ 188.667] (II) intel(0): switch to mode 1920x1200@60.0 on DP1 using pipe 1, position (0, 0), rotation normal, reflection none [ 188.677] (II) intel(0): Setting screen physical size to 508 x 317

Like most people her I'm having difficulty with sound. The hda hardware doesn't work at all and the ac79 got some crackling noise.
I've tried to play around with the qemu options but it doesn't make any difference. Even the QEMU_AUDIO_DRV=wav backend crackle. There is no crackle when playing steam trailers or music with audio players. I'm guessing games use a smaller audio buffer and are more susceptible.

I've tried forwarding my onboard mobo audio but it was never even detected. Has anyone got that to work? Last resort would be to buy an usb soundcard and forward that but I don't want to buy hardware to work around bugs in qemu.

Offline

#4517 2015-03-19 14:41:50

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Denso wrote:

How about NVIDIA ?

Weird. I have gigabyte GT610 which supports UEFI perfectly..
Anyway, try searching for NVidia's GOP technical presentations and etc - i've found one for AMD dating back to 2010, i think, saying "there will be an universal GOP for all cards". That presentation just proved the idea of copying GOP from one's GPU to another.

Also, try writing a support ticket to your hardware's vendor - in my ASUS case it just cleared up some things. But actually, they have an official update tool(which requires windowns installed..).

Also, maybe you just haven't got enough luck, since some nvidia cards may be converted to quadro just by firmware change.

You can also dump all UEFI-compatible ROMs from DB and try ripping uncompressed EFI driver same way we did it for AMD, it may suddenly be universal for all vendors and containing only GOP without any important bits. Supplying it as romfile= option will be a safe way of testing it out. The resulting file from firmware should be .efi image, you glue it with your normal vbios same way as AMD, using EfiRom(a tool from EDK2 btw). I don't recall properly, but i think EfiRom utility could decompress that efi image from ROM..

P.S. it wasn't a hex editor, it's an Elegant Weapon for a More Civilized Age of UEFI.

You are right , I have a GT610 too which has a UEFI ROM . But this new GT630 doesn't .

The card is from a vendor called "Point of View" , I'll try sending them a UEFI ROM request .

I did try ripping the UEFI partition from some ROMs I found in TPU , but that didn't succeed . Even AW's rom-parser didn't pick up the UEFI bits sad

Offline

#4518 2015-03-19 16:19:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Even AW's rom-parser didn't pick up the UEFI bits sad

Well, EfiRom will tell you something very alike. If AW's parser doesn't recognise any EFI drivers in ROM - then there is no EFI support in this ROM. At all.
Only one detail applies: nvidia cards sometimes have "pre-post rom". Adjust your plans accordingly.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4519 2015-03-19 19:28:34

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist strikes a menacing pose.

Laszlo, you are awesome. OVMF is awesome. Everything is awesome.

<domain type='kvm' id='10'> <name>vm1</name> <uuid>f6958833-174f-44e6-817b-2a6c73c6f041</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/vm1_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/hdd/qemu/windows7.iso'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <boot order='2'/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/hdd/qemu/virtio.iso'/> <backingStore/> <target dev='hdc' bus='ide'/> <readonly/> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/usr/share/edk2.git/ovmf-x64/UefiShell.iso'/> <backingStore/> <target dev='hdd' bus='ide'/> <readonly/> <alias name='ide0-1-1'/> <address type='drive' controller='0' bus='1' target='0' unit='1'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda'/> <backingStore/> <target dev='vda' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:f0:d5:39'/> <source network='default'/> <target dev='vnet0'/> <model type='rtl8139'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='usb'> <alias name='input0'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c624,c898</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c624,c898</imagelabel> </seclabel> </domain>

So, how did it go:
1. QXL graphics came online, i did all the installation tasks via QXL.
2. I tried to install AMD drivers for win8.1 on win7 and it didn't work, showing code 10 on both GPUs.
3. But then.. DL'ed win7 drivers and after a bootup logo(which is dublicated on the physical screen alike tianocore logo and OVMF menus) QXL screen froze...
4. In the beginning there was silence and darkness all across the world; Then came the Cursor and a loading screen...
5. Windows popped up a message about turning QXL off, and QXL became code-10'd, both GPUs code 0.
6. It's safe to get rid of QXL, but Spice won't work, libvirt won't provide a surface for input grabbing purposes.

Seems like finally libvirt+crossfire+windows7+vfio got along with each other. There's some issues left with the setup, namely disk performance(and safety too), cpu pinning and input, but otherwise it's fine. And there's a bunch of unneeded devices and disk attached.

UPD:
Benchmarks show negligible performance hit in furmark, otherwise system works as usual and very similar to bare metal setup. I wonder if i can actually boot this in bare metal, since i'm using the most crazy way of managing disks: whole /dev/sda to the VM.

UPD2:
There's some sound issues, as my sound just falls off VM. It works, VM reboot and there's no sound device anymore. But i'm more worried about strange shutdown behavior:
Windowns 7 starts shutting itself down, kills explorer and stuff, and stops. I have a wallpaper, a mouse cursor which i can move, but nothing else happens. And it also boots very slowly, i guess there's something to do with the disk access.
If i force it off, on next boot it stalls at boot, refusing to execute OVMF at least.
Only rebooting the host helps. There's nothing related to GPUs in dmesg, so i assume it's libvirt trickery that fails to work.
GPU reset problems? Huh? Why?.. It was working fine with that before when i used qemu without libvirt.

Last edited by Duelist (2015-03-20 17:13:44)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4520 2015-03-20 13:39:21

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How did you get Win7 to even load with more than 1 CPU with EFI boot? What is QXL?

My install would just freeze when the initial Windows Logo would appear after loading all the setup files (Black screen and bar loading on bottom).

Edit: Went back one page and found the post by Lersek: https://bbs.archlinux.org/viewtopic.php … 5#p1511585

Last edited by The_Moves (2015-03-20 14:11:18)

Offline

#4521 2015-03-20 17:12:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

How did you get Win7 to even load with more than 1 CPU with EFI boot? What is QXL?

My install would just freeze when the initial Windows Logo would appear after loading all the setup files (Black screen and bar loading on bottom).

Edit: Went back one page and found the post by Lersek: https://bbs.archlinux.org/viewtopic.php … 5#p1511585

Actually...
http://www.linux-kvm.org/downloads/lers … 770f8c.txt
Read all posts by lersek - he(and a bunch of other guys) develops OVMF. I mean, when we had initial troubles with VFIO, we asked alex(aw) for help, because he wrote it. Now we had problems with OVMF, and here he comes and explains it all.

I think, btw, his very first post and OVMF's whitepaper should be pinned into op-post, just for overall awareness.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4522 2015-03-20 18:19:25

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone got sound working well using qemu emulation.
I removed spice display and vga emulated driver and I only use assigned video cards and my sound is send to alsa but it does not sound that great.
I was thinking about hot plugin sound card with virsh is that possible I see there is  attach-device                  attach device from an XML file option

Offline

#4523 2015-03-20 18:46:16

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I think, btw, his very first post and OVMF's whitepaper should be pinned into op-post, just for overall awareness.

I agree! Thanks for posting this!

Offline

#4524 2015-03-20 22:24:50

VOT Productions
Member
Registered: 2011-10-22
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To everyone having issues with "Starting Windows" screen and it not progressing further:
Do -vga qxl while installing Windows. Then, install your GPU driver. Then put it back to -vga none, worked for me smile

However, I have a problem. I am trying to get Synergy to work, and this is my full QEMU command.

#!/bin/sh sudo brctl addbr br0 sudo ip addr add 172.20.0.1/16 dev br0 sudo ip link set br0 up sudo vfio-bind 0000:01:00.0 0000:01:00.1 cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa sudo qemu-system-x86_64 \ -enable-kvm -m 4096 -cpu host,kvm=off -smp cores=4,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/tmp/my_vars.fd \ -vga none \ -soundhw hda \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/1TB_LinuxMedia/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/mnt/1TB_LinuxMedia/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=/mnt/1TB_LinuxMedia/windows2.img,id=disk2,format=raw,if=none -device scsi-hd,drive=disk2 \ -usb -usbdevice host:04d9:a055 -usbdevice host:04d9:a04a \ -netdev user,id=net0 -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev bridge,br=br0,id=net1 -device e1000,netdev=net1,mac=52:54:00:12:34:57 \ -usbdevice tablet ip link set br0 down

It makes two virtual networks on Windows, one which can access the internet, and one internal. The trouble is, having that internal network on slows down browsing on Windows MASSIVELY, to the point that some websites don't load. If I disable that adapter in Windows, the websites load instantly like they should. If anyone knows how to connect Qemu to the network with a wlan0 interface (so I can't bridge) or knows how to fix the slowdown, please help me wink

Offline

#4525 2015-03-20 23:37:13

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It makes two virtual networks on Windows, one which can access the internet, and one internal. The trouble is, having that internal network on slows down browsing on Windows MASSIVELY, to the point that some websites don't load. If I disable that adapter in Windows, the websites load instantly like they should. If anyone knows how to connect Qemu to the network with a wlan0 interface (so I can't bridge) or knows how to fix the slowdown, please help me

I use the same scenario as you , my way for setting up the internal network is explained here

No issues at all .

Last edited by Denso (2015-03-20 23:37:40)

Offline

#4526 2015-03-21 03:06:15

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am using couple of R9 270 assigned to windows 8.1 and that seems to work perfectly.
I am testing nvidia 220 that I have laying around from using it with mythtv I am getting error 43.
From reading adding this option will work around the error code

<domain type='kvm'...> <features> <kvm> <hidden state='on'/> </kvm> </features> ... </domain>

Is this how I can get errror code 43 fixed ?
Did anyone tried to install windows 3.11 I only get black screen.

Last edited by tritron4 (2015-03-21 03:07:37)

Offline

#4527 2015-03-21 03:23:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

Did anyone tried to install windows 3.11 I only get black screen.

Please tell me you're joking.  It's not just hiding kvm, it's disabling all hyper-v extensions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4528 2015-03-21 03:23:04

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3.11? Three dot One One

Offline

#4529 2015-03-21 03:38:16

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Please tell me you're joking.

I just tried FreeDOS . It is working . (Needed to use "NVFlash" to flash my GPU's firmware through DOS and it worked) .

Last edited by Denso (2015-03-21 03:39:04)

Offline

#4530 2015-03-21 03:40:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flashing firmware in a guest on an assigned device is extremely dis-recommended.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4531 2015-03-21 03:46:31

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

3.11? Three dot One One

Yes for some old games fun 7th guest 11th hour and etc
I had found dos 6.22 flopy images and windows 3.11.
I had installed dos and cdrom drivers and I installed 3.11. There is a way to install network drivers and web browser also.
I can install windows 3.11 and it installs fine and I can exit to dos and start graphical interface but when I reboot guest win 3.11 starts with blackscreen.
I have the same issue with osx it boots to black screen when I try to install it.

Offline

#4532 2015-03-21 04:52:57

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Flashing firmware in a guest on an assigned device is extremely dis-recommended.

Yup . Although it worked for me , I wouldn't do it again . This GT630 without UEFI was pretty useless anyway .

Offline

#4533 2015-03-21 11:12:02

myweb
Member
Registered: 2013-07-13
Posts: 68

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

Did anyone try Linux kernel 4.0 or 3.19 with vfio?
I see only 3.18 packadge in first post - are there any issues with new kernels (3.19 & 4.0)

Thanks!

Offline

#4534 2015-03-21 12:07:14

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Flashing firmware in a guest on an assigned device is extremely dis-recommended.

OH GOD! Oh my god! Whoa.. Well.. That's a huge surprise he succeeded.

tritron4 wrote:

Yes for some old games fun 7th guest 11th hour and etc

Use DosBox, it's more suited for that purpose. Why would you want to passthrough something from 21 century into DOS?

Last edited by Duelist (2015-03-21 12:07:22)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4535 2015-03-21 13:01:19

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear All,

Did anyone try Linux kernel 4.0 or 3.19 with vfio?
I see only 3.18 packadge in first post - are there any issues with new kernels (3.19 & 4.0)

Thanks!

I am using 3.19.1 kernel it works fine.

Offline

#4536 2015-03-21 13:52:36

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear All,

Did anyone try Linux kernel 4.0 or 3.19 with vfio?
I see only 3.18 packadge in first post - are there any issues with new kernels (3.19 & 4.0)

Thanks!

Looks like someone is maintaining a linux-vfio kernel package on AUR, i will link it on the first page https://aur.archlinux.org/packages/?O=0&K=linux-vfio

Offline

#4537 2015-03-21 16:44:40

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello gentlemen, after some months of inactivity I decided to continue.
I noticed that after some hours the entire machine crashes (linux clock stops and keyboard leds doesn't change for example)
this happens on both windows 7 + seabios and windows 8.1 + OVMF pure

I have arch linux with unpatched linux-mainline 4.0 (ACS works without patches)
qemu from official git and propietary nvidia drivers.
430 as primary and 970 for passtrough (Asrock added primary video pci-slot selector for this motherboard as asked ^o^ )

There is any clue of what would be? journalctl and qemu monitor doesn't show anything when crash occurrs

Offline

#4538 2015-03-21 16:52:51

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Hello gentlemen, after some months of inactivity I decided to continue.
I noticed that after some hours the entire machine crashes (linux clock stops and keyboard leds doesn't change for example)
this happens on both windows 7 + seabios and windows 8.1 + OVMF pure

I have arch linux with unpatched linux-mainline 4.0 (ACS works without patches)
qemu from official git and propietary nvidia drivers.
430 as primary and 970 for passtrough (Asrock added primary video pci-slot selector for this motherboard as asked ^o^ )

There is any clue of what would be? journalctl and qemu monitor doesn't show anything when crash occurrs

Open dmesg -w in the background and throw a glimpse on it once in a while.
Also, try adding qemu debug console for a variant with OVMF.
It would be useful to monitor GPU's power state(including temperature) AND host CPU's temp and P-state.
Also, try pushing alt-sysrq-b when your linux host is frozen, maybe it's just X window system that freezes. Pushing Alt-SysRQ-B with "magic sysrq key" option enabled in kernel will force reboot your machine. It will work even if everything is dead, but kernel is still alive.
You could also try pinging that machine, to ensure that it's a complete freeze.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4539 2015-03-21 22:28:28

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi . So I discovered that I only get the following with GT610 (the card that doesn't allow VM rebooting) :

[Sat Mar 21 23:01:57 2015] irq 47: nobody cared (try booting with the "irqpoll" option) [Sat Mar 21 23:01:57 2015] CPU: 0 PID: 3 Comm: ksoftirqd/0 Not tainted 4.0.0-1-ARCH #1 [Sat Mar 21 23:01:57 2015] Hardware name: ASUS All Series/X99-DELUXE, BIOS 1305 01/26/2015 [Sat Mar 21 23:01:57 2015] 0000000000000000 000000002442baf4 ffff88083fc03ac0 ffffffff815636ba [Sat Mar 21 23:01:57 2015] 0000000000000000 ffff88081806b600 ffff88083fc03af0 ffffffff810d0186 [Sat Mar 21 23:01:57 2015] ffffffff8189a880 ffff88081806b600 0000000000000000 000000000000002f [Sat Mar 21 23:01:57 2015] Call Trace: [Sat Mar 21 23:01:57 2015] <IRQ> [<ffffffff815636ba>] dump_stack+0x4c/0x6e [Sat Mar 21 23:01:57 2015] [<ffffffff810d0186>] __report_bad_irq+0x36/0xd0 [Sat Mar 21 23:01:57 2015] [<ffffffff810d0527>] note_interrupt+0x257/0x2a0 [Sat Mar 21 23:01:57 2015] [<ffffffff810cda6e>] handle_irq_event_percpu+0xae/0x1f0 [Sat Mar 21 23:01:57 2015] [<ffffffff810cdbf1>] handle_irq_event+0x41/0x70 [Sat Mar 21 23:01:57 2015] [<ffffffff810d0ab6>] handle_fasteoi_irq+0x86/0x140 [Sat Mar 21 23:01:57 2015] [<ffffffff81018372>] handle_irq+0x22/0x40 [Sat Mar 21 23:01:57 2015] [<ffffffff8156ba4f>] do_IRQ+0x4f/0xf0 [Sat Mar 21 23:01:57 2015] [<ffffffff8156996d>] common_interrupt+0x6d/0x6d [Sat Mar 21 23:01:57 2015] [<ffffffff8115bea0>] ? end_page_writeback+0x60/0x60 [Sat Mar 21 23:01:57 2015] [<ffffffff81213732>] ? mpage_end_io+0x42/0x60 [Sat Mar 21 23:01:57 2015] [<ffffffff81285c8b>] bio_endio+0x6b/0xa0 [Sat Mar 21 23:01:57 2015] [<ffffffff8128d144>] blk_update_request+0x94/0x3a0 [Sat Mar 21 23:01:57 2015] [<ffffffffa00b1c53>] scsi_end_request+0x33/0x1d0 [scsi_mod] [Sat Mar 21 23:01:57 2015] [<ffffffffa00b372a>] scsi_io_completion+0xba/0x630 [scsi_mod] [Sat Mar 21 23:01:57 2015] [<ffffffffa0094561>] ? sd_done+0xb1/0x2e0 [sd_mod] [Sat Mar 21 23:01:57 2015] [<ffffffffa00ab17e>] scsi_finish_command+0xbe/0x100 [scsi_mod] [Sat Mar 21 23:01:57 2015] [<ffffffffa00b3011>] scsi_softirq_done+0x111/0x140 [scsi_mod] [Sat Mar 21 23:01:57 2015] [<ffffffff8129437b>] blk_done_softirq+0x8b/0xb0 [Sat Mar 21 23:01:57 2015] [<ffffffff812942fa>] ? blk_done_softirq+0xa/0xb0 [Sat Mar 21 23:01:57 2015] [<ffffffff81078a21>] __do_softirq+0xe1/0x2c0 [Sat Mar 21 23:01:57 2015] [<ffffffff81078d4e>] irq_exit+0x7e/0xa0 [Sat Mar 21 23:01:57 2015] [<ffffffff8156ba58>] do_IRQ+0x58/0xf0 [Sat Mar 21 23:01:57 2015] [<ffffffff8156996d>] common_interrupt+0x6d/0x6d [Sat Mar 21 23:01:57 2015] <EOI> [<ffffffff81092eda>] ? kthread_should_park+0x1a/0x30 [Sat Mar 21 23:01:57 2015] [<ffffffff81096408>] ? smpboot_thread_fn+0x58/0x230 [Sat Mar 21 23:01:57 2015] [<ffffffff810963b0>] ? SyS_setgroups+0x130/0x130 [Sat Mar 21 23:01:57 2015] [<ffffffff81092878>] kthread+0xd8/0xf0 [Sat Mar 21 23:01:57 2015] [<ffffffff810927a0>] ? kthread_create_on_node+0x1c0/0x1c0 [Sat Mar 21 23:01:57 2015] [<ffffffff81568dbc>] ret_from_fork+0x7c/0xb0 [Sat Mar 21 23:01:57 2015] [<ffffffff810927a0>] ? kthread_create_on_node+0x1c0/0x1c0 [Sat Mar 21 23:01:57 2015] handlers: [Sat Mar 21 23:01:57 2015] [<ffffffffa0475890>] vfio_intx_handler [vfio_pci] [Sat Mar 21 23:01:57 2015] Disabling IRQ #47

Could this lead to the reason for the VM rebooting failures ?

Any help is very much appreciated !

EDIT :

UGHHH ! setting : vfio_pci nointxmask=1  helped getting rid of the IRQ disabling message , but not the VM rebooting issue .

Last edited by Denso (2015-03-21 22:43:33)

Offline

#4540 2015-03-21 23:03:51

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

> Open dmesg -w in the background and throw a glimpse on it once in a while.

Its not supposed that journal stores the dmesg? or maybe crash occurs so fast that journal doesnt save?

> Also, try adding qemu debug console for a variant with OVMF.

what you mean, compiling qemu in debug mode? or downloading a debug variant of OVMF?

>It would be useful to monitor GPU's power state(including temperature) AND host CPU's temp and P-state.

Temp is at normal levels, I don't know how to see P-state or gpu power state trough

>Also, try pushing alt-sysrq-b when your linux host is frozen, maybe it's just X window system that freezes. Pushing Alt-SysRQ-B with "magic sysrq key" option enabled in kernel will force reboot your machine. It will work even if everything is dead, but kernel is still alive.
>You could also try pinging that machine, to ensure that it's a complete freeze.

Its a total freeze, even sysrq doesn't respond and ping timeouts

EDIT:

Got this while using win 8.1 and UEFI, it was constant message with 5 secs interval

mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu4 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu4 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:53 I-A-X99 kernel: kvm_get_msr_common: 218 callbacks suppressed mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu8 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu8 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:48 I-A-X99 kernel: kvm_get_msr_common: 202 callbacks suppressed mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu8 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu8 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu11 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm [28001]: vcpu9 unhandled rdmsr: 0x150 mar 22 01:19:43 I-A-X99 kernel: kvm_get_msr_common: 186 callbacks suppressed mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu6 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu6 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu6 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm [28001]: vcpu6 unhandled rdmsr: 0x150 mar 22 01:19:38 I-A-X99 kernel: kvm_get_msr_common: 206 callbacks suppressed mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu1 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu1 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:33 I-A-X99 kernel: kvm_get_msr_common: 230 callbacks suppressed mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu1 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu1 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:28 I-A-X99 kernel: kvm_get_msr_common: 230 callbacks suppressed mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu1 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu1 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu3 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150 mar 22 01:19:23 I-A-X99 kernel: kvm [28001]: vcpu0 unhandled rdmsr: 0x150

Last edited by Cubex (2015-03-22 00:33:53)

Offline

#4541 2015-03-22 09:55:28

tholin
Member
Registered: 2015-03-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tholin wrote:
aw wrote:

You seem to have a good grasp of the options.  Personally I'd opt for using IGD for the host and replacing your guest card with one that supports UEFI.  You seem to be averse to the VGA BIOS hacking route, but that's actually one that's likely to continue working once you figure it out, with no future i915 kernel patching.  Remember you don't need to flash the card with the new ROM, you can use an external file as the ROM for the VM.  If you're willing to settle for a x1 graphics card, you can find x1-to-x16 adapters that will allow you to plug any low-profile x16 card into the x1 slot.  Double check your Xorg.0.log for DRI, DRI3 may still be enabled, but DRI2 has arbiter issues... don't ask me what the difference is, I don't know.

I've been using the IGD for a few days and realized the drivers are too buggy for daily use. Maybe it's related to the i915 patch but many other people have the same problems. To be fair the open radeon drivers also got a lot of problems but I'm used to them and know how to work around the problems. I ordered that 1X radeon card instead.

Turn out there is a reason why 1X graphic cards are so rare. They are shit. 1X does not provide enough bandwidth for a graphic card. Scrolling in browsers, moving windows and even the kvm framebuffer is so slow it's unusable. Specially if you try to use dual or triple head. So the option of using two separate radeon cards are out.

Most of the problems I have with the Intel IGD are bugs and bugs can in theory be fixed but my experience with the intel drivers are bad. They introduce bugs at about the same rate as they are fixed. Turning off SNA and going back to using UXA acceleration fix a lot of problems but I still get graphic corruption and suspend resume doesn't work.

I tried to run without the i915 vgaarb patch so see what would happen and as expected the guest stalled in boot and the host got a corrupt color palette. The main problem I have with legacy free guest is I don't have a windows 8 license and I don't know of any way of getting one, except for buying one which is not going to happen. For now it looks like I'm stuck with Intel IGD + i915 vgaarb patch and all the bugs that brings.

While I was playing a game in the vm yesterday the entire system froze and after reboot I had machine check exception in the log. I can't interpret mce:s but it was some kind of bus error. I've never seen this problem before. Perhaps that 1X graphic card was defect and caused the mce or perhaps it was caused by something else? I took out the card and haven't seen any mce since. Lets see if that lasts.

Offline

#4542 2015-03-22 13:22:54

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Hello gentlemen, after some months of inactivity I decided to continue.
I noticed that after some hours the entire machine crashes (linux clock stops and keyboard leds doesn't change for example)
this happens on both windows 7 + seabios and windows 8.1 + OVMF pure

I have arch linux with unpatched linux-mainline 4.0 (ACS works without patches)
qemu from official git and propietary nvidia drivers.
430 as primary and 970 for passtrough (Asrock added primary video pci-slot selector for this motherboard as asked ^o^ )

There is any clue of what would be? journalctl and qemu monitor doesn't show anything when crash occurrs

So you're saying that you can passthrough devices in same group without ACS override patch in 4.0?

Offline

#4543 2015-03-22 13:37:42

Draizer
Member
Registered: 2015-03-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, by using OVMF instead of SeaBIOS I managed to install Windows, boot into it and install the drivers for my card, which is properly detected.  However, once I install the drivers and try to boot, I get stuck on the "Starting Windows" screen (and sometimes my whole system freezes along with it), unless I boot into safe mode or my card is disabled in device manager, which clearly shows where the issue comes from.

After checking up, I realized that my graphics card vbios (Sapphire ATI HD 7970 Ghz Edition Vapor-X 3Go) does NOT support UEFI GOP, which I suppose is the reason of my troubles. Unfortunately, since it is a custom PCB card, a generic 7970 vBIOS with UEFI support will not work (I've tried), so I've contacted Sapphire's support about this issue, as well as a Sapphire forum one-time poster who, in his unique post, complained about the same issues about the same card and apparently obtained a modified vBIOS with UEFI support. Hopefully one of these two will provide me the modified bios so that I can go forth (or at least confirm the vBIOS is not my issue). If anybody here needs it, I'll be willing to share if I get my hands on it.

I find it pretty outrageous that such a pricey card (when it came out) does not support this by default, but that's my fault for assuming it'd have UEFI support. The fact that other suppliers such as PowerColor DO support it on this model is sort of frustrating, though.

Last edited by Draizer (2015-03-22 13:41:20)

Offline

#4544 2015-03-22 13:47:54

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

So you're saying that you can passthrough devices in same group without ACS override patch in 4.0?

No, my new build has haswell-E and X99, which IOMMU groups works as intended, in the previous build with Z87 it was necesary to use the patch, 4.0 doesn't change anything

Offline

#4545 2015-03-22 15:07:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

> Open dmesg -w in the background and throw a glimpse on it once in a while.
Its not supposed that journal stores the dmesg? or maybe crash occurs so fast that journal doesnt save?

1. I don't know how journalctl works, i know that dmesg MIGHT show you something. And if it's a full freeze, having dmesg -w on the screen will give you logs immediately, on your framebuffer or console device with almost zero latency.

Cubex wrote:

> Also, try adding qemu debug console for a variant with OVMF.
what you mean, compiling qemu in debug mode? or downloading a debug variant of OVMF?

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

It'll create some more verbose logs. (maybe you'll have to use OVMF, i can't recall if seabios use that debug console)

Also, usually you may see those rdmsr errors when starting something like CPU-Z. Try it and see if the system freezes.

---

Draizer wrote:

So, by using OVMF instead of SeaBIOS I managed to install Windows, boot into it and install the drivers for my card, which is properly detected.

Unbelievable. Are you sure you used -pure-efi variant without CSM? Because if your GPU doesn't support GOP at all - it won't output graphics, moreover, it might not boot at all!

Draizer wrote:

my graphics card vbios (Sapphire ATI HD 7970 Ghz Edition Vapor-X 3Go) does NOT support UEFI GOP

I'm a little bit surprised that this issue is present on sapphire cards too. Crap. Seems like we have to buy only reference-cards made by AMD itself to have adequate support for them.
Anyway, what you can do with it:
https://bbs.archlinux.org/viewtopic.php … 3#p1504683
As i've found out, back in 2010 or 2012 or some years ago, AMD did a presentation on UEFI-forum, claiming that their GOP EFI Driver will be universal.
UPD: found the link for that presentation: http://www.uefi.org/sites/default/files … OP_AMD.pdf
Well. It is.
Even if you have a custom PCB weird no-name GPU, as long as you have it's own bios-without-GOP AND a working GOP efi driver, you can compress the latter and stuff it into the end of your image.
There MAY be problems with ROM Chip size, well, the modified image may not fit in. BUT! Since we're dealing with QEMU, physical capability of ROM chip doesn't matter.

I, myself, have HD7750 from ASUS, a silent, passively cooled variant. I highly doubt it's reference design.

UPD2:
http://forums.amd.com/game/messageview. … did=163119
Well, seems like AMD has failed once again:

Through AMD's extensive testing, a significant number of existing platforms experienced compatibility issues with the UEFI specifications that were implemented with the Hybrid VBIOS AMD developed.  Users could experience boot failures on incompatible systems.  To ensure compatibility, the standard VGA VBIOS continues to be used, and we cannot transition with the partner manufacturers yet.  It is a System BIOS implementation issue that can be addressed with a System BIOS update.

This is not an issue on OE system builds (HP, DELL, etc..) or integrated graphics components (where VBIOS is implemented directly into SBIOS), because full UEFI compatibility is set from the ground-up.

Also, some more links:
http://hardforum.com/showthread.php?t=1725316&page=4
http://www.overclock.net/t/1389206/do-y … -your-bios <<THAT'S AN AWFUL WAY of doing stuff, which EfiRom does.

Last edited by Duelist (2015-03-22 15:17:34)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4546 2015-03-22 15:53:23

Draizer
Member
Registered: 2015-03-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Unbelievable. Are you sure you used -pure-efi variant without CSM? Because if your GPU doesn't support GOP at all - it won't output graphics, moreover, it might not boot at all!

Well, I'm an idiot. Now that I've taken a bit of rest and see things more clearly I'm pretty sure the reason it worked is because Windows skipped the ATI card and used QEMU's emulated card. Using -vga none leads me straight into the QEMU console. With that said, Windows does detect the right card... it'll just get stuck on its starting screen if I try to use it.

And yes, I'm sure I'm using a pure-efi binary.

Duelist wrote:

I'm a little bit surprised that this issue is present on sapphire cards too. Crap. Seems like we have to buy only reference-cards made by AMD itself to have adequate support for them.
Anyway, what you can do with it:
https://bbs.archlinux.org/viewtopic.php … 3#p1504683
As i've found out, back in 2010 or 2012 or some years ago, AMD did a presentation on UEFI-forum, claiming that their GOP EFI Driver will be universal.
UPD: found the link for that presentation: http://www.uefi.org/sites/default/files … OP_AMD.pdf
Well. It is.
Even if you have a custom PCB weird no-name GPU, as long as you have it's own bios-without-GOP AND a working GOP efi driver, you can compress the latter and stuff it into the end of your image.
There MAY be problems with ROM Chip size, well, the modified image may not fit in. BUT! Since we're dealing with QEMU, physical capability of ROM chip doesn't matter.

I, myself, have HD7750 from ASUS, a silent, passively cooled variant. I highly doubt it's reference design.

I'll check that out, hopefully I'll get lucky.

Last edited by Draizer (2015-03-22 22:17:31)

Offline

#4547 2015-03-22 17:54:10

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

Quick question: I've seen few times in this topic information that Nvidia GPU's require to be put in 1st pci x16 slot. What does it say about using Nvidia for host and another Nvidia for guest? Would there be problems?

Currently I have GF 9800 GTX+ for host and Radeon 4850 for Win guest and I'm going to change Radeon for something nicer and was thinking about Nvidia GTX 780.

Offline

#4548 2015-03-23 00:39:48

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

Offline

#4549 2015-03-23 00:44:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

And radeon will give you reset problems.  We know how to work around the Code 43 issue (for now), we don't have a good solution for the resets.  dRaiser, whatever you're reading about the 1st slot and nvidia is wrong.  Slots come into play for ACS and if your motherboard vendor doesn't allow fine grained selection of the primary graphics device.  The only issue I'm aware of with using nvidia for the host is that the proprietary driver grabs the vga arbiter lock and doesn't release it.  So you either need to avoid vga or hack nvidia's wrapper code.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4550 2015-03-23 00:48:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
tritron4 wrote:

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

The only issue I'm aware of with using nvidia for the host is that the proprietary driver grabs the vga arbiter lock and doesn't release it.  So you either need to avoid vga or hack nvidia's wrapper code.

I think nvidia fixed this since i'm using the stock drivers without issue.

Offline

#4551 2015-03-23 00:57:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
aw wrote:
tritron4 wrote:

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

The only issue I'm aware of with using nvidia for the host is that the proprietary driver grabs the vga arbiter lock and doesn't release it.  So you either need to avoid vga or hack nvidia's wrapper code.

I think nvidia fixed this since i'm using the stock drivers without issue.

I use IGD for my host graphics, but I just download the latest driver, NVIDIA-Linux-x86_64-346.47  In there I find:

kernel/nv.c:nvidia_probe() ... #if defined(CONFIG_VGA_ARB) #if defined(VGA_DEFAULT_DEVICE) vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK); #endif vga_set_legacy_decoding(dev, VGA_RSRC_NONE); #endif

There's no call to vga_put() anywhere.  Still looks broken to me, but maybe this is dead code or there are corner cases.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4552 2015-03-23 01:21:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
nbhs wrote:

I think nvidia fixed this since i'm using the stock drivers without issue.

I use IGD for my host graphics, but I just download the latest driver, NVIDIA-Linux-x86_64-346.47  In there I find:

kernel/nv.c:nvidia_probe() ... #if defined(CONFIG_VGA_ARB) #if defined(VGA_DEFAULT_DEVICE) vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK); #endif vga_set_legacy_decoding(dev, VGA_RSRC_NONE); #endif

There's no call to vga_put() anywhere.  Still looks broken to me, but maybe this is dead code or there are corner cases.

Oh, maybe it is fixed.  Calling vga_set_legacy_decoding() clears the locked resources and does an implicit __vga_put().

EDIT: Embarrassing, it might be me who fixed it http://git.kernel.org/cgit/linux/kernel … 941affd75a

Last edited by aw (2015-03-23 01:28:02)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4553 2015-03-23 04:36:24

asd651651
Member
Registered: 2015-03-23
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,
I can pass through GPU successfully but after shutdown the VM and restart the VM, the host system stuck. Does anyone have that problem too?
It seems like the problem only occurs when passing through GPU, there is no such problem when passing through the USB controller.

Here’s my system environment:
    Kernel version: 3.18.8 (patched with linux-mainline)
    GPU: GTX 295
    Qemu: 2.2.0

Offline

#4554 2015-03-23 05:04:16

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I suffer from the booting problem too...
When I use the VFIO GPU passthrough in a VM, shutdown the VM, but I cannot boot the VM unless I restart my host system.
Does anyone know this problem?

Offline

#4555 2015-03-23 12:17:05

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
nbhs wrote:

I think nvidia fixed this since i'm using the stock drivers without issue.

I use IGD for my host graphics, but I just download the latest driver, NVIDIA-Linux-x86_64-346.47  In there I find:

kernel/nv.c:nvidia_probe() ... #if defined(CONFIG_VGA_ARB) #if defined(VGA_DEFAULT_DEVICE) vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK); #endif vga_set_legacy_decoding(dev, VGA_RSRC_NONE); #endif

There's no call to vga_put() anywhere.  Still looks broken to me, but maybe this is dead code or there are corner cases.

Oh, maybe it is fixed.  Calling vga_set_legacy_decoding() clears the locked resources and does an implicit __vga_put().

EDIT: Embarrassing, it might be me who fixed it http://git.kernel.org/cgit/linux/kernel … 941affd75a

Perhaps I misunderstood what a problem is, thanks for clarifying.

Offline

#4556 2015-03-23 13:51:28

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

Look up PCIe 'Function Level Reset'

Offline

#4557 2015-03-23 15:37:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To be honest, i'd totally go check that huge google spreadsheet with stats before changing existing or beginning a new setup.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4558 2015-03-23 15:42:26

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist, thank you - haven't seen that before. Great resource!

Offline

#4559 2015-03-23 15:51:46

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

@Duelist, thank you - haven't seen that before. Great resource!

While most of the data is okay, some setups may be semi-working(like my entry). If you're in doubt - you can search that user's posts in this thread using google site: search.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4560 2015-03-23 18:14:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A short tale about SELinux and VM audio:
QEMU is run as root(which is bad, but i'm too lazy to figure out all the permissions..i should really do this one day), i turned on qemu's VNC audio extension, and SELinux prevents that run-as-root process to access user's pulseaudio. Which is cool.
Setting SELinux mode to permissive(which is bad too) doesn't help too, as root isn't allowed to access user's pulse by something else.
Since i don't understand either SELinux or PulseAudio(when i set up my gentoo system, i didn't use SELinux and pulse audio simply didn't exist), it's easier for me to plug a dedicated sound device into my machine and passthrough it.

BUT!
There are GPU HDMI Audio devices! Does anybody know how do i fetch audio from them?
Do i really-really-really need to use some TV screen with HDMI Audio capability(since it's digital)? Or, maybe, there's some way of transferring audio through DVI, but my monitor doesn't support that?
Maybe there's some weird device which gets only HDMI's audio signal(like video capture device, only for audio) and feeds it into speakers/SPDIF/line in of the sound card?
Also, i remember some older GPUs had SPDIF out on them for that purpose, and you could plug it anywhere you need, including your motherboard. Do those still exist, or "if there's HDMI, then there's no SPDIF"?

Last edited by Duelist (2015-03-23 18:14:19)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4561 2015-03-23 18:38:45

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
dRaiser wrote:

@Duelist, thank you - haven't seen that before. Great resource!

While most of the data is okay, some setups may be semi-working(like my entry). If you're in doubt - you can search that user's posts in this thread using google site: search.

Well, I'm familiar with Google site search, thank you. wink

I've added my setup too. It's working very nicely, only it's a little outdated since Radeon 4850 is mostly good only for Blizzard games nowadays.

Offline

#4562 2015-03-23 18:46:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

BUT!
There are GPU HDMI Audio devices! Does anybody know how do i fetch audio from them?
Do i really-really-really need to use some TV screen with HDMI Audio capability(since it's digital)? Or, maybe, there's some way of transferring audio through DVI, but my monitor doesn't support that?
Maybe there's some weird device which gets only HDMI's audio signal(like video capture device, only for audio) and feeds it into speakers/SPDIF/line in of the sound card?
Also, i remember some older GPUs had SPDIF out on them for that purpose, and you could plug it anywhere you need, including your motherboard. Do those still exist, or "if there's HDMI, then there's no SPDIF"?

Using a "TV" is the easiest way to get audio out of HDMI.  If you're using nvidia, be sure to check my blog post about using MSI interrupts.  Nvidia audio works way better with MSI interrupts.  You can also do audio over DVI.  My HD8570 only has DP and DVI connections, my DP->HDMI adapter broke so I temporarily used a DVI->HDMI adapter and was quite surprised when I still had audio.  There are splitters and some (most?) receivers with HDMI input will allow you to channel the audio somewhere other than the TV.  HDCP copy protection comes into play when you start splitting the output, but that's only an issue if you're wanting to send protected content out to the screen.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4563 2015-03-23 20:26:06

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have no issues with my R9 270 I have A88X-pro asus motherboard and options vfio_iommu_type1 disable_hugepages=1 It works great. I can shutdown and restart guest without any issues.
Windows allows to select audio output over hdmi this will work with tv or lcd monitor with speakers. I guess you can use converter to extract audio out of hdmi. Other option is to use usb sound card.

Offline

#4564 2015-03-23 20:37:49

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

lcd monitor with speakers.

Some screens do not support audio over DVI, but have speakers. Usually, they have 3.5mm line in on their back.
---

Alright, i've got a problem..

Error starting domain: internal error: early end of file from monitor: possible problem: 2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy 2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7 2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7 2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed. 2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1388, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 999, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: early end of file from monitor: possible problem: 2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy 2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7 2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7 2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed. 2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized

Device or resource busy. But..

03:05.0 Multimedia audio controller: Device 1234:1880 (rev 02) Subsystem: Device 1234:2000 Flags: fast devsel, IRQ 1 I/O ports at <ignored> [disabled] Kernel driver in use: vfio-pci
cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1", "/dev/vfio/2", "/dev/vfio/7" ]

And nothing in dmesg.

I bet this is something obvious, but i just can't see it. Any hints where to look?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4565 2015-03-23 20:41:48

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you load modules ? I had seen this error when pci-stub was not loaded at boot.

Duelist wrote:
tritron4 wrote:

lcd monitor with speakers.

Some screens do not support audio over DVI, but have speakers. Usually, they have 3.5mm line in on their back.
---

Alright, i've got a problem..

Error starting domain: internal error: early end of file from monitor: possible problem: 2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy 2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7 2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7 2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed. 2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1388, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 999, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: early end of file from monitor: possible problem: 2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy 2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7 2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7 2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed. 2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized

Device or resource busy. But..

03:05.0 Multimedia audio controller: Device 1234:1880 (rev 02) Subsystem: Device 1234:2000 Flags: fast devsel, IRQ 1 I/O ports at <ignored> [disabled] Kernel driver in use: vfio-pci
cgroup_device_acl = [ "/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1", "/dev/vfio/2", "/dev/vfio/7" ]

And nothing in dmesg.

I bet this is something obvious, but i just can't see it. Any hints where to look?

Offline

#4566 2015-03-23 20:45:08

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you trying to pass through your onboard audio? When I tried doing this, my previous system with either hang or do this. I don't see any of the below in your recent previous posts:

cat /proc/cmdline
cat /etc/default/grub
lspci -nn

Might have this wrong as I don't have access to my comp at the moment.

Offline

#4567 2015-03-23 21:00:04

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That is a dedicated audio device. pci-stub shouldn't be needed, but i'll try adding it anyways.
Before VFIO-PCI driver bound the device, it wasn't bound to anything.
UPD:

BOOT_IMAGE=/vmlinuz-3.19.1-201.fc21.x86_64 root=UUID=86383473-85bd-44d7-9d15-0f15ec76da28 ro vconsole.font=latarcyrheb-sun16 nofb pci-stub.ids=1234:1880,1002:683f,1002:aab0,1002:683f,1002:683f enable_mtrr_cleanup iommu=pt acpi_enforce_resources=lax

Alright, as i thought, adding pci stub doesn't help at all.
IOMMU groups look same way as they look for devices 01:00.0 and 02:00.0 which work fine.
I wonder why there's some disabled IO range indicated..

Anyway, it's just a simple, dumb pci device! Why wouldn't it passthrough..

03:05.0 Multimedia audio controller [0401]: Device [1234:1880] (rev 02) Subsystem: Device [1234:2000] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 1 Region 0: I/O ports at <ignored> [disabled] Kernel driver in use: vfio-pci

If i recall correctly, it's Sound Blaster 16 PCI with rebadged ensoniq chip, made by creative. It's so old, that stock fedora kernel doesn't have drivers for it, however, it seemed to be AC'97-compliant.

00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Bus: primary=00, secondary=03, subordinate=03, sec-latency=64 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: fff00000-000fffff Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR+ BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

Upstream PCI bridge seems fine to me too.

Last edited by Duelist (2015-03-23 22:19:38)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4568 2015-03-24 02:31:44

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Read this https://lkml.org/lkml/2013/6/28/647

Did you try this ? options vfio_iommu_type1 disable_hugepages=1

http://spica-and-roid.blogspot.com.au/2 … rough.html
7. Necessary permission modifications [3]
According to my experience, you might encounter an error saying:
Device 'pci-assign' could not be initialized.

Before you try any solution, first make sure that modules such as kvm, kvm_intel, pci_stub are loaded. Also make sure that intel_iommu=on is in your grub configuration file (even if you see IOMMU from dmesg, it doesn't ensure that INTEL_IOMMU is enabled).

The command that fixed it was:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Last edited by tritron4 (2015-03-24 02:42:27)

Offline

#4569 2015-03-24 09:21:34

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

Read this https://lkml.org/lkml/2013/6/28/647

Did you try this ? options vfio_iommu_type1 disable_hugepages=1

http://spica-and-roid.blogspot.com.au/2 … rough.html
7. Necessary permission modifications [3]
According to my experience, you might encounter an error saying:
Device 'pci-assign' could not be initialized.

Before you try any solution, first make sure that modules such as kvm, kvm_intel, pci_stub are loaded. Also make sure that intel_iommu=on is in your grub configuration file (even if you see IOMMU from dmesg, it doesn't ensure that INTEL_IOMMU is enabled).

The command that fixed it was:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

I appreciate your help, but...
IOMMU is 100% online, since devices 01:00.0 and 02:00.0 are getting passed-through fine.
Hugepages aren't used(but i might try forcefully disabling them). and were disabled a long time ago.
QEMU is run as root, clear_emulator_capabilities are set to zero. ACS is working fine, the device is isolated in it's IOMMU group, and since this is an AMD system - there shouldn't be any problems with ACS.
But the idea of rebinding the device is interesting, i might give it a try.

Rebinding a device gives weird issues:

[ 821.585416] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.585427] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.594894] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.594903] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.598754] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.598762] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.602439] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.602446] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.604906] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.604914] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff]

I'm fine with that message, but it's repeated five times. Something is quite not right here.

UPD:

[ 71.303135] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 71.334630] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) [ 71.356269] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 [ 71.385082] pci-stub 0000:03:05.0: enabling device (0000 -> 0001) [ 71.638238] virbr0: port 2(vnet0) entered learning state [ 71.868569] pci-stub 0000:03:05.0: kvm assign device [ 73.644474] virbr0: topology change detected, propagating [ 73.644526] virbr0: port 2(vnet0) entered forwarding state [ 74.983914] kvm: zapping shadow pages for mmio generation wraparound

Changed VFIO to KVM pci-assign:

 <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x03' slot='0x05' function='0x0'/> </source> <alias name='hostdev4'/> <rom bar='on'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev>

And the device is passed-through, showing up in VM's device manager.
It looks bound to pci-stub from host's perspective, however, dmesg clearly shows that it's fine:

03:05.0 Multimedia audio controller [0401]: Device [1234:1880] (rev 02) Subsystem: Device [1234:2000] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 20 Region 0: I/O ports at 1000 [disabled] [size=128] Kernel driver in use: pci-stub

Last edited by Duelist (2015-03-24 11:05:01)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4570 2015-03-24 12:23:04

syshack
Member
Registered: 2015-03-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:
erganzi wrote:

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.


OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

Can u share your success experience with me?

Offline

#4571 2015-03-24 13:06:27

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I didn't even notice my kernel got updated from 3.18.x to 3.19.1. Thanks redhat folks, everything still works fine.
Some pages ago people asked about 3.19..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4572 2015-03-24 13:10:21

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So does it work than ?

Duelist wrote:
tritron4 wrote:

Read this https://lkml.org/lkml/2013/6/28/647

Did you try this ? options vfio_iommu_type1 disable_hugepages=1

http://spica-and-roid.blogspot.com.au/2 … rough.html
7. Necessary permission modifications [3]
According to my experience, you might encounter an error saying:
Device 'pci-assign' could not be initialized.

Before you try any solution, first make sure that modules such as kvm, kvm_intel, pci_stub are loaded. Also make sure that intel_iommu=on is in your grub configuration file (even if you see IOMMU from dmesg, it doesn't ensure that INTEL_IOMMU is enabled).

The command that fixed it was:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

I appreciate your help, but...
IOMMU is 100% online, since devices 01:00.0 and 02:00.0 are getting passed-through fine.
Hugepages aren't used(but i might try forcefully disabling them). and were disabled a long time ago.
QEMU is run as root, clear_emulator_capabilities are set to zero. ACS is working fine, the device is isolated in it's IOMMU group, and since this is an AMD system - there shouldn't be any problems with ACS.
But the idea of rebinding the device is interesting, i might give it a try.

Rebinding a device gives weird issues:

[ 821.585416] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.585427] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.594894] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.594903] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.598754] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.598762] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.602439] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.602446] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff] [ 821.604906] pci 0000:00:14.4: PCI bridge to [bus 03] [ 821.604914] pci 0000:00:14.4: bridge window [io 0x1000-0x1fff]

I'm fine with that message, but it's repeated five times. Something is quite not right here.

UPD:

[ 71.303135] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 71.334630] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003) [ 71.356269] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270 [ 71.385082] pci-stub 0000:03:05.0: enabling device (0000 -> 0001) [ 71.638238] virbr0: port 2(vnet0) entered learning state [ 71.868569] pci-stub 0000:03:05.0: kvm assign device [ 73.644474] virbr0: topology change detected, propagating [ 73.644526] virbr0: port 2(vnet0) entered forwarding state [ 74.983914] kvm: zapping shadow pages for mmio generation wraparound

Changed VFIO to KVM pci-assign:

 <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x03' slot='0x05' function='0x0'/> </source> <alias name='hostdev4'/> <rom bar='on'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev>

And the device is passed-through, showing up in VM's device manager.
It looks bound to pci-stub from host's perspective, however, dmesg clearly shows that it's fine:

03:05.0 Multimedia audio controller [0401]: Device [1234:1880] (rev 02) Subsystem: Device [1234:2000] Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 20 Region 0: I/O ports at 1000 [disabled] [size=128] Kernel driver in use: pci-stub

Offline

#4573 2015-03-24 13:16:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syshack wrote:
erganzi wrote:

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

Can u share your success experience with me?

RHEL7 supports GPU assignment of K-series Quadro (2000 and up), GRID, and Telsa.  This uses the non-VGA, secondary GPU model.  Configuration is documented here:

http://documentation-devel.engineering. … e-GPU.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4574 2015-03-24 13:54:04

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

asd651651 wrote:

hi,
I can pass through GPU successfully but after shutdown the VM and restart the VM, the host system stuck. Does anyone have that problem too?
It seems like the problem only occurs when passing through GPU, there is no such problem when passing through the USB controller.

Here’s my system environment:
    Kernel version: 3.18.8 (patched with linux-mainline)
    GPU: GTX 295
    Qemu: 2.2.0

Yes, I too experience this and have not found a solution.

Offline

#4575 2015-03-24 13:56:21

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
asd651651 wrote:

hi,
I can pass through GPU successfully but after shutdown the VM and restart the VM, the host system stuck. Does anyone have that problem too?
It seems like the problem only occurs when passing through GPU, there is no such problem when passing through the USB controller.

Here’s my system environment:
    Kernel version: 3.18.8 (patched with linux-mainline)
    GPU: GTX 295
    Qemu: 2.2.0

Yes, I too experience this and have not found a solution.

Any error messages in log ?

Offline

#4576 2015-03-24 13:57:35

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

Yes, I suffer from the booting problem too...
When I use the VFIO GPU passthrough in a VM, shutdown the VM, but I cannot boot the VM unless I restart my host system.
Does anyone know this problem?

Yes. Same issue here with an R9 280. Windows 7 VM works perfectly fine on 1st run, 2nd run locks up the Host system to the point where it needs a hard reboot.

I believe it to be a problem with AMD cards PCI bus reset on guest reboot. Please let me know if you find a solution.

Offline

#4577 2015-03-24 14:01:16

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
hotfunction wrote:

Yes, I suffer from the booting problem too...
When I use the VFIO GPU passthrough in a VM, shutdown the VM, but I cannot boot the VM unless I restart my host system.
Does anyone know this problem?

Yes. Same issue here with an R9 280. Windows 7 VM works perfectly fine on 1st run, 2nd run locks up the Host system to the point where it needs a hard reboot.

I believe it to be a problem with AMD cards PCI bus reset on guest reboot. Please let me know if you find a solution.

What cpus are you guys are runinng this on ? Intel or amd? I have A88X-PRO with amd cpu and it all works fine. The other poster has nvidia card so I dont think nvidia or ati are responsible.
I just shutdown and restarted both windows 10 and win 8.1 10 times no issues. I am using ovmf also. I have cople amd vidoe cards.

Last edited by tritron4 (2015-03-24 14:27:47)

Offline

#4578 2015-03-24 14:07:20

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone tried to install ubuntu 14.10 under kvm ? Any attempts to install it or try it shutdown guest

Offline

#4579 2015-03-24 14:11:10

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4580 2015-03-24 14:23:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Same version, thanks.  I thought I was being smart by trying to view it in incognito mode... but I was still on the vpn.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4581 2015-03-24 14:39:41

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:
Tyrewt wrote:
hotfunction wrote:

Yes, I suffer from the booting problem too...
When I use the VFIO GPU passthrough in a VM, shutdown the VM, but I cannot boot the VM unless I restart my host system.
Does anyone know this problem?

Yes. Same issue here with an R9 280. Windows 7 VM works perfectly fine on 1st run, 2nd run locks up the Host system to the point where it needs a hard reboot.

I believe it to be a problem with AMD cards PCI bus reset on guest reboot. Please let me know if you find a solution.

What cpus are you guys are runinng this on ? Intel or amd? I have A88X-PRO with amd cpu and it all works fine. The other poster has nvidia card so I dont think nvidia or ati are responsible.
I just shutdown and restarted both windows 10 and win 8.1 10 times no issues. I am using ovmf also. I have cople amd vidoe cards.

Intel i7 4790. Can replicate the issue every time on 2nd boot of Win7 guest.

Offline

#4582 2015-03-24 14:58:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, seems like there's no drivers for my ES1371 card for win7 x64. There are drivers in the linux kernel, but they aren't included in fedora21, so...

Did anybody try using NetJACK as a mean of transferring audio from guest to host? JACK is available for windows..

Last edited by Duelist (2015-03-24 15:12:17)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4583 2015-03-24 15:01:07

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Well, seems like there's no drivers for my ES1371 card for win7 x64. There's drivers in the linux kernel, but they aren't included in fedora21, so...

Did anybody try using NetJACK as a mean of transferring audio from guest to host? JACK is available for windows..

What about this driver http://www.driverscape.com/download/cre … -%28wdm%29

Offline

#4584 2015-03-24 15:09:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:
Duelist wrote:

Well, seems like there's no drivers for my ES1371 card for win7 x64. There's drivers in the linux kernel, but they aren't included in fedora21, so...

Did anybody try using NetJACK as a mean of transferring audio from guest to host? JACK is available for windows..

What about this driver http://www.driverscape.com/download/cre … -%28wdm%29

Dude, now that's offensive. I can definitely use google. There is no drivers for x64 version of windows 7. There's for xp x64, for xp and win7 x86.
Please, stop.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4585 2015-03-24 15:15:25

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
tritron4 wrote:
Duelist wrote:

Well, seems like there's no drivers for my ES1371 card for win7 x64. There's drivers in the linux kernel, but they aren't included in fedora21, so...

Did anybody try using NetJACK as a mean of transferring audio from guest to host? JACK is available for windows..

What about this driver http://www.driverscape.com/download/cre … -%28wdm%29

Dude, now that's offensive. I can definitely use google. There is no drivers for x64 version of windows 7. There's for xp x64, for xp and win7 x86.
Please, stop.

That is strange the download page clearly states Supported OS: Windows XP, Windows Vista 32 & 64bit, Windows 7 32 & 64bit, Windows 8 32 & 64bit. I have no experience with this card so that why I am posting the link I dont mean any disrespect.

You can buy this sound card
http://www.ebay.com/itm/Creative-Sound- … 27fbf1bea2

Last edited by tritron4 (2015-03-24 15:45:49)

Offline

#4586 2015-03-24 17:58:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:
Duelist wrote:
tritron4 wrote:

Dude, now that's offensive. I can definitely use google. There is no drivers for x64 version of windows 7. There's for xp x64, for xp and win7 x86.
Please, stop.

That is strange the download page clearly states Supported OS: Windows XP, Windows Vista 32 & 64bit, Windows 7 32 & 64bit, Windows 8 32 & 64bit. I have no experience with this card so that why I am posting the link I dont mean any disrespect.

You can buy this sound card
http://www.ebay.com/itm/Creative-Sound- … 27fbf1bea2

Well, seems like i have to take my words back.
I took that card out for proper examination, it turned out to be CB5880/CT4810 which is ES1371-based.
Reinserting the card back... changed it's PCI IDS, so it's detected properly now(but doesn't get along with vfio well):

03:05.0 Multimedia audio controller [0401]: Ensoniq 5880B [AudioPCI] [1274:5880] (rev 02) Subsystem: Ensoniq Creative Sound Blaster AudioPCI128 [1274:2000] Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort+ <MAbort+ >SERR- <PERR- INTx- Latency: 32 (3000ns min, 32000ns max) Interrupt: pin A routed to IRQ 20 Region 0: I/O ports at c000 [size=64] Capabilities: <access denied> Kernel driver in use: pci-stub Kernel modules: snd_ens1371

After a numerous tries of some generic drivers, that driver that you've linked(and i've found it before many times), which was from "VMware VMaudio (VMAUDIO) (WDM)", installed and worked.

Weird ass legacy. Hell, i thought i was going insane. That card seems to be some fakey copy, since it lacks TDA 2x6w amplifier and it has a different PCB layout than others found on the interwebs, and a different audio codec also.

The old Gentoo wisdom remains true once again: if something doesn't work quite right, try recompiling it.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4587 2015-03-24 18:13:03

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just ordered sound card on ebay so I will let you know if that works with 8.1.
I wonder if we can improve sound from qemu by using asound.conf. I found out that spice outputs almost perfect sound. 

Duelist wrote:
tritron4 wrote:
Duelist wrote:

Dude, now that's offensive. I can definitely use google. There is no drivers for x64 version of windows 7. There's for xp x64, for xp and win7 x86.
Please, stop.

That is strange the download page clearly states Supported OS: Windows XP, Windows Vista 32 & 64bit, Windows 7 32 & 64bit, Windows 8 32 & 64bit. I have no experience with this card so that why I am posting the link I dont mean any disrespect.

You can buy this sound card
http://www.ebay.com/itm/Creative-Sound- … 27fbf1bea2

Well, seems like i have to take my words back.
I took that card out for proper examination, it turned out to be CB5880/CT4810 which is ES1371-based.
Reinserting the card back... changed it's PCI IDS, so it's detected properly now(but doesn't get along with vfio well):

03:05.0 Multimedia audio controller [0401]: Ensoniq 5880B [AudioPCI] [1274:5880] (rev 02) Subsystem: Ensoniq Creative Sound Blaster AudioPCI128 [1274:2000] Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort+ <MAbort+ >SERR- <PERR- INTx- Latency: 32 (3000ns min, 32000ns max) Interrupt: pin A routed to IRQ 20 Region 0: I/O ports at c000 [size=64] Capabilities: <access denied> Kernel driver in use: pci-stub Kernel modules: snd_ens1371

After a numerous tries of some generic drivers, that driver that you've linked(and i've found it before many times), which was from "VMware VMaudio (VMAUDIO) (WDM)", installed and worked.

Weird ass legacy. Hell, i thought i was going insane. That card seems to be some fakey copy, since it lacks TDA 2x6w amplifier and it has a different PCB layout than others found on the interwebs, and a different audio codec also.

The old Gentoo wisdom remains true once again: if something doesn't work quite right, try recompiling it.

Offline

#4588 2015-03-24 21:22:00

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have no issues passing through my Sound Blaster Z to a Windows vm. Once I install the drivers, everything comes up properly. No issues what so ever.

Offline

#4589 2015-03-24 22:06:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My 2 cents... If you're looking at purchasing a sound card, please default to a pci-express card.  Not necessarily a Z, but something more like it than that old ebay card above.  Conventional PCI often has issues with broken interrupt masking (the feature wasn't even added to the spec until PCI2.3) and conventional PCI devices do not have a concept of requester IDs, which is what the IOMMU uses to distinguish devices.  Not even the ACS override patch will separate devices behind a conventional PCI bridge, nor would it work correctly if it was hacked to do so.  You may need to unload drivers to get an exclusive interrupt for the device if masking is broken.  I've tested with an old Sound Blaster (Live!) and it works, but it wouldn't be my first choice to use regularly.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4590 2015-03-25 00:06:07

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh wow, I didn't even look to see what their card was, only that its slot ID was strange. This is mine, PCI-Express, no issues:

http://www.soundblaster.com/products/so … ter-z.aspx

Last edited by The_Moves (2015-03-25 00:06:24)

Offline

#4591 2015-03-25 00:15:21

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Crashes still occurs randomly, dmesg -w doesn't show anything relevant when crash occur, debug log neigher, i'm starting to suspect that might be a hardware lockup?
Seems to occur more when using CPU intensively but still happens in normal use, Im without clue ATM

Also discovered something interesting accidentally and I need someone to confirm this, when I compiled the lastest linux-mainline 4.0 rc5.1 I forgot to enable pci stub module, so after shutting down machine Dmesg showed "released" devices loading their respective module, after reboot I discovered that pci stub was not enabled and devices where working normaly before and after running KVM...

seems that we can grab PCI-E devices runtime without pci-stub? is this really safe? I passed secondary gpu + audio, USB controller and Net adapter, and 4 devices where restored aparently... dmesg of testing:

0000:02:00.0 is the GTX970
0000:04:00.0 is the USB controller
0000:05:00.0 is the NET adapter

[ 115.651756] usbcore: registered new interface driver snd-usb-audio [ 115.687194] usb 1-14.1: current rate 33186 is different from the runtime rate 16000 [ 115.699927] usb 1-14.1: current rate 33186 is different from the runtime rate 16000 [ 115.724838] usb 1-14.1: current rate 198 is different from the runtime rate 16000 [ 115.732585] usb 1-14.1: current rate 8 is different from the runtime rate 16000 [ 115.734399] usb 1-14.1: 3:1: cannot get min/max values for control 2 (id 3) [ 300.233368] mce: [Hardware Error]: Machine check events logged --QEMU started here-- [ 464.991218] VFIO - User Level meta-driver version: 0.3 [ 465.261701] xhci_hcd 0000:04:00.0: remove, state 4 [ 465.261710] usb usb4: USB disconnect, device number 1 [ 465.261891] xhci_hcd 0000:04:00.0: USB bus 4 deregistered [ 465.261897] xhci_hcd 0000:04:00.0: remove, state 1 [ 465.261901] usb usb3: USB disconnect, device number 1 [ 465.261903] usb 3-1: USB disconnect, device number 2 [ 465.323138] xhci_hcd 0000:04:00.0: USB bus 3 deregistered [ 465.400432] usbcore: deregistering interface driver ov534 [ 465.499432] tun: Universal TUN/TAP device driver, 1.6 [ 465.499437] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 465.500357] device tap0 entered promiscuous mode [ 465.500534] hostbr0: port 1(tap0) entered listening state [ 465.500543] hostbr0: port 1(tap0) entered listening state [ 465.501220] IPv6: ADDRCONF(NETDEV_UP): hostbr0: link is not ready [ 465.543875] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable [ 467.240268] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x258 [ 467.240281] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x900 [ 480.529995] hostbr0: port 1(tap0) entered learning state [ 487.466676] kvm: zapping shadow pages for mmio generation wraparound [ 495.569983] hostbr0: topology change detected, propagating [ 495.569991] hostbr0: port 1(tap0) entered forwarding state [ 495.570023] IPv6: ADDRCONF(NETDEV_CHANGE): hostbr0: link becomes ready --Shutdown of QEMU-- [ 598.207298] hostbr0: port 1(tap0) entered disabled state [ 598.207420] device tap0 left promiscuous mode [ 598.207440] hostbr0: port 1(tap0) entered disabled state --Nouveau starts spilling-- [ 601.701150] nouveau 0000:02:00.0: fb1: nouveaufb frame buffer device [ 601.701157] [drm] Initialized nouveau 1.2.1 20120801 for 0000:02:00.0 on minor 1 [ 601.715319] snd_hda_intel 0000:02:00.1: Disabling MSI [ 601.715336] snd_hda_intel 0000:02:00.1: Handle VGA-switcheroo audio client [ 601.725718] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 601.725725] r8169 0000:05:00.0: can't disable ASPM; OS doesn't have ASPM control [ 601.725731] r8169 0000:05:00.0: enabling device (0400 -> 0403) [ 601.728245] r8169 0000:05:00.0 enp5s0: renamed from eth0 [ 601.758725] xhci_hcd 0000:04:00.0: enabling device (0400 -> 0402) [ 601.758804] xhci_hcd 0000:04:00.0: xHCI Host Controller [ 601.758811] xhci_hcd 0000:04:00.0: new USB bus registered, assigned bus number 3 --Usbs connected to controller starts popping--

Offline

#4592 2015-03-25 07:20:01

hotfunction
Member
Registered: 2015-02-05
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:
tritron4 wrote:
Tyrewt wrote:

Yes. Same issue here with an R9 280. Windows 7 VM works perfectly fine on 1st run, 2nd run locks up the Host system to the point where it needs a hard reboot.

I believe it to be a problem with AMD cards PCI bus reset on guest reboot. Please let me know if you find a solution.

What cpus are you guys are runinng this on ? Intel or amd? I have A88X-PRO with amd cpu and it all works fine. The other poster has nvidia card so I dont think nvidia or ati are responsible.
I just shutdown and restarted both windows 10 and win 8.1 10 times no issues. I am using ovmf also. I have cople amd vidoe cards.

Intel i7 4790. Can replicate the issue every time on 2nd boot of Win7 guest.


Intel i5-3470. I have tried GTX-560 and GTX-295 with Ubuntu 14.04 guest, but both fail to boot on 2nd start.

Offline

#4593 2015-03-25 07:58:57

Kingd
Member
Registered: 2014-04-04
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I followed post 1 but I am stuck with the following error message, after testing if the system works:

qemu-system-x86_64: -device vfio-pci,host=01:00.0: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=01:00.0: vfio: failed to get group 1 qemu-system-x86_64: -device vfio-pci,host=01:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0: Device 'vfio-pci' could not be initialized

My primary VGA is the build in Intel HD Graphics 4000 and I want to passthrough my PowerColor HD 7970 (UEFI).

Any suggestions? I am thankful for every hint smile

Last edited by Kingd (2015-03-25 08:00:07)

Offline

#4594 2015-03-25 09:00:34

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guys, you with 2nd boot problem, dump bios from your card with gpuz, and use it in qemu/libvirt with romfile= option.

@Kingd your iommu doesn't seem to be enabled? cat /proc/cmdline? intel_iommu=on?
or you need ACS patch, you need to bind all devices in same group aka gpu and gpu audio= same group....

Last edited by slis (2015-03-25 09:02:01)

Offline

#4595 2015-03-25 10:05:57

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

NVIDIA's drivers refuse to initialize on 2nd boot for me . Anyone having this issue ?

The card reboots fine as long as I don't install NVIDIA's drivers . As soon as I install them , rebooting results in black screen when Windows is about to show the desktop .

This happens with GT610 only , GTX770 behaves perfectly during reboots .

Microsoft Basic Display Driver is AWESOME , except for the resoluion .

Offline

#4596 2015-03-25 10:50:17

asd651651
Member
Registered: 2015-03-23
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

NVIDIA's drivers refuse to initialize on 2nd boot for me . Anyone having this issue ?

The card reboots fine as long as I don't install NVIDIA's drivers . As soon as I install them , rebooting results in black screen when Windows is about to show the desktop .

This happens with GT610 only , GTX770 behaves perfectly during reboots .

Microsoft Basic Display Driver is AWESOME , except for the resoluion .

I have that issue too. It works fine when guest not installing the nvidia driver on ubuntu 14.04, but after the guest installing the nvidia driver, i can't boot the guest again after shutting down it.

No error message output in terminal and dmesg.

Last edited by asd651651 (2015-03-25 10:51:47)

Offline

#4597 2015-03-25 12:36:50

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone notice the time is always out of sync on their Windows 7 guests?  Though configured correctly and using NTP, the time is always off by 3 hours.

Offline

#4598 2015-03-25 14:19:27

kraghot
Member
Registered: 2015-03-25
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm interested in configuring this type of setup, and I hope I'll be building a new computer soon. What parts would you recommend for this type of build?

Offline

#4599 2015-03-25 14:26:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tyrewt wrote:

Anyone notice the time is always out of sync on their Windows 7 guests?  Though configured correctly and using NTP, the time is always off by 3 hours.

This sounds like you're setting the rtc to utc (default) instead of local time


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4600 2015-03-25 14:30:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kraghot wrote:

I'm interested in configuring this type of setup, and I hope I'll be building a new computer soon. What parts would you recommend for this type of build?

https://bbs.archlinux.org/viewtopic.php … 6#p1495466


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4601 2015-03-25 14:44:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Crashes still occurs randomly, dmesg -w doesn't show anything relevant when crash occur, debug log neigher, i'm starting to suspect that might be a hardware lockup?
Seems to occur more when using CPU intensively but still happens in normal use, Im without clue ATM

Also discovered something interesting accidentally and I need someone to confirm this, when I compiled the lastest linux-mainline 4.0 rc5.1 I forgot to enable pci stub module, so after shutting down machine Dmesg showed "released" devices loading their respective module, after reboot I discovered that pci stub was not enabled and devices where working normaly before and after running KVM...

seems that we can grab PCI-E devices runtime without pci-stub? is this really safe? I passed secondary gpu + audio, USB controller and Net adapter, and 4 devices where restored aparently... dmesg of testing:

0000:02:00.0 is the GTX970
0000:04:00.0 is the USB controller
0000:05:00.0 is the NET adapter

[ 115.651756] usbcore: registered new interface driver snd-usb-audio [ 115.687194] usb 1-14.1: current rate 33186 is different from the runtime rate 16000 [ 115.699927] usb 1-14.1: current rate 33186 is different from the runtime rate 16000 [ 115.724838] usb 1-14.1: current rate 198 is different from the runtime rate 16000 [ 115.732585] usb 1-14.1: current rate 8 is different from the runtime rate 16000 [ 115.734399] usb 1-14.1: 3:1: cannot get min/max values for control 2 (id 3) [ 300.233368] mce: [Hardware Error]: Machine check events logged --QEMU started here-- [ 464.991218] VFIO - User Level meta-driver version: 0.3 [ 465.261701] xhci_hcd 0000:04:00.0: remove, state 4 [ 465.261710] usb usb4: USB disconnect, device number 1 [ 465.261891] xhci_hcd 0000:04:00.0: USB bus 4 deregistered [ 465.261897] xhci_hcd 0000:04:00.0: remove, state 1 [ 465.261901] usb usb3: USB disconnect, device number 1 [ 465.261903] usb 3-1: USB disconnect, device number 2 [ 465.323138] xhci_hcd 0000:04:00.0: USB bus 3 deregistered [ 465.400432] usbcore: deregistering interface driver ov534 [ 465.499432] tun: Universal TUN/TAP device driver, 1.6 [ 465.499437] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> [ 465.500357] device tap0 entered promiscuous mode [ 465.500534] hostbr0: port 1(tap0) entered listening state [ 465.500543] hostbr0: port 1(tap0) entered listening state [ 465.501220] IPv6: ADDRCONF(NETDEV_UP): hostbr0: link is not ready [ 465.543875] kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable [ 467.240268] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1e@0x258 [ 467.240281] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x900 [ 480.529995] hostbr0: port 1(tap0) entered learning state [ 487.466676] kvm: zapping shadow pages for mmio generation wraparound [ 495.569983] hostbr0: topology change detected, propagating [ 495.569991] hostbr0: port 1(tap0) entered forwarding state [ 495.570023] IPv6: ADDRCONF(NETDEV_CHANGE): hostbr0: link becomes ready --Shutdown of QEMU-- [ 598.207298] hostbr0: port 1(tap0) entered disabled state [ 598.207420] device tap0 left promiscuous mode [ 598.207440] hostbr0: port 1(tap0) entered disabled state --Nouveau starts spilling-- [ 601.701150] nouveau 0000:02:00.0: fb1: nouveaufb frame buffer device [ 601.701157] [drm] Initialized nouveau 1.2.1 20120801 for 0000:02:00.0 on minor 1 [ 601.715319] snd_hda_intel 0000:02:00.1: Disabling MSI [ 601.715336] snd_hda_intel 0000:02:00.1: Handle VGA-switcheroo audio client [ 601.725718] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 601.725725] r8169 0000:05:00.0: can't disable ASPM; OS doesn't have ASPM control [ 601.725731] r8169 0000:05:00.0: enabling device (0400 -> 0403) [ 601.728245] r8169 0000:05:00.0 enp5s0: renamed from eth0 [ 601.758725] xhci_hcd 0000:04:00.0: enabling device (0400 -> 0402) [ 601.758804] xhci_hcd 0000:04:00.0: xHCI Host Controller [ 601.758811] xhci_hcd 0000:04:00.0: new USB bus registered, assigned bus number 3 --Usbs connected to controller starts popping--

Hmm, check your CPU for bugs. I have AMD family 15h cpu, and there's a revision guide for it saying that "hardware lockups may happen because of an interrupt" and stuff like that.

[ 300.233368] mce: [Hardware Error]: Machine check events logged

This is very, very bad. Check what ever it is. It may be a simple overheat, or something worse.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4602 2015-03-25 14:46:09

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Tyrewt wrote:

Anyone notice the time is always out of sync on their Windows 7 guests?  Though configured correctly and using NTP, the time is always off by 3 hours.

This sounds like you're setting the rtc to utc (default) instead of local time

 <clock offset='localtime'>

...why windows is so lame on this side?...


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4603 2015-03-25 14:49:27

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kraghot wrote:

I'm interested in configuring this type of setup, and I hope I'll be building a new computer soon. What parts would you recommend for this type of build?

And check first post for huge spreadsheet with stats when determining what GPUs you probably want to use. But that is a loosy list, as it doesn't reflect amounts of quirks applied on listed systems.
If you want a system that is 100% working - just copy aw's one;)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4604 2015-03-25 20:20:49

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I thought AW was using a Dell Venue 8 Pro?

Offline

#4605 2015-03-25 20:54:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

I thought AW was using a Dell Venue 8 Pro?

Heh, had to google for that one.  My hardware is nothing special, I just live within its limitations and don't try to use multiple slots off the processor root ports.  One card off the processor root ports, one card off the PCH root ports, GPUs new enough to support UEFI, and each connected to a small craigslist TV to be able to use the audio function directly.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4606 2015-03-25 21:16:04

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That would be neet to run kvm on dell venue 8 pro any one had tried
I have AMD A8-6600K APU  A88X-pro and couple of R9 270 and it work great. Thanks to

options vfio_iommu_type1 disable_hugepages=1 

Last edited by tritron4 (2015-03-25 21:16:47)

Offline

#4607 2015-03-26 03:05:17

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi aw,

I was just curious, since some PCIe root ports without ACS support but with vendor verification of disabled peer-to-peer interaction, some devices were quirked in the kernel:

static const u16 pci_quirk_intel_pch_acs_ids[] = { /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Ibexpeak PCH */ 0x3b42, 0x3b43, 0x3b44, 0x3b45, 0x3b46, 0x3b47, 0x3b48, 0x3b49, 0x3b4a, 0x3b4b, 0x3b4c, 0x3b4d, 0x3b4e, 0x3b4f, 0x3b50, 0x3b51, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Cougarpoint PCH */ 0x1c10, 0x1c11, 0x1c12, 0x1c13, 0x1c14, 0x1c15, 0x1c16, 0x1c17, 0x1c18, 0x1c19, 0x1c1a, 0x1c1b, 0x1c1c, 0x1c1d, 0x1c1e, 0x1c1f, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Pantherpoint PCH */ 0x1e10, 0x1e11, 0x1e12, 0x1e13, 0x1e14, 0x1e15, 0x1e16, 0x1e17, 0x1e18, 0x1e19, 0x1e1a, 0x1e1b, 0x1e1c, 0x1e1d, 0x1e1e, 0x1e1f, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Lynxpoint-H PCH */ 0x8c10, 0x8c11, 0x8c12, 0x8c13, 0x8c14, 0x8c15, 0x8c16, 0x8c17, 0x8c18, 0x8c19, 0x8c1a, 0x8c1b, 0x8c1c, 0x8c1d, 0x8c1e, 0x8c1f, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Lynxpoint-LP PCH */ 0x9c10, 0x9c11, 0x9c12, 0x9c13, 0x9c14, 0x9c15, 0x9c16, 0x9c17, 0x9c18, 0x9c19, 0x9c1a, 0x9c1b, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Wildcat PCH */ 0x9c90, 0x9c91, 0x9c92, 0x9c93, 0x9c94, 0x9c95, 0x9c96, 0x9c97, 0x9c98, 0x9c99, 0x9c9a, 0x9c9b, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Patsburg (X79) PCH */ 0x1d10, 0x1d12, 0x1d14, 0x1d16, 0x1d18, 0x1d1a, 0x1d1c, 0x1d1e, };

Now basically what I'm wondering, is if adding other root port PCI ids to this list would effectively be the same as applying previous ACS override kernel patches and providing override ID through kernel options? Also, would this be the same for adding the ID for the processor's own PCIe root ports, and not just PCH root ports?

I think I've posted something like this in the past, but for many Z97 motherboards, the system looks something like:
VtAr2ni.png
where the only GPU pci-express slots are provided by the processor, and none are available from the PCH, so the only option would be to force separation of the processor root port. What are the ramifications of doing this through the existing quirked list, instead of using your previous ACS patches (besides the obvious possible effects of ACS patches in general)

Last edited by mutiny (2015-03-26 03:14:09)

Offline

#4608 2015-03-26 03:36:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mutiny

The ACS override patch is just a quirk, like the code you're point to.  It just happens to be a dynamic quirk that let's you specify devices via the kernel command line.  It's certainly possible to add static entries for processor root ports, but that table is not the right place to do it.  That table triggers not only reporting the isolation of the device, but enabling of isolation.  So you wouldn't want to trigger that on anything except devices that follow the same programming model.

Have you read my blog post on IOMMU groups?  I think that gives a pretty good overview of why groups take the shape they do and the dangers of assuming isolation where there is none.  The difference between the in-kernel quirks and the command line version is that the in-kernel quirks have some assurance from the hardware vendor that there is actually isolation.  You assume all risk when you patch-in and use the override.  The mechanism by which they work is roughly the same though.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4609 2015-03-26 03:39:13

syshack
Member
Registered: 2015-03-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,aw
I followed this article step by step( https://access.redhat.com/documentation … e-GPU.html ) to passthrough Grid K2 to my Win7 vm and got code 43 after installed drivers.

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# /usr/libexec/qemu-kvm --version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-60.el7), Copyright (c) 2003-2008 Fabrice Bellard
[root@localhost ~]# lspci |grep K2
05:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)

grub:
        linux16 /boot/vmlinuz-3.10.0-123.el7.x86_64 root=UUID=aa5139f6-942e-474e-853c-3d5b2cc49125 ro vconsole.keymap=us crashkernel=auto  vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_US.UTF-8 intel_iommu=on rdblacklist=nouveau nouveau.modeset=0

[root@localhost ~]# lsmod
Module                  Size  Used by
vhost_net              33961  1
macvtap                18302  1 vhost_net
macvlan                19046  1 macvtap
tun                    27183  5 vhost_net
vfio_iommu_type1       17636  2
vfio_pci               36438  2
vfio                   20810  8 vfio_iommu_type1,vfio_pci
bnep                   19704  2
bluetooth             372662  7 bnep
rfkill                 26536  3 bluetooth
fuse                   87661  3
ipt_MASQUERADE         12880  3
iptable_nat            13011  1
nf_nat_ipv4            13263  1 iptable_nat
nf_nat                 21798  3 ipt_MASQUERADE,nf_nat_ipv4,iptable_nat
nf_conntrack_ipv4      14862  2
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
xt_conntrack           12760  1
nf_conntrack          101024  6 ipt_MASQUERADE,nf_nat,nf_nat_ipv4,xt_conntrack,iptable_nat,nf_conntrack_ipv4
ipt_REJECT             12541  2
xt_CHECKSUM            12549  1
iptable_mangle         12695  1
ip6table_filter        12815  0
ip6_tables             27025  1 ip6table_filter
iptable_filter         12810  1
ip_tables              27239  3 iptable_filter,iptable_mangle,iptable_nat
ebtable_nat            12807  0
ebtables               30913  1 ebtable_nat
bridge                110196  0
stp                    12976  1 bridge
llc                    14552  2 stp,bridge
sg                     36533  0
iTCO_wdt               13480  0
iTCO_vendor_support    13718  1 iTCO_wdt
dm_mirror              22135  0
dm_region_hash         20862  1 dm_mirror
dm_log                 18411  2 dm_region_hash,dm_mirror
dm_mod                102999  2 dm_log,dm_mirror
coretemp               13435  0
kvm_intel             138567  12
kvm                   441119  1 kvm_intel
crct10dif_pclmul       14289  0
crc32_pclmul           13113  0
crc32c_intel           22079  0
ghash_clmulni_intel    13259  0
aesni_intel            55624  0
lrw                    13286  1 aesni_intel
gf128mul               14951  1 lrw
glue_helper            13990  1 aesni_intel
ablk_helper            13597  1 aesni_intel
cryptd                 20359  3 ghash_clmulni_intel,aesni_intel,ablk_helper
serio_raw              13462  0
pcspkr                 12718  0
sb_edac                22344  0
edac_core              62330  1 sb_edac
lpc_ich                16977  0
i2c_i801               18135  0
mfd_core               13435  1 lpc_ich
mei_me                 18568  0
mei                    77872  1 mei_me
ntb                    35932  0
shpchp                 37032  0
ipmi_si                53257  0
ipmi_msghandler        45306  1 ipmi_si
mperf                  12667  0
nfsd                  284378  1
auth_rpcgss            59368  1 nfsd
nfs_acl                12837  1 nfsd
lockd                  93977  1 nfsd
sunrpc                293453  5 nfsd,auth_rpcgss,lockd,nfs_acl
uinput                 17625  0
ext4                  528957  1
mbcache                14958  1 ext4
jbd2                   98341  1 ext4
sd_mod                 45373  3
crc_t10dif             12714  1 sd_mod
crct10dif_common       12595  2 crct10dif_pclmul,crc_t10dif
usb_storage            66305  0
sr_mod                 22416  0
cdrom                  42556  1 sr_mod
ast                    60327  2
syscopyarea            12529  1 ast
sysfillrect            12701  1 ast
sysimgblt              12640  1 ast
i2c_algo_bit           13413  1 ast
drm_kms_helper         52758  1 ast
ttm                    83948  1 ast
mlx4_core             223339  0
isci                  137622  0
mpt2sas               193927  2
ahci                   25819  0
drm                   297829  4 ast,ttm,drm_kms_helper
e1000e                258529  0
libahci                32009  1 ahci
libsas                 83532  1 isci
libata                219478  3 ahci,libahci,libsas
raid_class             13554  1 mpt2sas
ptp                    18933  1 e1000e
i2c_core               40325  5 ast,drm,i2c_i801,drm_kms_helper,i2c_algo_bit
pps_core               19106  1 ptp
scsi_transport_sas     41034  3 isci,mpt2sas,libsas

Offline

#4610 2015-03-26 03:50:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syshack wrote:

hi,aw
I followed this article step by step( https://access.redhat.com/documentation … e-GPU.html ) to passthrough Grid K2 to my Win7 vm and got code 43 after installed drivers.

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# /usr/libexec/qemu-kvm --version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-60.el7), Copyright (c) 2003-2008 Fabrice Bellard
[root@localhost ~]# lspci |grep K2
05:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)

Looks like you're running RHEL, please file a bugzilla and add me to the cc - alex.williamson@redhat.com

Please include the type of hardware that you're using (Nvidia is rather particular about being on a GRID qualified system), lspci -vvv, domain xml, libvirt log for the domain (/var/log/libvirt/qemu/$DOMAIN.log), and host dmesg.  Thanks


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4611 2015-03-26 04:10:56

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@mutiny

The ACS override patch is just a quirk, like the code you're point to.  It just happens to be a dynamic quirk that let's you specify devices via the kernel command line.  It's certainly possible to add static entries for processor root ports, but that table is not the right place to do it.  That table triggers not only reporting the isolation of the device, but enabling of isolation.  So you wouldn't want to trigger that on anything except devices that follow the same programming model.

Have you read my blog post on IOMMU groups?  I think that gives a pretty good overview of why groups take the shape they do and the dangers of assuming isolation where there is none.  The difference between the in-kernel quirks and the command line version is that the in-kernel quirks have some assurance from the hardware vendor that there is actually isolation.  You assume all risk when you patch-in and use the override.  The mechanism by which they work is roughly the same though.

Thank you for the response and information aw.

I've gone ahead and read your blog post again. I do have a better understanding of the overall idea and possible risks. However, if I did want to go ahead and assume that there is some notion of isolation or at least some isolation capabilities, what would be the best approach to force the issue (even hard coding IDs specific to my system)? Mosty because I am stuck with the hardware I do have, and don't mind experimenting and testing for a while, even if there are risks.

(1) For the Z97 PCH root ports (also a Wildcat Point PCH), if I made the same assumption that this particular chipset also happens to belong to the category of PCH root ports with vendor verification, would the code I referenced be the appropriate place for that?
(2) What would be the best approach for static entries for the Core i7 processor's root port, since it probably does not follow the same model as PCH root ports? Would something like the original ACS patches be more appropriate for the processor root ports?

Thanks for the information and all that you've contributed! smile

Last edited by mutiny (2015-03-26 04:25:42)

Offline

#4612 2015-03-26 04:35:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:

I've gone ahead and read your blog post again. I do have a better understanding of the overall idea and possible risks. However, if I did want to go ahead and assume that there is some notion of isolation or at least some isolation capabilities, what would be the best approach to force the issue (even hard coding IDs specific to my system)? For the Z97 PCH root ports (also a Wildcat Point PCH), if I made the same assumption that this particular chipset also happens to belong to the category of PCH root ports with vendor verification, would the code I posted be the appropriate place for that? And then, what would be the best approach for the Core i7 processor's root port, since it probably does not follow the same model as PCH root ports? Would the original ACS patches be more appropriate for the processor root ports? Thanks for the information and all that you've contributed! smile

I don't think I'm understanding the first half of your question.  You're running on a Z97 PCH, aka Wildcat Point, which is already covered by the quirks you quoted.  Why would you need to add IDs to the table for your system for PCH root ports?  Do you have different device IDs or is something else below the root port causing undesirable grouping?  If you have something in the same family as the PCH devices we currently quirk, it's plausible to think that adding IDs to that table will work correctly, but I can't confirm it.  I thought we were already including all the major chipsets though.  If we're missing something, let me know and I can ask Intel.

For Core i5/7 or Xeon E3 processor root ports, we have no indication that there's any isolation nor programming information to enable whatever isolation the ports may provide.  Intel is not interested in providing quirks for these ports.  You could add a fixed quirk for your specific IDs, but I fail to see how that's any better than the ACS override patch and command line control.  Are you simply looking for a smaller patch that's more portable between kernel versions?  Note that the ACS override patch does allow you to specify specific vendor and device IDs if you want to be more surgical than enabling all downstream ports.

Last edited by aw (2015-03-26 04:38:37)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4613 2015-03-26 04:37:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gah, accidental quote, ignore me

Last edited by aw (2015-03-26 04:38:19)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4614 2015-03-26 05:07:07

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I don't think I'm understanding the first half of your question.  You're running on a Z97 PCH, aka Wildcat Point, which is already covered by the quirks you quoted.  Why would you need to add IDs to the table for your system for PCH root ports?  Do you have different device IDs or is something else below the root port causing undesirable grouping?  If you have something in the same family as the PCH devices we currently quirk, it's plausible to think that adding IDs to that table will work correctly, but I can't confirm it.  I thought we were already including all the major chipsets though.  If we're missing something, let me know and I can ask Intel.

For Core i5/7 or Xeon E3 processor root ports, we have no indication that there's any isolation nor programming information to enable whatever isolation the ports may provide.  Intel is not interested in providing quirks for these ports.  You could add a fixed quirk for your specific IDs, but I fail to see how that's any better than the ACS override patch and command line control.  Are you simply looking for a smaller patch that's more portable between kernel versions?  Note that the ACS override patch does allow you to specify specific vendor and device IDs if you want to be more surgical than enabling all downstream ports.

Yes, you are right, I am asking about the Z97 PCH because my particular IDs don't seem to exist in the current quirked list. The current list under Wildcat PCH in quirks.c all have IDs of 0x9cXX, whereas the PCH IDs on my motherboard are 0x8c90, 0x8c94, 0x8c96, and 0x8c9c. The motherboard is a Gigabyte GA-Z97X-UD5H-BK.

I guess I was assuming there was something in the most recent kernel versions where it would be as easy as adding the processor root ports to a list, like the PCH root ports quirks, because as I understand it, without any support from Intel I'll basically be adding these quirks/patches to each kernel udpate for the rest of time. You are completely correct that I am just looking for something simpler to modify between kernel updates. If the ACS patch still best applies in this situation, then I'll go that route for the Core i7 root ports.

Offline

#4615 2015-03-26 06:12:08

Kingd
Member
Registered: 2014-04-04
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

@Kingd your iommu doesn't seem to be enabled? cat /proc/cmdline? intel_iommu=on?
or you need ACS patch, you need to bind all devices in same group aka gpu and gpu audio= same group....

Nope IOMMU is activated.

$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-linux root=UUID=... rw quiet intel_iommu=on pci-stub.ids=1002:6798

I haven`t tried the ACS pactch, but I will give the linux-vfio kernel from the AUR a shot. Maybe that will solve my problem.

//edit: Same error message with the new linux-vfio kernel sad any suggestions?


Here ist my qemu syntax, which caused the vfio error:

qemu-system-x86_64 -enable-kvm -m 8024 -cpu host,kvm=off -smp 4 -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd -device vfio-pci,host=01:00.0 -vga none

My discrete graphics card is assigned to 01:00.0 wit the id 1002:6798.

$ lspci | grep VGA 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] $ lspci -n ... 01:00.0 0300: 1002:6798 01:00.1 0403: 1002:aaa0 ...

Last edited by Kingd (2015-03-26 08:55:24)

Offline

#4616 2015-03-26 06:16:09

syshack
Member
Registered: 2015-03-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
syshack wrote:

hi,aw
I followed this article step by step( https://access.redhat.com/documentation … e-GPU.html ) to passthrough Grid K2 to my Win7 vm and got code 43 after installed drivers.

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# /usr/libexec/qemu-kvm --version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-60.el7), Copyright (c) 2003-2008 Fabrice Bellard
[root@localhost ~]# lspci |grep K2
05:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)

Looks like you're running RHEL, please file a bugzilla and add me to the cc - alex.williamson@redhat.com

Please include the type of hardware that you're using (Nvidia is rather particular about being on a GRID qualified system), lspci -vvv, domain xml, libvirt log for the domain (/var/log/libvirt/qemu/$DOMAIN.log), and host dmesg.  Thanks

i'v filed a bugzia,pls check,thx. ( https://bugzilla.redhat.com/show_bug.cgi?id=1206006)
if need more details,let me know and i will provid as soon as possible.

Offline

#4617 2015-03-26 07:48:13

syshack
Member
Registered: 2015-03-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syshack wrote:
aw wrote:
syshack wrote:

hi,aw
I followed this article step by step( https://access.redhat.com/documentation … e-GPU.html ) to passthrough Grid K2 to my Win7 vm and got code 43 after installed drivers.

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# /usr/libexec/qemu-kvm --version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-60.el7), Copyright (c) 2003-2008 Fabrice Bellard
[root@localhost ~]# lspci |grep K2
05:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)

Looks like you're running RHEL, please file a bugzilla and add me to the cc - alex.williamson@redhat.com

Please include the type of hardware that you're using (Nvidia is rather particular about being on a GRID qualified system), lspci -vvv, domain xml, libvirt log for the domain (/var/log/libvirt/qemu/$DOMAIN.log), and host dmesg.  Thanks

i'v filed a bugzia,pls check,thx. ( https://bugzilla.redhat.com/show_bug.cgi?id=1206006)
if need more details,let me know and i will provid as soon as possible.

im sorry ,i make a stupid mistake. the fails caused by power shortage.

Offline

#4618 2015-03-26 12:59:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:
aw wrote:

I don't think I'm understanding the first half of your question.  You're running on a Z97 PCH, aka Wildcat Point, which is already covered by the quirks you quoted.  Why would you need to add IDs to the table for your system for PCH root ports?  Do you have different device IDs or is something else below the root port causing undesirable grouping?  If you have something in the same family as the PCH devices we currently quirk, it's plausible to think that adding IDs to that table will work correctly, but I can't confirm it.  I thought we were already including all the major chipsets though.  If we're missing something, let me know and I can ask Intel.

For Core i5/7 or Xeon E3 processor root ports, we have no indication that there's any isolation nor programming information to enable whatever isolation the ports may provide.  Intel is not interested in providing quirks for these ports.  You could add a fixed quirk for your specific IDs, but I fail to see how that's any better than the ACS override patch and command line control.  Are you simply looking for a smaller patch that's more portable between kernel versions?  Note that the ACS override patch does allow you to specify specific vendor and device IDs if you want to be more surgical than enabling all downstream ports.

Yes, you are right, I am asking about the Z97 PCH because my particular IDs don't seem to exist in the current quirked list. The current list under Wildcat PCH in quirks.c all have IDs of 0x9cXX, whereas the PCH IDs on my motherboard are 0x8c90, 0x8c94, 0x8c96, and 0x8c9c. The motherboard is a Gigabyte GA-Z97X-UD5H-BK.

I guess I was assuming there was something in the most recent kernel versions where it would be as easy as adding the processor root ports to a list, like the PCH root ports quirks, because as I understand it, without any support from Intel I'll basically be adding these quirks/patches to each kernel udpate for the rest of time. You are completely correct that I am just looking for something simpler to modify between kernel updates. If the ACS patch still best applies in this situation, then I'll go that route for the Core i7 root ports.

Double checking ark, Z97 is actually Lynx Point, which I thought was covered by the two sets of Lynx Point entries we have, but apparently not.  The datasheet confirms your device ID values.  I'll ask Intel about these.  Chances are good that they will follow the same programming model as existing PCH root ports and could be added to the table, but we'll need to see if Intel is willing to do the legwork to verify this.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4619 2015-03-26 13:01:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

syshack wrote:
syshack wrote:

i'v filed a bugzia,pls check,thx. ( https://bugzilla.redhat.com/show_bug.cgi?id=1206006)
if need more details,let me know and i will provid as soon as possible.

im sorry ,i make a stupid mistake. the fails caused by power shortage.

Glad you got it working


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4620 2015-03-26 14:26:59

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
[ 300.233368] mce: [Hardware Error]: Machine check events logged

This is very, very bad. Check what ever it is. It may be a simple overheat, or something worse.

Seems to occur always in 300 secs mark after boot up, I tried to read using mcelog program but Im not sure how to read the event.

Duelist wrote:

Hmm, check your CPU for bugs. I have AMD family 15h cpu, and there's a revision guide for it saying that "hardware lockups may happen because of an interrupt" and stuff like that.

My CPU is i7 5820k, an 2011 v3 socket Haswell-E processor, this seems to be the documentation, I didn't found anything too much relevant about any posible cause of these hangs
http://www.intel.com/content/dam/www/pu … -vol-1.pdf

Also I tried to use kdump mechanism for "capturing" the kernel panic but seems that computer completelly crashes, maybe even before a kernel panic actually occurs? I had a feeling that reset button was unresponsible for a sec or two

Last edited by Cubex (2015-03-26 14:28:49)

Offline

#4621 2015-03-26 14:39:40

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:
Duelist wrote:
[ 300.233368] mce: [Hardware Error]: Machine check events logged

This is very, very bad. Check what ever it is. It may be a simple overheat, or something worse.

Seems to occur always in 300 secs mark after boot up, I tried to read using mcelog program but Im not sure how to read the event.

Duelist wrote:

Hmm, check your CPU for bugs. I have AMD family 15h cpu, and there's a revision guide for it saying that "hardware lockups may happen because of an interrupt" and stuff like that.

My CPU is i7 5820k, an 2011 v3 socket Haswell-E processor, this seems to be the documentation, I didn't found anything too much relevant about any posible cause of these hangs
http://www.intel.com/content/dam/www/pu … -vol-1.pdf

Also I tried to use kdump mechanism for "capturing" the kernel panic but seems that computer completelly crashes, maybe even before a kernel panic actually occurs? I had a feeling that reset button was unresponsible for a sec or two

https://www.kernel.org/doc/Documentation/sysrq.txt

I've not tried this yet, but I plan to.

Last edited by The_Moves (2015-03-26 14:40:39)

Offline

#4622 2015-03-26 14:56:48

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have HP server and I am trying assign raid controller. It is being blocked by dma write patch. I wonder if I add my motherboard controller pci can be   added to this and i can override the error I am getting

mutiny wrote:

Hi aw,

I was just curious, since some PCIe root ports without ACS support but with vendor verification of disabled peer-to-peer interaction, some devices were quirked in the kernel:

static const u16 pci_quirk_intel_pch_acs_ids[] = { /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Ibexpeak PCH */ 0x3b42, 0x3b43, 0x3b44, 0x3b45, 0x3b46, 0x3b47, 0x3b48, 0x3b49, 0x3b4a, 0x3b4b, 0x3b4c, 0x3b4d, 0x3b4e, 0x3b4f, 0x3b50, 0x3b51, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Cougarpoint PCH */ 0x1c10, 0x1c11, 0x1c12, 0x1c13, 0x1c14, 0x1c15, 0x1c16, 0x1c17, 0x1c18, 0x1c19, 0x1c1a, 0x1c1b, 0x1c1c, 0x1c1d, 0x1c1e, 0x1c1f, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Pantherpoint PCH */ 0x1e10, 0x1e11, 0x1e12, 0x1e13, 0x1e14, 0x1e15, 0x1e16, 0x1e17, 0x1e18, 0x1e19, 0x1e1a, 0x1e1b, 0x1e1c, 0x1e1d, 0x1e1e, 0x1e1f, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Lynxpoint-H PCH */ 0x8c10, 0x8c11, 0x8c12, 0x8c13, 0x8c14, 0x8c15, 0x8c16, 0x8c17, 0x8c18, 0x8c19, 0x8c1a, 0x8c1b, 0x8c1c, 0x8c1d, 0x8c1e, 0x8c1f, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Lynxpoint-LP PCH */ 0x9c10, 0x9c11, 0x9c12, 0x9c13, 0x9c14, 0x9c15, 0x9c16, 0x9c17, 0x9c18, 0x9c19, 0x9c1a, 0x9c1b, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Wildcat PCH */ 0x9c90, 0x9c91, 0x9c92, 0x9c93, 0x9c94, 0x9c95, 0x9c96, 0x9c97, 0x9c98, 0x9c99, 0x9c9a, 0x9c9b, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Patsburg (X79) PCH */ 0x1d10, 0x1d12, 0x1d14, 0x1d16, 0x1d18, 0x1d1a, 0x1d1c, 0x1d1e, };

Now basically what I'm wondering, is if adding other root port PCI ids to this list would effectively be the same as applying previous ACS override kernel patches and providing override ID through kernel options? Also, would this be the same for adding the ID for the processor's own PCIe root ports, and not just PCH root ports?

I think I've posted something like this in the past, but for many Z97 motherboards, the system looks something like:
http://i.imgur.com/VtAr2ni.png
where the only GPU pci-express slots are provided by the processor, and none are available from the PCH, so the only option would be to force separation of the processor root port. What are the ramifications of doing this through the existing quirked list, instead of using your previous ACS patches (besides the obvious possible effects of ACS patches in general)

Offline

#4623 2015-03-26 15:02:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I have HP server and I am trying assign raid controller. It is being blocked by dma write patch. I wonder if I add my motherboard controller pci can be   added to this and i can override the error I am getting

I don't know what dma write patch you're talking about, but assigning the integrated raid controller on HP systems is just asking for trouble and will likely be prevented on newer kernels that exclude devices associated with VT-d RMRRs.  PS - please quote appropriately and avoid top-posting.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4624 2015-03-26 15:25:56

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
tritron4 wrote:

I have HP server and I am trying assign raid controller. It is being blocked by dma write patch. I wonder if I add my motherboard controller pci can be   added to this and i can override the error I am getting

I don't know what dma write patch you're talking about, but assigning the integrated raid controller on HP systems is just asking for trouble and will likely be prevented on newer kernels that exclude devices associated with VT-d RMRRs.  PS - please quote appropriately and avoid top-posting.

It is pci-e sas raid controller and  not build in  and Rmrr is preventing assignment of raid card. Is there a way to force assignment of such card ?
Everything was working fine with xen.

Offline

#4625 2015-03-26 15:55:07

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:
aw wrote:
tritron4 wrote:

I have HP server and I am trying assign raid controller. It is being blocked by dma write patch. I wonder if I add my motherboard controller pci can be   added to this and i can override the error I am getting

I don't know what dma write patch you're talking about, but assigning the integrated raid controller on HP systems is just asking for trouble and will likely be prevented on newer kernels that exclude devices associated with VT-d RMRRs.  PS - please quote appropriately and avoid top-posting.

It is pci-e sas raid controller and  not build in  and Rmrr is preventing assignment of raid card. Is there a way to force assignment of such card ?
Everything was working fine with xen.

As the error message says, contact your vendor.  The problems with RMRRs are even worse that ACS and "working fine" may just mean you were lucky.  We'll have a white paper published on this shortly, but in the meantime you can get an understanding of the problems from this thread - https://www.marc.info/?l=kvm&m=142561921430906


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4626 2015-03-26 16:17:18

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

https://www.kernel.org/doc/Documentation/sysrq.txt

I've not tried this yet, but I plan to.

I had sysrq enabled all time but does nothing at crash time

Offline

#4627 2015-03-26 16:25:30

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I get this errors

vfio-pci 0000:0a:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. [ 1561.682642] vfio-pci 0000:0a:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. [ 1561.728758] xenbr1: port 2(vnet0) entered disabled state
0a:00.0 0104: 103c:323b (rev 01) Subsystem: 103c:3352 Physical Slot: 2 Flags: fast devsel, IRQ 53 Memory at fbe00000 (64-bit, non-prefetchable) [size=1M] Memory at fbdf0000 (64-bit, non-prefetchable) [size=1K] I/O ports at 5000 [size=256] [virtual] Expansion ROM at fbd00000 [disabled] [size=512K] Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable- Count=1/32 Maskable+ 64bit+ Capabilities: [b0] MSI-X: Enable- Count=64 Masked- Capabilities: [c0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [300] #19 Kernel driver in use: vfio-pci Kernel modules: hpsa

Offline

#4628 2015-03-26 16:56:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I get this errors

vfio-pci 0000:0a:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. [ 1561.682642] vfio-pci 0000:0a:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. [ 1561.728758] xenbr1: port 2(vnet0) entered disabled state

Yep, that's the problem we're talking about.  Since this is a Smart Array controller, you're basically out of luck as I don't think HP is going to be able to provide you with a firmware update that disables RMRRs.  The problem is that the raid controller and the platform management controller communicate via back-channel DMA defined by RMRRs.  That carves out a region or IOVA space for the device that's incompatible with PCI device assignment.  We can neither ignore it nor support it and its existence implies a lack of isolation between the raid controller and the management controller.

The other question is why do you want to do this?  Block devices are generally not as latency sensitive as network devices and direct assignment has diminishing returns.  A virtio block devices is likely to provide nearly native performance for the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4629 2015-03-26 19:19:08

chinaski
Member
Registered: 2015-03-26
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

First thanks for this great guide.

I'm testing  (in Fedora 21) with a Tyan S7025 board and a GT610 card, and usb passthrough works once the install process begin, but the keyboard and mouse don't work before, so if I press F12 nothing happens and also can not select an option in the grub menu....How can I solve this?

Thank you

Last edited by chinaski (2015-03-26 19:20:01)

Offline

#4630 2015-03-26 19:58:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chinaski wrote:

Hello,

First thanks for this great guide.

I'm testing  (in Fedora 21) with a Tyan S7025 board and a GT610 card, and usb passthrough works once the install process begin, but the keyboard and mouse don't work before, so if I press F12 nothing happens and also can not select an option in the grub menu....How can I solve this?

Thank you

If you're doing just qemu commandline, i suggest you adding -device qxl just for input purposes. Also, there's a way to set up VNC connection. Maybe spice too, but i don't know for sure.
If you're doing libvirt way - you should have either spice or VNC graphics doing your input.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4631 2015-03-27 14:58:13

chinaski
Member
Registered: 2015-03-26
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
chinaski wrote:

Hello,

First thanks for this great guide.

I'm testing  (in Fedora 21) with a Tyan S7025 board and a GT610 card, and usb passthrough works once the install process begin, but the keyboard and mouse don't work before, so if I press F12 nothing happens and also can not select an option in the grub menu....How can I solve this?

Thank you

If you're doing just qemu commandline, i suggest you adding -device qxl just for input purposes. Also, there's a way to set up VNC connection. Maybe spice too, but i don't know for sure.
If you're doing libvirt way - you should have either spice or VNC graphics doing your input.


Thank you,

I'm using qemu commandline and tried adding a pci card with two usb3.0 ports. I could make it work and select f12 in bootmenu but only once, at second reboot of the host I got this now:

[  161.148722] dmar: DRHD: handling fault status reg 2
[  161.148731] dmar: DMAR:[DMA Read] Request device [0c:00.0] fault addr ee000
DMAR:[fault reason 06] PTE Read access is not set

0c:00.0 is the usb card

Strange, but only worked once Im using the vm (windows or linux). I have also tried change the card from pci slot with same result.

Offline

#4632 2015-03-27 17:54:26

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

First, just for installing & then booting a Win7 or Windows 2008 R2 guest, please refer to the following sections in the whitepaper:

  • Supported guest operating systems | Notes about Windows Server 2008 R2
  • Compatibility Support Module (CSM)

    -- strictly as background material only, you do not need to do anything with this

  • Select features | Video driver

Windows 2008 R2 is a UEFI system "in general", it's just its stock (default) video driver that has this inconvenient dependency on Int10h. So, OVMF (the "pure" build, no CSM) fakes an Int10h handler for just this OS, which is pretty raw, but enough for the user to install the guest, and then to update / install "real" video drivers that do not have this Int10h dependency (and then other resolutions and S3 etc become available.)

In short,

  1. You should create a virtual machine with two video adapters. The first should be QXL, the second should be your passthru GPU. Use the "pure UEFI" build of OVMF from Gerd's package.

  2. QXL will allow you to install Win7 / Win2008R2 in 1024x768x32.

  3. Update the guest's video driver for the QXL card from the stock windows driver (which OVMF fools, with regard to Int10h, see the material above) to the real QXL XDDM driver. Look at http://www.spice-space.org/download.html , under Guest | Windows binaries.

  4. Install the driver for your secondary (passthru) GPU.

  5. Potentially mark the secondary (passthru) GPU as primary desktop provider.

Admittedly, I did not test the "GPU passthru" part with Windows 2008 R2 (only with Windows Server 2012 R2), but the OS installation I definitely tested, and I don't see any reason why GPU passthrough (and the dual display) should not work.

Finally, an unrelated, but nonetheless important point -- I recommend that everyone use libvirt. It supports OVMF very nicely. Please refer to the following section in the whitepaper:

Installation of OVMF guests with virt-manager and virt-install

.

Hi and thanks for posting!

This may be an unrelated issue, but when I use -device qxl-vga my entire screen, other workspaces and even my console gets mangled by the color monster (16 colors maybe?). Using i3-wm.

Offline

#4633 2015-03-27 22:41:01

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ohh this is bad, long story short:

I had to build a serial cable from scratch because the motherboard serial conector was not the same pinout as the cable I bough (cable fault), install linux on a Pentium 3 box and configure to dump the console from serial

There is the log:

[ 2987.618210] PANIC: double fault, error_code: 0x0 [ 2987.619023] CPU: 10 PID: 2591 Comm: qemu-system-x86 Tainted: P O 4.0.0-1-mainline #12 [ 2987.619023] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99 Extreme6, BIOS L1.74 02/02/2015 [ 2987.619023] task: ffff8804813b14c0 ti: ffff88048107c000 task.ti: ffff88048107c000 [ 2987.619023] RIP: 0010:[<ffffffff8158460d>] [<ffffffff8158460d>] page_fault+0xd/0x30 [ 2987.619023] RSP: 0018:00007f3bd1ff8f88 EFLAGS: 00010016 [ 2987.619023] RAX: 000000000000b940 RBX: 0000000000000000 RCX: ffffffff81583277 [ 2987.619023] RDX: 000000000000b940 RSI: 0000000000000000 RDI: 00007f3bd1ff90c8 [ 2987.619023] RBP: 00007f3bd1ff9088 R08: 00007f3beeccb380 R09: 0000000000000000 [ 2987.619023] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000b940 [ 2987.619023] R13: 00007f3bd1ff90c8 R14: 0000000000000000 R15: 00007f3bee80a1c0 [ 2987.619023] FS: 00007f3bd1ffb700(0000) GS:ffff88049fd40000(0000) knlGS:0000000000000000 [ 2987.619023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2987.619023] CR2: 00007f3bd1ff8f78 CR3: 00000004813d4000 CR4: 00000000001427e0 [ 2987.619023] Stack: [ 2987.619023] BUG: unable to handle kernel paging request at 00007f3bd1ff8f88 [ 2987.619023] IP: [<ffffffff81018a73>] show_stack_log_lvl+0xf3/0x1b0 [ 2987.619023] PGD 48a3c7067 PUD 489c50067 PMD 48b961067 PTE 0 [ 2987.619023] Oops: 0000 [#1] PREEMPT SMP [ 2987.619023] Modules linked in: tun mousedev hid_logitech_hidpp hid_logitech_dj arc4 ecb md4 md5 hmac nls_utf8 cifs dns_resolver ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc cfg80211 gspca_zc3xx gspca_main videodev rfkill media coretemp hwmon snd_hda_codec_hdmi intel_rapl snd_hda_codec_realtek iosf_mbi x86_pkg_temp_thermal snd_hda_codec_generic intel_powerclamp nls_iso8859_1 nls_cp437 kvm_intel vfat nvidia(PO) snd_usb_audio kvm snd_hda_intel snd_hda_controller fuse iTCO_wdt crct10dif_pclmul mxm_wmi iTCO_vendor_support fat crc32_pclmul snd_hda_codec snd_usbmidi_lib crc32c_intel ghash_clmulni_intel e1000e snd_rawmidi aesni_intel snd_hwdep snd_seq_device evdev drm aes_x86_64 mei_me snd_pcm lrw gf128mul snd_timer mac_hid r8169 glue_helper snd sb_edac ablk_helper i2c_i801 mii cryptd ptp tpm_tis pcspkr serio_raw edac_core mei i2c_core lpc_ich soundcore shpchp tpm pps_core wmi processor button sch_fq_codel nfs lockd grace sunrpc fscache ext4 crc16 mbcache jbd2 hid_generic sr_mod cdrom uas usb_storage sd_mod usbhid hid atkbd libps2 ahci libahci libata scsi_mod xhci_pci xhci_hcd usbcore usb_common i8042 serio [ 2987.619023] CPU: 10 PID: 2591 Comm: qemu-system-x86 Tainted: P O 4.0.0-1-mainline #12 [ 2987.619023] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99 Extreme6, BIOS L1.74 02/02/2015 [ 2987.619023] task: ffff8804813b14c0 ti: ffff88048107c000 task.ti: ffff88048107c000 [ 2987.619023] RIP: 0010:[<ffffffff81018a73>] [<ffffffff81018a73>] show_stack_log_lvl+0xf3/0x1b0 [ 2987.619023] RSP: 0018:ffff88049fd44e38 EFLAGS: 00010046 [ 2987.619023] RAX: 000000000000000c RBX: 0000000000000000 RCX: ffff88049fd43fc0 [ 2987.619023] RDX: 00007f3bd1ff8f88 RSI: ffffffff817f02bc RDI: ffffffff817ff1c0 [ 2987.619023] RBP: ffff88049fd44e98 R08: ffff88049fd3ffc0 R09: 00000000000008a7 [ 2987.619023] R10: 0000000000000000 R11: 00000000000008a7 R12: ffff88049fd44f58 [ 2987.619023] R13: 0000000000000000 R14: ffffffff817c9406 R15: 0000000000000000 [ 2987.619023] FS: 00007f3bd1ffb700(0000) GS:ffff88049fd40000(0000) knlGS:0000000000000000 [ 2987.619023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2987.619023] CR2: 00007f3bd1ff8f88 CR3: 00000004813d4000 CR4: 00000000001427e0 [ 2987.619023] Stack: [ 2987.619023] ffff88049fd44e98 ffffffff8157c000 0000000000000008 ffff88049fd44ea8 [ 2987.619023] ffff88049fd44e68 00007f3bd1ff8f88 ffff88048107c000 ffff88049fd44f58 [ 2987.619023] 00007f3bd1ff8f88 0000000000000040 0000000000000000 00007f3bee80a1c0 [ 2987.619023] Call Trace: [ 2987.619023] <#DF> [ 2987.619023] [<ffffffff8157c000>] ? printk+0x55/0x6b [ 2987.619023] [<ffffffff81018bba>] show_regs+0x8a/0x250 [ 2987.619023] [<ffffffff81059007>] df_debug+0x27/0x40 [ 2987.619023] [<ffffffff810170b7>] do_double_fault+0x87/0x100 [ 2987.619023] [<ffffffff81584048>] double_fault+0x28/0x30 [ 2987.619023] [<ffffffff81583277>] ? native_iret+0x7/0x7 [ 2987.619023] [<ffffffff8158460d>] ? page_fault+0xd/0x30 [ 2987.619023] <<EOE>> [ 2987.619023] <UNK> Code: 90 06 00 00 48 89 55 c8 eb 88 66 0f 1f 44 00 00 e8 24 84 2b 00 eb af 48 39 ca 0f 84 90 00 00 00 45 85 ff 74 06 41 f6 c7 03 74 5a <48> 8b 32 48 8d 42 08 4c 89 45 b0 41 83 c7 01 48 89 45 c0 48 c7 [ 2987.619023] RIP [<ffffffff81018a73>] show_stack_log_lvl+0xf3/0x1b0 [ 2987.619023] RSP <ffff88049fd44e38> [ 2987.619023] CR2: 00007f3bd1ff8f88

Offline

#4634 2015-03-28 19:49:05

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

How did you get Win7 to even load with more than 1 CPU with EFI boot?

I'll just follow up on this part -- the rest of your comment has been addressed already, but this is indeed a different issue.

If you want Win7 + OVMF + multiple VCPUs, then you have to turn off HYPER-V features. References:

Offline

#4635 2015-03-28 20:24:44

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
lersek wrote:

After that you'll probably want something more than just 1 resolution, 1024x768x32. Normally that "more" means the QXL XDDM driver, but in this case, the passthru GPU and its drivers are just as fine, for "more".

HA! I've tried setting up windows 8 on a hardware UEFI platform, and i burst out laughing when..

(I understand that you connected the above two just for comical effect, but lest we confuse others, let's point out that the quote from me relates to Windows 7 / Windows Server 2008 R2. The QXL XDDM driver which I referenced is also for Windows 7 / Windows Server 2008 R2 exclusively. (Windows 8 has a revisited driver model called WDDM, hence the QXL guest driver for Windows 8 is called "QXL WDDM".) So, beyond the ironic association between the above two, ie. the limited graphics resolution, there's no actual technical connection. Anyway, I digress...)

Duelist wrote:

1.It boots into 1024x768x32.
2.I try to download drivers(or, at least, a browser), starting Internet exploder..
3.And it says "Screen resolution is too small, internet will not be exploded for today"
4.No drivers - no resolution change.
5.No resolution change - no drivers.

So you've got to have some CD with drivers when you try installing windows 8 or some other UEFI-compatible windowns. Or you can't download drivers.

I understand that this Windows 8 installation targeted a hardware UEFI platform, but the moral you've drawn from the experience doesn't equally apply to Windows 8 on OVMF (and it also may not apply to other hardware UEFI platforms). Namely, if you can convince your firmware to switch to a higher GOP resolution before it boots Windows 8, then Windows 8 will "inherit" that higher resolution, and it will just use the linear framebuffer like that. It won't be accelerated or anything, but a high resolution may certainly be available.

And OVMF happens to support this. Just use QXL, give it enough video RAM (the default video RAM size should be fine for most users), and then go to Device Manager | OVMF Platform Configuration in the firmware. Select your preferred resolution, and reboot. (You can reboot for example under Boot Maintenance Manager | Reset System.) On next boot, the setup TUI resolution won't change, but the OVMF splash screen and the guest OS's inherited resolution should reflect your preference. Then you can start Internet Explorer, and type the magical first URL into its URL bar. (Which is getfirefox.com, obviously :))

This is documented in the whitepaper under: (a) "Notes about Windows Server 2012 and Windows 8", and (b) "Platform Driver".

Offline

#4636 2015-03-28 20:53:54

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tritron4 wrote:

I am running into issue where efi-e1000.rom is not found. Ho to tell libvirt proper location ? I have the files in /usr/share/qemu

The efi-e1000.rom file should come from the qemu installation (qemu bundles some iPXE binaries), and should reside under a path that qemu automatically finds.

That said, you won't have much luck netbooting VMs with iPXE oproms under OVMF at the moment. Please see the following messages / threads:

My personal opinion is that you should

  • either apply those two patches to iPXE and rebuild the oproms (hard),

  • or set the NIC model of your VM to virtio-net, and disable the external oprom for the NIC (easy).

You can disable oprom loading in QEMU with

-device virtio-net-pci,netdev=...,romfile=

(yes, the equal sign is the last character of the option). The same can be done in the libvirt domain XML with

 <interface type='network'> <model type='virtio'/> <rom file=''/> </interface>

This will cause QEMU not to load the iPXE oprom into the virtio-net NIC, which in turn will allow OVMF's builtin virtio-net driver to bind the NIC. (This is documented in OvmfPkg/README.) Then PXE boot should "just work", assuming you have an otherwise correct PXE / DHCP setup on the guest's network.

(If you're interested, please refer to bug 1181980 in the RH bugzilla, and its duplicates.)

This method does necessitate installing the virtio-net driver in the guest OS as well, but that shouldn't be a problem.

Offline

#4637 2015-03-29 12:09:55

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Forum,
This is my first posting and before I get to the purpose of itself I sadly have to start with an excuse sad
I've tried for the last ~30 hours to get this configuration working and I failed terrible ... Even thogh I've read and kept reading, I'm at a point where all is messed up in my head. The questions I'm asking are probably most, if not all answered in this monster of a thread, but where is the function to search through it? Either I'm blind, or I have a reason to feel as stupid as I do right now hmm

My Hardware in use:
Asus Sabertooth Z87
Intel I7 4770K
Intel HD4600 IGM (For the Arch host)
Nvidia GTX 970 (For the Windows guest)
16GB RAM
Intel PRO/1000 PT (82571EB) Dualport

Yesterday I failed at the vfio point with some Error -22 stating the recource could not be found, but I'm not even sure anymore what I actually need ... and thats why I'm here:

Even though I think my Hardware is sufficient. Are the components able to run this configuration?
What patches, modules and settings vor qemu do I really need on my system with the stated HW?
Do I have to install both vfio and ovmf?

Those are prob. dumb questions, I know. I'm desperate!

Best regards,
Beer Belly

P.S. I almost forgot the most Important Part: Thank you for creating this Guide and all your contributed time!

P.P.S. After going outside for a smoke to clear my head a little I think I've found the wall I'm banging my head against ... the i74770K only supports VT-x, not VT-d?!
Pls correct me if I'm wrong...

Last edited by bierbauch (2015-03-29 12:47:20)

Offline

#4638 2015-03-29 12:44:52

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Pls delete this Posting!

Last edited by bierbauch (2015-03-29 12:45:32)

Offline

#4639 2015-03-29 14:03:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bierbauch wrote:

P.P.S. After going outside for a smoke to clear my head a little I think I've found the wall I'm banging my head against ... the i74770K only supports VT-x, not VT-d?!
Pls correct me if I'm wrong...

Correct, your CPU isn't going to work, no VT-d - http://ark.intel.com/products/75123/Int … o-3_90-GHz


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4640 2015-03-29 14:23:37

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thx for the verification mate,
Just checked my "closet" and only found a E3 1220L which is most likely nothing i want to use in this scenario :'(.
I guess I'll buy myself a 4790k then ...

Offline

#4641 2015-03-29 16:34:42

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again,

currently I'm trying to create a windows 7 UEFI VM. Unfortunately the installation process seems to hang at the very beginning when the screen only shows "Starting Windows".
I think the installation image should be okay, since I successfully used it for a seabios windows 7 vm.

Can someone can give me some hint, what could go wrong?

My quemu script looks as follows (no q35! smile ):

taskset 0x0000003F \ qemu-system-x86_64 \ -name win7uefi \ -enable-kvm \ -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -m 4096 \ -mem-path /dev/hugepages \ -mem-prealloc \ -balloon none \ -k de \ -drive if=pflash,format=raw,readonly,file=ovmf/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=ovmf/ovmf_vars_x64.bin \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=vmdata/win7uefi.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -drive file=vmdata/de_win_7_prosp1_x64.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=vmdata/virtio_drv.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -spice port=5930,disable-ticketing \ -device virtio-serial-pci \ -device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \ -chardev spicevmc,id=spicechannel0,name=vdagent \ -rtc base=utc

Thanks in advance!

Last edited by apex8 (2015-03-29 16:50:21)

Offline

#4642 2015-03-29 16:51:39

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
-device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \

Thanks in advance!

Read back a few pages (181 i think), you need to first install the OS on the normal vga qxl without the gpu being used. then install the qxl driver, then poweroff and add the gpu with vfio-pci.
PS I didnt need -vga none, that actually breaks my screen. I just dont use a -vga setting at all.

Offline

#4643 2015-03-29 17:16:24

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks Naruni, I can process the installation now. Maybe this hint about QXL is worth to added on the first post?

Offline

#4644 2015-03-29 17:48:03

Draizer
Member
Registered: 2015-03-17
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I finally got everything to work, however I'm running into a small issue : for some reason, the screen resolution is somewhat borked on my VM. While it's still in the BIOS, it's "true" 1920x1080, takes up my whole screen, as it should. However, once it boots onto Windows 8, for some reason, the screen shrinks (I end up with black bars on all 4 sides of the VM screen, I guess around an inch wide). Under W8 the resolution is marked as 1920x1080. Drivers are installed for my card as well.

I've looked it up but found nothing on this, did anyone else here run into that particular issue?

Offline

#4645 2015-03-29 17:48:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Thanks Naruni, I can process the installation now. Maybe this hint about QXL is worth to added on the first post?

It's documented in the OVMF whitepaper linked on the OP

Offline

#4646 2015-03-29 18:48:05

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Draizer wrote:

Hello,

I finally got everything to work, however I'm running into a small issue : for some reason, the screen resolution is somewhat borked on my VM. While it's still in the BIOS, it's "true" 1920x1080, takes up my whole screen, as it should. However, once it boots onto Windows 8, for some reason, the screen shrinks (I end up with black bars on all 4 sides of the VM screen, I guess around an inch wide). Under W8 the resolution is marked as 1920x1080. Drivers are installed for my card as well.

I've looked it up but found nothing on this, did anyone else here run into that particular issue?

I've seen this... on a hardware platform with win7 and win8 both. That hardware platform didn't even support UEFI at all!
I was using AMD HD7750 GPU on 27" fullhd screen, it was 1920x1080 and it was pixel-perfect, BUT there were black stripes all around.
Try bothering the driver's configuration panel and screen's menu. I seriously have no idea why this happens.
Maybe the screen has some weird PPI?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4647 2015-03-29 19:07:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bierbauch wrote:

P.P.S. After going outside for a smoke to clear my head a little I think I've found the wall I'm banging my head against ... the i74770K only supports VT-x, not VT-d?!
Pls correct me if I'm wrong...

Correct, your CPU isn't going to work, no VT-d - http://ark.intel.com/products/75123/Int … o-3_90-GHz

You know what's hilarious?
4770   supports vt-d.
4770S supports vt-d
4770HD supports vt-d
4770R supports vt-d
4770T supports vt-d
Same thing with 3770 series, only 3770K doesn't support vt-d.
Moreover, there's Xeons with vt-d based on 4770 series.

Unlocked multiplier? More like disabled VT-d.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4648 2015-03-29 23:42:21

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Haswell-E has overclock and VT-D support, for example the 5820K Im using

Offline

#4649 2015-03-30 02:46:32

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Here's an interesting thing.  I ran the ovmf script from page 1 with the edk2.git ovmf from lersek's initial post p180-1? for win7 + Uefi + ovmf.  Turns out that it works with 440fx ( with qxl and removing the hyper-v stuff) but hangs with a black screen with q35.  I've had a working q35 and 440fx using the seabios/x-vga non uefi setup. I'm passing a quadro k2000. The other weird thing is that although I put -smp 6, the 440fx/ovmf/Uefi version reports only two processors. I'd rather use q35 because some usb devices only work with q35 and not with 440fx, eg thumb drives.

Offline

#4650 2015-03-30 02:50:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Here's an interesting thing.  I ran the ovmf script from page 1 with the edk2.git ovmf from lersek's initial post p180-1? for win7 + Uefi + ovmf.  Turns out that it works with 440fx ( with qxl and removing the hyper-v stuff) but hangs with a black screen with q35.  I've had a working q35 and 440fx using the seabios/x-vga non uefi setup. I'm passing a quadro k2000. The other weird thing is that although I put -smp 6, the 440fx/ovmf/Uefi version reports only two processors. I'd rather use q35 because some usb devices only work with q35 and not with 440fx, eg thumb drives.

Try using -device nec-usb-xhci for usb

Offline

#4651 2015-03-30 03:08:50

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried that, window does BSOD with error 7b, remove the device and it works again.

Offline

#4652 2015-03-30 03:20:31

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Tried that, window does BSOD with error 7b, remove the device and it works again.

this one? https://msdn.microsoft.com/en-us/librar … 85%29.aspx, weird, works for me on 8.1

-device nec-usb-xhci,id=xhci -device usb-host,vendorid=0x046e,productid=0x530a,bus=xhci.0

Last edited by nbhs (2015-03-30 03:21:12)

Offline

#4653 2015-03-30 07:44:15

Chaohong Hu
Member
Registered: 2014-12-29
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is that possible to passthrough graphic card on a laptop? My laptop has a intel igd and a nvidia card.

If the answer is tricky or just impossible, please tell me why.

If some special laptops support passthrough, what is the features?

Offline

#4654 2015-03-30 08:46:19

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Chaohong Hu wrote:

Is that possible to passthrough graphic card on a laptop? My laptop has a intel igd and a nvidia card.

If the answer is tricky or just impossible, please tell me why.

If some special laptops support passthrough, what is the features?

Interessting Question. Speaking from my experience gathered last Weekend with a desktop workstation I would say: Maybe!

It should be possible in theory but 3 things must allign for that to happen:
Intel CPU supports VT-d (i.e. i7 4500U)
Chipset supports VT-d (i.e. QM87)
and probably the most troubling:
The provided BIOS must support that option too ...

I would imagine 2 points of that list troublesome:
I pretty much never see what chipset is used in the product description.
The BIOS is pretty much never mentioned or looked at in tests and reviews, so without thinking too much about it I would say this could be a frustrating try and error phase ...

But this is just me talking. Have no practical experience. Never tried ...

Offline

#4655 2015-03-30 11:48:53

tholin
Member
Registered: 2015-03-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm thinking about upgrading my Radeon HD 6770 to something more powerful. I don't like nvidia so the cards I'm considering with good price performance ratio are the following.
Radeon R9 270X
Radeon R9 285
Radeon R9 280
Any pros or cons to using any of them? Can anyone confirm that they work? What about the reset problem?

A new patch went into kernel 4.0 to work around the reset problem but it requires a switch between the cards and the root hub. My 16x lanes connect to some PLX PEX 8747 switches before the CPU root port so reset shouldn't be a problem for me?

I see that Powercolor, MSI and Sapphire got official UEFI GOP support lists for their cards but what about the other manufacturers? I can't find any lists, just speculations on forums.

Offline

#4656 2015-03-30 13:04:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tholin wrote:

I'm thinking about upgrading my Radeon HD 6770 to something more powerful. I don't like nvidia so the cards I'm considering with good price performance ratio are the following.
Radeon R9 270X
Radeon R9 285
Radeon R9 280
Any pros or cons to using any of them? Can anyone confirm that they work? What about the reset problem?

A new patch went into kernel 4.0 to work around the reset problem but it requires a switch between the cards and the root hub. My 16x lanes connect to some PLX PEX 8747 switches before the CPU root port so reset shouldn't be a problem for me?

I see that Powercolor, MSI and Sapphire got official UEFI GOP support lists for their cards but what about the other manufacturers? I can't find any lists, just speculations on forums.

I've heard that R9 285 is based on a new chip(so it's not rebadged 7xxx series card) and shouldn't have reset problems.
Aaand try preferring MSI or Powercolor or somebody else, because saphhire have some problems with UEFI too. Basically, if your card doesn't support UEFI natively, out of the box, then you have high chances of not getting it to work.

For stats you can check out that huge google spreadsheet from the op-post, but remember the "old" names of rebadged cards(i.e. 270x is 7970 if i recall properly).

Chaohong Hu wrote:

Is that possible to passthrough graphic card on a laptop? My laptop has a intel igd and a nvidia card.

If the answer is tricky or just impossible, please tell me why.

If some special laptops support passthrough, what is the features?

Well, some notebooks(i've stumbled with lenovo myself) may have weird nvidia optimus technology, which broke everything up. The laptop that i've observed had unknown header 7f in lspci when the nvidia card is offline, and NoSoftRst+ when the card is online. Hell, that machine even couldn't go to STR and back without having that card broken. I highly doubt that VFIO will work on that setup.

Second problem is video output! As you might know, all video outputs, including the screen itself, are switched between intel and discrete gpu. How does that switch works - almost nobody knows, because it is a highly proprietary stuff. Ask guys with bumblebee, there are problems of getting it to work just like it's supposed to work. This problem applies to intel-amd(WORST COMBINATION EVER, hello rare hybrid drivers) and amd-amd notebooks too.

You can construct or buy and external pci-e 16x slot in kind of dock-station way, connecting it somewhere like pcmcia or inside to mini-pci-e or somewhere else, and plug a desktop GPU. That way it should work good, but you'll need too connect an external screen to it, or try fiddling with nvidia optimus(or some alike from AMD).


Conclusion: I wouldn't even bother. We have problems on desktops, and you're bringing a whole new opportunity to suffer from proprietary hardware&software.

P.S. I'm not sure if AMD-NVIDIA notebooks even exist on this planet. And it's not some eurocom notebooks, which are whatever you want.

Last edited by Duelist (2015-03-30 13:07:42)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4657 2015-03-30 14:42:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tholin wrote:

A new patch went into kernel 4.0 to work around the reset problem but it requires a switch between the cards and the root hub.

What patch is this?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4658 2015-03-30 15:23:58

tholin
Member
Registered: 2015-03-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
tholin wrote:

A new patch went into kernel 4.0 to work around the reset problem but it requires a switch between the cards and the root hub.

What patch is this?

Well, you wrote it.

commit d84f31744643e2c439466d513ebc1bc81c4d9186
Author: Alex Williamson <alex.williamson@redhat.com>
Date:   Fri Nov 21 11:24:14 2014 -0700

PCI: Mark AMD/ATI VGA devices that don't reset on D3hot->D0 transition

Some AMD/ATI GPUs report NoSoftRst- to indicate that they perform a reset
when software transitions them from D3hot to D0, but there is no apparent
effect on the device: the monitor remains synced and the framebuffer
contents are retained.

Callers of pci_reset_function() don't necessarily have a way to validate
whether a reset was effective, so we don't want to rely on NoSoftRst if
it's known to be inaccurate.  Returning an error in such cases appears to
be the better option.  For users like vfio-pci, this allows the driver to
escalate to the bus reset interfaces.

If a device lives on the root bus, there's really no further
escalation path, so we exempt PM reset as potentially better than
nothing.

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index e52356a..31e7972 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3042,6 +3042,27 @@ static void quirk_no_bus_reset(struct pci_dev *dev) */ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030, quirk_no_bus_reset); +static void quirk_no_pm_reset(struct pci_dev *dev) +{ + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html We can't do a bus reset on root bus devices, but an ineffective + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html PM reset may be better than nothing. + */ + if (!pci_is_root_bus(dev->bus)) + dev->dev_flags |= PCI_DEV_FLAGS_NO_PM_RESET; +} + +/* + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Some AMD/ATI GPUS (HD8570 - Oland) report that a D3hot->D0 transition + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html causes a reset (i.e., they advertise NoSoftRst-). This transition seems + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html to have no effect on the device: it retains the framebuffer contents and + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html monitor sync. Advertising this support makes other layers, like VFIO, + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html assume pci_reset_function() is viable for this device. Mark it as + 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_18.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html unavailable to skip it when testing reset methods. + */ +DECLARE_PCI_FIXUP_CLASS_HEADER(PCI_VENDOR_ID_ATI, PCI_ANY_ID, + PCI_CLASS_DISPLAY_VGA, 8, quirk_no_pm_reset); +

Offline

#4659 2015-03-30 15:47:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tholin wrote:
aw wrote:
tholin wrote:

A new patch went into kernel 4.0 to work around the reset problem but it requires a switch between the cards and the root hub.

What patch is this?

Well, you wrote it.

commit d84f31744643e2c439466d513ebc1bc81c4d9186

Yep, that's the one I was afraid you were looking at.  This is only a fix for a limited number of devices that expose that they do a reset on transition from D3hot to D0.  My HD8570 is one such card.  It mostly makes the finishing reset-on-close work so the screen turns off rather than leaving the last image displayed by the guest.  This does not fix the reset problem on cards like the HD7790 or various R7/9 cards that can't even manage to reboot the guest within the same instance of the VM.  We still do not have a solution for that reset problem.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4660 2015-03-30 16:56:37

tholin
Member
Registered: 2015-03-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yep, that's the one I was afraid you were looking at.  This is only a fix for a limited number of devices that expose that they do a reset on transition from D3hot to D0.  My HD8570 is one such card.  It mostly makes the finishing reset-on-close work so the screen turns off rather than leaving the last image displayed by the guest.  This does not fix the reset problem on cards like the HD7790 or various R7/9 cards that can't even manage to reboot the guest within the same instance of the VM.  We still do not have a solution for that reset problem.

That's disappointing. Which cards are affected? Just the Bonaire XT (HD7790) or all the GCN 1.1 based cards? Are the GCN 1.2 based R9 285 unaffected like Duelist suggested?

Offline

#4661 2015-03-30 17:08:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tholin wrote:
aw wrote:

Yep, that's the one I was afraid you were looking at.  This is only a fix for a limited number of devices that expose that they do a reset on transition from D3hot to D0.  My HD8570 is one such card.  It mostly makes the finishing reset-on-close work so the screen turns off rather than leaving the last image displayed by the guest.  This does not fix the reset problem on cards like the HD7790 or various R7/9 cards that can't even manage to reboot the guest within the same instance of the VM.  We still do not have a solution for that reset problem.

That's disappointing. Which cards are affected? Just the Bonaire XT (HD7790) or all the GCN 1.1 based cards? Are the GCN 1.2 based R9 285 unaffected like Duelist suggested?

Dunno


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4662 2015-03-30 17:08:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
tholin wrote:
aw wrote:

What patch is this?

Well, you wrote it.

commit d84f31744643e2c439466d513ebc1bc81c4d9186

Yep, that's the one I was afraid you were looking at.  This is only a fix for a limited number of devices that expose that they do a reset on transition from D3hot to D0.  My HD8570 is one such card.  It mostly makes the finishing reset-on-close work so the screen turns off rather than leaving the last image displayed by the guest.  This does not fix the reset problem on cards like the HD7790 or various R7/9 cards that can't even manage to reboot the guest within the same instance of the VM.  We still do not have a solution for that reset problem.

Huh? The behaviour you've described i've observed on HD7750 when i force-off'd the guest machine. I thought it was logical for it to behave like that. Like, the machine is killed, and the device is left in the last state it was.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4663 2015-03-30 18:18:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:
tholin wrote:

Well, you wrote it.

commit d84f31744643e2c439466d513ebc1bc81c4d9186

Yep, that's the one I was afraid you were looking at.  This is only a fix for a limited number of devices that expose that they do a reset on transition from D3hot to D0.  My HD8570 is one such card.  It mostly makes the finishing reset-on-close work so the screen turns off rather than leaving the last image displayed by the guest.  This does not fix the reset problem on cards like the HD7790 or various R7/9 cards that can't even manage to reboot the guest within the same instance of the VM.  We still do not have a solution for that reset problem.

Huh? The behaviour you've described i've observed on HD7750 when i force-off'd the guest machine. I thought it was logical for it to behave like that. Like, the machine is killed, and the device is left in the last state it was.

It might be useful for debugging, but it's not intended.  We really want to return the device to the host in a similar state that we received it.  It's also nice that the monitor will turn off if the VM isn't running.  Not to mention that the data in the framebuffer is guest state that we shouldn't just leave laying around if we can help it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4664 2015-03-30 19:43:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
aw wrote:

Yep, that's the one I was afraid you were looking at.  This is only a fix for a limited number of devices that expose that they do a reset on transition from D3hot to D0.  My HD8570 is one such card.  It mostly makes the finishing reset-on-close work so the screen turns off rather than leaving the last image displayed by the guest.  This does not fix the reset problem on cards like the HD7790 or various R7/9 cards that can't even manage to reboot the guest within the same instance of the VM.  We still do not have a solution for that reset problem.

Huh? The behaviour you've described i've observed on HD7750 when i force-off'd the guest machine. I thought it was logical for it to behave like that. Like, the machine is killed, and the device is left in the last state it was.

It might be useful for debugging, but it's not intended.  We really want to return the device to the host in a similar state that we received it.  It's also nice that the monitor will turn off if the VM isn't running.  Not to mention that the data in the framebuffer is guest state that we shouldn't just leave laying around if we can help it.

Okay, i know that this feature is good, but you've said "only for selected devices". But reading the patch seems like every ATI device will be reset. Is it true? If so - that's neat.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4665 2015-03-30 20:03:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Okay, i know that this feature is good, but you've said "only for selected devices". But reading the patch seems like every ATI device will be reset. Is it true? If so - that's neat.

We've been resetting devices as they're released for a while (v3.17), it's not AMD specific.  What we're doing in this commit is simply blacklisting PM reset for the small population of AMD GPUs that claim to support it (NoSoftRst-).  Most GPUs, AMD or otherwise, do not claim to support PM reset, so there's no change.  When there are no function-level resets available for a device, the device remains tagged as dirty and when vfio-pci has the opportunity to do a bus reset, it will.  For this to work, all of the devices affected by a bus reset need to be bound to vfio-pci.  For instance, if you don't assign the audio function to the VM and leave it bound to pci-stub, this would be a reason to bind it to vfio-pci instead.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4666 2015-03-31 01:50:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

Okay, i know that this feature is good, but you've said "only for selected devices". But reading the patch seems like every ATI device will be reset. Is it true? If so - that's neat.

We've been resetting devices as they're released for a while (v3.17), it's not AMD specific.  What we're doing in this commit is simply blacklisting PM reset for the small population of AMD GPUs that claim to support it (NoSoftRst-).  Most GPUs, AMD or otherwise, do not claim to support PM reset, so there's no change.  When there are no function-level resets available for a device, the device remains tagged as dirty and when vfio-pci has the opportunity to do a bus reset, it will.  For this to work, all of the devices affected by a bus reset need to be bound to vfio-pci.  For instance, if you don't assign the audio function to the VM and leave it bound to pci-stub, this would be a reason to bind it to vfio-pci instead.

Thanks for clarification. Aaand my GPU is affected by this patch, it reports NoSoftRst- but doesn't reset. Well, half a year waiting untill 4.0 will be released..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4667 2015-03-31 01:53:54

syshack
Member
Registered: 2015-03-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
syshack wrote:
syshack wrote:

i'v filed a bugzia,pls check,thx. ( https://bugzilla.redhat.com/show_bug.cgi?id=1206006)
if need more details,let me know and i will provid as soon as possible.

im sorry ,i make a stupid mistake. the fails caused by power shortage.

Glad you got it working

Thank you and erganzi!

Offline

#4668 2015-03-31 01:59:47

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just received Creative Labs Sound Blaster Audigy2 ZS PCI Sound Card and I had no issues passing it to windows 8.1

Offline

#4669 2015-03-31 17:08:37

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

... for win7 ... although I put -smp 6, the 440fx/ovmf/Uefi version reports only two processors.

That seems to be a limitation of Windows 7.

http://en.wikipedia.org/wiki/Windows_7#Processor_limits

You can try experimenting with the -smp sub-options. I think plain

-smp 6

says "six single-core, single-thread processors", and that is beyond the Windows 7 limitation. Try to pass "two triple-core, single-thread processors" instead, with

-smp sockets=2,cores=3,threads=1

Offline

#4670 2015-03-31 17:43:08

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
mostlyharmless wrote:

... for win7 ... although I put -smp 6, the 440fx/ovmf/Uefi version reports only two processors.

That seems to be a limitation of Windows 7.

http://en.wikipedia.org/wiki/Windows_7#Processor_limits

You can try experimenting with the -smp sub-options. I think plain

-smp 6

says "six single-core, single-thread processors", and that is beyond the Windows 7 limitation. Try to pass "two triple-core, single-thread processors" instead, with

-smp sockets=2,cores=3,threads=1

I used to have -smp 4 and it worked just fine with win7, showing four cores with one thread each in one socket. Now i have libvirt, and "<vcpu placement='static'>4</vcpu>" and it works fine with(or without) OVMF and win7.

Also, check it out:
windows server 2008r2 on an unknown hypervisor
I might be able to dig out some more info on this machine. All i know - this is a server machine, obviously.

UPD:
Table of testing:
S_C_T_Number of windows cores
2_2_1_4 (two sockets, two cores per socket, one thread per core, four logical cpus as seen by windows)
2_4_1_8
4_1_1_2
4_4_1_8

Weird, as wikipedia states that the limit is one socket, not two. OH. RIGHT. I've forgot what edition of windowns i have. I should go to hell for this...

Last edited by Duelist (2015-03-31 20:19:17)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4671 2015-03-31 20:46:20

dante82
Member
Registered: 2015-03-31
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there,

I finally got to the point of succeeding in passing through my graphics card using the marvelous step-by-step guide in the first comment, thanks for keeping this also updated!!!

However, I'm experiencing heavy lags after installing the NVidia graphics card driver. I already tried the 340.52 version that doesn't have the error 43 issue. I don't experience these lags without the graphics driver (fallback to system drivers). I googled for that issue but couldn't find any leap to follow. Any ideas from the audience?

My setup is passing through a GeForce 750Ti on a Windows 8.1 64Bit VM with the following qemu call:

sudo qemu-system-x86_64 \ -enable-kvm -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -M q35 \ -m 4096 -smp 2 \ -device vfio-pci,host=01:00.0,x-vga=on \ -device vfio-pci,host=01:00.1 \ -device vfio-pci,host=03:00.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/vir-pool/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/mnt/vir-pool/kvm-win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -net nic,model=virtio,macaddr=52:54:00:00:00:01 -net bridge,br=br0 \ -vnc :1 -vga none

Offline

#4672 2015-03-31 21:02:36

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So somehow I managed to break it again, after it's been working great for a while.

My main graphics card (a GTX980) broke, so I sent it to RMA. In the process of de-terming what was wrong with my pc (before I found out it was the GPU), i've reinstalled my Arch Host and switched to the linux-ovmf kernel in the AUR. I've gotten it back yesterday and plugged everything in again. Added the correct video card (a GTX770) to pci-stub again.

lspci -nn

00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 980] [10de:13c0] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 04:00.0 Network controller [0280]: Broadcom Corporation BCM4352 802.11ac Wireless Network Adapter [14e4:43b1] (rev 03)

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pci-stub.ids=8086:153b,14e4:43b1,10de:1184,10de:0e0a"

As my motherboard has 2 ethernet slots, i've also passthroughed(?) the ethernet adapter and the wireless adapter for Bluetooth.

To me it seems everything is correct here. However:
dmesg | grep pci-stub

[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-vfio root=UUID=5ff3e9bb-e7b1-494f-8e42-6c5a1f8e7aa6 rw quiet intel_iommu=on pci-stub.ids=8086:153b,14e4:43b1,10de:1184,10de:0e0a i915.enable_hd_vgaarb=1 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-vfio root=UUID=5ff3e9bb-e7b1-494f-8e42-6c5a1f8e7aa6 rw quiet intel_iommu=on pci-stub.ids=8086:153b,14e4:43b1,10de:1184,10de:0e0a i915.enable_hd_vgaarb=1 [ 0.451397] pci-stub: add 8086:153B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.451402] pci-stub 0000:00:19.0: claimed by stub [ 0.451405] pci-stub: add 14E4:43B1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.451409] pci-stub 0000:04:00.0: claimed by stub [ 0.451411] pci-stub: add 10DE:1184 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.451415] pci-stub 0000:02:00.0: claimed by stub [ 0.451417] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.451421] pci-stub 0000:02:00.1: claimed by stub [ 3.153520] pci-stub 0000:02:00.0: enabling device (0000 -> 0003)

So everything seems correctly blocked by pci-stub, except for 02:00.0 (which is the graphics card itself, not the audio). Somehow it re-enables itself?

Also /etc/vfio-pci.cfg:

DEVICES="0000:00:19.0 0000:02:00.0 0000:02:00.1 0000:04:00.0"

Now when I run the VM script it gives the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

The script:

QEMU_AUDIO_DRV=pa qemu-system-x86_64 -L . -enable-kvm -m 8096 -cpu host,kvm=off -smp 6,sockets=1,cores=6,threads=1 -vga none \ -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -device vfio-pci,host=00:19.0 -device vfio-pci,host=04:00.0 \ -device virtio-scsi-pci,id=scsi -drive file=/dev/sdb,id=ssddisk,format=raw,if=none -device scsi-hd,drive=ssddisk \ -usb -usbdevice host:0738:2215 -usbdevice host:13d3:3404 -usbdevice host:2516:0011 -usbdevice host:1532:0033 -soundhw hda

Offline

#4673 2015-03-31 21:06:39

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Now when I run the VM script it gives the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device

That means the IOMMU is not enabled.  Maybe it's not enabled in the kernel config.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4674 2015-03-31 21:13:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dante82 wrote:

Hi there,

I finally got to the point of succeeding in passing through my graphics card using the marvelous step-by-step guide in the first comment, thanks for keeping this also updated!!!

However, I'm experiencing heavy lags after installing the NVidia graphics card driver. I already tried the 340.52 version that doesn't have the error 43 issue. I don't experience these lags without the graphics driver (fallback to system drivers). I googled for that issue but couldn't find any leap to follow. Any ideas from the audience?

My setup is passing through a GeForce 750Ti on a Windows 8.1 64Bit VM with the following qemu call:

sudo qemu-system-x86_64 \ -enable-kvm -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -M q35 \ -m 4096 -smp 2 \ -device vfio-pci,host=01:00.0,x-vga=on \ -device vfio-pci,host=01:00.1 \ -device vfio-pci,host=03:00.0 \ -device virtio-scsi-pci,id=scsi \ -drive file=/mnt/vir-pool/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/mnt/vir-pool/kvm-win7.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -net nic,model=virtio,macaddr=52:54:00:00:00:01 -net bridge,br=br0 \ -vnc :1 -vga none

Any messages in dmesg maybe? Like IO_PAGE_FAULT?
What motherboard and CPU do you have?

And, aw, wasn't there some problem with one of hyper-v extensions and nvidia?

Last edited by Duelist (2015-03-31 21:22:34)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4675 2015-03-31 21:14:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

And, aw, wasn't there some problem wioth one of hyper-v extensions and nvidia?

I didn't feel like looking it up, I assume they're using the version that detects KVM, but not Hyper-V


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4676 2015-03-31 21:31:48

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
PureTryOut wrote:

Now when I run the VM script it gives the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device

That means the IOMMU is not enabled.  Maybe it's not enabled in the kernel config.

Well that would be a bad thing, considering this kernel is made to enable pci passthrough... Well i'll report it in the AUR then.

Offline

#4677 2015-03-31 21:34:42

dante82
Member
Registered: 2015-03-31
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

And, aw, wasn't there some problem wioth one of hyper-v extensions and nvidia?

I didn't feel like looking it up, I assume they're using the version that detects KVM, but not Hyper-V

First of all, thanks for the quick replies!

I tried both variants, with and without hv-extensions. With latest driver version I get the error 43, with the 340.52 a horribly slow desktop experience.

Any messages in dmesg maybe? Like IO_PAGE_FAULT?
What motherboard and CPU do you have?
And, aw, wasn't there some problem with one of hyper-v extensions and nvidia?

Nothing unusual in dmesg. I'm currently changing graphics against an old 8800GTS I have spare here. Will post the dmesg output and results shortly

In the meantime further data that I can provide:
Mainboard: Intel DH87RL
CPU: Intel 4570
RAM: 32 GB

Offline

#4678 2015-03-31 21:41:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dante82

Does the lag go away without the audio function assigned?  If so: http://vfio.blogspot.com/2014/09/vfio-i … ndows.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4679 2015-03-31 21:47:02

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:
aw wrote:
PureTryOut wrote:

Now when I run the VM script it gives the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device

That means the IOMMU is not enabled.  Maybe it's not enabled in the kernel config.

Well that would be a bad thing, considering this kernel is made to enable pci passthrough... Well i'll report it in the AUR then.

It's unlikely its disabled in the kernel config since its based on -ARCH kernel:

try this and check if its enabled:

zcat /proc/config.gz | grep -i iommu

if not, check if its enabled in your bios

Offline

#4680 2015-03-31 21:56:22

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dante82 wrote:

In the meantime further data that I can provide:
Mainboard: Intel DH87RL
CPU: Intel 4570
RAM: 32 GB


Well, your hardware should support IOMMU, H87 chipset supports vt-d and intel i5-4570(i assume you have that CPU) supports vt-d.
That's a good thing. Try following aw's advice.

Last edited by Duelist (2015-03-31 21:56:40)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4681 2015-03-31 22:13:10

dante82
Member
Registered: 2015-03-31
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

to answer your comments first:

Does the lag go away without the audio function assigned?  If so: http://vfio.blogspot.com/2014/09/vfio-i … ndows.html

From what I see in the qemu call, I don't have audio assigned. Correct me if I'm wrong there, it's already late here.
By Audio, do you mean the Audio device of the GeForce (PCI ID 1:00.1)?

Well, your hardware should support IOMMU, H87 chipset supports vt-d and intel i5-4570(i assume you have that CPU) supports vt-d.

That's correct and I tried to choose my HW setup wisely considering the idle power consumption as well. Otherwise I would have gone with an Asrock ;-)


here are the results. With the GF 8800 GTS I experience the same lagging with the drivers and dmesg gives me the following output:

[ 815.599961] device tap0 entered promiscuous mode [ 815.600081] br0: port 2(tap0) entered forwarding state [ 815.600090] br0: port 2(tap0) entered forwarding state [ 815.950618] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403) [ 815.978917] vfio-pci 0000:03:00.0: enabling device (0500 -> 0502) [ 816.545577] irq 16: nobody cared (try booting with the "irqpoll" option) [ 816.546138] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.19.2-1-vfio #1 [ 816.546139] Hardware name: /DH87RL, BIOS RLH8710H.86A.0327.2014.0924.1645 09/24/2014 [ 816.546140] 0000000000000000 1e87b86f7f3a999a ffff88082fa83e28 ffffffff8155d0cf [ 816.546142] 0000000000000000 ffff88080a6dac00 ffff88082fa83e58 ffffffff810ce6a6 [ 816.546143] ffff88080a0b4b50 ffff88080a6dac00 0000000000000000 0000000000000010 [ 816.546145] Call Trace: [ 816.546146] <IRQ> [<ffffffff8155d0cf>] dump_stack+0x4c/0x6e [ 816.546155] [<ffffffff810ce6a6>] __report_bad_irq+0x36/0xd0 [ 816.546157] [<ffffffff810cea47>] note_interrupt+0x257/0x2a0 [ 816.546159] [<ffffffff810cbfae>] handle_irq_event_percpu+0xae/0x1f0 [ 816.546161] [<ffffffff810cc131>] handle_irq_event+0x41/0x70 [ 816.546162] [<ffffffff810cefd6>] handle_fasteoi_irq+0x86/0x140 [ 816.546166] [<ffffffff81018252>] handle_irq+0x22/0x40 [ 816.546169] [<ffffffff815654bf>] do_IRQ+0x4f/0xf0 [ 816.546171] [<ffffffff8156346d>] common_interrupt+0x6d/0x6d [ 816.546172] <EOI> [<ffffffff81421ac2>] ? poll_idle+0x42/0x80 [ 816.546176] [<ffffffff81421519>] cpuidle_enter_state+0x49/0x190 [ 816.546177] [<ffffffff81421747>] cpuidle_enter+0x17/0x20 [ 816.546179] [<ffffffff810b49c8>] cpu_startup_entry+0x388/0x410 [ 816.546182] [<ffffffff8104b55f>] start_secondary+0x1af/0x1f0 [ 816.546183] handlers: [ 816.546384] [<ffffffffa004da70>] usb_hcd_irq [usbcore] [ 816.546817] [<ffffffffa047a780>] vfio_intx_handler [vfio_pci] [ 816.547298] Disabling IRQ #16 [ 818.531373] irq 16: nobody cared (try booting with the "irqpoll" option) [ 818.531875] CPU: 3 PID: 501 Comm: qemu-system-x86 Not tainted 3.19.2-1-vfio #1 [ 818.531876] Hardware name: /DH87RL, BIOS RLH8710H.86A.0327.2014.0924.1645 09/24/2014 [ 818.531877] 0000000000000000 00000000d16d99f1 ffff88082fb83e28 ffffffff8155d0cf [ 818.531879] 0000000000000000 ffff88080a6dac00 ffff88082fb83e58 ffffffff810ce6a6 [ 818.531880] ffff88080a0b4b50 ffff88080a6dac00 0000000000000000 0000000000000010 [ 818.531882] Call Trace: [ 818.531883] <IRQ> [<ffffffff8155d0cf>] dump_stack+0x4c/0x6e [ 818.531892] [<ffffffff810ce6a6>] __report_bad_irq+0x36/0xd0 [ 818.531894] [<ffffffff810cea47>] note_interrupt+0x257/0x2a0 [ 818.531896] [<ffffffff810cbfae>] handle_irq_event_percpu+0xae/0x1f0 [ 818.531897] [<ffffffff810cc131>] handle_irq_event+0x41/0x70 [ 818.531899] [<ffffffff810cefd6>] handle_fasteoi_irq+0x86/0x140 [ 818.531902] [<ffffffff81018252>] handle_irq+0x22/0x40 [ 818.531904] [<ffffffff815654bf>] do_IRQ+0x4f/0xf0 [ 818.531906] [<ffffffff8156346d>] common_interrupt+0x6d/0x6d [ 818.531907] <EOI> [<ffffffff813c9f28>] ? intel_iommu_unmap+0x1a8/0x1e0 [ 818.531911] [<ffffffff813c9eb4>] ? intel_iommu_unmap+0x134/0x1e0 [ 818.531913] [<ffffffff813baedc>] iommu_unmap+0xac/0x1c0 [ 818.531919] [<ffffffffa040fa13>] vfio_remove_dma+0xc3/0x1b0 [vfio_iommu_type1] [ 818.531921] [<ffffffffa0410204>] vfio_iommu_type1_ioctl+0x464/0xaae [vfio_iommu_type1] [ 818.531926] [<ffffffffa05d08aa>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 818.531929] [<ffffffffa05d0d05>] ? kvm_vm_ioctl+0x445/0x790 [kvm] [ 818.531931] [<ffffffffa0404fc7>] vfio_fops_unl_ioctl+0x77/0x2c0 [vfio] [ 818.531935] [<ffffffff811e4e48>] do_vfs_ioctl+0x2f8/0x500 [ 818.531937] [<ffffffff811ef592>] ? __fget+0x72/0xb0 [ 818.531939] [<ffffffff811e50d1>] SyS_ioctl+0x81/0xa0 [ 818.531941] [<ffffffff81562909>] system_call_fastpath+0x12/0x17 [ 818.531941] handlers: [ 818.532119] [<ffffffffa004da70>] usb_hcd_irq [usbcore] [ 818.532505] [<ffffffffa047a780>] vfio_intx_handler [vfio_pci] [ 818.532936] Disabling IRQ #16 [ 822.210072] kvm: zapping shadow pages for mmio generation wraparound [ 830.651754] br0: port 2(tap0) entered forwarding state

With re-attaching the GF 750TI I got an error message from KVM:

KVM internal error. Suberror: 1 emulation failure RAX=0000000000000170 RBX=00000000003c2000 RCX=0000000000010008 RDX=00000000003c2000 RSI=0000000080000005 RDI=fffff80004560180 RBP=0000000000000001 RSP=ffffd000232e72e8 R8 =0000000000000000 R9 =0000000000001000 R10=ffffd000232e7490 R11=0000000000000000 R12=0000000000000000 R13=0000000000000002 R14=ffffd000232e7488 R15=0000000000000001 RIP=ffffffffffd03000 RFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA] CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA] FS =0053 0000000066076000 00003c00 0040f300 DPL=3 DS [-WA] GS =002b fffff80004560000 ffffffff 00c0f300 DPL=3 DS [-WA] LDT=0000 0000000000000000 ffffffff 00c00000 TR =0040 fffff80005f01080 00000067 00008b00 DPL=0 TSS64-busy GDT= fffff80005f00000 0000007f IDT= fffff80005f00080 00000fff CR0=80050031 CR2=000000001026efd8 CR3=000000016bb4a000 CR4=001506f8 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000d01 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <0f> 01 c1 c3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

After a restart of the VM, the 340 driver shows an error 43 message (which didn't come up before).

BUT: After reinstalling the driver and rebooting the VM again, the lagging has gone.
I don't understand why but here's typical output I see from dmesg:

[ 144.160027] device tap0 entered promiscuous mode [ 144.160181] br0: port 2(tap0) entered forwarding state [ 144.160189] br0: port 2(tap0) entered forwarding state [ 144.522094] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258 [ 144.522107] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 144.525417] vfio-pci 0000:03:00.0: enabling device (0500 -> 0502) [ 150.145059] kvm: zapping shadow pages for mmio generation wraparound [ 159.222102] br0: port 2(tap0) entered forwarding state

I'm happy for now but the stability of the complete setup still worries me. Any further ideas on where to investigate?

Last edited by dante82 (2015-03-31 22:19:37)

Offline

#4682 2015-03-31 22:18:18

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dante82:

-device vfio-pci,host=01:00.1 \

That's your line, and it passes your gpu's audio function.
First dmesg error just asking for aw's fix about MSI - IRQ #16 happened, but nobody cared. Something hints me that this irq 16 belong to 01:00.1 host device.

The solutions are:
1.Ditch out host device 01:00.1
2.Follow aw's advice on forcing it use MSI
3.Ditch the software part of driver installation doing anything with audio.
4.Get the IRQ #16 disabled via kernel error everytime.

Oh, and what the hell is host device 03:00.0? Usb controller? Studying your logs further gives me some idea about usb.

Last edited by Duelist (2015-03-31 22:21:08)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4683 2015-03-31 22:22:15

dante82
Member
Registered: 2015-03-31
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

@dante82:

-device vfio-pci,host=01:00.1 \

That's your line, and it passes your gpu's audio function.
First dmesg error just asking for aw's fix about MSI - IRQ #16 happened, but nobody cared. Something hints me that this irq 16 belong to 01:00.1 host device.

The solutions are:
1.Ditch out host device 01:00.1
2.Follow aw's advice on forcing it use MSI
3.Ditch the software part of driver installation doing anything with audio.
4.Get the IRQ #16 disabled via kernel error everytime.

I just had the same idea (see my edited post). ;-)

Thanks a lot for your advices! I will investigate further tomorrow and see how it goes.

Offline

#4684 2015-03-31 22:25:05

dante82
Member
Registered: 2015-03-31
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Oh, and what the hell is host device 03:00.0? Usb controller? Studying your logs further gives me some idea about usb.

Yeah right, I forgot to mention that small detail:

03:00.0 USB controller: VIA Technologies, Inc. Device 3483 (rev 01) (prog-if 30 [XHCI]) Subsystem: VIA Technologies, Inc. Device 3483 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+ Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 33 Region 0: Memory at f7800000 (64-bit, non-prefetchable) [size=4K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: xhci_pci

Offline

#4685 2015-03-31 22:27:18

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

It's unlikely its disabled in the kernel config since its based on -ARCH kernel:

try this and check if its enabled:

zcat /proc/config.gz | grep -i iommu

if not, check if its enabled in your bios

Well it seemed my BIOS got reset somehow without mentioning me, probably when I plugged the host GPU in again.
Vt-d was disabled, enabled it, and it worked again. Thanks for your help guys!

Offline

#4686 2015-04-01 02:05:19

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

I use linux-3.18.6 and qemu-2.1.3 and can passthrough AMD R5000 successful.
but I can not see the VM bios startup message on screen, it direct go to the windows 7 startup process.

give me some idea, thanks.

Offline

#4687 2015-04-01 02:14:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@erganzi

-boot menu=on[,splash-time=$timeout_ms]  ?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4688 2015-04-01 05:17:33

darkstego
Member
Registered: 2014-08-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

I was able to get Windows 8.1 running with VFIO and OVMF, but I have an odd issue.

After the windows install, OVMF will not boot windows if I remove the drive with the install media

Here is my running script.

sudo qemu-system-x86_64 \ -L $HOME/Desktop/ovmf -bios bios.bin \ -enable-kvm -m 8192 -smp 4,cores=2,threads=2,sockets=1 \ -cpu host,kvm=off \ -device virtio-scsi-pci,id=scsi \ -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \ -drive file=/dev/sdb,id=disk_primary,format=raw,if=none -device scsi-hd,drive=disk_primary \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -vga none \ -rtc base=localtime,clock=host \ -soundhw hda \ -device vfio-pci,host=05:00.0

The trouble is if I omit the

 -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install 

line or move it after the disk drive the EFI will not find anything to boot.

I am sure the windows installed in EFI mode (GPT partition and everything), but I can't seem to convince OVMF to boot into it without the iso present.

I would appreciate any advice on the matter.

Offline

#4689 2015-04-01 07:18:10

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@erganzi

-boot menu=on[,splash-time=$timeout_ms]  ?

Try:
-boot menu=on,splash-time=20000

It still display nothing, just black screen, then turned into windows startup.

Offline

#4690 2015-04-01 07:49:45

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any idea why this command:

virt-install --name=win7 --os-type=windows --disk path=/dev/mapper/VMssd-win7 --cdrom=/vm/win7x64.iso --graphics spice --ram=2048 --boot loader_type=pflash,loader_ro=yes,loader=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd,nvram_template=/vm/win7_VARS-pure-efi.fd

Results in this:

ERROR Unknown options ['loader_ro', 'nvram_template', 'loader_type']

Offline

#4691 2015-04-01 08:51:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:

Any idea why this command:

virt-install --name=win7 --os-type=windows --disk path=/dev/mapper/VMssd-win7 --cdrom=/vm/win7x64.iso --graphics spice --ram=2048 --boot loader_type=pflash,loader_ro=yes,loader=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd,nvram_template=/vm/win7_VARS-pure-efi.fd

Results in this:

ERROR Unknown options ['loader_ro', 'nvram_template', 'loader_type']

Maybe it's /etc/libvirtd/qemu.conf error, and there's no OVMF line there as mentioned in OVMF whitepaper?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4692 2015-04-01 17:03:29

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

That's in /etc/libvirt/qemu.conf

It almost seems like the error is telling me that virt-install isnt capable of those options (loader_ro, nvram_template and loader_type) even though it is documented here and here.

I think you are right duelist, because I have noticed in vart-manager I am lacking the option for setting OVMF instead of BIOS in overview.

Edit: virt-install not virsh

Even using the whitepaper as an example, I get the same error:

LDR="loader=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,loader_ro=yes,loader_type=pflash" virt-install \ > --name fedora20 \ > --memory 2048 \ > --vcpus 2 \ > --os-variant fedora20 \ > --boot hd,cdrom,$LDR \ > --disk size=20 \ > --disk path=Fedora-Live-Xfce-x86_64-20-1.iso,device=cdrom,bus=scsi
ERROR Unknown options ['loader_ro', 'loader_type']

Yes, I have a problem:

$ virt-install --boot=? --boot options: arch cdrom clearxml domain_type dtb emulator fd hd init initargs initrd kernel kernel_args loader machine menu network os_type useserial

I should have loader, loader_ro, loader_type, nvram and nvram_template...

Last edited by Naruni (2015-04-01 17:59:01)

Offline

#4693 2015-04-01 19:04:54

tritron4
Member
Registered: 2012-04-14
Posts: 146

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well libvirt git from aur lets you select ovmf. I am using ovmf pure from rpm and virt-manager works fine with ovmf

Naruni wrote:
nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

That's in /etc/libvirt/qemu.conf

It almost seems like the error is telling me that virt-install isnt capable of those options (loader_ro, nvram_template and loader_type) even though it is documented here and here.

I think you are right duelist, because I have noticed in vart-manager I am lacking the option for setting OVMF instead of BIOS in overview.

Edit: virt-install not virsh

Even using the whitepaper as an example, I get the same error:

LDR="loader=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,loader_ro=yes,loader_type=pflash" virt-install \ > --name fedora20 \ > --memory 2048 \ > --vcpus 2 \ > --os-variant fedora20 \ > --boot hd,cdrom,$LDR \ > --disk size=20 \ > --disk path=Fedora-Live-Xfce-x86_64-20-1.iso,device=cdrom,bus=scsi
ERROR Unknown options ['loader_ro', 'loader_type']

Yes, I have a problem:

$ virt-install --boot=? --boot options: arch cdrom clearxml domain_type dtb emulator fd hd init initargs initrd kernel kernel_args loader machine menu network os_type useserial

I should have loader, loader_ro, loader_type, nvram and nvram_template...

Offline

#4694 2015-04-01 19:56:05

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

That's in /etc/libvirt/qemu.conf

It almost seems like the error is telling me that virt-install isnt capable of those options (loader_ro, nvram_template and loader_type) even though it is documented here and here.

Y'sure that your software is updated? Seems like old libvirtd on host to me.

[user@crossfire ~]$ yum info libvirt Loaded plugins: langpacks Available Packages Name : libvirt Arch : x86_64 Version : 1.2.9.2 Release : 1.fc21 Size : 43 k Repo : updates/21/x86_64

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4695 2015-04-01 20:44:34

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
it's me again ... :'(
I got the 4790k today and have a fresh arch on a drive but sadly no success with my setup sad - I have no Idea what to try next, can't figure out how to create the iommu group other than with the vfio-bind script provided by the op.

This is the provided error msg:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

// journalctl:
Apr 01 22:23:36 teststation kernel: Intel-IOMMU: enabled
Apr 01 22:23:36 teststation kernel: pci-stub: add 10DE:13C2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
Apr 01 22:23:36 teststation kernel: pci-stub 0000:01:00.0: claimed by stub
Apr 01 22:23:36 teststation kernel: pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
Apr 01 22:23:36 teststation kernel: pci-stub 0000:01:00.1: claimed by stub
Apr 01 22:24:04 teststation kernel: vfio-pci: probe of 0000:01:00.0 failed with error -22
Apr 01 22:24:04 teststation kernel: vfio-pci: probe of 0000:01:00.0 failed with error -22
Apr 01 22:24:04 teststation kernel: vfio-pci: probe of 0000:01:00.1 failed with error -22


A short Rundown of my configuration:

i7 4970K
Nvidia GTX 970
Asus Sabertooth Z87
2 SSD Mirrors (one for arch, one for windows)
16 GB RAM

Monitor1 --> Intel HD4600 HDMI
Monitor2 --> Intel HD4600 DisplayPort
Monitor3 --> GTX 970 Displayport

$ lspci -nnv | grep NVIDIA
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation Device [10de:1116]
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:1116]

$ sed -n 4p /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb"

$ sed -n 7p /etc/mkinitcpio.conf
MODULES="pci-stub"

$lsmod|egrep -i "vfio|pci_stub|kvm"
vfio_pci               35525  0
vfio_iommu_type1       17118  0
vfio                   18477  2 vfio_iommu_type1,vfio_pci
kvm_intel             143417  0
kvm                   435299  1 kvm_intel
pci_stub               12429  0

If you need any further Info on my system, pls let me know ...
Regards.

Edit: Added System Log.

Last edited by bierbauch (2015-04-01 20:51:16)

Offline

#4696 2015-04-01 20:57:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bierbauch wrote:

This is the provided error msg:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

Look back a page or two; VT-d is not enabled, probably in you BIOS.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4697 2015-04-01 21:06:50

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bierbauch wrote:

This is the provided error msg:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

Look back a page or two; VT-d is not enabled, probably in you BIOS.

Damn  ...
You are right. The only good thing is, I figured it out before I read your post smile

MAAAAN I feel so dumb ^^ - The CPU swap disabled the feature and I forgot to recheck!

Got it working. Let the fun begin!

Offline

#4698 2015-04-02 20:45:49

marcosonoio
Member
Registered: 2014-04-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
thanks to all for this useful topic.
in these days i'm trying to have a working VM with windows 8.1, but i have some problems.
with this config i can install windows, boot up and use windows, but when i try to install nvidia driver (340.52), i received a BSOD (with a video driver's error):

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0,romfile=/home/marco/image.rom,multifunction=on \
-device vfio-pci,host=01:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/marco/it_windows_8.1_with_update_x64_dvd_4048528.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/marco/Scaricati/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/marco/win81.raw,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-rtc base=localtime,clock=host \
-soundhw hda \
-vga none

so i want to try with the q25 chipset, i can't boot up windows, but i can install the system with the "-vga std" option:

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 -M q35 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,romfile=/home/marco/image.rom,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/marco/it_windows_8.1_with_update_x64_dvd_4048528.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/marco/Scaricati/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/marco/win81.raw,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-rtc base=localtime,clock=host \
-soundhw hda \
-vga none

when i start the VM i find this in the dmesg output(always):

[   47.485338] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb000 flags=0x0010]
[   47.485345] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb3c0 flags=0x0010]
[   47.485347] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb040 flags=0x0010]
[   47.485348] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb400 flags=0x0010]
[   47.485350] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb440 flags=0x0010]
[   47.485351] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb080 flags=0x0010]
[   47.485353] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb680 flags=0x0010]
[   47.485354] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb0c0 flags=0x0010]
[   47.485355] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb480 flags=0x0010]
[   47.485356] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb100 flags=0x0010]
[   47.485358] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb4c0 flags=0x0010]
[   47.485359] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb140 flags=0x0010]
[   47.485360] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb500 flags=0x0010]
[   47.485361] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb180 flags=0x0010]
[   47.485363] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb540 flags=0x0010]
[   47.485364] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb1c0 flags=0x0010]
[   47.485365] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb580 flags=0x0010]
[   47.485367] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb200 flags=0x0010]

my system:

amd a10 7850k
gigabyte GA-F2A88XN-WIFI
2x4gb ddr3
ssd 128gb 840pro
nvidia gtx 750 ti
updated fedora 21 + libvirt preview repo

i want to install the 340.52 to not have problem with the hyperv(nvidia related).
what's wrong?

thanks you

Last edited by marcosonoio (2015-04-02 20:49:14)

Offline

#4699 2015-04-02 21:55:12

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Update:  so the -smp answers were helpful, thanks.  Apparently Win 7 under q35 and 440fx deals somewhat differently re number of processors.  First I had looked under Computer->Properties, then under Device Manager->Processors.   Both setups show 6 processors under Device Manager, which I guess is the important thing.  Under Computer->Properties the q35 just mentions the processor name without a number; presumably that means 1 physical processor 6 virtual cores. Under the 440fx setup it says 2 processors; presumably 2 "physical" with 3 virtual each.  Weird and probably unimportant.  If I get a chance I'll see if it changes the "windows experience" processor score.

As far as the USB xhci driver vs passthrough: I haven't had a chance to experiment much yet, but thanks for the extra parameters to try.  I'm pretty sure Win 8/8.1/server 2012 and the rest of the UEFI aware OSes are probably not a good indicator of success with Win 7.

Offline

#4700 2015-04-03 04:12:35

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, arch's current libvirt is version 1.2.14-1-x86_64
I wonder if I'll go to AUR libvirt...

Offline

#4701 2015-04-03 12:54:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

marcosonoio wrote:

Hi,
thanks to all for this useful topic.
in these days i'm trying to have a working VM with windows 8.1, but i have some problems.
with this config i can install windows, boot up and use windows, but when i try to install nvidia driver (340.52), i received a BSOD (with a video driver's error):

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \ -device vfio-pci,host=01:00.0,romfile=/home/marco/image.rom,multifunction=on \ -device vfio-pci,host=01:00.1 \ ..snip.. -vga none

so i want to try with the q25 chipset, i can't boot up windows, but i can install the system with the "-vga std" option:

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 -M q35 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,romfile=/home/marco/image.rom,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ ..snip.. -vga none

when i start the VM i find this in the dmesg output(always):

[ 47.485338] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb000 flags=0x0010] [ 47.485345] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb3c0 flags=0x0010] ..snip.. [ 47.485367] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb200 flags=0x0010]

my system:

amd a10 7850k gigabyte GA-F2A88XN-WIFI 2x4gb ddr3 ssd 128gb 840pro nvidia gtx 750 ti updated fedora 21 + libvirt preview repo

i want to install the 340.52 to not have problem with the hyperv(nvidia related).
what's wrong?

thanks you

First hint where to look:
Your CPU has bugs in IOMMU, they're listed there if i recall your CPU family number right.
That, i believe(haven't red IOMMU architectural guide yet), yelds those IO_PAGE_FAULT errors.
Also, seems like you're using your integrated card as the host gpu, there may be problems with that, but there's none i've heard of.
And you shouldn't absolutely need x-vga option when using OVMF, as stated in aw's presentation.
Try disabling x-vga.

The -vga std option just connects a virtual, emulated graphics card, making your GPU secondary in windows but not in OVMF setup menus.

EDIT:
Forgot to add a reassuring note: i have amd athlon x4 750k, which is based on a similar architecture as yours(and opteron 6300 series too), and asus F2A55 motherboard, and it works with OVMF only.

Last edited by Duelist (2015-04-03 14:22:05)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4702 2015-04-03 14:55:07

larrabee
Member
Registered: 2014-08-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello. I have troubles with vfio.
I have:
Motherboard ASRock Z87 Extreme4
CPU Intel core i7 4771
video AMD 6450
video IGD
SAS adapter Adaptec 7805h
OS: Arch
kernel: stock  (3.19.2-1)
I connect video to vfio and power up VM, but gave error «vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.»

lspci output:

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] 02:00.0 Serial Attached SCSI controller: Adaptec PMC-Sierra PM8018 SAS HBA [Series 7H] (rev 06) 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03)

lspci -n output:

00:00.0 0600: 8086:0c00 (rev 06) 00:01.0 0604: 8086:0c01 (rev 06) 00:01.1 0604: 8086:0c05 (rev 06) 00:02.0 0300: 8086:0412 (rev 06) 00:03.0 0403: 8086:0c0c (rev 06) 00:14.0 0c03: 8086:8c31 (rev 04) 00:16.0 0780: 8086:8c3a (rev 04) 00:19.0 0200: 8086:153b (rev 04) 00:1a.0 0c03: 8086:8c2d (rev 04) 00:1b.0 0403: 8086:8c20 (rev 04) 00:1c.0 0604: 8086:8c10 (rev d4) 00:1c.3 0604: 8086:8c16 (rev d4) 00:1c.6 0604: 8086:244e (rev d4) 00:1d.0 0c03: 8086:8c26 (rev 04) 00:1f.0 0601: 8086:8c44 (rev 04) 00:1f.2 0106: 8086:8c02 (rev 04) 00:1f.3 0c05: 8086:8c22 (rev 04) 01:00.0 0300: 1002:6779 01:00.1 0403: 1002:aa98 02:00.0 0107: 9005:8088 (rev 06) 04:00.0 0200: 8086:1539 (rev 03) 05:00.0 0604: 1b21:1080 (rev 03)

cmdline:

initrd=\intel-ucode.img initrd=\initramfs-linux.img root=/dev/sda2 rw intel_iommu=on iommu=on elevator=noop pci-stub.ids=1002:6779,1002:aa98

ll /sys/bus/pci/devices/*/iommu_group:

lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:00.0/iommu_group -> ../../../kernel/iommu_groups/0/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:01.0/iommu_group -> ../../../kernel/iommu_groups/1/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:01.1/iommu_group -> ../../../kernel/iommu_groups/1/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:02.0/iommu_group -> ../../../kernel/iommu_groups/2/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:03.0/iommu_group -> ../../../kernel/iommu_groups/3/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:14.0/iommu_group -> ../../../kernel/iommu_groups/4/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:16.0/iommu_group -> ../../../kernel/iommu_groups/5/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:19.0/iommu_group -> ../../../kernel/iommu_groups/6/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1a.0/iommu_group -> ../../../kernel/iommu_groups/7/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1b.0/iommu_group -> ../../../kernel/iommu_groups/8/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1c.0/iommu_group -> ../../../kernel/iommu_groups/9/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1c.3/iommu_group -> ../../../kernel/iommu_groups/10/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1c.6/iommu_group -> ../../../kernel/iommu_groups/11/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1d.0/iommu_group -> ../../../kernel/iommu_groups/12/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1f.0/iommu_group -> ../../../kernel/iommu_groups/13/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1f.2/iommu_group -> ../../../kernel/iommu_groups/13/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:00:1f.3/iommu_group -> ../../../kernel/iommu_groups/13/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:01:00.0/iommu_group -> ../../../../kernel/iommu_groups/1/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:01:00.1/iommu_group -> ../../../../kernel/iommu_groups/1/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:02:00.0/iommu_group -> ../../../../kernel/iommu_groups/1/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:04:00.0/iommu_group -> ../../../../kernel/iommu_groups/14/ lrwxrwxrwx 1 root root 0 апр 3 10:57 /sys/bus/pci/devices/0000:05:00.0/iommu_group -> ../../../../kernel/iommu_groups/11/

Video card and SAS adapter in same iommu group (1)

I try:
Use kernel with ACS override patch
Use option vfio_iommu_type1 allow_unsafe_interrupts=1
It doesnot help.

How i can fix this error, please help. Thank you.

Offline

#4703 2015-04-03 15:35:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

larrabee wrote:

Hello. I have troubles with vfio.
I have:
Motherboard ASRock Z87 Extreme4
CPU Intel core i7 4771
video AMD 6450
video IGD
SAS adapter Adaptec 7805h
OS: Arch
kernel: stock  (3.19.2-1)
I connect video to vfio and power up VM, but gave error «vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.»

..snip..
Video card and SAS adapter in same iommu group (1)

I try:
Use kernel with ACS override patch
Use option vfio_iommu_type1 allow_unsafe_interrupts=1
It doesnot help.

How i can fix this error, please help. Thank you.

You sure you enabled your acs override mode? There's a kernel command line parameter for that:

+ pcie_acs_override = + [PCIE] Override missing PCIe ACS support for: + downstream + All downstream ports - full ACS capabilties + multifunction + All multifunction devices - multifunction ACS subset + id:nnnn:nnnn + Specfic device - full ACS capabilities + Specified as vid:did (vendor/device ID) in hex

And are you sure you need that "allow_unsafe_interrupts=1"?
Did you read aw's blog post about it?(question 8)

Last edited by Duelist (2015-04-03 15:36:11)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4704 2015-04-03 17:53:01

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.
2: It appears that I can passthrough my three devices (USB 3.0 controller, graphics adapter, sound card) only the first time I start the VM. When the VM turns off as described in [1] the devices are missing in the VM. The VM is still accessible via RDP, but in the device manager the devices aren't listed.

Is this related to the OVMF Setup? Do I need to release the devices in vfio explicitly?

Thanks in advance!

taskset 0x0000003F \ qemu-system-x86_64 \ -nodefaults \ -name win7uefi \ -enable-kvm \ -machine type=pc,accel=kvm \ -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -m 4096 \ -mem-path /dev/hugepages \ -mem-prealloc \ -balloon none \ -k de \ -drive if=pflash,format=raw,readonly,file=ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=ovmf_vars_x64.bin \ -drive if=none,file=/dev/sdb3,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=on \ -drive if=none,file=/dev/sdb7,id=vdisk2,cache=none,format=raw,aio=native \ -device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk2,id=disk2,x-data-plane=on \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -device vfio-pci,host=03:00.0 \ -device vfio-pci,host=07:00.0 \ -net nic,macaddr=52:54:00:12:34:57,model=virtio -net bridge,br=kvmbr0 \ -spice port=5930,disable-ticketing \ -device virtio-serial-pci \ -device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \ -chardev spicevmc,id=spicechannel0,name=vdagent \ -rtc base=utc \ -monitor unix:user.sock,server,nowait \ -daemonize

Offline

#4705 2015-04-03 18:06:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.
2: It appears that I can passthrough my three devices (USB 3.0 controller, graphics adapter, sound card) only the first time I start the VM. When the VM turns off as described in [1] the devices are missing in the VM. The VM is still accessible via RDP, but in the device manager the devices aren't listed.

Is this related to the OVMF Setup? Do I need to release the devices in vfio explicitly?

Thanks in advance!

taskset 0x0000003F \ qemu-system-x86_64 \ -nodefaults \ -name win7uefi \ -enable-kvm \ -machine type=pc,accel=kvm \ -cpu host,kvm=off \ -smp 6,sockets=1,cores=3,threads=2 \ -m 4096 \ -mem-path /dev/hugepages \ -mem-prealloc \ -balloon none \ -k de \ -drive if=pflash,format=raw,readonly,file=ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=ovmf_vars_x64.bin \ -drive if=none,file=/dev/sdb3,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=on \ -drive if=none,file=/dev/sdb7,id=vdisk2,cache=none,format=raw,aio=native \ -device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk2,id=disk2,x-data-plane=on \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -device vfio-pci,host=03:00.0 \ -device vfio-pci,host=07:00.0 \ -net nic,macaddr=52:54:00:12:34:57,model=virtio -net bridge,br=kvmbr0 \ -spice port=5930,disable-ticketing \ -device virtio-serial-pci \ -device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \ -chardev spicevmc,id=spicechannel0,name=vdagent \ -rtc base=utc \ -monitor unix:user.sock,server,nowait \ -daemonize

Errr, did you use -device qxl-vga as stated in OVMF whitepaper for installing the drivers stage?

I'm currently using libvirt, my VM starts, showing me "starting windowns" screen on QXL graphics in a window on host, then, after some black screen time, it fires up the main GPUs and they start outputting the video signal to the screen properly.

And what do you mean "patching the drivers"?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4706 2015-04-03 18:19:41

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes I needed qxl-vga during the setup. But after I installed the nvidia drivers I removed this device. This is working so far. The last thing I added to the startup script was adding the second disk partition. I removed that one and now I got a fine startup with all devices working well. Is something wrong with that parameter, maybe bus related?
Anyway I do not know if this is a oneshot or if that "guest moved index from 0 to ..." issue still persists.

EDIT: I meant patching windows (updating) and installing device drivers.

EDIT2: Sorry, I had a Seabios VM before and thus I might got something wrong. I was sure that removing the qxl-vga was intended after the drivers of the dedicated graphics device are installed. But I think what you wanted to tell me, that they always better should be present in the VM. Right?

Last edited by apex8 (2015-04-03 18:45:01)

Offline

#4707 2015-04-03 18:51:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Yes I needed qxl-vga during the setup. But after I installed the nvidia drivers I removed this device. This is working so far. The last thing I added to the startup script was adding the second disk partition. I removed that one and now I got a fine startup with all devices working well. Is something wrong with that parameter, maybe bus related?
Anyway I do not know if this is a oneshot or if that "guest moved index from 0 to ..." issue still persists.

EDIT: I meant patching windows (updating) and installing device drivers.

EDIT2: Sorry, I had a Seabios VM before and thus I might got something wrong. I was sure that removing the qxl-vga was intended after the drivers of the dedicated graphics device are installed. But I think what you wanted to tell me, that they always better should be present in the VM. Right?

Well, even if nothing works, qxl-vga still works. Windows turns off QXL for me. For some reason, i couldn't make my VM output "starting windows" message on the main GPU, it works only on qxl-vga. I don't know why, but let it be. Well, it's intended to remove the QXL.

Try adding -device virtio-blk-pci for the second partition and see if that makes it work. Seems like i'm unable help you anymore.

Last edited by Duelist (2015-04-03 18:51:13)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4708 2015-04-04 12:17:01

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it normal for windows guests, that the red hat QXL graphics device is reported with code 43 in the device manager? However the dedicated nvidia device is working normally.

Offline

#4709 2015-04-04 13:24:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Is it normal for windows guests, that the red hat QXL graphics device is reported with code 43 in the device manager? However the dedicated nvidia device is working normally.

It's strange, but totally fine.
Actually, you can just disconnect it totally.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4710 2015-04-04 14:42:56

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

darkstego wrote:

Hey guys,

I was able to get Windows 8.1 running with VFIO and OVMF, but I have an odd issue.

After the windows install, OVMF will not boot windows if I remove the drive with the install media

Here is my running script.

sudo qemu-system-x86_64 \ -L $HOME/Desktop/ovmf -bios bios.bin \ ... -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \ -drive file=/dev/sdb,id=disk_primary,format=raw,if=none -device scsi-hd,drive=disk_primary \ ...

The trouble is if I omit the

 -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install 

line or move it after the disk drive the EFI will not find anything to boot.

Please stop using -L and -bios. Refer to the wiki or the example in the whitepaper wrt. the pflash drives instead.

Please capture the debug output with "-debugcon file:debug.log -global isa-debugcon.iobase=0x402"; if there's a problem with processing QEMU's "bootorder" fw_cfg file, the debug log should help us understand it.

Also, how recent is your OVMF build?

Offline

#4711 2015-04-04 14:46:21

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:

Any idea why this command:

virt-install --name=win7 --os-type=windows --disk path=/dev/mapper/VMssd-win7 --cdrom=/vm/win7x64.iso --graphics spice --ram=2048 --boot loader_type=pflash,loader_ro=yes,loader=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd,nvram_template=/vm/win7_VARS-pure-efi.fd

Results in this:

ERROR Unknown options ['loader_ro', 'nvram_template', 'loader_type']

Your virt-manager (or virt-install) package (or build) is too old.

Offline

#4712 2015-04-04 14:57:07

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:

@erganzi

-boot menu=on[,splash-time=$timeout_ms]  ?

Try:
-boot menu=on,splash-time=20000

It still display nothing, just black screen, then turned into windows startup.

Please describe the symptoms in more detail: command line or domain XML, OVMF debug log (see capture notes above), what displays are configured, which one is the display that you see stuff on, etc.

Also, I doubt qemu 2.1.3 is recent enough (Alex will know better); as far as I remember, for my Nvidia GTX750 it was too old. I needed commit fe08275d, which implies 2.2.

Other than that, OVMF indeed supports '-boot menu=on[,splash-time=$timeout_ms]'. The behavior is not identical to that of SeaBIOS (as the qemu manual explains, these options have firmware-specific behavior), but it should be a close approximation. See https://github.com/tianocore/edk2/commit/9253c14d and its surrounding commits.

Offline

#4713 2015-04-04 15:12:52

larrabee
Member
Registered: 2014-08-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You sure you enabled your acs override mode? There's a kernel command line parameter for that:
...
And are you sure you need that "allow_unsafe_interrupts=1"?
Did you read aw's blog post about it?(question 8)

Duelist, thank you.
Option pcie_acs_override=downstream resolve it. Now SAS and video card in different groups.
Thank you.

Offline

#4714 2015-04-04 15:14:21

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Offline

#4715 2015-04-04 15:25:37

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

i couldn't make my VM output "starting windows" message on the main GPU

If you mean the colorful animation: I think that's presented by the windows 7 / windows server 2008 r2 boot loader, which is a UEFI application (or else, it could be early runtime OS code too, but it certainly inherits the video framebuffer from one of the GOPs (graphics output protocols) installed before ExitBootServices()).

It looks like this animation code only uses the first GOP that it finds. The order in which it finds GOPs is unspecified. You could experiment with replacing the guest-visible PCI addresses that are assigned to QXL and to the passthru card with each other on the qemu command line. That might invert the order in which these devices are enumerated in the firmware, and consequently the order in which their respective drivers install their GOPs.

I never tried this -- as you say there's some "blank screen time" to endure on the passthru display, but it isn't overly annoying.

Offline

#4716 2015-04-04 16:19:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
Duelist wrote:

i couldn't make my VM output "starting windows" message on the main GPU

If you mean the colorful animation: I think that's presented by the windows 7 / windows server 2008 r2 boot loader, which is a UEFI application (or else, it could be early runtime OS code too, but it certainly inherits the video framebuffer from one of the GOPs (graphics output protocols) installed before ExitBootServices()).

It looks like this animation code only uses the first GOP that it finds. The order in which it finds GOPs is unspecified. You could experiment with replacing the guest-visible PCI addresses that are assigned to QXL and to the passthru card with each other on the qemu command line. That might invert the order in which these devices are enumerated in the firmware, and consequently the order in which their respective drivers install their GOPs.

I never tried this -- as you say there's some "blank screen time" to endure on the passthru display, but it isn't overly annoying.

Well, i kinda knew that it grabs the first GPU it finds. Nice idea of switching the addresses that way real GPUs would be presented first.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4717 2015-04-04 17:52:36

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Yes, I use it because it is recommended for best IO performance and I havn't had any problems when I was using a seabios VM. If dataplane would be the cause of this problem in the OVMF VM, how much would performance decrease if I switch it off?

Offline

#4718 2015-04-04 19:24:14

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
lersek wrote:
apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Yes, I use it because it is recommended for best IO performance and I havn't had any problems when I was using a seabios VM. If dataplane would be the cause of this problem in the OVMF VM, how much would performance decrease if I switch it off?

I have no reason to imply that dataplane was the "culprit" (or just "trigger") here; I only raised it because it was the only "unusual" option to my eye, and because I vaguely recall the dataplane code having/being a separate "virtio server" implementation. (But, this may not be true any longer, and it may not have been true ever.)

I never used dataplane myself (and I certainly never tried to test it with any remotely "scientific" method), so I can't give you numbers, unfortunately.

Anyway it shouldn't be hard to test your setup with dataplane off.

Offline

#4719 2015-04-05 22:21:29

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Need some help. I have read most of the thread; however, cannot shake this.

The error: vfio: error no iommu_group for device

# qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -vga none qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device # dmesg | grep probe [ 478.084005] vfio-pci: probe of 0000:02:00.0 failed with error -22

1. VT-d is supported and enabled in my bios [ASUS P9x79 LE (bios 4801/4701) i7-3930k (C2 stepping)].
2. Intel io_mmu is enabled.
3. pci-stub has blocked nouveau from taking control of the card
4. I am using pcie_acs_override=downstream

# dmesg | grep -e DMAR -e IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
#dmesg | grep pci-stub [ 0.700460] pci-stub 0000:02:00.0: claimed by stub [ 0.700501] pci-stub 0000:02:00.1: claimed by stub
# dmesg | grep -i DMAR #

Using the linux-vfio kernel out of the AUR: BOOT_IMAGE=/boot/vmlinuz-linux-vfio
I believe my kernel boot flags are correct:

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:1184,10de:0e0a emulate_invalid_guest_state=0

I have checked the following modules have loaded:

# cat /etc/modules-load.d/vga-passthrough.conf pci_stub vfio vfio_iommu_type1 vfio_pci kvm kvm-intel

I am not using the i915 pach because I do not have Intel graphics. I am using a GTX 560ti as primary, trying to pass through a GTX 770.

I have read through most of the thread and tons of pages on the internet, and almost everyone who has seen this problem either 1) cpu/MB did not support vt-d or 2) forgot to enable it in bios or via kernel option. I have tried many, many variations of fixes; however, nothing.

What am I missing here?

Last edited by logix (2015-04-05 22:52:38)

Offline

#4720 2015-04-05 23:15:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@logix

The IOMMU is not enabled, your first grep should produce lots more output.  Does /sys/firmware/acpi/tables/DMAR exist on your system?  A DMAR reference should have shown in the grep.  Without a DMAR table, VT-d cannot be configured, which suggests a BIOS issue.

EDIT: I've seen cases where a hard power cycle is required after enabling VT-d in the BIOS to make it actually take effect.

Last edited by aw (2015-04-05 23:17:36)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4721 2015-04-05 23:25:39

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@logix

The IOMMU is not enabled, your first grep should produce lots more output.  Does /sys/firmware/acpi/tables/DMAR exist on your system?  A DMAR reference should have shown in the grep.  Without a DMAR table, VT-d cannot be configured, which suggests a BIOS issue.

EDIT: I've seen cases where a hard power cycle is required after enabling VT-d in the BIOS to make it actually take effect.

I do not have a DMAR reference

$ sudo ls -al /sys/firmware/acpi/tables/DMAR ls: cannot access /sys/firmware/acpi/tables/DMAR: No such file or directory

I had a feeling it was a problem with the firmware, and I guess that confirms it. I have hard-reset many times. In fact, disabling and enabling VT-d does a hard reset automatically. I will try downgrading the bios-- as there is a post with it working for old bios.

Thanks!

Edit: Tried 4 different bios dating from 1/2013 to most current. Cannot get DMAR tables to appear. Sad.

Last edited by logix (2015-04-06 14:45:01)

Offline

#4722 2015-04-06 07:03:10

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
apex8 wrote:
lersek wrote:

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Yes, I use it because it is recommended for best IO performance and I havn't had any problems when I was using a seabios VM. If dataplane would be the cause of this problem in the OVMF VM, how much would performance decrease if I switch it off?

I have no reason to imply that dataplane was the "culprit" (or just "trigger") here; I only raised it because it was the only "unusual" option to my eye, and because I vaguely recall the dataplane code having/being a separate "virtio server" implementation. (But, this may not be true any longer, and it may not have been true ever.)

I never used dataplane myself (and I certainly never tried to test it with any remotely "scientific" method), so I can't give you numbers, unfortunately.

Anyway it shouldn't be hard to test your setup with dataplane off.

I switched off x-data-plane...

-drive if=none,file=/dev/sdb3,id=vdisk1,cache=none,format=raw,aio=native \ -device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=off \

...but the result was the same.

qemu-system-x86_64: Guest moved used index from 14 to 0
dmesg [ 139.746997] device tap0 entered promiscuous mode [ 139.747123] kvmbr0: port 2(tap0) entered listening state [ 139.747131] kvmbr0: port 2(tap0) entered listening state [ 139.747490] qemu-system-x86: sending ioctl 5326 to a partition! [ 139.747523] qemu-system-x86: sending ioctl 80200204 to a partition! [ 140.179757] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 140.179889] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 [ 140.181864] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002) [ 140.183020] vfio-pci 0000:03:00.0: enabling device (0100 -> 0102) [ 144.698904] kvm: zapping shadow pages for mmio generation wraparound [ 153.360553] kvmbr0: port 2(tap0) entered disabled state [ 153.360595] device tap0 left promiscuous mode [ 153.360601] kvmbr0: port 2(tap0) entered disabled state

Is it possible to gather some more data to find out whats happening in the background?

Offline

#4723 2015-04-06 15:38:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
qemu-system-x86_64: Guest moved used index from 14 to 0

I don't think this is as unusual as Laszlo seems to think it is.  I was seeing it roughly one in every three boots, but it doesn't seem like I can reproduce it anymore.  I'd try updating kernel and QEMU.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4724 2015-04-06 15:42:10

darkstego
Member
Registered: 2014-08-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
darkstego wrote:

Hey guys,

I was able to get Windows 8.1 running with VFIO and OVMF, but I have an odd issue.

After the windows install, OVMF will not boot windows if I remove the drive with the install media

Here is my running script.

sudo qemu-system-x86_64 \ -L $HOME/Desktop/ovmf -bios bios.bin \ ... -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \ -drive file=/dev/sdb,id=disk_primary,format=raw,if=none -device scsi-hd,drive=disk_primary \ ...

The trouble is if I omit the

 -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install 

line or move it after the disk drive the EFI will not find anything to boot.

Please stop using -L and -bios. Refer to the wiki or the example in the whitepaper wrt. the pflash drives instead.

Please capture the debug output with "-debugcon file:debug.log -global isa-debugcon.iobase=0x402"; if there's a problem with processing QEMU's "bootorder" fw_cfg file, the debug log should help us understand it.

Also, how recent is your OVMF build?

Thanks,

The reason I am using the -L and -bios options is it happened to be the only one I could get to work for the install.

I have now switched back to pflash and it seems the missing part was  "bootindex=0" option added to the primary disk.

Everything works well now.

Offline

#4725 2015-04-06 17:03:11

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can't use keyboard before windows begins to load (after I can use it without problems), but I need it to press button to load from DVD.

Nevermind, with qemu (not -git) works ok

Last edited by walkindude (2015-04-06 17:11:18)

Offline

#4726 2015-04-06 17:12:59

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

walkindude wrote:

Can't use keyboard before windows begins to load (after I can use it without problems), but I need it to press button to load from DVD.

Have a look at the arch wiki pages for qemu and look for information about setting up a spice server. You will get an empty black window, which makes you able to press keys in the VM. The other option is to use VNC in your setup and connect to the VM via a VNC client.

Offline

#4727 2015-04-06 17:27:23

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@apex8
Don't want to use VNC. Besides it's working for now, so I guess I stick to qemu (without -git)
But more serious problem remains: sometimes it cannot boot at all, just seabios screen -> black for 1-2 second -> seabios screen -> black -> seabios... If I then use hard reset, and then turn VM on again windows boots as expected. It happening on 2.2.1 and on git version.

Offline

#4728 2015-04-06 17:31:46

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@logix

The IOMMU is not enabled, your first grep should produce lots more output.  Does /sys/firmware/acpi/tables/DMAR exist on your system?  A DMAR reference should have shown in the grep.  Without a DMAR table, VT-d cannot be configured, which suggests a BIOS issue.

EDIT: I've seen cases where a hard power cycle is required after enabling VT-d in the BIOS to make it actually take effect.

logix wrote:

I do not have a DMAR reference

$ sudo ls -al /sys/firmware/acpi/tables/DMAR ls: cannot access /sys/firmware/acpi/tables/DMAR: No such file or directory

I had a feeling it was a problem with the firmware, and I guess that confirms it. I have hard-reset many times. In fact, disabling and enabling VT-d does a hard reset automatically. I will try downgrading the bios-- as there is a post with it working for old bios.

Thanks!

Edit: Tried 4 different bios dating from 1/2013 to most current. Cannot get DMAR tables to appear. Sad.


I think that this info is wrong. If I recall correctly, ACPI DMAR Tables are for the AMD IOMMU implementation, AMD-Vi. Intel ACPI tables are called IVRS. Basically, you should not have DMAR on Intel or IVRS on AMD, it makes no sense.


About your vfio: error no iommu_group for device, if my common sense is right, you should check if you have populated the /sys/kernel/iommu_groups location. Try this:

find /sys/kernel/iommu_groups/

It should return lots of results, something like...

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1

If your iommu_groups directory is not populated, maybe you require something else in your Boot Loader besides intel_iommu=on. There were a few more parameters that some guys required to have the iommu_groups directory populated, one was related to unsafe interrupts or something like that.

Offline

#4729 2015-04-06 17:35:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zir_blazer wrote:

I think that this info is wrong. If I recall correctly, ACPI DMAR Tables are for the AMD IOMMU implementation, AMD-Vi. Intel ACPI tables are called IVRS. Basically, you should not have DMAR on Intel or IVRS on AMD, it makes no sense.

You have this completely backwards.

If your iommu_groups directory is not populated, maybe you require something else in your Boot Loader besides intel_iommu=on. There were a few more parameters that some guys required to have the iommu_groups directory populated, one was related to unsafe interrupts or something like that.

allow_unsafe_interrupts has absolutely no bearing on whether iommu_groups is populated.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4730 2015-04-06 17:43:32

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
zir_blazer wrote:

I think that this info is wrong. If I recall correctly, ACPI DMAR Tables are for the AMD IOMMU implementation, AMD-Vi. Intel ACPI tables are called IVRS. Basically, you should not have DMAR on Intel or IVRS on AMD, it makes no sense.

You have this completely backwards.

It seems you're right. Intel mentions DMAR and AMD IVRS, I got it backwards.

https://www.kernel.org/doc/Documentatio … -IOMMU.txt
http://ubuntuforums.org/showthread.php?t=2254677

Offline

#4731 2015-04-06 19:10:23

SoftWinds
Member
Registered: 2015-04-02
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to passthrough my nVidia 970 card to my Windows guest using OVMF, and everything's working perfectly except for one weird thing: my mainboard is a Supermicro X10SAE, and I've configured it to use my integrated GPU (Intel P4600) as the main GPU, and pci-stub to stub out the GTX 970. This all works fine if I don't have the HDMI cable connect to the nVidia card when the computer boots up; however if it's connected, weird thing starts happening - it seems that nVidia "hijacks" the boot process somehow - GRUB doesn't display the entries correctly, and Arch starts showing on the HDMI output even though it shouldn't. The OS can't even get to show the display manager in this case, it just stays in the console. Does anyone have any suggestions? Thanks!

Offline

#4732 2015-04-06 20:58:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SoftWinds wrote:

I've managed to passthrough my nVidia 970 card to my Windows guest using OVMF, and everything's working perfectly except for one weird thing: my mainboard is a Supermicro X10SAE, and I've configured it to use my integrated GPU (Intel P4600) as the main GPU, and pci-stub to stub out the GTX 970. This all works fine if I don't have the HDMI cable connect to the nVidia card when the computer boots up; however if it's connected, weird thing starts happening - it seems that nVidia "hijacks" the boot process somehow - GRUB doesn't display the entries correctly, and Arch starts showing on the HDMI output even though it shouldn't. The OS can't even get to show the display manager in this case, it just stays in the console. Does anyone have any suggestions? Thanks!

Maybe it's your HDMI audio function? Is it stubbed too?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4733 2015-04-07 14:07:27

SoftWinds
Member
Registered: 2015-04-02
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
SoftWinds wrote:

I've managed to passthrough my nVidia 970 card to my Windows guest using OVMF, and everything's working perfectly except for one weird thing: my mainboard is a Supermicro X10SAE, and I've configured it to use my integrated GPU (Intel P4600) as the main GPU, and pci-stub to stub out the GTX 970. This all works fine if I don't have the HDMI cable connect to the nVidia card when the computer boots up; however if it's connected, weird thing starts happening - it seems that nVidia "hijacks" the boot process somehow - GRUB doesn't display the entries correctly, and Arch starts showing on the HDMI output even though it shouldn't. The OS can't even get to show the display manager in this case, it just stays in the console. Does anyone have any suggestions? Thanks!

Maybe it's your HDMI audio function? Is it stubbed too?

Yes it is stubbed as well. I also use it to output audio in the VM so I know it's passed through correctly.

Offline

#4734 2015-04-07 14:35:09

dhiru1602
Member
Registered: 2015-01-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Offline

#4735 2015-04-07 14:55:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dhiru1602 wrote:

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Maybe an interesting endeavor, if your time and electricity are free.  A 9600GT is not an impressive card anymore.  Conservatively, a GT740 has 2x the passmark score of a 9600GT, runs about 2/3 the power of a single 9600GT, and can be purchased for under $100.  For not much more, something like a GTX750 doubles the performance again, at a lower power point and better performance per dollar margin.  It's fun to use the hardware you've got laying around, but there's all sorts of unknown proprietary bits of Nvidia SLI and in this case it just doesn't seem worth it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4736 2015-04-07 21:12:05

dhiru1602
Member
Registered: 2015-01-28
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
dhiru1602 wrote:

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Maybe an interesting endeavor, if your time and electricity are free.  A 9600GT is not an impressive card anymore.  Conservatively, a GT740 has 2x the passmark score of a 9600GT, runs about 2/3 the power of a single 9600GT, and can be purchased for under $100.  For not much more, something like a GTX750 doubles the performance again, at a lower power point and better performance per dollar margin.  It's fun to use the hardware you've got laying around, but there's all sorts of unknown proprietary bits of Nvidia SLI and in this case it just doesn't seem worth it.

I totally agree with you. I stopped gaming after I moved permanently to linux few years ago and I haven't considered upgrading my GPU. Thanks to your great work with vfio, I was able to passthrough my GPU to a virtual machine and combined with a monitor and keyboard mouse pair, I was able to create a second standalone PC for family use and save electricity. wink

Putting together a PC with a beefy hardware for linux finally makes sense. Thanks to you smile

Last edited by dhiru1602 (2015-04-07 21:13:21)

Offline

#4737 2015-04-07 22:26:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dhiru1602 wrote:
aw wrote:
dhiru1602 wrote:

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Maybe an interesting endeavor, if your time and electricity are free.  A 9600GT is not an impressive card anymore.  Conservatively, a GT740 has 2x the passmark score of a 9600GT, runs about 2/3 the power of a single 9600GT, and can be purchased for under $100.  For not much more, something like a GTX750 doubles the performance again, at a lower power point and better performance per dollar margin.  It's fun to use the hardware you've got laying around, but there's all sorts of unknown proprietary bits of Nvidia SLI and in this case it just doesn't seem worth it.

I totally agree with you. I stopped gaming after I moved permanently to linux few years ago and I haven't considered upgrading my GPU. Thanks to your great work with vfio, I was able to passthrough my GPU to a virtual machine and combined with a monitor and keyboard mouse pair, I was able to create a second standalone PC for family use and save electricity. wink

Putting together a PC with a beefy hardware for linux finally makes sense. Thanks to you smile

Using two 9600GTs you could... do three PCs in one, but you'll probably will run out of memory or CPU power or some other IOMMU-related resource like iommu groups.

Also, seems like amd crossfire works overall better than nvidia SLI, considering the fact that back in the day you would have to install some hypervisor-like software called HyperSLI on a real hardware platform, let alone virtual machine.

Offtopic:
just accidentally found out that spice or VNC passes through ACPI power button event. Seems like my virtual PC works so well and integrated, i began "blurring the edge between virtuality and reality".

Last edited by Duelist (2015-04-07 22:30:10)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4738 2015-04-07 22:52:44

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, is it possible to have a "passed through" physical hard drive without any virtual disk drive/image? Basically, instead of having an OS raw or qcow2 image file on the host's hard disk/filesystem, or instead of even passing an entire block device (drive /dev/sda for example), can we attach a physical hard drive to a motherboard's 2nd SATA controller, and pass through the SATA controller to the guest and have all disk operations performed by the guest alone through that controller? Will OVMF and the guest see the hard disk and be able to install OS, boot off it, etc?

Offline

#4739 2015-04-07 23:09:10

SoftWinds
Member
Registered: 2015-04-02
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Actually I do have another question, but I'm not really sure if it's related to GPU passthrough or just general KVM/QEMU: When I start the VM the first time everything's fine, but if I shut down the VM and try to start it again I run into the error:

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Here's my qemu command:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/mapper/VolGroup00-vm,id=disk,format=raw,if=none \ -device scsi-hd,drive=disk \ -net nic -net bridge,br=br0 \ -rtc base=localtime,clock=host \ -vga none

Offline

#4740 2015-04-07 23:56:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SoftWinds wrote:

Actually I do have another question, but I'm not really sure if it's related to GPU passthrough or just general KVM/QEMU: When I start the VM the first time everything's fine, but if I shut down the VM and try to start it again I run into the error:

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Sounds like you're hitting a memory leak, what kernel/qemu versions are you using?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4741 2015-04-08 00:00:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:

Just curious, is it possible to have a "passed through" physical hard drive without any virtual disk drive/image? Basically, instead of having an OS raw or qcow2 image file on the host's hard disk/filesystem, or instead of even passing an entire block device (drive /dev/sda for example), can we attach a physical hard drive to a motherboard's 2nd SATA controller, and pass through the SATA controller to the guest and have all disk operations performed by the guest alone through that controller? Will OVMF and the guest see the hard disk and be able to install OS, boot off it, etc?

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4742 2015-04-08 00:03:45

SoftWinds
Member
Registered: 2015-04-02
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
SoftWinds wrote:

Actually I do have another question, but I'm not really sure if it's related to GPU passthrough or just general KVM/QEMU: When I start the VM the first time everything's fine, but if I shut down the VM and try to start it again I run into the error:

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Sounds like you're hitting a memory leak, what kernel/qemu versions are you using?

I'm running qemu 2.2.1-2 on Arch with kernel 3.19.2-1. I think it's a memory leak as well, since I checked all the process in htop and nothing seems out of the ordinary, and when I sum the memory usage up it's nowhere near enough to fill up my RAM. Is there anything I can do to debug this?

By the way, thanks aw for your work on this!

Last edited by SoftWinds (2015-04-08 00:05:23)

Offline

#4743 2015-04-08 20:00:51

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to figure out whether or not my Nvidia 780 supports EFI. I would think it would, but when I try to dump the rom:

$ cd /sys/bus/pci/devices/000:02:00.0 $ echo 1 > rom $ cat rom cat: rom: Input/output error $ dmesg | tail ... [ 4850.407367] pci-stub 0000:02:00.0: Invalid ROM contents

The card has been taken as a stub:

$ dmesg | grep stub [ 0.435187] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.435192] pci-stub 0000:02:00.0: claimed by stub [ 0.435195] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.435199] pci-stub 0000:02:00.1: claimed by stub

My relevant kernel boot parameters:

$ cat /proc/cmdline intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a pcie_acs_override=downstream

And my lspci:

$ lspci 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04) 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d4) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4) 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1) 04:00.0 Network controller: Qualcomm Atheros AR9462 Wireless Network Adapter (rev 01) 05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 06:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

In the past I've been able to successfully do the hacky VGA-based passthrough, but I'd like to use the OVMF/EFI method instead.

Offline

#4744 2015-04-08 20:05:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:

I'm trying to figure out whether or not my Nvidia 780 supports EFI. I would think it would, but when I try to dump the rom:

There's no harm in just trying it and it seems likely that it would.  If you get the TianoCore bootsplash, the answer is yes.

Last edited by aw (2015-04-08 20:05:37)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4745 2015-04-08 22:21:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity wrote:
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] 02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)

In the past I've been able to successfully do the hacky VGA-based passthrough, but I'd like to use the OVMF/EFI method instead.

I'd say, both of your GPUs support UEFI.
EFI GOP driver part of my VBIOS is taken from barts GPU vbios.

There are some problems with nvidia ROMs, like pre-POST/post-POST ROMs and stuff. You could actually try lurking in TPU's VBIOS database or try fetching the VBIOS from windows using GPU-z software, i've heard recent versions of it can handle stuff well.

Last edited by Duelist (2015-04-08 22:22:46)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4746 2015-04-09 04:30:43

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

I gave this a shot today just to experiment. I connected a spare 1TB SATA drive to the motherboard's Marvell SATA controller (non-chipset 2nd controller) and passed that through qemu to play around with. OVMF shows no signs of seeing a drive/controller (not sure if I should even be seeing anything really), but booting off a Windows install ISO shows the hard drive perfectly fine, and allows me to install to it. However, after the install completes and the guest system reboots, OVMF just dumps back at the Shell> prompt. OVMF's Boot Manager shows a "Windows Boot Manager" entry with device path: HD(2,GPT,UUID#)/\EFI\Microsoft\Boot\bootmgfw.efi so it seems like Windows at least installed boot information to the drive and to the firmware VARS I think? Not sure what to do from here or if it is a dead end. Maybe others have experimented with trying something like this (this is just all out of curiosity's sake). Probably has something to do with HD index showing as being at 2, wonder if this could be manually edited.

Last edited by mutiny (2015-04-09 04:44:03)

Offline

#4747 2015-04-09 10:59:40

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I encountered small problem when reinstalling my hypervisor OS recently, hope someone can point me to most robust solution.

My setup is basically headless host running two guest Windows 7 instances, each with own GPU and other peripherals (including native sound). One of guest instances takes primary GPU and the other takes second; hypervisor is controlled via serial console and/or ssh. The problem seems to be that "basically headless" is not as good as (planned) "actually headless".

What seems to be happening is that pci-stub is unable to attach itself to primary GPU because it is used by vgaarb early on kernel startup. Because pci-stub is unable to attach itself to primary GPU, snd_hda_intel comes along later and grabs HDMI audio output of this card (readeon is blacklisted), and this leads to all sort of trouble when guest starts up. Of course simple configuration tweak is to blacklist this module (which I did) but I'd prefer it pci-stub just did what it was configured to do in the first place i.e. prevent any driver from attaching itself to primary GPU (e.g. just in case some other driver came along in newer Linux version).

I think for this I need to disable vgaarb or set configuration such that vgaarb "knows" that the machine is meant to be running headless and no VGA output is available to host OS. Solutions coming to my mind are:
* disable CONFIG_VGA_CONSOLE in configuration and recompile kernel
* or disable CONFIG_VGA_ARB (same drawback as above)
* or set "install vgaarb /bin/false" in /etc/modprobe.d/ (assuming vgaarb can be built as module?)

I wonder what is the right way to approach the problem; ideally I'd rather use configuration thus making my setup less dependent on correct set of configuration switches when building the kernel. If only I could somehow set kernel parameters such as to force screen_info.orig_video_isVGA=0 ... perhaps boot parameter "earlyprintk=serial,..." might help?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4748 2015-04-09 12:58:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

vgaarb is not a "device driver" and cannot be built as a module.  It's a kernel service.  It's not consuming resources of the device and won't prevent pci-stub from attaching.  Have you tried building CONFIG_PCI_STUB=y?  If you build pci-stub as a module then you need to go through extra effort to make sure it loads before other modules.  See the modprobe.d softdep option for example.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4749 2015-04-09 13:48:46

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Bronek

vgaarb is not a "device driver" and cannot be built as a module.  It's a kernel service.  It's not consuming resources of the device and won't prevent pci-stub from attaching.  Have you tried building CONFIG_PCI_STUB=y?  If you build pci-stub as a module then you need to go through extra effort to make sure it loads before other modules.  See the modprobe.d softdep option for example.

Thanks aw; yes I build with CONFIG_PCI_STUB=y and yet somehow option to claim my primary GPU is ignored by pci-stub while all other PCI resources are claimed with no problems (USB cards, secondary GPU, motherboard sound, motherboard USB). I can use primary GPU for passthrough despite it, but this does not seem very robust. From dmesg I can see that vgaarb makes this GPU its default and I thought this is related. My kernel is 3.18.11 , I will post more details later today when I get home. At this moment I'm tempted to build without VGA_ARB but I'd prefer runtime option instead if there is one ...


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4750 2015-04-09 15:40:27

Child_of_Sun
Member
Registered: 2014-07-16
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:
aw wrote:

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

I gave this a shot today just to experiment. I connected a spare 1TB SATA drive to the motherboard's Marvell SATA controller (non-chipset 2nd controller) and passed that through qemu to play around with. OVMF shows no signs of seeing a drive/controller (not sure if I should even be seeing anything really), but booting off a Windows install ISO shows the hard drive perfectly fine, and allows me to install to it. However, after the install completes and the guest system reboots, OVMF just dumps back at the Shell> prompt. OVMF's Boot Manager shows a "Windows Boot Manager" entry with device path: HD(2,GPT,UUID#)/\EFI\Microsoft\Boot\bootmgfw.efi so it seems like Windows at least installed boot information to the drive and to the firmware VARS I think? Not sure what to do from here or if it is a dead end. Maybe others have experimented with trying something like this (this is just all out of curiosity's sake). Probably has something to do with HD index showing as being at 2, wonder if this could be manually edited.

I try the "same" thing with my Motherbord Sata Controller it's a SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]. The Windows 8.1 DVD won't boot and the Windows 10 Tech Preview freezes.

Ovmf has no boot option for this controller, even when i use uefi-raid mode.

I hope someone can help with that.

Offline

#4751 2015-04-09 16:12:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek :

I am passing my primary GPU to a Windows VM successfully . I boot the host with UEFI , and also use OVMF for booting VM (Complete EFI setup) . What I did :

1 - Boot the host with 

video=efifb:off

2 - The GPU you want to passthrough must be first installed as a seconday GPU on the host , then use nvagetbios to dump the ROM from it .
3 - Edit your VM to use the ROM you just dumped .
4 - Shutdown the system and move the card to the first slot to become primary (In case your motherboard doesn't allow you to set which card to use as a primary) .
5 - Boot the host and test the VM .

I passed my GTX770 to a VM thus making my host headless .

Good luck .

Offline

#4752 2015-04-09 17:01:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:

...


OVMF doesn't boot from sata controllers yet,  there are some patches posted on the edk2 mailing list where you can find them, I also posted a link to a patched binary a few pages back, there's also ovmf-fishman-git on AUR which contains said patches

EDIT: OVMF-sata.tar.gz

Last edited by nbhs (2015-04-09 19:09:01)

Offline

#4753 2015-04-09 20:31:06

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@Bronek :
I am passing my primary GPU to a Windows VM successfully . I boot the host with UEFI , and also use OVMF for booting VM (Complete EFI setup) . What I did :

1 - Boot the host with 

video=efifb:off

that's a good hint. I am not using EFI, but I will try all of

nofb nomodeset video=vesafb:off

and see if any of those make any difference. As for the other points, I found that my card works just fine with its own ROM as initialised by computer BIOS, thus no need to fiddle.

Basically my problem is not that passthrough does not work on primary GPU (it does work, most of the time), but that despite this

 pci-stub.ids=. . .,1002:6749,1002:aa90,1002:6707,1002:aa80

the card is not claimed by pci-stub at boot, thus allowing other modules to claim it which caused me some trouble.

My dmesg says

[ 0.963355] pci 0000:03:00.0: [1002:6707] type 00 class 0x030000 [ 0.963367] pci 0000:03:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref] [ 0.963376] pci 0000:03:00.0: reg 0x18: [mem 0xdfd20000-0xdfd3ffff 64bit] [ 0.963382] pci 0000:03:00.0: reg 0x20: [io 0x7000-0x70ff] [ 0.963393] pci 0000:03:00.0: reg 0x30: [mem 0xdfd00000-0xdfd1ffff pref] [ 0.963426] pci 0000:03:00.0: supports D1 D2 [ 0.963468] pci 0000:03:00.1: [1002:aa80] type 00 class 0x040300 [ 0.963479] pci 0000:03:00.1: reg 0x10: [mem 0xdfd40000-0xdfd43fff 64bit] [ 0.963534] pci 0000:03:00.1: supports D1 D2 . . . [ 1.024920] vgaarb: setting as boot device: PCI:0000:03:00.0 [ 1.024922] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none [ 1.024933] vgaarb: device added: PCI:0000:82:00.0,decodes=io+mem,owns=none,locks=none [ 1.024938] vgaarb: loaded [ 1.024939] vgaarb: bridge control possible 0000:82:00.0 [ 1.024940] vgaarb: bridge control possible 0000:03:00.0 . . . [ 1.082113] pci 0000:03:00.0: Video device with shadowed ROM . . . [ 1.185892] pci-stub: add 1002:6749 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.185917] pci-stub 0000:82:00.0: claimed by stub [ 1.185929] pci-stub: add 1002:AA90 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.185947] pci-stub 0000:82:00.1: claimed by stub

Note that 1002:6707,1002:aa80 are not claimed. I assumed its vgaarb preventing pci-stub from claiming the card. Incorrectly as it turns out, but now I do not know what does it.

Denso wrote:

Good luck .

Thanks

Last edited by Bronek (2015-04-09 20:31:22)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4754 2015-04-09 21:10:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek:
Can you give us lspci -v output?

Because...

[ 0.373329] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.373447] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.373563] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none [ 0.373677] vgaarb: loaded [ 0.373751] vgaarb: bridge control possible 0000:04:00.0 [ 0.373825] vgaarb: bridge control possible 0000:02:00.0 [ 0.373900] vgaarb: bridge control possible 0000:01:00.0

and then...

[ 0.914831] pci-stub: add 1022:780D sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.914951] pci-stub 0000:00:14.2: claimed by stub [ 0.915030] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.915156] pci-stub 0000:01:00.0: claimed by stub [ 0.915236] pci-stub 0000:02:00.0: claimed by stub [ 0.915312] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.915432] pci-stub 0000:01:00.1: claimed by stub [ 0.915511] pci-stub 0000:02:00.1: claimed by stub [ 0.915588] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.915703] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

is what i see in logs on my system.
And it gets bound to pci-stub fine. Then vfio-bind systemd service starts, binding the devices to vfio-pci

pci-stub.ids=1022:780d,1002:683f,1002:aab0,1002:683f,1002:683f

Note: I have 2 identical cards, BUT, as you can see, i've made a typo(just noted, lol), you don't need to enter it multiple times. It's not possible to passthrough one of two identical cards without using a special pci-stub version which binds by PCI BDF address rather than ven_id:dev_id pair. Even if it didn't bind for the first time for some weird reason, pci-stub tries to bind it again.

Therefore, i'm not sure that your devices aren't bound to pci-stub or vfio-pci. Maybe they are?

BTW:

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Subsystem: ASUSTeK Computer Inc. Device aab0 Flags: bus master, fast devsel, latency 0, IRQ 15 Memory at fe360000 (64-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel

As you can see, i've got audio device bound as well.

EDIT:
An idea struck me suddenly.
Maybe you didn't update your grub config, containing the new kernel command-line?

su -c "grub2-mkconfig > /boot/grub/grub.cfg"

Try doing cat /proc/cmdline to find out what's your current kernel commandline.

Last edited by Duelist (2015-04-09 21:25:58)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4755 2015-04-09 21:35:30

Marius_Linux
Member
Registered: 2015-04-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, after a full week of trying to get this working, I don't know what to do anymore right now.
I'm running Fedora 21(latest Updates, Kernel 3.19.3-200.fc21.x86_64, QEMU qemu-2.1.3-3.fc21), trying to passthrough my two NVIDIA GTX 680(one is a Palit Jetstream, the other one a normal Gainward) to a VM, and I'm using OVMF.
My CPU is an Intel i7 4770 (without K). supporting VT-d/x.
Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), but after reboot of the VM, Windows reports them with Code 43, and they are not working without NVIDIA drivers.
This is my lspci -nn:

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09) 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09) 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0162] (rev 09) 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04) 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 [8086:1e3a] (rev 04) 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04) 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4) 00:1c.3 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 4 [8086:1e16] (rev c4) 00:1c.4 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 [8086:1e18] (rev c4) 00:1c.5 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 [8086:1e1a] (rev c4) 00:1c.6 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 [8086:1e1c] (rev c4) 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4) 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04) 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04) 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04) 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller [8086:1e22] (rev 04) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) 05:00.0 Ethernet controller [0200]: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe [14e4:16b1] (rev 10) 06:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01) 08:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa) 09:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] 0a:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042] 

I'm using virt-manager, so this is my config XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8.1</name> <uuid>e7d551c9-a77e-4a90-8974-9c881f7662ad</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>8</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader type='pflash'>/var/lib/libvirt/images/win81-OVMF.fd</loader> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/win8.1.qcow2'/> <target dev='hda' bus='ide'/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/marius/Downloads/de_windows_8_1_x64_dvd_2707227.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/win81-OVMF.fd'/> <target dev='sda' bus='usb'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-vga'/> <qemu:arg value='none'/> </qemu:commandline> </domain>

It produces this command:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name win8.1 -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+erms,+smep,+fsgsbase,+rdrand,+f16c,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,kvm=off -drive file=/var/lib/libvirt/images/win81-OVMF.fd,if=pflash,format=raw,unit=0 -m 4096 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid e7d551c9-a77e-4a90-8974-9c881f7662ad -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win8.1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/win8.1.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/home/marius/Downloads/de_windows_8_1_x64_dvd_2707227.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=1 -drive file=/var/lib/libvirt/images/win81-OVMF.fd,if=none,id=drive-usb-disk0,format=raw -device usb-storage,drive=drive-usb-disk0,id=usb-disk0,removable=off -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x7 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x8 -device vfio-pci,host=02:00.0,id=hostdev2,bus=pci.0,addr=0x9 -device vfio-pci,host=02:00.1,id=hostdev3,bus=pci.0,addr=0xa -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xb -vga none -msg timestamp=on 

Anybody who could help me please?
Many thanks in advance
Marius

Offline

#4756 2015-04-09 21:40:11

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

@Bronek:
Can you give us lspci -v output?

Here

# lspci -vn -s 0000:03:00.0 03:00.0 0300: 1002:6707 (prog-if 00 [VGA controller]) Subsystem: 1002:0b06 Flags: bus master, fast devsel, latency 0, IRQ 162 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at dfd20000 (64-bit, non-prefetchable) [size=128K] I/O ports at 7000 [size=256] Expansion ROM at dfd00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: radeon

as you can see, I'm running guest at this moment which is why module in use is vfio-pci. Before I started guest there was no driver (any) in use for primary card (radeon is blacklisted), while for the other card it was "Kernel driver in use: pci-stub" (consistent with dmesg)

Duelist wrote:

Because...

[ 0.373329] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.373447] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.373563] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none [ 0.373677] vgaarb: loaded [ 0.373751] vgaarb: bridge control possible 0000:04:00.0 [ 0.373825] vgaarb: bridge control possible 0000:02:00.0 [ 0.373900] vgaarb: bridge control possible 0000:01:00.0

that interesting, you do not have " vgaarb: setting as boot device". I guess that's because booting from EFI does not need vgaarb.

As for audio, I do not use HDMI (I dont even passthrough function .1), I have USB ODAC instead attached to one of USB cards passed through to guest. I like this sound much better smile

Duelist wrote:

EDIT:
An idea struck me suddenly.
Maybe you didn't update your grub config, containing the new kernel command-line?

that's exactly why I'm not using GRUB big_smile Syslinux here, much simpler.

Last edited by Bronek (2015-04-09 21:40:59)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4757 2015-04-09 22:11:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Duelist wrote:

@Bronek:
Can you give us lspci -v output?

Here

# lspci -vn -s 0000:03:00.0 03:00.0 0300: 1002:6707 (prog-if 00 [VGA controller]) Subsystem: 1002:0b06 Flags: bus master, fast devsel, latency 0, IRQ 162 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at dfd20000 (64-bit, non-prefetchable) [size=128K] I/O ports at 7000 [size=256] Expansion ROM at dfd00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: radeon

as you can see, I'm running guest at this moment which is why module in use is vfio-pci. Before I started guest there was no driver (any) in use for primary card (radeon is blacklisted), while for the other card it was "Kernel driver in use: pci-stub" (consistent with dmesg)

Duelist wrote:

Because...

[ 0.373329] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none [ 0.373447] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.373563] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none [ 0.373677] vgaarb: loaded [ 0.373751] vgaarb: bridge control possible 0000:04:00.0 [ 0.373825] vgaarb: bridge control possible 0000:02:00.0 [ 0.373900] vgaarb: bridge control possible 0000:01:00.0

that interesting, you do not have " vgaarb: setting as boot device". I guess that's because booting from EFI does not need vgaarb.

As for audio, I do not use HDMI (I dont even passthrough function .1), I have USB ODAC instead attached to one of USB cards passed through to guest. I like this sound much better smile

Duelist wrote:

EDIT:
An idea struck me suddenly.
Maybe you didn't update your grub config, containing the new kernel command-line?

that's exactly why I'm not using GRUB big_smile Syslinux here, much simpler.

[ 0.373252] vgaarb: setting as boot device: PCI:0000:01:00.0

Apparently, it got clipped. I do have vgaarb and i'm booting host with csm. Too lazy to redo.

Do cat /proc/cmdline. Maybe it's syslinux's fault. Also, i think, you can make a file in /etc/modprobe.d/ and put pci-stub options there.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4758 2015-04-09 22:24:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Marius_Linux wrote:

Hi, after a full week of trying to get this working, I don't know what to do anymore right now.
I'm running Fedora 21(latest Updates, Kernel 3.19.3-200.fc21.x86_64, QEMU qemu-2.1.3-3.fc21), trying to passthrough my two NVIDIA GTX 680(one is a Palit Jetstream, the other one a normal Gainward) to a VM, and I'm using OVMF.
My CPU is an Intel i7 4770 (without K). supporting VT-d/x.
Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), but after reboot of the VM, Windows reports them with Code 43, and they are not working without NVIDIA drivers.
This is my lspci -nn:

..snip.. 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) ..snip..

I'm using virt-manager, so this is my config XML:
..snip..

It produces this command:
..snip..

Anybody who could help me please?
Many thanks in advance
Marius

First of all - if you're going to try using SLI - there may be problems with 440fx being "compatible" with SLI. In the real, hardware world you must have "SLI-compatible" certified motherboard for SLI to work. Moreover, back in the day, you must've use HyperSLI (based on VT-d) third party software to make SLI work on non-SLI-certified motherboards. The software is outdated, but still available, i wonder if it will work inside a VM using nested virtualization. That would be crazy.
List of things you can do:
1. Try Q35 chipset model.
2. Provide us dmesg to see if there's correct interrupt assigning messages and all the needed stuff.
3. Feed GTX680s with ROMs dumped from them in case of "Invalid ROM contents" messages. Also you could get the needed ROMs here.
4. "Quadrify" your GPUs using modified vfio-pci with fake PCI IDs and (maybe) ROMs. You could quadrify it hardware-way, but that's too hardcore and pointless if the culprit is somewhere else. That'll make all the problems with drivers detecting hypervisor go away, as it's 100% legit to passthrough a Quadro device. That way you could use more fresh drivers.
5. As i can see, you have Z77 chipset. Lurk a few pages back, there was some guy with that chipset and some problems with nvidia. I don't remember what the problem was, actually.
6. Try using legacy KVM pci passthrough: here's an example of using it(driver option). It doesn't work with VGA, but since you're using OVMF anyway...
7. Wait for aw to come, heh.

Last edited by Duelist (2015-04-09 22:26:38)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4759 2015-04-09 22:47:57

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Do cat /proc/cmdline.

Thanks, that was a very good advice. It turns out I am an idiot (don't tell my family, they already know), I wrote new boot parameters in the wrong file.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4760 2015-04-09 22:55:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

I wrote new boot parameters in the wrong file.

Bronek wrote:

that's exactly why I'm not using GRUB big_smile Syslinux here, much simpler.

Kek.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4761 2015-04-10 00:11:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Marius_Linux wrote:

Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), ...

Is this a typo?  You only need pre-338.77 to avoid all hypervisor detection and pre-344.11 for the hyper-v stuff.  Try a newer driver.  Start with a single card for the guest.  Dump your <qemu:args> section, you don't need it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4762 2015-04-10 03:42:24

Slabity
Member
Registered: 2013-12-29
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I managed to successfully pass through my GPU using the OVMF method, but I'm struggling to get audio to work over ALSA. I am using libvirt, and have added the following to my VM config:

<qemu:commandline> <qemu:env name='QEMU_AUDIO_DRV' value='alsa'/> <qemu:env name='QEMU_ALSA_DAC_PERIOD_SIZE' value='170'/> <qemu:env name='QEMU_ALSA_DAC_BUFFER_SIZE' value='512'/> </qemu:commandline>

But my host does not show any ALSA client appearing. Any ideas where I should look?

Offline

#4763 2015-04-10 14:44:58

Marius_Linux
Member
Registered: 2015-04-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Marius_Linux wrote:

Hi, after a full week of trying to get this working, I don't know what to do anymore right now.
I'm running Fedora 21(latest Updates, Kernel 3.19.3-200.fc21.x86_64, QEMU qemu-2.1.3-3.fc21), trying to passthrough my two NVIDIA GTX 680(one is a Palit Jetstream, the other one a normal Gainward) to a VM, and I'm using OVMF.
My CPU is an Intel i7 4770 (without K). supporting VT-d/x.
Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), but after reboot of the VM, Windows reports them with Code 43, and they are not working without NVIDIA drivers.
This is my lspci -nn:

..snip.. 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1) 02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) ..snip..

I'm using virt-manager, so this is my config XML:
..snip..

It produces this command:
..snip..

Anybody who could help me please?
Many thanks in advance
Marius

First of all - if you're going to try using SLI - there may be problems with 440fx being "compatible" with SLI. In the real, hardware world you must have "SLI-compatible" certified motherboard for SLI to work. Moreover, back in the day, you must've use HyperSLI (based on VT-d) third party software to make SLI work on non-SLI-certified motherboards. The software is outdated, but still available, i wonder if it will work inside a VM using nested virtualization. That would be crazy.
List of things you can do:
1. Try Q35 chipset model.
2. Provide us dmesg to see if there's correct interrupt assigning messages and all the needed stuff.
3. Feed GTX680s with ROMs dumped from them in case of "Invalid ROM contents" messages. Also you could get the needed ROMs here.
4. "Quadrify" your GPUs using modified vfio-pci with fake PCI IDs and (maybe) ROMs. You could quadrify it hardware-way, but that's too hardcore and pointless if the culprit is somewhere else. That'll make all the problems with drivers detecting hypervisor go away, as it's 100% legit to passthrough a Quadro device. That way you could use more fresh drivers.
5. As i can see, you have Z77 chipset. Lurk a few pages back, there was some guy with that chipset and some problems with nvidia. I don't remember what the problem was, actually.
6. Try using legacy KVM pci passthrough: here's an example of using it(driver option). It doesn't work with VGA, but since you're using OVMF anyway...
7. Wait for aw to come, heh.

aw wrote:
Marius_Linux wrote:

Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), ...

Is this a typo?  You only need pre-338.77 to avoid all hypervisor detection and pre-344.11 for the hyper-v stuff.  Try a newer driver.  Start with a single card for the guest.  Dump your <qemu:args> section, you don't need it.

Thanks for your help you two, I was surprised how quick you answered :-)
I've just checked my Palit card using your tool, and as it seems, it doesn't have EFI support:

Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 1180, class: 030000 PCIR: revision 0, vendor revision: 1 Last image 

Currently, I am looking for a BIOS for this Card(and my Gainward GTX 680), which support EFI. Does anybody know where to look, or how to check if the ROM is compatible with my card?

Last edited by Marius_Linux (2015-04-10 14:45:36)

Offline

#4764 2015-04-10 17:04:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Marius_Linux wrote:

Currently, I am looking for a BIOS for this Card(and my Gainward GTX 680), which support EFI. Does anybody know where to look, or how to check if the ROM is compatible with my card?

Duelist wrote:

3. Feed GTX680s with ROMs dumped from them in case of "Invalid ROM contents" messages. Also you could get the needed ROMs here.

TechPowerUp GPU BIOS database.
Official site of your GPU vendor is also a good place to look for GPU bios updates. Since my GT610 does support UEFI, it's very strange that your GTX680 doesn't.
Something hints me that all 680 are alike to each other being reference designs, and they all can be quadrified and/or their VBIOS images are compatible.
Of course, vendor may change the reference design, ruining compatibility, but i haven't heard of that for 680s.

How to check.. Well, add it and observe. If the card will(for some weird and unlikely reason) start smoking - quick CTRL-C will save it. If you see Tianocore boot splash - everything's working fine.

<rom bar='on' file='/path/to/your/romfile.rom'/>

P.S.
AFAIR, SLI won't double your video memory, but XDMA CF will.

Last edited by Duelist (2015-04-10 17:06:08)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4765 2015-04-10 23:55:17

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Child_of_Sun wrote:
mutiny wrote:
aw wrote:

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

I gave this a shot today just to experiment. I connected a spare 1TB SATA drive to the motherboard's Marvell SATA controller (non-chipset 2nd controller) and passed that through qemu to play around with. OVMF shows no signs of seeing a drive/controller (not sure if I should even be seeing anything really), but booting off a Windows install ISO shows the hard drive perfectly fine, and allows me to install to it. However, after the install completes and the guest system reboots, OVMF just dumps back at the Shell> prompt. OVMF's Boot Manager shows a "Windows Boot Manager" entry with device path: HD(2,GPT,UUID#)/\EFI\Microsoft\Boot\bootmgfw.efi so it seems like Windows at least installed boot information to the drive and to the firmware VARS I think? Not sure what to do from here or if it is a dead end. Maybe others have experimented with trying something like this (this is just all out of curiosity's sake). Probably has something to do with HD index showing as being at 2, wonder if this could be manually edited.

I try the "same" thing with my Motherbord Sata Controller it's a SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]. The Windows 8.1 DVD won't boot and the Windows 10 Tech Preview freezes.

Ovmf has no boot option for this controller, even when i use uefi-raid mode.

I hope someone can help with that.

Try virtio-scsi block passthrough. I found a concise (libvirt based) example here: https://blueprints.launchpad.net/nova/+ … ce-mapping

From http://libvirt.org/formatdomain.html, "... generic SCSI commands from the guest are accepted and passed through to the physical device ...". This can be used for CD-ROM/DVD burning passthrough etc (http://wiki.qemu.org/Features/VirtioSCSI).

Also, OVMF's builtin virtio-scsi driver will be able to handle & boot off such devices. The qemu command line is:

 -device virtio-scsi-pci,id=scsi0 \ -drive file=/dev/disk/by-id/...,if=none,id=virtio-scsi-passthrough0,format=raw \ -device scsi-block,bus=scsi0.0,drive=virtio-scsi-passthrough0

Offline

#4766 2015-04-11 06:01:03

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I'm on ubuntu 14.10 x64 and I'm trying to get this to work. So far I run the qemu script and I get a small console where I can input commands to qemu, but no output on my display hooked up to my guest GPU, nor any errors.

My specs:

Gigabyte GA-Z97-HD3 (VT-d enabled)
Intel Core i7-4790K (intel ark says VT-d compatible)
Host GPU: Intel HD Graphics 4600
Guest GPU: MSI TwinFrozr GTX 770

output of "lspci -nn | grep NVIDIA"

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)


This is my grub command line options

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_acs_override=downstream intel_iommu=on,igfx_on i915.enable_hd_vgaarb=1"

I have nouveau blacklisted, I have the following in /etc/initramfs-tools/modules (And I updated initramfs after)

pci_stub ids=10de:1184,10de:0e0a

I ran the vfio-bind script with no errors. I ran it with

sudo ./vfio-bind 0000:01:00.0 0000:01:00.1

These are the contents of it

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

I run qemu with this script

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 \
-vga none -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

I don't get any visible errors. Just a blank, usable qemu console and no output at all on the display hooked up to the guest GPU.

Am I missing something?

Offline

#4767 2015-04-11 10:55:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

Hello, I'm on ubuntu 14.10 x64 and I'm trying to get this to work. So far I run the qemu script and I get a small console where I can input commands to qemu, but no output on my display hooked up to my guest GPU, nor any errors.

My specs:

Gigabyte GA-Z97-HD3 (VT-d enabled)
Intel Core i7-4790K (intel ark says VT-d compatible)
Host GPU: Intel HD Graphics 4600
Guest GPU: MSI TwinFrozr GTX 770

output of "lspci -nn | grep NVIDIA"

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

This is my grub command line options

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_acs_override=downstream intel_iommu=on,igfx_on i915.enable_hd_vgaarb=1"

..snip..
I run qemu with this script

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 \
-vga none -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

I don't get any visible errors. Just a blank, usable qemu console and no output at all on the display hooked up to the guest GPU.

Am I missing something?

Try checking dmesg. If there are any errors in KVM or VFIO work - it'll be there.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4768 2015-04-11 13:20:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

Am I missing something?

Whenever it's not explicitly mentioned, my first suspicion is whether the i915 patch is actually applied to the kernel.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4769 2015-04-11 13:43:26

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

These keep piling in dmesg :

[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018 [Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID) [Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: device [8086:2f08] error status/mask=00000040/00002000 [Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: [ 6] Bad TLP

Using 4.0-rc5 and also happened on previous kernels . When it starts appearing , any VM rebooting or even host rebooting causes the host to hard crash . This is on X99 platform , where 00:03.0 is a :

00:03.0 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) (prog-if 00 [Normal decode])

Offline

#4770 2015-04-11 13:50:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

These keep piling in dmesg :

[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018 [Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID) [Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: device [8086:2f08] error status/mask=00000040/00002000 [Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: [ 6] Bad TLP

You're getting bad packets on the link, the root port is the receiver of the packet.  This could indicate some sort of hardware problem.  Are you overclocking?  If you move the device behind this root port to another slot, do the errors follow it?  It could be a bad endpoint device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4771 2015-04-11 14:38:52

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You're getting bad packets on the link, the root port is the receiver of the packet.  This could indicate some sort of hardware problem.  Are you overclocking?  If you move the device behind this root port to another slot, do the errors follow it?  It could be a bad endpoint device.

So it is the GPU and not the port itself that causes the problem .

Let me understand this correctly . I have these root ports :

00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) 00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)

Are they numbered in a relative way to their physical position on the motherboard ? If that's the case , then 00:03.0 is the fourth slot from the top which is occupied by my GT610 (the card of legendary headaches thus far) .

Offline

#4772 2015-04-11 14:43:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

You're getting bad packets on the link, the root port is the receiver of the packet.  This could indicate some sort of hardware problem.  Are you overclocking?  If you move the device behind this root port to another slot, do the errors follow it?  It could be a bad endpoint device.

So it is the GPU and not the port itself that causes the problem .

Let me understand this correctly . I have these root ports :

00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) 00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)

Are they numbered in a relative way to their physical position on the motherboard ? If that's the case , then 00:03.0 is the fourth slot from the top which is occupied by my GT610 (the card of legendary headaches thus far) .

The slot to root port mapping depends on how your motherboard is wired.  There are ways that the firmware can provide more human readable slot names via ACPI, but they're often not used on desktop boards.  lspci -t is useful for seeing a tree view of the hardware for more easily mapping which endpoint address is behind which root port.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4773 2015-04-11 14:49:38

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The slot to root port mapping depends on how your motherboard is wired.  There are ways that the firmware can provide more human readable slot names via ACPI, but they're often not used on desktop boards.  lspci -t is useful for seeing a tree view of the hardware for more easily mapping which endpoint address is behind which root port.

Forgive me , but I couldn't make sense of this output :

-+-[0000:ff]-+-0b.0 | +-0b.1 | +-0b.2 | +-0c.0 | +-0c.1 | +-0c.2 | +-0c.3 | +-0c.4 | +-0c.5 | +-0f.0 | +-0f.1 | +-0f.4 | +-0f.5 | +-0f.6 | +-10.0 | +-10.1 | +-10.5 | +-10.6 | +-10.7 | +-12.0 | +-12.1 | +-13.0 | +-13.1 | +-13.2 | +-13.3 | +-13.4 | +-13.5 | +-13.6 | +-13.7 | +-14.0 | +-14.1 | +-14.2 | +-14.3 | +-14.6 | +-14.7 | +-15.0 | +-15.1 | +-15.2 | +-15.3 | +-16.0 | +-16.6 | +-16.7 | +-17.0 | +-17.4 | +-17.5 | +-17.6 | +-17.7 | +-1e.0 | +-1e.1 | +-1e.2 | +-1e.3 | +-1e.4 | +-1f.0 | \-1f.2 \-[0000:00]-+-00.0 +-01.0-[03-08]----00.0-[04-08]--+-01.0-[05]----00.0 | +-05.0-[06]----00.0 | +-07.0-[07]----00.0 | \-09.0-[08]----00.0 +-01.1-[09]-- +-02.0-[02]--+-00.0 | \-00.1 +-03.0-[01]--+-00.0 | \-00.1 +-05.0 +-05.1 +-05.2 +-05.4 +-11.0 +-11.4 +-14.0 +-16.0 +-19.0 +-1a.0 +-1b.0 +-1c.0-[0a]-- +-1c.3-[0b-10]----00.0-[0c-10]--+-01.0-[0d]----00.0 | +-02.0-[0e]----00.0 | +-03.0-[0f]-- | \-04.0-[10]-- +-1c.4-[11]----00.0 +-1c.6-[12]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3

Offline

#4774 2015-04-11 14:56:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
 \-[0000:00]-+-00.0 +-01.0-[03-08]----00.0-[04-08]--+-01.0-[05]----00.0 | +-05.0-[06]----00.0 | +-07.0-[07]----00.0 | \-09.0-[08]----00.0 +-01.1-[09]-- +-02.0-[02]--+-00.0 | \-00.1 +-03.0-[01]--+-00.0 | \-00.1 ... +-1c.0-[0a]-- +-1c.3-[0b-10]----00.0-[0c-10]--+-01.0-[0d]----00.0 | +-02.0-[0e]----00.0 | +-03.0-[0f]-- | \-04.0-[10]-- +-1c.4-[11]----00.0 +-1c.6-[12]----00.0

Devices 01:00.0 and 01:00.1 are behind root port 00:03.0


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4775 2015-04-11 15:02:17

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Ohhh I get it now ! The first number is the Port number , the numbers after it are the devices numbers . So devices 02:00.0 , 02:00.1 are behind Port 00:02.0 . Device 11:00.0 is behind 00:1c.4 , and so on .

Thanks Alex ! It is GTX770 then , which is the host's primary GPU . I'll try to troubleshoot it and post my findings if I ever got it fixed hopefully .

EDIT :

Rebooted the host , entered BIOS and loaded optimized defaults , also I added "nomodeset" &  "nofb" to the boot parameters . Uptil now , no errors in dmesg . Will see if it can survive overnight .

Last edited by Denso (2015-04-11 16:13:30)

Offline

#4776 2015-04-11 17:47:20

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any ideas on why I can't get IOMMU working? My original post: https://bbs.archlinux.org/viewtopic.php … 4#p1517314

I cannot get DMAR tables populated:

# ls /sys/firmware/acpi/tables/ APIC BGRT DSDT dynamic FACP FACS FPDT HPET MCFG SSDT

I have everything enabled as mentioned in previous post  . Only thing I can think of is hardware issues; however, I've switched motherboards twice.

i7-3930k w/ C2 Stepping (SR0KY)
ASUS P9x79 LE (Tried 4 different bios)
ASUS RAMPAGE IV GENE( Tried 3 different bios)

Internet research shows people have successfully used this hardware. Any ideas?

Last edited by logix (2015-04-11 17:49:41)

Offline

#4777 2015-04-11 18:13:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@logix

It's still a BIOS issue, nothing we can do here.  Maybe Asus doesn't detect the C2 stepping as having VT-d.  It is possible to pass tables into the kernel, but creating and passing that table is beyond the scope of this thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4778 2015-04-11 19:00:30

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone tried the UEFI new machine install on virt-manager? There's supposed to be a patch to allow selecting Overview|Firmware and then UEFI or BIOS while "Customizing before install" when making a new machine. All I see is the 440fx or Q35 choice using the latest libvirtd/virt-manager on Arch.

Offline

#4779 2015-04-11 19:40:29

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Anyone tried the UEFI new machine install on virt-manager? There's supposed to be a patch to allow selecting Overview|Firmware and then UEFI or BIOS while "Customizing before install" when making a new machine. All I see is the 440fx or Q35 choice using the latest libvirtd/virt-manager on Arch.

You should have the OVMF package installed, and its files (firmware binary and varstore template) listed as a pair in the "nvram" stanza of /etc/libvirtd/qemu.comf. IIRC virt-manager queries libvirtd, and the UEFI choice means the first entry in that list of pairs (assuming there is a pair). My memories might be off though.

If you edit qemu.conf, don't forget to restart libvirtd.

Offline

#4780 2015-04-11 19:54:51

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah, followed the instructions in the white paper, restarted, still no joy. Any other troubleshooting I can try?

Offline

#4781 2015-04-11 21:22:38

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@logix
My functioning system with a RIVE IV and a 4930K, bios 4901 shows the same output as you have, namely:

ls /sys/firmware/acpi/tables/ APIC BGRT DMAR DSDT dynamic FACP FACS FPDT HPET MCFG SSDT

so I don't think that indicates anything. However you should see something like

#dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 0x00000000AC4D3690 0000B4 (v01 A M I OEMDMAR 00000001 INTL 00000001) [ 0.000000] Intel-IOMMU: enabled [ 0.337843] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de [ 0.337938] IOAPIC id 0 under DRHD base 0xfbffc000 IOMMU 0 [ 0.337938] IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 0 [ 1.001689] IOMMU: dmar0 using Queued invalidation [ 1.001692] IOMMU: Setting RMRR: [ 1.001702] IOMMU: Setting identity map for device 0000:00:1a.0 [0xac39b000 - 0xac3a8fff] [ 1.001727] IOMMU: Setting identity map for device 0000:00:1d.0 [0xac39b000 - 0xac3a8fff] [ 1.001741] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 1.001749] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 10.885128] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com> [ 10.885129] AMD IOMMUv2 functionality not available on this system

Note in particular the first line.  Support for your hardware should be there since bios 1005, see http://rog.asus.com/61782011/news/uefi- … -and-gene/ although your bios could be up to date at 4901 as well.  The only other thing I see that's different is  "pcie_acs_override=downstream" which I don't use.  My kernel line has intel_iommu=on pci-stub.ids=10de:0ffe,10de:0e1b modprobe.blacklist=nouveau,nvidia.  The other weird thing is that your command

#dmesg | grep pci-stub [ 0.700460] pci-stub 0000:02:00.0: claimed by stub [ 0.700501] pci-stub 0000:02:00.1: claimed by stub

doesn't actually show pci-stub in your kernel command line.  Is there a typo somewhere? Extra whitespace? Or did you edit the output?
Here's the result on my machine:

dmesg | grep pci-stub [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=/dev/mapper/cryptvg2-arch rw intel_iommu=on pci-stub.ids=10de:0ffe,10de:0e1b modprobe.blacklist=nouveau,nvidia radeon.dpm=1 radeon.modeset=1 cryptdevice=/dev/sdb1:cryptvg2 cryptkey=/dev/sdc:vfat:/keyfile [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=/dev/mapper/cryptvg2-arch rw intel_iommu=on pci-stub.ids=10de:0ffe,10de:0e1b modprobe.blacklist=nouveau,nvidia radeon.dpm=1 radeon.modeset=1 cryptdevice=/dev/sdb1:cryptvg2 cryptkey=/dev/sdc:vfat:/keyfile [ 1.039022] pci-stub: add 10DE:0FFE sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.039030] pci-stub 0000:03:00.0: claimed by stub [ 1.039037] pci-stub: add 10DE:0E1B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 1.039045] pci-stub 0000:03:00.1: claimed by stub

You'd have to have horrible luck to have two bad motherboards, so a hardware problem seems unlikely. I am using the standard Arch kernel.

Offline

#4782 2015-04-11 21:39:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

@logix
My functioning system with a RIVE IV and a 4930K, bios 4901 shows the same output as you have, namely:

ls /sys/firmware/acpi/tables/ APIC BGRT DMAR DSDT dynamic FACP FACS FPDT HPET MCFG SSDT

You're missing that logix's output doesn't include the DMAR table, which is essential to VT-d functionality.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4783 2015-04-11 21:54:57

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, missed that; still hope the other suggestions are helpful.

Offline

#4784 2015-04-11 22:59:42

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've made a bit more progress from a blank output. I've tried OVMF and am now correctly passing through my GTX 770 and I'm getting a display!

The snag I'm hitting now is that I can only seem to boot into the default EFI shell. I have no idea what to do from there. I can't seem to boot from my Windows 8 ISO file. I tried typing "bootx64.efi" to see if I could manually trigger a boot but that didn't work.

This is my QEMU command

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
-vga none  -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -drive file=/home/anthony/virtio.iso,id=virtiocd,if=none \
-device ide-cd,bus=ide.1,drive=virtiocd

What am I missing? Anything obvious?

Offline

#4785 2015-04-11 23:19:12

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

I've made a bit more progress from a blank output. I've tried OVMF and am now correctly passing through my GTX 770 and I'm getting a display!

The snag I'm hitting now is that I can only seem to boot into the default EFI shell. I have no idea what to do from there. I can't seem to boot from my Windows 8 ISO file. I tried typing "bootx64.efi" to see if I could manually trigger a boot but that didn't work.

This is my QEMU command

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
-vga none  -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -drive file=/home/anthony/virtio.iso,id=virtiocd,if=none \
-device ide-cd,bus=ide.1,drive=virtiocd

What am I missing? Anything obvious?

All I can provide is my working Drives configuration , replace yours with these and see if it works :

-drive file=/home/anthony/windows.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/anthony/win8.iso,id=iso_install,if=none,format=raw \ -device scsi-cd,drive=iso_install \ -cdrom /home/anthony/virtio.iso \

When the UEFI shell appears , type "exit" and hit enter , you will see a menu choose "boot devices" if I remember correctly , then choose the SCSI device .

Hope you'd find your path around ! Good luck .

Offline

#4786 2015-04-12 00:01:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

I've made a bit more progress from a blank output. I've tried OVMF and am now correctly passing through my GTX 770 and I'm getting a display!

The snag I'm hitting now is that I can only seem to boot into the default EFI shell. I have no idea what to do from there. I can't seem to boot from my Windows 8 ISO file. I tried typing "bootx64.efi" to see if I could manually trigger a boot but that didn't work.

This is my QEMU command

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
-vga none  -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -drive file=/home/anthony/virtio.iso,id=virtiocd,if=none \
-device ide-cd,bus=ide.1,drive=virtiocd

What am I missing? Anything obvious?

Create a 4GB fat32 image and copy your windows installation files there, it should work

Offline

#4787 2015-04-12 03:30:45

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, I tried that, I didn't get very far though.

This is my new qemu command

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
-vga none \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/anthony/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

win8boot.img is a fat32 formatted image with all the files from the win8.iso copied to it.


By the way, when I boot the VM, I get msgs of it trying to boot from EFI floppy, EFI floppy 2, EFI DVD/CDROM and all failing.

It then tries to boot from network, fails, then drops me to an EFI shell.

Last edited by Helios747 (2015-04-12 03:31:03)

Offline

#4788 2015-04-12 03:38:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

Alright, I tried that, I didn't get very far though.

This is my new qemu command

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
-vga none \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/anthony/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

win8boot.img is a fat32 formatted image with all the files from the win8.iso copied to it.


By the way, when I boot the VM, I get msgs of it trying to boot from EFI floppy, EFI floppy 2, EFI DVD/CDROM and all failing.

It then tries to boot from network, fails, then drops me to an EFI shell.

First change this:

-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-hd,drive=isocd \

Second:

Make sure the file /efi/boot/bootx64.efi exists on the image

Offline

#4789 2015-04-12 04:34:53

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you mean to paste the changes to be made on the second line?

I think you just copied my win8boot.img line twice.

Offline

#4790 2015-04-12 04:35:50

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

Did you mean to paste the changes to be made on the second line?

I think you just copied my win8boot.img line twice.

-device scsi-cd != -device scsi-hd

this

-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-cd,drive=isocd \

to this

-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-hd,drive=isocd \

Last edited by nbhs (2015-04-12 04:36:34)

Offline

#4791 2015-04-12 16:36:59

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello. I am trying to pass through an AMD Radeon R9 290 to an x86-64 Windows 7 guest using qemu with kvm. I have had significant success so far, the card passes through and runs with excellent performance, however I am having an issue with the guest freezing and/or crashing.

The first issue I had encountered was a crash (BSOD) every time while booting the guest. I believe it was code 0x116 in atikmpag.sys, however I cannot recall for certain. I was able to fix this issue by ensuring the host's module vfio_iommu_type1 was loaded with the parameter "disable_hugepages=1".

For my current issue, there appears to be two cases where the guest fails in (at least slightly) different ways.

Case A

While booting the VM, during the windows logo on black background screen, the following appears in dmesg -w:

[ 803.389766] AMD-Vi: Event logged [INVALID_DEVICE_REQUEST device=00:00.1 address=0x000000fdf8010020 flags=0x0a00]

The guest runs fine until some point, usually seems to be associated with high memory or cpu usage. At this point, the following messages appear, in one quick burst, in dmesg -w:

[ 2517.239350] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d400 flags=0x0010] [ 2517.239357] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d480 flags=0x0010] [ 2517.239359] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d440 flags=0x0010] [ 2517.239361] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d4c0 flags=0x0010] [ 2517.239363] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d500 flags=0x0010] [ 2517.239364] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d540 flags=0x0010] [ 2517.239365] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d580 flags=0x0010] [ 2517.239367] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d5c0 flags=0x0010] [ 2517.239368] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d540 flags=0x0010] [ 2517.239370] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:00.0 domain=0x0002 address=0x000000009822d580 flags=0x0010] (etc, about 47 of these)

At this point, most (all?) applications in the guest (appear to) stop responding to window events, and do not appear to render to their windows anymore. Some, but not all, applications stop producing audio. After 2 to 4 seconds, the mouse cursor freezes. At this point, the guest is unusable except remotely. The keyboard (ps2-over-usb usb passthrough) responds to numlock, some applications continue to run. For example, I have been running a Zandronum game server (UDP) and it continues to receive/send packets, run its game simulation, accept new connections, etc. The client I have running on the same guest appears to stop functioning, however its software MIDI synthesizing still runs. I have been unable to get any -soundhw working without serious quality and skipping/choppiness issues, so I have written a program (user-mode) for the guest to record the audio and then send it over the (virtual) network to the host which then plays it back (this happens over UDP). This program, too, continues to run without issue.

The guest at this point does not respond to system_powerdown from the qemu console. If I press alt-tab or ctrl-alt-delete in the guest, it enters a further frozen state where the keyboard no longer responds to numlock, and the guest appears to no longer get incoming packets (or perhaps the server has frozen/crashed/terminated?). The MIDI synth continues to run, however, and the audio software I have written continues to record the audio and send it correctly.

Case B

When booting the VM, the above error message does not appear in dmesg. Eventually, the system freezes, however no messages appear in dmesg, and there is a chance the screen will turn black for a few seconds and then display a BSOD with the following message:

Attempt to reset the display driver and recover from timeout failed. 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html STOP: 0x00000116 (0xFFFFFA8006257010,0xFFFFF88003C0B848,0x0000000000000000,0 x0000000000000002) 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_19.html topic_1.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html atikmpag.sys - Address FFFFF88003C0B848 base at FFFFF88003C00000, DateStamp 546e9eb6

In either case, I can reset the guest using system_reset (or just quit and then run it later) without doing any kind of reboot/reset/suspend-resume with the host.

The first time running the guest after booting the host, I get case A every time, however from that point on I will (at least it seems) either always get case A or always get case B, until I reboot the host. I could be mistaken about this, however.

The devices I am looking at are these:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Root Complex ( there does not appear to actually be a 00:00.1 ) 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) I/O Memory Management Unit 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 02:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce 9800 GT] (rev a2) 03:07.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV280 [Radeon 9200 PRO] (rev 01) 03:07.1 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] RV280 [Radeon 9200 PRO] (Secondary) (rev 01)

00:02.x are in the same iommu group as 01:00.x which is the card I am passing through. 02:00.0 is the graphics card I'm using for the host. I have blacklisted radeon so the device does not come up aside from the BIOS when booting. The card I am passing through is selected by the BIOS as primary, which I thought might be the issue so I installed a PCI radeon card (03:07.x) and set PCI as the primary VGA display. This did not appear to have any effect on my problem at all.

The script I use to launch the VM is the following:

#!/bin/sh export QEMU_AUDIO_DRV=alsa export QEMU_ALSA_DAC_FIXED_FREQ=48000 export QEMU_ALSA_DAC_BUFFER_SIZE=8192 #export QEMU_ALSA_DAC_PERIOD_SIZE=170 qemu-system-x86_64 \ -enable-kvm \ -cpu host,kvm=off \ -m 4096 \ -smp cores=2,threads=1,sockets=1 \ -drive file=/dev/mapper/urw-win,format=raw,if=virtio \ -device vfio-pci,host=01:00.0,x-vga=on \ -device vfio-pci,host=01:00.1 \ -usb \ -usbdevice host:13ba:0018 \ -usbdevice host:046d:c040 \ -usbdevice host:0810:0001 \ -vga none \ -soundhw hda \ $*

I have tried various changes to the script with no success:

  • -m 2048, -smp cores=1,threads=1,sockets=1 (didn't fix anything)

  • -smp cores=1,threads=3,sockets=1 (didn't fix anything)

  • removing x-vga=on (system does not appear to actually boot, suspect either seabios or windows can't handle not having any graphics)

  • removing x-vga=on, adding -vga cirrus (system boots, seems more sluggish, did not fix issue)

  • adding multi-function=on to 01:00.0 (had this before, didn't work then either)

  • no soundhw (actually had this before, didn't fix anything)

  • adding vfio-pci devices for 00:02.0, 00:02.1 (error in dmesg unable to reset one of the devices, boots fine but didn't fix anything)

  • without kvm=off (didn't fix anything)

  • -realtime mlock=on (looked hopeful but didn't fix anything)

  • -realtime mlock=off (nope)

I have included some dmesg lines from booting the host, which may be relevant (blank lines where I have cut):

[ 0.000000] Linux version 3.19.3-3-ARCH (builduser@tobias) (gcc version 4.9.2 20150304 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 8 14:10:00 CEST 2015 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=4f2e65ef-ef21-4628-bbd7-219d7855732b rw quiet cryptdevice=UUI D=17b9c41f-ab59-4a68-a5ec-b7b3e2776bf6:calue [ 0.000000] tseg: 009f800000 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000009bc9bfff] usable [ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./FM2A88X+ Killer, BIOS P2.10 07/31/2014 [ 0.000000] ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20141107/tbfadt-649) [ 0.000000] AGP: Checking aperture... [ 0.000000] AGP: No AGP bridge found [ 0.000000] AGP: Node 0: aperture [bus addr 0x00000000-0x01ffffff] (32MB) [ 0.000000] AGP: Your BIOS doesn't leave a aperture memory hole [ 0.000000] AGP: Please enable the IOMMU option in the BIOS setup (<- it is enabled) [ 0.000000] AGP: This costs you 64MB of RAM [ 0.000000] AGP: Mapping aperture over RAM [mem 0x90000000-0x93ffffff] (65536KB) [ 0.365445] ACPI: Using IOAPIC for interrupt routing [ 0.365463] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.365501] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.365837] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.401076] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none [ 0.401080] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none [ 0.401083] vgaarb: setting as boot device: PCI:0000:03:07.0 [ 0.401085] vgaarb: device added: PCI:0000:03:07.0,decodes=io+mem,owns=io+mem,locks=none [ 0.401086] vgaarb: loaded [ 0.401088] vgaarb: bridge control possible 0000:03:07.0 [ 0.401089] vgaarb: bridge control possible 0000:02:00.0 [ 0.401090] vgaarb: bridge control possible 0000:01:00.0 [ 0.401183] PCI: Using ACPI for IRQ routing [ 0.841509] ERROR: Unable to locate IOAPIC for GSI 56 (<- this one appears on the screen during my cryptsetup password entry prompt) [ 0.841566] Failed to set pin attr for GSI56 [ 0.841569] pci 0000:00:00.2: PCI INT A: failed to register GSI [ 0.842108] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 0.842109] AMD-Vi: Extended features: PPR GT IA PC [ 0.842112] AMD-Vi: Interrupt remapping enabled [ 0.853022] AMD-Vi: Using passthrough domain for device 0000:01:00.0 [ 0.854512] AMD-Vi: Lazy IO/TLB flushing enabled [ 0.854893] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 0.854896] software IO TLB [mem 0x97c9c000-0x9bc9c000] (64MB) mapped at [ffff880097c9c000-ffff88009bc9bfff] [ 0.858586] pcieport 0000:00:02.1: Signaling PME through PCIe PME interrupt [ 0.858588] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 0.858589] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt [ 0.858592] pcie_pme 0000:00:02.1:pcie01: service driver pcie_pme loaded [ 0.858604] pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt [ 0.858605] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 0.858607] pcie_pme 0000:00:03.1:pcie01: service driver pcie_pme loaded [ 0.858619] pcieport 0000:00:15.0: Signaling PME through PCIe PME interrupt [ 0.858622] pcie_pme 0000:00:15.0:pcie01: service driver pcie_pme loaded [ 0.858635] pcieport 0000:00:15.2: Signaling PME through PCIe PME interrupt [ 0.858636] pci 0000:05:00.0: Signaling PME through PCIe PME interrupt [ 0.858639] pcie_pme 0000:00:15.2:pcie01: service driver pcie_pme loaded [ 0.858651] pcieport 0000:00:15.3: Signaling PME through PCIe PME interrupt [ 0.858652] pci 0000:06:00.0: Signaling PME through PCIe PME interrupt [ 0.858655] pcie_pme 0000:00:15.3:pcie01: service driver pcie_pme loaded [ 0.858660] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.858675] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.858716] GHES: HEST is not enabled! [ 11.604037] input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.1/0000:01:00.1/sound/card1/input8 [ 11.604117] input: HD-Audio Generic HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.1/0000:01:00.1/sound/card1/input9 [ 11.604189] input: HD-Audio Generic HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.1/0000:01:00.1/sound/card1/input10 [ 11.604258] input: HD-Audio Generic HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.1/0000:01:00.1/sound/card1/input11 [ 11.604329] input: HD-Audio Generic HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:02.1/0000:01:00.1/sound/card1/input12 [ 11.604396] input: HD-Audio Generic HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:02.1/0000:01:00.1/sound/card1/input13

I also get these while starting the vm:

[ 124.786012] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 124.786149] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 124.786155] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 [ 128.333852] vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=none:owns=none

At some point while troubleshooting I stopped using pci-stub on the kernel command line. Enabling it again has not fixed the problem:

[ 0.917064] pci-stub: add 1002:67B1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.917083] pci-stub 0000:01:00.0: claimed by stub [ 0.917091] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.917100] pci-stub 0000:01:00.1: claimed by stub

I hope this post is not too long, but I really don't know where to go from here. If anyone has any answers or advice, please. I'm dying here.

Offline

#4792 2015-04-12 16:53:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ughman
Just a sidenote: instead of writing a userspace tool for audio transfer, you could use NetJACK instead. Works the same way, pushing audio into UDP.

I can't help you with the main problems, because they're very complex. But i'll just cut some of the info you've provided:

[ 0.917064] pci-stub: add 1002:67B1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.917083] pci-stub 0000:01:00.0: claimed by stub [ 0.917091] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 0.917100] pci-stub 0000:01:00.1: claimed by stub

This is 100% fine.

[ 124.786012] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) [ 124.786149] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 [ 124.786155] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 [ 128.333852] vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=none:owns=none

This is fine, but vgaarb line looks strange. AW may provide more info on this, if he'll come.

00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425

This looks very weird. What kernel version are you using? I suppose you should have something alike to this:

00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port [1022:1412] 00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port [1022:1414]

And since we're sharing same CPU BugList(i've got fam15h mod10h-1fh), and i've been experiencing same weird issues on my x-vga VM too...

I'd suggest you to move to OVMF. That helped me much. In fact, that's the only way i'm able to run VM stable.

P.S.
And why do you have three different GPUs in your system?..

Last edited by Duelist (2015-04-12 17:03:57)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4793 2015-04-12 17:58:03

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Just a sidenote: instead of writing a userspace tool for audio transfer, you could use NetJACK instead. Works the same way, pushing audio into UDP.

Thank you, I will look into that.

Duelist wrote:
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425

This looks very weird. What kernel version are you using? I suppose you should have something alike to this:

00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port [1022:1412] 00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port [1022:1414]

I am running "3.19.3-3-ARCH #1 SMP PREEMPT Wed Apr 8 14:10:00 CEST 2015 x86_64 GNU/Linux" according to uname -a. The complete lspci list is here:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Root Complex 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) I/O Memory Management Unit 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Root Port 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424 00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 09) 00:10.1 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 09) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11) 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11) 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11) 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11) 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 16) 00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD] FCH IDE Controller 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 01) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11) 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40) 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11) 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 0) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 2) 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 3) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 30h-3fh) Processor Function 5 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 02:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce 9800 GT] (rev a2) 03:07.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV280 [Radeon 9200 PRO] (rev 01) 03:07.1 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] RV280 [Radeon 9200 PRO] (Secondary) (rev 01) 05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller 06:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 10)
Duelist wrote:

I'd suggest you to move to OVMF. That helped me much. In fact, that's the only way i'm able to run VM stable.

I will work on this then. Will this require me to get the guest OS UEFI-bootable? At the moment it is installed on an MBR-partitioned virtual drive.

Duelist wrote:

P.S.
And why do you have three different GPUs in your system?..

I only installed the third so that I could get the BIOS to select a card other than the one I'm using for passthrough to not be the primary VGA adapter. The setup screen only allows me to select "PCI Express", "PCI", or "Onboard". The other two are both pci-e and I'm pretty sure I don't actually have onboard graphics because I'm running a non-APU on an FM2+ board.

Offline

#4794 2015-04-12 19:28:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ughman wrote:

I will work on this then. Will this require me to get the guest OS UEFI-bootable? At the moment it is installed on an MBR-partitioned virtual drive.

Yes, the OS will need to be UEFI-bootable. Using OVMF whitepaper you can get Windows 7 to boot.

I'm running a non-APU on an FM2+ board.

AH THE PAIN I FEEL.
So you have AMD Athlon 860K(kaveri) or something and you are experiencing same problems as i've had on 750K(trinity based).
God damn it...
And you even have A88X-based motherboard...
Damn it!
I've considered upgrading my system to 860K+A88X, but seems like my system can not be saved.

So, the only way to run vfio-pci stable on AMD APU-based Athlon CPUs or APUs is OVMF.

UPD:
Just checked AMD's revision guide on fam15h mod30h-3fh - they've deleted a lot of bugs, but the IOMMU ones are still present. Still not quite sure what do they mean.

Last edited by Duelist (2015-04-12 20:06:59)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4795 2015-04-12 19:55:04

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Helios747 wrote:

Did you mean to paste the changes to be made on the second line?

I think you just copied my win8boot.img line twice.

-device scsi-cd != -device scsi-hd

this

-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-cd,drive=isocd \

to this

-drive file=/home/anthony/win8boot.img,id=isocd,if=none -device scsi-hd,drive=isocd \

THANK YOU. IT BOOTS!

I installed windows 8.1, and nVidia drivers installed just fine! Device Manager shows my 770 running properly!

All I need now is sound, but that should be simple enough.

Also, for anybody lurking this thread, I got this all working in Arch Linux with the vfio kernel from AUR. I had trouble getting in working in Ubuntu or Fedora so I said screw it and installed Arch.

Offline

#4796 2015-04-12 20:29:35

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Helios747 wrote:

I had trouble getting in working in Ubuntu or Fedora so I said screw it and installed Arch.

Well, you do know that aw(who wrote vfio-pci) is a redhat developer and he's using.. well.. redhat products?
As my long experience reading this thread says, it's either hardware or people, rarely software problems. So this should work distribution-independent.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4797 2015-04-12 21:27:01

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

We'll I'm finally up and running with a successful GTX 770 pass (what a long adventure).

Looking for some performance tuning, performance is a bit choppy while gaming (med settings, easy to run game). Currently, I cannot get Windows 8 to boot with cpu=host. I get the SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error. I have to use cpu=qemu64 to boot.

Any advice for my boot script?

sudo qemu-system-x86_64 -enable-kvm -m 10240 -cpu qemu64,kvm=off -smp 4,sockets=1,cores=4,threads=4 \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 \ -device vfio-pci,host=00:1f.2 \ -device vfio-pci,host=06:00.0 \ -net nic -net bridge,br=br0 \ -rtc base=localtime,clock=host \ -soundhw hda \ -vga none

I am just passing my SATA controller (00:1f.2) directly to the guest in order to boot, because it has Windows already installed on the ssd. 03:00.0 is GTX 770 and 06:00.0 is 1 of 3 USB controllers.

Edit: I'm using a KVM switch and my monitors digital/analog input selector to switch between host and guest. I noticed that when my host goes to sleep, I cannot wake it back up. Any ideas on why?

Last edited by logix (2015-04-12 21:34:45)

Offline

#4798 2015-04-12 23:20:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

logix wrote:

We'll I'm finally up and running with a successful GTX 770 pass (what a long adventure).

Looking for some performance tuning, performance is a bit choppy while gaming (med settings, easy to run game). Currently, I cannot get Windows 8 to boot with cpu=host. I get the SYSTEM_THREAD_EXCEPTION_NOT_HANDLED error. I have to use cpu=qemu64 to boot.

Any advice for my boot script?

sudo qemu-system-x86_64 -enable-kvm -m 10240 -cpu qemu64,kvm=off -smp 4,sockets=1,cores=4,threads=4 \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 \ -device vfio-pci,host=00:1f.2 \ -device vfio-pci,host=06:00.0 \ -net nic -net bridge,br=br0 \ -rtc base=localtime,clock=host \ -soundhw hda \ -vga none

I am just passing my SATA controller (00:1f.2) directly to the guest in order to boot, because it has Windows already installed on the ssd. 03:00.0 is GTX 770 and 06:00.0 is 1 of 3 USB controllers.

Edit: I'm using a KVM switch and my monitors digital/analog input selector to switch between host and guest. I noticed that when my host goes to sleep, I cannot wake it back up. Any ideas on why?

There's a lot of options for -cpu. Choose one that will represent your cpu best, usually i do -cpu host.
Also, when -smp cores=NUM or sockets=NUM or threads=NUM argument is supplied, the first -smp NUMBER is ignored.
If you want the most simple config - use -smp 4(that'll give you 4 cores).
If you want to follow your cpu's logical core topology as stated in lscpu, you'd do something like -smp sockets=1,cores=4,thread=2(NUM threads is NUM threads per core), that'll give your guest EIGHT logical cpus, like on a real core i7-alike CPU with hyperthreading.
If you'll specify, let's say, one socket, four cores-per-socket and four threads-per-core while having a regular four core host, you'd have a huge performance hit because every one physical core would need to work as two logical cores. When using libvirt, there's a warning issued about that.

So, currently your machine is having one socket, four cores per socket and four threads per core, which sums up into 16 logical cores. Y' sure that you have that much? What does your VM's task manager say about that?

You could also try enabling hugepages or npt.
And you can try pinning your cpus, since looks like you have a lot of cpu cores. Usually that improves latencies and performance alot.

Regarding your edit:
I guess your KVM switch is passive, and just disconnects your mouse and keyboard from the machine. Of course when it's disconnected, your firmware lefts those ports unpowered and it doesn't wait for any signals to come from them. When the power is given, KVM switch is switching somewhere.
Actually, that's a way i've considered going back in the day, but i've stayed on libvirt's VNC "display" - works like a charm, even ACPI power button event is transfered into VM.
But being kind of dumb and retarded, instead of going netjack way or any other way of software audio output, i've built a hardware audio mixer just4lulz and plugged a separate sound card for host needs. So your mileage may vary.

P.S.
Windows 7 is limited to one or two sockets depending on the edition, but it's not limited on overall core or thread core, so it's legal to put two opteron-6300s CPUs and have 32 cores on a real machine, and of course it's possible to do so in a VM.

Last edited by Duelist (2015-04-12 23:27:42)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4799 2015-04-12 23:29:06

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Helios747 wrote:

I had trouble getting in working in Ubuntu or Fedora so I said screw it and installed Arch.

Well, you do know that aw(who wrote vfio-pci) is a redhat developer and he's using.. well.. redhat products?
As my long experience reading this thread says, it's either hardware or people, rarely software problems. So this should work distribution-independent.

It was simpler for me because I didn't have to translate many steps listed in this guide to work with various distros. All the steps he gave here worked without modification in Arch Linux.

Offline

#4800 2015-04-13 00:32:06

Helios747
Member
Registered: 2012-11-17
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One last issue I'm having, is the sound in the Windows 8.1 VM is choppy. I'm thinking about working around this by simply giving the VM my g930 through a USB passthrough, but if I can I'd like to fix this.

This is my command line

#!/bin/sh

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=2,threads=2 \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
-vga none \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/SSD/lvWinRoot,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/dev/HDD/lvWinStorage,id=disk2,format=raw,if=none -device scsi-hd,drive=disk2 \
-usb -usbdevice host:046d:c31c -usbdevice host:062a:3621 -usbdevice host:1532:0201 \
-soundhw hda

The audio device Linux is routing this through is "Analog output" on my Logitech G930 headset. I tried the pulseaudio driver but the problem was even worse.

The OP said I'd have to mess around with those env variables to get sound working right but I don't really know how to mess with them. Should I increase the values? Decrease?

Last edited by Helios747 (2015-04-13 00:32:50)

Offline

#4801 2015-04-13 16:29:57

Iksf
Member
Registered: 2015-04-13
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any way to have two identical GPU's, one as pass-through one not.

lspci | grep NVIDIA 01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) lspci -nn 01:00.0 0300: 10de:13c2 (rev a1) 01:00.1 0403: 10de:0fbb (rev a1) 02:00.0 0300: 10de:13c2 (rev a1) 02:00.1 0403: 10de:0fbb (rev a1)

They have the same id's so I can't seem to separate them.

Offline

#4802 2015-04-13 17:09:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Iksf wrote:

Is there any way to have two identical GPU's, one as pass-through one not.

lspci | grep NVIDIA 01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) 02:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) lspci -nn 01:00.0 0300: 10de:13c2 (rev a1) 01:00.1 0403: 10de:0fbb (rev a1) 02:00.0 0300: 10de:13c2 (rev a1) 02:00.1 0403: 10de:0fbb (rev a1)

They have the same id's so I can't seem to separate them.

Search the thread for xen-pciback.hide, which for our purposes is similar to pci-stub, but allows devices to be specified by address rather than vendor and device ID.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4803 2015-04-13 19:41:00

zopilote
Member
Registered: 2014-11-02
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Since my GT610 does support UEFI, it's very strange that your GTX680 doesn't.

Hi, could you provide me detailed info about your GT 610? I'm very interested about getting it cause my evga gt 630 doesn't support uefi, thank you so much in advance!

Offline

#4804 2015-04-13 20:41:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zopilote wrote:
Duelist wrote:

Since my GT610 does support UEFI, it's very strange that your GTX680 doesn't.

Hi, could you provide me detailed info about your GT 610? I'm very interested about getting it cause my evga gt 630 doesn't support uefi, thank you so much in advance!

Have you contacted EVGA support?  My EVGA GTX660 didn't come with UEFI support either, but they sent me a new BIOS when I asked.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4805 2015-04-14 14:59:52

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zopilote wrote:
Duelist wrote:

Since my GT610 does support UEFI, it's very strange that your GTX680 doesn't.

Hi, could you provide me detailed info about your GT 610? I'm very interested about getting it cause my evga gt 630 doesn't support uefi, thank you so much in advance!

Looks like i've got this piece of cra hardware. I've got it(somewhere a year ago) very cheap, for something like 20$ or so. It was used, but on a warranty. So far so good, it works despite being plugged into 16x-1x riser card. There's pci-e x1 version of that card available, but i couldn't get it.
But the performance is, well, worth the price - it's possible to play quake 3 or play full hd video on this, so i'm satisfied.
If i could - i'd swap my CPU with APU, that'd give me one GPU less without losing host output.

Last edited by Duelist (2015-04-14 15:00:33)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4806 2015-04-14 17:26:14

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nevermind

Last edited by paperinick (2015-04-14 18:12:43)

Offline

#4807 2015-04-15 03:21:09

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I tracked down the source of my poor Windows guest performance. I noticed that whenever I move my mouse the CPU usage, as noted by task manager in the guest, shoots up to 30-40% usage (idle sits around 2-4%).

I'm passing my USB controller though to the guest, which should give me the best performance. Possibly bad interrupt handling?

Offline

#4808 2015-04-15 07:52:30

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A quick question - I need to downgrade my kernel from 3.18 to 3.14 (former does not play well with ZFS, due to changes in vfs, and the latter is second-best LTS kernel). Is there any patch I should apply on 3.14.38 (current 3.14) for full vfio functionality, as required for full GPU passthrough functionality and qemu 2.2.1 ? In particular I'm thinking https://lkml.org/lkml/2014/5/25/94 , is there anything else?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4809 2015-04-15 19:53:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

logix wrote:

Possibly bad interrupt handling?

USB HID devices do not generate interrupts, instead, they're polled by the operating system every so often.
Try changing your display and pointing device in QEMU. I'd try VNC as a 100% working solution, but you could just add -device qxl to your QEMU command-line startup script and have SDL graphics window which somehow passes your mouse input into VM.

Aren't you the guy with 8 virtual cores on 4 real cores?.. Yes, you're that guy. Did you sort out your strange CPU configuration? Because, well, overallocating vCPUs is a major performance killer.

Last edited by Duelist (2015-04-15 19:55:12)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4810 2015-04-15 23:25:49

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I've been trying to get this working, but been running into some problems.
I have 2 differend Radeon HD cards, one plugged into the main PCIex16 slot, another in the PCIex4 slot.
The card I want to passthrough is the first one.
PCI-Stub claims the primary card and no radeon driver gets loaded for it, but video is still only output on this one.
How can i switch to the other card or boot with it to begin with? (To clarify: I want the arch host to use the graphics card in the second slot, which is currently not doing anything even though lspci -v shows a loaded radeon driver.)

(My MB doesn't allow me to choose the primary card.)


Is what I'm trying to do possible, or do I have to passthrough the second card? (Not ideal in this case, as I think the x4 slot would hinder the cards performance).

Last edited by fseven (2015-04-16 00:12:25)

Offline

#4811 2015-04-16 07:40:53

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:

Hello,
Is what I'm trying to do possible, or do I have to passthrough the second card? (Not ideal in this case, as I think the x4 slot would hinder the cards performance).

Preferably you should switch BIOS (or UEFI) motherboard configuration option to select PCIe slot for primary VGA card, or to disable ROM of the card in 16x slot (which will have the same effect). In case you cannot do that, it is perfectly possible to passthrough primary GPU card, that's what I do (my host is running headless, both primary and secondary GPUs are for passed to two VMs).

Last edited by Bronek (2015-04-16 07:42:49)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4812 2015-04-16 07:43:42

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

Preferably you should switch BIOS (or UEFI) motherboard configuration option to select PCIe slot for primary VGA card, or to disable ROM of the card in 16x slot (which will have the same effect).


As I said, I fear i don't have the option to switch the primary slot. (MB is a Gigabyte  970A DS3P).

Could you elaborate on the other thing? How would I disable the ROM?

Offline

#4813 2015-04-16 09:17:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:
Bronek wrote:

Preferably you should switch BIOS (or UEFI) motherboard configuration option to select PCIe slot for primary VGA card, or to disable ROM of the card in 16x slot (which will have the same effect).


As I said, I fear i don't have the option to switch the primary slot. (MB is a Gigabyte  970A DS3P).

Could you elaborate on the other thing? How would I disable the ROM?

Usually gigabyte has that option. Whatever.
You can edit your xorg.conf to choose the device manually by busid, like

Section "Device" Identifier "Device0" Driver "YOUR DRIVER" VendorName "LIKE AMD OR SOMETHING" BoardName "MAY BE YOUR MODEL" BusID "PCI:02:00:00" <<THAT'S YOUR PCI BDF ADDRESS, AS FOUND IN LSPCI EndSectio

You can use fglrx and hope it's fixed now or radeon. Just try and see if radeon ignores pci-stub like fglrx does. I think it shouldn't do that, and you don't have to explicitly blacklist it.

So, edit the xorg.conf, load a driver for your GPU and it should be fine. Or maybe you don't even need to add that BusID line at all.

Last edited by Duelist (2015-04-16 09:20:09)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4814 2015-04-16 11:58:51

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
ughman wrote:

I will work on this then. Will this require me to get the guest OS UEFI-bootable? At the moment it is installed on an MBR-partitioned virtual drive.

Yes, the OS will need to be UEFI-bootable. Using OVMF whitepaper you can get Windows 7 to boot.

I believe I have the guest booting now with OVMF. This is my current qemu launch script:

#!/bin/sh export QEMU_AUDIO_DRV=alsa export QEMU_ALSA_DAC_FIXED_FREQ=48000 export QEMU_ALSA_DAC_BUFFER_SIZE=11054 #export QEMU_ALSA_DAC_PERIOD_SIZE=170 qemu-system-x86_64 \ -enable-kvm \ -cpu host,kvm=off \ -m 4096 \ -smp cores=2,threads=1,sockets=1 \ -drive file=OVMF.fd,format=raw,if=pflash \ -drive file=/dev/mapper/mass-win--uefi,format=raw,if=virtio \ -usb \ -usbdevice host:13ba:0018 \ -usbdevice host:0810:0001 \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -soundhw hda \ -vga none \ $*

Unfortunately, this hasn't fixed the problem either. I have once gotten what is I assume to be a bsod, but nothing came up on the screen after it went black and turned off, as well as freezes on the later attempts.

Offline

#4815 2015-04-16 14:09:57

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried passing Radeon 295 X2 ? Asking because it apparently has builtin PLX chip, thus I'm guessing that PCIe topology can be a little "funky".


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4816 2015-04-16 17:01:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ughman wrote:
Duelist wrote:
ughman wrote:

I will work on this then. Will this require me to get the guest OS UEFI-bootable? At the moment it is installed on an MBR-partitioned virtual drive.

Yes, the OS will need to be UEFI-bootable. Using OVMF whitepaper you can get Windows 7 to boot.

I believe I have the guest booting now with OVMF. This is my current qemu launch script:

#!/bin/sh export QEMU_AUDIO_DRV=alsa export QEMU_ALSA_DAC_FIXED_FREQ=48000 export QEMU_ALSA_DAC_BUFFER_SIZE=11054 #export QEMU_ALSA_DAC_PERIOD_SIZE=170 qemu-system-x86_64 \ -enable-kvm \ -cpu host,kvm=off \ -m 4096 \ -smp cores=2,threads=1,sockets=1 \ -drive file=OVMF.fd,format=raw,if=pflash \ -drive file=/dev/mapper/mass-win--uefi,format=raw,if=virtio \ -usb \ -usbdevice host:13ba:0018 \ -usbdevice host:0810:0001 \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -soundhw hda \ -vga none \ $*

Unfortunately, this hasn't fixed the problem either. I have once gotten what is I assume to be a bsod, but nothing came up on the screen after it went black and turned off, as well as freezes on the later attempts.

What? Does the OS boots? Or you see a black screen?
What windows do you have? Have you added -device qxl-vga for win7/srv2008r2 as stated in the whitepaper?

You should see tianocore logo with memcheck in upper right corner, then a black screen, after some time "starting windowns" logo animation will show up.
Of course, OS should be installed from UEFI, because it overwrites ESP and UEFIVARS.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4817 2015-04-16 17:24:57

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

What? Does the OS boots? Or you see a black screen?

Sorry, it does boot fine. Nothing appears on the screen at first but windows eventually appears with a login prompt.

Duelist wrote:

What windows do you have?

Windows 7 Ultimate x64

Duelist wrote:

Have you added -device qxl-vga for win7/srv2008r2 as stated in the whitepaper?

I was not using that when I posted, but after reading your post I have now tried it and eventually got a freeze regardless. This time it happened while I was scrolling through some text, which appeared to be corrupted once it froze. The display on the passed-through card went black after a few seconds, and then the qxl output went black a few seconds after that.

Duelist wrote:

You should see tianocore logo with memcheck in upper right corner, then a black screen, after some time "starting windowns" logo animation will show up.

That I do (when using -device qxl-vga). Afterwards, the qxl output stays at the logo animation, and the login screen appears on the display attached to the card.

Duelist wrote:

Of course, OS should be installed from UEFI, because it overwrites ESP and UEFIVARS.

Wait, I'm supposed to reinstall the OS after switching to UEFI? That's rather annoying, I was hoping I could keep the same install, as this setup was previously running on bare metal for several months before I moved it into the VM.

Offline

#4818 2015-04-16 18:46:19

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ughman wrote:

Windows 7 Ultimate x64

Duelist wrote:

Have you added -device qxl-vga for win7/srv2008r2 as stated in the whitepaper?

I was not using that when I posted, but after reading your post I have now tried it and eventually got a freeze regardless. This time it happened while I was scrolling through some text, which appeared to be corrupted once it froze. The display on the passed-through card went black after a few seconds, and then the qxl output went black a few seconds after that.

Duelist wrote:

You should see tianocore logo with memcheck in upper right corner, then a black screen, after some time "starting windowns" logo animation will show up.

That I do (when using -device qxl-vga). Afterwards, the qxl output stays at the logo animation, and the login screen appears on the display attached to the card.

Duelist wrote:

Of course, OS should be installed from UEFI, because it overwrites ESP and UEFIVARS.

Wait, I'm supposed to reinstall the OS after switching to UEFI? That's rather annoying, I was hoping I could keep the same install, as this setup was previously running on bare metal for several months before I moved it into the VM.

Seeing a logo stuck on qxl-vga is 100% fine, you can drop the qxl-vga device once you've installed the drivers.
Also, if it boots and works - you don't need to reinstall it. Maybe reinstall some device drivers, but nothing much. There may be a hidden windows "hardware migration" problem somewhere, but if it boots - i think it should work.

I don't know what causes the freeze, but here's my trinity-based system for general reference:

options kvm-amd npt=1 nested=0 options kvm ignore_msrs=0 allow_unsafe_assigned_interrupts=0 options vfio_iommu_type1 disable_hugepages=1 allow_unsafe_interrupts=0
GRUB_CMDLINE_LINUX="vconsole.font=latarcyrheb-sun16 $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) nofb pci-stub.ids=1022:780d,1002:683f,1002:aab0 enable_mtrr_cleanup iommu=pt acpi_enforce_resources=lax"
<domain type='kvm' id='2'> <name>vm1</name> <uuid>f6958833-174f-44e6-817b-2a6c73c6f041</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/var/lib/libvirt/qemu/nvram/vm1_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='directsync' io='native'/> <source dev='/dev/sda'/> <backingStore/> <target dev='vda' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='network'> <mac address='52:54:00:f0:d5:39'/> <source network='default'/> <target dev='vnet0'/> <model type='rtl8139'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/2'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='usb'> <alias name='input0'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x00' slot='0x14' function='0x2'/> </source> <alias name='hostdev4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> </redirdev> <redirdev bus='usb' type='spicevmc'> <alias name='redir1'/> </redirdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c127,c267</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c127,c267</imagelabel> </seclabel> </domain>

^^^^
that libvirt xml generates this
v v v v

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name vm1 -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu Opteron_G5,+bmi1,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+tce,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme -drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/vm1_VARS.fd,if=pflash,format=raw,unit=1 -m 4096 -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 -uuid f6958833-174f-44e6-817b-2a6c73c6f041 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/dev/sda,if=none,id=drive-virtio-disk0,format=raw,cache=directsync,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xc,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f0:d5:39,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-mouse,id=input0 -vnc 127.0.0.1:0 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x4 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x8,rombar=1,romfile=/mnt/hdd/qemu/hybridmagic.rom -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device vfio-pci,host=02:00.0,id=hostdev2,bus=pci.0,addr=0xa,rombar=1,romfile=/mnt/hdd/qemu/hybridmagic.rom -device vfio-pci,host=02:00.1,id=hostdev3,bus=pci.0,addr=0xb -device pci-assign,configfd=24,host=00:14.2,id=hostdev4,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
(qemu-2.1.3-5.fc21) Name : edk2.git-ovmf-x64 Release : 20150416.b962.ged6fde4 3.19.3-200.fc21.x86_64
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 21 Model: 16 Model name: AMD Athlon(tm) X4 750K Quad Core Processor Stepping: 1 CPU MHz: 1900.000 CPU max MHz: 3400.0000 CPU min MHz: 1400.0000 BogoMIPS: 6834.29 Virtualization: AMD-V L1d cache: 16K L1i cache: 64K L2 cache: 2048K NUMA node0 CPU(s): 0-3
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Complex [1022:1410] +-00.2 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) I/O Memory Management Unit [1022:1419] +-02.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] +-04.0-[02]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] +-11.0 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7801] +-12.0 Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller [1022:7807] +-12.2 Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller [1022:7808] +-13.0 Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller [1022:7807] +-13.2 Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller [1022:7808] +-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:780b] +-14.2 Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller [1022:780d] +-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:780e] +-14.4-[03]----05.0 Ensoniq 5880B [AudioPCI] [1274:5880] +-15.0-[04]--+-00.0 NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] | \-00.1 NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] +-15.1-[05]----00.0 ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042] +-15.2-[06]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] +-16.0 Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller [1022:7807] +-16.2 Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller [1022:7808] +-18.0 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 0 [1022:1400] +-18.1 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 1 [1022:1401] +-18.2 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 2 [1022:1402] +-18.3 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 3 [1022:1403] +-18.4 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 4 [1022:1404] \-18.5 Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 5 [1022:1405]

Try running dmesg -w in some other window while trying to "freeze" the VM.
You can also plug in ISA debug console to check logs, like this:

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Last edited by Duelist (2015-04-16 18:53:53)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4819 2015-04-17 07:25:14

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw
the AMD Firepro R5000 can passthrough successfully, but when I install vga driver, it restart with BSOD like atikmpag.sys error

my kernel:
linux-3.18.6
qemu:
qemu-2.1.3

commandline:
/usr/libexec/qemu-kvm-passthrough -cpu host -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/usr/share/qemu/win-pre-install-with-slic.bin \
-drive file=/var/lib/libvirt/images/fwq_clean_win7_64bit-patched.img,id=virtio-scsi-disk0 \
-device virtio-blk-pci,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=06:d8:e8:00:00:1b,addr=0x3 \
-net tap,ifname=tap0,script=/root/qemu-ifup \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.2 \
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.3 \
-device vfio-pci,host=06:00.2,bus=root.1,addr=00.4 \
-monitor stdio

Offline

#4820 2015-04-17 08:40:36

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
when I want to use a newer version of the ovmf-svn package in my VM, is it possible to keep the part where the settings are stored in from the old build?

Thanks
apex

Offline

#4821 2015-04-17 08:57:15

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using a different board now and got it working fine, but I have another question:
Is it possible to passthrough a single SATA port? Or how would passing through the disk like in the first place not impact performance? (Because then it wouldn't be in AHCI mode.)
I can't passthrough an entire controller, since I only have one.

Offline

#4822 2015-04-17 09:19:30

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:

I'm using a different board now and got it working fine, but I have another question:
Is it possible to passthrough a single SATA port? Or how would passing through the disk like in the first place not impact performance? (Because then it wouldn't be in AHCI mode.)
I can't passthrough an entire controller, since I only have one.

I don´t believe passing through a single port is possible. The best would be installing a separate controller, if that is an option for you.

Offline

#4823 2015-04-17 10:02:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi,
when I want to use a newer version of the ovmf-svn package in my VM, is it possible to keep the part where the settings are stored in from the old build?

Thanks
apex

When using EDK2 GIT builds, you have the following files:

/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /usr/share/edk2.git/ovmf-x64/OVMF-with-csm.fd /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd /usr/share/edk2.git/ovmf-x64/OVMF_CODE-with-csm.fd /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /usr/share/edk2.git/ovmf-x64/OVMF_VARS-with-csm.fd

If you will add both OVMF_CODE and OVMF_VARS, you can update only OVMF_CODE(just copy the file, or libvirt does it's stuff) without changing the VARS. lersek may provide more info on this.


@fseven: you can achieve a similar effect when doing -device virtio-blk-pci with drive /dev/sda or even /dev/sda4(but that will bork the partition table, as the VM will think it's a whole disk). If you poke windows and it's EFI-loader the right way, you may even boot baremetal from that partitions.

I have /dev/sda1 as 500~GB junkyard FS mounted to linux, and windows knows about it(since it has full /dev/sda), but neither OS can mount other's FS. Linux prevents you, windows doesn't do ext4.
There are some caching and scheduling issues present, but they don't affect the performance much.

I haven't heard any success-stories about dedicating a single port to some VM.

Last edited by Duelist (2015-04-17 10:03:15)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4824 2015-04-17 10:03:33

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
fseven wrote:

I'm using a different board now and got it working fine, but I have another question:
Is it possible to passthrough a single SATA port? Or how would passing through the disk like in the first place not impact performance? (Because then it wouldn't be in AHCI mode.)
I can't passthrough an entire controller, since I only have one.

I don´t believe passing through a single port is possible. The best would be installing a separate controller, if that is an option for you.

I'm looking into that, but they do seem to cost a fair bit, not sure if it's worth it.
I get about 90MB/s write on Guest and 380MB/s on host, which is quite a big difference. At this speed I might as well be using a normal HD.

Update: Just tried passing it through using the -hda option instead of using a virtio-scsi device like in the first post, performance seems to be within 10-15% which is completely acceptable.

Offline

#4825 2015-04-17 11:58:54

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
apex8 wrote:

Hi,
when I want to use a newer version of the ovmf-svn package in my VM, is it possible to keep the part where the settings are stored in from the old build?

Thanks
apex

When using EDK2 GIT builds, you have the following files:

/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /usr/share/edk2.git/ovmf-x64/OVMF-with-csm.fd /usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd /usr/share/edk2.git/ovmf-x64/OVMF_CODE-with-csm.fd /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /usr/share/edk2.git/ovmf-x64/OVMF_VARS-with-csm.fd

@fseven: you can achieve a similar effect when doing -device virtio-blk-pci with drive /dev/sda or even /dev/sda4(but that will bork the partition table, as the VM will think it's a whole disk). If you poke windows and it's EFI-loader the right way, you may even boot baremetal from that partitions.

Can you give some more details about "poking windows and it´s EFI-loader the right way"? wink Furthermore I would expect Windows to complain about changed chipset/driver mismatch and crashing, or is this an issue which doesn't exist in newer Windows versions?

Last edited by apex8 (2015-04-17 12:07:19)

Offline

#4826 2015-04-17 14:27:02

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

@fseven: you can achieve a similar effect when doing -device virtio-blk-pci with drive /dev/sda or even /dev/sda4(but that will bork the partition table, as the VM will think it's a whole disk). If you poke windows and it's EFI-loader the right way, you may even boot baremetal from that partitions.

I have /dev/sda1 as 500~GB junkyard FS mounted to linux, and windows knows about it(since it has full /dev/sda), but neither OS can mount other's FS. Linux prevents you, windows doesn't do ext4.
There are some caching and scheduling issues present, but they don't affect the performance much.

I haven't heard any success-stories about dedicating a single port to some VM.

Right now I'm trying out the software raid method from the wiki, which seems to have quite passable performance (better with just the -hda option than with  -device virtio-blk-pci).
The host can see and mount the partition as well (though not at once (well I haven't tried, but it wouldn't be smart anyways).).
There are a few tweaks (see here for example  http://wiki.mikejung.biz/KVM_/_Xen?Page … VM_Guests) that might further improve performance, I'm trying that next.

apex8 wrote:

Can you give some more details about "poking windows and it´s EFI-loader the right way"? wink Furthermore I would expect Windows to complain about changed chipset/driver mismatch and crashing, or is this an issue which doesn't exist in newer Windows versions?

In my experience if you pass the complete disk, you can try booting but windows always bluescreened for me (no wonder I guess, it's quite a different environment.). I don't think it's possible to boot both directly from the disk and via Qemu.

Last edited by fseven (2015-04-17 14:29:57)

Offline

#4827 2015-04-17 20:57:48

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Used to be with Windows XP "hardware profiles", that you could dual boot a VM and bare metal from the same installation...as if you had a docking station.  You had to have a patch to one of the .sys files if I recall correctly.  Haven't tried 7+

Offline

#4828 2015-04-17 21:04:07

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Offline

#4829 2015-04-17 21:35:03

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Check the CPU. And disks. I've sudden FPS drops when host heavily loads the hard drive, or spikes when host is changing CPU's frequency.
Dedicating(pinning) CPUs will help.

@mostlyharmless
In win7+ you do this same way as XP - enable the "generic" hard drive controller driver, boot up in a different environment and install the needed drivers. The system won't delete drivers for unplugged hardware, so you should be able to boot both ways.


As for "poking EFI loader the right way" - usually that involves copying EFIVARS from bare metal to VM and modifying them to point to the same ESP(partition with EFI loader) in different environments.

Last edited by Duelist (2015-04-17 22:53:15)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4830 2015-04-17 23:26:04

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
fseven wrote:

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Check the CPU. And disks. I've sudden FPS drops when host heavily loads the hard drive, or spikes when host is changing CPU's frequency.
Dedicating(pinning) CPUs will help.

@mostlyharmless
In win7+ you do this same way as XP - enable the "generic" hard drive controller driver, boot up in a different environment and install the needed drivers. The system won't delete drivers for unplugged hardware, so you should be able to boot both ways.


As for "poking EFI loader the right way" - usually that involves copying EFIVARS from bare metal to VM and modifying them to point to the same ESP(partition with EFI loader) in different environments.

The link in the first post is broken, where can I read up on dedicating/pinning CPUs?

Last edited by fseven (2015-04-17 23:28:48)

Offline

#4831 2015-04-18 00:04:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:
Duelist wrote:
fseven wrote:

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Check the CPU. And disks. I've sudden FPS drops when host heavily loads the hard drive, or spikes when host is changing CPU's frequency.
Dedicating(pinning) CPUs will help.

@mostlyharmless
In win7+ you do this same way as XP - enable the "generic" hard drive controller driver, boot up in a different environment and install the needed drivers. The system won't delete drivers for unplugged hardware, so you should be able to boot both ways.


As for "poking EFI loader the right way" - usually that involves copying EFIVARS from bare metal to VM and modifying them to point to the same ESP(partition with EFI loader) in different environments.

The link in the first post is broken, where can I read up on dedicating/pinning CPUs?

Apart from ITT, you can read redhat's guide. It's very useful when you're pursuiting high performance.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4832 2015-04-18 00:45:20

fkoehler
Member
Registered: 2015-04-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \ -name win81 \ -enable-kvm \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -cpu host,kvm=off \ -smp 4,sockets=1 \ -m 8G \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/fabian/win81.iso,id=cd0,if=none \ -device scsi-cd,drive=cd0 \ -drive file=/home/fabian/virtio.iso,id=cd1,if=none \ -device ide-cd,bus=ide.1,drive=cd1 \ -drive file=/dev/sda3,id=disk0,format=raw,if=none \ -device scsi-hd,drive=disk0 \ -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

Offline

#4833 2015-04-18 10:20:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fkoehler wrote:

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \ -name win81 \ -enable-kvm \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -cpu host,kvm=off \ -smp 4,sockets=1 \ -m 8G \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/fabian/win81.iso,id=cd0,if=none \ -device scsi-cd,drive=cd0 \ -drive file=/home/fabian/virtio.iso,id=cd1,if=none \ -device ide-cd,bus=ide.1,drive=cd1 \ -drive file=/dev/sda3,id=disk0,format=raw,if=none \ -device scsi-hd,drive=disk0 \ -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

Type exit or quit(whatever works) in the shell and you'll get into the interactive menu, where you'll find your way around.
The problem is - OVMF and windows are very picky about what type of controller you use to connect .iso images to the VM.
OVMF whitepaper suggests this way:

 # The Fedora installer disk appears as an IDE CD-ROM in the guest. This is # the 2nd boot option. OPTS="$OPTS -drive id=cd0,if=none,format=raw,readonly" OPTS="$OPTS,file=Fedora-Live-Xfce-x86_64-20-1.iso" OPTS="$OPTS -device ide-cd,bus=ide.1,drive=cd0,bootindex=1"

And you should connect virtio windows drivers iso(virtio.iso) in some similar way for windows to be able to use virtio.

P.S.
-cpu kvm=off is needed only for nvidia drivers which detect KVM and shut themselves down. Since you're using radeon, you can drop that line and activate hyper-v enlightenments instead.

Last edited by Duelist (2015-04-18 10:22:39)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4834 2015-04-18 22:28:17

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi,
when I want to use a newer version of the ovmf-svn package in my VM, is it possible to keep the part where the settings are stored in from the old build?

Thanks
apex

The split files for OVMF have been implemented for exactly this reason. When you upgrade the OVMF package on the host, the firmware binary (OVMF_CODE.fd) and the varstore template (OVMF_VARS.fd) are both updated. The code part is expected to see a change (that's what motivates the release of an updated OVMF package in the first place). The OVMF_VARS.fd file (the varstore template) is not expected to change.

In any case, since your virtual machines use their private variable stores (which had been originally copied from the then-installed OVMF_VARS.fd file, and then modified by the individual virtual machines in different, individual ways, and are all located under /var/lib/libvirt/qemu/nvram/ in a libvirt setup), such an OVMF package upgrade affects the firmware binary for *all* virtual machines (that's the goal), and it affects the varstores of *no* virtual machines (that is also the goal). The "new" OVMF_VARS.fd file (if it changed during the update, but it really doesn't) would only affect virtual machines that you'd *create* after the pkg update.

So, the short version is, if you use the split files (which you should), and your package maintainer is not doing crazy things (like changing from "secure boot disabled build" to "secure boot enabled build" in a simple package upgrade), then you should be all set; just upgrade the OVMF package and continue using your VM's own private varstore with the new binary.

(Note that I have no clue if "ovmf-svn" ships the split files.)

There's obviously one issue that needs to be mentioned here: what if the firmware binary (OVMF_CODE.fd) is modified in such a way that it becomes incompatible with the earlier varstore template (and consequently, with all individual varstores that had been instantiated from that earlier varstore template)? In that case the new package will (well... should!) install the firmware binary under a different name (ie. not OVMF_CODE.fd, but something else).

This way your old virtual machines won't even boot (because qemu won't find the central firmware binary under the original name), which is certainly favorable to crashing. There are two options then: either move your virtual machine to the new firmware (by deleting its old varstore, and creating the new varstore from the new template -- libvirt should do this for you automatically, if you delete the old varstore and also remove the nvram element from the domain XML), *or* you can keep the OVMF_CODE.fd file from the earlier package around, in a different location, and re-point the domain XML for all affected VMs to the new location of the old firmware binary.

If you pick option #1, then the VM will clearly lose its non-volatile variables. That should be no problem though, as described here:

http://blog.uncooperative.org/blog/2014 … partition/

installed UEFI OSes have (... should have) a fallback utility on their ESPs that get loaded if everything else fails, and they should regenerate some sane Boot#### options.

Offline

#4835 2015-04-18 22:42:58

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fkoehler wrote:

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \ -name win81 \ -enable-kvm \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -cpu host,kvm=off \ -smp 4,sockets=1 \ -m 8G \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/fabian/win81.iso,id=cd0,if=none \ -device scsi-cd,drive=cd0 \ -drive file=/home/fabian/virtio.iso,id=cd1,if=none \ -device ide-cd,bus=ide.1,drive=cd1 \ -drive file=/dev/sda3,id=disk0,format=raw,if=none \ -device scsi-hd,drive=disk0 \ -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

First, you're using the varstore *template* as the VM's private variable store. That's wrong. Please use libvirt, or else instantiate the VM's own varstore from the template manually. See the "cp" command in the wiki. https://wiki.archlinux.org/index.php/PC … U_commands

Second, I think your virtio-scsi setup may not be right.  For -device scsi-hd and -device scsi-cd, I'd add the bus=scsi.0 property.

Finally, add the bootindex=0 property to your scsi-hd device, and bootindex=1 to scsi-cd. (If your target disk is empty / unformatted, this is the most convenient setting.) Otherwise, reverse the order.

I'd have to see your OVMF debug log (-debugcon file:debug.log -global isa-debugcon.iobase=0x402) to know more.

Offline

#4836 2015-04-18 22:46:10

wikavalier
Member
Registered: 2013-01-22
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fkoehler wrote:

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \ -name win81 \ -enable-kvm \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -cpu host,kvm=off \ -smp 4,sockets=1 \ -m 8G \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/fabian/win81.iso,id=cd0,if=none \ -device scsi-cd,drive=cd0 \ -drive file=/home/fabian/virtio.iso,id=cd1,if=none \ -device ide-cd,bus=ide.1,drive=cd1 \ -drive file=/dev/sda3,id=disk0,format=raw,if=none \ -device scsi-hd,drive=disk0 \ -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

I also spent many hours figuring out what I did wrong!!! But the thing is that you just need to type "exit" and an interactive menu will appear. Then select something like boot from file, search for the windows dvd and browse the EFI folder.

I cant exactly tell since you only need to do it for the installation. After that, it seems that the uefi knows from where to boot.

One question: is there any way to use the nvidia/nouveau drivers with vfio, like lxc or cgroups? I would like to passthrough and use the nvidia card for linux, not at the same time, but without rebooting

EDIT
hmmmmm.... perhaps kernel 4.0?

Last edited by wikavalier (2015-04-19 00:10:35)

Offline

#4837 2015-04-19 14:00:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

wikavalier wrote:

One question: is there any way to use the nvidia/nouveau drivers with vfio, like lxc or cgroups? I would like to passthrough and use the nvidia card for linux, not at the same time, but without rebooting

EDIT
hmmmmm.... perhaps kernel 4.0?

You can write up a small script which will unbind the devices from vfio-pci after the VM shutdown and bind them back to whatever thre driver you need.

You could make a systemd unit which binds the devices and runs the VM, and also shuts down the VM and rebinds the devices back on unit shutdown.
I think that may be possible to implement this via libvirt, but i'm not sure.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4838 2015-04-19 16:49:17

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've got a pretty simple stable instance running, not everything is tested for full performance.

Here is the script:

#!/bin/bash #kvm opt='-enable-kvm ' #Memory opt+='-m 8192 ' #cpu opt+='-cpu host,kvm=off ' opt+='-smp 4,cores=2,threads=2,sockets=1 ' #vga #opt+='-vga qxl ' #gpu bind opt+='-device vfio-pci,host=01:00.0 ' #ovmf opt+='-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd ' opt+='-drive if=pflash,format=raw,file=/vm/7_vars.fd ' #virtio scsi controller opt+='-device virtio-scsi-pci,id=scsi ' #disk opt+='-drive file=/dev/mapper/vmdisks-win7,id=disk,format=raw,if=none -device scsi-hd,drive=disk ' #virtio drivers #opt+='-drive file=/vm/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd ' #install cdrom #opt+='-cdrom /vm/X17-58997.iso ' qemu-system-x86_64 $opt

Offline

#4839 2015-04-19 17:16:46

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You can write up a small script which will unbind the devices from vfio-pci after the VM shutdown and bind them back to whatever thre driver you need.

You could make a systemd unit which binds the devices and runs the VM, and also shuts down the VM and rebinds the devices back on unit shutdown.
I think that may be possible to implement this via libvirt, but i'm not sure.

I tried that, and here's something I ran into (sorry for the funky column alignment):

$echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/unbind $ls /sys/bus/pci/devices/0000\:01\:00.0/ boot_vga config device enable iommu_group local_cpus numa_node rescan resource1 resource3_wc subsystem uevent broken_parity_status consistent_dma_mask_bits dma_mask_bits firmware_node irq modalias power resource resource1_wc resource5 subsystem_device vendor class d3cold_allowed driver_override iommu local_cpulist msi_bus remove resource0 resource3 rom subsystem_vendor

*Assumption disclaimer* There is no more driver file in the device's dir because pci-stub claimed it. So in order for the host to use it, the device would need to be bound to a vfio group and utilized by the host somehow? I've some more testing to do since I would like the same thing wikavalier wants.

Offline

#4840 2015-04-19 17:53:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:
Duelist wrote:

You can write up a small script which will unbind the devices from vfio-pci after the VM shutdown and bind them back to whatever thre driver you need.

You could make a systemd unit which binds the devices and runs the VM, and also shuts down the VM and rebinds the devices back on unit shutdown.
I think that may be possible to implement this via libvirt, but i'm not sure.

I tried that, and here's something I ran into (sorry for the funky column alignment):

$echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/unbind $ls /sys/bus/pci/devices/0000\:01\:00.0/ boot_vga config device enable iommu_group local_cpus numa_node rescan resource1 resource3_wc subsystem uevent broken_parity_status consistent_dma_mask_bits dma_mask_bits firmware_node irq modalias power resource resource1_wc resource5 subsystem_device vendor class d3cold_allowed driver_override iommu local_cpulist msi_bus remove resource0 resource3 rom subsystem_vendor

*Assumption disclaimer* There is no more driver file in the device's dir because pci-stub claimed it. So in order for the host to use it, the device would need to be bound to a vfio group and utilized by the host somehow? I've some more testing to do since I would like the same thing wikavalier wants.

Whaat?
You do the unbind. Then you do lspci to be sure that you've unbound your device.
And then you bind it to whatever driver you want.
pci-stub is a device driver too.
It is needed only for safety reasons, to make sure device doesn't get grabbed by some crazy driver which is loaded before vfio-bind systemd unit is launched.
When the system is fully booted, you can unbind the device from all drivers and have no kernel driver loaded for it.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4841 2015-04-19 20:06:28

fkoehler
Member
Registered: 2015-04-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
fkoehler wrote:

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \ -name win81 \ -enable-kvm \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -cpu host,kvm=off \ -smp 4,sockets=1 \ -m 8G \ -device vfio-pci,host=01:00.0 \ -device vfio-pci,host=01:00.1 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/home/fabian/win81.iso,id=cd0,if=none \ -device scsi-cd,drive=cd0 \ -drive file=/home/fabian/virtio.iso,id=cd1,if=none \ -device ide-cd,bus=ide.1,drive=cd1 \ -drive file=/dev/sda3,id=disk0,format=raw,if=none \ -device scsi-hd,drive=disk0 \ -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

First, you're using the varstore *template* as the VM's private variable store. That's wrong. Please use libvirt, or else instantiate the VM's own varstore from the template manually. See the "cp" command in the wiki. https://wiki.archlinux.org/index.php/PC … U_commands

Second, I think your virtio-scsi setup may not be right.  For -device scsi-hd and -device scsi-cd, I'd add the bus=scsi.0 property.

Finally, add the bootindex=0 property to your scsi-hd device, and bootindex=1 to scsi-cd. (If your target disk is empty / unformatted, this is the most convenient setting.) Otherwise, reverse the order.

I'd have to see your OVMF debug log (-debugcon file:debug.log -global isa-debugcon.iobase=0x402) to know more.

Still having the UEFI issue. I am using vga=std to get everything to boot first. I made changes to my qemu start according to Duelist's and lersek's replies. My startup script is below. If I use an ubuntu iso it is getting detected automatically and I a land on the iso's grub (cannot boot as the kernel crashes for some reason). Tried archlinux iso which does not get detected. Therefore I type exit to get to the interactive menu where I go to Boot Maintenance Manager where the Archlinux iso was selectable I chose the x64 efi (pressed enter) but landed back on the Boot Maintenance Manager screen. My windows does not show up at all. debug.log is created but empty.

I also have another question that will only be relevant if I finally get it to work: using the vga=none graphics I get the output on another screen. To use mouse or keyboard I can give the vm control over the usb devices. But then there will be no way to switch it back to the host or will it?


#!/bin/bash QEMU=qemu-system-x86_64 QEMU_OPTS="" function add() { for opt in $*; do QEMU_OPTS+=" ${opt}" done } # name add "-name win81" # debug options add "-debugcon file:debug.log -global isa-debugcon.iobase=0x402" # enable kvm virtualization add "-enable-kvm" # UEFI via ovmf add "-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin" add "-drive if=pflash,format=raw,file=/tmp/ovmf_vars.bin" # cpu add "-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" add "-smp 4,sockets=1" # memory add "-m 8G" # pci add "-device vfio-pci,host=01:00.0" add "-device vfio-pci,host=01:00.1" # graphics mode #add "-vga none" # scsi controller add "-device virtio-scsi-pci,id=scsi" # hard drives add "-drive file=/dev/sda3,id=disk0,format=raw,if=none -device scsi-hd,drive=disk0,bus=scsi.0,bootindex=1" # cd drives add "-drive file=/home/fabian/win81.iso,id=cd0,if=none,format=raw,readonly -device ide-cd,bus=ide.1,drive=cd0,bootindex=0" add "-drive file=/home/fabian/virtio.iso,id=cd1,if=none,format=raw,readonly -device scsi-cd,bus=scsi.0,drive=cd1" # usb #add "-usb" # logitech receiver #add "-usbdevice host:046d:c52b" # sound hardware # add "-soundhw hda" # boot opitions add "-boot menu=on" cp /usr/share/ovmf/x64/ovmf_vars_x64.bin /tmp/ovmf_vars.bin echo "${QEMU} ${QEMU_OPTS}" exec ${QEMU} ${QEMU_OPTS}

Offline

#4842 2015-04-19 21:48:53

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fkoehler wrote:

I also have another question that will only be relevant if I finally get it to work: using the vga=none graphics I get the output on another screen. To use mouse or keyboard I can give the vm control over the usb devices. But then there will be no way to switch it back to the host or will it?

You can use -device qxl to make qemu provide you a small SDL-based surface for mouse and keyboard capturing purposes.
It won't show anything, and the device won't work, but who cares since we got the input.
In libvirt i have Graphics->VNC, and it gives me the same small window for same purposes.
Also, there is synergy and hardware KVM switches, spice and various stuff to try.
You could've just make qemu's VNC server running and connect to it from the host.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4843 2015-04-20 09:20:16

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, all:

my environment:
kernel: 3.18.6 with i915 patches.
qemu: 2.1.3
arch: Intel Xeon series
type: legacy VGA passthrough

There is a problem when I passthrough AMD HD7850 card to VM. the catd can passthrough successfully.
but when I installed driver, the vm crashed with follow error:
pcieport 0000:00:03.0: AER: Multiple Uncorrected (Fatal) error received: id=0000
vfio-pci 0000:04:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0400(Unregistered Agent ID)
vfio-pci 0000:04:00.1: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0401(Unregistered Agent ID)

qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

If I just passthrough the VGA card, without hdmi audio, the VM worked well with driver. I don't know why, somebody can explain to me, thanks.

Offline

#4844 2015-04-20 09:38:13

acehege
Member
Registered: 2015-04-20
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everybody,

I have gotten KVM VGA-Passthrough to work using this awesome forum and all of your experiences, code snippets and comments.

BUT, everytime i reboot/shutdown a VM with my passed-through Nvidia Cards ( GT630 ), it gives me the dreaded "Error 43" code, and the only fix is to reboot the arch linux host.

--- Info ---

1. I am running the linux-vfio kernel from AUR

2. I have four identical Nvidia GT630 cards

3. I am using OVFM in the VMs

4. The VMs are running Windows 8.1

5. kvm=off is set.

6. I have tried with and without x-vga=on

7. I have tried with and without the following in grub: pcie_acs_override=downstream i915.enable_hd_vgaarb=1

8. I have tried with virt-manager and also commandline qemu, same symptoms.

My current kernel version is 3.19.3.

--- /Info ---

Any help would be highly appreciated!

If any of you need more information or snippets from the system, just ask, and I will deliver wink

Offline

#4845 2015-04-20 16:43:26

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You can use -device qxl to make qemu provide you a small SDL-based surface for mouse and keyboard capturing purposes.
It won't show anything, and the device won't work, but who cares since we got the input.
In libvirt i have Graphics->VNC, and it gives me the same small window for same purposes.
Also, there is synergy and hardware KVM switches, spice and various stuff to try.
You could've just make qemu's VNC server running and connect to it from the host.

I was wondering if it is technically possible to emulate a kvm switch? Kind of like attaching and detaching the usb device, but leaving the slot active (real kvm switches do something similar so you dont have to wait 10 seconds for windows to load the drivers every time).
Because the network based options arent't really ideal when latency is important and switching by attaching/detaching always takes a few seconds.

Last edited by fseven (2015-04-20 16:45:03)

Offline

#4846 2015-04-20 18:44:13

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

acehege wrote:

Hey everybody,

I have gotten KVM VGA-Passthrough to work using this awesome forum and all of your experiences, code snippets and comments.

BUT, everytime i reboot/shutdown a VM with my passed-through Nvidia Cards ( GT630 ), it gives me the dreaded "Error 43" code, and the only fix is to reboot the arch linux host.

--- Info ---

1. I am running the linux-vfio kernel from AUR

2. I have four identical Nvidia GT630 cards

3. I am using OVFM in the VMs

4. The VMs are running Windows 8.1

5. kvm=off is set.

6. I have tried with and without x-vga=on

7. I have tried with and without the following in grub: pcie_acs_override=downstream i915.enable_hd_vgaarb=1

8. I have tried with virt-manager and also commandline qemu, same symptoms.

My current kernel version is 3.19.3.

--- /Info ---

Any help would be highly appreciated!

If any of you need more information or snippets from the system, just ask, and I will deliver wink

Have you tried loading your gpu rom with the romfile= parameter? check out this post https://bbs.archlinux.org/viewtopic.php … 7#p1511177

Offline

#4847 2015-04-20 19:05:15

akawoz
Member
Registered: 2015-02-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great success here - my gear / versions list.

Any questions please let me know - been battling this for ages.

Hardware:
Asrock 990FX Extreme9
AMD FX-8370 8-core CPU
32GB DDR3
1x GeForce GT 730 (host)
1x GeForce GTX 960 (VM)
1x Renasas 4x USB 3 PCIe 1x (VM)
no overclocking or special BIOS changes except:
IO-MMU=ON
USB 3 legacy support = OFF

Host OS:
Arch Linux stable repos
Kernel 3.19.3-3-vfio from: https://aur.archlinux.org/packages/linux-vfio/
No patches

Guest OS:
Windows 8.1 x64
Latest NVIDIA Experience drivers

[root@cloud ~]# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D)
00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E)
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F)
00:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port G)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A)
00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link)
00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI Express bridge for GPP2 port 1
00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port B)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5
01:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 730] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
03:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
05:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
07:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
07:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)
08:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
08:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
09:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
0b:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)
0c:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection

[root@cloud ~]# cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES="piix ide_disk reiserfs"
MODULES="nouveau pci-stub btrfs"

[root@cloud ~]# cat /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=noagp pci-stub.ids=10de:1401,10de:0fba,1912:0014"

[root@cloud qemu]# cat gpu-bios2.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit gpu-bios2
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>gpu-bios2</name>
  <uuid>dd9ac642-5098-4c65-9c87-8d7d33ed32b6</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='off'/>
      <vapic state='off'/>
      <spinlocks state='off'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/data/images/gpu-bios.qcow2'/>
      <target dev='hda' bus='ide'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='ide' index='0'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:99:bf:7d'/>
      <source dev='enp12s0' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:79:67:c4'/>
      <source network='isolated'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </sound>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=06:00.0'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,kvm=off'/>
  </qemu:commandline>
</domain>

Last edited by akawoz (2015-04-20 19:10:38)

Offline

#4848 2015-04-20 19:12:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@akawoz

You're so close to having a pretty clean setup, why ruin it with q35 and all those <qemu:arg> statements?  You're already telling libvirt to use kvm=off with the hidden option, no need to specify it again.  If you swap to 440fx you can just use normal <hostdev> options for the GPU, audio, and USB.  And with a win8.1 guest and card that new, I'm not sure why you wouldn't use OVMF rather than BIOS to drop x-vga=on.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4849 2015-04-20 19:47:32

akawoz
Member
Registered: 2015-02-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah agreed - but I've been playing around with this trying to get it working for months (started with an X10SAT, Radeon 7850) so I'm just pleased to finally have it working.

Good point re duplication of kvm=off statements.

Will take a look at <hostdev>.

Tried with OVMF but Windows kept freezing at boot - so followed the path of least resistance. Now I've got it running and stable I'd try other settings and options. Cheers.

aw wrote:

@akawoz

You're so close to having a pretty clean setup, why ruin it with q35 and all those <qemu:arg> statements?  You're already telling libvirt to use kvm=off with the hidden option, no need to specify it again.  If you swap to 440fx you can just use normal <hostdev> options for the GPU, audio, and USB.  And with a win8.1 guest and card that new, I'm not sure why you wouldn't use OVMF rather than BIOS to drop x-vga=on.

Offline

#4850 2015-04-20 21:06:16

fkoehler
Member
Registered: 2015-04-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am back. I still had no luck with ovmf-svn from the AUR. ubuntu would boot (only without the graphics card as the radeon driver would crash), none of the archlinux entries would boot and for the windows iso I do not even have an option to boot from file. debug.log remained empty all the time. So I tried ovmf from this repo as described in the wiki. Now I get a black screen but at least the debug.log is not empty. The error messages lead me to old edk2 issues that were fixed years ago. Maybe you understand them?

startup script

#!/bin/bash QEMU=qemu-system-x86_64 QEMU_OPTS="" function add() { for opt in $*; do QEMU_OPTS+=" ${opt}" done } # name add "-name win81" # debug options add "-debugcon file:debug.log -global isa-debugcon.iobase=0x402" # enable kvm virtualization add "-enable-kvm" # uefi add "-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd" add "-drive if=pflash,format=raw,file=/tmp/ovmf_vars.fd" # cpu add "-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" add "-smp 4,sockets=1" # memory add "-m 16G" # pci add "-device vfio-pci,host=01:00.0" add "-device vfio-pci,host=01:00.1" # graphics mode add "-vga none" # scsi controller add "-device virtio-scsi-pci,id=scsi" # hard drives add "-drive file=/dev/sda3,id=disk0,format=raw,if=none -device scsi-hd,drive=disk0,bus=scsi.0,bootindex=0" # cdrom add "-drive file=/home/fabian/win81.iso,id=cd0,if=none,format=raw,readonly -device ide-cd,bus=ide.1,drive=cd0,bootindex=1" add "-drive file=/home/fabian/virtio.iso,id=cd1,if=none,format=raw,readonly -device scsi-cd,bus=scsi.0,drive=cd1" add "-boot menu=on" # start cp /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /tmp/ovmf_vars.fd echo "${QEMU} ${QEMU_OPTS}" exec ${QEMU} ${QEMU_OPTS}

ovmf errors from debug.log

SecCoreStartupWithStack(0xFFFCC000, 0x818000) Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000 Register PPI Notify: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Register PPI Notify: EA7CA24B-DED5-4DAD-A389-BF827E8F9B38 Install PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: DBE23AA9-A345-4B97-85B6-B226F1617389 Loading PEIM at 0x00000838EA0 EntryPoint=0x00000839100 PcdPeim.efi Install PPI: 06E81C58-4AD7-44BC-8390-F10265F72480 Install PPI: 01F34D25-4DE2-23AD-3FF3-36353FF323F1 Install PPI: 4D8B155B-C059-4C8F-8926-06FD4331DB8A Install PPI: A60C6B59-E459-425D-9C69-0BCC9CB27D81 Loading PEIM at 0x000008421A0 EntryPoint=0x00000842400 StatusCodePei.efi Install PPI: 229832D3-7A30-4B36-B827-F40CB7D45436 Loading PEIM at 0x00000847EA0 EntryPoint=0x00000848100 PlatformPei.efi Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Platform PEIM Loaded CMOS: 00: 56 00 04 00 21 00 02 20 04 15 26 02 10 80 00 00 10: 40 00 00 00 07 80 02 FF FF 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: FF FF 20 00 00 BF 00 20 30 00 00 00 00 12 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 40 03 00 03 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Select Item: 0x19 Select Item: 0x25 S3 support was detected on QEMU Install PPI: 7408D748-FC8C-4EE6-9288-C4BEC092A410 PeiInstallPeiMemory MemoryBegin 0xBC000000, MemoryLength 0x4000000 QemuInitializeRam called Allocated Memory unaligned: Address = 0xBFFD0000, Pages = 0x30, Type = 6 After aligning to 0x10000 bytes: Address = 0xBFFD0000, Pages = 0x20 Updated aligned-mem HOB with BaseAddress = BFFD0000, Length = 20000, MemoryType = 6 Created after-mem HOB with BaseAddress = BFFF0000, Length = 10000, MemoryType = 7 Reserved variable store memory: 0xBFFD0000; size: 128kb Platform PEI Firmware Volume Initialization Install PPI: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Notify: PPI Guid: 49EDB1C1-BF21-4761-BB12-EB0031AABB39, Peim notify entry point: 8242CA The 1th FV start address is 0x00000900000, size is 0x00800000, handle is 0x900000 Temp Stack : BaseAddress=0x814000 Length=0x4000 Temp Heap : BaseAddress=0x810000 Length=0x1980 Total temporary memory: 32768 bytes. temporary memory stack ever used: 16384 bytes. temporary memory heap used: 6528 bytes. Old Stack size 16384, New stack size 131072 Stack Hob: BaseAddress=0xBC000000 Length=0x20000 Heap Offset = 0xBB810000 Stack Offset = 0xBB808000 TemporaryRamMigration(0x810000, 0xBC01C000, 0x8000) Loading PEIM at 0x000BFFB6000 EntryPoint=0x000BFFB6260 PeiCore.efi Reinstall PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Reinstall PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A Reinstall PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: F894643D-C449-42D1-8EA8-85BDD8C65BDE Loading PEIM at 0x000BFFAC000 EntryPoint=0x000BFFAC260 DxeIpl.efi Install PPI: 0AE8CE5D-E448-4437-A8D7-EBF5F194F731 Install PPI: 1A36E4E7-FAB6-476A-8E75-695A0576FDD7 Loading PEIM at 0x000BFFA1000 EntryPoint=0x000BFFA1260 S3Resume2Pei.efi Install PPI: 6D582DBC-DB85-4514-8FCC-5ADF6227B147 DXE IPL Entry Loading PEIM at 0x000BFF64000 EntryPoint=0x000BFF64260 DxeCore.efi Loading DXE CORE at 0x000BFF64000 EntryPoint=0x000BFF64260 Install PPI: 605EA650-C65C-42E1-BA80-91A52AB618C6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BFF938C8 HOBLIST address in DXE = 0xBFD19018 Memory Allocation 0x0000000A 0x818000 - 0x81FFFF Memory Allocation 0x0000000A 0x810000 - 0x817FFF Memory Allocation 0x0000000A 0x807000 - 0x807FFF Memory Allocation 0x0000000A 0x800000 - 0x805FFF Memory Allocation 0x0000000A 0x806000 - 0x806FFF Memory Allocation 0x00000006 0xBFFD0000 - 0xBFFEFFFF Memory Allocation 0x00000007 0xBFFF0000 - 0xBFFFFFFF Memory Allocation 0x0000000A 0x820000 - 0x8FFFFF Memory Allocation 0x00000004 0x900000 - 0x10FFFFF Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF Memory Allocation 0x00000004 0xBFFCF000 - 0xBFFCFFFF Memory Allocation 0x00000004 0xBFFB6000 - 0xBFFCEFFF Memory Allocation 0x00000004 0xBFFAC000 - 0xBFFB5FFF Memory Allocation 0x00000004 0xBFFA1000 - 0xBFFABFFF Memory Allocation 0x00000004 0xBFF64000 - 0xBFFA0FFF Memory Allocation 0x00000003 0xBFF64000 - 0xBFFA0FFF Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF Memory Allocation 0x00000004 0xBFF02000 - 0xBFF43FFF Memory Allocation 0x00000004 0xBC000000 - 0xBC01FFFF FV Hob 0x900000 - 0x10FFFFF InstallProtocolInterface: D8117CFE-94A6-11D4-9A3A-0090273FC14D BFF93360 InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFD15270 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BFD15658 InstallProtocolInterface: 220E73B6-6BDB-4413-8405-B974B108619A BFD14C70 InstallProtocolInterface: EE4E5898-3914-4259-9D6E-DC7BD79403CF BFF93588 Loading driver 9B680FCE-AD6B-4F3A-B60B-F59899003443 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF732180 Loading driver at 0x000BFE8C000 EntryPoint=0x000BFE8C2AF DevicePathDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF732E58 InstallProtocolInterface: 0379BE4E-D706-437D-B037-EDB82FB772A4 BFE98200 InstallProtocolInterface: 8B843E20-8132-4852-90CC-551A4E4A7F1C BFE98240 InstallProtocolInterface: 05C99A21-C70F-4AD2-8A5F-35DF3343F51E BFE98250 Loading driver 80CF7257-87AB-47F9-A3FE-D50B76D89541 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7325C0 Loading driver at 0x000BFE7F000 EntryPoint=0x000BFE7F2AF PcdDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724018 InstallProtocolInterface: 11B34006-D85B-4D0A-A290-D5A571310EF7 BFE87C60 InstallProtocolInterface: 13A3F0F6-264A-3EF0-F2E0-DEC512342F34 BFE87D60 InstallProtocolInterface: 5BE40F57-FA68-4610-BBBF-E9C5FCDAD365 BFE87DF0 InstallProtocolInterface: FD0F4478-0EFD-461D-BA2D-E58C45FD5F5E BFE87E10 Loading driver 733CBAC2-B23F-4B92-BC8E-FB01CE5907B7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727040 Loading driver at 0x000BFEC3000 EntryPoint=0x000BFEC32AF FvbServicesRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724698 QEMU Flash: Attempting flash detection at FFE00000 QemuFlashDetected => FD behaves as FLASH QemuFlashDetected => Yes Variable FV header is not valid. It will be reinitialized. Installing QEMU flash FVB InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFEF0DF0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF727D98 Loading driver FEDE0A1B-BCA2-4A9F-BB2B-D9FD7DEC2E9F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727340 Loading driver at 0x000BFEB9000 EntryPoint=0x000BFEB92AF StatusCodeRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF727598 InstallProtocolInterface: D2B2B828-0826-48A7-B3DF-983C006024F0 BFEBF600 Loading driver B601F8C4-43B7-4784-95B1-F4226CB40CEE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72FBC0 Loading driver at 0x000BFEB1000 EntryPoint=0x000BFEB12AF RuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F298 InstallProtocolInterface: B7DFB4E1-052F-449F-87BE-9818FC91B733 BFEB56C0 Loading driver F80697E9-7FD6-4665-8646-88E33EF71DFC InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E040 Loading driver at 0x000BFDF9000 EntryPoint=0x000BFDF92AF SecurityStubDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F698 InstallProtocolInterface: 94AB2F58-1438-4EF1-9152-18941A3A0E68 BFE57B70 InstallProtocolInterface: A46423E3-4617-49F1-B9FF-D1BFA9115839 BFE57B68 Loading driver 13AC6DD0-73D0-11D4-B06B-00AA00BD6DE7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E400 Loading driver at 0x000BFDED000 EntryPoint=0x000BFDED2AF EbcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72EA18 InstallProtocolInterface: 13AC6DD1-73D0-11D4-B06B-00AA00BD6DE7 BF72E798 InstallProtocolInterface: 2755590C-6F3C-42FA-9EA4-A3BA543CDA25 BF72DE58 InstallProtocolInterface: AAEACCFD-F27B-4C17-B610-75CA1F2DFB52 BF72DE18 Loading driver 79CA4208-BBA1-4A9A-8456-E1E66A81484E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D140 Loading driver at 0x000BFDE8000 EntryPoint=0x000BFDE82AF Legacy8259.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72D8D8 InstallProtocolInterface: 38321DBA-4FE0-4E17-8AEC-413055EAEDC1 BFDEB040 Loading driver A19B1FE7-C1BC-49F8-875F-54A5D542443F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D500 Loading driver at 0x000BFDE2000 EntryPoint=0x000BFDE22AF CpuIo2Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72CF98 InstallProtocolInterface: AD61F191-AE5F-4C0E-B9FA-E869D288C64F BFDE53A0 Loading driver 1A1E4886-9517-440E-9FDE-3BE44CEE2136 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72CB00 Loading driver at 0x000BFDCF000 EntryPoint=0x000BFDCF2AF CpuDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72C998 InstallProtocolInterface: 26BACCB1-6F42-11D4-BCE7-0080C73C8881 BFDDBF20 Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Loading driver 6F0198AA-1F1D-426D-AE3E-39AB633FCC28 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72BE00 Loading driver at 0x000BFEAB000 EntryPoint=0x000BFEAB2AF KbcReset.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728D18 InstallProtocolInterface: 27CFAC88-46CC-11D4-9A38-0090273FC14D 0 Loading driver C8339973-A563-4561-B858-D8476F9DEFC4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7288C0 Loading driver at 0x000BFDC9000 EntryPoint=0x000BFDC92AF Metronome.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728A98 InstallProtocolInterface: 26BACCB2-6F42-11D4-BCE7-0080C73C8881 BFDCBF20 Loading driver 22DC2B60-FE40-42AC-B01F-3AB1FAD9AAD8 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72B9C0 Loading driver at 0x000BFEA3000 EntryPoint=0x000BFEA32AF EmuVariableFvbRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF726F98 EMU Variable FVB Started Disabling EMU Variable FVB since flash variables appear to be supported. Error: Image at 000BFEA3000 start failed: Aborted Loading driver 2226F30F-3D5B-402D-9936-A97184EB4516 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF726040 Loading driver at 0x000BED8A000 EntryPoint=0x000BED8A2AF VariableAuthRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72BA98 Variable Store header is corrupted ASSERT_EFI_ERROR (Status = Volume Corrupt) ASSERT /var/lib/jenkins/jobs/edk2/workspace/rpmbuild/rpm/BUILD/edk2.git/SecurityPkg/VariableAuthenticated/RuntimeDxe/VariableDxe.c(449): !EFI_ERROR (Status)

Offline

#4851 2015-04-20 22:10:19

Fidelix
Member
Registered: 2015-04-19
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@akawoz, my Windows screen also freezes with OVMF if I start with more than 7.8GB of RAM. Don't know what else to do, but other than that everything is working perfectly with my GTX 970.

I only wish I was able to turn on Hyper-v Enlightenments. Damn Nvidia

Offline

#4852 2015-04-20 22:38:40

fkoehler
Member
Registered: 2015-04-17
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fkoehler wrote:

I am back. I still had no luck with ovmf-svn from the AUR. ubuntu would boot (only without the graphics card as the radeon driver would crash), none of the archlinux entries would boot and for the windows iso I do not even have an option to boot from file. debug.log remained empty all the time. So I tried ovmf from this repo as described in the wiki. Now I get a black screen but at least the debug.log is not empty. The error messages lead me to old edk2 issues that were fixed years ago. Maybe you understand them?

startup script

#!/bin/bash QEMU=qemu-system-x86_64 QEMU_OPTS="" function add() { for opt in $*; do QEMU_OPTS+=" ${opt}" done } # name add "-name win81" # debug options add "-debugcon file:debug.log -global isa-debugcon.iobase=0x402" # enable kvm virtualization add "-enable-kvm" # uefi add "-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd" add "-drive if=pflash,format=raw,file=/tmp/ovmf_vars.fd" # cpu add "-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" add "-smp 4,sockets=1" # memory add "-m 16G" # pci add "-device vfio-pci,host=01:00.0" add "-device vfio-pci,host=01:00.1" # graphics mode add "-vga none" # scsi controller add "-device virtio-scsi-pci,id=scsi" # hard drives add "-drive file=/dev/sda3,id=disk0,format=raw,if=none -device scsi-hd,drive=disk0,bus=scsi.0,bootindex=0" # cdrom add "-drive file=/home/fabian/win81.iso,id=cd0,if=none,format=raw,readonly -device ide-cd,bus=ide.1,drive=cd0,bootindex=1" add "-drive file=/home/fabian/virtio.iso,id=cd1,if=none,format=raw,readonly -device scsi-cd,bus=scsi.0,drive=cd1" add "-boot menu=on" # start cp /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /tmp/ovmf_vars.fd echo "${QEMU} ${QEMU_OPTS}" exec ${QEMU} ${QEMU_OPTS}

ovmf errors from debug.log

SecCoreStartupWithStack(0xFFFCC000, 0x818000) Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000 Register PPI Notify: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Register PPI Notify: EA7CA24B-DED5-4DAD-A389-BF827E8F9B38 Install PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: DBE23AA9-A345-4B97-85B6-B226F1617389 Loading PEIM at 0x00000838EA0 EntryPoint=0x00000839100 PcdPeim.efi Install PPI: 06E81C58-4AD7-44BC-8390-F10265F72480 Install PPI: 01F34D25-4DE2-23AD-3FF3-36353FF323F1 Install PPI: 4D8B155B-C059-4C8F-8926-06FD4331DB8A Install PPI: A60C6B59-E459-425D-9C69-0BCC9CB27D81 Loading PEIM at 0x000008421A0 EntryPoint=0x00000842400 StatusCodePei.efi Install PPI: 229832D3-7A30-4B36-B827-F40CB7D45436 Loading PEIM at 0x00000847EA0 EntryPoint=0x00000848100 PlatformPei.efi Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Platform PEIM Loaded CMOS: 00: 56 00 04 00 21 00 02 20 04 15 26 02 10 80 00 00 10: 40 00 00 00 07 80 02 FF FF 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: FF FF 20 00 00 BF 00 20 30 00 00 00 00 12 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 40 03 00 03 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Select Item: 0x19 Select Item: 0x25 S3 support was detected on QEMU Install PPI: 7408D748-FC8C-4EE6-9288-C4BEC092A410 PeiInstallPeiMemory MemoryBegin 0xBC000000, MemoryLength 0x4000000 QemuInitializeRam called Allocated Memory unaligned: Address = 0xBFFD0000, Pages = 0x30, Type = 6 After aligning to 0x10000 bytes: Address = 0xBFFD0000, Pages = 0x20 Updated aligned-mem HOB with BaseAddress = BFFD0000, Length = 20000, MemoryType = 6 Created after-mem HOB with BaseAddress = BFFF0000, Length = 10000, MemoryType = 7 Reserved variable store memory: 0xBFFD0000; size: 128kb Platform PEI Firmware Volume Initialization Install PPI: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Notify: PPI Guid: 49EDB1C1-BF21-4761-BB12-EB0031AABB39, Peim notify entry point: 8242CA The 1th FV start address is 0x00000900000, size is 0x00800000, handle is 0x900000 Temp Stack : BaseAddress=0x814000 Length=0x4000 Temp Heap : BaseAddress=0x810000 Length=0x1980 Total temporary memory: 32768 bytes. temporary memory stack ever used: 16384 bytes. temporary memory heap used: 6528 bytes. Old Stack size 16384, New stack size 131072 Stack Hob: BaseAddress=0xBC000000 Length=0x20000 Heap Offset = 0xBB810000 Stack Offset = 0xBB808000 TemporaryRamMigration(0x810000, 0xBC01C000, 0x8000) Loading PEIM at 0x000BFFB6000 EntryPoint=0x000BFFB6260 PeiCore.efi Reinstall PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Reinstall PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A Reinstall PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: F894643D-C449-42D1-8EA8-85BDD8C65BDE Loading PEIM at 0x000BFFAC000 EntryPoint=0x000BFFAC260 DxeIpl.efi Install PPI: 0AE8CE5D-E448-4437-A8D7-EBF5F194F731 Install PPI: 1A36E4E7-FAB6-476A-8E75-695A0576FDD7 Loading PEIM at 0x000BFFA1000 EntryPoint=0x000BFFA1260 S3Resume2Pei.efi Install PPI: 6D582DBC-DB85-4514-8FCC-5ADF6227B147 DXE IPL Entry Loading PEIM at 0x000BFF64000 EntryPoint=0x000BFF64260 DxeCore.efi Loading DXE CORE at 0x000BFF64000 EntryPoint=0x000BFF64260 Install PPI: 605EA650-C65C-42E1-BA80-91A52AB618C6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BFF938C8 HOBLIST address in DXE = 0xBFD19018 Memory Allocation 0x0000000A 0x818000 - 0x81FFFF Memory Allocation 0x0000000A 0x810000 - 0x817FFF Memory Allocation 0x0000000A 0x807000 - 0x807FFF Memory Allocation 0x0000000A 0x800000 - 0x805FFF Memory Allocation 0x0000000A 0x806000 - 0x806FFF Memory Allocation 0x00000006 0xBFFD0000 - 0xBFFEFFFF Memory Allocation 0x00000007 0xBFFF0000 - 0xBFFFFFFF Memory Allocation 0x0000000A 0x820000 - 0x8FFFFF Memory Allocation 0x00000004 0x900000 - 0x10FFFFF Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF Memory Allocation 0x00000004 0xBFFCF000 - 0xBFFCFFFF Memory Allocation 0x00000004 0xBFFB6000 - 0xBFFCEFFF Memory Allocation 0x00000004 0xBFFAC000 - 0xBFFB5FFF Memory Allocation 0x00000004 0xBFFA1000 - 0xBFFABFFF Memory Allocation 0x00000004 0xBFF64000 - 0xBFFA0FFF Memory Allocation 0x00000003 0xBFF64000 - 0xBFFA0FFF Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF Memory Allocation 0x00000004 0xBFF02000 - 0xBFF43FFF Memory Allocation 0x00000004 0xBC000000 - 0xBC01FFFF FV Hob 0x900000 - 0x10FFFFF InstallProtocolInterface: D8117CFE-94A6-11D4-9A3A-0090273FC14D BFF93360 InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFD15270 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BFD15658 InstallProtocolInterface: 220E73B6-6BDB-4413-8405-B974B108619A BFD14C70 InstallProtocolInterface: EE4E5898-3914-4259-9D6E-DC7BD79403CF BFF93588 Loading driver 9B680FCE-AD6B-4F3A-B60B-F59899003443 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF732180 Loading driver at 0x000BFE8C000 EntryPoint=0x000BFE8C2AF DevicePathDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF732E58 InstallProtocolInterface: 0379BE4E-D706-437D-B037-EDB82FB772A4 BFE98200 InstallProtocolInterface: 8B843E20-8132-4852-90CC-551A4E4A7F1C BFE98240 InstallProtocolInterface: 05C99A21-C70F-4AD2-8A5F-35DF3343F51E BFE98250 Loading driver 80CF7257-87AB-47F9-A3FE-D50B76D89541 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7325C0 Loading driver at 0x000BFE7F000 EntryPoint=0x000BFE7F2AF PcdDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724018 InstallProtocolInterface: 11B34006-D85B-4D0A-A290-D5A571310EF7 BFE87C60 InstallProtocolInterface: 13A3F0F6-264A-3EF0-F2E0-DEC512342F34 BFE87D60 InstallProtocolInterface: 5BE40F57-FA68-4610-BBBF-E9C5FCDAD365 BFE87DF0 InstallProtocolInterface: FD0F4478-0EFD-461D-BA2D-E58C45FD5F5E BFE87E10 Loading driver 733CBAC2-B23F-4B92-BC8E-FB01CE5907B7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727040 Loading driver at 0x000BFEC3000 EntryPoint=0x000BFEC32AF FvbServicesRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724698 QEMU Flash: Attempting flash detection at FFE00000 QemuFlashDetected => FD behaves as FLASH QemuFlashDetected => Yes Variable FV header is not valid. It will be reinitialized. Installing QEMU flash FVB InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFEF0DF0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF727D98 Loading driver FEDE0A1B-BCA2-4A9F-BB2B-D9FD7DEC2E9F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727340 Loading driver at 0x000BFEB9000 EntryPoint=0x000BFEB92AF StatusCodeRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF727598 InstallProtocolInterface: D2B2B828-0826-48A7-B3DF-983C006024F0 BFEBF600 Loading driver B601F8C4-43B7-4784-95B1-F4226CB40CEE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72FBC0 Loading driver at 0x000BFEB1000 EntryPoint=0x000BFEB12AF RuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F298 InstallProtocolInterface: B7DFB4E1-052F-449F-87BE-9818FC91B733 BFEB56C0 Loading driver F80697E9-7FD6-4665-8646-88E33EF71DFC InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E040 Loading driver at 0x000BFDF9000 EntryPoint=0x000BFDF92AF SecurityStubDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F698 InstallProtocolInterface: 94AB2F58-1438-4EF1-9152-18941A3A0E68 BFE57B70 InstallProtocolInterface: A46423E3-4617-49F1-B9FF-D1BFA9115839 BFE57B68 Loading driver 13AC6DD0-73D0-11D4-B06B-00AA00BD6DE7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E400 Loading driver at 0x000BFDED000 EntryPoint=0x000BFDED2AF EbcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72EA18 InstallProtocolInterface: 13AC6DD1-73D0-11D4-B06B-00AA00BD6DE7 BF72E798 InstallProtocolInterface: 2755590C-6F3C-42FA-9EA4-A3BA543CDA25 BF72DE58 InstallProtocolInterface: AAEACCFD-F27B-4C17-B610-75CA1F2DFB52 BF72DE18 Loading driver 79CA4208-BBA1-4A9A-8456-E1E66A81484E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D140 Loading driver at 0x000BFDE8000 EntryPoint=0x000BFDE82AF Legacy8259.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72D8D8 InstallProtocolInterface: 38321DBA-4FE0-4E17-8AEC-413055EAEDC1 BFDEB040 Loading driver A19B1FE7-C1BC-49F8-875F-54A5D542443F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D500 Loading driver at 0x000BFDE2000 EntryPoint=0x000BFDE22AF CpuIo2Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72CF98 InstallProtocolInterface: AD61F191-AE5F-4C0E-B9FA-E869D288C64F BFDE53A0 Loading driver 1A1E4886-9517-440E-9FDE-3BE44CEE2136 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72CB00 Loading driver at 0x000BFDCF000 EntryPoint=0x000BFDCF2AF CpuDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72C998 InstallProtocolInterface: 26BACCB1-6F42-11D4-BCE7-0080C73C8881 BFDDBF20 Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Loading driver 6F0198AA-1F1D-426D-AE3E-39AB633FCC28 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72BE00 Loading driver at 0x000BFEAB000 EntryPoint=0x000BFEAB2AF KbcReset.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728D18 InstallProtocolInterface: 27CFAC88-46CC-11D4-9A38-0090273FC14D 0 Loading driver C8339973-A563-4561-B858-D8476F9DEFC4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7288C0 Loading driver at 0x000BFDC9000 EntryPoint=0x000BFDC92AF Metronome.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728A98 InstallProtocolInterface: 26BACCB2-6F42-11D4-BCE7-0080C73C8881 BFDCBF20 Loading driver 22DC2B60-FE40-42AC-B01F-3AB1FAD9AAD8 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72B9C0 Loading driver at 0x000BFEA3000 EntryPoint=0x000BFEA32AF EmuVariableFvbRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF726F98 EMU Variable FVB Started Disabling EMU Variable FVB since flash variables appear to be supported. Error: Image at 000BFEA3000 start failed: Aborted Loading driver 2226F30F-3D5B-402D-9936-A97184EB4516 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF726040 Loading driver at 0x000BED8A000 EntryPoint=0x000BED8A2AF VariableAuthRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72BA98 Variable Store header is corrupted ASSERT_EFI_ERROR (Status = Volume Corrupt) ASSERT /var/lib/jenkins/jobs/edk2/workspace/rpmbuild/rpm/BUILD/edk2.git/SecurityPkg/VariableAuthenticated/RuntimeDxe/VariableDxe.c(449): !EFI_ERROR (Status)

There was one line wrong in my startup. Here is the updated version:

#!/bin/bash QEMU=qemu-system-x86_64 QEMU_OPTS="" function add() { for opt in $*; do QEMU_OPTS+=" ${opt}" done } # name add "-name win81" # debug options add "-debugcon file:debug.log -global isa-debugcon.iobase=0x402" # enable kvm virtualization add "-enable-kvm" # uefi add "-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd" add "-drive if=pflash,format=raw,file=/tmp/ovmf_vars.fd" # cpu add "-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" add "-smp 4,sockets=1" # memory add "-m 16G" # pci add "-device vfio-pci,host=01:00.0" add "-device vfio-pci,host=01:00.1" # graphics mode # add "-vga none" # scsi controller add "-device virtio-scsi-pci,id=scsi" # hard drives add "-drive file=/dev/sda3,id=disk0,format=raw,if=none -device scsi-hd,drive=disk0,bus=scsi.0,bootindex=0" # cdrom add "-drive file=/home/fabian/win81.iso,id=cd0,if=none,format=raw,readonly -device ide-cd,bus=ide.1,drive=cd0,bootindex=1" add "-drive file=/home/fabian/virtio.iso,id=cd1,if=none,format=raw,readonly -device scsi-cd,bus=scsi.0,drive=cd1" add "-boot menu=on" # start cp -f /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/ovmf_vars.fd echo "${QEMU} ${QEMU_OPTS}" exec ${QEMU} ${QEMU_OPTS}

Now I land on the UEFI Shell. But ther is still no entry for windows in the boot from file menu. Here is my lengthy debug.log

SecCoreStartupWithStack(0xFFFCC000, 0x818000) Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000 Register PPI Notify: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Register PPI Notify: EA7CA24B-DED5-4DAD-A389-BF827E8F9B38 Install PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: DBE23AA9-A345-4B97-85B6-B226F1617389 Loading PEIM at 0x00000838EA0 EntryPoint=0x00000839100 PcdPeim.efi Install PPI: 06E81C58-4AD7-44BC-8390-F10265F72480 Install PPI: 01F34D25-4DE2-23AD-3FF3-36353FF323F1 Install PPI: 4D8B155B-C059-4C8F-8926-06FD4331DB8A Install PPI: A60C6B59-E459-425D-9C69-0BCC9CB27D81 Loading PEIM at 0x000008421A0 EntryPoint=0x00000842400 StatusCodePei.efi Install PPI: 229832D3-7A30-4B36-B827-F40CB7D45436 Loading PEIM at 0x00000847EA0 EntryPoint=0x00000848100 PlatformPei.efi Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Platform PEIM Loaded CMOS: 00: 42 00 35 00 22 00 02 20 04 15 26 02 10 80 00 00 10: 40 00 00 00 07 80 02 FF FF 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: FF FF 20 00 00 BF 00 20 30 00 00 00 00 12 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 40 03 00 03 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Select Item: 0x19 Select Item: 0x25 S3 support was detected on QEMU Install PPI: 7408D748-FC8C-4EE6-9288-C4BEC092A410 PeiInstallPeiMemory MemoryBegin 0xBC000000, MemoryLength 0x4000000 QemuInitializeRam called Allocated Memory unaligned: Address = 0xBFFD0000, Pages = 0x30, Type = 6 After aligning to 0x10000 bytes: Address = 0xBFFD0000, Pages = 0x20 Updated aligned-mem HOB with BaseAddress = BFFD0000, Length = 20000, MemoryType = 6 Created after-mem HOB with BaseAddress = BFFF0000, Length = 10000, MemoryType = 7 Reserved variable store memory: 0xBFFD0000; size: 128kb Platform PEI Firmware Volume Initialization Install PPI: 49EDB1C1-BF21-4761-BB12-EB0031AABB39 Notify: PPI Guid: 49EDB1C1-BF21-4761-BB12-EB0031AABB39, Peim notify entry point: 8242CA The 1th FV start address is 0x00000900000, size is 0x00800000, handle is 0x900000 Temp Stack : BaseAddress=0x814000 Length=0x4000 Temp Heap : BaseAddress=0x810000 Length=0x1980 Total temporary memory: 32768 bytes. temporary memory stack ever used: 16384 bytes. temporary memory heap used: 6528 bytes. Old Stack size 16384, New stack size 131072 Stack Hob: BaseAddress=0xBC000000 Length=0x20000 Heap Offset = 0xBB810000 Stack Offset = 0xBB808000 TemporaryRamMigration(0x810000, 0xBC01C000, 0x8000) Loading PEIM at 0x000BFFB6000 EntryPoint=0x000BFFB6260 PeiCore.efi Reinstall PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3 Reinstall PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A Reinstall PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6 Install PPI: F894643D-C449-42D1-8EA8-85BDD8C65BDE Loading PEIM at 0x000BFFAC000 EntryPoint=0x000BFFAC260 DxeIpl.efi Install PPI: 0AE8CE5D-E448-4437-A8D7-EBF5F194F731 Install PPI: 1A36E4E7-FAB6-476A-8E75-695A0576FDD7 Loading PEIM at 0x000BFFA1000 EntryPoint=0x000BFFA1260 S3Resume2Pei.efi Install PPI: 6D582DBC-DB85-4514-8FCC-5ADF6227B147 DXE IPL Entry Loading PEIM at 0x000BFF64000 EntryPoint=0x000BFF64260 DxeCore.efi Loading DXE CORE at 0x000BFF64000 EntryPoint=0x000BFF64260 Install PPI: 605EA650-C65C-42E1-BA80-91A52AB618C6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BFF938C8 HOBLIST address in DXE = 0xBFD19018 Memory Allocation 0x0000000A 0x818000 - 0x81FFFF Memory Allocation 0x0000000A 0x810000 - 0x817FFF Memory Allocation 0x0000000A 0x807000 - 0x807FFF Memory Allocation 0x0000000A 0x800000 - 0x805FFF Memory Allocation 0x0000000A 0x806000 - 0x806FFF Memory Allocation 0x00000006 0xBFFD0000 - 0xBFFEFFFF Memory Allocation 0x00000007 0xBFFF0000 - 0xBFFFFFFF Memory Allocation 0x0000000A 0x820000 - 0x8FFFFF Memory Allocation 0x00000004 0x900000 - 0x10FFFFF Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF Memory Allocation 0x00000004 0xBFFCF000 - 0xBFFCFFFF Memory Allocation 0x00000004 0xBFFB6000 - 0xBFFCEFFF Memory Allocation 0x00000004 0xBFFAC000 - 0xBFFB5FFF Memory Allocation 0x00000004 0xBFFA1000 - 0xBFFABFFF Memory Allocation 0x00000004 0xBFF64000 - 0xBFFA0FFF Memory Allocation 0x00000003 0xBFF64000 - 0xBFFA0FFF Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF Memory Allocation 0x00000004 0xBFF02000 - 0xBFF43FFF Memory Allocation 0x00000004 0xBC000000 - 0xBC01FFFF FV Hob 0x900000 - 0x10FFFFF InstallProtocolInterface: D8117CFE-94A6-11D4-9A3A-0090273FC14D BFF93360 InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFD15270 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BFD15658 InstallProtocolInterface: 220E73B6-6BDB-4413-8405-B974B108619A BFD14C70 InstallProtocolInterface: EE4E5898-3914-4259-9D6E-DC7BD79403CF BFF93588 Loading driver 9B680FCE-AD6B-4F3A-B60B-F59899003443 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF732180 Loading driver at 0x000BFE8C000 EntryPoint=0x000BFE8C2AF DevicePathDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF732E58 InstallProtocolInterface: 0379BE4E-D706-437D-B037-EDB82FB772A4 BFE98200 InstallProtocolInterface: 8B843E20-8132-4852-90CC-551A4E4A7F1C BFE98240 InstallProtocolInterface: 05C99A21-C70F-4AD2-8A5F-35DF3343F51E BFE98250 Loading driver 80CF7257-87AB-47F9-A3FE-D50B76D89541 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7325C0 Loading driver at 0x000BFE7F000 EntryPoint=0x000BFE7F2AF PcdDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724018 InstallProtocolInterface: 11B34006-D85B-4D0A-A290-D5A571310EF7 BFE87C60 InstallProtocolInterface: 13A3F0F6-264A-3EF0-F2E0-DEC512342F34 BFE87D60 InstallProtocolInterface: 5BE40F57-FA68-4610-BBBF-E9C5FCDAD365 BFE87DF0 InstallProtocolInterface: FD0F4478-0EFD-461D-BA2D-E58C45FD5F5E BFE87E10 Loading driver 733CBAC2-B23F-4B92-BC8E-FB01CE5907B7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727040 Loading driver at 0x000BFEC3000 EntryPoint=0x000BFEC32AF FvbServicesRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724698 QEMU Flash: Attempting flash detection at FFE00010 QemuFlashDetected => FD behaves as FLASH QemuFlashDetected => Yes Installing QEMU flash FVB InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFEF0DF0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF727D98 Loading driver FEDE0A1B-BCA2-4A9F-BB2B-D9FD7DEC2E9F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727340 Loading driver at 0x000BFEB9000 EntryPoint=0x000BFEB92AF StatusCodeRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF727598 InstallProtocolInterface: D2B2B828-0826-48A7-B3DF-983C006024F0 BFEBF600 Loading driver B601F8C4-43B7-4784-95B1-F4226CB40CEE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72FBC0 Loading driver at 0x000BFEB1000 EntryPoint=0x000BFEB12AF RuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F298 InstallProtocolInterface: B7DFB4E1-052F-449F-87BE-9818FC91B733 BFEB56C0 Loading driver F80697E9-7FD6-4665-8646-88E33EF71DFC InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E040 Loading driver at 0x000BFDF9000 EntryPoint=0x000BFDF92AF SecurityStubDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F698 InstallProtocolInterface: 94AB2F58-1438-4EF1-9152-18941A3A0E68 BFE57B70 InstallProtocolInterface: A46423E3-4617-49F1-B9FF-D1BFA9115839 BFE57B68 Loading driver 13AC6DD0-73D0-11D4-B06B-00AA00BD6DE7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E400 Loading driver at 0x000BFDED000 EntryPoint=0x000BFDED2AF EbcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72EA18 InstallProtocolInterface: 13AC6DD1-73D0-11D4-B06B-00AA00BD6DE7 BF72E798 InstallProtocolInterface: 2755590C-6F3C-42FA-9EA4-A3BA543CDA25 BF72DE58 InstallProtocolInterface: AAEACCFD-F27B-4C17-B610-75CA1F2DFB52 BF72DE18 Loading driver 79CA4208-BBA1-4A9A-8456-E1E66A81484E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D140 Loading driver at 0x000BFDE8000 EntryPoint=0x000BFDE82AF Legacy8259.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72D8D8 InstallProtocolInterface: 38321DBA-4FE0-4E17-8AEC-413055EAEDC1 BFDEB040 Loading driver A19B1FE7-C1BC-49F8-875F-54A5D542443F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D500 Loading driver at 0x000BFDE2000 EntryPoint=0x000BFDE22AF CpuIo2Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72CF98 InstallProtocolInterface: AD61F191-AE5F-4C0E-B9FA-E869D288C64F BFDE53A0 Loading driver 1A1E4886-9517-440E-9FDE-3BE44CEE2136 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72CB00 Loading driver at 0x000BFDCF000 EntryPoint=0x000BFDCF2AF CpuDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72C998 InstallProtocolInterface: 26BACCB1-6F42-11D4-BCE7-0080C73C8881 BFDDBF20 Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Flushing GCD Loading driver 6F0198AA-1F1D-426D-AE3E-39AB633FCC28 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72BE00 Loading driver at 0x000BFEAB000 EntryPoint=0x000BFEAB2AF KbcReset.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728D18 InstallProtocolInterface: 27CFAC88-46CC-11D4-9A38-0090273FC14D 0 Loading driver C8339973-A563-4561-B858-D8476F9DEFC4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7288C0 Loading driver at 0x000BFDC9000 EntryPoint=0x000BFDC92AF Metronome.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728A98 InstallProtocolInterface: 26BACCB2-6F42-11D4-BCE7-0080C73C8881 BFDCBF20 Loading driver 22DC2B60-FE40-42AC-B01F-3AB1FAD9AAD8 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72B9C0 Loading driver at 0x000BFEA3000 EntryPoint=0x000BFEA32AF EmuVariableFvbRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF726F98 EMU Variable FVB Started Disabling EMU Variable FVB since flash variables appear to be supported. Error: Image at 000BFEA3000 start failed: Aborted Loading driver 2226F30F-3D5B-402D-9936-A97184EB4516 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF726040 Loading driver at 0x000BED8A000 EntryPoint=0x000BED8A2AF VariableAuthRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72BA98 Variable driver common space: 0xDF9C 0xDF9C 0xDF9C InstallProtocolInterface: CD3D0A05-9E24-437C-A891-1EE053DB7638 BEDF2F48 InstallProtocolInterface: AF23B340-97B4-4685-8D4F-A3F28169B21D BEDF2F50 InstallProtocolInterface: 1E5668E2-8481-11D4-BCF1-0080C73C8881 0 Loading driver 79E4A61C-ED73-4312-94FE-E3E7563362A9 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF725040 Loading driver at 0x000BFDC4000 EntryPoint=0x000BFDC42AF PrintDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF726B98 InstallProtocolInterface: F05976EF-83F1-4F3D-8619-F7595D41E538 BFDC70E0 Loading driver 348C4D62-BFBD-4882-9ECE-C80BB1C4783B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7252C0 Loading driver at 0x000BFD9D000 EntryPoint=0x000BFD9D2AF HiiDatabase.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF7259D8 InstallProtocolInterface: E9CA4775-8657-47FC-97E7-7ED65A084324 BFDBDF28 InstallProtocolInterface: 0FD96974-23AA-4CDC-B9CB-98D17750322A BFDBDF70 InstallProtocolInterface: EF9FC172-A1B2-4693-B327-6D32FC416042 BFDBDF98 InstallProtocolInterface: 587E72D7-CC50-4F79-8209-CA291FC1A10F BFDBDFF0 InstallProtocolInterface: 31A6406A-6BDF-4E46-B2A2-EBAA89C40920 BFDBDF48 Loading driver 96B5C032-DF4C-4B6E-8232-438DCF448D0E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF31DA40 Loading driver at 0x000BFD97000 EntryPoint=0x000BFD972AF NullMemoryTestDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF31D118 InstallProtocolInterface: 309DE7F1-7F5E-4ACE-B49C-531BE5AA95EF BFD9A2C0 Loading driver F9D88642-0737-49BC-81B5-6889CD57D9EA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF31D240 Loading driver at 0x000BFD8D000 EntryPoint=0x000BFD8D2AF SmbiosDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF31D498 InstallProtocolInterface: 03583FF6-CB36-4940-947E-B9B39F4AFAF7 BFD935B0 Loading driver 9622E42C-8E38-4A08-9E8F-54F784652F6B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3129C0 Loading driver at 0x000BFD7E000 EntryPoint=0x000BFD7E2AF AcpiTableDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF312C18 InstallProtocolInterface: FFE06BDD-6107-46A6-7BB2-5A9C7EC5275C BF312228 Loading driver BDCE85BB-FBAA-4F4E-9264-501A2C249581 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF312380 Loading driver at 0x000BFD73000 EntryPoint=0x000BFD732AF S3SaveStateDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF31C018 InstallProtocolInterface: E857CAF6-C046-45DC-BE3F-EE0765FBA887 BFD7ABE0 Loading driver A210F973-229D-4F4D-AA37-9895E6C9EABA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF31C200 Loading driver at 0x000BFD6D000 EntryPoint=0x000BFD6D2AF DpcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF31C998 InstallProtocolInterface: 480F8AE9-0C46-4AA9-BC89-DB9FBA619806 BFD700C0 Loading driver F2765DEC-6B41-11D5-8E71-00902707B35E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF316040 Loading driver at 0x000BFD68000 EntryPoint=0x000BFD682AF Timer.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF316218 InstallProtocolInterface: 26BACCB3-6F42-11D4-BCE7-0080C73C8881 BFD6B060 Loading driver 2383608E-C6D0-4E3E-858D-45DFAC3543D5 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3162C0 Loading driver at 0x000BFD5D000 EntryPoint=0x000BFD5D2AF PciHostBridge.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF3169D8 InstallProtocolInterface: CF8034BE-6768-4D8B-B739-7CCE683A9FBE BF316548 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BFD64C50 InstallProtocolInterface: 2F707EBB-4A1A-11D4-9A38-0090273FC14D BF31B178 Loading driver FE5CEA76-4F72-49E8-986F-2CD899DFFE5D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF31B380 Loading driver at 0x000BFD50000 EntryPoint=0x000BFD502AF FaultTolerantWriteDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF31BA98 Ftw: FtwWorkSpaceLba - 0xF, WorkBlockSize - 0x1000, FtwWorkSpaceBase - 0x0 Ftw: FtwSpareLba - 0x10, SpareBlockSize - 0x1000 Ftw: NumberOfWorkBlock - 0x1, FtwWorkBlockLba - 0xF Ftw: WorkSpaceLbaInSpare - 0x0, WorkSpaceBaseInSpare - 0x0 Ftw: Remaining work space size - FE0 InstallProtocolInterface: 3EBD9E82-2C78-4DE6-9786-8D4BFCB7C881 BF319028 Variable PK does not exist. Variable SetupMode is 1 Variable SecureBoot is 0 Variable SecureBootEnable is 0 Variable CustomMode is 0 Variable VendorKeys is 1 InstallProtocolInterface: 6441F818-6362-4E44-B570-7DBA31DD2453 0 Loading driver EBF342FE-B1D3-4EF8-957C-8048606FF671 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF318040 Loading driver at 0x000BFD24000 EntryPoint=0x000BFD242AF SetupBrowser.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF31B998 InstallProtocolInterface: B9D4C360-BCFB-4F9B-9298-53C136982258 BFD47670 InstallProtocolInterface: A770C357-B693-4E6D-A6CF-D21C728E550B BFD476A0 InstallProtocolInterface: 1F73B18D-4630-43C1-A1DE-6F80855D7DA4 BFD47680 Loading driver 4110465D-5FF3-4F4B-B580-24ED0D06747A InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF318540 Loading driver at 0x000BECB6000 EntryPoint=0x000BECB62AF SmbiosPlatformDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF318798 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Select Item: 0x19 Select Item: 0x19 Select Item: 0x22 Select Item: 0x21 SmbiosAdd: Smbios type 1 with size 0x51 is added to 32-bit table SmbiosAdd: Smbios type 1 with size 0x51 is added to 64-bit table SmbiosCreateTable: Initialize 32-bit entry point structure SmbiosCreateTable() re-allocate SMBIOS 32-bit table SmbiosCreateTable: Initialize 64-bit entry point structure SmbiosCreate64BitTable() re-allocate SMBIOS 64-bit table SmbiosAdd: Smbios type 3 with size 0x29 is added to 32-bit table SmbiosAdd: Smbios type 3 with size 0x29 is added to 64-bit table SmbiosAdd: Smbios type 4 with size 0x44 is added to 32-bit table SmbiosAdd: Smbios type 4 with size 0x44 is added to 64-bit table SmbiosAdd: Smbios type 16 with size 0x19 is added to 32-bit table SmbiosAdd: Smbios type 16 with size 0x19 is added to 64-bit table SmbiosAdd: Smbios type 17 with size 0x35 is added to 32-bit table SmbiosAdd: Smbios type 17 with size 0x35 is added to 64-bit table SmbiosAdd: Smbios type 19 with size 0x21 is added to 32-bit table SmbiosAdd: Smbios type 19 with size 0x21 is added to 64-bit table SmbiosAdd: Smbios type 19 with size 0x21 is added to 32-bit table SmbiosAdd: Smbios type 19 with size 0x21 is added to 64-bit table SmbiosAdd: Smbios type 32 with size 0xD is added to 32-bit table SmbiosAdd: Smbios type 32 with size 0xD is added to 64-bit table SmbiosAdd: Smbios type 0 with size 0x48 is added to 32-bit table SmbiosAdd: Smbios type 0 with size 0x48 is added to 64-bit table Loading driver 49970331-E3FA-4637-9ABC-3B7868676970 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3170C0 Loading driver at 0x000BECA8000 EntryPoint=0x000BECA82AF AcpiPlatform.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF315E58 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. AcpiPlatformEntryPoint: PCI enumeration pending, registered callback Loading driver 378D7B65-8DA9-4773-B6E4-A47826A833E1 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF315140 Loading driver at 0x000BFEA1000 EntryPoint=0x000BFEA12AF PcRtc.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF315858 InstallProtocolInterface: 27CFAC87-46CC-11D4-9A38-0090273FC14D 0 Loading driver F0E6A44F-7195-41C3-AC64-54F202CD0A21 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3140C0 Loading driver at 0x000BEC87000 EntryPoint=0x000BEC872AF SecureBootConfigDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF314ED8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC9D610 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BF314CE0 InstallProtocolInterface: F0E6A44F-7195-41C3-AC64-54F202CD0A21 BF314CD8 Loading driver F099D67F-71AE-4C36-B2A3-DCEB0EB2B7D8 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF314480 Loading driver at 0x000BFD1F000 EntryPoint=0x000BFD1F2AF WatchdogTimer.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF311358 InstallProtocolInterface: 665E3FF5-46CC-11D4-9A38-0090273FC14D BFD21F40 Loading driver AD608272-D07F-4964-801E-7BD3B7888652 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3115C0 Loading driver at 0x000BEC81000 EntryPoint=0x000BEC812AF MonotonicCounterRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF310018 InstallProtocolInterface: 1DA97072-BDDC-4B30-99F1-72A0B56FFF2A 0 Loading driver 42857F0A-13F2-4B21-8A23-53D3F714B840 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF310200 Loading driver at 0x000BEC77000 EntryPoint=0x000BEC772AF CapsuleRuntimeDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF310918 InstallProtocolInterface: 5053697E-2CBC-4819-90D9-0580DEEE5754 0 Loading driver FC5C7020-1A48-4198-9BE2-EAD5ABC8CF2F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2BD040 Loading driver at 0x000BEC24000 EntryPoint=0x000BEC242AF BdsDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2BDE58 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. InstallProtocolInterface: 665E3FF6-46CC-11D4-9A38-0090273FC14D BEC64D38 Loading driver E660EA85-058E-4B55-A54B-F02F83A24707 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF30EB40 Loading driver at 0x000BEC03000 EntryPoint=0x000BEC032AF DisplayEngine.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2BD298 InstallProtocolInterface: 9BBE29E9-FDA1-41EC-AD52-452213742D2E BEC1D2D0 Loading driver 6B79BBC0-26B9-4FE9-B631-551D8AB078C6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF30E440 Loading driver at 0x000BEBFB000 EntryPoint=0x000BEBFB2AF AcpiS3SaveDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF30B018 Select Item: 0x0 FW CFG Signature: 0x554D4551 Select Item: 0x1 FW CFG Revision: 0x1 QemuFwCfg interface is supported. Select Item: 0x19 Select Item: 0x25 InstallProtocolInterface: 125F2DE1-FB85-440C-A54C-4D99358A8D38 BEC00220 InstallProtocolInterface: BD445D79-B7AD-4F04-9AD8-29BD2040EB3C 0 Loading driver D9DCC5DF-4007-435E-9098-8970935504B2 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF30B200 Loading driver at 0x000BEBF0000 EntryPoint=0x000BEBF02AF PlatformDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF30B998 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEBF69F0 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEBF6A10 Loading driver 93B80004-9FB3-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF30A0C0 Loading driver at 0x000BEBD1000 EntryPoint=0x000BEBD12AF PciBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF30A318 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBEB040 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBEB300 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBEB320 InstallProtocolInterface: 19CB87AB-2CB9-4665-8360-DDCF6054F79D BEBEB098 Loading driver 33CB97AF-6C33-4C42-986B-07581FA366D4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF307200 Loading driver at 0x000BEBCA000 EntryPoint=0x000BEBCA2AF BlockMmioToBlockIoDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF307998 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBCDF80 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBCE280 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBCE2A0 Loading driver 83DD3B39-7CAF-4FAC-A542-E050B767E3A7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FF040 Loading driver at 0x000BEBC3000 EntryPoint=0x000BEBC32AF VirtioPciDeviceDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FF218 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBC7640 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBC76A0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBC76C0 Loading driver 11D92DFB-3CA9-4F93-BA2E-4780ED3E03B5 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FF2C0 Loading driver at 0x000BEBBB000 EntryPoint=0x000BEBBB2AF VirtioBlkDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FFA58 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBBFF40 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBBFFA0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBBFFC0 Loading driver FAB5D4F4-83C0-4AAF-8480-442D11DF6CEA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF306040 Loading driver at 0x000BEBB3000 EntryPoint=0x000BEBB32AF VirtioScsiDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF306218 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBB83C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBB8420 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBB8440 Loading driver CF569F50-DE44-4F54-B4D7-F4AE25CDA599 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3062C0 Loading driver at 0x000BEBAD000 EntryPoint=0x000BEBAD2AF XenIoPciDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF306A58 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBB0960 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBB09C0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBB09E0 Loading driver 565EC8BA-A484-11E3-802B-B8AC6F7D65E6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF305040 Loading driver at 0x000BEB9C000 EntryPoint=0x000BEB9C2AF XenBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF305218 Error: Image at 000BEB9C000 start failed: Aborted Loading driver 8C2487EA-9AF3-11E3-B966-B8AC6F7D65E6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF305040 Loading driver at 0x000BEBA2000 EntryPoint=0x000BEBA22AF XenPvBlkDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF305ED8 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEBA92C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEBA9450 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEBA9470 Loading driver 51CCF399-4FDF-4E55-A45B-E123F84D456A InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3052C0 Loading driver at 0x000BEB9A000 EntryPoint=0x000BEB9A2AF ConPlatformDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF305518 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB9EBA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB9ED80 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB9EDA0 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB9EBE0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB9ED80 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB9EDA0 Loading driver 408EDCEC-CF6D-477C-A5A8-B4844E3DE281 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3049C0 Loading driver at 0x000BEB8A000 EntryPoint=0x000BEB8A2AF ConSplitterDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF304C18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB95A80 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB96030 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB96050 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB95B40 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB96070 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB96090 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB95B80 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB960B0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB960D0 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB95AC0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB960F0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB96110 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB95B00 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB96130 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB96150 InstallProtocolInterface: 387477C1-69C7-11D2-8E39-00A0C969723B BEB95690 InstallProtocolInterface: DD9E7534-7762-4698-8C14-F58517A625AA BEB956C0 InstallProtocolInterface: 31878C87-0B75-11D5-9A4F-0090273FC14D BEB95718 InstallProtocolInterface: 8D59D32B-C655-4AE9-9B15-F25904992A43 BEB95770 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B BEB95850 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B BEB95970 Loading driver CCCB0C28-4B24-11D5-9A5A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF301880 Loading driver at 0x000BEB7D000 EntryPoint=0x000BEB7D2AF GraphicsConsoleDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF304118 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB84880 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB84B40 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB84B60 Loading driver 9E863906-A40F-4875-977F-5B93FF237FC6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF3007C0 Loading driver at 0x000BEB6D000 EntryPoint=0x000BEB6D2AF TerminalDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF300658 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB784C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB78C20 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB78C40 Loading driver 6B38F7B4-AD98-40E9-9093-ACA2B5A253C4 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FE0C0 Loading driver at 0x000BEB63000 EntryPoint=0x000BEB632AF DiskIoDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FEE58 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB69AA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB6A080 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB6A0A0 Loading driver 1FA1F39E-FEFF-4AAE-BD7B-38A070A3B609 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FE880 Loading driver at 0x000BEB57000 EntryPoint=0x000BEB572AF PartitionDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FE318 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB5F880 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB5FA30 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB5FA50 Loading driver CD3BAFB6-50FB-4FE8-8E4E-AB74D2C1A600 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FD040 Loading driver at 0x000BEB51000 EntryPoint=0x000BEB512AF EnglishDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FDE98 InstallProtocolInterface: 1D85CD7F-F43D-11D2-9A0C-0090273FC14D BEB54220 InstallProtocolInterface: A4C751FC-23AE-4C3E-92E9-4964CF63F349 BEB54260 Loading driver 0167CCC4-D0F7-4F21-A3EF-9E64B7CDCE8B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FD380 Loading driver at 0x000BEB47000 EntryPoint=0x000BEB472AF ScsiBus.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FDBD8 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB4D560 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB4D690 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB4D6B0 Loading driver 0A66E322-3740-4CCE-AD62-BD172CECCA35 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FC0C0 Loading driver at 0x000BEB39000 EntryPoint=0x000BEB392AF ScsiDisk.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FCE58 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB43300 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB438D0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB438F0 Loading driver 69FD8E47-A161-4550-B01A-5594CEB2B2B2 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FC880 Loading driver at 0x000BEB23000 EntryPoint=0x000BEB232AF IdeBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FCC18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB347E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB34C10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB34C30 InstallProtocolInterface: 0784924F-E296-11D4-9A49-0090273FC14D BEB34AE0 InstallProtocolInterface: 4D330321-025F-4AAC-90D8-5ED900173B63 BEB34AF0 Loading driver 99549F44-49BB-4820-B9D2-901329412D67 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FB0C0 Loading driver at 0x000BEB1C000 EntryPoint=0x000BEB1C2AF IdeController.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FBC58 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB1FA20 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB1FC00 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB1FC20 Loading driver 38A0EC22-FBE7-4911-8BC1-176E0D6C1DBD InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2FB400 Loading driver at 0x000BEB15000 EntryPoint=0x000BEB152AF IsaAcpi.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FB318 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB18D20 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB18FF0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB18FD0 Loading driver 240612B5-A063-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F80C0 Loading driver at 0x000BEB0C000 EntryPoint=0x000BEB0C2AF IsaBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2FB658 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB12360 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB12480 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB124A0 Loading driver 93B80003-9FB3-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F8340 Loading driver at 0x000BEB01000 EntryPoint=0x000BEB012AF IsaSerialDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F8518 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEB08EA0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEB091A0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEB091C0 Loading driver 3DC82376-637B-40A6-A8FC-A565417F2C38 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F70C0 Loading driver at 0x000BEAF4000 EntryPoint=0x000BEAF42AF Ps2KeyboardDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F7B18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEAFC900 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEAFCB00 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEAFCB20 Loading driver 0ABD8284-6DA3-4616-971A-83A5148067BA InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F7880 Loading driver at 0x000BEAE8000 EntryPoint=0x000BEAE82AF IsaFloppyDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F7458 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEAEFD20 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEAEFF10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEAEFF30 Loading driver FA20568B-548B-4B2B-81EF-1BA08D4A3CEC InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F60C0 Loading driver at 0x000BEACE000 EntryPoint=0x000BEACE2AF BootScriptExecutorDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F6B18 Loading driver 961578FE-B6B7-44C3-AF35-6BC705CD2B1F InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F6940 Loading driver at 0x000BEAC7000 EntryPoint=0x000BEAC756C InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F6858 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEAC7360 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEAC7438 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEAC7450 Loading driver A2F436EA-A127-4EF8-957C-8048606FF670 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2E6040 Loading driver at 0x000BEAB8000 EntryPoint=0x000BEAB82AF SnpDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2E6BD8 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEAC2F00 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEAC37D0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEAC37F0 Loading driver 025BBFC7-E6A9-4B8B-82AD-6815A1AEAF4A InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2E6940 Loading driver at 0x000BEAA3000 EntryPoint=0x000BEAA32AF MnpDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2E6D98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEAB2460 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEAB3950 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEAB3970 Loading driver E4F61863-FE2C-4B56-A8F4-08519BC439DF InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2E7040 Loading driver at 0x000BEA95000 EntryPoint=0x000BEA952AF VlanConfigDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2E7BD8 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA9E5A0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA9EBC0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA9EBE0 Loading driver 529D3F93-E8E9-4E73-B1E1-BDF6A9D50113 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2E7940 Loading driver at 0x000BEA87000 EntryPoint=0x000BEA872AF ArpDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2E7D98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA90520 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA90860 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA90880 Loading driver 94734718-0BBC-47FB-96A5-EE7A5AE6A2AD InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F5040 Loading driver at 0x000BEA72000 EntryPoint=0x000BEA722AF Dhcp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F5BD8 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA818C0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA82060 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA82080 Loading driver 26841BDE-920A-4E7A-9FBE-637F477143A6 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F5940 Loading driver at 0x000BEA61000 EntryPoint=0x000BEA612AF Ip4ConfigDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F5D98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA6C8E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA6CFB0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA6CFD0 Loading driver 9FB1A1F3-3B71-4324-B39A-745CBB015FFF InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F4040 Loading driver at 0x000BEA47000 EntryPoint=0x000BEA472AF Ip4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F4BD8 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA5B720 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA5C000 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA5C020 Loading driver DC3641B8-2FA8-4ED3-BC1F-F9962A03454B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F4940 Loading driver at 0x000BEA34000 EntryPoint=0x000BEA342AF Mtftp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F4D98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA42240 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA423D0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA423F0 Loading driver 6D6963AB-906D-4A65-A7CA-BD40E5D6AF2B InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F3040 Loading driver at 0x000BEA21000 EntryPoint=0x000BEA212AF Udp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F3F18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA2F220 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA2F3A0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA2F3C0 Loading driver 6D6963AB-906D-4A65-A7CA-BD40E5D6AF4D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F3A00 Loading driver at 0x000BE9FF000 EntryPoint=0x000BE9FF2AF Tcp4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F3898 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BEA186A0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BEA19C10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BEA19C30 Loading driver 3B1DEAB5-C75D-442E-9238-8E2FFB62B0BB InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F2040 Loading driver at 0x000BE9EC000 EntryPoint=0x000BE9EC2AF UefiPxe4BcDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F2F18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE9FABE0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE9FAD10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE9FAD30 Loading driver 4579B72D-7EC4-4DD4-8486-083C86B182A7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F2A00 Loading driver at 0x000BE9CB000 EntryPoint=0x000BE9CB2AF IScsi4Dxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F2898 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE9E5200 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE9E53E0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE9E5400 InstallProtocolInterface: 59324945-EC44-4C0D-B1CD-9DB139DF070C BE9E60F0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BE9E54C0 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BF2F1E68 Loading driver A92CDB4B-82F1-4E0B-A516-8A655D371524 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2EF380 Loading driver at 0x000BE9C1000 EntryPoint=0x000BE9C12AF VirtioNetDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F1A98 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE9C7A80 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE9C7940 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE9C7960 Loading driver 2FB92EFA-2EE0-4BAE-9EB6-7464125E1EF7 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2EF780 Loading driver at 0x000BE9B2000 EntryPoint=0x000BE9B22AF UhciDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F0D18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE9BD140 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE9BD3F0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE9BD410 Loading driver BDFE430E-8F2A-4DB0-9991-6F856594777E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2F0240 Loading driver at 0x000BE9A0000 EntryPoint=0x000BE9A02AF EhciDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2F0818 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE9AD600 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE9AEC10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE9AEC30 Loading driver B7F50E91-A759-412C-ADE4-DCD03E7F7C28 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2EE040 Loading driver at 0x000BE986000 EntryPoint=0x000BE9862AF XhciDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2EEF18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE99A0E0 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE99C110 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE99C130 Loading driver 240612B7-A063-11D4-9A3A-0090273FC14D InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2EE440 Loading driver at 0x000BE973000 EntryPoint=0x000BE9732AF UsbBusDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2EEA18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE980A60 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE9814B0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE9814D0 Loading driver 2D2E62CF-9ECF-43B7-8219-94E7FC713DFE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2ED040 Loading driver at 0x000BE965000 EntryPoint=0x000BE9652AF UsbKbDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2EDF18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE96E580 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE96F3A0 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE96F3C0 Loading driver 9FB4B4A7-42C0-4BCD-8540-9BCC6711F83E InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2ED440 Loading driver at 0x000BE959000 EntryPoint=0x000BE9592AF UsbMassStorageDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2EDA18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE961180 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE961E70 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE961E90 Loading driver E3752948-B9A1-4770-90C4-DF41C38986BE InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2EC040 Loading driver at 0x000BE948000 EntryPoint=0x000BE9482AF QemuVideoDxe.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2ECF18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE950720 InstallProtocolInterface: 107A772C-D5E1-11D4-9A46-0090273FC14D BE951520 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE951540 InstallProtocolInterface: 5C198761-16A8-4E69-972C-89D67954F81D BE950A00 [BdsDxe] Locate Variable Lock protocol - Success [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:PlatformLangCodes [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:LangCodes [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:BootOptionSupport [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:HwErrRecSupport [Variable] Lock: 8BE4DF61-93CA-11D2-AA0D-00E098032B8C:OsIndicationsSupported Variable Driver Auto Update Lang, Lang:eng, PlatformLang:en Status: Success InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC66BB0 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEC66B68 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC66BD0 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEC66B80 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC67480 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEC674B8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC66800 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEC667E0 PlatformBdsInit Registered NotifyDevPath Event PlatformBdsPolicyBehavior PCI Bus First Scanning PciBus: Discovered PCI @ [00|00|00] PciBus: Discovered PCI @ [00|01|00] PciBus: Discovered PCI @ [00|01|01] BAR[4]: Type = Io32; Alignment = 0xF; Length = 0x10; Offset = 0x20 PciBus: Discovered PCI @ [00|01|03] PciBus: Discovered PCI @ [00|02|00] BAR[0]: Type = PMem32; Alignment = 0xFFFFFF; Length = 0x1000000; Offset = 0x10 BAR[2]: Type = Mem32; Alignment = 0xFFF; Length = 0x1000; Offset = 0x18 PciBus: Discovered PCI @ [00|03|00] BAR[0]: Type = Mem32; Alignment = 0x1FFFF; Length = 0x20000; Offset = 0x10 BAR[1]: Type = Io32; Alignment = 0x3F; Length = 0x40; Offset = 0x14 PciBus: Discovered PCI @ [00|04|00] BAR[0]: Type = PMem64; Alignment = 0xFFFFFFF; Length = 0x10000000; Offset = 0x10 BAR[1]: Type = Mem64; Alignment = 0x3FFFF; Length = 0x40000; Offset = 0x18 BAR[2]: Type = Io32; Alignment = 0xFF; Length = 0x100; Offset = 0x20 PciBus: Discovered PCI @ [00|05|00] BAR[0]: Type = Mem64; Alignment = 0x3FFF; Length = 0x4000; Offset = 0x10 PciBus: Discovered PCI @ [00|06|00] BAR[0]: Type = Io32; Alignment = 0x3F; Length = 0x40; Offset = 0x10 BAR[1]: Type = Mem32; Alignment = 0xFFF; Length = 0x1000; Offset = 0x14 PciBus: Discovered PCI @ [00|00|00] PciBus: Discovered PCI @ [00|01|00] PciBus: Discovered PCI @ [00|01|01] BAR[4]: Type = Io32; Alignment = 0xF; Length = 0x10; Offset = 0x20 PciBus: Discovered PCI @ [00|01|03] PciBus: Discovered PCI @ [00|02|00] BAR[0]: Type = PMem32; Alignment = 0xFFFFFF; Length = 0x1000000; Offset = 0x10 BAR[2]: Type = Mem32; Alignment = 0xFFF; Length = 0x1000; Offset = 0x18 PciBus: Discovered PCI @ [00|03|00] BAR[0]: Type = Mem32; Alignment = 0x1FFFF; Length = 0x20000; Offset = 0x10 BAR[1]: Type = Io32; Alignment = 0x3F; Length = 0x40; Offset = 0x14 PciBus: Discovered PCI @ [00|04|00] BAR[0]: Type = PMem64; Alignment = 0xFFFFFFF; Length = 0x10000000; Offset = 0x10 BAR[1]: Type = Mem64; Alignment = 0x3FFFF; Length = 0x40000; Offset = 0x18 BAR[2]: Type = Io32; Alignment = 0xFF; Length = 0x100; Offset = 0x20 PciBus: Discovered PCI @ [00|05|00] BAR[0]: Type = Mem64; Alignment = 0x3FFF; Length = 0x4000; Offset = 0x10 PciBus: Discovered PCI @ [00|06|00] BAR[0]: Type = Io32; Alignment = 0x3F; Length = 0x40; Offset = 0x10 BAR[1]: Type = Mem32; Alignment = 0xFFF; Length = 0x1000; Offset = 0x14 PciBus: HostBridge->SubmitResources() - Success PciBus: HostBridge->NotifyPhase(AllocateResources) - Success PciBus: Resource Map for Root Bridge PciRoot(0x0) Type = Io16; Base = 0xC000; Length = 0x1000; Alignment = 0xFFF Base = 0xC000; Length = 0x100; Alignment = 0xFF; Owner = PCI [00|04|00:20] Base = 0xC100; Length = 0x40; Alignment = 0x3F; Owner = PCI [00|06|00:10] Base = 0xC140; Length = 0x40; Alignment = 0x3F; Owner = PCI [00|03|00:14] Base = 0xC180; Length = 0x10; Alignment = 0xF; Owner = PCI [00|01|01:20] Type = Mem32; Base = 0xC0000000; Length = 0x11100000; Alignment = 0xFFFFFFF Base = 0xC0000000; Length = 0x10000000; Alignment = 0xFFFFFFF; Owner = PCI [00|04|00:10] Base = 0xD0000000; Length = 0x1000000; Alignment = 0xFFFFFF; Owner = PCI [00|02|00:10] Base = 0xD1000000; Length = 0x40000; Alignment = 0x3FFFF; Owner = PCI [00|04|00:18] Base = 0xD1040000; Length = 0x20000; Alignment = 0x1FFFF; Owner = PCI [00|03|00:10] Base = 0xD1060000; Length = 0x4000; Alignment = 0x3FFF; Owner = PCI [00|05|00:10] Base = 0xD1064000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [00|06|00:14] Base = 0xD1065000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [00|02|00:18] InstallProtocolInterface: 30CFE3E7-3DE1-4586-BE20-DEABA1B3B793 0 OnPciEnumerated: PCI enumeration complete, installing ACPI tables Select Item: 0x19 Select Item: 0x27 Select Item: 0x19 Select Item: 0x29 Select Item: 0x19 Select Item: 0x26 InstallQemuFwCfgTables: installed 6 tables InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2E1FD8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2DE028 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2E88D8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2DE568 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2DEFD8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D7028 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2E8E18 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D7568 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2D7FD8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D6028 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2EB558 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D6568 InstallProtocolInterface: 4006C0C1-FCB3-403E-996D-4A6C8724E06D BF2D6630 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2E8A40 Loading driver at 0x000BE8AA000 EntryPoint=0x000BE8AB7D9 8086100e.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2E1F18 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE8C4C10 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE8C4BE0 InstallProtocolInterface: 3BC1B285-8A15-4A82-AABF-4D7D13FB3265 BF2D6618 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2D6FD8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D5028 InstallProtocolInterface: 4006C0C1-FCB3-403E-996D-4A6C8724E06D BF2D50F0 InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF2C91C0 Loading driver at 0x000BE891000 EntryPoint=0x000BE8912C8 AmdGop.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF2C9618 InstallProtocolInterface: 18A031AB-B443-4D1A-A5C0-0C09261E9F71 BE8A87A8 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE8A86D0 InstallProtocolInterface: 5C198761-16A8-4E69-972C-89D67954F81D BE8A87D8 InstallProtocolInterface: 3BC1B285-8A15-4A82-AABF-4D7D13FB3265 BF2D50D8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2D5FD8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D5568 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2E85D8 InstallProtocolInterface: 4CF5B200-68B8-4CA5-9EEC-B23E3F50029A BF2D4028 PlatformBdsPolicyBehavior: not restoring NvVars from disk since flash variables appear to be supported. PlatformBdsGetDriverOption Boot Mode:0 Found LPC Bridge device BdsPlatform.c+232: COM1 DevPath: PciRoot(0x0)/Pci(0x1,0x0)/Serial(0x0)/Uart(115200,8,N,1)/VenPcAnsi() BdsPlatform.c+261: COM2 DevPath: PciRoot(0x0)/Pci(0x1,0x0)/Serial(0x1)/Uart(115200,8,N,1)/VenPcAnsi() Found PCI VGA device QemuVideo: QEMU Standard VGA detected QemuVideo: Using mmio bar @ 0xD1065000 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2C7018 Adding Mode 0 as Bochs Internal Mode 0: 640x480, 32-bit, 60 Hz Adding Mode 1 as Bochs Internal Mode 1: 800x480, 32-bit, 60 Hz Adding Mode 2 as Bochs Internal Mode 2: 800x600, 32-bit, 60 Hz Adding Mode 3 as Bochs Internal Mode 3: 832x624, 32-bit, 60 Hz Adding Mode 4 as Bochs Internal Mode 4: 960x640, 32-bit, 60 Hz Adding Mode 5 as Bochs Internal Mode 5: 1024x600, 32-bit, 60 Hz Adding Mode 6 as Bochs Internal Mode 6: 1024x768, 32-bit, 60 Hz Adding Mode 7 as Bochs Internal Mode 7: 1152x864, 32-bit, 60 Hz Adding Mode 8 as Bochs Internal Mode 8: 1152x870, 32-bit, 60 Hz Adding Mode 9 as Bochs Internal Mode 9: 1280x720, 32-bit, 60 Hz Adding Mode 10 as Bochs Internal Mode 10: 1280x760, 32-bit, 60 Hz Adding Mode 11 as Bochs Internal Mode 11: 1280x768, 32-bit, 60 Hz Adding Mode 12 as Bochs Internal Mode 12: 1280x800, 32-bit, 60 Hz Adding Mode 13 as Bochs Internal Mode 13: 1280x960, 32-bit, 60 Hz Adding Mode 14 as Bochs Internal Mode 14: 1280x1024, 32-bit, 60 Hz Adding Mode 15 as Bochs Internal Mode 15: 1360x768, 32-bit, 60 Hz Adding Mode 16 as Bochs Internal Mode 16: 1366x768, 32-bit, 60 Hz Adding Mode 17 as Bochs Internal Mode 17: 1400x1050, 32-bit, 60 Hz Adding Mode 18 as Bochs Internal Mode 18: 1440x900, 32-bit, 60 Hz Adding Mode 19 as Bochs Internal Mode 19: 1600x900, 32-bit, 60 Hz Adding Mode 20 as Bochs Internal Mode 20: 1600x1200, 32-bit, 60 Hz Adding Mode 21 as Bochs Internal Mode 21: 1680x1050, 32-bit, 60 Hz Adding Mode 22 as Bochs Internal Mode 22: 1920x1080, 32-bit, 60 Hz Adding Mode 23 as Bochs Internal Mode 23: 1920x1200, 32-bit, 60 Hz Adding Mode 24 as Bochs Internal Mode 24: 1920x1440, 32-bit, 60 Hz Adding Mode 25 as Bochs Internal Mode 25: 2000x2000, 32-bit, 60 Hz Adding Mode 26 as Bochs Internal Mode 26: 2048x1536, 32-bit, 60 Hz Adding Mode 27 as Bochs Internal Mode 27: 2048x2048, 32-bit, 60 Hz Adding Mode 28 as Bochs Internal Mode 28: 2560x1440, 32-bit, 60 Hz Adding Mode 29 as Bochs Internal Mode 29: 2560x1600, 32-bit, 60 Hz InitializeBochsGraphicsMode: 640x480 @ 32 PixelBlueGreenRedReserved8BitPerColor FrameBufferBase: 0xD0000000, FrameBufferSize: 0x12C000 0: shl:0 shr:0 mask:FF0000 1: shl:0 shr:0 mask:FF00 2: shl:0 shr:0 mask:FF Bytes per pixel: 4 InstallProtocolInterface: 9042A9DE-23DC-4A38-96FB-7ADED080516A BF2C70B8 InstallVbeShim: VBE shim installed PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor Found PCI VGA device InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2C7358 InstallProtocolInterface: 1C0C34F6-D380-41FA-A049-8AD06C1A66AA BF2C32F0 InstallProtocolInterface: BD8C1056-9F36-44EC-92A8-A6337F817986 BF2C3300 InstallProtocolInterface: 9042A9DE-23DC-4A38-96FB-7ADED080516A BF2C3040 InstallProtocolInterface: 64A892DC-5561-4536-92C7-799BFC183355 BF2C57E8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2C5458 InstallProtocolInterface: 7EE2BD44-3DA0-11D4-9A38-0090273FC14D BF30A568 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2BF018 InstallProtocolInterface: 7EE2BD44-3DA0-11D4-9A38-0090273FC14D BF30A7A8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2BF998 InstallProtocolInterface: 7EE2BD44-3DA0-11D4-9A38-0090273FC14D BF2BFD68 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2BF6D8 InstallProtocolInterface: 7EE2BD44-3DA0-11D4-9A38-0090273FC14D BF2BF8E8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2BF4D8 InstallProtocolInterface: 7EE2BD44-3DA0-11D4-9A38-0090273FC14D BF2BF228 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2BEF18 InstallProtocolInterface: 7EE2BD44-3DA0-11D4-9A38-0090273FC14D BF2BEE68 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF2BE998 InstallProtocolInterface: BB25CF6F-F1D4-11D2-9A0C-0090273FC1FD BF2C5568 InstallProtocolInterface: 9E863906-A40F-4875-977F-5B93FF237FC6 BF26BD18 Terminal - Mode 0, Column = 80, Row = 25 Terminal - Mode 1, Column = 80, Row = 50 Terminal - Mode 2, Column = 100, Row = 25 Terminal - Mode 3, Column = 100, Row = 31 Terminal - Mode 4, Column = 104, Row = 32 Terminal - Mode 5, Column = 120, Row = 33 Terminal - Mode 6, Column = 128, Row = 31 Terminal - Mode 7, Column = 128, Row = 40 Terminal - Mode 8, Column = 144, Row = 45 Terminal - Mode 9, Column = 160, Row = 37 Terminal - Mode 10, Column = 160, Row = 40 Terminal - Mode 11, Column = 160, Row = 42 Terminal - Mode 12, Column = 160, Row = 50 Terminal - Mode 13, Column = 160, Row = 53 Terminal - Mode 14, Column = 170, Row = 40 Terminal - Mode 15, Column = 175, Row = 55 Terminal - Mode 16, Column = 180, Row = 47 Terminal - Mode 17, Column = 200, Row = 47 Terminal - Mode 18, Column = 200, Row = 63 Terminal - Mode 19, Column = 210, Row = 55 Terminal - Mode 20, Column = 240, Row = 56 Terminal - Mode 21, Column = 240, Row = 63 Terminal - Mode 22, Column = 240, Row = 75 Terminal - Mode 23, Column = 250, Row = 105 Terminal - Mode 24, Column = 256, Row = 80 Terminal - Mode 25, Column = 256, Row = 107 Terminal - Mode 26, Column = 320, Row = 75 Terminal - Mode 27, Column = 320, Row = 84 Terminal - Mode 28, Column = 320, Row = 107 Terminal - Mode 29, Column = 350, Row = 110 Terminal - Mode 30, Column = 400, Row = 126 Terminal - Mode 31, Column = 480, Row = 113 Terminal - Mode 32, Column = 512, Row = 113 Terminal - Mode 33, Column = 960, Row = 227 Terminal - Mode 34, Column = 1024, Row = 227 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF26B098 InstallProtocolInterface: 387477C1-69C7-11D2-8E39-00A0C969723B BF26B400 InstallProtocolInterface: DD9E7534-7762-4698-8C14-F58517A625AA BF26B4D0 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B BF26B418 InstallProtocolInterface: D3B36F2B-D551-11D4-9A46-0090273FC14D 0 InstallProtocolInterface: D3B36F2C-D551-11D4-9A46-0090273FC14D 0 InstallProtocolInterface: D3B36F2D-D551-11D4-9A46-0090273FC14D 0 PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor InitializeBochsGraphicsMode: 800x600 @ 32 PixelBlueGreenRedReserved8BitPerColor FrameBufferBase: 0xD0000000, FrameBufferSize: 0x1D4C00 0: shl:0 shr:0 mask:FF0000 1: shl:0 shr:0 mask:FF00 2: shl:0 shr:0 mask:FF Bytes per pixel: 4 GraphicsConsole video resolution 800 x 600 Graphics - Mode 0, Column = 80, Row = 25 Graphics - Mode 1, Column = 0, Row = 0 Graphics - Mode 2, Column = 100, Row = 31 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B BF3013F0 InstallProtocolInterface: D3B36F2C-D551-11D4-9A46-0090273FC14D 0 PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor PixelBlueGreenRedReserved8BitPerColor InstallProtocolInterface: 9042A9DE-23DC-4A38-96FB-7ADED080516A BEB958E0 GraphicsConsole video resolution 800 x 600 Graphics - Mode 0, Column = 80, Row = 25 Graphics - Mode 1, Column = 0, Row = 0 Graphics - Mode 2, Column = 100, Row = 31 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B BF255E70 InstallProtocolInterface: D3B36F2C-D551-11D4-9A46-0090273FC14D 0 InstallProtocolInterface: 387477C1-69C7-11D2-8E39-00A0C969723B BF247028 InstallProtocolInterface: DD9E7534-7762-4698-8C14-F58517A625AA BF247040 InstallProtocolInterface: D3B36F2B-D551-11D4-9A46-0090273FC14D 0 PlatformBdsDiagnostics PlatformBdsConnectSequence InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B BF242528 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF240020 InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B BF240AE8 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF240660 InstallProtocolInterface: A1E37052-80D9-4E65-A317-3E9A55C43EC9 BEB1FA60 InstallProtocolInterface: 69FD8E47-A161-4550-B01A-5594CEB2B2B2 BF2402D8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF22FC18 InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B BF22F2A8 InstallProtocolInterface: D432A67F-14DC-484B-B3BB-3F0291849327 BF22F308 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF22F120 BlockSize : 2048 LastBlock : 0 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF22F120 BlockSize : 2048 LastBlock : 1EEEDE InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF22E398 InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B BF20CDF0 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF20DCA0 BlockSize : 2048 LastBlock : 7 BlockSize : 2048 LastBlock : 7 InstallProtocolInterface: A19832B9-AC25-11D3-9A2D-0090273FC14D BE8C7370 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BE8C77E8 InstallProtocolInterface: E18541CD-F755-4F73-928D-643C8A79B229 BE8C7698 InstallProtocolInterface: 1ACED566-76ED-4218-BC81-767F1F977A89 BE8C7698 InstallProtocolInterface: 6A7A5CFF-E8D9-4F70-BADA-75AB3025CE14 BE8C76C0 InstallProtocolInterface: 56EC3091-954C-11D2-8E3F-00A0C969723B BE8C76D8 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BE8C76E0 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 9E23D768-D2F3-4366-9FC3-3A7ABA864374 BF20CA70 InstallProtocolInterface: F36FF770-A7E1-42CF-9ED2-56F0F271F44C BEFEAF40 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEFEAB98 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEFEA9A0 InstallProtocolInterface: E4F61863-FE2C-4B56-A8F4-08519BC439DF BEFEA998 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 7AB33A91-ACE5-4326-B572-E7EE33D39F16 BEFEC940 InstallProtocolInterface: F44C00EE-1F2C-4A00-AA09-1C9F3E0800A3 BEFEF8A0 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 7AB33A91-ACE5-4326-B572-E7EE33D39F16 BEFF9940 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEFFA9D8 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEFFA8D0 InstallProtocolInterface: 3B95AA31-3793-434B-8667-C8070892E05E BEFFA8B8 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 7AB33A91-ACE5-4326-B572-E7EE33D39F16 BEFFE8C0 InstallProtocolInterface: C51711E7-B4BF-404A-BFB8-0A048EF1FFE4 BEFE9020 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF0089E0 InstallProtocolInterface: 83F01464-99BD-45E5-B383-AF6305D8E9E6 BF007AA0 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF0168A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF015938 InstallProtocolInterface: 9D9A39D8-BD42-4A73-A4D5-8EE94BE11380 BF013920 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF02C8A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF02A9F8 InstallProtocolInterface: 2FE800BE-8F01-4AA6-946B-D71388E1833F BF029920 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF02E9E0 InstallProtocolInterface: 00720665-67EB-4A99-BAF7-D3C33A1C7CC9 BF02DA30 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: F4B427BB-BA21-4F16-BC4E-43E416AB619C BF039CF0 InstallProtocolInterface: 8A219718-4EF5-4761-91C8-C0F04BDA9E56 BF03AC60 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF03B920 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF03E8A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF03C938 InstallProtocolInterface: 78247C57-63DB-4708-99C2-A8B4A9A61F6B BF03CC70 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF0418A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF0409B8 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF043AA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF0438B8 InstallProtocolInterface: 03C4E603-AC28-11D3-9A2D-0090273FC14D BEFDE068 InstallProtocolInterface: 56EC3091-954C-11D2-8E3F-00A0C969723B BEFDE0D8 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: 143B7632-B81B-4CB7-ABD3-B625A5B9BFFE BEFE8048 InstallProtocolInterface: 65530BC7-A359-410F-B010-5AADC7EC2B62 BEFE8EF8 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF03C920 InstallProtocolInterface: 4579B72D-7EC4-4DD4-8486-083C86B182A7 BEFE8030 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: FA920010-6785-4941-B6EC-498C579F160A BF03CB20 InstallProtocolInterface: 143B7632-B81B-4CB7-ABD3-B625A5B9BFFE BF03EC18 InstallProtocolInterface: 0167CCC4-D0F7-4F21-A3EF-9E64B7CDCE8B BF044BA0 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF044A18 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF040BA8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF044CD8 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF20B368 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF03DBD8 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF03DC68 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF040A18 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF040AA8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF043998 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF043AA8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF046918 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF0469A8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF047898 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF047B28 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF048998 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF048CA8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF049C58 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF049CE8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04A898 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04AB28 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04AA18 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04B8A8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04BB18 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04BBA8 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04CA98 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04CB28 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04DD18 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04DB28 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04EA98 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04ED28 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF04F998 InstallProtocolInterface: 932F47E6-2362-4002-803E-3CD54B138F85 BF04FCA8 InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B BF0518A8 InstallProtocolInterface: D432A67F-14DC-484B-B3BB-3F0291849327 BF051990 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF052A60 BlockSize : 512 LastBlock : 150A7A8E InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B BF0578A8 InstallProtocolInterface: D432A67F-14DC-484B-B3BB-3F0291849327 BF057990 InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B BF058D20 BlockSize : 2048 LastBlock : 856C BlockSize : 2048 LastBlock : 1EEEDE BlockSize : 2048 LastBlock : 7 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: F4B427BB-BA21-4F16-BC4E-43E416AB619C BF05FCB0 InstallProtocolInterface: 8A219718-4EF5-4761-91C8-C0F04BDA9E56 BF05F8A0 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BEFBC920 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF0628A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF060938 InstallProtocolInterface: 78247C57-63DB-4708-99C2-A8B4A9A61F6B BF060C70 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF0658A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF064938 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF0678A0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF0669B8 InstallProtocolInterface: 03C4E603-AC28-11D3-9A2D-0090273FC14D BEFB1068 InstallProtocolInterface: 56EC3091-954C-11D2-8E3F-00A0C969723B BEFB10D8 Support(): UNDI3.1 found on handle BF20B918 BlockSize : 512 LastBlock : 150A7A8E BlockSize : 2048 LastBlock : 856C Select Item: 0x8 Select Item: 0x17 qemu -kernel was not used. BdsLibConnectAll BlockSize : 2048 LastBlock : 1EEEDE BlockSize : 2048 LastBlock : 7 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: F4B427BB-BA21-4F16-BC4E-43E416AB619C BF0619B0 InstallProtocolInterface: 8A219718-4EF5-4761-91C8-C0F04BDA9E56 BF064920 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BEFBD8A0 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF069AA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF0698B8 InstallProtocolInterface: 78247C57-63DB-4708-99C2-A8B4A9A61F6B BEFBDB30 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF06CAA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF06C8B8 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF06EB60 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF06E8B8 InstallProtocolInterface: 03C4E603-AC28-11D3-9A2D-0090273FC14D BEFB1068 InstallProtocolInterface: 56EC3091-954C-11D2-8E3F-00A0C969723B BEFB10D8 Support(): UNDI3.1 found on handle BF20B918 BlockSize : 512 LastBlock : 150A7A8E BlockSize : 2048 LastBlock : 856C BlockSize : 2048 LastBlock : 1EEEDE BlockSize : 2048 LastBlock : 7 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: F4B427BB-BA21-4F16-BC4E-43E416AB619C BF0619B0 InstallProtocolInterface: 8A219718-4EF5-4761-91C8-C0F04BDA9E56 BF064920 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BEFBDDE0 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF069AA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF0698B8 InstallProtocolInterface: 78247C57-63DB-4708-99C2-A8B4A9A61F6B BEFBDB30 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF06CAA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF06C8B8 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF06EB60 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF06E8B8 InstallProtocolInterface: 03C4E603-AC28-11D3-9A2D-0090273FC14D BEFB1068 InstallProtocolInterface: 56EC3091-954C-11D2-8E3F-00A0C969723B BEFB10D8 Support(): UNDI3.1 found on handle BF20B918 BlockSize : 512 LastBlock : 150A7A8E BlockSize : 2048 LastBlock : 856C BlockSize : 2048 LastBlock : 1EEEDE BlockSize : 2048 LastBlock : 7 BlockSize : 2048 LastBlock : 7 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 Support(): UNDI3.1 found on handle BF20B918 InstallProtocolInterface: F4B427BB-BA21-4F16-BC4E-43E416AB619C BF0619B0 InstallProtocolInterface: 8A219718-4EF5-4761-91C8-C0F04BDA9E56 BF064920 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BEFBDDE0 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF069AA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF0698B8 InstallProtocolInterface: 78247C57-63DB-4708-99C2-A8B4A9A61F6B BEFBDB30 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF06CAA0 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF06C8B8 InstallProtocolInterface: 41D94CD2-35B6-455A-8258-D4E51334AADD BF06EB60 InstallProtocolInterface: 3AD9DF29-4501-478D-B1F8-7F7FE70E50F3 BF06E8B8 InstallProtocolInterface: 03C4E603-AC28-11D3-9A2D-0090273FC14D BEFB1068 InstallProtocolInterface: 56EC3091-954C-11D2-8E3F-00A0C969723B BEFB10D8 Support(): UNDI3.1 found on handle BF20B918 BlockSize : 512 LastBlock : 150A7A8E BlockSize : 2048 LastBlock : 856C Buffer: EFI DVD/CDROM Select Item: 0x19 Select Item: 0x2A SetBootOrderFromQemu: setting BootOrder: success Select Item: 0xE Select Item: 0x19 Start showing progress bar... Press any key to stop it! ...Zzz.... Showing progress bar...Remaining 3 second! Showing progress bar...Remaining 2 second! Showing progress bar...Remaining 1 second! S3Ready! AcpiS3Context: AcpiFacsTable is 0xBFEFE000 AcpiS3Context: IdtrProfile is 0xBFEF3000 AcpiS3Context: S3NvsPageTableAddress is 0x 0 AcpiS3Context: S3DebugBufferAddress is 0xBFEF1000 InstallProtocolInterface: 60FF8964-E906-41D0-AFED-F241E974E08E 0 InstallProtocolInterface: FA20568B-548B-4B2B-81EF-1BA08D4A3CEC 0 Initialize variable error flag (FF) Memory Previous Current Next Type Pages Pages Pages ====== ======== ======== ======== 0A 00000004 00000023 0000002B 09 00000008 0000000D 00000010 00 00000004 00000027 00000030 06 00000024 000000F8 00000136 05 00000030 000000CD 00000100 03 00000180 00000595 000006FA 04 00000F00 00000EB3 00000F00 Booting EFI SCSI Device 1 BlockSize : 512 LastBlock : 150A7A8E PlatformBdsBootFail S3Ready! Memory Previous Current Next Type Pages Pages Pages ====== ======== ======== ======== 0A 0000002B 00000023 0000002B 09 00000010 0000000D 00000010 00 00000030 00000027 00000030 06 00000136 000000F8 00000136 05 00000100 000000CD 00000100 03 000006FA 00000595 000006FA 04 00000F00 00000EB3 00000F00 Booting EFI DVD/CDROM BlockSize : 2048 LastBlock : 1EEEDE BlockSize : 2048 LastBlock : 7 PlatformBdsBootFail S3Ready! Memory Previous Current Next Type Pages Pages Pages ====== ======== ======== ======== 0A 0000002B 00000023 0000002B 09 00000010 0000000D 00000010 00 00000030 00000027 00000030 06 00000136 000000F8 00000136 05 00000100 000000CD 00000100 03 000006FA 00000595 000006FA 04 00000F00 00000EB3 00000F00 Booting EFI Internal Shell InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF06EB80 Loading driver at 0x000BE5D8000 EntryPoint=0x000BE5D8260 Shell.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF070B18 InstallProtocolInterface: 387477C2-69C7-11D2-8E39-00A0C969723B BF07E8A0 InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA BF07EC58 InstallProtocolInterface: 6302D008-7F9B-4F30-87AC-60C9FEF5DA4E BE685D80 Image Return Status = Success PlatformBdsBootSuccess InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC67500 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEE81068 InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BEC67520 InstallProtocolInterface: 330D4706-F2A0-4E4F-A369-B66FA8D54385 BEE82948 InstallProtocolInterface: 348C4D62-BFBD-4882-9ECE-C80BB1C4783B 0 Buffer: EFI DVD/CDROM InstallProtocolInterface: 348C4D62-BFBD-4882-9ECE-C80BB1C4783B 0 InstallProtocolInterface: 348C4D62-BFBD-4882-9ECE-C80BB1C4783B 0

Offline

#4853 2015-04-20 22:40:24

akawoz
Member
Registered: 2015-02-22
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mine was doing that with only 4GB RAM so you're doing better than me! Seems so many variables that make for a stable or not experience. :-)

Fidelix wrote:

@akawoz, my Windows screen also freezes with OVMF if I start with more than 7.8GB of RAM. Don't know what else to do, but other than that everything is working perfectly with my GTX 970.

I only wish I was able to turn on Hyper-v Enlightenments. Damn Nvidia

Offline

#4854 2015-04-21 00:34:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fidelix wrote:

@akawoz, my Windows screen also freezes with OVMF if I start with more than 7.8GB of RAM.

How much RAM do you have?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4855 2015-04-21 07:36:28

acehege
Member
Registered: 2015-04-20
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
acehege wrote:

Hey everybody,

I have gotten KVM VGA-Passthrough to work using this awesome forum and all of your experiences, code snippets and comments.

BUT, everytime i reboot/shutdown a VM with my passed-through Nvidia Cards ( GT630 ), it gives me the dreaded "Error 43" code, and the only fix is to reboot the arch linux host.

--- Info ---

1. I am running the linux-vfio kernel from AUR

2. I have four identical Nvidia GT630 cards

3. I am using OVFM in the VMs

4. The VMs are running Windows 8.1

5. kvm=off is set.

6. I have tried with and without x-vga=on

7. I have tried with and without the following in grub: pcie_acs_override=downstream i915.enable_hd_vgaarb=1

8. I have tried with virt-manager and also commandline qemu, same symptoms.

My current kernel version is 3.19.3.

--- /Info ---

Any help would be highly appreciated!

If any of you need more information or snippets from the system, just ask, and I will deliver wink

Have you tried loading your gpu rom with the romfile= parameter? check out this post https://bbs.archlinux.org/viewtopic.php … 7#p1511177

I tried earlier with just the rom dumped from gpu-z, but I had not seen the post you linked to!

So I tried it, but unfortunately adding the romfile from nvagetbios does not help me, as i still get "Error 43" when rebooting the VM, and to fix it requires me to reboot the host arch linux.

Offline

#4856 2015-04-21 08:15:30

Mike1Q84
Member
Registered: 2015-04-21
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear hackers,

Thank you for this awesome solution to get rid of dual booting for software that can't run efficiently under Linux at the moment. I really appreciate the time and energy you guys invested in this project.

Would you be so kind as to help me sorting out some issues related to ViDock and KVM VGA passthrough?

Under my setup, the GPUs are housed by the ViDock 4 Plus, and connected to my laptop through ExpressCard interface.

The KVM guest can see the GPUs with lspci. But,

With GT650,
With -vga none and nouveau driver loaded, the GT650 can display graphic output on external monitor under LiveCD environments of Ubuntu 14.04.2 LTS and Mint 17 installation ISO, very slow response though.
After nvidia driver installed under Mint 17 and Archlinux KVM guest, the nvidia driver never loaded. Evan with modprobe commands, it will display Error: Driver 'nvlink' is already registered, and the nvidia module will never be inserted.

With 7870
Whenever Seabios kicked in, the external monitor will display messy output of multiple colored rectangles.
And when Windows 7 installation ISO loaded, it will result blue screen of death.
The 7870 itself is in good shape though.

Any help will be greatly appreciated,
Mike

Last edited by Mike1Q84 (2015-04-21 08:20:24)

Offline

#4857 2015-04-21 12:07:20

Fidelix
Member
Registered: 2015-04-19
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Fidelix wrote:

@akawoz, my Windows screen also freezes with OVMF if I start with more than 7.8GB of RAM.

How much RAM do you have?

24 GB. I'd like to give Windows at least 12 GB

Offline

#4858 2015-04-21 15:00:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fidelix wrote:
aw wrote:
Fidelix wrote:

@akawoz, my Windows screen also freezes with OVMF if I start with more than 7.8GB of RAM.

How much RAM do you have?

24 GB. I'd like to give Windows at least 12 GB

I don't have any trouble using larger guests.  If you're not using libvirt to manage the user locked memory limits, you'll need to set a limit matching the expected guest size.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4859 2015-04-21 15:48:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a feeling that guys with "windows freeze at startup after OVMF" may misunderstand the "freeze".
Maybe it doesn't just freeze, but waits for something or actually boots without showing any video output due to "special" way of output?
I remember someone has confused the "freeze" or BSOD with just an unshown loading animation on windows7+OVMF...


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4860 2015-04-21 15:52:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:

Because the network based options arent't really ideal when latency is important and switching by attaching/detaching always takes a few seconds.

Network based options.. provide you the latency of pinging the localhost. It should be 1-2ms or less. If there's a huge overhead causing lags - check your setup.
I'm using VNC for input, and there is no latency or problems.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4861 2015-04-21 16:35:39

Naruni
Member
Registered: 2010-08-05
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GPU is working smoothly on win7, audio is working fine when not being overused (one sound at a time), then it is constantly popping, clicking and causing severe system lag in games.  It can play a youtube video without any problems.  When first using only ac97, not even the standard windows bootup sound could play without severe stuttering.  I looked at the solution of

modprobe i810_audio clocking=48000

but I use neither i810 or OSS (that I know of).

Host audio:

00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller

Audio options for qemu:

export QEMU_AUDIO_DRV=alsa -soundhw all

Edit:
Go figure. Changing to -soundhw ac97 and adding the i810 manual clocking module parameter fixes almost all of the audio popping. Now I'm getting just a couple hiccups if there are multiple voices playing.

Last edited by Naruni (2015-04-21 16:55:44)

Offline

#4862 2015-04-21 17:10:52

Fidelix
Member
Registered: 2015-04-19
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I have a feeling that guys with "windows freeze at startup after OVMF" may misunderstand the "freeze".
Maybe it doesn't just freeze, but waits for something or actually boots without showing any video output due to "special" way of output?
I remember someone has confused the "freeze" or BSOD with just an unshown loading animation on windows7+OVMF...

After 3 minutes or so the Windows loading animation appears, but if I leave it there for 30min it keeps spinning.

aw wrote:
Fidelix wrote:
aw wrote:

How much RAM do you have?

24 GB. I'd like to give Windows at least 12 GB

I don't have any trouble using larger guests.  If you're not using libvirt to manage the user locked memory limits, you'll need to set a limit matching the expected guest size.

How do I use virsh or virt-manager to "manage the user locked memory limits"?
Thanks

Offline

#4863 2015-04-21 17:13:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fidelix wrote:
aw wrote:
Fidelix wrote:

24 GB. I'd like to give Windows at least 12 GB

I don't have any trouble using larger guests.  If you're not using libvirt to manage the user locked memory limits, you'll need to set a limit matching the expected guest size.

How do I use virsh or virt-manager to "manage the user locked memory limits"?
Thanks

It happen automatically if you manage the VM through libvirt and expose the device as a <hostdev> (ie. don't hide it behind <qemu:args>)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4864 2015-04-21 17:37:42

Fidelix
Member
Registered: 2015-04-19
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It happen automatically if you manage the VM through libvirt and expose the device as a <hostdev> (ie. don't hide it behind <qemu:args>)

Then I suppose something else is wrong. I created it and manage the VM through libvirt/virt-manager (and sometimes virsh) and I have no qemu:args. The log has nothing useful.

This is the relevant memory stuff on the XML:

  <memory unit='KiB'>7668000</memory>
  <currentMemory unit='KiB'>7668000</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

Offline

#4865 2015-04-21 17:39:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fidelix wrote:
aw wrote:

It happen automatically if you manage the VM through libvirt and expose the device as a <hostdev> (ie. don't hide it behind <qemu:args>)

Then I suppose something else is wrong. I created it and manage the VM through libvirt/virt-manager (and sometimes virsh) and I have no qemu:args. The log has nothing useful.

This is the relevant memory stuff on the XML:

  <memory unit='KiB'>7668000</memory>
  <currentMemory unit='KiB'>7668000</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

And you've allocated sufficient hugepages for the VM?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4866 2015-04-21 19:21:10

Fidelix
Member
Registered: 2015-04-19
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Fidelix wrote:
aw wrote:

It happen automatically if you manage the VM through libvirt and expose the device as a <hostdev> (ie. don't hide it behind <qemu:args>)

Then I suppose something else is wrong. I created it and manage the VM through libvirt/virt-manager (and sometimes virsh) and I have no qemu:args. The log has nothing useful.

This is the relevant memory stuff on the XML:

  <memory unit='KiB'>7668000</memory>
  <currentMemory unit='KiB'>7668000</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

And you've allocated sufficient hugepages for the VM?

With the VM shut down:

AnonHugePages:    604160 kB
HugePages_Total:    5000
HugePages_Free:     5000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Something wrong?

Offline

#4867 2015-04-21 20:07:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fidelix wrote:

With the VM shut down:

AnonHugePages:    604160 kB
HugePages_Total:    5000
HugePages_Free:     5000
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Something wrong?

That should get you to a 10G guest


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4868 2015-04-21 20:37:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4869 2015-04-21 23:49:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Just curious, if all the PCI IDS for Hawaii(and hawaii xt) and Bonaire cards are known and listed in pci.ids file, why you've made this patch non-device-specific?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4870 2015-04-21 23:53:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Just curious, if all the PCI IDS for Hawaii(and hawaii xt) and Bonaire cards are known and listed in pci.ids file, why you've made this patch non-device-specific?

Because I thought people here would want to try it asap.  As the patch indicates, it's a prototype, it works for me, and I don't really want to limit it unnecessarily for those willing to try it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4871 2015-04-22 16:40:06

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello all, I have the follow setup:
1. intel motherboard with Xeon CPU (E3)
2. NVidia Quadro 2000 (no UEFI support)

With the IGD as primary, I tried applying the patches on a Ubuntu server installation though an older kernel (3.13) as explained here: http://ubuntuforums.org/showthread.php?t=2266916 . Everything works fine until I assign an additional PCI device to the VM. In which case the host just completely hangs after the VM shutdown. If I don't assign extra PCI devices to the machine, it works fine, reboots as expected, etc.
I added an AMD/ATI graphic card and set it as primary display. I disabled the NVIDIA display and when I launch the VM everything works fine, but the VM doesn't survive a reboot. It seems that at the first reboot the NVidia doesn't properly reset.

Any hint on what to investigate?
Thanks!

Offline

#4872 2015-04-22 18:37:22

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Do you think that could help with reset of AMD W7100 ?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4873 2015-04-22 18:44:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Do you think that could help with reset of AMD W7100 ?

Dunno, try it.  That's a Tonga GPU, who knows, maybe the bug extends further than Bonaire/Hawaii.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4874 2015-04-23 10:14:58

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Anything for Nvidia's GPUs with reset issues ? XD

Offline

#4875 2015-04-23 12:43:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Anything for Nvidia's GPUs with reset issues ? XD

I'm not aware of any widespread reset issues on nvidia.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4876 2015-04-23 12:56:43

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Denso wrote:
aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Anything for Nvidia's GPUs with reset issues ? XD

I'm not aware of any widespread reset issues on nvidia.

Well . It only happens after I install the nvidia's drivers . Using the Microsoft Basic one is fine . When the drivers are installed , rebooting the VM works , I can see the OVMF screen + Windows logo , then just before it shows the desktop it goes black . The VM doesn't crash nor the host . No dmesg errors . One guest boot per host boot . GT610 .

lspci -vvv : 02:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Gigabyte Technology Co., Ltd Device 3549 Physical Slot: 6 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 29 Region 0: Memory at f8000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M] Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at d000 [size=128] Expansion ROM at f9000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [128 v1] Power Budgeting <?> Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Kernel driver in use: vfio-pci Kernel modules: nouveau

Offline

#4877 2015-04-23 13:01:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:
Denso wrote:

Anything for Nvidia's GPUs with reset issues ? XD

I'm not aware of any widespread reset issues on nvidia.

Well . It only happens after I install the nvidia's drivers . Using the Microsoft Basic one is fine . When the drivers are installed , rebooting the VM works , I can see the OVMF screen + Windows logo , then just before it shows the desktop it goes black . The VM doesn't crash nor the host . No dmesg errors . One guest boot per host boot . GT610 .

widespread...

Sorry, one user having trouble with a low-end Fermi-based card does not make for a class problem.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4878 2015-04-23 14:13:10

Parzival
Member
Registered: 2015-04-23
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with passing through Intel HD 5000 on Intel NUC. I've followed the guide on the first pinned page of the thread:

  • Installed linux-vfio kernel from aur

  • loaded pci-stub on boot

  • added the right boot options

This is the device:

00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09) DeviceName: CPU1 Subsystem: Intel Corporation Device 2054 Kernel driver in use: pci-stub Kernel modules: i915

The resulting fault message:

qemu-system-i386: -device vfio-pci,host=0000:00:02.0: vfio: failed to set iommu for container: Operation not permitted qemu-system-i386: -device vfio-pci,host=0000:00:02.0: vfio: failed to setup container for group 1 qemu-system-i386: -device vfio-pci,host=0000:00:02.0: vfio: failed to get group 1 qemu-system-i386: -device vfio-pci,host=0000:00:02.0: Device initialization failed. qemu-system-i386: -device vfio-pci,host=0000:00:02.0: Device 'vfio-pci' could not be initialized

dmesg

vfio-pci 0000:00:02.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

So, I'm interested to know :

  • whether anyone had the same problem and was able to solve it

  • what/how should I ask the vendor (Intel I suppose) so I can resolve the above issue

Thanks

Offline

#4879 2015-04-23 14:14:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Well . It only happens after I install the nvidia's drivers . Using the Microsoft Basic one is fine . When the drivers are installed , rebooting the VM works , I can see the OVMF screen + Windows logo , then just before it shows the desktop it goes black . The VM doesn't crash nor the host . No dmesg errors . One guest boot per host boot . GT610 .

 Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress-

Does the TC/VC map reported in the Virtual Channel capability (01 above) change from the first host boot to after a guest failure?  I had to add VC save/restore for platforms that use non-default TC/VC maps.  Kernels 3.14 and newer should already include this though.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4880 2015-04-23 14:15:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Parzival wrote:

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4881 2015-04-23 14:24:56

Parzival
Member
Registered: 2015-04-23
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Parzival wrote:

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

Offline

#4882 2015-04-23 14:32:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Parzival wrote:
aw wrote:
Parzival wrote:

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

Because not only is IGD not a discrete device in the sense that it's integrated, but also it's current driver requirements extend beyond the individual PCI device.  It requires various operation regions outside of the device to be poked through, it requires registers on the host bridge to be assigned, it requires specific PCI IDs matching the host chipset on the host bridge and PCH device, etc.  There are proof-of-concept level patches available, but I wouldn't recommend them unless you plan to contribute to their development.  Likely the only way we'll see progress is in a new driver from Intel that confines the platform requirements for IGD to the PCI device itself.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4883 2015-04-23 14:39:01

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Does the TC/VC map reported in the Virtual Channel capability (01 above) change from the first host boot to after a guest failure?  I had to add VC save/restore for platforms that use non-default TC/VC maps.  Kernels 3.14 and newer should already include this though.

No , nothing in the GPU or the integrated HDMI audio devices changes when the guest got rebooted .

On a side note , ASUS released a new BIOS update for X99-Deluxe . After applying it , I no longer recieve the "AER: Multiple Corrected error" in dmesg anymore smile

It seems to be a widespread issue with X99 . This platform's stability is not good compared to my old Z77 .

EDIT :

I am passing through the Intel Audio device to the same VM . This device gets TC/VC changes after guest reboot :

Before VM reboot : VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=2 ArbSelect=Fixed TC/VC=04 Status: NegoPending- InProgress- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel
After VM reboot : VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=2 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 

I'll reboot the host & VM without attaching it , and will see if it is to blame .

Last edited by Denso (2015-04-23 14:46:27)

Offline

#4884 2015-04-23 14:43:52

Parzival
Member
Registered: 2015-04-23
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Parzival wrote:
aw wrote:

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

Because not only is IGD not a discrete device in the sense that it's integrated, but also it's current driver requirements extend beyond the individual PCI device.  It requires various operation regions outside of the device to be poked through, it requires registers on the host bridge to be assigned, it requires specific PCI IDs matching the host chipset on the host bridge and PCH device, etc.  There are proof-of-concept level patches available, but I wouldn't recommend them unless you plan to contribute to their development.  Likely the only way we'll see progress is in a new driver from Intel that confines the platform requirements for IGD to the PCI device itself.

Thanks.
I know that this could be a broad answer, but still, I'll appreciate if you can explain why I do able to pass through IGD using Xen? What's the difference?

Offline

#4885 2015-04-23 14:47:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Parzival wrote:
aw wrote:
Parzival wrote:

Can you explain why? Will it be supported in the near future?

Because not only is IGD not a discrete device in the sense that it's integrated, but also it's current driver requirements extend beyond the individual PCI device.  It requires various operation regions outside of the device to be poked through, it requires registers on the host bridge to be assigned, it requires specific PCI IDs matching the host chipset on the host bridge and PCH device, etc.  There are proof-of-concept level patches available, but I wouldn't recommend them unless you plan to contribute to their development.  Likely the only way we'll see progress is in a new driver from Intel that confines the platform requirements for IGD to the PCI device itself.

Thanks.
I know that this could be a broad answer, but still, I'll appreciate if you can explain why I do able to pass through IGD using Xen? What's the difference?

Because Xen is willing to hack their VM to bits, regardless of the support-ability of the solution in order to make something work.  When they tried to merge those hacks into upstream QEMU is when we started talking to Intel about how to do this correctly.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4886 2015-04-23 15:23:51

Tyrewt
Member
Registered: 2014-09-13
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

I'm willing to test, but Ubuntu 14.04 LTS is at qemu-kvm 2.0.0+dfsg-2ubuntu1.9, would this patch still be applicable?

Offline

#4887 2015-04-23 17:31:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Parzival wrote:
aw wrote:
Parzival wrote:

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

You can try looking into KVMGT... As i understood their talk and presentation - they're willing to allow VMs access intel's IGD and stuff. But there's only a prototype yet.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4888 2015-04-23 17:53:40

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This a random comment, but I suppose it may be useful for someone. I have -finally- discovered what the Intel VT-d enabled Chipsets (Q and C Series) are useful for when compared to the non VT-d ones.

This is my lspci output, with IOMMU Group data manually added:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06) -- GROUP 0 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) -- GROUP 1 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06) -- GROUP 2 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) -- GROUP 3 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) -- GROUP 4 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) -- GROUP 5 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05) -- GROUP 6 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) -- GROUP 7 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) -- GROUP 8 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) -- GROUP 9 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) -- GROUP 10 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) -- GROUP 11 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) -- GROUP 12 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) -- GROUP 13 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05) -- GROUP 14 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) -- GROUP 14 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) -- GROUP 14 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) -- GROUP 14 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770] -- GROUP 1 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] -- GROUP 1 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) -- GROUP 15 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) -- GROUP 16 04:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) -- GROUP 17 04:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) -- GROUP 18 04:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) -- GROUP 19 04:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) -- GROUP 20 04:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) -- GROUP 21 08:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) -- GROUP 22 09:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01) -- GROUP 23 0a:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01) -- GROUP 23 0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) -- GROUP 24

This is how a Z97 Chipset IOMMU Grouping looks like:

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 OEM] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1) ### Group 2 ### 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) ### Group 3 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 4 ### 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller ### Group 5 ### 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 ### Group 6 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V ### Group 7 ### 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 ### Group 8 ### 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller ### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) 03:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 04:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 04:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 04:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 04:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 09:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller ### Group 10 ### 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 ### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Offline

#4889 2015-04-23 17:59:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zir_blazer wrote:

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Update your kernel, quirks for Lynxpoint PCH root ports were added in 3.15.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4890 2015-04-23 18:24:46

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
zir_blazer wrote:

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Update your kernel, quirks for Lynxpoint PCH root ports were added in 3.15.

My computer is the first one with a C226, the other is from here:
http://ubuntuforums.org/showthread.php? … st13237568

I'm using Kernel 3.17.1, the other guy says he has 3.16. I wasn't aware that the grouping was related to a quirck. Got a more recent IOMMU Group list on a non-Q or C Chipset too see it fixed?
So far, I was unable to find what makes the VT-d support on these Chipsets useful for. I through than the IOMMU Grouping difference made sense.

Last edited by zir_blazer (2015-04-23 18:27:28)

Offline

#4891 2015-04-23 19:01:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zir_blazer wrote:
aw wrote:
zir_blazer wrote:

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Update your kernel, quirks for Lynxpoint PCH root ports were added in 3.15.

My computer is the first one with a C226, the other is from here:
http://ubuntuforums.org/showthread.php? … st13237568

I'm using Kernel 3.17.1, the other guy says he has 3.16. I wasn't aware that the grouping was related to a quirck. Got a more recent IOMMU Group list on a non-Q or C Chipset too see it fixed?
So far, I was unable to find what makes the VT-d support on these Chipsets useful for. I through than the IOMMU Grouping difference made sense.

Ah, sorry I keyed on the 8-series chipset, that one is quirked and I'm currently harassing Intel about the 9-series.  TBH, I have no idea what you're talking about with Q vs C chipsets.  The reason that anything without native ACS allows smaller IOMMU groups is because we have quirks in the kernel that enable non-standard isolation within the chipset/device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4892 2015-04-23 19:27:04

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm talking about this: http://ark.intel.com/compare/75522,7501 … 0711,27215

Intel says that for LGA 1150, only the Q87 and C222, C224 and C226 Chipsets supports VT-d. The point is, what you actually need VT-d Chipset support for? Before I builded my current system (Xeon E3-1245V3 and a Supermicro X10SAT), I spend around six months researching if you needed Chipset support or not, since there was a lot of contradicting info. Some people said and proved that with only Processor VT-d support you could get Passthrough working, but you had manufacturers like ASUS that instead claimed that they didn't supported VT-d on typical H or Z-Chipsets based Motherboard since a VT-d Chipset was required too. I decided to play safe and go Server.
What I found out by looking at a few lists of IOMMU Groups like the one I linked, is that on consumer Motherboards, they usually have a single huge group for the Chipset's PCIe Lanes. Instead, on mine, nearly each device has its own group. Keep in mind that this is using only intel_iommu=on on the Boot Loader config file, there is no Kernel patch or anything else applied, its pretty much a very dull Arch Linux install, since I am using it as Dom0 for Xen, but I'm looking at KVM in case I get a new GeForce. Because during an entire year I found no other explanation with a practical example, I concluded that Intel calls VT-d support on the Chipset, one that implements IOMMU for the devices connected to its PCIe Controller. I think I saw some people that applied the ACS override patch and their IOMMU Group list looked like mine, but not out of the box. But I didn't paid attention to other details like the Kernel, or quircks like the one that you mention, so I don't know if my theory is true or not.

Offline

#4893 2015-04-23 19:41:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zir_blazer wrote:

I'm talking about this: http://ark.intel.com/compare/75522,7501 … 0711,27215

Intel says that for LGA 1150, only the Q87 and C222, C224 and C226 Chipsets supports VT-d. The point is, what you actually need VT-d Chipset support for? Before I builded my current system (Xeon E3-1245V3 and a Supermicro X10SAT), I spend around six months researching if you needed Chipset support or not, since there was a lot of contradicting info. Some people said and proved that with only Processor VT-d support you could get Passthrough working, but you had manufacturers like ASUS that instead claimed that they didn't supported VT-d on typical H or Z-Chipsets based Motherboard since a VT-d Chipset was required too. I decided to play safe and go Server.
What I found out by looking at a few lists of IOMMU Groups like the one I linked, is that on consumer Motherboards, they usually have a single huge group for the Chipset's PCIe Lanes. Instead, on mine, nearly each device has its own group. Keep in mind that this is using only intel_iommu=on on the Boot Loader config file, there is no Kernel patch or anything else applied, its pretty much a very dull Arch Linux install, since I am using it as Dom0 for Xen, but I'm looking at KVM in case I get a new GeForce. Because during an entire year I found no other explanation with a practical example, I concluded that Intel calls VT-d support on the Chipset, one that implements IOMMU for the devices connected to its PCIe Controller. I think I saw some people that applied the ACS override patch and their IOMMU Group list looked like mine, but not out of the box. But I didn't paid attention to other details like the Kernel, or quircks like the one that you mention, so I don't know if my theory is true or not.

I guess I should stop using VT-d on my H67 desktop then, Intel® Virtualization Technology for Directed I/O (VT-d):    No

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.  In the early days of VT-d, back when there was an MCH (Memory Controller HUB) and IOH (I/O Hub), VT-d support in the chipset was required.  The processor wasn't involved in the IOMMU.  The IOMMU needed to live between I/O and memory, which the processor was not.  However, when the memory controller was pulled into the CPU, along with (sometimes) integrated graphics, and high speed root ports, VT-d also needed to move into the processor.  I don't really know what Intel is trying to say with that entry in ARK, whether it's any indication of actual support, or just a glitch in their database, but I'm pretty sure the difference between your 8-series having separate IOMMU groups and the 9-series not is software, not hardware.  Maybe Intel will be less willing to provide us with quirks if there's no server versions of a given chipset.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4894 2015-04-23 21:58:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.

So we're back to basics, eh?
"If you want stability and even some support from the manufacturer - use expensive server-class hardware."
Cheapest Xeon E5 costs more that my whole PC.
Are you sure that this breakage and lack of isolation and more quirks coming from newer hardware isn't "accidental" like it was with nvidia?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4895 2015-04-23 22:18:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.

So we're back to basics, eh?
"If you want stability and even some support from the manufacturer - use expensive server-class hardware."
Cheapest Xeon E5 costs more that my whole PC.
Are you sure that this breakage and lack of isolation and more quirks coming from newer hardware isn't "accidental" like it was with nvidia?

Oh come on, is invoking Nvidia the new Godwin's law for this thread?  ACS and device isolation is a high-end feature.  I think it's caught Intel by surprise that yes, they do need to advertise the isolation of devices and no, VT-d isn't much use without it.  We've also been quirking Intel 10G NICs to expose isolation, along with workstation and server class chipsets like X79 and X99.  How does that fit into your conspiracy theory?  If we've sparked change in the consumer class products, it may be a couple years before those products start leaving the fab.  In the meantime, we're getting pretty good cooperation.  Maybe there are technical reasons why we're not getting quirks for client CPUs.  Maybe the thing that enables SLI/Crossfire is lack of isolation.  I don't know.

Furthermore, the cheapest E5 Xeons I see on newegg are around $200US, so if your PC is that crappy, maybe you could use an upgrade.

Last edited by aw (2015-04-23 22:22:48)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4896 2015-04-24 01:16:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
aw wrote:

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.

So we're back to basics, eh?
"If you want stability and even some support from the manufacturer - use expensive server-class hardware."
Cheapest Xeon E5 costs more that my whole PC.
Are you sure that this breakage and lack of isolation and more quirks coming from newer hardware isn't "accidental" like it was with nvidia?

Oh come on, is invoking Nvidia the new Godwin's law for this thread?  ACS and device isolation is a high-end feature.  I think it's caught Intel by surprise that yes, they do need to advertise the isolation of devices and no, VT-d isn't much use without it.  We've also been quirking Intel 10G NICs to expose isolation, along with workstation and server class chipsets like X79 and X99.  How does that fit into your conspiracy theory?  If we've sparked change in the consumer class products, it may be a couple years before those products start leaving the fab.  In the meantime, we're getting pretty good cooperation.  Maybe there are technical reasons why we're not getting quirks for client CPUs.  Maybe the thing that enables SLI/Crossfire is lack of isolation.  I don't know.

Furthermore, the cheapest E5 Xeons I see on newegg are around $200US, so if your PC is that crappy, maybe you could use an upgrade.

I just find it strange that AMD, being the main and the only "rival" of intel on HPC and desktop markets have no problems with this... right? I agree, sometimes they have different problems, like minor or major bugs or firmware support. Their problems are different.
If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

I just can't conclude anything viable from this situation - all I see now is a lot of quirks and patches and cooperation related solely to intel.
Maybe their CPUs are more widespread?.. Or they are more "open" to the community(considering the open-source work they do)? Or, maybe, their hardware is more troublesome? Or this ACS isn't even present on AMD systems, and nobody noticed it?

P.S.
Cheapest modern Xeon E5 in my areacountry costs around 1000$.
P.P.S.
Godwin's law, heh, I see what you did there.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4897 2015-04-24 01:39:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I just find it strange that AMD, being the main and the only "rival" of intel on HPC and desktop markets have no problems with this... right? I agree, sometimes they have different problems, like minor or major bugs or firmware support. Their problems are different.

False, while AMD did manage to get ACS on root ports, we still have ACS quirks for other AMD devices.

If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

False, while AMD-IOMMU specs were maybe out before VT-d, it was incredibly difficult to get actual AMD-Vi hardware until well after Intel.  You certainly won't find it on Athlon 64 hardware with AMD-Vi.

I just can't conclude anything viable from this situation - all I see now is a lot of quirks and patches and cooperation related solely to intel.
Maybe their CPUs are more widespread?.. Or they are more "open" to the community(considering the open-source work they do)? Or, maybe, their hardware is more troublesome? Or this ACS isn't even present on AMD systems, and nobody noticed it?

Intel is far more widespread than AMD and they make far more devices.  They not only have more chipsets, but they rev them more often.  Take for instance the AMD 900-series chipset that started shipping in 2011 and is still the latest chipset for non-APU AMD processors.  That was about the same time Intel was shipping the 6-series chipset.  Since then Intel also shipped the 7-series, 8-series, and 9-series as well as X79 and X99.  Not to mention all the peripherals.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4898 2015-04-24 06:05:43

mqddb
Member
Registered: 2015-04-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone:
      Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.

Offline

#4899 2015-04-24 06:49:30

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I don't really know what Intel is trying to say with that entry in ARK, whether it's any indication of actual support, or just a glitch in their database, but I'm pretty sure the difference between your 8-series having separate IOMMU groups and the 9-series not is software, not hardware.  Maybe Intel will be less willing to provide us with quirks if there's no server versions of a given chipset.

aw wrote:

ACS and device isolation is a high-end feature.

That is my point. What I try to say is that is intentional that on consumer Chipsets you get a single group with the PCI Controller Ports and everything else below it. On C226 I got it working out of the box since its a Workstation/Server Chipset, I didn't had to apply patchs or anything to see nearly everything in its own group. It seems that what you're applying quircks to, is to force to enable it anyways (Similar to Intel only allowing you to use the Unlocked Multiplier of their K Series Processors on Z Chipset Motherboards, while some manufacturers discovered ways to bypass that to overclock in cheaper Chipsets: http://www.xbitlabs.com/news/mainboards … oards.html ). Since you were unaware that Intel sells VT-d support as an extra feature in some Chipsets at the first place, its probable that these quircks are related to that.
What I'm missing is having a lot of IOMMU Groups examples on different platforms. I know how mine in a C226 looks. I know the guy with the Z97 that I posted (Which I think he mentioned to not be using the ACS Override patch), and recall a few more consumer platforms with that same arrangement on other Desktop 8-Series. It is not in-depth enough to confirm this, but I'm inclined to believe than that is what VT-d support on the Chipset means for Intel. It seems that the IOMMU isolation capabilities are exclusive for the Chipset PCIe Controller instead of a global one.

aw wrote:

along with workstation and server class chipsets like X79 and X99.

These are high end Desktop. Workstation and Server class are their same-silicon and more expensive C counterparts, C202, C204 and C206 for Sandy Bridge-E/Ivy Bridge-E, and C216 for Haswell-E.


Duelist wrote:

If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

AMD-Vi was first supported on Desktop Chipsets with the 890FX, that was during the Phenom II era. 990FX supports it too. However, Bulldozer and derivated processors also incorporate their own IOMMU, so you don't need Chipset (Similar to Intel).

Last edited by zir_blazer (2015-04-24 06:50:40)

Offline

#4900 2015-04-24 12:50:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mqddb wrote:

Hello everyone:
      Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.

No, an assigned GPU does not support any of the -display methods.  Remote displays with GPU assignment requires guest-based solutions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4901 2015-04-24 13:05:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

zir_blazer wrote:

That is my point. What I try to say is that is intentional that on consumer Chipsets you get a single group with the PCI Controller Ports and everything else below it. On C226 I got it working out of the box since its a Workstation/Server Chipset, I didn't had to apply patchs or anything to see nearly everything in its own group.

I'm sorry to blow your theory, but you got it to work on C226 because quirks are already in place for your PCH root ports and you apparently don't have or aren't using multiple processor-based root ports.  There's no difference between the consumer 8-series chipsets and the "workstation" version with respect to isolation.

It seems that what you're applying quircks to, is to force to enable it anyways (Similar to Intel only allowing you to use the Unlocked Multiplier of their K Series Processors on Z Chipset Motherboards, while some manufacturers discovered ways to bypass that to overclock in cheaper Chipsets: http://www.xbitlabs.com/news/mainboards … oards.html ). Since you were unaware that Intel sells VT-d support as an extra feature in some Chipsets at the first place, its probable that these quircks are related to that.
What I'm missing is having a lot of IOMMU Groups examples on different platforms. I know how mine in a C226 looks. I know the guy with the Z97 that I posted (Which I think he mentioned to not be using the ACS Override patch), and recall a few more consumer platforms with that same arrangement on other Desktop 8-Series. It is not in-depth enough to confirm this, but I'm inclined to believe than that is what VT-d support on the Chipset means for Intel. It seems that the IOMMU isolation capabilities are exclusive for the Chipset PCIe Controller instead of a global one.

Trust me, I know IOMMU groups.  You don't need to apply the ACS patch because a) there's already a kernel-based workaround in place for PCH root ports and b) you're not using the processor based root ports in a way that exposes the lack of isolation.  You have not discovered some secret decoder for device isolation in ark.  Don't believe me, run 'dmesg | grep "Intel PCH root port ACS workaround enabled"' <-- that's why your IOMMU groups look the way they do.  You're welcome.

aw wrote:

along with workstation and server class chipsets like X79 and X99.

These are high end Desktop. Workstation and Server class are their same-silicon and more expensive C counterparts, C202, C204 and C206 for Sandy Bridge-E/Ivy Bridge-E, and C216 for Haswell-E.

No, if anything it's the other way, the C series are just rebranded desktop chipsets, the X series actually add more high-end features and are different silicon.

Duelist wrote:

If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

AMD-Vi was first supported on Desktop Chipsets with the 890FX, that was during the Phenom II era. 990FX supports it too. However, Bulldozer and derivated processors also incorporate their own IOMMU, so you don't need Chipset (Similar to Intel).

This we can agree on, I think.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4902 2015-04-24 16:25:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

False, while AMD did manage to get ACS on root ports, we still have ACS quirks for other AMD devices.

Did you mean this?

aw wrote:

You certainly won't find it on Athlon 64 hardware with AMD-Vi.

What about K10.5(Kuma) athlons, which have their microarch "backported" from phenoms II? Except they were AM2+ socket-based, so their chipsets were only 7XX, which, i believe, do not support IOMMU?


Well, now it makes sense. FX-line is semi-dead(Hail Opteron-X, it is an APU), and all APUs have Unified North Bridge and will only differ by FCH, which isn't so important in IOMMU working, right?

So if intel would make their new chipsets as rare as AMD does, there wouldn't be such a great difference in quirks per secondtime.


P.S.
Why do I remember quirks.c file having only seven lines?..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4903 2015-04-24 16:48:22

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paperinick wrote:

Hello all, I have the follow setup:
1. intel motherboard with Xeon CPU (E3)
2. NVidia Quadro 2000 (no UEFI support)

With the IGD as primary, I tried applying the patches on a Ubuntu server installation though an older kernel (3.13) as explained here: http://ubuntuforums.org/showthread.php?t=2266916 . Everything works fine until I assign an additional PCI device to the VM. In which case the host just completely hangs after the VM shutdown. If I don't assign extra PCI devices to the machine, it works fine, reboots as expected, etc.
I added an AMD/ATI graphic card and set it as primary display. I disabled the NVIDIA display and when I launch the VM everything works fine, but the VM doesn't survive a reboot. It seems that at the first reboot the NVidia doesn't properly reset.

Any hint on what to investigate?
Thanks!

I did a lots of experiments with the patches and an extra VGA. So the issues boils down to this and maybe a bug (in Qemu?) :
if the VM starts, the GPU gets in a "weird" internal state. If I kill the VM before Windows starts the NVidia driver, the VM won't boot anymore until the host is reset. What is worst is that any attempt to reboot the guest VM will cause the host to apparently freeze (I don't know if it's just the graphic, I will try to ssh to the host in the frozen state). If Windows start the NVidia driver the GPU is properly put into a "bootable" state and all is nice and fine.
Later I will try to see what happens by using virsh to boot and shutdown the VM.
Has anybody experienced anything similar?
Thanks (more details to follow).

Offline

#4904 2015-04-24 17:13:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

False, while AMD did manage to get ACS on root ports, we still have ACS quirks for other AMD devices.

Did you mean this?

Yeah, that added to a set of quirks that were already there for devices attached to the AMD southbridge

aw wrote:

You certainly won't find it on Athlon 64 hardware with AMD-Vi.

What about K10.5(Kuma) athlons, which have their microarch "backported" from phenoms II? Except they were AM2+ socket-based, so their chipsets were only 7XX, which, i believe, do not support IOMMU?

I'd guess not.

Well, now it makes sense. FX-line is semi-dead(Hail Opteron-X, it is an APU), and all APUs have Unified North Bridge and will only differ by FCH, which isn't so important in IOMMU working, right?

Yeah, they look a lot more similar to an Intel processor+PCH model.  Isolation is not required for the IOMMU itself, but we require isolation to make device assignment safe.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4905 2015-04-24 17:16:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paperinick wrote:
paperinick wrote:

Hello all, I have the follow setup:
1. intel motherboard with Xeon CPU (E3)
2. NVidia Quadro 2000 (no UEFI support)

With the IGD as primary, I tried applying the patches on a Ubuntu server installation though an older kernel (3.13) as explained here: http://ubuntuforums.org/showthread.php?t=2266916 . Everything works fine until I assign an additional PCI device to the VM. In which case the host just completely hangs after the VM shutdown. If I don't assign extra PCI devices to the machine, it works fine, reboots as expected, etc.
I added an AMD/ATI graphic card and set it as primary display. I disabled the NVIDIA display and when I launch the VM everything works fine, but the VM doesn't survive a reboot. It seems that at the first reboot the NVidia doesn't properly reset.

Any hint on what to investigate?
Thanks!

I did a lots of experiments with the patches and an extra VGA. So the issues boils down to this and maybe a bug (in Qemu?) :
if the VM starts, the GPU gets in a "weird" internal state. If I kill the VM before Windows starts the NVidia driver, the VM won't boot anymore until the host is reset. What is worst is that any attempt to reboot the guest VM will cause the host to apparently freeze (I don't know if it's just the graphic, I will try to ssh to the host in the frozen state). If Windows start the NVidia driver the GPU is properly put into a "bootable" state and all is nice and fine.
Later I will try to see what happens by using virsh to boot and shutdown the VM.
Has anybody experienced anything similar?
Thanks (more details to follow).

You have a Quadro 2000, just assigned it as a secondary device and be done with it.  https://access.redhat.com/documentation … e-GPU.html

If it doesn't work, update kernel and/or QEMU.  Note that only K-series Quadros are actually supported, but Fermi are likely to work.  Quadro cards do not work well when configured as the primary display for a VM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4906 2015-04-25 10:30:21

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

There's no difference between the consumer 8-series chipsets and the "workstation" version with respect to isolation.

There MUST be a difference somewhere. Both ASUS and Supermicro previously did some comments about that. In the case of ASUS, their excuse to never bother making proper DMAR Tables for Intel Motherboards was that non-Q/C Chipsets didn't "fully supported VT-d", which they stated numerous times quite adamantly. Examples here:


http://rog.asus.com/forum/showthread.ph … post412844

Raja@ASUS wrote:

Thats because the chipset does not officially support it either. Some of the tests will pass so some motherboard vendors will claim to have support, BUT if you search you'll find the same vendors have issues with certain Vt-d features also. We don't want to play that game.

For an ablsoute guarantee of Vt-d one needs to purchase a Q series chipset.


http://www.overclock.net/t/1488891/asus … t_22328891

Raja@ASUS wrote:

Just how Intel have done it for a few generations now. I have seen no documentation (yet) that suggests that VT-d IO is supported on the Z series of chipsets. Z87 was the same. Q series only. If that has changed its news to me (but not impossible).

http://ark.intel.com/products/82012 Does not show any Vt-d support there yet does it?

Some of the other vendors have claimed Vt-D for a few generations, but all 50 Vt-d tests did not pass on their boards. We won't claim support when that happens. Other vendors have more lax rules.


http://www.overclock.net/t/1488891/asus … t_22329055

Raja@ASUS wrote:

The Vt-D support was not full directed IO and all tests will not validate on the Z series chipsets in the past. SO take that for what you will.

Nothing has chanhed as far as I am aware.


I end up having to go in this same circle with users on every gen. The CPU supporting and Intel fully enabling it on the chipset are two different things.

And yes any user finding it works partially inthe usage scenario is lucky, as all tests will not validate - so its a crapshoot. For us the risk in claiming partial support outweighs being honest about it.

I'm missing one more Post from him where he said that non-Q Chipsets "fails a battery of 25 of 50 tests" or something like that.


When I asked about VT-d support in a Z87 Motherboard to Supermicro mail support, they replied this:

Since Z87 chipset does not support VT-d,  onboard LAN will not support it either because it is connected to PCH PCIe port.  One workaround is to use a VT-d enabled PCIe device and plug it into CPU based PCIe-port on board.  Along with a VT-d enabled CPU the above workaround should work per Intel.


Also, I just checked the 8-Series Chipset Datasheet, here: http://www.intel.com/content/dam/www/pu … asheet.pdf

It mentions this about VT-d support in Chipsets (Page 253):

5.29.2 Intel® VT-d Features Supported
• The following devices and functions support FLR in the PCH:
— High Definition Audio (Device 27: Function 0)
— SATA Host Controller 1 (Device 31: Function 2)
— SATA Host Controller 2 (Device 31: Function 5)
— USB2 (EHCI) Host Controller 1 (Device 29: Function 0)
— USB2 (EHCI) Host Controller 2 (Device 26: Function 0)
— GbE Lan Host Controller (Device 25: Function 0)
• Interrupt virtualization support for IOxAPIC
• Virtualization support for HPETs

However, VT-d itself does not appear on the feature matrix at Page 52. It mentions this instead:

2. Table above shows feature differences between the PCH SKUs. If a feature is not listed in the table it is
considered a Base feature that is included in all SKUs.

...But that contradicts previous statements.
It may be possible that the IOMMU Groups isolation is unrelated to Q/C Chipsets, however, they are NOT the same. There is a difference somewhere. If you have good contacts with Intel, it may be useful if you get a definite answer about this so you can stop a collection of 3 year old claims.



aw wrote:

No, if anything it's the other way, the C series are just rebranded desktop chipsets, the X series actually add more high-end features and are different silicon.

You have two C-Series Chipsets per generation, the ones based on the consumer platform (LGA 1155/1150), like my C226 for Haswell, which you can say that are rebranded consumer Chipsets with a slighty different feature set, and the one for LGA 2011/2011-R3, which is identical silicon to its X counterpart (C612 and X99 for Haswell-E, C602/604/606 and X79 for Sandy Bridge-E/Ivy Bridge-E). However, in that case, the X79/X99 is the most crippled of the two, since they disable the enterprise level features: http://ark.intel.com/compare/81761,81759
Besides, you will not find a Dual Socket X99 Motherboards, that's C612s segment, asserting its highendness.


Also, here is another example in a Datasheet: http://www.intel.com/content/dam/www/pu … asheet.pdf

On Page 49 and 50 you see mentions to VT-d and X79 NOT supporting it, reinforcing that it was supposed to be a Server feature. Moreover, some of the C6xx had an integrated SAS Controller, which X79 did not. A Motherboard manufacturer even managed to hack their way to include it in a X79 based product: http://www.legitreviews.com/ecs-enables … board_1833

Yes, I know that you can get VT-d working anyways. But there is a missing piece in the puzzle.

Last edited by zir_blazer (2015-04-25 12:12:00)

Offline

#4907 2015-04-25 15:10:59

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Googling around revealed that AER Errors are quite common when using Nvidia on X99 .

Also it is reported that using this kernel parameter makes the error go away :

pci=nommconf

Does this option affect the working of VFIO & GPU passthrough ?

Offline

#4908 2015-04-25 15:26:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

Googling around revealed that AER Errors are quite common when using Nvidia on X99 .

Also it is reported that using this kernel parameter makes the error go away :

pci=nommconf

Does this option affect the working of VFIO & GPU passthrough ?

mmconf is the mechanism to get to PCI extended configuration space.  AER is an extended capability, so disabling access to it disables AER.  nommconf is a rather large hammer though, why not just disable AER with pci=noaer?  nommconf won't prevent VFIO, but ACS is also an extended capability, so you may find more isolation issues using it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4909 2015-04-25 16:03:50

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yup . In fact , pci=nommconf results in Code 43 .

Will try pci=noaer now smile

EDIT : There are a couple of PCI-E options in BIOS that I enabled now . Hopefully they would make the issue go away . If not I will try pci=noaer .

EDIT 2 : When I launch both of my GT610 VM and GTX770 VM , I notice that GTX770 got ecap hiding message in dmesg :

vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

but there is nothing similar for the GT610 . Could this be a cause for the reset issues I'm having with GT610 ?

Last edited by Denso (2015-04-25 16:15:23)

Offline

#4910 2015-04-25 19:21:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

EDIT 2 : When I launch both of my GT610 VM and GTX770 VM , I notice that GTX770 got ecap hiding message in dmesg :

vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

but there is nothing similar for the GT610 . Could this be a cause for the reset issues I'm having with GT610 ?

That's just an informational message.  If you don't see the same on the GT610, it's because the card doesn't have that capability.  I can't imagine how it would contribute to the reset problem.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4911 2015-04-25 19:38:00

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks !

It seems that getting rid of it and get a 900 series GPU is the next move  smile

Offline

#4912 2015-04-26 08:37:00

scarface_the_fox
Member
Registered: 2015-04-26
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone

i have been for the past 4 days trying to install windows 7 using the method in the op's post and have the vga-passthru working and also passthru the sata controller sofar i have the vga passthru working as it is supposed to but with the sata controller everytime the vm resets the controller is no longer detected and the vm hangs just saying SeaBIOS (version 1.7.5-20140531_171129-lamiak) and i have to restart the entire pc any ideas

i have totally run out of ideas myself i have made sure that the sata controllers in bios are AHCI and that i even turned hotswap off.

the init script i use is as follows:
#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
        echo $line | grep ^# >/dev/null 2>&1 && continue
                vfiobind $line
done

sudo qemu-system-x86_64 -vga none -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
-boot menu=on \
-smp 4,sockets=1,cores=1,threads=1 \
-usb -usbdevice host:046d:c313 \
-device vfio-pci,host=02:00.0,bus=pcie.0,x-vga=on \
-device vfio-pci,host=02:00.1 \
-device vfio-pci,host=0a:00.0 \
-device virtio-scsi-pci,id=scsi \
-drive id=windcd,format=raw,media=cdrom,file=/home/cassandra/Downloads/en_windo$
-drive id=virtio,format=raw,media=cdrom,file=/home/cassandra/Downloads/virtio-w$
-net nic,model=virtio -net user \
-vga none \
-boot menu=on
exit 0

the pc specs are :
MOTHERBOARD : Asus sabertooth x79
Processor           : Intel Core-I7-3820 @ 3.60 (VT-d enabled)
ram                     : 16gb ddr3
HOST gfx            : Nvidia geforce 7950
guest GFX           : Nvidia geforce GTX 660
guest sata           : Asmedia ASM1062 Serial ATA controller
guest HDD          : hitachi 120gb sata (connected to eSATA)

Offline

#4913 2015-04-26 09:30:10

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

scarface_the_fox wrote:

Hello everyone

i have been for the past 4 days trying to install windows 7 using the method in the op's post and have the vga-passthru working and also passthru the sata controller sofar i have the vga passthru working as it is supposed to but with the sata controller everytime the vm resets the controller is no longer detected and the vm hangs just saying SeaBIOS (version 1.7.5-20140531_171129-lamiak) and i have to restart the entire pc any ideas

i have totally run out of ideas myself i have made sure that the sata controllers in bios are AHCI and that i even turned hotswap off.

the init script i use is as follows:
#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
        echo $line | grep ^# >/dev/null 2>&1 && continue
                vfiobind $line
done

sudo qemu-system-x86_64 -vga none -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
-boot menu=on \
-smp 4,sockets=1,cores=1,threads=1 \
-usb -usbdevice host:046d:c313 \
-device vfio-pci,host=02:00.0,bus=pcie.0,x-vga=on \
-device vfio-pci,host=02:00.1 \
-device vfio-pci,host=0a:00.0 \
-device virtio-scsi-pci,id=scsi \
-drive id=windcd,format=raw,media=cdrom,file=/home/cassandra/Downloads/en_windo$
-drive id=virtio,format=raw,media=cdrom,file=/home/cassandra/Downloads/virtio-w$
-net nic,model=virtio -net user \
-vga none \
-boot menu=on
exit 0

the pc specs are :
MOTHERBOARD : Asus sabertooth x79
Processor           : Intel Core-I7-3820 @ 3.60 (VT-d enabled)
ram                     : 16gb ddr3
HOST gfx            : Nvidia geforce 7950
guest GFX           : Nvidia geforce GTX 660
guest sata           : Asmedia ASM1062 Serial ATA controller
guest HDD          : hitachi 120gb sata (connected to eSATA)

Use virtio-blk or virtio-scsi instead of passingthrough the controller. It's working just fine.

Offline

#4914 2015-04-26 13:39:43

han5
Member
Registered: 2015-04-26
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm almost there, I guess, but now Windows doesn't seem to play along. Windows is starting to boot, though, but shortly after the startup screen (the one looking like this: http://upload.wikimedia.org/wikipedia/c … ooting.png ) appears, the VM (or at least video output) freezes (i.e. the dots stop spinning and nothing else is happening - no bluescreen).

Did anyone else had similar issues and could point me in the right direction?

-------- details --------

Windows installation was straight forward. After installing virtio drivers the hdd was found, the installer created partitions and copied files. I later mounted the data partition to check it's contents and it superficially looked like a common windows installation.

Once the installer was done, it rebooted. Then, according to the message in the following startup screen, it did some further "setting up devices". At the end of that process, it seemed like the VM hang already but after a few seconds it rebooted again. And now it hangs for real...

I'm running Ubuntu 15.04 3.16.0-36-generic on host and Windows 8.1 professional in the VM. When setting everything up I laregly followed http://ubuntuforums.org/showthread.php?t=2266916, supplemented with info from this thread.

The VM is launched with:
qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 6G \
-mem-path /dev/hugepages \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-usb -usbdevice host:046d:c52b \
-bios /home/user/vm/OVMF-pure-efi.fd \
-device virtio-scsi-pci,id=scsi0 \
-drive file=/dev/sdc,id=disk,format=raw,if=none -device scsi-hd,bus=scsi0.0,drive=disk \
-drive file=/home/user/iso/de_windows_8_1_x64_dvd_xxxxxxx.iso,id=isowindows,if=none -device scsi-cd,bus=scsi0.0,drive=isowindows \
-drive file=/home/user/iso/virtio-win-0.1-100.iso,id=isovirtio,if=none -device ide-cd,bus=ide.1,drive=isovirtio \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-boot order=cd,menu=on \
-k de

Hardware:
Xeon E5-2630v3
X99 board
ATI R5 230 (host)
nvidia GTX960 (passed through)
SSD solely for VM (sdc)
USB mouse/keyboard combo passed to VM

Offline

#4915 2015-04-26 20:06:09

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Bronek wrote:
aw wrote:

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Do you think that could help with reset of AMD W7100 ?

Dunno, try it.  That's a Tonga GPU, who knows, maybe the bug extends further than Bonaire/Hawaii.

is it independent from http://article.gmane.org/gmane.linux.kernel.pci/40663 , or should I patch my kernel 3.18.12 as well? Or perhaps try with kernel 4.0?

Last edited by Bronek (2015-04-26 20:06:45)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4916 2015-04-26 20:09:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:

is it independent from http://article.gmane.org/gmane.linux.kernel.pci/40663 , or should I patch my kernel 3.18.12 as well? Or perhaps try with kernel 4.0?

Not at all, it should be independent of kernel version.  I've also decided that the reset is not reliable enough for the kernel, so this is the patch proposed for QEMU: https://lists.gnu.org/archive/html/qemu … 03128.html

EDIT: Note that this version only enables the reset for specific device IDs.  If you're looking to try it on something else, you'll need to add the ID for your hardware to the list.

Last edited by aw (2015-04-26 20:11:30)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4917 2015-04-26 21:04:12

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm currently trying to get this working on Ubuntu 14.10 on kernel 3.19 and am having a lot of issues. When I try to boot it with qemu, a see a small black window pop up with:

compat_monitor0 console
QEMU 2.2.1 monitor - type 'help' for more information
(qemu)

and no output from the second monitor (my tv). I have also tried using Virtual Machine Manager, which, after passing through my GTX 760, gives me a virtual window on my main monitor. I installed windows 8.1 on it and saw that I had a code 43 on my GTX 760. I tried installing drivers for it on the guest machine (a few different releases back to 335.23) and none of them solved the issue.

Here's my execution script:

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
-smp 4,sockets=1,cores=1,threads=1 \
-bios /usr/share/seabios/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/carter/Downloads/EVGA.GTX760.4096.130607.rom \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1 \
-drive file=/home/carter/windows8.img,cache=writeback,if=none,id=drive0,aio=native \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/carter/Windows_8.1_Pro_X64_Activated.iso,id=iso_install,if=none \
-device scsi-cd,drive=iso_install \
-boot menu=on

exit 0

My output of dmesg | grep pci-stub:

[    0.514450] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.514462] pci-stub 0000:07:00.1: claimed by stub
[    0.514469] pci-stub: add 10DE:1187 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.514474] pci-stub 0000:07:00.0: claimed by stub

The contents of /etc/modules:

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

The contents of /etc/default/grub:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT="0"
#GRUB_HIDDEN_TIMEOUT="0"
GRUB_HIDDEN_TIMEOUT_QUIET="true"
GRUB_TIMEOUT="10"
GRUB_DISTRIBUTOR="`lsb_release -i -s 2> /dev/null || echo Debian`"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream i915.enable_hd_vgaarb=1"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL="console"

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE="640x480"

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID="true"

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

I have tried the ACS and i915 patches, neither of which helped. However, I'm not entirely sure I did it right because I haven't found a clear quide on how to apply them. I tried it the way done in this guide.

My specs are:
Gigabyte H77 motherboard (with vt-d turned on)
NVIDIA GTX 560 for the host
EVGA GTX 760 for the guest
Intel i5 Quad Core

Would anybody care to help me out? smile

Offline

#4918 2015-04-26 21:07:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anybody actually had success following the Ubuntu guide?  Seems like a flood of people following whatever misinformation it contains and landing here.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4919 2015-04-26 21:17:42

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

m5a97 pro
fx-8320
hd 6850
8 gb

qemu-2.2.1
seabios-1.8.1

windows works, no problem to launch some games or benchmarks, but...
a lot of hangs during startup (looks like 1 in 10 it can successfully, other times - just logo, or black screen after it), nothing in dmesg or /var/log that could be helpful. If I look in virt-manager it shows CPU usage as a straight line from the moment of hang.

Could the change of the motherboard helps me? What about sabertooth 990fx r2.0? Or better try Intel?

<domain type='kvm'>
  <name>windows</name>
  <uuid>80831171-02e1-4c05-be41-011401d13184</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <topology sockets='1' cores='6' threads='1'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='skinit'/>
    <feature policy='require' name='tce'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='topoext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='ibs'/>
    <feature policy='require' name='wdt'/>
    <feature policy='require' name='extapic'/>
    <feature policy='require' name='osvw'/>
    <feature policy='require' name='nodeid_msr'/>
    <feature policy='require' name='perfctr_nb'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='lwp'/>
    <feature policy='require' name='invtsc'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-wrapper</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='directsync' io='threads'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='directsync' io='threads'/>
      <source file='/home/walkindude/driveD.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/Downloads/en_windows_10_pro_10061_x64_dvd.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/virtio-win-0.1-100.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:c3:bc:db'/>
      <source network='network'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ac97'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x09da'/>
        <product id='0x0260'/>
      </source>
      <boot order='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x09da'/>
        <product id='0xf643'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Last edited by walkindude (2015-04-26 21:31:24)

Offline

#4920 2015-04-27 01:47:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@carterghill

rombar=0,romfile=/home/carter/Downloads/EVGA.GTX760.4096.130607.rom

Do not pass Go, do not collect $200.  This is a bogus combination of arguments that causes QEMU to load the ROM into a legacy ROM space, disassociating it from the device.  I'd remove both arguments and only add back romfile if you need it.

You don't need the i915 patch since you don't have Intel host graphics, which means you also don't need this:  i915.enable_hd_vgaarb=1

Chances are you also don't need this: vfio_iommu_type1.allow_unsafe_interrupts=1

Personally, given your guest and hardware combination, I'd use a standard 440FX machine, OVMF guest firmware, and you should be able to manage it with libvirt.

You might even be able to move one of your cards to a PCH root port (probably listed as a PCIe 2.0 port) and avoid needing the ACS override patch.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4921 2015-04-27 01:57:03

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@han5

Try installing TightVNC server in the guest so you can get to the desktop w/o the display and determine if the guest is running or really hung.  You also shouldn't need x-vga=on since you're using OVMF (and you're multifunction=on option isn't really doing anything since you're not specifying the addr= to put a secondary function on the device).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4922 2015-04-27 02:06:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@walkindude

It looks like you're running an 8G guest on an 8G host.  Device assignment VMs cannot be overcommitted.  See if you still have problems reducing the VM to 6G or 7G.  The host could be swapping like crazy and timing glitches resulting from that could cause the problems you're seeing.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4923 2015-04-27 04:39:57

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
sorry to be not clear mysefl:
Host: 16G
Guest: 8G

Also in /etc/sysctl.d/99-sysctl.conf:

vm.nr_hugepages = 4400
hugepage size: 2048 kB

Tried to set it to 5500 for example.
After boot:

/proc/meminfo

HugePages_Total: 5500
HugePages_Free: 1404
HugePages_Rsvd: 0
HugePages_Surp: 0

This remained the same after reboots. For now refreshed windows boots as expected, but for how long.

Last edited by walkindude (2015-04-27 05:18:48)

Offline

#4924 2015-04-27 08:30:15

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Bronek wrote:

is it independent from http://article.gmane.org/gmane.linux.kernel.pci/40663 , or should I patch my kernel 3.18.12 as well? Or perhaps try with kernel 4.0?

Not at all, it should be independent of kernel version.  I've also decided that the reset is not reliable enough for the kernel, so this is the patch proposed for QEMU: https://lists.gnu.org/archive/html/qemu … 03128.html

EDIT: Note that this version only enables the reset for specific device IDs.  If you're looking to try it on something else, you'll need to add the ID for your hardware to the list.

Thanks for heads up, I found this:

+ switch (vendor) { + case 0x1002: + switch (device) {

I will amend if necessary for W7100 (don't remember its device code ATM) and will let you know if it worked. FWIW, http://article.gmane.org/gmane.linux.kernel.pci/40663 patched on kernel 3.18.11 (with qemu 2.2.1) have not fixed my problems with W7100 reset, so I'm keen to try something else.

I wouldn't have this problem if only R9 290X fitted in single PCIe slot (and one bracket, too)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4925 2015-04-27 10:52:41

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nope, just turned on my PC, and on the first VM boot got black screen
but

HugePages_Total: 5500
HugePages_Free: 1404
HugePages_Rsvd: 0
HugePages_Surp: 0

free -m

total: 15977
used: 11962
free 2550
shared: 86
buff/cache: 1466
available: 3608

Could it be errors in DSDT? Some time ago I've tried to do https://wiki.archlinux.org/index.php/DSDT, out of scientific interest wink Had fixed errors and warnings, but then during my first tries on passthrough, didn't get after seabios - black screen. Then I tried to do without it. System boots for some time, like 4-8 times, and then shit happens. Didn't install to much: just Visual Studio, Qt, Steam...

Also in dmesg (not always see that, maybe it's a culprit):

[    5.173608] kvm: Nested Virtualization enabled
[    5.173615] kvm: Nested Paging enabled <- tried with and without, nothing seems changed
[   83.226853] kvm: zapping shadow pages for mmio generation wraparound
[   92.263251] kvm [3109]: vcpu3 unimplemented perfctr wrmsr: 0xc0010004 data 0xffffffffffffd8f0
[   92.263255] kvm [3109]: vcpu4 unimplemented perfctr wrmsr: 0xc0010004 data 0xffffffffffffd8f0
[   92.263256] kvm [3109]: vcpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[   92.263259] kvm [3109]: vcpu5 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[   92.263260] kvm [3109]: vcpu1 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[   92.263262] kvm [3109]: vcpu4 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[   92.263296] kvm [3109]: vcpu3 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[  451.669332] kvm [3109]: vcpu3 unimplemented perfctr wrmsr: 0xc0010004 data 0xffffffffffffd8f0
[  451.669335] kvm [3109]: vcpu4 unimplemented perfctr wrmsr: 0xc0010004 data 0xffffffffffffd8f0
[  451.669337] kvm [3109]: vcpu0 unimplemented perfctr wrmsr: 0xc0010004 data 0xffffffffffffd8f0
[  451.669341] kvm [3109]: vcpu2 unimplemented perfctr wrmsr: 0xc0010004 data 0xffffffffffffd8f0
[  451.669345] kvm [3109]: vcpu5 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[  451.669347] kvm [3109]: vcpu4 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[  451.669349] kvm [3109]: vcpu0 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[  451.669352] kvm [3109]: vcpu2 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076
[  451.669361] kvm [3109]: vcpu3 unimplemented perfctr wrmsr: 0xc0010000 data 0x530076

Last edited by walkindude (2015-04-27 12:43:16)

Offline

#4926 2015-04-27 16:44:41

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey, so I have been poking around the thread trying to get this working, is the first page's guide still considered correct? I have seen mention of the linux-vfio aur package, but no other posts about how it works, and what needs to be configured on top of it. I guess my confusion revolves specifically around the pci-vfio bind script as I don't remember having to use that or that it was a dead end last time I got this working.

I have gotten this working before, but it's been at least 6 months and a hundred pages since I last read through the thread.

Offline

#4927 2015-04-27 16:57:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would not recommend the vfio bind script to anyone.  The better option is to use libvirt to manage the VM.  If you're using OVMF for your guest (preferred), you can manage everything natively within the libvirt xml.  If you need to go the seabios route due to old guest or old hardware, use a wrapper script to add the x-vga option to the vfio-pci device in the resulting QEMU commandline.  If you insist on using q35 (not recommended except for Linux guests), you can also use the wrapper to move the GPU to the root complex or create and use root ports.  This usage is really only a workaround until libvirt gets sane support for q35 topology though.  Using <qemu:arg> options in the xml is generally not recommended or necessary and causes significant headaches if used to specify the assigned device since it's effectively hidden from libvirt.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4928 2015-04-27 17:28:55

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the fast reply!

Ok, so I am planning on using OVMF(I have used it in the past in conjunction with libvirt and know the gotchas that apply to nvidia cards) but before I get there, is there anything with the kernel that I need to do? I have downloaded and compiled linux-vfio, and am about to add a boot configuration for it, but this is where my memory fails me. I don't remember if this is something I had to do to get OVMF to attach to the video card correctly, or if it was something I did that turned out to not apply. And is the required boot option just to enable iommu in the boot parameters on a stock kernel?

Setup:
Intel 4790k(Supports VT-d)
Intel IGD
Nvidia GTX760(Supports EFI)

Offline

#4929 2015-04-27 17:45:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Punkbob

Given your hardware and intended (non-VGA) usage, the only potential patch you'd need vs stock is the ACS override.  You can probably avoid that patch as well by being careful not to share processor root ports with other devices or worst case, moving the Nvidia card to a PCH root port.  I'm sorry, I don't know what the linux-vfio package is, I don't run Arch myself.  Personally I'd try to use the stock kernel for your configuration.  I run a GTX750 guest (UEFI) with host IGD on stock Fedora.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4930 2015-04-27 21:27:30

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Bronek wrote:

is it independent from http://article.gmane.org/gmane.linux.kernel.pci/40663 , or should I patch my kernel 3.18.12 as well? Or perhaps try with kernel 4.0?

Not at all, it should be independent of kernel version.  I've also decided that the reset is not reliable enough for the kernel, so this is the patch proposed for QEMU: https://lists.gnu.org/archive/html/qemu … 03128.html

EDIT: Note that this version only enables the reset for specific device IDs.  If you're looking to try it on something else, you'll need to add the ID for your hardware to the list.

@aw

I added my device to your patch and it seems to work, I was able to restart guest (without crashing host) although tried only once. More testing needed because after first restart I noticed brief artifacts on screen of this card, until driver was loaded.

More details of my card, in case you are interested

root@gdansk ~ # lspci -nnv -s 0000:82:00.0 82:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga PRO GL [FirePro W7100] [1002:692b] (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:030c] Flags: fast devsel, IRQ 11 Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at f0000000 (64-bit, prefetchable) [disabled] [size=2M] I/O ports at f000 [disabled] [size=256] Memory at fbd00000 (32-bit, non-prefetchable) [disabled] [size=256K] Expansion ROM at fbd40000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [200] #15 Capabilities: [270] #19 Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] #13 Capabilities: [2d0] #1b Capabilities: [328] Alternative Routing-ID Interpretation (ARI) root@gdansk ~ # lspci -nnv -s 0000:82:00.1 82:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aad8] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aad8] Flags: bus master, fast devsel, latency 0, IRQ 10 Memory at fbd60000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [328] Alternative Routing-ID Interpretation (ARI) Kernel modules: snd_hda_intel

Here is change of your patch:

+ case 0x67B9: + case 0x67BA: + case 0x67BE: + /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Tonga */ + case 0x692B: /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Tonga PRO GL [Radeon W7100] */ + vdev->resetfn = vfio_radeon_reset; + break; + } + break; + } +}

FWIW, I noticed that I am able to cleanly (probably - didn't try to build yet) apply your patches from mainline kernel on 3.18.12 which I'm using, would you recommend or discourage applying any of those? Assuming I am not merging any other patches from mainline

bronek@gdansk ~/Sources/linux-stable (git)-[v3.18.12] % git log --oneline --reverse -n 21 43f497a Linux 3.18.12 8659c14 PCI: Add flag for devices that don't reset on D3hot->D0 transition d3aa82b PCI: Mark AMD/ATI VGA devices that don't reset on D3hot->D0 transition 92e46be vfio/type1: DMA unmap chunking def9cee vfio/type1: Chunk contiguous reserved/invalid page mappings f7fd50e vfio/type1: Add conditional rescheduling 87498ef vfio: Add device tracking during unbind afe7fd7 vfio: Tie IOMMU group reference to vfio group e22eb3a vfio: Add and use device request op for vfio bus drivers 1d8e0e3 vfio-pci: Generalize setup of simple eventfds e0919875 vfio-pci: Add device request interface 65c825f iommu/vt-d: Detach domain *only* from attached iommus 05b36e6 PCI: Add ACS quirks for Intel 1G NICs 6988212 iommu/vt-d: Remove unused variable 9971924 vgaarb: Stub vga_set_legacy_decoding() 7f56323 vfio-pci: Add module option to disable VGA region access b7316ee vfio-pci: Add VGA arbiter client b3d9280 vfio-pci: Allow PCI IDs to be specified as module options 0802afb vfio-pci: Remove warning if try-reset fails 42ee7c2 vfio-pci: Move idle devices to D3hot power state faadced vfio-pci: Fix use after free

Thank you for your hard work on vfio and PCI subsystem :)

Last edited by Bronek (2015-04-27 21:30:28)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4931 2015-04-27 23:11:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

I would not recommend the vfio bind script to anyone.

Thanks for reminding me to remove it from my systemd and system overall - i've moved to libvirt ever since and totally forgot about that script.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4932 2015-04-28 00:16:08

blakx
Member
Registered: 2015-04-27
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm interested in building a new system for this, with the Code 43 for Nvidia issue and the Radeon Hawaii/Bonaire reset issue, what is the recommended performant card to go with? Wait for the next line of Radeon cards?

Last edited by blakx (2015-04-28 00:16:26)

Offline

#4933 2015-04-28 00:29:17

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the help, just one more quick question:

Anyone have experience with getting code 10 errors on the HDMI driver for an Nvidia card?

Edit: NVM, just fixed it.

Last edited by Punkbob (2015-04-28 00:30:48)

Offline

#4934 2015-04-28 01:55:54

mqddb
Member
Registered: 2015-04-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
mqddb wrote:

Hello everyone:
      Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.

No, an assigned GPU does not support any of the -display methods.  Remote displays with GPU assignment requires guest-based solutions.

@aw

    If any solutions to use spice remote display when an GPU assignment. I am thinking cp the assignment GPU's FB to qxl vga, if windows guest support two vga working at the same time? If you want add this VGA-Passthrough feature to ovirt or openstack, this spice problem must be resolve, right? Do you have any plan to do this?

Offline

#4935 2015-04-28 02:23:55

Linuxlookup
Member
Registered: 2015-04-27
Posts: 2
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Has anybody actually had success following the Ubuntu guide?  Seems like a flood of people following whatever misinformation it contains and landing here.

I've had success with Ubuntu. Though I did not follow any guide, my configurations are based on documentation and general experiences from users in various forums. I'm planning to evaluate other hardware configurations and document my method in a HowTo article once satisfied with the results. For those interested in seeing Ubuntu in action, recently I published a video which demonstrates Windows Gaming with QEMU/KVM on Ubuntu Linux.

Offline

#4936 2015-04-28 03:14:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Linuxlookup wrote:
aw wrote:

Has anybody actually had success following the Ubuntu guide?  Seems like a flood of people following whatever misinformation it contains and landing here.

I've had success with Ubuntu. Though I did not follow any guide, my configurations are based on documentation and general experiences from users in various forums. I'm planning to evaluate other hardware configurations and document my method in a HowTo article once satisfied with the results. For those interested in seeing Ubuntu in action, recently I published a video which demonstrates Windows Gaming with QEMU/KVM on Ubuntu Linux.

R9 280 also has reset problems?  That ought to be Tahiti based.  Have you tried updating the Bonaire/Hawaii patch as Bronek did to add your ID?  https://lists.gnu.org/archive/html/qemu … 03128.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4937 2015-04-28 03:16:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mqddb wrote:
aw wrote:
mqddb wrote:

Hello everyone:
      Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.

No, an assigned GPU does not support any of the -display methods.  Remote displays with GPU assignment requires guest-based solutions.

@aw

    If any solutions to use spice remote display when an GPU assignment. I am thinking cp the assignment GPU's FB to qxl vga, if windows guest support two vga working at the same time? If you want add this VGA-Passthrough feature to ovirt or openstack, this spice problem must be resolve, right? Do you have any plan to do this?

Nope.  Scraping the framebuffer requires knowing where the framebuffer lives, which requires far more knowledge of proprietary drivers than I'm interested in.  There are client based solutions for this, NICE DCV, HP RGS, etc...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4938 2015-04-28 03:18:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

List looks like a pretty complete sync to v4.1-rc1.  There's one more fix in the queue for e0919875, it's been posted, but not yet pushed.  I'll try to get it in for rc2.  Please update with further testing on Tonga.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4939 2015-04-28 09:04:17

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blakx wrote:

I'm interested in building a new system for this, with the Code 43 for Nvidia issue and the Radeon Hawaii/Bonaire reset issue, what is the recommended performant card to go with? Wait for the next line of Radeon cards?

Radeon R9 290X (more specifically, Saphire Tri-X UEFI enabled) worked very well for me, but now I'm running two virtual machines and need space for 2 GPUs and a bunch of other cards, so it was simply too large. It is slightly larger than two slots, so it had to go, for only this reason. I do not have problems with AMD FirePro V series but they are quite expensive for what they do, similarly more expensive model of NVidia Quaddro should work for you since they are "enabled" for GPU pass-through (e.g. K2200 , K4200 etc) but again they are very expensive for what they do. I'd guess Radeon R9 should work for you and is economical choice, but this is also a question of lucky/good choice of vendor and model.

Last edited by Bronek (2015-04-28 09:05:24)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4940 2015-04-28 19:17:32

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Bronek

List looks like a pretty complete sync to v4.1-rc1.  There's one more fix in the queue for e0919875, it's been posted, but not yet pushed.  I'll try to get it in for rc2.  Please update with further testing on Tonga.


Something is not right, but I suspect it's not with qemu but vfio (or something else in kernel). Occasionally when I shutdown VM guest and then start it up again, it seems like its trying to use wrong PCI device at the moment when driver gets loaded. Not a GPU card, something entirely different. At first Windows start appears as normal, then it blanks and a cursor appears on black background (which is normal for short time when driver starts), at which moment "dmesg -w" starts showing errors from a different PCIe card and its driver (slot 84:00.0 is Intel P3700, driver nvme) and Windows guest remains with blank screen.

[80149.889741] dmar: DRHD: handling fault status reg 2 [80149.894627] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr ffe0e000 DMAR:[fault reason 06] PTE Read access is not set [80149.908554] dmar: DRHD: handling fault status reg 102 [80149.913614] dmar: DMAR:[DMA Write] Request device [84:00.0] fault addr ffff9000 DMAR:[fault reason 05] PTE Write access is not set [80150.052146] usb 3-1: reset low-speed USB device number 2 using xhci_hcd [80150.340244] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4600 [80150.340251] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4540 [80150.340259] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [80150.340265] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [80150.485412] dmar: DRHD: handling fault status reg 202 [80150.490475] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fff24000 DMAR:[fault reason 06] PTE Read access is not set [80150.622368] usb 3-1: reset low-speed USB device number 2 using xhci_hcd [80150.910482] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4600 [80150.910489] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4540 [80150.910497] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [80150.910502] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [80150.966996] dmar: DRHD: handling fault status reg 302 [80150.972052] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr ffeca000 DMAR:[fault reason 06] PTE Read access is not set [80151.560200] dmar: DRHD: handling fault status reg 402 [80151.565259] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fffca000 DMAR:[fault reason 06] PTE Read access is not set [80155.227534] dmar: DRHD: handling fault status reg 502 [80155.232595] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fffad000 DMAR:[fault reason 06] PTE Read access is not set [80155.246330] dmar: DRHD: handling fault status reg 602 [80155.251410] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fff02000 DMAR:[fault reason 06] PTE Read access is not set [80170.201208] dmar: DRHD: handling fault status reg 702 [80170.206278] dmar: DMAR:[DMA Write] Request device [84:00.0] fault addr ffff9000 DMAR:[fault reason 05] PTE Write access is not set [80180.114179] nvme 0000:84:00.0: Aborting I/O 0 QID 1 [80180.114186] nvme 0000:84:00.0: Aborting I/O 1 QID 1 [80180.114194] nvme 0000:84:00.0: Aborting I/O 0 QID 16 [80180.114197] nvme 0000:84:00.0: Aborting I/O 1 QID 16 [80180.114279] dmar: DRHD: handling fault status reg 2 [80180.119169] dmar: DMAR:[DMA Write] Request device [84:00.0] fault addr fffff000 DMAR:[fault reason 05] PTE Write access is not set

FWIW my current kernel does not any of your newer patches beyond 3.18.12

Last edited by Bronek (2015-04-28 19:24:19)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4941 2015-04-28 19:42:13

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

Are you using the ACS override patch?  What's the topology (lspci -tv)?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4942 2015-04-28 23:33:27

Linuxlookup
Member
Registered: 2015-04-27
Posts: 2
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Linuxlookup wrote:
aw wrote:

Has anybody actually had success following the Ubuntu guide?  Seems like a flood of people following whatever misinformation it contains and landing here.

I've had success with Ubuntu. Though I did not follow any guide, my configurations are based on documentation and general experiences from users in various forums. I'm planning to evaluate other hardware configurations and document my method in a HowTo article once satisfied with the results. For those interested in seeing Ubuntu in action, recently I published a video which demonstrates Windows Gaming with QEMU/KVM on Ubuntu Linux.

R9 280 also has reset problems?  That ought to be Tahiti based.  Have you tried updating the Bonaire/Hawaii patch as Bronek did to add your ID?  https://lists.gnu.org/archive/html/qemu … 03128.html

Yes, I too have experienced a reset issue with AMD R9 280. Few months back I saw your Qemu-devel post from 12/02/14 and experimented with information provided. Unsuccessful, I haven't pursued any further solution myself.

Offline

#4943 2015-04-29 12:26:00

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have tried several more things trying to fix this freezing/bsod issue. Somewhere along the line it seems it's changed so that I no longer get freezes, just BSOD. At one point I did get a host freeze, unsure if this was caused by the VM or not.

  • Made a new clean UEFI install of Windows 7, GPT on LVM on LUKS on mdadm raid1. BSOD now reboots the VM, likely the default behavior for windows. Event viewer says code is 0x116.

  • Changed to kvm-pci-assign. Issue still persisted, changed back to vfio-pci after.

  • Changed some settings to match closer to Duelist's posted configuration.

    root=UUID=4f2e65ef-ef21-4628-bbd7-219d7855732b rw quiet cryptdevice=UUID=17b9c41f-ab59-4a68-a5ec-b7b3e2776bf6:calue pci-stub.ids=1002:67b1,1002:aac8 iommu=pt acpi_enforce_resources=lax
    options vfio_iommu_type1 disable_hugepages=1 options kvm-amd npt=1 nested=0
    #!/bin/sh export QEMU_AUDIO_DRV=sdl export QEMU_AUDIO_DAC_FIXED_SETTINGS=1 export QEMU_AUDIO_DAC_FIXED_FREQ=44100 export QEMU_AUDIO_DAC_FIXED_FMT=S16 export QEMU_SDL_SAMPLES=1024 qemu-system-x86_64 \ -machine accel=kvm \ -machine pc-i440fx-2.1,accel=kvm,usb=off \ -cpu Opteron_G5,+bmi1,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+tce,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme \ -drive file=OVMF-newinstall.fd,format=raw,if=pflash \ -drive file=/dev/mapper/urw-win,format=raw,if=ide \ -m 4096 \ -realtime mlock=on \ -smp 3 \ -no-user-config \ -nodefaults \ -rtc base=localtime,driftfix=slew \ -global kvm-pit.lost_tick_policy=discard \ -no-hpet \ -global PIIX4_PM.disable_s3=1 \ -global PIIX4_PM.disable_s4=1 \ -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \ -device usb-host,id=usbkbdms,bus=usb.0,vendorid=0x13ba,productid=0x0018 \ -netdev user,id=net \ -device rtl8139,netdev=net,bus=pci.0,addr=0x6 \ -device vfio-pci,host=01:00.0,bus=pci.0,addr=0x8 \ -device vfio-pci,host=01:00.1,bus=pci.0,addr=0x9 \ -vga none \ -soundhw hda \ $*

    (I did manage to mostly fix my audio issues by using sdl instead of alsa, and changing asound.conf to use 44100 Hz for dmix.)
    Issue still persists.

  • Removed Radeon 9250 PCI card from motherboard, wasn't using it anyway. Didn't fix.

  • Removed "blacklist radeon" from modprobe.d. Guest card already stubbed through pci-stub.

  • Ran windows update on guest. Didn't fix issue.

  • Started running VM from remote system over SSH with -nographic. Didn't fix issue.

  • Removed host's card (GeForce 9800 GT), no longer running X on host, running VM over ssh. Didn't fix issue.

I am going now to move the guest-assigned card to another PCI-e x16 slot and attempt again, not expecting any success. I have a powered x1 to x16 riser which I had been running this card on previously on an older motherboard, I may try that later, not expecting any success.

I don't really understand why this is an issue. I can even shutdown the guest, unbind the card from vfio and bind it to radeon, then log in and launch X on the host and use it without an issue. I'm thinking this could be a bug in the guest radeon drivers. Perhaps I might try some older versions of the driver. I'm running out of things left to test. I could run linux within the VM and attempt to get anything on the guest's dmesg output, but I don't know.

It's really frustrating being this close to success and not being able to do anything about it.

Offline

#4944 2015-04-29 13:45:01

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You have a Quadro 2000, just assigned it as a secondary device and be done with it.  https://access.redhat.com/documentation … e-GPU.html

If it doesn't work, update kernel and/or QEMU.  Note that only K-series Quadros are actually supported, but Fermi are likely to work.  Quadro cards do not work well when configured as the primary display for a VM.

I am trying to build a machine where the host is a server so basically no UI except SSH. Anyway, I was pointing to the fact that there is a bug somewhere that is causing the host to crash/freeze. Given the advanced experimentation going on, not really surprised, just wondering on what is the best action, if there is a bug tracker, etc.

I am planning to get a UEFI bios for my card, PNY technical support claims that they have a version of the BIOS with UEFI support, so trying that route. Thank you very much, I am impressed by how much of this stuff works anyway!!

Offline

#4945 2015-04-29 15:21:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paperinick wrote:
aw wrote:

You have a Quadro 2000, just assigned it as a secondary device and be done with it.  https://access.redhat.com/documentation … e-GPU.html

If it doesn't work, update kernel and/or QEMU.  Note that only K-series Quadros are actually supported, but Fermi are likely to work.  Quadro cards do not work well when configured as the primary display for a VM.

I am trying to build a machine where the host is a server so basically no UI except SSH. Anyway, I was pointing to the fact that there is a bug somewhere that is causing the host to crash/freeze. Given the advanced experimentation going on, not really surprised, just wondering on what is the best action, if there is a bug tracker, etc.

Your kernel is too old to be interesting for upstream bugs.  You can see if the problem still occurs on current upstream or file a bug with your distro.

I am planning to get a UEFI bios for my card, PNY technical support claims that they have a version of the BIOS with UEFI support, so trying that route. Thank you very much, I am impressed by how much of this stuff works anyway!!

That doesn't solve the problem that Quadro cards don't generally behave as VM primary displays, if you use the secondary display approach, it doesn't matter whether you have a legacy or UEFI BIOS for the card.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4946 2015-04-29 17:30:36

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That doesn't solve the problem that Quadro cards don't generally behave as VM primary displays

Where can I find more info about this? If it boots, runs and handle Premier 6 CUDA acceleration, I am happy. Investigating about the last part, but I am interested all there is to know about this. Thanks a lot!

Offline

#4947 2015-04-29 18:14:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

paperinick wrote:

That doesn't solve the problem that Quadro cards don't generally behave as VM primary displays

Where can I find more info about this? If it boots, runs and handle Premier 6 CUDA acceleration, I am happy. Investigating about the last part, but I am interested all there is to know about this. Thanks a lot!

Try it, you'll see.  Given that you have the hardware to run in a mode that Nvidia actually claims to support, I'm not sure why you wouldn't want to run in that mode.  Some people just like to make things more difficult...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4948 2015-04-29 18:22:17

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Bronek

Are you using the ACS override patch?  What's the topology (lspci -tv)?

No ACS override patch here, only patch on top of 3.18.12 I have at this moment is from "git diff c143c2333c4..3ccb354d641" (VFS change in kernel mainline), which is a workaround for ZFS issue https://github.com/zfsonlinux/zfs/issues/3030

Here is topology (with list of vfio devices):

root@gdansk ~ # cat /proc/cmdline BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw spl.spl_hostid=0xa8c08502 console=ttyS0,115200N8R nomodeset udev.children-max=32 edac_core.edac_mc_panic_on_ue=1 intel_iommu=on pci-stub.ids=8086:1d2d,8086:1d20,8086:1d26,1912:0014,1002:692b,1002:aad8,1002:6707,1002:aa80 isolcpus=2,3,4,5,6,7,8,9,10,11,14,15,16,17,18,19,20,21,22,23 initrd=../initramfs-linux.img root@gdansk ~ # lspci -nnvt -+-[0000:ff]-+-08.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [8086:0e80] | +-09.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 [8086:0e90] | +-0a.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 [8086:0ec0] | +-0a.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 [8086:0ec1] | +-0a.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 [8086:0ec2] | +-0a.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 [8086:0ec3] | +-0b.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1e] | +-0b.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1f] | +-0c.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee0] | +-0c.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee2] | +-0c.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee4] | +-0d.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee1] | +-0d.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee3] | +-0d.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee5] | +-0e.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0ea0] | +-0e.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0e30] | +-0f.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers [8086:0ea8] | +-0f.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers [8086:0e71] | +-0f.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eaa] | +-0f.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eab] | +-0f.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eac] | +-0f.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0ead] | +-10.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 [8086:0eb0] | +-10.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 [8086:0eb1] | +-10.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 [8086:0eb2] | +-10.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 [8086:0eb3] | +-10.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 [8086:0eb4] | +-10.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 [8086:0eb5] | +-10.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 [8086:0eb6] | +-10.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 [8086:0eb7] | +-13.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e1d] | +-13.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e34] | +-13.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers [8086:0e81] | +-13.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring [8086:0e36] | +-16.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder [8086:0ec8] | +-16.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0ec9] | \-16.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0eca] +-[0000:80]-+-01.0-[81]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] | +-02.0-[82]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Tonga PRO GL [FirePro W7100] [1002:692b] | | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aad8] | +-03.0-[83]----00.0 LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] [1000:005b] | +-03.2-[84]----00.0 Intel Corporation PCIe Data Center SSD [8086:0953] | +-04.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 [8086:0e20] | +-04.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 [8086:0e21] | +-04.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 [8086:0e22] | +-04.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 [8086:0e23] | +-04.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 [8086:0e24] | +-04.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 [8086:0e25] | +-04.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 [8086:0e26] | +-04.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 [8086:0e27] | +-05.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc [8086:0e28] | +-05.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS [8086:0e2a] | \-05.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC [8086:0e2c] +-[0000:7f]-+-08.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [8086:0e80] | +-09.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 [8086:0e90] | +-0a.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 [8086:0ec0] | +-0a.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 [8086:0ec1] | +-0a.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 [8086:0ec2] | +-0a.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 [8086:0ec3] | +-0b.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1e] | +-0b.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1f] | +-0c.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee0] | +-0c.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee2] | +-0c.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee4] | +-0d.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee1] | +-0d.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee3] | +-0d.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee5] | +-0e.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0ea0] | +-0e.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0e30] | +-0f.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers [8086:0ea8] | +-0f.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers [8086:0e71] | +-0f.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eaa] | +-0f.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eab] | +-0f.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eac] | +-0f.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0ead] | +-10.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 [8086:0eb0] | +-10.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 [8086:0eb1] | +-10.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 [8086:0eb2] | +-10.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 [8086:0eb3] | +-10.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 [8086:0eb4] | +-10.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 [8086:0eb5] | +-10.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 [8086:0eb6] | +-10.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 [8086:0eb7] | +-13.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e1d] | +-13.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e34] | +-13.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers [8086:0e81] | +-13.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring [8086:0e36] | +-16.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder [8086:0ec8] | +-16.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0ec9] | \-16.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0eca] \-[0000:00]-+-00.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 [8086:0e00] +-01.0-[01]----00.0 LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0086] +-02.0-[02]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] +-03.0-[03]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cayman LE GL [FirePro V5900] [1002:6707] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80] +-04.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 [8086:0e20] +-04.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 [8086:0e21] +-04.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 [8086:0e22] +-04.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 [8086:0e23] +-04.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 [8086:0e24] +-04.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 [8086:0e25] +-04.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 [8086:0e26] +-04.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 [8086:0e27] +-05.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc [8086:0e28] +-05.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS [8086:0e2a] +-05.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC [8086:0e2c] +-11.0-[04]----00.0 Intel Corporation C602 chipset 4-Port SATA Storage Control Unit [8086:1d6b] +-16.0 Intel Corporation C600/X79 series chipset MEI Controller #1 [8086:1d3a] +-16.1 Intel Corporation C600/X79 series chipset MEI Controller #2 [8086:1d3b] +-1a.0 Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 [8086:1d2d] +-1b.0 Intel Corporation C600/X79 series chipset High Definition Audio Controller [8086:1d20] +-1c.0-[05-06]--+-00.0 Intel Corporation I350 Gigabit Network Connection [8086:1521] | \-00.1 Intel Corporation I350 Gigabit Network Connection [8086:1521] +-1c.5-[07]----00.0 NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] +-1c.6-[08]----00.0 NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] +-1d.0 Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 [8086:1d26] +-1e.0-[09]-- +-1f.0 Intel Corporation C600/X79 series chipset LPC Controller [8086:1d41] +-1f.2 Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller [8086:1d02] +-1f.3 Intel Corporation C600/X79 series chipset SMBus Host Controller [8086:1d22] \-1f.6 Intel Corporation C600/X79 series chipset Thermal Management Controller [8086:1d24]

I think I had these occasional problems with 84:00 before, but I do not remember whether these happened when I was using other graphic card (I also have FirePro V4900) or not.

Last edited by Bronek (2015-04-29 18:24:03)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4949 2015-04-30 01:21:02

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everyone! I have been trying to set this up on my system for hours now and have came to a standstill, lol.

After I add the kernel parameter

"pci-stub.ids=10de:13c0,10de:0fbb" 

my system fails to get passed the "Reached target Graphical Interface" tab while booting. If I remove that parameter it boots fine.
Also when running,

"qemu-system-x86_64 -enable-kvm -m 8000 -cpu host,kvm=off -vga none -device vfio-pci,host=07:00.0 -smp 4,sockets=1,cores=4,threads=2 -hda /games/win.cow"

I get the error

"qemu-system-x86_64: -device vfio-pci,host=07:00.0: vfio: error opening /dev/vfio/23: No such file or directory qemu-system-x86_64: -device vfio-pci,host=07:00.0: vfio: failed to get group 23 qemu-system-x86_64: -device vfio-pci,host=07:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=07:00.0: Device 'vfio-pci' could not be initialized"

I realize that the error for qemu is probably from the system not booting with the pci-stub kernel parameter, but I can't boot while its on.

Last edited by Zerqz (2015-04-30 01:21:40)

Offline

#4950 2015-04-30 01:37:11

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zerqz wrote:

Hey everyone! I have been trying to set this up on my system for hours now and have came to a standstill, lol.

After I add the kernel parameter

"pci-stub.ids=10de:13c0,10de:0fbb" 

my system fails to get passed the "Reached target Graphical Interface" tab while booting. If I remove that parameter it boots fine.

Do you have.. a second GPU that should be left for host?
What distrib, kernel and hardware are you using? You sure that IOMMU is on?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4951 2015-04-30 01:48:36

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Zerqz wrote:

Hey everyone! I have been trying to set this up on my system for hours now and have came to a standstill, lol.

After I add the kernel parameter

"pci-stub.ids=10de:13c0,10de:0fbb" 

my system fails to get passed the "Reached target Graphical Interface" tab while booting. If I remove that parameter it boots fine.

Do you have.. a second GPU that should be left for host?
What distrib, kernel and hardware are you using? You sure that IOMMU is on?


I have 2 gtx980's the first in 01:00.0 and the second in 07:00.0.
and a x64 AMD cpu

Running arch, kernel 4.0.1-1, and iommu=on is in my kernel parameters as well. It is on.

and I have added pci-stub into /etc/mkinitcpio.conf and rebuilt and all that.

Last edited by Zerqz (2015-04-30 02:36:20)

Offline

#4952 2015-04-30 05:21:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Zerqz

The vfio messages suggest the IOMMU is not enabled or not present.  Generally the correct option on AMD is amd_iommu=on.  If you have two identical cards then the pci-stub.ids= option is maybe not the best choice for you as it will claim both devices.  This is why you're failing to start graphics on the host, pci-stub owns the device rather than the host graphics driver.  You'll either need to create some scripts to unbind the host device from pci-stub and give it to the host, something like:

echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/unbind echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/remove_id echo 0000:01:00.0 > /sys/bus/pci/drivers_probe echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/new_id

(repeat for audio function/ID)  You could actually replace the drivers_probe line with modprobe --ignore-install of the host driver and specify the script as an install option in modprobe.d.  Another option that you'll find used previously in this very long thread is to use the xen-pciback module in place of pci-stub since it has the ability to claim devices based on bus address rather than device IDs.

You also seem to be trying to use VGA-mode assignment (I'd recommend UEFI/OVMF), but you're not specifying x-vga=on as an option for the vfio-pci device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4953 2015-04-30 05:28:36

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Zerqz

The vfio messages suggest the IOMMU is not enabled or not present.  Generally the correct option on AMD is amd_iommu=on.  If you have two identical cards then the pci-stub.ids= option is maybe not the best choice for you as it will claim both devices.  This is why you're failing to start graphics on the host, pci-stub owns the device rather than the host graphics driver.  You'll either need to create some scripts to unbind the host device from pci-stub and give it to the host, something like:

echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/unbind echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/remove_id echo 0000:01:00.0 > /sys/bus/pci/drivers_probe echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/new_id

(repeat for audio function/ID)  You could actually replace the drivers_probe line with modprobe --ignore-install of the host driver and specify the script as an install option in modprobe.d.  Another option that you'll find used previously in this very long thread is to use the xen-pciback module in place of pci-stub since it has the ability to claim devices based on bus address rather than device IDs.

You also seem to be trying to use VGA-mode assignment (I'd recommend UEFI/OVMF), but you're not specifying x-vga=on as an option for the vfio-pci device.

I thought that pci-stub.ids=10de:13c0,10de:0fbb only pointed to the card in pci slot 7, not slot 1, because the numbers beside the card in slot 1 are all different. That's why I can't just blacklist nvidia, because I still need the nvidia drivers for the first 980.

I did do some command to check if amd-vi was on and it was successful, but I will change to amd_iommu=on to see.

Offline

#4954 2015-04-30 05:34:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Zerqz

Then perhaps you don't really have two GTX980s, because they would both be 10de:13c0.  Maybe you could provide us with the actual data you're working from.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4955 2015-04-30 07:05:33

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since some time ago USB ports on the front of PC stopped working when booted. They work fine when booting though (can boot from usb drive connected to said ports). I noticed this:

On GRUB command line:

pci-stub.ids=1002:4397,1002:4396,1002:6810,1002:aab0
~ % lspci -n | grep 1002:4397 00:12.0 0c03: 1002:4397 00:13.0 0c03: 1002:4397 00:16.0 0c03: 1002:4397 ~ % lspci -n | grep 1002:4396 00:12.2 0c03: 1002:4396 00:13.2 0c03: 1002:4396 00:16.2 0c03: 1002:4396 ~ % lspci -n | grep 1002:6810 06:00.0 0300: 1002:6810 ~ % lspci -n | grep 1002:aab0 06:00.1 0403: 1002:aab0
~ % lspci ... 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller ... 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller ... 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ...

How can it be that multiple usb controllers have same id (is it called id?) 1002:4396? It makes me believe assigning 1002:4396 to pci-stub kills my front usb ports as well as card reader. Does anyone have any idea how could i diagnose and possibly fix this? In the past during manual testing i found out that ports i want passed through to VM are on controllers 00:13.0 and 00:13.2.

Offline

#4956 2015-04-30 07:14:13

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

Hi, all:

my environment:
kernel: 3.18.6 with i915 patches.
qemu: 2.1.3
arch: Intel Xeon series
type: legacy VGA passthrough

There is a problem when I passthrough AMD HD7850 card to VM. the catd can passthrough successfully.
but when I installed driver, the vm crashed with follow error:
pcieport 0000:00:03.0: AER: Multiple Uncorrected (Fatal) error received: id=0000
vfio-pci 0000:04:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0400(Unregistered Agent ID)
vfio-pci 0000:04:00.1: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0401(Unregistered Agent ID)

qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

If I just passthrough the VGA card, without hdmi audio, the VM worked well with driver. I don't know why, somebody can explain to me, thanks.

Just ignored the HDMI Audio completely (or adding it to pcie.0 bus works as well).

Offline

#4957 2015-04-30 07:14:51

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

Hi, aw
the AMD Firepro R5000 can passthrough successfully, but when I install vga driver, it restart with BSOD like atikmpag.sys error

my kernel:
linux-3.18.6
qemu:
qemu-2.1.3

commandline:
/usr/libexec/qemu-kvm-passthrough -cpu host -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/usr/share/qemu/win-pre-install-with-slic.bin \
-drive file=/var/lib/libvirt/images/fwq_clean_win7_64bit-patched.img,id=virtio-scsi-disk0 \
-device virtio-blk-pci,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=06:d8:e8:00:00:1b,addr=0x3 \
-net tap,ifname=tap0,script=/root/qemu-ifup \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.2 \
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.3 \
-device vfio-pci,host=06:00.2,bus=root.1,addr=00.4 \
-monitor stdio

just ignored the HDMI Audio completely (or adding it to pcie.0 bus works as well).

Offline

#4958 2015-04-30 08:59:00

fseven
Member
Registered: 2015-04-15
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone found a way to disable/power down the passthrough device when the VM is not in use?
Having an additional graphics card drawing power and producing heat/noise isn't ideal.

Offline

#4959 2015-04-30 14:16:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:

Has anyone found a way to disable/power down the passthrough device when the VM is not in use?
Having an additional graphics card drawing power and producing heat/noise isn't ideal.

There's no way to do this without hardware support for slot power control.  The best we can do generically is put the device into a D3 power state.  The vfio-pci kernel driver will do this automatically for unused device bound to the driver starting in v4.1-rc1.  Don't expect a massive difference, but it saves a few watts.  Unfortunately it doesn't do anything about the fan speed unless the device goes to a lower fan speed in D3.  We would need device specific power management drivers to do anything more effective, but the better alternative to that is probably to make the host graphics drivers more robust with binding and unbinding devices so we can simply give the device back to the host and let the native driver put it into a low power state.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4960 2015-04-30 14:26:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

How can it be that multiple usb controllers have same id (is it called id?) 1002:4396? It makes me believe assigning 1002:4396 to pci-stub kills my front usb ports as well as card reader. Does anyone have any idea how could i diagnose and possibly fix this? In the past during manual testing i found out that ports i want passed through to VM are on controllers 00:13.0 and 00:13.2.

See previous suggestions to Zerqz about pci-stub and multiple devices with the same device ID.  You either need to switch to xen-pciback, use a script to better manage pci-stub, or since these are just usb controllers and manage unbinding and re-binding to host drivers just fine, dynamically move the device to vfio-pci just for the VM, which is what libvirt would do if you're using it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4961 2015-04-30 15:04:49

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Zerqz

Then perhaps you don't really have two GTX980s, because they would both be 10de:13c0.  Maybe you could provide us with the actual data you're working from.

lspci

01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) 02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04) 06:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] 07:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) 07:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) 0b:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) 0c:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 0d:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

lspci -n

01:00.0 0300: 10de:13c0 (rev a1) 01:00.1 0403: 10de:0fbb (rev a1) 02:00.0 0106: 1b21:0612 (rev 01) 03:00.0 0106: 1b21:0612 (rev 01) 04:00.0 0c03: 1b21:1042 05:00.0 0604: 1b21:1080 (rev 04) 06:04.0 0401: 13f6:8788 07:00.0 0300: 10de:13c0 (rev a1) 07:00.1 0403: 10de:0fbb (rev a1) 0b:00.0 0200: 10ec:8168 (rev 09)

My mistake.

I'll try to unbind it and see what happens soon as I can.

You'll either need to create some scripts to unbind the host device from pci-stub and give it to the host, something like: echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/unbind echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/remove_id echo 0000:01:00.0 > /sys/bus/pci/drivers_probe echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/new_id (repeat for audio function/ID) You could actually replace the drivers_probe line with modprobe --ignore-install of the host driver and specify the script as an install option in modprobe.d. Another option that you'll find used previously in this very long thread is to use the xen-pciback module in place of pci-stub since it has the ability to claim devices based on bus address rather than device IDs. You also seem to be trying to use VGA-mode assignment (I'd recommend UEFI/OVMF), but you're not specifying x-vga=on as an option for the vfio-pci device.

Offline

#4962 2015-04-30 15:17:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Off-topic a bit:
We're steadily going to page #200, and approaching 5000 posts in this thread. We really need to do something about it...

It makes me somewhat envy seeing people with two GTX980 not running SLI for maximum lulz... Would be awesome to quadrify and push them both into the VM.. Sadly, there's no quadro-counterpart yet.

Last edited by Duelist (2015-04-30 15:31:48)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4963 2015-04-30 16:20:09

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I posted about this issue here, but I'm having issues with windows freezing as I'm trying to boot in ovmf. Could anyone help me?

Offline

#4964 2015-04-30 16:43:22

Omar007
Member
Registered: 2015-04-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@noctlos; I'm having the same problem. I think I once read somewhere that it had to do with the Q35 chipset not allowing Windows 8 and later to boot properly but I can't seem to find it so I'm not certain that is still the case or how to solve it.

@Duelist; a physical Q35 chipset doesn't support SLI/CF so I guess you can't use it in the VM even if you do pass through 2 GPU's.. Did anyone try to do this?

Offline

#4965 2015-04-30 17:04:47

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Omar007; I have another image on the same computer, with a similar windows installation. It boots fine under the same arguments. That it won't boot with a new image doesn't compute for me.

Edit: nevertheless, it appears that removing references to q35 chipsets resolved this. I'll do some more experimenting.

Last edited by noctlos (2015-04-30 17:09:56)

Offline

#4966 2015-04-30 21:05:52

chillum
Member
Registered: 2014-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! I'm using Linux Sabayon 3.9.0-sabayon

sysfs (/sys/) in this system isn't compatible with the sysfs folder structure used in vfio-bind on the front page; It used to work on my gentoo system, but then I upgraded to Sabayon instead of trying to figure zfs (and many other reasons), now I'm stuck, quite an important bit of my work was done on Windoze, so I'd like it back (and well, games of course), any help? please?

Offline

#4967 2015-04-30 21:17:00

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw this time different card seem to have received communication intended for FirePro W7100 at 82:00 (see lspci -nnvt output above for my configuration)

[175157.337000] vfio-pci 0000:82:00.0: irq 160 for MSI/MSI-X [175157.354768] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.354957] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.354969] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.355076] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.355082] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.355088] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X [175157.355212] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.355218] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.355223] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X [175157.355228] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X [175157.355379] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.355385] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.355390] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X [175157.355396] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X [175157.355401] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X [175157.355571] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.355577] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.355582] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X [175157.355588] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X [175157.355593] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X [175157.355599] vfio-pci 0000:81:00.0: irq 166 for MSI/MSI-X [175157.355790] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.355796] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.355802] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X [175157.355807] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X [175157.355812] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X [175157.355818] vfio-pci 0000:81:00.0: irq 166 for MSI/MSI-X [175157.355824] vfio-pci 0000:81:00.0: irq 167 for MSI/MSI-X [175157.356036] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X [175157.356042] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X [175157.356047] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X [175157.356052] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X [175157.356057] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X [175157.356062] vfio-pci 0000:81:00.0: irq 166 for MSI/MSI-X [175157.356068] vfio-pci 0000:81:00.0: irq 167 for MSI/MSI-X [175157.356076] vfio-pci 0000:81:00.0: irq 168 for MSI/MSI-X [175157.377567] dmar: DRHD: handling fault status reg 2 [175157.382540] dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 43e39b000 DMAR:[fault reason 06] PTE Read access is not set [175159.030150] usb 3-1: reset low-speed USB device number 2 using xhci_hcd [175159.319282] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d05120 [175159.319287] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d050c0 [175159.319295] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [175159.319299] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [175159.607427] usb 3-1: reset low-speed USB device number 2 using xhci_hcd [175159.896485] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d05120 [175159.896489] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d050c0 [175159.896495] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [175159.896499] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [175160.184610] usb 3-1: reset low-speed USB device number 2 using xhci_hcd [175160.472711] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d05120 [175160.472715] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d050c0 [175160.472720] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes [175160.472723] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes [175160.764503] dmar: DRHD: handling fault status reg 102 [175160.769652] dmar: DMAR:[DMA Write] Request device [82:00.0] fault addr 43de38000 DMAR:[fault reason 05] PTE Write access is not set

Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#4968 2015-04-30 21:21:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chillum wrote:

Hi! I'm using Linux Sabayon 3.9.0-sabayon

sysfs (/sys/) in this system isn't compatible with the sysfs folder structure used in vfio-bind on the front page; It used to work on my gentoo system, but then I upgraded to Sabayon instead of trying to figure zfs (and many other reasons), now I'm stuck, quite an important bit of my work was done on Windoze, so I'd like it back (and well, games of course), any help? please?

Why is sysfs incompatible?  Is Sabayon based on a 3.9 kernel?  That's way too old for VGA assignment.  Have they messed with the sysfs mount point?  If so, they're broken, http://git.kernel.org/cgit/linux/kernel … -rules.txt

- sysfs is always at /sys
  Parsing /proc/mounts is a waste of time. Other mount points are a
  system configuration bug you should not try to solve. For test cases,
  possibly support a SYSFS_PATH environment variable to overwrite the
  application's behavior, but never try to search for sysfs. Never try
  to mount it, if you are not an early boot script.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4969 2015-04-30 21:33:54

chillum
Member
Registered: 2014-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, sorry, I messed about trying to find a kernel that would compile and 3.9.0 was the only one that compiled, problem solved, too old a kernel thanks!

Offline

#4970 2015-05-01 00:19:36

gg
Member
Registered: 2015-03-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw (also, thanks @Bronek)

I'm using your patch onto qemu v2.3.0 with kernel v4.1-rc1 (also with libvirt and OVMF) and my R9 290 will reset if I shutdown the machine and start it again, but not restart the machine reliably (see below).

This works both whether the Quadro 6000 is used by pci-stub, or vfio-pci and assigned together with the R9 290. I'm not using the ACS quirk patch for the root PCH.

-[0000:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller +-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 +-01.1-[02]--+-00.0 NVIDIA Corporation GF100GL [Quadro 6000] | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller

Before I installed the Windows drivers for the R9 290 (with only the R9 290 passed through) I was able to the restart the machine once. After installing them and going from a safe resolution to my monitor native resolution (accelerated) and then having the driver crash I haven't been able to restart (but I can shutdown and start the machine again).

After a driver crash,  when I restart I get the 'blinking cursor on black background' as mentioned by @Bronek. I'm not getting messages via dmesg similar to what @Bronek is reporting when he/she also gets the blinking cursor. Per the notes in the patch, might it be the SMC firmware is not running or in some state the patch doesn't work with?

I'm not sure why the driver has been crashing (I'd not had the issue before with the R9 290 passed through but some other variables have changed too). I've been unable to relate the driver crashes to anything yet and I don't know if that's what is preventing restarting. Being unable to restart the machine is a bit inconvenient when upgrading Windows 10 releases. I'll have to try again sometime with Windows 8.1 perhaps.

Just wanted to let you know that being able to shutdown and start the machine has made the R9 290 worlds more practical.

Thank you!!

Offline

#4971 2015-05-01 01:35:45

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm still having that code 43 issue on Windows 8.1. Here's my libvirt XML:

<domain type='kvm'>
  <name>Win</name>
  <uuid>f4f827da-ef6c-11e4-a668-8f0730658845</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='off'/>
    </kvm>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win8.1.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/carter/Windows_8.1_Pro_X64_Activated.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:a7:85:28'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Host: Ubuntu 14.10 with GTX 560
Guest: Windows 8.1 GTX 760

Offline

#4972 2015-05-01 02:36:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@carterghill

Remove the <graphics> and <video> sections.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4973 2015-05-01 02:40:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@gg

I can't really figure out whether the patch is an improvement for you.  There's really not much difference for VM shutdown vs VM reset as far as the types of resets that occur for the device.  My testing on a Bonaire seemed to indicate that resets vs shutdown/restart had about the same success rate, ie. mostly working but occasionally not.  The SMC firmware must be running for the reset to work, otherwise it reverts to previous behavior, which in my experience gives you a black screen rather than bsod.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4974 2015-05-01 02:57:43

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@carterghill

Remove the <graphics> and <video> sections.

Sorry I should've clarified, I have tried without them as well. I only had them in just now so that I could actually use windows. Without those tags, there is still no output on my second monitor.

Last edited by carterghill (2015-05-01 04:56:15)

Offline

#4975 2015-05-01 03:48:12

gg
Member
Registered: 2015-03-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@gg

I can't really figure out whether the patch is an improvement for you.  There's really not much difference for VM shutdown vs VM reset as far as the types of resets that occur for the device.  My testing on a Bonaire seemed to indicate that resets vs shutdown/restart had about the same success rate, ie. mostly working but occasionally not.  The SMC firmware must be running for the reset to work, otherwise it reverts to previous behavior, which in my experience gives you a black screen rather than bsod.

Before using the patch on qemu 2.3 I was using 2.1 and was not once able to restart or shutdown|start a guest machine without suspending or rebooting the host. It seems from what I'm experiencing that the patch has finally provided a reset (under certain conditions). Is there something else in qemu, the kernel, and the vfio-pci module in particular, that might be making this possible if not the patch you provided for qemu?

Offline

#4976 2015-05-01 04:10:36

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

carterghill wrote:
aw wrote:

@carterghill

Remove the <graphics> and <video> sections.

Sorry I should've clarified, I have tried without them as well. I only I had them in just now so that I could actually use windows. Without those tags, there is still no output on my second monitor.

Isn't is kvm hidden state=ON? not OFF? And you must get output from passthrough gpu regradless of driver working.

Offline

#4977 2015-05-01 04:10:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gg wrote:
aw wrote:

@gg

I can't really figure out whether the patch is an improvement for you.  There's really not much difference for VM shutdown vs VM reset as far as the types of resets that occur for the device.  My testing on a Bonaire seemed to indicate that resets vs shutdown/restart had about the same success rate, ie. mostly working but occasionally not.  The SMC firmware must be running for the reset to work, otherwise it reverts to previous behavior, which in my experience gives you a black screen rather than bsod.

Before using the patch on qemu 2.3 I was using 2.1 and was not once able to restart or shutdown|start a guest machine without suspending or rebooting the host. It seems from what I'm experiencing that the patch has finally provided a reset (under certain conditions). Is there something else in qemu, the kernel, and the vfio-pci module in particular, that might be making this possible if not the patch you provided for qemu?

Nope, if there's an improvement, it's definitely from this patch.  I just couldn't figure out from the previous post whether things were actually better for you with the patch.  Thanks


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4978 2015-05-01 04:13:50

gg
Member
Registered: 2015-03-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
gg wrote:
aw wrote:

@gg

I can't really figure out whether the patch is an improvement for you.  There's really not much difference for VM shutdown vs VM reset as far as the types of resets that occur for the device.  My testing on a Bonaire seemed to indicate that resets vs shutdown/restart had about the same success rate, ie. mostly working but occasionally not.  The SMC firmware must be running for the reset to work, otherwise it reverts to previous behavior, which in my experience gives you a black screen rather than bsod.

Before using the patch on qemu 2.3 I was using 2.1 and was not once able to restart or shutdown|start a guest machine without suspending or rebooting the host. It seems from what I'm experiencing that the patch has finally provided a reset (under certain conditions). Is there something else in qemu, the kernel, and the vfio-pci module in particular, that might be making this possible if not the patch you provided for qemu?

Nope, if there's an improvement, it's definitely from this patch.  I just couldn't figure out from the previous post whether things were actually better for you with the patch.  Thanks

Yes, was definitely a huge step. Thanks!

Offline

#4979 2015-05-01 04:16:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
carterghill wrote:
aw wrote:

@carterghill

Remove the <graphics> and <video> sections.

Sorry I should've clarified, I have tried without them as well. I only I had them in just now so that I could actually use windows. Without those tags, there is still no output on my second monitor.

Isn't is kvm hidden state=ON? not OFF? And you must get output from passthrough gpu regradless of driver working.

Yes, hidden state='on'  Good catch


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4980 2015-05-01 05:38:33

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
slis wrote:
carterghill wrote:

Sorry I should've clarified, I have tried without them as well. I only I had them in just now so that I could actually use windows. Without those tags, there is still no output on my second monitor.

Isn't is kvm hidden state=ON? not OFF? And you must get output from passthrough gpu regradless of driver working.

Yes, hidden state='on'  Good catch

Ah okay, I changed that and there's still no output on my second monitor hmm. I sort of expected this, seeing as how I purposely tried with earlier drivers initially and had no output then either. So we know that it's being passed through because I can see it in the device manager when I view it through VNC, right? And since apparently I should get output regardless of drivers (I should've known that...) what else could be causing this issue? Oh, and thanks for your help so far you guys smile

Last edited by carterghill (2015-05-01 05:38:55)

Offline

#4981 2015-05-01 06:11:09

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you dont have output your i915 patch is not working, or you need to patch host nvidia driver for vga, or your gpu does not have uefi  bios for ovmf.

Offline

#4982 2015-05-01 15:10:23

ZaoX64
Member
Registered: 2015-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hellow im finish my simple config. 
IT WORKS...
Sound terible crakling*^&$$. I am working on it wink
Ungine Valley and Heaven benchmark pass.

MSI Z77  + i7-3770 VT-d
NVIDIA EVGA GTX660 ( rom flash to support UEFI )

Host on 3.19-3-3-ARCH witch QEMU 2.2.1 + OVMF + intel GPU + 2 monitor  (old one and new model  with multi input 27''  )
Guest OS  Windows Server 2012 x64. with GTX660 + 1  monitor (shared with Host OS)
I have 2 sets of mouse and keyboard, one for OS.

my simple bash script:

#!/bin/bash export QEMU_ALSA_DAC_BUFFER="512" export QEMU_ALSA_DAC_PERIOD_SIZE="1024" export QEMU_AUDIO_DRV="alsa" cp -f /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_var.fd qemu-system-x86_64 \ -enable-kvm \ -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \ -smp 4,sockets=1,cores=4,threads=1 \ -m 4096 \ -soundhw hda \ -vga none \ -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \ -drive if=pflash,format=raw,readonly,file="/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd" \ -drive if=pflash,format=raw,file="./efi_var.fd" \ -hda ./winbox01.img \ -cdrom /dev/sr0 \ -usb -usbdevice host:046d:c316 -usbdevice host:1bcf:0005 

   

What do you think -M q35 will be bettr for my VM ?

Last edited by ZaoX64 (2015-05-01 19:26:05)


[MSI Z77A-G43 i7-3770 32GB GTX660]

Offline

#4983 2015-05-01 15:33:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ZaoX64 wrote:
 -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \

Are you running an older Nvidia driver to be able to use the hyper-v extensions?  Generally all those hv_foo options need to be removed to avoid Code 43 on latest Nvidia driver.

What do you think -m q35 will be bettr for my VM ?

100% no


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4984 2015-05-01 15:34:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

If you dont have output your i915 patch is not working, or you need to patch host nvidia driver for vga, or your gpu does not have uefi  bios for ovmf.

i915 patch shouldn't be relevant since they're not using IGD, but I do agree that something is wrong with vga arbitration/routing since there isn't any output to the screen.  I'd try a newer kernel.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4985 2015-05-01 15:53:14

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
slis wrote:

If you dont have output your i915 patch is not working, or you need to patch host nvidia driver for vga, or your gpu does not have uefi  bios for ovmf.

i915 patch shouldn't be relevant since they're not using IGD, but I do agree that something is wrong with vga arbitration/routing since there isn't any output to the screen.  I'd try a newer kernel.

I am on the latest kernel (to my knowledge) 3.19. Maybe I should try the i915 patch again. How exactly do I run the patch? I've had a lot of trouble finding a decent guide that shows how to run a patch, and most of these threads just say where do find the patch and not what to do with it... I know that GTX 760 should work, I've seen other people succeed with it. How would I patch my host nvidia driver for VGA?

Last edited by carterghill (2015-05-01 15:56:16)

Offline

#4986 2015-05-01 19:10:24

ZaoX64
Member
Registered: 2015-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Are you running an older Nvidia driver to be able to use the hyper-v extensions?  Generally all those hv_foo options need to be removed to avoid Code 43 on latest Nvidia driver.

Currently NV driver version are 331.58 (I know to old - but safe) but i planing to successivly update driver to higher posible.
I don't konw highest working version.
(mayby someday all nv driver 350+ will work )

I would like use this enviroment to develop my multiplatform game.

Last edited by ZaoX64 (2015-05-01 19:17:25)


[MSI Z77A-G43 i7-3770 32GB GTX660]

Offline

#4987 2015-05-01 19:25:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ZaoX64 wrote:
aw wrote:

Are you running an older Nvidia driver to be able to use the hyper-v extensions?  Generally all those hv_foo options need to be removed to avoid Code 43 on latest Nvidia driver.

Currently NV driver version are 331.58 (I know to old - but safe) but i planing to successivly update driver to higher posible.
I don't konw highest working version.
(mayby someday all nv driver 350+ will work )

I would like use this enviroment to develop my multiplatform game.

At 331.58, you don't even need kvm=off, that change didn't happen until 338.77.  If you go up to 344.11 you'll need to remove all those hv_foo enablers.  IME, the graphics performance gain from newer drivers trumps the benefit of the hyper-v extensions.  If graphics performance isn't your #1 priority then maybe there are cases for using older drivers.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4988 2015-05-01 19:49:46

ZaoX64
Member
Registered: 2015-05-01
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
ZaoX64 wrote:
aw wrote:

Are you running an older Nvidia driver to be able to use the hyper-v extensions?  Generally all those hv_foo options need to be removed to avoid Code 43 on latest Nvidia driver.

Currently NV driver version are 331.58 (I know to old - but safe) but i planing to successivly update driver to higher posible.
I don't konw highest working version.
(mayby someday all nv driver 350+ will work )

I would like use this enviroment to develop my multiplatform game.

At 331.58, you don't even need kvm=off, that change didn't happen until 338.77.  If you go up to 344.11 you'll need to remove all those hv_foo enablers.  IME, the graphics performance gain from newer drivers trumps the benefit of the hyper-v extensions.  If graphics performance isn't your #1 priority then maybe there are cases for using older drivers.

I remove Hyper-V and use 347.88. With good result.
I will try 350...

Last edited by ZaoX64 (2015-05-01 19:50:41)


[MSI Z77A-G43 i7-3770 32GB GTX660]

Offline

#4989 2015-05-02 07:35:50

toxster
Member
Registered: 2015-04-28
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey just wanted to post a small success story,

Intel 5960X w/ Asus Strix 970 GTX (radeon HD as primary), latest nvidia drivers works when using kvm=off, cannot get Geforce Experience to recognize that i have a nvidia card, but the driver works, have anyone got passed this? edit: it just started working, weird, maybe some windows update...

Win 8.1 using ovmf (ovmf-git/svn and qemu 2.3.50 (built from git 3 nights ago)

GTA 5 running quite smooth @ 4K High, ~47fps

however, there seem to be some 60 fps limitation, even if i set heaven and valley benchmarks at ridiculously low resolution with all settings on low or none, it cannot get past 60 fps?!

Thanks for the good job guys, now I can have a gaming rig, docker host, and dev setup on the same computer smile

Host:
Kernel: 4.0.1-1-ARCH (no patches!)
MB: Gigabyte X99 Gaming 5
CPU: Intel 5960X
Guest GFX: ASUS STRIX Geforce 970 GTX
Host GFX: Radeon HD4350 (set as primary in BIOS)

Last edited by toxster (2015-05-04 13:21:06)

Offline

#4990 2015-05-03 05:44:22

shawnt
Member
Registered: 2014-02-07
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Same as above, wanna share some progress so far I got.

Hardware: msi z97 gaming-5, i5-4690k, evga gtx 750 ti

What have succeeded:
stock kernel, qemu-system-x86_64, ovmf, windows 8.1 64bit, nvidia driver 338
linux-vfio, qemu-system-x86_64, seabios, windows 8.1 64bit, nvidia driver 338
linux-vfio, qemu-system-i386, seabios, windows 8.1 32bit, nvidia driver 338

stock kernel, qemu-system-x86_64, ovmf, windows 10 64bit 9926 build, nvidia driver 350
linux-vfio, qemu-system-x86_64, seabios, windows 10 64bit 9926 build, nvidia driver 350
linux-vfio, qemu-system-i386, seabios, windows 10 32bit 9926 build, nvidia driver 350

What have failed:
Windows 10 32bit/64bit 10041 build in all conditions fail to install as BSOD "system_thread_exception_not_handled". The install went through without vfio passthrough under both seabios and ovmf (still not able to boot if switch to vfio after installation gets done).
Windows 10 9926 build with 338 driver has some glitches during my planetside 2 gaming. It got fixed after upgrading to 344 driver, although I'm not sure if it's caused by the 338 driver or something else. Haven't got a chance to verify.
Nvidia 352 beta under windows 10 9926 build gets code 43 error with kvm=off and no hv_* (arms racing has begun?).
It seems ovmf under qemu-system-i386 doesn't work with vfio enabled, that's why 32bit with ovmf is missing in the list.

Thanks to all the people working on qemu and vfio project, especially to the OP for working on this guide. Really blows my mind how much potential power I can get from my archlinux PC.
Sorry about my english as a non-native speaker

Update: from the post on page 209, Windows 10 beyond 9926 would need CPU either core2duo or kvm64 with +lahf_lm,+cx16. Nvidia driver 352 and 358 requires 10041+ build to run.

Last edited by shawnt (2015-05-31 07:23:11)

Offline

#4991 2015-05-03 07:36:13

scoobydog
Member
Registered: 2014-04-14
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

with your help, I run Battlefield 4 and Battlefield Hardline in KVM Win 7 Pro x64 on a Debian 8.0 Jessie, without patches or special modifications, with good FPS, 50-60 in FHD and 30-40 in UHD.

here is a short video https://youtu.be/N9QwDmYBRgg

Info's

Host:
  Debian 8.0 Jessie 01.05.2015
Guest:
  KVM Win7 Pro amd64 with Nvidia whql Driver 350.12 64bit
Hardware:
  CPU i7 4820K socket 2011
  Mainboard Gigabyte GA-X79-UP4 Rev. 1.0
  Graphic card:
    Host: Gigabyte GV-N75TOC-2GI
    Guest: Asus GTX780-DC2OC-3GD5
Patches:
  none, Debian standard
Modifications:
  /etc/default/grub
    GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pci_stub.ids=(ids from GTX780)"
  /etc/initramfs-tools/modules
    add "pci_stub" at the end
after change, start this as root:
  update-initramfs -u
  update-grub2
  and reboot

After boot:
  First bind graphic card to vfio:
    lspci give this information for my GTX780
    04:00.0 VGA compatible controller: NVIDIA Corporation GK110 GeForce GTX 780 rev a1
    04:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio rev a1
    vfio-bind from https://bbs.archlinux.org/viewtopic.php?id=162768
    as root start "vfio-bind 0000:04:00.0 0000:04:00.1"

now you can start the virtual machine ---

qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-machine q35,accel=kvm \
-soundhw hda \
-device vfio-pci,host=0x:00.0,rombar=1,x-vga=on \
-device vfio-pci,host=0x:00.1 \
-bios /usr/share/seabios/bios.bin \
-device virtio-net-pci,netdev=user.0,mac=52:54:00:03:02:01 \
-netdev user,id=user.0 \
-drive file=win7-x64_system.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-drive file=win7-games.qcow2,if=none,id=drive-virtio-disk1,format=qcow2 \
-device virtio-blk-pci,scsi=off,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 \
-rtc base=localtime,driftfix=slew \
-usbdevice host:x.x \
-usbdevice host:y.y \
-vga none

Thanks for your Help.

Last edited by scoobydog (2015-05-03 12:24:51)

Offline

#4992 2015-05-04 06:24:00

gg
Member
Registered: 2015-03-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gg wrote:
aw wrote:
gg wrote:

Before using the patch on qemu 2.3 I was using 2.1 and was not once able to restart or shutdown|start a guest machine without suspending or rebooting the host. It seems from what I'm experiencing that the patch has finally provided a reset (under certain conditions). Is there something else in qemu, the kernel, and the vfio-pci module in particular, that might be making this possible if not the patch you provided for qemu?

Nope, if there's an improvement, it's definitely from this patch.  I just couldn't figure out from the previous post whether things were actually better for you with the patch.  Thanks

Yes, was definitely a huge step. Thanks!

Today I'm using a fresh install of Windows 8.1 (and kernel 4.1-rc2, qemu 2.3, and OVMF) and the R9 290 is resetting on each reboot whether or not a Quadro 6000 is also passed-through.

My longstanding issue with resetting Hawaii appears to be largely resolved! smile

Offline

#4993 2015-05-04 13:11:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gg wrote:

Today I'm using a fresh install of Windows 8.1 (and kernel 4.1-rc2, qemu 2.3, and OVMF) and the R9 290 is resetting on each reboot whether or not a Quadro 6000 is also passed-through.

My longstanding issue with resetting Hawaii appears to be largely resolved! smile

QEMU 2.3 or qemu.git.  The reset patch didn't go in until after 2.3.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4994 2015-05-04 19:24:05

gg
Member
Registered: 2015-03-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
gg wrote:

Today I'm using a fresh install of Windows 8.1 (and kernel 4.1-rc2, qemu 2.3, and OVMF) and the R9 290 is resetting on each reboot whether or not a Quadro 6000 is also passed-through.

My longstanding issue with resetting Hawaii appears to be largely resolved! smile

QEMU 2.3 or qemu.git.  The reset patch didn't go in until after 2.3.

That was with qemu.git branch/tag v2.3.0 and your patch. I've now built again from qemu.git head, which includes your patch, and it also works the same (well).

Offline

#4995 2015-05-05 11:26:54

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ughman wrote:

I am going now to move the guest-assigned card to another PCI-e x16 slot and attempt again, not expecting any success.

It seems this has fixed my issue completely. I am now successfully passing through an AMD R9 290 while leaving an NVIDIA 9800 GT as the host device (nouveau driver).

For future reference, this is the exact hardware I am using:

  • Athlon X4 860K - AMD AD860KXBJABOX

  • Radeon R9 290 - Sapphire "tri-x oc" 100362-2SR

  • A88X Bolton D4 motherboard - ASRock "fatal1ty" 90-MXGT60-A0UAY1Z

I had previously had my card installed in the PCI-e x16 3.0 slot; my fix was to move it to the x4 2.0 slot (physically x16, labeled as "x4 mode"). The labels in the motherboard user guide for these slots are PCIE2 and PCIE4 respectively. It is unfortunate that I could not get the R9 into the 3.0 slot since the other card can't do 3.0, and I don't do much graphics-wise on the host. Still, I am very pleased with the results.

This is my current working QEMU script, using the older windows install which had been previously running on a bare metal AM2+ phenom system

#!/bin/sh export QEMU_AUDIO_DRV=sdl export QEMU_AUDIO_DAC_FIXED_SETTINGS=1 export QEMU_AUDIO_DAC_FIXED_FREQ=44100 export QEMU_AUDIO_DAC_FIXED_FMT=S16 export QEMU_SDL_SAMPLES=1024 qemu-system-x86_64 \ -machine pc-i440fx-2.1,accel=kvm,usb=off \ -cpu host \ -drive file=OVMF.fd,format=raw,if=pflash \ -drive file=/dev/mapper/mass-win--uefi,format=raw,if=virtio \ -m 4096 \ -smp 3 \ -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \ -device usb-host,id=usb-keyboard-mouse,bus=usb.0,vendorid=0x13ba,productid=0x0018 \ -netdev user,id=net \ -device rtl8139,netdev=net,bus=pci.0,addr=0x6 \ -device vfio-pci,host=02:00.0,bus=pci.0,addr=0x8 \ -device vfio-pci,host=02:00.1,bus=pci.0,addr=0x9 \ -vga none \ -serial none \ -soundhw hda \ -nographic \ $*

Thanks for this thread, and thanks Duelist for providing a lot of help and information.

Offline

#4996 2015-05-05 13:26:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ughman wrote:

...
I had previously had my card installed in the PCI-e x16 3.0 slot; my fix was to move it to the x4 2.0 slot (physically x16, labeled as "x4 mode"). The labels in the motherboard user guide for these slots are PCIE2 and PCIE4 respectively. It is unfortunate that I could not get the R9 into the 3.0 slot since the other card can't do 3.0, and I don't do much graphics-wise on the host. Still, I am very pleased with the results.

This is my current working QEMU script, using the older windows install which had been previously running on a bare metal AM2+ phenom system

...

Thanks for this thread, and thanks Duelist for providing a lot of help and information.

AFAIR, all pci-e slots on Kaveri C/APUs are 3.0, and x1 3.0 has the same bandwidth as x16 2.0 or something. Trinity and Richland, on the other hand, can't do PCI-E 3.0 at all.
Try running lspci -vv as root to see the actual pci-e bandwidth speed. I think it won't bottleneck much.

 LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB

This is HD7750 as 01:00.0 @ first 16x 2.0 on 750K Trinity in F2A55(Hudson D2?..).

 LnkCap: Port #1, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB

This is the second HD7750 as 02:00.0 @ second 16x 2.0. It's routed as 8x, although i can't recall if there was any warning about it being only an x4 port. Since all the power lines are fed through the first 12 or so pins, it'd be weird to route the port, but not use it's full bandwidth.

 LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <4us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB

And this is GT610 as 04:00.0 @ first 1x 2.0 through the powered(12V line with molex aka IDE power connector) 1x->16x riser.

I'm just curious what it looks like in your system. And if you will consider putting R9 290 in a riser card, please, could you give me a photo of result? It's quite a lenghty card, so i suppose it'd be sitting is some weird place and/or position inside the case. I've stuffed my GT610 that way that it's metal bracket is centimeter away from the CPU fan, this fact gives me some lulz.

Also, how did you move the baremetal setup to the VM? Was there BSOD 7B? Migrating to ESP+EFI loader?

OH! And don't forget to fix your entry!

Last edited by Duelist (2015-05-05 13:34:53)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4997 2015-05-05 14:44:40

ughman
Member
Registered: 2015-04-12
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Try running lspci -vv as root to see the actual pci-e bandwidth speed. I think it won't bottleneck much.

 Capabilities: [78] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

This is the 9800 GT host card as 01:00.0 @ x16 3.0

 Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

This is the R9 290 as 02:00.0 @ x16 2.0 "x4 mode". Looking at this it seems like it's actually running at x1? That even worse.

The board's user manual itself says this:

PCIE2 (PCIe 3.0 x16 slot) is used for PCI Express x16 lane width graphics cards.
PCIE4 (PCIe 2.0 x16 slot) is used for PCI Express x4 lane width graphics cards.

Duelist wrote:

And if you will consider putting R9 290 in a riser card, please, could you give me a photo of result? It's quite a lenghty card, so i suppose it'd be sitting is some weird place and/or position inside the case.

The case is pretty big, there's 10 slots in the back so I just use the spare slots and stick a piece of cardboard under the riser to insulate it. I had a string of cable ties through the plastic fan cooling assembly thing to hold the card straight with more than just the backplate since the riser sits loose in the case. I'll take a picture of it when I get around to trying that setup again.

Duelist wrote:

Also, how did you move the baremetal setup to the VM? Was there BSOD 7B? Migrating to ESP+EFI loader?

The original setup was a ~80 G partition on a 1TB MBR drive, with some kind of 100M system partiton alongside it. Ignoring the 100M partition, I created a new LVM volume just large enough for the first partition and an MBR part table. I used fdisk on that and then copied the original partition right in. Apparently you can't just dd NTFS filesystems around wherever you want, there's something in the NTFS header which specifies the first drive sector/block/whatever for the filesystem, so I ended up modifying some part of the FS header with a hex editor to get it to work. Since I only copied that partition and not the 100M other thing, I used some boot repair tools on the win7 install disc which copied in the boot manager garbage and got it working after a few runs of it. I fixed the 0x7B BSOD by making some registry edit change using regedit from the install disc. I did something similar to this http://jsmcomputers.biz/wp/?p=249, also the same trick to get virtio working but changing the virtio key in that location.

Switching to UEFI, I made a new 100G LV, formatted it in fdisk as gpt and added the original partition to the start and a 100M "EFI System Partition" to the end, which is apparently fat32 with a special UUID. Running the same boot fixing thing on the win7 disc a couple times got that working, I don't remember running any command line programs but I might have. This time I used gparted instead of pv (or dd) which probably did make some changes to the partition table NTFS header, so I didn't need to use a hex editor this time.

I have to say the windows boot stuff is really rather ugly. I suppose it's better than it was for 2k and XP, but it's still pretty bad.

Duelist wrote:

OH! And don't forget to fix your entry!

Interesting, I don't think I submitted anything, it was probably added by someone else? The main page says the sheet is locked because of vandalism or something, so I don't think I can edit it myself.

Last edited by ughman (2015-05-05 14:53:43)

Offline

#4998 2015-05-05 15:13:27

wulfspider
Member
Registered: 2015-05-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am new to this topic. So I have some questions.

What is "better"? The qemu default chipset or q35? Which one is better for Windows 8?

Which bios should I use? Seabios or OVMF? Which one is better for Windows 8? (I want to import my Windows 8 Key from the ACPI table)

Which combination would have the best performance?

In a lot of posts of qemu commands I saw something like this:

qemu-system-x86_64 --enable-kvm -M q35 -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

What is the function of the bus and the addr parameter?

I read somewhere if I use the OVMF bios I wouldn't need the following parameter. Is that right?

multifunction=on,x-vga=on

Thanks you smile

Edit:
One more question: Why should I add a rom parameter to the graphics card?

Last edited by wulfspider (2015-05-05 15:17:14)

Offline

#4999 2015-05-05 16:39:49

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There is no better, performance is same, 440fx is recommended.

Seabios is for old legacy vga mode (needs kernel patch maybe), ovmf is uefi (no patches needed).

Bus and addr tells qemu where to put it on virtual bus/port or smth like that smile

Yes x-vga can be avoided using ovmf.

Sometimes rebooting vm doesn't work well, and using rom helps. It did with my old 560ti.

Offline

#5000 2015-05-05 16:42:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

5000!  (sorry, had to do it)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5001 2015-05-05 16:43:26

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hahahaha.

Offline

#5002 2015-05-05 16:53:12

wulfspider
Member
Registered: 2015-05-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just for notice:

What's the difference between q35 and 440fx (I think this is the default one)?

Offline

#5003 2015-05-05 16:58:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

wulfspider wrote:

Just for notice:

What's the difference between q35 and 440fx (I think this is the default one)?

Q35 is a PCI-express chipset from around 2007, i440FX is a conventional PCI chipset from around 1996 used on Pentium Pro systems.  QEMU emulates both of these, 440FX better than Q35, IMO.  If you're running a Windows guest and can use OVMF (UEFI support for GPU and guest OS), then use the default 440FX, it will cause you far fewer problems and you can use libvirt without any hacks.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5004 2015-05-05 17:13:03

wulfspider
Member
Registered: 2015-05-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to look for the EFI support of my graphics card (GTX 680) like this:

# cd /sys/bus/pci/devices/0000:01:00.0/ # echo 1 > rom # cat rom > /tmp/image.rom # echo 0 > rom

But I get an Input/Output error at the step with cat. I added the ids to the pci-stub boot parameter and added the nouveau to modprobe blacklist. Should I remove one of them?

Offline

#5005 2015-05-05 19:44:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

wulfspider wrote:

I am trying to look for the EFI support of my graphics card (GTX 680) like this:

# cd /sys/bus/pci/devices/0000:01:00.0/ # echo 1 > rom # cat rom > /tmp/image.rom # echo 0 > rom

But I get an Input/Output error at the step with cat. I added the ids to the pci-stub boot parameter and added the nouveau to modprobe blacklist. Should I remove one of them?

Just try a try an OVMF VM with it, if you get the TianoCore splash screen it supports UEFI.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5006 2015-05-05 20:51:11

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

wulfspider wrote:

But I get an Input/Output error at the step with cat.

Try using something like nvagetram or how was that software called.. There's a special piece of software to extract nvidia pre-POST ROMs from cards. Or you could get yours from baremetal GPU-z or techpowerup database. Or you can just follow aw's wise advise of just trying to kickstart OVMF without that unnecessary for GTX6XX cards check.


aw wrote:

5000!  (sorry, had to do it)

Time to celebrate.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5007 2015-05-05 21:21:39

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yo. I just upgraded to kernel 4.0.1, and applied the same old i915 patch....
But it doesn't seem to work anymore, even though the patch was successful.
I'm getting nothing anymore on my card output, and a bunch of "unclaimed register" errors in dmesg coming from i915.

Any tips ? Do anyone has a more recent version of the arbitration patch ?

Thanks smile

Edit: I've looked into OVMF, but sadly that means a whole system reinstall.
Actually been trying to reinstall on full EFI with OVMF, but I can't manage to boot any install ISO from OVMF.
VGA passthrough works perfect though.

Last edited by Fira (2015-05-05 22:46:04)

Offline

#5008 2015-05-06 01:20:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fira wrote:

Yo. I just upgraded to kernel 4.0.1, and applied the same old i915 patch....
But it doesn't seem to work anymore, even though the patch was successful.
I'm getting nothing anymore on my card output, and a bunch of "unclaimed register" errors in dmesg coming from i915.

Any tips ? Do anyone has a more recent version of the arbitration patch ?

Thanks smile

Edit: I've looked into OVMF, but sadly that means a whole system reinstall.
Actually been trying to reinstall on full EFI with OVMF, but I can't manage to boot any install ISO from OVMF.
VGA passthrough works perfect though.

Whole guest system re-install.  Well worth it to not be patching every new host kernel you get IMO.  I don't know where these ISO boot problems are coming from, but see the recent series on my blog, part 4 covers installing Windows 8.1 with OVMF using virt-manager.  If you're using Windows 7 with OVMF, I think some people have gotten that to work, but I haven't gotten that far in my howto series, maybe next week.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5009 2015-05-06 05:30:13

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In fact I've done the exact same as you did in your blog, but on QEMU command line directly..
I've tried with Windows 8.1 and 7 images from MSDNAA as you did, and even an ArchLinux ISO.
Every time, OVMF fails to boot both with pure-EFI and CSM, then drops you to the shell indicating there is a valid GPT Label on the disks.
Any ideas ? I didn't have any trouble booting these under Seabios, and AFAIK they should all be EFI-aware.

Offline

#5010 2015-05-06 07:39:42

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

5000!  (sorry, had to do it)

You worked hard for the privilege :-P


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5011 2015-05-06 07:45:09

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw I think my FirePro W7100 is either not meant to work with passthrough at all or is just faulty, since I have occasional artefacts on screen even before restarting the guest and when the card is secondary GPU in the system (i.e. should not be referred by host in any way). I think its not worth your time to try diagnosing my problems, I will switch to something else and put this card to other use.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5012 2015-05-06 12:09:29

toxster
Member
Registered: 2015-04-28
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again,

since my last post i have managed to go fully libvirt! no more vfio-bind scripts and long command lines...

For reference: the Nvidia Controlpanel defaults to vsync on (since i have a g-sync monitor) hence why i capped out at 60fps. I can get higher FPS in games now if i turn it off.

Here is my libvirt (since i was struggling to find a sample to base my own upon) i ended up creating a new machine in virt-manager and the virsh edit <name> to add the OVMF, kvm off, etc, required parts, left to do is trim out the unnecessary parts, serial port, ballooning etc.

<domain type='kvm' id='2'> <name>win8.1</name> <uuid>d9aaa8dd-6b43-421b-b165-706926e9181e</uuid> <memory unit='KiB'>16760832</memory> <currentMemory unit='KiB'>16760832</currentMemory> <vcpu placement='static' current='8'>32</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_code_x64.bin</loader> <nvram template='/usr/share/ovmf/x64/ovmf_vars_x64.bin'>/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd</nvram> </os> <features> <acpi/> <apic/> <pae/> <!-- if using NVIDIA, remove all elements inside of hyperv (Code 43 else) --> <hyperv> </hyperv> <!-- this kvm element was added to be able to use NVIDIA drivers. (Code 43 else) --> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='8'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> <!-- NVIDIA: must be set to no --> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdb3'/> <backingStore/> <target dev='sda' bus='scsi'/> <alias name='scsi0-0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda'/> <backingStore/> <target dev='sdb' bus='scsi'/> <boot order='1'/> <alias name='scsi0-0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/root/vm/virtio-win-0.1-100.iso'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:f8:f3:dd'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <product id='0x1006'/> <address bus='1' device='6'/> </source> <alias name='hostdev1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0x0a01'/> <address bus='1' device='2'/> </source> <alias name='hostdev2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc07d'/> <address bus='1' device='5'/> </source> <alias name='hostdev3'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <product id='0x0250'/> <address bus='1' device='7'/> </source> <alias name='hostdev4'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

I hope this helps anyone..

Is there anything i can add to increase performance? Huge pages?

Offline

#5013 2015-05-06 13:25:52

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes hugepages,
and change <topology sockets='1' cores='4' threads='8'/> to <topology sockets='1' cores='8' threads='1'/> or something like that, using threads slows cpu down atleast for me...

and you should do static pinning on real cores no ht ones, and maybe emulatorpin, read more at

https://libvirt.org/formatdomain.html#elementsCPUTuning

Offline

#5014 2015-05-06 14:23:26

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

toxster wrote:

Hi again,

since my last post i have managed to go fully libvirt! no more vfio-bind scripts and long command lines...

For reference: the Nvidia Controlpanel defaults to vsync on (since i have a g-sync monitor) hence why i capped out at 60fps. I can get higher FPS in games now if i turn it off.

Here is my libvirt (since i was struggling to find a sample to base my own upon) i ended up creating a new machine in virt-manager and the virsh edit <name> to add the OVMF, kvm off, etc, required parts, left to do is trim out the unnecessary parts, serial port, ballooning etc.

<domain type='kvm' id='2'> <name>win8.1</name> <uuid>d9aaa8dd-6b43-421b-b165-706926e9181e</uuid> <memory unit='KiB'>16760832</memory> <currentMemory unit='KiB'>16760832</currentMemory> <vcpu placement='static' current='8'>32</vcpu> . . .

I hope this helps anyone..

Is there anything i can add to increase performance? Huge pages?

Yes go with hugepages. Also, giving your VM 32 vcpus will slow it down quite a bit, unless you actually have this many threads in host machine to give away!


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5015 2015-05-06 18:04:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

5000!  (sorry, had to do it)

And it's also exaclty 2 years old

Offline

#5016 2015-05-06 18:48:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
aw wrote:

5000!  (sorry, had to do it)

And it's also exaclty 2 years old

That means a thousand posts per year. Not so much, if you think about it.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5017 2015-05-06 18:54:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:
aw wrote:

5000!  (sorry, had to do it)

And it's also exaclty 2 years old

That means a thousand posts per year. Not so much, if you think about it.

MATHS!


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5018 2015-05-06 18:57:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
nbhs wrote:

And it's also exaclty 2 years old

That means a thousand posts per year. Not so much, if you think about it.

MATHS!

Crap. 2500. Still not so much.

Damn, that was awkward.

BTW, 772 of them are yours, 234 are mine.


Aaaalso, cruising through your blogposts.. libvirt doesn't support x-vga YET? Or there was a claim that it won't support it at all? That part 5 with selinux looks strange..

Last edited by Duelist (2015-05-06 19:04:45)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5019 2015-05-06 19:29:34

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw: I looked further into your post and checked my system and... it appears that EDK2/OVMF doesn't detect the FAT filesystems on my ISOs. I'm only getting BLK mappings, no FS mappings, and therefore can't boot since it can't read EFI partitions..
Could you tell me which version you've been using ? I believe i'm on the latest from Jenkins. Stable releases don't appear to boot at all here.

Offline

#5020 2015-05-06 19:54:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just found an article with benchmarks of Citrix XenApp+Grid K2 vGPU.

BEWARE, IT'S IN RUSSIAN.

Anyway, there's 1131/1183 pts in Unigine on GTX 650, and ~1400pts on K2 per instance.
So, like, get an expensive K2, and you can do like 4 VMs comparable in performance as... my setup on VFIO+2 HD7750. I get ~1200 when using crossfire. Duct-tape TightVNC server to it, provide tons of bandwidth, and voila.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5021 2015-05-06 20:03:51

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've seen some people here have GT610 with EFI support. Can anybody share the rom? My card comes only with VGA and I can't contact the vendor's support to get a newer rom from them.

Offline

#5022 2015-05-06 20:51:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Aaaalso, cruising through your blogposts.. libvirt doesn't support x-vga YET? Or there was a claim that it won't support it at all? That part 5 with selinux looks strange..

The x- in front of the x-vga option is a big warning flag to libvirt to not support this feature because it's experimental.  It's experimental because VGA arbitration is so broken on Linux, mostly with the i915 driver.  I don't see how that's going to get fixed, so I don't have any immediate plans to make it non-experimental and I hope that with UEFI becoming more common, VGA may just go away.  Last I tried the selinux stuff worked, I didn't re-try it for that post though.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5023 2015-05-06 20:52:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fira wrote:

@aw: I looked further into your post and checked my system and... it appears that EDK2/OVMF doesn't detect the FAT filesystems on my ISOs. I'm only getting BLK mappings, no FS mappings, and therefore can't boot since it can't read EFI partitions..
Could you tell me which version you've been using ? I believe i'm on the latest from Jenkins. Stable releases don't appear to boot at all here.

I'm on the latest from Gerd's repo too, I did a fresh update before doing that demo.  If you've got it setup like me, I'd suspect your media vs the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5024 2015-05-06 21:21:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Just found an article with benchmarks of Citrix XenApp+Grid K2 vGPU.

BEWARE, IT'S IN RUSSIAN.

Anyway, there's 1131/1183 pts in Unigine on GTX 650, and ~1400pts on K2 per instance.
So, like, get an expensive K2, and you can do like 4 VMs comparable in performance as... my setup on VFIO+2 HD7750. I get ~1200 when using crossfire. Duct-tape TightVNC server to it, provide tons of bandwidth, and voila.

A GTX 650 has 384 CUDA cores, a GRID K2 has two separate processor cores, each with 1536 CUDA cores.  Each core is really therefore more equivalent to a GTX 680.  According to the article (google translation), a K280Q configuration is effectively giving each VM a full GPU core, K260Q is half a core, K240Q is a quarter core.  So only in that last config is the core count equivalent to a GTX 650.  Also factor in that a GTX 650 is about $75 USD and a K2 is about $3000 USD and GeForce biting into those GRID margins, if we can run it in a VM, might make you suspect that Nvidia really is trying to prevent GeForce assignment.  But look at it from the other side that it would be completely ridiculous to put eight GTX 650s into a system while a GRID card is only a single double-slot card and you'd think that they'd embrace GeForce assignment as a gateway to a realistic density and far more flexible solution with GRID.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5025 2015-05-06 21:36:51

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

Just found an article with benchmarks of Citrix XenApp+Grid K2 vGPU.

BEWARE, IT'S IN RUSSIAN.

Anyway, there's 1131/1183 pts in Unigine on GTX 650, and ~1400pts on K2 per instance.
So, like, get an expensive K2, and you can do like 4 VMs comparable in performance as... my setup on VFIO+2 HD7750. I get ~1200 when using crossfire. Duct-tape TightVNC server to it, provide tons of bandwidth, and voila.

A GTX 650 has 384 CUDA cores, a GRID K2 has two separate processor cores, each with 1536 CUDA cores.  Each core is really therefore more equivalent to a GTX 680.  According to the article (google translation), a K280Q configuration is effectively giving each VM a full GPU core, K260Q is half a core, K240Q is a quarter core.  So only in that last config is the core count equivalent to a GTX 650.  Also factor in that a GTX 650 is about $75 USD and a K2 is about $3000 USD and GeForce biting into those GRID margins, if we can run it in a VM, might make you suspect that Nvidia really is trying to prevent GeForce assignment.  But look at it from the other side that it would be completely ridiculous to put eight GTX 650s into a system while a GRID card is only a single double-slot card and you'd think that they'd embrace GeForce assignment as a gateway to a realistic density and far more flexible solution with GRID.

I agree. I've tried to reply or add something, but failed multiple times.


deniv wrote:

I've seen some people here have GT610 with EFI support. Can anybody share the rom? My card comes only with VGA and I can't contact the vendor's support to get a newer rom from them.

Using envytools' nvagetbios -s prom, here's my GT610's VBIOS. It's a Gigabyte Gigabyte(sic!) GT610.
After 90 days the file will be automagically deleted, sorry about that.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5026 2015-05-06 22:42:06

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, after lots of testing and much annoying of the EDK2 developpers, i realized that my DreamSpark Win8.1 and Win7 ISO are in UDF Format - which is currently not supported by EDK2/OVMF. I'm still not sure to understand why in 200 pages, there's been only one other occurence of the problem reported, I can only assume the majority of the Win8 ISOs are in fact no UDF...

After trying to convert the image to FAT32 and ISO9660 i got bootable images on the EFI side, only to get hit by a crash on the Windows Bootloader.
I just realized I could work around this whole mess by making ... a bootable USB key and ... passing it through to the VM... Yeah.
Leaving this here in case the experiment goes wrong. I'll report on wether it worked or not if I'm still alive by then.

Offline

#5027 2015-05-06 22:46:01

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I recently switched to i3 for everyday use, and I noticed my virtual machine has no sound anymore.
Pulseaudio seems to be running, I can correctly change volumes and outputs with pavucontrol, and every other applications plays  sound normally.

However when I switch to a full-on DE, Mate in my case, I do have sound in the vm. I'm not sure what's up here.

My launch script:

QEMU_AUDIO_DRV=all qemu-system-x86_64 -L . -enable-kvm -m 8096 -cpu host,kvm=off -smp 6,sockets=1,cores=6,threads=1 -vga none \ -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -device vfio-pci,host=00:19.0 -device vfio-pci,host=04:00.0 \ -device virtio-scsi-pci,id=scsi1 -drive file=/dev/sdb,id=ssddisk,format=raw,if=none -device scsi-hd,drive=ssddisk \ -device virtio-scsi-pci,id=scsi2 -drive file=/dev/sdd,id=hdddisk,format=raw,if=none -device scsi-hd,drive=hdddisk \ -usb -usbdevice host:0738:2215 -usbdevice host:0738:a215 -usbdevice host:13d3:3404 -usbdevice host:2516:0011 -usbdevice host:1532:0033 \ -usbdevice host:054c:0268 -soundhw hda

Offline

#5028 2015-05-06 22:58:56

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fira wrote:

I'm still not sure to understand why in 200 pages, there's been only one other occurence of the problem reported, I can only assume the majority of the Win8 ISOs are in fact no UDF...

...because this thread isn't related to "installing windowns into QEMU virtual machine"?

And why do i have the habit of tearing apart windowns' install media and actually understanding how it's supposed to boot?..
You can tear that darn install media into two partitions on two separate disks, and it'll still work, if you put enough duct tape everywhere. Actually, that's how it works when using the USB thumb drive method.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5029 2015-05-06 23:30:27

Fira
Member
Registered: 2011-09-24
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Fira wrote:

I'm still not sure to understand why in 200 pages, there's been only one other occurence of the problem reported, I can only assume the majority of the Win8 ISOs are in fact no UDF...

...because this thread isn't related to "installing windowns into QEMU virtual machine"?

I'm not sure what your problem is or why you're even saying this. 
Am just reporting that there are potential, although uncommon, issues with Windows 7/8 installation on OVMF due to unusual ISOs, which is pretty much a requirement for effective VGA passthrough ?

Duelist wrote:

And why do i have the habit of tearing apart windowns' install media and actually understanding how it's supposed to boot?..

I'm terribly sorry, i thought this was a Community Forum, I clearly didn't realize this was your personal mailbox.

Edit: If anyone even cares about the whole UDF mess,
- Microsoft's USB Creator won't take the ISO because it's non-standard
- LiLi USB Creator is not UDF aware and just prints out a README
- Rufus seems to actually do the job, up to the moment you try to boot it and it doesn't work

Last edited by Fira (2015-05-06 23:39:49)

Offline

#5030 2015-05-07 02:06:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fira wrote:

Edit: If anyone even cares about the whole UDF mess,
- Microsoft's USB Creator won't take the ISO because it's non-standard
- LiLi USB Creator is not UDF aware and just prints out a README
- Rufus seems to actually do the job, up to the moment you try to boot it and it doesn't work

Where did you even get this image? DreamSpark you say?.. Isn't that Microsoft's Live File System images, intended for something like live, bootable images? I mean, technically you can add files to the burned disc, that would be somewhat useful for a live system or some weird updates of the install media.
Can you just extract all the files from that image, and make ESP(containing /BOOT/BOOTX64.EFI or how was that file should be named) and data partition manually? That should be somewhat simpler to do than dealing with various usb creating software.

If you have dreamspark access, why you didn't go straight to MSDN? MSDN is known to supply normal, ISO9660 images. God damn microsoft, subscription is paid-only.

UPD:Looking at the images, there's Server2012R2 with ISO9660 and UDF1.02 available on dreamspark. That looks weird.
Seems like the only way around is to tear them up manually.

Fira wrote:

Windows 7/8 installation on OVMF due to unusual ISOs, which is pretty much a requirement for effective VGA passthrough

Why, you could always install linux into your VM, it'd just be another usecase of this tech. For example, i need to provide a remote workplace for some 3D-guy working in blender and i don't like buying citrix xenapp or vsphere or whatever... ... darn, i should definitely try doing that to see the bandwidth issues.

Last edited by Duelist (2015-05-07 10:25:36)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5031 2015-05-07 11:06:03

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
deniv wrote:

I've seen some people here have GT610 with EFI support. Can anybody share the rom? My card comes only with VGA and I can't contact the vendor's support to get a newer rom from them.

Using envytools' nvagetbios -s prom, here's my GT610's VBIOS. It's a Gigabyte Gigabyte(sic!) GT610.
After 90 days the file will be automagically deleted, sorry about that.

Thanks a bunch! It works with my https://www.asus.com/us/Graphics_Cards/GT610SL1GD3L/ So, yet another working win7 efi setup.

A pity rebooting doesn't work (I think I have the same problem as you). Perhaps it's somehow related to the VBIOS. Anybody else mind sharing their VBIOS for gt610?

P.S. Wish QXL wouldn't stop working when a passed card is active. It works in win 8.1! I get two screens one on a physical card and one on a qxl card. Why win 7 can't do it? :x

Offline

#5032 2015-05-07 16:55:10

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

ist it only possible to passthrough whole iommu groups?
VGA passthrough works like charm for two months now.
Recently i´ve experimenting to get dual boot to work with a native installed Win7 on /dev/sdg.
So i have to passthrough ASM1602 SATA controller.
The problem is sde and sdf shoud be accessible on host.

ls -l /sys/block/ | grep sd. lrwxrwxrwx 1 root root 0 7. Mai 16:53 sda -> ../devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda lrwxrwxrwx 1 root root 0 7. Mai 16:53 sdb -> ../devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb lrwxrwxrwx 1 root root 0 7. Mai 16:53 sdc -> ../devices/pci0000:00/0000:00:1f.2/ata3/host2/target2:0:0/2:0:0:0/block/sdc lrwxrwxrwx 1 root root 0 7. Mai 16:53 sdd -> ../devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sdd lrwxrwxrwx 1 root root 0 7. Mai 16:53 sde -> ../devices/pci0000:00/0000:00:1c.3/0000:04:00.0/0000:05:03.0/0000:07:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sde lrwxrwxrwx 1 root root 0 7. Mai 16:53 sdf -> ../devices/pci0000:00/0000:00:1c.3/0000:04:00.0/0000:05:03.0/0000:07:00.0/ata8/host7/target7:0:0/7:0:0:0/block/sdf lrwxrwxrwx 1 root root 0 7. Mai 16:53 sdg -> ../devices/pci0000:00/0000:00:1c.3/0000:04:00.0/0000:05:07.0/0000:09:00.0/ata9/host8/target8:0:0/8:0:0:0/block/sdg lrwxrwxrwx 1 root root 0 7. Mai 17:43 sdh -> ../devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5:1.0/host10/target10:0:0/10:0:0:0/block/sdh

This is the big Z97 group

### Group 9 ### 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 05:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 0a:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

Is it somehow possible to passthrough only 09:00.0?

Thank you in advance

Offline

#5033 2015-05-07 17:01:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

9-series PCH root port quirks will probably go in for kernel 4.2


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5034 2015-05-07 17:50:29

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:
Duelist wrote:
deniv wrote:

I've seen some people here have GT610 with EFI support. Can anybody share the rom? My card comes only with VGA and I can't contact the vendor's support to get a newer rom from them.

Using envytools' nvagetbios -s prom, here's my GT610's VBIOS. It's a Gigabyte Gigabyte(sic!) GT610.
After 90 days the file will be automagically deleted, sorry about that.

Thanks a bunch! It works with my https://www.asus.com/us/Graphics_Cards/GT610SL1GD3L/ So, yet another working win7 efi setup.

A pity rebooting doesn't work (I think I have the same problem as you). Perhaps it's somehow related to the VBIOS. Anybody else mind sharing their VBIOS for gt610?

P.S. Wish QXL wouldn't stop working when a passed card is active. It works in win 8.1! I get two screens one on a physical card and one on a qxl card. Why win 7 can't do it? :x

Rebooting doesn't work?.. Haven't heard about it. My Radeon-based VM system reboots just fine.
Anyway, i can conclude that Asus can not into UEFI properly.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5035 2015-05-07 19:11:37

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
deniv wrote:
Duelist wrote:

Using envytools' nvagetbios -s prom, here's my GT610's VBIOS. It's a Gigabyte Gigabyte(sic!) GT610.
After 90 days the file will be automagically deleted, sorry about that.

Thanks a bunch! It works with my https://www.asus.com/us/Graphics_Cards/GT610SL1GD3L/ So, yet another working win7 efi setup.

A pity rebooting doesn't work (I think I have the same problem as you). Perhaps it's somehow related to the VBIOS. Anybody else mind sharing their VBIOS for gt610?

P.S. Wish QXL wouldn't stop working when a passed card is active. It works in win 8.1! I get two screens one on a physical card and one on a qxl card. Why win 7 can't do it? :x

Rebooting doesn't work?.. Haven't heard about it.

I recall somebody in this topic said their GT610 didn't survive reboots. I thought it was you, but it seems like I was wrong.

My Radeon-based VM system reboots just fine.

My Radeon-based VM reboots fine as well.

Anyway, i can conclude that Asus can not into UEFI properly.

But I used your Gigabyte UEFI. big_smile Anyway, ASUS does have a windows-only utility to update vbios; I just can't use it since I have no bare-metal Windows installs.

Offline

#5036 2015-05-07 19:33:50

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

So I recently switched to i3 for everyday use, and I noticed my virtual machine has no sound anymore.
Pulseaudio seems to be running, I can correctly change volumes and outputs with pavucontrol, and every other applications plays  sound normally.

However when I switch to a full-on DE, Mate in my case, I do have sound in the vm. I'm not sure what's up here.

My launch script:

QEMU_AUDIO_DRV=all qemu-system-x86_64 -L . -enable-kvm -m 8096 -cpu host,kvm=off -smp 6,sockets=1,cores=6,threads=1 -vga none \ -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -device vfio-pci,host=00:19.0 -device vfio-pci,host=04:00.0 \ -device virtio-scsi-pci,id=scsi1 -drive file=/dev/sdb,id=ssddisk,format=raw,if=none -device scsi-hd,drive=ssddisk \ -device virtio-scsi-pci,id=scsi2 -drive file=/dev/sdd,id=hdddisk,format=raw,if=none -device scsi-hd,drive=hdddisk \ -usb -usbdevice host:0738:2215 -usbdevice host:0738:a215 -usbdevice host:13d3:3404 -usbdevice host:2516:0011 -usbdevice host:1532:0033 \ -usbdevice host:054c:0268 -soundhw hda

Shouldn't "QEMU_AUDIO_DRV=alsa" set it to alsa? Even that doesn't seem to work while, again, in Mate it works.
i3 shouldn't be messing with my sound right?

Last edited by PureTryOut (2015-05-07 21:58:32)

Offline

#5037 2015-05-08 15:14:03

wulfspider
Member
Registered: 2015-05-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again,

OVMF works. Started the ovmf with the qemu command and ovmf and TianoCore appears. But after I added ovmf to my libvirt xml I get "Permission denied". My OVMF is self compiled

XML config:

... <os> .... <loader readonly='yes' type='pflash'>/opt/ovmf/OVMF_CODE.fd</loader> <nvram template='/opt/ovmf/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd</nvram> .... </os> ...

Error:

qemu-system-x86_64: -drive file=/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd,if=pflash,format=raw,unit=1: Could not open '/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd': Permission denied

ls -lah:

drwxr-x--- 7 libvirt-qemu kvm 4,0K Mai 8 17:11 .. -rwxr-x--- 1 root root 128K Mai 6 11:29 win8.1_VARS.fd

Last edited by wulfspider (2015-05-08 15:17:20)

Offline

#5038 2015-05-08 15:23:44

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Remove /var/lib/libvirt/qemu/nvram/win8.1_VARS.fd, which will leave:

<nvram template='/opt/ovmf/OVMF_VARS.fd'></nvram>

When you start the VM, it will generate a new nvram file.

Offline

#5039 2015-05-08 15:27:49

wulfspider
Member
Registered: 2015-05-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hm, I removed the template parameter but nothing has changed.

I also changed the permissions of this file from root:root to libvirt-qemu:kvm, but after I executed virsh start win8.1 the permissions are resetted to root:root

Last edited by wulfspider (2015-05-08 15:28:09)

Offline

#5040 2015-05-08 16:44:41

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

i just need a way to get win7 running from another disk currently "sdg", configured by vmm or xml. I´m not sure if there is a need for me to passthrough SATA controller at all.

Or pass through the other SATA controller from Intel?

### Group 11 ### 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller

and plug the disks

lrwxrwxrwx 1 root root 0 8. Mai 17:30 sda -> ../devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda lrwxrwxrwx 1 root root 0 8. Mai 17:30 sdb -> ../devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb lrwxrwxrwx 1 root root 0 8. Mai 17:30 sdc -> ../devices/pci0000:00/0000:00:1f.2/ata3/host2/target2:0:0/2:0:0:0/block/sdc lrwxrwxrwx 1 root root 0 8. Mai 17:30 sdd -> ../devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sdd

on the ASM1062 witch failed yesterday.

There are just too many disks to wire.
Just need the easiest solution, because Group 9 passthrough isn´t working. VirtIO disks worked perfectly for me in the past.

Anyone?

Last edited by Schlunze (2015-05-08 16:49:18)

Offline

#5041 2015-05-08 17:14:46

PureTryOut
Member
Registered: 2014-09-23
Posts: 63

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PureTryOut wrote:

Shouldn't "QEMU_AUDIO_DRV=alsa" set it to alsa? Even that doesn't seem to work while, again, in Mate it works.
i3 shouldn't be messing with my sound right?

Well I found out why I had no sound with Pulseaudio, still not sure why it works fine in Mate though.
Pulseaudio is running as the current user, but i'm launching Qemu with sudo rights.

I'm launching Qemu with sudo because I boot directly from hard disk (to get full ssd speeds), to open these I need sudo rights. I guess I could chown the disks to my user every time but I rather not do that.
Also, even when chowning it to my user, and then launching Qemu without root, it complains about not being able to open /dev/vfio/16 because it's missing permissions.

I thought switching to OVMF might help (do note, i'm a noob in all of this), but of course it didn't.

So how can I run Qemu as root, while still using Pulseaudio?

Offline

#5042 2015-05-09 00:57:06

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

deniv wrote:

I recall somebody in this topic said their GT610 didn't survive reboots. I thought it was you, but it seems like I was wrong.

It is me sad

Been trying with this cursed card for more than a month , and nothing would fix it .

The issue is when you reboot the VM , it succeeds , you can see OVMF screen and Windows loading logo then an infinite black screen . Hopefully , I'll get another newer card soon .

Can anyone suggest a newer architecture SINGLE-SLOT card from NVIDIA ? Couldn't find anything other than a GTX 750 .

Offline

#5043 2015-05-09 10:47:52

deniv
Member
Registered: 2013-10-16
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

The issue is when you reboot the VM , it succeeds , you can see OVMF screen and Windows loading logo then an infinite black screen .

Yep, I have the same problem. And even the host hangs when the VM reboots.

Offline

#5044 2015-05-09 12:23:12

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

The issue is when you reboot the VM , it succeeds , you can see OVMF screen and Windows loading logo then an infinite black screen .

Idk if this will be of any help but similar thing happens when emulated vga is used as primary and physical vga is used as secondary. windows logo would be seen on emulated vga window and then it goes black as windows switches to more powerful physical vga. maybe something similar is happening there?

Offline

#5045 2015-05-09 14:03:07

wulfspider
Member
Registered: 2015-05-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can somebody tell me if my hardware supports VT-d? Or how can I check this? I don't get working the pass through and I am slowly dispairing.

Motherboard: Asus P8Z77-V Pro
Processor: i7-3770

I know that my processor and the Z77 chipset support VT-d, but don't know if it really works or if Asus disabled this function in my model. Under the CPU section in my bios I enabled "Intel Virtualisation technology". Is this the same? Another Option like "VT-d" is not available.

Last edited by wulfspider (2015-05-09 14:13:52)

Offline

#5046 2015-05-09 17:37:46

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

wulfspider wrote:

Can somebody tell me if my hardware supports VT-d? Or how can I check this? I don't get working the pass through and I am slowly dispairing.

Motherboard: Asus P8Z77-V Pro
Processor: i7-3770

I know that my processor and the Z77 chipset support VT-d, but don't know if it really works or if Asus disabled this function in my model. Under the CPU section in my bios I enabled "Intel Virtualisation technology". Is this the same? Another Option like "VT-d" is not available.

/proc/cpuinfo flags #

/proc/cpuinfo will tell you if the processor supports virtualization and if it is enabled (but it could not show up in flags because it is disabled — more on this below).

The flags to look out for are:

    vmx — Intel VT-x, basic virtualization
    svm — AMD SVM, basic virtualization
    ept — Extended Page Tables, an Intel feature to make emulation of guest page tables faster.
    vpid — VPID, an Intel feature to make expensive TLB flushes unnecessary when context switching between guests.
    npt — AMD Nested Page Tables, similar to EPT.
    tpr_shadow and flexpriority — Intel feature that reduces calls into the hypervisor when accessing the Task Priority Register, which helps when running certain types of SMP guests.
    vnmi — Intel Virtual NMI feature which helps with certain sorts of interrupt events in guests.

from http://virt-tools.org/learning/check-hardware-virt/

Offline

#5047 2015-05-09 18:10:27

stokbaek
Member
Registered: 2014-09-20
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I desided to give this a go since I had to reinstall pc anyways.

I managed to get myself stuck and hope some of you guys would be able to help out of my problem. My issue is when I start the vm does a QUEMU window open but I never get any signal on the screen that is in the GFX card that is linked to it.
If I add the following line to the vmcreate script am I able to VNC into the VM. It can see the gfx card and installed the drivers for it, but still aint able to get its screen working even after I have installed the video driver to it.

-vnc :0 -vga vmware

This PC is running Linux Mint 17.1
kernel:

3.13.0-37-generic

PC Setup:

 description: Motherboard product: B75M-D3H vendor: Gigabyte Technology Co., Ltd. physical id: 0 version: x.x serial: To be filled by O.E.M. slot: To be filled by O.E.M. description: CPU product: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz vendor: Intel Corp. physical id: 43 bus info: cpu@0 version: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz slot: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz size: 3801MHz capacity: 3801MHz width: 64 bits clock: 100MHz

/etc/modules:

lp rtc pci_stub vfio vfio_iommu_type1 vfio_pci kvm kvm_intel

/etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"

lspci -nn | grep NVIDIA:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

iniramfs/modules:

pci_stub ids=10DE:0E0A,10DE:1187

dmesg | grep pci-stub:

[ 2.737698] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.737712] pci-stub 0000:01:00.1: claimed by stub [ 2.737718] pci-stub: add 10DE:1187 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000 [ 2.737721] pci-stub 0000:01:00.0: claimed by stub

/etc/vfio-pci1.cfg:

0000:01:00.0 0000:01:00.1

/usr/vm1:

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/media/christoffers/Games/vm_machines/windows8.1.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/christoffers/Downloads/windows8.1.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on exit 0

kvm-ok:

INFO: /dev/kvm exists KVM acceleration can be used

Please do tell if I need to supply more information

Last edited by stokbaek (2015-05-09 19:58:30)

Offline

#5048 2015-05-09 21:30:27

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

Idk if this will be of any help but similar thing happens when emulated vga is used as primary and physical vga is used as secondary. windows logo would be seen on emulated vga window and then it goes black as windows switches to more powerful physical vga. maybe something similar is happening there?

That's not the case , I am passing through the GT610 as the primary and only GPU to the VM (-vga none) .

Maybe it will work if I passed it as a secondary one tongue Let's try and see !

EDIT : Nope . Gonna replace it with an ASUS GTX 750 soon .

Last edited by Denso (2015-05-09 22:56:27)

Offline

#5049 2015-05-10 00:11:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

EDIT : Nope . Gonna replace it with an ASUS GTX 750 soon .

There was a passively-cooled version of GTX750, and maybe even 750Ti from Zotac. Sorry for my obsession, none of my GPUs are actively cooled..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5050 2015-05-10 00:44:01

Omar007
Member
Registered: 2015-04-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been moving from QEMU bash scripts to libvirt/virt-manager and have had no problems with the main setup in either yet.

Since the basic setup worked without issues for me, I tried to create some more exotic configurations and use cases. What I've found during the process:
- The latest nVidia host drivers (349.16-2) can not be used if you want to passthrough a GPU (at least not without modifications and I don't know how to do that).
- The 'intel_iommu=on' kernel parameter may prevent the system from booting into X if you are using *.conf files for X (though maybe this depends on hardware).
- If X is running when you unbind or rebind a GPU, it'll freeze your whole system.
- Stubbing or binding to vfio-pci is not needed when using libvirt/virt-manager (it binds the GPU when you start the VM and unbinds/rebinds to host after it shuts down)
- SLI is currently not possible on either chipset (at least I did not manage to get this working. I'll be happy to hear if someone did!).
- Standard sound configuration of libvirt doesn't work if the VM is started without an active X session (the sound won't play over the host speakers).
- Onboard audio unbinding and rebinding works perfectly and can be done during active X sessions as well.
- When emulating an audio device it is not possible to define how much channels it should have?? (ie. 2.1, 5.1 or 7.1 instead of the default stereo)
- Windows 8.1 will not boot with the Q35 chipset (it freezes during boot at the logo).
- No working QXL/Spice drivers available for Windows 8.1 yet so you get a wonderful 800x600 resolution!
- The OVMF version on extra stops the VM from rebooting (it just shows a white _ in the top left). The ovmf-svn package on the AUR works fine and rebooting works.


Used hardware
MoBo: ASUS X99-A
GPU: GeForce GTX 970 (x2)
CPU: Intel i7 5930K
UEFI boot mode only (CSM/Legacy support disabled)

Software:
- kernel 4.0.1-1 (latest on core)
- qemu 2.2.1-4 (latest on extra)
- ovmf-svn 17281-1 (aur)
- libvirt 1.2.15-1 (latest on community)
- virt-manager 1.2.0-1 (latest on community)

Last edited by Omar007 (2015-05-10 01:02:06)

Offline

#5051 2015-05-10 01:01:00

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Omar007 wrote:

Used hardware
MoBo: ASUS X99-A

My greatest mistake since my virtualization journey started was to "upgrade" to X99-Deluxe from ASUS . period .

My previous 990FXA from GigaByte was rock stable for passing through GPUs . I got rid of it because the AMD FX-8350 was causing most heavy games to lag , and emulators were a nightmare to play on .

Offline

#5052 2015-05-10 01:06:30

Omar007
Member
Registered: 2015-04-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How so? What problems did you face? Maybe it was just bad luck and you received a bad board?

So far I've not had any real problems with mine. Unless one of the points I mentioned is caused by the board but I kinda doubt that tbh.

I wish Nouveau worked properly with my cards already! Or that nVidia releases a driver version for Wayland and which doesn't take possession of the VGA arbitration thingie.
I'd really like to give Wayland a shot to see how that fares with dynamic unbinding and rebinding of GPUs tongue

Last edited by Omar007 (2015-05-10 01:09:23)

Offline

#5053 2015-05-10 02:17:18

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Omar007 wrote:

How so? What problems did you face? Maybe it was just bad luck and you received a bad board?

Well , I bought it 2 months after its release . It used to crash randomly ALOT during normal operation on my Windows VM . Secondly , when I moved to this platform my GT610 started to have reset issues and I am certain that it didn't have any issues running atop 990FXA .

Anyway , to be honest , recent ASUS updates fixed 99% of the stability issues (No more random crashing) .

My plan is IF (a big IF here) AMD's Zen performed near Haswell's single-threaded performance , I'll seriously consider moving to Zen+ (Zen's second gen.) platfrom .

Cheers ! smile

Offline

#5054 2015-05-10 23:23:52

mikeabrahamsen
Member
Registered: 2013-04-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone been able to get SLI working? I have 2 gtx 780ti cards being passed through to a windows guest. The cards are recognize by the OS but the nvidia control panel does not have the option to enable SLI (however I can set one card as a PhysX processor). Any suggestions?

Hardware:
MoBo: Asus x99 Deluxe
Processor: Intel i7 5820k
Host Graphics: AMD R7 240
Guest Graphics: 2x Nvidia gtx 780ti

Currently using virt-manager to configure vm.

Offline

#5055 2015-05-11 00:11:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mikeabrahamsen wrote:

Has anyone been able to get SLI working? I have 2 gtx 780ti cards being passed through to a windows guest. The cards are recognize by the OS but the nvidia control panel does not have the option to enable SLI (however I can set one card as a PhysX processor). Any suggestions?

Hardware:
MoBo: Asus x99 Deluxe
Processor: Intel i7 5820k
Host Graphics: AMD R7 240
Guest Graphics: 2x Nvidia gtx 780ti

Currently using virt-manager to configure vm.

I can't remember anyone running SLI for, well, 75 pages at least.
Anyway, i think nvidia drivers detect your VM as non SLI-compatible motherboard and refuse to work.
It happens on physical machines too, so there's a crutch for that: HyperSLI. Maybe it will work if you'll enable nested virtualization. Or maybe it won't.

In the broad scale, i can't see any obvious technical reasons for it to not work - i've got my XDMA crossfire working since a long time. Maybe aw will tell something, i don't know.

Glad to see you have both cards running independently, at least.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5056 2015-05-11 16:25:49

mikeabrahamsen
Member
Registered: 2013-04-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I can't remember anyone running SLI for, well, 75 pages at least.
Anyway, i think nvidia drivers detect your VM as non SLI-compatible motherboard and refuse to work.
It happens on physical machines too, so there's a crutch for that: HyperSLI. Maybe it will work if you'll enable nested virtualization. Or maybe it won't.

In the broad scale, i can't see any obvious technical reasons for it to not work - i've got my XDMA crossfire working since a long time. Maybe aw will tell something, i don't know.

Glad to see you have both cards running independently, at least.

Thanks for the response. I tried out nested virtualization with both HyperSLI and DifferentSLI (seems to be what most people were using in more recent posts from the thread you linked). Neither worked for me, it gave me another avenue to try at least. For now I will just settle for using one card.

Offline

#5057 2015-05-11 18:00:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mikeabrahamsen wrote:

For now I will just settle for using one card.

Or you could run two VMs simultaneously to try to "emulate" GRID K2 setup. Oh, btw, i think it's possible to quadrify your GPUs, check that. That may open you some possibilities.

Last edited by Duelist (2015-05-11 18:01:19)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5058 2015-05-12 02:17:34

strunberg
Member
Registered: 2015-05-12
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, this is my first post because I came to this forum because of this thread.

https://www.pugetsystems.com/labs/artic … 4-KVM-585/
https://www.pugetsystems.com/labs/artic … Setup-564/

I'm going to  send some emails to AMD,Intel,GPU manufactures,and the motherboard manufactures.

I want to create a database of computer component compatibility to make things easier for my self,others,and to pad my resume.

What compatibility questions(Which of your mother board support X?) should I ask ?

Please list everything that I need to ask.

Offline

#5059 2015-05-12 08:32:39

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There is google spreadsheet of known working hardware in this thread.

https://docs.google.com/spreadsheet/ccc … _web#gid=0

Offline

#5060 2015-05-12 16:00:03

bierbauch
Member
Registered: 2015-03-28
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all,
After I showed my current setup around I now have the honor to play with a different config...

The system has two GTX 970. One is supposed to work with the host machine and one is for a Windows guest. Since both have the same vendor ID (10de:13c2,...) I'm expecting this to not work correctly, or is there a way to bind the specific device to linux after boot. Or can I filter with PCI stub for the spot ID in addition to the vendor one?

Pls excuse my grammar, in my the phone ...

Best regards,
Beerbelly

Offline

#5061 2015-05-12 16:38:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bierbauch wrote:

Hey all,
After I showed my current setup around I now have the honor to play with a different config...

The system has two GTX 970. One is supposed to work with the host machine and one is for a Windows guest. Since both have the same vendor ID (10de:13c2,...) I'm expecting this to not work correctly, or is there a way to bind the specific device to linux after boot. Or can I filter with PCI stub for the spot ID in addition to the vendor one?

Pls excuse my grammar, in my the phone ...

http://vfio.blogspot.com/2015/05/vfio-g … -host.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5062 2015-05-12 21:53:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
bierbauch wrote:

Hey all,
After I showed my current setup around I now have the honor to play with a different config...

The system has two GTX 970. One is supposed to work with the host machine and one is for a Windows guest. Since both have the same vendor ID (10de:13c2,...) I'm expecting this to not work correctly, or is there a way to bind the specific device to linux after boot. Or can I filter with PCI stub for the spot ID in addition to the vendor one?

Pls excuse my grammar, in my the phone ...

http://vfio.blogspot.com/2015/05/vfio-g … -host.html

Y U NO XEN PCI-BACK?

Some users have found the xen-pciback module to be a suitable stand-in for pci-stub with the additional feature that the "hide" option for this module takes device addresses rather than device IDs.  I can't load this module on Fedora, so here's my solution that I like a bit better.

Oh.. I guess you need to install xen for that, right? Is it possible to tear that module from xen and make it something like pci-id-stub or vfio-stub or something?

Last edited by Duelist (2015-05-12 21:57:26)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5063 2015-05-12 22:16:26

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

I saw that you used a "generic" pattern to bind every NVIDIA & AMD devices to VFIO without the need to specify the exact IDs :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

Can I use that and SKIP the binding script altogether ? I know my question is confusing , basically what I'm doing now is this :

1 - Binding the devices to pci-stub.ids from the bootloader .

2 - Binding the same devices to VFIO using the binding script (from OP) like this :

vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1 0000:05:00.0 0000:06:00.0 0000:07:00.0 0000:08:00.0 0000:00:1b.0 0000:00:19.0

3 - Starting VMs .

Wouldn't steps 1&2 become unnecessary if I use :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

in modprobe.d ?

A second question if I may :

Can I break the "options vfio-pci ids=" into multiple lines ? like this :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=8086:8d20,1912:0015,8086:15a1

Thanks !

Last edited by Denso (2015-05-12 22:22:49)

Offline

#5064 2015-05-12 22:38:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

I saw that you used a "generic" pattern to bind every NVIDIA & AMD devices to VFIO without the need to specify the exact IDs :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

Can I use that and SKIP the binding script altogether ? I know my question is confusing , basically what I'm doing now is this :

1 - Binding the devices to pci-stub.ids from the bootloader .

2 - Binding the same devices to VFIO using the binding script (from OP) like this :

vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1 0000:05:00.0 0000:06:00.0 0000:07:00.0 0000:08:00.0 0000:00:1b.0 0000:00:19.0

3 - Starting VMs .

Wouldn't steps 1&2 become unnecessary if I use :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

in modprobe.d ?

Yes, vfio-pci.ids option added in 4.1 is meant to negate the need to use pci-stub and bind directly to vfio-pci.  The only difference is that pci-stub is often built statically into the kernel while vfio-pci is more typically loaded as a module, so you'll need to configure your initramfs to include all the vfio modules and trigger it to load vfio-pci (the dependencies should all autoload the other modules).

A second question if I may :

Can I break the "options vfio-pci ids=" into multiple lines ? like this :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=8086:8d20,1912:0015,8086:15a1

Thanks !

AFAIK, no


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5065 2015-05-12 22:46:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Y U NO XEN PCI-BACK?

Some users have found the xen-pciback module to be a suitable stand-in for pci-stub with the additional feature that the "hide" option for this module takes device addresses rather than device IDs.  I can't load this module on Fedora, so here's my solution that I like a bit better.

Oh.. I guess you need to install xen for that, right? Is it possible to tear that module from xen and make it something like pci-id-stub or vfio-stub or something?

The problem that xen ignores with the pciback module is that the kernel has every right to re-number your PCI bus.  What's at 01:00.0 now could be at 02:00.0 in the next kernel release if a tweak were made to bus enumeration.  Things like the VT-d spec take this into account, providing paths to devices from known starting points, like the root bus where the bus number is fixed by the platform firmware.  So while the xen-pciback method of picking devices usually works and nobody is likely to change PCI bus enumeration for the fun of it, it's flawed and I'm not willing to propose it as an alternative claiming method for vfio-pci.  Besides, I showed in the blog post how trivial it is to create a script that moves the problem to userspace and makes use of a less problematic interface than binding drivers by PCI device IDs.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5066 2015-05-12 22:47:51

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Yes, vfio-pci.ids option added in 4.1 is meant to negate the need to use pci-stub and bind directly to vfio-pci.  The only difference is that pci-stub is often built statically into the kernel while vfio-pci is more typically loaded as a module, so you'll need to configure your initramfs to include all the vfio modules and trigger it to load vfio-pci (the dependencies should all autoload the other modules).

Thanks , I did add vfio-pci to initramfs (mkinitcpio.conf in Arch) . Pci-stub is also a module in Arch's stock kernel . I had to add it to initramfs too .

aw wrote:

AFAIK, no

Thanks ! smile

EDIT :

Breaking the "options vfio-pci ids=" to multiple lines is possible . I just broke mine to 3 lines , rebooted and everything is working like a charm :

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=8086:8d20,1912:0015,8086:15a1

This makes adding devices and maintaining the binding list a lot easier and cleaner . smile

Last edited by Denso (2015-05-12 23:18:39)

Offline

#5067 2015-05-12 23:55:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

EDIT :

Breaking the "options vfio-pci ids=" to multiple lines is possible . I just broke mine to 3 lines , rebooted and everything is working like a charm :

Cool


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5068 2015-05-13 02:14:06

JeromeDGDG
Member
Registered: 2015-05-13
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I Arrived to PCI passthrough my Nvidia GTX 750 (from EVGA) on ASrock Z97 extrem 6 motherboard with intel 4470 proc by the linux-vfio dedied kernel and add: iommu, ACS, i915 options support on bootloader.
I just would like to add this material ability to the support list: https://docs.google.com/spreadsheet/ccc … _web#gid=0
But how to add it ?
Thank you.

Offline

#5069 2015-05-13 06:27:54

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Good morning
I've followed every hint I've found on the web but I can't make my VGA passthrough work
I'm on Debian testing with official kernel (3.16.0.4 - 3.16.7-ckt9-3), with ACS and i915 patches, distro's QEMU (2.1+dfsg-11) and distro's seabios (1.8.1-2)
This is my PC

Sapphire PURE Platinum H61
Intel i7 3770
Nvidia GTX660 Ti (Zotac, the one I want to passthrough)
8GB of DDR3

I can successfully load the vfio and vfio-pvi modules and associate the GPU and the HDMI audio to it with

modprobe vfio
modprobe vfio-pci
echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

With lspci -nnv I can see that the two devices are used by the driver vfio-pci
With these qemu's argument I can successfully run Windows 7 and the GTX660 Ti is visible from there

#!/bin/sh
export QEMU_AUDIO_DRV=alsa
DISKIMG="/media/nicola/data/qemu/WindowsVM.img"
qemu-system-x86_64 --enable-kvm -M q35 -smp 8,sockets=1,cores=4,threads=2 -drive file=${DISKIMG},if=virtio  -m 6144 \
-usbdevice tablet -soundhw ac97 -cpu host,kvm=off \
-net nic,model=virtio -net user,hostname=windowsvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-device vfio-pci,host=01:00.0,bus=root.1,multifunction=on,x-vga=on,romfile=/home/nicola/Desktop/Zotac.GTX660Ti.2048.120723.rom \
-device vfio-pci,host=01:00.1,bus=root.1 -vga vmware

4gj0i.jpg

But if I set -vga none I don't have any output on the discrete GPU. Do you have any hints?

Last edited by corna (2015-05-13 13:13:36)

Offline

#5070 2015-05-13 07:33:39

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Your i915 patch is not working/not enabled in kernel boot options.

edit: you got h61 mbo, i don't think it supports vt-d?

Last edited by slis (2015-05-13 07:35:22)

Offline

#5071 2015-05-13 10:28:13

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In my BIOS configuration there's a "VT-d" switch and dmesg does not output any IOMMU error; this should mean that the VT-d is supported, right?
When I'll get home I'll double-check the BIOS configuration and the GRUB kernel's boot options

Offline

#5072 2015-05-13 10:55:28

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got this in my dmesg:

root@HOST:~# dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.012702] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.012707] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.012778] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.398237] IOMMU: dmar0 using Queued invalidation
[    0.398238] IOMMU: dmar1 using Queued invalidation
[    0.398239] IOMMU: Setting RMRR:
[    0.398248] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[    0.398526] IOMMU: Setting identity map for device 0000:00:14.0 [0xc95cb000 - 0xc95d7fff]
[    0.398542] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc95cb000 - 0xc95d7fff]
[    0.398554] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc95cb000 - 0xc95d7fff]
[    0.398562] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.398568] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

and this

root@HOST:~# dmesg | grep vgaarb
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.19.0-mast root=UUID=d8c8db8d-2f38-479e-9aba-8b9e004cf0b0 ro transparent_hugepage=always intel_iommu=on i915.enable_hd_vgaarb=1 elevator=deadline quiet
[    0.166672] vgaarb: setting as boot device: PCI:0000:00:02.0
[    0.166673] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.166676] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.166677] vgaarb: loaded
[    0.166678] vgaarb: bridge control possible 0000:01:00.0
[    0.166679] vgaarb: no bridge control possible 0000:00:02.0
[    1.488098] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Offline

#5073 2015-05-13 13:17:21

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've checked and both the "Intel Virtualization Technology" and "VT-d" switches are on in my BIOS
This is the output of your commands, almost identical

nicola@debianDesktop:~$ dmesg | grep IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] Intel-IOMMU: enabled
[    0.015015] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.015019] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.015091] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.506913] IOMMU 0 0xfed90000: using Queued invalidation
[    0.506914] IOMMU 1 0xfed91000: using Queued invalidation
[    0.506915] IOMMU: Setting RMRR:
[    0.506924] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[    0.507212] IOMMU: Setting identity map for device 0000:00:1a.0 [0xca1fa000 - 0xca216fff]
[    0.507230] IOMMU: Setting identity map for device 0000:00:1d.0 [0xca1fa000 - 0xca216fff]
[    0.507241] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.507247] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.512879] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[    0.512881] AMD IOMMUv2 functionality not available on this system

nicola@debianDesktop:~$ dmesg | grep vgaarb
[    0.000000] Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt9-3~deb8u1+acsoverridei915vgaarbiter1 (2
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=UUID=cf3edd70-0d97-4e7a-af69-a5b6beff2e6e ro intel_iommu=on pcie_acs_override=downstream i915.enable_hd_vgaarb=1 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=UUID=cf3edd70-0d97-4e7a-af69-a5b6beff2e6e ro intel_iommu=on pcie_acs_override=downstream i915.enable_hd_vgaarb=1 quiet
[    0.256781] vgaarb: setting as boot device: PCI:0000:00:02.0
[    0.256782] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.256785] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.256786] vgaarb: loaded
[    0.256787] vgaarb: bridge control possible 0000:01:00.0
[    0.256788] vgaarb: no bridge control possible 0000:00:02.0
[    2.203712] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io

I'm running the qemu's script with kdesudo; is there a way to run it without sudo privileges?
Ideas?

Last edited by corna (2015-05-13 13:20:06)

Offline

#5074 2015-05-13 13:40:17

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah it all looks good, you should have output on your gpu... old qemu maybe? i don't know.

edit: try ovmf? try libvirt? check aw's blog.

Last edited by slis (2015-05-13 13:42:19)

Offline

#5075 2015-05-13 22:06:35

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The output of rom-parser (http://vfio.blogspot.it/2014/08/does-my … t-efi.html) is this one

 Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 1183, class: 030000 PCIR: revision 0, vendor revision: 1 Last image

so I suppose I can't use ovmf...

What about libvirt? Can you link me the blog?
QEMU's version is 2.1, a recent one

Last edited by corna (2015-05-13 22:07:24)

Offline

#5076 2015-05-14 00:15:37

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:

so I suppose I can't use ovmf...

Have you looked at a different rom?

http://www.techpowerup.com/vgabios/inde … =&memSize=

Last edited by The_Moves (2015-05-14 00:15:53)

Offline

#5077 2015-05-14 07:54:53

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yesterday I ran QEMU without specifing any GPU ROM and it worked!
Will I experience any performance loss/problem without a GPU ROM?
Thanks for your help btw :-)

Last edited by corna (2015-05-14 07:55:28)

Offline

#5078 2015-05-14 08:59:31

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:

Yesterday I ran QEMU without specifing any GPU ROM and it worked!
Will I experience any performance loss/problem without a GPU ROM?
Thanks for your help btw :-)

No at all; using GPU ROM only makes sense as a workaround in certain situations.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5079 2015-05-14 10:53:23

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello again,

i know that this topic is about vga-passthrough in the first place, but it´s the most experienced community in virtualisation topics out there!

I have absolutly no idea how to install Windows 7 UEFI correctly as VM or on bare metal.

I had some success passing through the intel sata controller to VM.

Installing Windows 7 VM with UEFI using ovmf/libvirt/qemu on the passed through sata controller with attached ssd works fine, until installer reboots.
I´ve not configured any hd´s in vmmanager, just the sata controller, loaded all virtio drivers from iso (hd was visible without driver).
After installation reboot, Windows 7 UEFI VM end up in ovmf EFI-Shell. Partitioning scheme and data is written to disk, just Windows Boot Manager fails to boot in ovmf.

I´ve read some older posts about the fat32 uefi partition and i think i´m really close to get the VM to work without the option to dual boot on bare metal. Only missing a little detail about configuring ovmf/qemu/libvirt to boot from passed through SATA controller with attached disk that contains the uefi fat32 partiton.

Does qemu / libvirt / ovmf need a configured disk to boot?
I use ovmf 16229-1. Do i need a special ovmf svn package from AUR?

A bunch of another off topic questions is Windows 7 UEFI bare metal installation in this case (dual boot part).
Looks like nowadays UEFI Windows installations are not as simple as they should be, or me missing very much basic knowledge about UEFI Systems.

My MB is Asrock Z97Extreme6.

I´ve found no useful article that describes my usecase for basic hd UEFI configuration in main bios.
- 1 linux gpt disk with syslinux on SATA A0
- 3 gpt lvm disks (storage for home network) on SATA A1-3
- 2 lvm disks without partition table (unplugged for testing) SATA1-2
- 1 windows7 uefi gpt disk (for win7 installation unformatted, or like later described with fully installed win7 uefi) on SATA0
- 1 DVDRW on SATA A4

SATA connectors without A are the main ones from the Intel sata-controller.

In the good old non UEFI times just the disk configuration in main BIOS decided which disk is started, after Windows 7 UEFI installation i´ve got this strange Windows Boot Manager entry on main bios.

Do UEFI-Systems read data from disks for general hw initiation / OS initiation?
Why there is this new Windows entry in my BIOS instead of the disk description in main bios?
Is gummiboot a replacement for Windows Boot Manager or does it have to be the main bootloader that "chainloads" other OSses (WBM; syslinux...)?

These disks are my requirements to make use of UEFI on bare metal Asrock Bios. But im not able to install Win7 in this - i think - pretty common configuration.

It only installs and works, if i unplug all other disks. If i plug in the disks back on this previous working installation:
Windows Boot Manager fails to boot,
syslinux from linux gpt disk is second option in boot legacy, syslinux works normally from this point on.

I have absolutly no clue why a normal Windows 7 UEFI installation on bare metal does not work with all my disks attached.
All disks have gpt like described, except two lvm disks, they have no partition table. I´ve unpluged them for this test. I´ve tested with various disks unplugged, no luck.

Maybe someone of u have some experiences with this kinda UEFI stuff.

Ssd for Win7 is connected to sata0 on intel sata controller. Linux is on another disk (A0) with syslinux as another boot option in main bios.
With almost all necessary disks, exept the two disks without partition table (unplugged) the installation can´t be completed after reboot on bare metal.
Maybe its a wrong main bios configuration?

Maybe gummiboot is needed on bare metal to "chainload?" uefi OS and normal syslinux as another boot option, as main bootloader not the Asrock one.

Back to the VM - part.

Can somebody share a configuration example who managed to get a sata controller passthrough to work (with win7 uefi install, ovmf, libvirt xml)?

Thank you in advance.

Last edited by Schlunze (2015-05-14 10:57:19)

Offline

#5080 2015-05-14 10:56:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:

The output of rom-parser (http://vfio.blogspot.it/2014/08/does-my … t-efi.html) is this one

 Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 1183, class: 030000 PCIR: revision 0, vendor revision: 1 Last image

so I suppose I can't use ovmf...

What about libvirt? Can you link me the blog?
QEMU's version is 2.1, a recent one

Try checking your manufacturer's site for VBIOS updates - maybe there'll be an UEFI-compatible(with GOP) one. Or lurk through the techpowerup database.
Because my gigabyte GT610 supports GOP.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5081 2015-05-14 11:37:31

protake
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The iommu group of my second card contains a lot of stuff, for example my network card.

ls /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ 0000:00:1c.0 0000:00:1c.5 0000:00:1c.6 0000:02:00.0 0000:02:00.1 0000:03:00.0 0000:04:00.0

Unless I bind all these devices, qemu won't start. Is there anything I can do about this? This prevents my network connection from working, for example. Is my motherboard at fault here?

Last edited by protake (2015-05-14 11:38:28)

Offline

#5082 2015-05-14 11:51:19

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
corna wrote:

The output of rom-parser (http://vfio.blogspot.it/2014/08/does-my … t-efi.html) is this one

 Valid ROM signature found @0h, PCIR offset 190h PCIR: type 0, vendor: 10de, device: 1183, class: 030000 PCIR: revision 0, vendor revision: 1 Last image

so I suppose I can't use ovmf...

What about libvirt? Can you link me the blog?
QEMU's version is 2.1, a recent one

Try checking your manufacturer's site for VBIOS updates - maybe there'll be an UEFI-compatible(with GOP) one. Or lurk through the techpowerup database.
Because my gigabyte GT610 supports GOP.

I've tried rom-parser with this Gigabyte BIOS and this is the output

Valid ROM signature found @400h, PCIR offset 190h
    PCIR: type 0, vendor: 10de, device: 1183, class: 030000
    PCIR: revision 0, vendor revision: 1
    Last image

but the BIOS description says

5. UEFI BIOS

so I suppose that the rom-parser script is broken. Later I'll try if it works with OVMF

Offline

#5083 2015-05-14 12:03:23

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

protake wrote:

The iommu group of my second card contains a lot of stuff, for example my network card.

ls /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ 0000:00:1c.0 0000:00:1c.5 0000:00:1c.6 0000:02:00.0 0000:02:00.1 0000:03:00.0 0000:04:00.0

Unless I bind all these devices, qemu won't start. Is there anything I can do about this? This prevents my network connection from working, for example. Is my motherboard at fault here?

You need ACS override patch for kernel.

Offline

#5084 2015-05-14 13:00:21

protake
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
protake wrote:

The iommu group of my second card contains a lot of stuff, for example my network card.

ls /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ 0000:00:1c.0 0000:00:1c.5 0000:00:1c.6 0000:02:00.0 0000:02:00.1 0000:03:00.0 0000:04:00.0

Unless I bind all these devices, qemu won't start. Is there anything I can do about this? This prevents my network connection from working, for example. Is my motherboard at fault here?

You need ACS override patch for kernel.

Hm, strange. I was using the linux-vfio kernel yesterday, and I'm pretty sure it was the same. I don't mind trying it again if it works, though...

Offline

#5085 2015-05-14 13:06:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

protake wrote:
slis wrote:
protake wrote:

The iommu group of my second card contains a lot of stuff, for example my network card.

ls /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ 0000:00:1c.0 0000:00:1c.5 0000:00:1c.6 0000:02:00.0 0000:02:00.1 0000:03:00.0 0000:04:00.0

Unless I bind all these devices, qemu won't start. Is there anything I can do about this? This prevents my network connection from working, for example. Is my motherboard at fault here?

You need ACS override patch for kernel.

Hm, strange. I was using the linux-vfio kernel yesterday, and I'm pretty sure it was the same. I don't mind trying it again if it works, though...

There are kernel commandline options required to enable the acs override patch.  What chipset is this?  We have quirks in place for nearly all Intel PCH root ports, 9-series is coming in kernel 4.2


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5086 2015-05-14 13:19:38

protake
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

There are kernel commandline options required to enable the acs override patch.  What chipset is this?  We have quirks in place for nearly all Intel PCH root ports, 9-series is coming in kernel 4.2

Intel H97.

Offline

#5087 2015-05-14 13:23:54

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

protake wrote:

Intel H97.

aw wrote:

9-series is coming in kernel 4.2

Offline

#5088 2015-05-14 13:36:16

protake
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
protake wrote:

Intel H97.

aw wrote:

9-series is coming in kernel 4.2

Yeah, I understood as much. I just wanted to mention it. Thanks for all the replies!

edit: Is there anything I could do until 4.2 hits?

Last edited by protake (2015-05-14 16:43:29)

Offline

#5089 2015-05-14 17:29:03

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the problem that google play movies doesn't work anymore on my arch host since around February this year. I didn't find a solution yet, also tried to use the windows version of firefox in wine, but it won't work.
Now I'm wondering why it also doesn't work in my Windows 7 VM. I havn't installed much there. Basically it's a Firefox, flash player, Steam and drivers. Are there any host requirements or some qemu configuration I need to respect for that?

BR
apex

Offline

#5090 2015-05-14 18:07:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

I have the problem that google play movies doesn't work anymore on my arch host since around February this year. I didn't find a solution yet, also tried to use the windows version of firefox in wine, but it won't work.
Now I'm wondering why it also doesn't work in my Windows 7 VM. I havn't installed much there. Basically it's a Firefox, flash player, Steam and drivers. Are there any host requirements or some qemu configuration I need to respect for that?

BR
apex

First thing that comes to my mind - some weird DRM system. You suuure that everything else works fine?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5091 2015-05-14 18:28:33

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
apex8 wrote:

I have the problem that google play movies doesn't work anymore on my arch host since around February this year. I didn't find a solution yet, also tried to use the windows version of firefox in wine, but it won't work.
Now I'm wondering why it also doesn't work in my Windows 7 VM. I havn't installed much there. Basically it's a Firefox, flash player, Steam and drivers. Are there any host requirements or some qemu configuration I need to respect for that?

BR
apex

First thing that comes to my mind - some weird DRM system. You suuure that everything else works fine?

You mean in the VM? Gaming is totally fine. Couldn't think of anything in particular, which is missing..

Offline

#5092 2015-05-14 19:11:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
Duelist wrote:
apex8 wrote:

I have the problem that google play movies doesn't work anymore on my arch host since around February this year. I didn't find a solution yet, also tried to use the windows version of firefox in wine, but it won't work.
Now I'm wondering why it also doesn't work in my Windows 7 VM. I havn't installed much there. Basically it's a Firefox, flash player, Steam and drivers. Are there any host requirements or some qemu configuration I need to respect for that?

BR
apex

First thing that comes to my mind - some weird DRM system. You suuure that everything else works fine?

You mean in the VM? Gaming is totally fine. Couldn't think of anything in particular, which is missing..

Oh, it doesn't work on the host...
What hardware do you have on the host? Maybe it's just some messed up codecs(google likes using "new" codecs like vp8/9 or something)?


Slightly OT:
guys, please, save me from going insane - i'm considering buying a third asus HD7750 silent GPU. It's located ~180km away, one of the cheapest 7750s, and i don't even have space in the case to fit it yet. But something shivers inside me when i look at it. Just imagine running a tri-way crossfire(however, amd support once said that crossfire requires an even number of cards) in a VM. Even if crossfire won't work, i'll just set up another VM... Maybe running it will require some hardcore case modding butchering and fitting a riser-card to run it from x1 pci-e, or maybe i'll need to switch to kaveri C/APUs for pci-e3.0... But daaamn, i want that card, while understanding that the idea is bizarre.

Last edited by Duelist (2015-05-14 19:20:39)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5093 2015-05-14 21:08:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Slightly OT:
guys, please, save me from going insane - i'm considering buying a third asus HD7750 silent GPU...

That's certainly a strange obsession you've got for a relatively mediocre card.  I've got my own, so I won't rib you on it too hard.  It's at least silent, but if you're already got two of these power sucking cards in your system, do you really want to add 50% more?  Personally I don't see why CF/SLI is even on the radar for most people, single card performance on a recent card should handily beat multi-card performance from a few years ago.  I don't know about AMD, but Nvidia is doing some pretty impressive work on power efficiency with Maxwell.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5094 2015-05-14 23:54:19

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

protake wrote:

edit: Is there anything I could do until 4.2 hits?

Sure , compile a kernel with ACS patch , add :

pcie_acs_override=downstream

to your boot parameters .

Cheers !

Offline

#5095 2015-05-15 18:04:19

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to use OVMF but after reboot the VM hangs with a black screen (even if I don't have VGA passthrough enabled). This happens even with some Linux distros as guest.
My fear is that I might be hitting some random daily OVMF build bug. If you have a successful setup with OVMF would you mind sharing the OVMF image you are using? Thanks!!

Offline

#5096 2015-05-15 21:19:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

Slightly OT:
guys, please, save me from going insane - i'm considering buying a third asus HD7750 silent GPU...

That's certainly a strange obsession you've got for a relatively mediocre card.  I've got my own, so I won't rib you on it too hard.  It's at least silent, but if you're already got two of these power sucking cards in your system, do you really want to add 50% more?  Personally I don't see why CF/SLI is even on the radar for most people, single card performance on a recent card should handily beat multi-card performance from a few years ago.  I don't know about AMD, but Nvidia is doing some pretty impressive work on power efficiency with Maxwell.

Each card consumes only 55W full-throttle.Yes, that's almost twice as less than CPU's TDP. Considering ULPS and zerocore and simple declocking - when half-loaded or idle they should consume even less.
50% more? Not 50%. Second card gave me 1.3x-1.9x performance(unigine benchmark makes a good use of CF, heh). The reason i'm obsessed with this - i can't sell them due to... multiple reasons.
But it's not enough performance-wise(and as my mind hints me - it'll never be), and having a somewhat unique and bizarre config makes me feel funny. Oh, and 2gb of video ram isn't quite awesome for my taste, however i don't like that tendency to bloat video ram sizes(12GB?! 512mb was enough-for-everyone some years ago)

Oh, and that HD7750 is still good on performance/power/money chart. Of course, GTX750 overtakes it, but still.
Due to some magic, i could've continued mining litecoins and still have profit(because 55w of electricity was equal to zero cents per month. I'm serious) four months ago. That's how weird these cards are.

paperinick wrote:

I am trying to use OVMF but after reboot the VM hangs with a black screen (even if I don't have VGA passthrough enabled). This happens even with some Linux distros as guest.
My fear is that I might be hitting some random daily OVMF build bug. If you have a successful setup with OVMF would you mind sharing the OVMF image you are using? Thanks!!

Weeeeeeird. I have libvirt and kraxel-org-based jenkins-git images, and it works fine there.

Last edited by Duelist (2015-05-15 21:20:55)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5097 2015-05-15 21:42:30

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

...because 55w of electricity was equal to zero cents per month....

How do you figure that?  Free electric?  Surplus production?  55 watts 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 24 hours = 1.32kW, which is 10 cents per day from my utility even at the lowest tier


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5098 2015-05-15 22:13:33

sweetsuicide
Member
Registered: 2015-05-15
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everybody here,
this is some awesome work done here. I built a computer specifically for running windows inside a vm with secondary GPU passthough. I followed instructions from the main page here mixed with the similar https://wiki.archlinux.org/index.php/PC … h_via_OVMF guide. This means that I have the builds from Hoffmann's repository. I have not stepped up to libvirt yet, mainly because I haven't installed any DE or WM.
So here's my problem: everything looks fine (qemu boots up, secondary gpu fires up with qxl), but I cannot install windows. Windows8 iso is not even browsable through the efi shell. Windows 7 is readable but it stops immediately with an error (boot related, 0xc000000d code something).
I have no I idea what's happening.
Setup is following:
i7-4770
ASRock Z87 Extreme4
16GB RAM
two ssd's
R9 280x

radeon module was blacklisted (even it doesn't seem like it's necessary, as video card powers up when I start qemu). Qemu command:

qemu-system-x86_64\ -enable-kvm\ -cpu host,kvm=off\ -nographic -m 8192 -vga qxl\ -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:1b.0\ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd\ -drive if=pflash,format=raw,file=/home/luca/this.fd\ -device virtio-scsi-pci,id=scsi -drive file=/mnt/SAND/win.cow,id=disk,if=none,cache=writeback\ -device scsi-hd,drive=disk -drive file=/home/user/Windows81.iso,id=isocd,if=none -device scsi-cd,drive=isocd\ -drive file=/home/luca/virtio-win-0.1.96.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd\ -usb -usbdevice host:045e:07f8 -usbdevice host:046d:c408

Command given as root (not much choice here)

Offline

#5099 2015-05-15 22:35:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sweetsuicide wrote:

secondary gpu fires up with qxl

What does that even mean?

qemu-system-x86_64\ -enable-kvm\ -cpu host,kvm=off\ -nographic -m 8192 -vga qxl\

Huh?  QXL is an emulated graphics card.  It might make sense to have it since you claim to be wanting to setup a secondary graphics configuration, but what's supposed to happen with -nographic?  Where is the QXL graphics supposed to display?  Your choices are SDL, VNC, or Spice, but if you leave -nographic, you might as well dump the QXL device with -vga none.

Others have had trouble with the ISO format, that may be what you're running into with the install image, something about UDF vs ISO9660.  I've never had a problem with images from MSDN.

Your setup seems textbook enough that I'd follow the howto guide in my blog (link in sig) once you get your ISO format straightened out (libvirt does not require a DE/WM.  Even the graphical tools like virt-manager can be run remotely if your system is headless).

BTW, you don't need kvm=off for an AMD GPU.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5100 2015-05-15 23:11:22

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Offline

#5101 2015-05-15 23:56:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Are both slots for a single VM or split between host and guest?  Do they both need to be PCIe 3.0?  Integrated graphics or discrete for host?  Any other cards to install (for host or guest)?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5102 2015-05-16 08:18:30

scoobydog
Member
Registered: 2014-04-14
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I saw my system is in the Google Doc list, nice :-)

Here are the missing informations about my System:

Linux  3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) x86_64 GNU/Linux

QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11), Copyright (c) 2003-2008 Fabrice Bellard

No special Paches, I'm using the kernel from Debian.

And it's also working with a Asus strix GTX970 OC Graphikcard, see a benchmark of it here https://www.youtube.com/watch?v=v1cFWrKtWis .

Offline

#5103 2015-05-16 10:08:51

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
apex8 wrote:

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Are both slots for a single VM or split between host and guest?  Do they both need to be PCIe 3.0?  Integrated graphics or discrete for host?  Any other cards to install (for host or guest)?

I guess its not necessary, that both slots are for a single VM. Now I'm curious: does that make a difference? However I think it would be nice if they have PCIe 3.0.
Integrated Graphics is always a nice to have, I think. Two separete SATA controllers would be nice to have. Integrated or as extra card is not so important for me.
Also at the moment I use a dedicated sound card for the VM, but I think there isn't much benefit in this, is it? OK after all in my setup the speaker out goes directly into the line in of the integrated sound device used in the host smile

Last edited by apex8 (2015-05-16 10:13:41)

Offline

#5104 2015-05-16 10:57:03

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

...because 55w of electricity was equal to zero cents per month....

How do you figure that?  Free electric?  Surplus production?  55 watts 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 24 hours = 1.32kW, which is 10 cents per day from my utility even at the lowest tier

Currency exchange rate allows me to say this. And I don't have the lowest prices, but they're quite low for my area. But that currency-exchange-magic works the other way, so i can't really sell my cards or buy a new one. Hence the obsession.

apex8 wrote:

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Let's start from the top:
What CPU do you want to use?
AMD? Use whatever with 990FX and tons of pci-e lanes. Intel? AW will give you more advice on this, especially about 9-series chipsets and ACS quirks.

And also... PCI-E 3.0 is needed when you don't have enough bandwidth in PCI-E 2.0. If your current cards run fine on pci-e 2.0 - don't bother. Maaaybe you might get some slight performance increase due to additional bandwidth. Especially you'll notice this if you're using some weird config(like me) or doing crossfire-over-pci-e aka XDMA CF(again, like me).
ATM most GPUs already support PCI-E 3.0, and there's rarely a motherboard-CPU combo without pci-e 3.0.(usually that's AMD)

Dedicated sound card for the VM... I admit, i've set that up too, but that's too much hassle(actually, now i continue using that card because of gameport and a joystick) in mixing host and guest sounds, so i'd suggest you digging to NetJack way of mixing outputs. Since we have localhost tun/tap network, we shouldn't stumble into bandwidth or lag issues, and something hints me that it should work better that embedded pulse audio or alsa stuff we have now. I'm not a linux sound specialist, so YMMV.

And why would you want to have a separate SATA controller? Why?
Virtio-pci gives you A LOT of speed and in some cases you can even benefit from host system caching. Oh noes, SMART and TRIM aren't working? Use virtio-scsi or whatever it was the controller variant.
Usually you'd want to have a separate SATA controller for the VM just to reduce the amount of guest-system-trickery with drivers(like windows having BSOD 7B). Otherwise it's not worth it, IMO.
And also, that controller, if it isn't a separate PCI card, may fall into a clustered IOMMU-group, requesting you to passthrough other stuff that you don't want to passthrough, like USB or ethernet controllers.

And also, i've exceeded my post quota on "And also" usage.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5105 2015-05-16 11:25:45

apex8
Member
Registered: 2014-03-29
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK maybe I'm a bit too pessimistic on the disk performance, because currently I already use virt io with data-x-plane and the performance could be better. However the host runs on a SSD and this could be something I could do with the VM too.
At the sound configuration, you mentioned something like NetJack. Can you go into a bit more detail? Currently I'm using pulseaudio, but I suspect this thing of being responsible for trouble with google play movies and audio crackle. Why do I suspect pulseaudio for those issues? Both things worked in the past and the only major changes I did to the system was switching from KDE to LXDE and installing pulseaudio.

Last edited by apex8 (2015-05-16 11:35:42)

Offline

#5106 2015-05-16 13:52:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In the first post of my recent how-to series I talk about the hardware I use, not that I'd recommend it, but to talk about why it works for my config - http://vfio.blogspot.com/2015/05/vfio-g … dware.html

I won't be as generous as Duelist, skip AMD platforms.  They've admittedly given up on the single core performance race, you don't know what you're getting on an APU system for isolation, and their latest performance/workstation class chipset is quickly coming up on its four year anniversary since launch.

So that leaves Intel.  I assume that most people do not want to patch their kernel long term, so that means we want a config that avoids the i915 VGA arbiter issues and doesn't require the ACS override patch.  The VGA issue is easy, use a UEFI capable GPU and guest or use discrete graphics for the host.  The isolation problem is more complicated.  The reason I ask about PCIe 3.0 requirements is that PCIe 3.0 slots are pretty much always sourced from the processor root ports rather than the chipset root ports.  If you use a "client" processor, ie. Core i5/i7, or even a Xeon E3-1200 series, there's not going to be isolation between the processor root ports.  That means that whatever you plug into those root port slots either needs to be used exclusively by the host or exclusively by a single VM.  That leaves a lot of workable configs, but you must be conscious of the restrictions.  We have pretty good coverage for isolation of the PCH root ports, 9-series is coming in kernel 4.2, the reset should already be there.  This means that you get finer granularity of what you can assign for devices connected via the PCH root ports, which are generally the PCIe 2.0 slots as well as the integrated components.

If you don't want to sorry about this at all and want the most flexibility in your system config without patches, get a Xeon E5 or better with and X79 or X99 chipset.

If you really want to assign a disk controller to a VM, you're into risky territory, mostly because the cheap ones are all crap.  If you try to assign an integrated controller, the boot ROM for it may be integrated into the system BIOS, so you'll have no way to boot the VM unless you can extract it... and hope it supports UEFI if you're going the OVMF route with the VM.  You may have better luck with a discrete HBA, but you're still in the unknown/crap space.  Personally I find virtio-scsi w/ multi-queue and a fast backing setup to meet my needs.

If you want to assign a NIC, make it an Intel NIC, Realtek NICs are crap for assignment.  Broadcom can be ok, but it's a bit more of a gamble.  For anything else you might want to assign, try to avoid conventional PCI.  All conventional PCI devices are indistinguishable by the IOMMU, so there is no way to split them between host and guest and they often have poor support for interrupt masking.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5107 2015-05-16 14:14:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

If you want to assign a NIC, make it an Intel NIC, Realtek NICs are crap for assignment.  Broadcom can be ok, but it's a bit more of a gamble.  For anything else you might want to assign, try to avoid conventional PCI.  All conventional PCI devices are indistinguishable by the IOMMU, so there is no way to split them between host and guest and they often have poor support for interrupt masking.

Oh god, I have Atheros flashbacks. Oh god. No, not the L1 again. Oh god... This card was incompatible with.. everything. Are they dead yet?

And why do you skip AMD platforms? Some people use debian, being the not-so-fresh-but-stable software, why not combine it with 4-years-old-but-already-patched-everywhere AMD chipset?

Some people prefer i5-2500k for it's insane overclocking abilities and cheapness, and it's 3-4 years old too. And what, it makes it a bad motherboard-cpu combo?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5108 2015-05-16 14:40:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

And why do you skip AMD platforms? Some people use debian, being the not-so-fresh-but-stable software, why not combine it with 4-years-old-but-already-patched-everywhere AMD chipset?

Some people prefer i5-2500k for it's insane overclocking abilities and cheapness, and it's 3-4 years old too. And what, it makes it a bad motherboard-cpu combo?

I'm not sure how 2500k is relevant here, it doesn't support VT-d, so for anything in this thread, yes it does make a bad combo.

AMD is just not interesting to me, so certainly factor in some personal bias.  990FX has reasonably good isolation, I have one myself.  Mine is also a dog compared to my Intel boxes.  If you head towards the newer APU-based AMD system, you're in a world of unknown.  We don't know what the isolation capabilities are of the chipset and I don't have a relationship with AMD to try to fix any gaps with quirks.  There's also no resource like ark to determine the various features of a given combo of AMD CPU and chipset.  We also have users reporting that they need to do strange things like disable hugepages on some AMD systems which implies there's some software or hardware problem out there, but we don't have enough data points, or interest, to figure it out.  So no, I can't in good faith recommend an AMD platform.

BTW, I forgot to mention that Gigabyte boards seem to have the best BIOS configuration for primary graphics, so if discrete host graphics was on my radar in choosing a motherboard, it would probably be a Gigabyte board.  I obviously can't guarantee anything there though.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5109 2015-05-17 01:57:16

ceri
Member
Registered: 2013-10-12
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

ISo that leaves Intel.  I assume that most people do not want to patch their kernel long term, so that means we want a config that avoids the i915 VGA arbiter issues and doesn't require the ACS override patch.  The VGA issue is easy, use a UEFI capable GPU and guest or use discrete graphics for the host.  The isolation problem is more complicated.  The reason I ask about PCIe 3.0 requirements is that PCIe 3.0 slots are pretty much always sourced from the processor root ports rather than the chipset root ports.  If you use a "client" processor, ie. Core i5/i7, or even a Xeon E3-1200 series, there's not going to be isolation between the processor root ports.  That means that whatever you plug into those root port slots either needs to be used exclusively by the host or exclusively by a single VM.  That leaves a lot of workable configs, but you must be conscious of the restrictions.  We have pretty good coverage for isolation of the PCH root ports, 9-series is coming in kernel 4.2, the reset should already be there.  This means that you get finer granularity of what you can assign for devices connected via the PCH root ports, which are generally the PCIe 2.0 slots as well as the integrated components.

I don't really know/understand the architecture of the interaction between PCI/e ports and the CPU & MB, and so I'm confused about the ACS isolation stuff, but from what you've written here and on your blog, would I be correct in assuming that:

a) if i wanted to use discrete graphics (w/ a UEFI bios) on a single guest, and intel integrated graphics on the host (with no other PCI/PCIe devices), I wouldn't have to worry about using any patches or isolation, nor the type of CPU/motherboard I get (except for CPU/GFX compatibility with the MB and VT-d, etc)?

b) in the case of (a) but also with a PCIe 1.0+ sound card for use on the host (e.g. put in a PCIe 2.0 slot), I wouldn't have to worry about patches or isolation with a pre-9 series chipset (or 9 series from linux 4.2+)?

c) in the case of (a), except that I would be using a second discrete GFX card for the host, I *would* have to worry about patches and isolation?

Last edited by ceri (2015-05-17 01:58:40)

Offline

#5110 2015-05-17 03:43:44

marca311
Member
From: Canada
Registered: 2014-02-25
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I have been following this thread for several weeks now and after a couple weeks of trying things on my computer, I have hit a brick wall.
Before we get any further, here are my specs:

Arch stock Kernel 4.0.2-1 or 3.19.3-1-ck+ACS override patch Intel i7-5820k MSI Gaming 7 X99S Motherboard Intel X99 chipset 32GB RAM EVGA 980 Superclocked QEMU-git (pulled and compiled 16 May 2015) OVMF-svn (pulled and compiled last week)

I have tried with both libvirt and straight up qemu.
Here is my QEMU file:

#!/bin/bash vfio-bind 0000:03:00.0 0000:03:00.1 export QEMU_AUDIO_DAC_FIXED_SETTINGS=1 export QEMU_AUDIO_DAC_FIXED_FREQ=44100 export QEMU_AUDIO_DAC_FIXED_FMT=S16 export QEMU_SDL_SAMPLES=1024 qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \ -soundhw hda \ -mem-path /dev/hugepages \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -boot d /home/marcus/qemu/win8.1/c_drive.qcow2 \ -net nic \ -net user,smb=/run/Game-Drive \ -smp 4,sockets=1,cores=4,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on \ -device vfio-pci,host=03:00.1 \ -vga none \ -usb \ -device nec-usb-xhci,id=xhci \ -usbdevice host:046d:c52f \ -usbdevice host:045e:00b4

And here is my libvirt config:

<domain type='kvm'> <name>win8.1</name> <uuid>d9aaa8dd-6b43-421b-b165-706926e9181e</uuid> <memory unit='KiB'>8290304</memory> <currentMemory unit='KiB'>8290304</currentMemory> <vcpu placement='static' current='8'>32</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_code_x64.bin</loader> <nvram template='/usr/share/ovmf/x64/ovmf_vars_x64.bin'>/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='8'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/marcus/qemu/win8.1/c_drive.qcow2'/> <target dev='hdb' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/marcus/qemu/win8.1/Witcher3.qcow2'/> <target dev='hdc' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/marcus/qemu/win8.1/GTAV.qcow2'/> <target dev='hdd' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:03:6e:f7'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </interface> <sound model='ac97'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <sound model='es1370'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </sound> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </sound> <sound model='ich9'> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </sound> <sound model='pcspk'/> <sound model='sb16'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc52f'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x00b4'/> </source> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

Here is what libvirt starts up

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name win8.1 -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu host,kvm=off -drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd,if=pflash,format=raw,unit=1 -m 8096 -realtime mlock=off -smp 8,maxcpus=32,sockets=1,cores=4,threads=8 -uuid d9aaa8dd-6b43-421b-b165-706926e9181e -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win8.1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/home/marcus/qemu/win8.1/c_drive.qcow2,if=none,id=drive-ide0-0-1,format=qcow2 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=1 -drive file=/home/marcus/qemu/win8.1/Witcher3.qcow2,if=none,id=drive-ide0-1-0,format=qcow2 -device ide-hd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/home/marcus/qemu/win8.1/GTAV.qcow2,if=none,id=drive-ide0-1-1,format=qcow2 -device ide-hd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:03:6e:f7,bus=pci.0,addr=0x9 -device AC97,id=sound0,bus=pci.0,addr=0x4 -device ES1370,id=sound1,bus=pci.0,addr=0xa -device intel-hda,id=sound2,bus=pci.0,addr=0xb -device hda-duplex,id=sound2-codec0,bus=sound2.0,cad=0 -device ich9-intel-hda,id=sound3,bus=pci.0,addr=0xc -device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 -soundhw pcspk -device sb16,id=sound5 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=03:00.1,id=hostdev1,bus=pci.0,addr=0x3 -device usb-host,hostbus=1,hostaddr=6,id=hostdev2 -device usb-host,hostbus=1,hostaddr=5,id=hostdev3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

Now then, that's out of the way.
My issue is in a few places.
The biggest one is that when I shut down the VM, the X server on the host freezes and if I try to shut it down, nothing happens.
If I try to shut the computer down, everything just freezes and I have to hard reboot. Finally, I have to shut off my power supply for approximately 15 mins before the motherboard even acknowledges me pressing the power button. I'm wondering if I'm damaging my hardware.

Error I get on shutdown right before the freeze (with qemu only, not libvirt):

qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: error, group 28 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to get group 28 qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device initialization failed qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

Yes, I do have the kernel arguments for PCI-Stub set up.

This issue happens with both qemu and libvirt, however qemu requires mouse something on the display to update first before it freezes, libvirt just freezes immediately.
I have been unable to find anything in journalctl or any relevant logs. Any ideas?

Last edited by marca311 (2015-05-17 03:51:00)

Offline

#5111 2015-05-17 11:56:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

marca311 wrote:

Hi all!

I have been following this thread for several weeks now and after a couple weeks of trying things on my computer, I have hit a brick wall.
Before we get any further, here are my specs:

Arch stock Kernel 4.0.2-1 or 3.19.3-1-ck+ACS override patch Intel i7-5820k MSI Gaming 7 X99S Motherboard Intel X99 chipset 32GB RAM EVGA 980 Superclocked QEMU-git (pulled and compiled 16 May 2015) OVMF-svn (pulled and compiled last week)

..SNIP...

Error I get on shutdown right before the freeze (with qemu only, not libvirt):

qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: error, group 28 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to get group 28 qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device initialization failed qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

Yes, I do have the kernel arguments for PCI-Stub set up.

This issue happens with both qemu and libvirt, however qemu requires mouse something on the display to update first before it freezes, libvirt just freezes immediately.
I have been unable to find anything in journalctl or any relevant logs. Any ideas?

List your iommu groups by doing

tree /sys/kernel/iommu_groups/

and check what's in group 28. Or just stuff it all here. Chances are that you have problems with device isolation, needing ACS patch and general 9-series intel chipset support that will be brought upstream in linux 4.2.

Last edited by Duelist (2015-05-17 11:57:11)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5112 2015-05-17 13:40:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ceri wrote:

I don't really know/understand the architecture of the interaction between PCI/e ports and the CPU & MB, and so I'm confused about the ACS isolation stuff, but from what you've written here and on your blog, would I be correct in assuming that:

a) if i wanted to use discrete graphics (w/ a UEFI bios) on a single guest, and intel integrated graphics on the host (with no other PCI/PCIe devices), I wouldn't have to worry about using any patches or isolation, nor the type of CPU/motherboard I get (except for CPU/GFX compatibility with the MB and VT-d, etc)?

Yes, if the discrete graphics card for the guest is the only plugin card, then pretty much anything will work.  If your chipset is not quirked for PCH root port isolation, then you'd need to install it in a processor root port slot to keep it from getting grouped with onboard devices, if it is quirked, then any slot will work.

b) in the case of (a) but also with a PCIe 1.0+ sound card for use on the host (e.g. put in a PCIe 2.0 slot), I wouldn't have to worry about patches or isolation with a pre-9 series chipset (or 9 series from linux 4.2+)?

This is also an easy configuration, but you do somewhat need to worry about isolation and which card goes where.  If the guest discrete graphics is installed in a processor root port slot, then the host sound card would need to be installed in a PCH root port slot.  If you have a pre-9 series chipset, both could be installed in PCH root port slots.

c) in the case of (a), except that I would be using a second discrete GFX card for the host, I *would* have to worry about patches and isolation?

It's still achievable without patches, but you're potentially sacrificing performance.  The rule is that Intel Client processors (i5/i7) and Xeon E3-1200 series do not have isolation between the processor root ports.  Therefore anything plugged into those slots will be part of the same IOMMU group.  We cannot have split ownership of an IOMMU group between host and guest.  Most motherboard will only allow you to specify "PCIe" for graphics without slot granularity (Gigabyte seems to be the exception here), which means that you need to use the processor root port slots for your host graphics.  That means any other processor root port slots need to be used for host owned devices, for example you can put your x1 sound card in another processor slot.  The guest discrete graphics would need to use a PCH root port slot which is already quirked for isolation to avoid grouping with onboard devices (pre-9 series until kernel 4.2).  With BIOSes that allow slot selection of primary graphics, there's a bit more freedom that you can choose whether to put the host graphics in the processor or PCH root port, which can favor guest graphics performance.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5113 2015-05-17 14:00:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist, X99 is not a 9-series chipset, X99 quirks went in for 4.0

@marca311, your log doesn't make much sense, the group not viable error is a fatal qemu error, the VM shouldn't start but you're saying you get that error on shutdown.  I don't see how that's possible.  You say you have the kernel commandline enabled for pci-stub, but is it effective?  Does /proc/cmdline show the option?  Is pci-stub built statically into your kernel?  If pci-stub is a module then it becomes non-deterministic which pci-stub gets a shot at rebinding to the GPU before the native driver, it all depends on which module loaded first.  But event that's still confusing because your raw qemu script doesn't appear to be unbinding devices when finished.  I don't recommend vfio-bind anymore, and that itself should be causing you trouble if it's doing more than binding endpoint devices to vfio-pci.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5114 2015-05-17 19:45:24

marca311
Member
From: Canada
Registered: 2014-02-25
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@marca311, your log doesn't make much sense, the group not viable error is a fatal qemu error, the VM shouldn't start but you're saying you get that error on shutdown.  I don't see how that's possible.  You say you have the kernel commandline enabled for pci-stub, but is it effective?  Does /proc/cmdline show the option?  Is pci-stub built statically into your kernel?  If pci-stub is a module then it becomes non-deterministic which pci-stub gets a shot at rebinding to the GPU before the native driver, it all depends on which module loaded first.  But event that's still confusing because your raw qemu script doesn't appear to be unbinding devices when finished.  I don't recommend vfio-bind anymore, and that itself should be causing you trouble if it's doing more than binding endpoint devices to vfio-pci.

So I double-checked my kernel config and it turns out that CONFIG_PCI_STUB was set to module, so I changed it and recompiled.
I haven't had a crash yet, but I haven't tested as much yet. Thanks so far though!

However this brings me to my next couple of issues: sound and CPU performance.
Running 3DMark gives me bad scores on my CPU (http://www.3dmark.com/3dm/7000311?) (See the physics score, the average for my processor is around 11940).
In addition, the GTA V benchmark gives me scores around 15-20fps on the GTX 980.
I have 4100 2048KB hugepages set up and QEMU is using nearly all of them.
Is this a chipset issue and I just have to be patient for kernel 4.2? Is there a patch other than the ACS one in the OP?

And finally on the sound front, I am getting heavily distorted sound from standalone QEMU setup, I'm using hda for the qemu soundhw option and the following flags:

export QEMU_ALSA_DAC_BUFFER_SIZE=512 export QEMU_ALSA_DAC_PERIOD_SIZE=170 export QEMU_AUDIO_DRV=alsa export QEMU_AUDIO_DAC_FIXED_SETTINGS=1 export QEMU_AUDIO_DAC_FIXED_FREQ=48000 export QEMU_AUDIO_DAC_FIXED_FMT=S16 export QEMU_SDL_SAMPLES=1024

However on VM boot, I get the following errors:

alsa: Requested buffer size 512 was rejected, using 2048 alsa: Requested period size 170 was rejected, using 1024

Again, both of these are probably small things I'm missing, but I've been working on all these problems combined for a couple of weeks now, so I'm at my wits end.

Last edited by marca311 (2015-05-17 19:46:40)

Offline

#5115 2015-05-17 19:58:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@marca311, it looks like you've done no tuning aside from hugepages and host CPU type.  You're using emulated audio, default IDE disk, default rtl network, user mode networking, not attempting to pin vCPU threads...  you've got work to do before complaining about performance.  Nothing you're seeing has anything to do with chipset or the patch in 4.2 that's not even relevant to you.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5116 2015-05-17 21:24:25

Jimi
Member
From: Brooklyn, NY
Registered: 2009-09-25
Posts: 125
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have this working with an LGA 1150 CPU/mobo?

Offline

#5117 2015-05-17 21:33:40

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

patch in 4.2

If it's not X79 and X99, then what is 9-series chipsets?.. Those for LGA 1151(yeah, that one pin was very necessary) skylake(i7-5XXX) like H150 or what was it? Or what? Haven't seen that chipset belonging to people here...
Here: "On May 12, 2014, Intel announced the release of two 9-series chipsets, H97 and Z97."
Oh wait, then what is X79/X99?..

@marca311

aw wrote:

You're using emulated audio, default IDE disk, default rtl network, user mode networking, not attempting to pin vCPU threads...

Emulated audio: i don't know what he meant...
Default IDE disk: Use virtio-blk-pci like -device virtio-blk-pci,drive=disk1 and try playing with x-data-plane and caching, maybe others will suggest you their best options combos.
Default rtl network and user mode networking: tun/tap networking is very fast and somewhat simple to manage: here, have a link to read, i'm not sure if it's not outdated, but you get the idea.
vCPU pinning: redhat's awesome guide, but it's for libvirt - for qemu command-line there was cpuset and even core NoHZ mode trickery with isolcpus, with which i'm not so familiar(since i personally don't have spare cores to disable from host usage). I think there was something about it in the op-post.

In general - try swooping through the redhat's guide, it's darn awesome for performance improvements.

Jimi wrote:

Does anyone have this working with an LGA 1150 CPU/mobo?

Have you checked there?
I think filtering by the motherboard will help, though i can't help you much with that question.

Last edited by Duelist (2015-05-17 21:41:19)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5118 2015-05-17 21:39:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jimi wrote:

Does anyone have this working with an LGA 1150 CPU/mobo?

See part 1 of the guide in my sig


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5119 2015-05-17 21:42:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Jimi wrote:

Does anyone have this working with an LGA 1150 CPU/mobo?

See part 1 of the guide in my sig

I've checked your blog a second after he posted this, and your xeon is 1155. Or am i misunderstanding something?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5120 2015-05-17 21:44:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:
Jimi wrote:

Does anyone have this working with an LGA 1150 CPU/mobo?

See part 1 of the guide in my sig

I've checked your blog a second after he posted this, and your xeon is 1155. Or am i misunderstanding something?

Oops, you're right.  I do have another B85/Haswell setup, that works, but it has very limited I/O.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5121 2015-05-17 23:40:07

Jimi
Member
From: Brooklyn, NY
Registered: 2009-09-25
Posts: 125
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
aw wrote:

See part 1 of the guide in my sig

I've checked your blog a second after he posted this, and your xeon is 1155. Or am i misunderstanding something?

Oops, you're right.  I do have another B85/Haswell setup, that works, but it has very limited I/O.

I found a bunch of setups listed in https://docs.google.com/spreadsheet/ccc … _web#gid=2

Looks like my odds are good finally having Linux as the main OS on my home gaming PC.

Offline

#5122 2015-05-18 01:55:25

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, managed to convert my existing qemu script Win 7 Bios setup to a OVMF/libvirt/Uefi setup without data loss or reinstallation.  However I note that Win XP Mode no longer works.  Anyone else trying to run Win XP mode under KVM? I tried hiding the VM with the <KVM> hidden=on without success.  Realize it's a little OT.

Offline

#5123 2015-05-18 02:15:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

So, managed to convert my existing qemu script Win 7 Bios setup to a OVMF/libvirt/Uefi setup without data loss or reinstallation.  However I note that Win XP Mode no longer works.  Anyone else trying to run Win XP mode under KVM? I tried hiding the VM with the <KVM> hidden=on without success.  Realize it's a little OT.

Doesn't XP mode just require nested virt?  So you need to add options kvm_intel nested=1 (or kvm_amd) to a modprobe.d conf file, then expose the feature on the CPU using:

<cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='require' name='vmx'/>
  </cpu>

I haven't tried XP mode, but that makes cpu-z show the vt-x flag.

EDIT: Seems like just enabling the module option is enough, cpu-z shows vt-x regardless of the extra feature flag as long as nested is enabled in the kernel module.

Last edited by aw (2015-05-18 02:28:30)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5124 2015-05-18 09:40:57

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

regarding motherboard selection, I have good experience with SuperMicro X9DA7 (which probably is not what you want because it's getting long in tooth and is 2 socket). From this experience I would guess that newer workstation class (single socket) X10 motherboards would be worth a try. Thinking about X10SRA in particular; its chipset is C612

Last edited by Bronek (2015-05-18 10:09:49)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5125 2015-05-18 15:34:29

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'll give the VMT option a try, but Windows isn't complaining about lack of virtualization ability.  It has a DCOM error in the Event viewer similar to http://superuser.com/questions/282994/w … ion-failed Bear in mind that the same installation worked under qemu script.

Offline

#5126 2015-05-18 15:43:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

I'll give the VMT option a try, but Windows isn't complaining about lack of virtualization ability.  It has a DCOM error in the Event viewer similar to http://superuser.com/questions/282994/w … ion-failed Bear in mind that the same installation worked under qemu script.

You can find the resulting qemu command line that libvirt uses in /var/log/libvirt (or simply using ps while the VM is running) to compare to your script


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5127 2015-05-18 23:34:27

lefta
Member
Registered: 2015-05-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there,

Some months ago, I managed to make a virtual machine with Qemu/KVM/VGA passthrough. Everything went fine.
One or two months ago, I started my VM, but Windows 7 froze right after openning a session. I couldn't find a way to get past this freeze.
On freeze, Qemu outputs "main-loop: WARNING: I/O thread spun for 1000 iterations".
Waiting past the loop is not an option, as Windows crashes complaining for missed clock interrupts.
However, the installation CD does not crash, but freezes every few seconds (without redisplaying the error message though).

I stripped down the command line to:

qemu-system-x86_64 \ -m 1G \ -vga none \ -boot d \ -drive file=/home/ced/Documents/iso/windows7.iso,id=virtdrv,if=none,format=raw -device ide-cd,bus=ide.1,drive=virtdrv \ -enable-kvm \ -cpu host -smp 1,sockets=1,cores=1,threads=1

dmesg:

[ 1684.510717] kvm: zapping shadow pages for mmio generation wraparound

Prentend installing QEmu (-x means disabled, it is a gentoo)

app-emulation/qemu-2.3.0::gentoo USE="aio alsa bluetooth caps curl fdt jpeg ncurses nls opengl pin-upstream-blobs png python seccomp threads usb uuid vhost-net vnc -accessibility -debug -filecaps -glusterfs -gtk -gtk2 -infiniband -iscsi -lzo -nfs -numa -pulseaudio -rbd -sasl -sdl (-selinux) -smartcard -snappy -spice -ssh -static -static-softmmu -static-user -systemtap -tci {-test} -tls -usbredir -vde -virtfs -xattr -xen -xfs" PYTHON_TARGETS="python2_7" QEMU_SOFTMMU_TARGETS="x86_64 -aarch64 -alpha -arm -cris -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -moxie -or32 -ppc -ppc64 -ppcemb -s390x -sh4 -sh4eb -sparc -sparc64 -unicore32 -xtensa -xtensaeb" QEMU_USER_TARGETS="-aarch64 -alpha -arm -armeb -cris -i386 -m68k -microblaze -microblazeel -mips -mips64 -mips64el -mipsel -mipsn32 -mipsn32el -or32 -ppc -ppc64 -ppc64abi32 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -unicore32 -x86_64"

Before this problem, I updated my system. As I don't update it so often, I had about 200 updates (1/3 of my system), so I suspect an upgrade to be the culprit. But if I'm right, which one?
Google did not give any usefull result.

Hints :
- SystemRescueCD (x86) and Gentoo live CD (amd64) do not have the problem.
- Windows 7 installation CD (x86 & amd64) freezes.
- Windows 7 installation CD (x86) without KVM freezes even faster.
- All 2.x QEmu versions in portage tree suffer from the same problem (so from 2.1 to 2.3).
- Seabios 1.7.5 and 1.7.4 don't work.
- Linux kernels from 3.18.2 to 4.0.3 have the problem too.

My guesses :
- It is not KVM related (obviously).
- KVM seems to delay the freeze (why?).
- It does not seem to be directly QEmu related (but I could be wrong).
- It should be QEmu dependency related (but I'm not sure at all).

What can make Windows freeze but not Linux? ( Trolls, keep away please... )

Specs :
Gentoo Linux
QEmu 2.3.0 (I can downgrade/upgrade if needed)
Seabios 1.7.5 (I know 1.8 is out, but I couldn't install it on my gentoo)
CPU Intel i5-3470

I know it is not VGA passthrough related, but I though you were the most able to help me.

Offline

#5128 2015-05-19 01:05:46

ceri
Member
Registered: 2013-10-12
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
ceri wrote:

I don't really know/understand the architecture of the interaction between PCI/e ports and the CPU & MB, and so I'm confused about the ACS isolation stuff, but from what you've written here and on your blog, would I be correct in assuming that:

a) if i wanted to use discrete graphics (w/ a UEFI bios) on a single guest, and intel integrated graphics on the host (with no other PCI/PCIe devices), I wouldn't have to worry about using any patches or isolation, nor the type of CPU/motherboard I get (except for CPU/GFX compatibility with the MB and VT-d, etc)?

Yes, if the discrete graphics card for the guest is the only plugin card, then pretty much anything will work.  If your chipset is not quirked for PCH root port isolation, then you'd need to install it in a processor root port slot to keep it from getting grouped with onboard devices, if it is quirked, then any slot will work.

*snip*

Thanks for the clarifications!

Offline

#5129 2015-05-19 15:52:55

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again
First, thanks for your help! Now I can run a Windows 7 guest for gaming and everything works almost perfectly!
The only problem is due to the i915 vga arbitration patch: enabling it causes lots of animation problems on my GNOME host. I'm trying to understand if I can use OVMF
Currently I can't run QEMU with OVMF:
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html if I don't specify any vga passthrough in my QEMU configuration it works fine and it outputs everything in its window
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html if I specify a vga passthrough it does not produce any output
I have a Zotac GTX660 Ti and, using this tool, it seems that it doesn't support UEFI (but I can run Windows in UEFI mode without the virtualization, does it matter?). But according to a post on this thread (that I can't find now), there is a chance of making a non-UEFI card work with UEFI by using a romfile of another card. The problem is that, also in BIOS mode, if I specify ANY romfile the vga passthrough does not work anymore... Do you have any idea?
This is my system:

MB: Sapphire H61 Pure Platinum ITX
CPU: i7-3770
RAM: 8GB DDR3 1600MHz
GPU: Zotac Nvidia GTX660 Ti

and this is my QEMU configuration (in BIOS mode)

#!/bin/sh
QEMU_AUDIO_DRV=alsa
DISKIMG="/media/nicola/data/qemu/WindowsVM.img"
qemu-system-x86_64 --enable-kvm -M q35 -smp 8,sockets=1,cores=4,threads=2 -drive file=${DISKIMG},if=virtio  -m 6144 \
-soundhw hda -cpu host,kvm=off \
-net nic,model=virtio -net user,hostname=windowsvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1 -vga none

Thanks!

Last edited by corna (2015-05-19 15:54:06)

Offline

#5130 2015-05-19 17:19:59

Jimi
Member
From: Brooklyn, NY
Registered: 2009-09-25
Posts: 125
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried to pass two GPUs and use SLI on a VM?

Offline

#5131 2015-05-19 18:08:37

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

can somebody share a startup.nsh script or link for swiching to pci-e graphic on boot?

Thanks in advance.

Offline

#5132 2015-05-19 21:22:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Jimi wrote:

Has anyone tried to pass two GPUs and use SLI on a VM?

Not SLI, but XDMA Crossfire setup is running fine on my machine:
ASUS F2A55+Athlon X4 750K(trinity)+2xASUS HD7750 Silent. They do crossfire. Performance improvement is same as bare-metal. PCI-E 3.0 needed for improvement.

And i highly doubt SLI running - i've suggested trying HyperSLI to some guy with two GTX980 some pages back, and he failed doing the magic, so i think SLI won't work.

I don't know about regular crossfire, my crossfire setup doesn't require a separate bridge and relies solely on pci-e, so it's simpler to manage.

Last edited by Duelist (2015-05-19 21:30:29)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5133 2015-05-19 21:29:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:

Hi again
First, thanks for your help! Now I can run a Windows 7 guest for gaming and everything works almost perfectly!
The only problem is due to the i915 vga arbitration patch: enabling it causes lots of animation problems on my GNOME host. I'm trying to understand if I can use OVMF
Currently I can't run QEMU with OVMF:
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html if I don't specify any vga passthrough in my QEMU configuration it works fine and it outputs everything in its window
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html if I specify a vga passthrough it does not produce any output
I have a Zotac GTX660 Ti and, using this tool, it seems that it doesn't support UEFI (but I can run Windows in UEFI mode without the virtualization, does it matter?). But according to a post on this thread (that I can't find now), there is a chance of making a non-UEFI card work with UEFI by using a romfile of another card. The problem is that, also in BIOS mode, if I specify ANY romfile the vga passthrough does not work anymore... Do you have any idea?
This is my system:

MB: Sapphire H61 Pure Platinum ITX
CPU: i7-3770
RAM: 8GB DDR3 1600MHz
GPU: Zotac Nvidia GTX660 Ti

and this is my QEMU configuration (in BIOS mode)

#!/bin/sh
QEMU_AUDIO_DRV=alsa
DISKIMG="/media/nicola/data/qemu/WindowsVM.img"
qemu-system-x86_64 --enable-kvm -M q35 -smp 8,sockets=1,cores=4,threads=2 -drive file=${DISKIMG},if=virtio  -m 6144 \
-soundhw hda -cpu host,kvm=off \
-net nic,model=virtio -net user,hostname=windowsvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1 -vga none

Thanks!

Well... Running windows UEFI on bare metal won't help, chances are you're just having CSM on and it continues to use VGA.

Try dumping your card's ROM using envytools(and recheck for UEFI-compatibility) and adding it as a romfile option to your non-OVMF setup. If it works - try searching(i'd ask manufacturer first since my GT610 has GOP(UEFI) in it's shipped ROM) for UEFI-compatible ROM somewhere there.
When you start up your OVMF-VM, it WON'T USE YOUR GRAPHICS UNTIL WINDOWS DRIVERS KICK IN.
So it boots, shows you all the animations of windows logos and stuff, and only then fires up your GPU and starts outputting.
Basically, Schlunze in his posts request a .nsh EFI shell script to make OVMF use PT-d GPU from the very beginning, not relying on QXL(i didn't even think it's possible!).

You won't get accelerated QEMU Windows Graphics when not doing secondary passthrough.

And if i recall correctly, i915-patch actually breaks DRI, so... yeah.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5134 2015-05-19 21:43:48

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
corna wrote:

Hi again
First, thanks for your help! Now I can run a Windows 7 guest for gaming and everything works almost perfectly!
The only problem is due to the i915 vga arbitration patch: enabling it causes lots of animation problems on my GNOME host. I'm trying to understand if I can use OVMF
Currently I can't run QEMU with OVMF:
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html if I don't specify any vga passthrough in my QEMU configuration it works fine and it outputs everything in its window
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_20.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html if I specify a vga passthrough it does not produce any output
I have a Zotac GTX660 Ti and, using this tool, it seems that it doesn't support UEFI (but I can run Windows in UEFI mode without the virtualization, does it matter?). But according to a post on this thread (that I can't find now), there is a chance of making a non-UEFI card work with UEFI by using a romfile of another card. The problem is that, also in BIOS mode, if I specify ANY romfile the vga passthrough does not work anymore... Do you have any idea?
This is my system:

MB: Sapphire H61 Pure Platinum ITX
CPU: i7-3770
RAM: 8GB DDR3 1600MHz
GPU: Zotac Nvidia GTX660 Ti

and this is my QEMU configuration (in BIOS mode)

#!/bin/sh
QEMU_AUDIO_DRV=alsa
DISKIMG="/media/nicola/data/qemu/WindowsVM.img"
qemu-system-x86_64 --enable-kvm -M q35 -smp 8,sockets=1,cores=4,threads=2 -drive file=${DISKIMG},if=virtio  -m 6144 \
-soundhw hda -cpu host,kvm=off \
-net nic,model=virtio -net user,hostname=windowsvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1 -vga none

Thanks!

Well... Running windows UEFI on bare metal won't help, chances are you're just having CSM on and it continues to use VGA.

Try dumping your card's ROM using envytools(and recheck for UEFI-compatibility) and adding it as a romfile option to your non-OVMF setup. If it works - try searching(i'd ask manufacturer first since my GT610 has GOP(UEFI) in it's shipped ROM) for UEFI-compatible ROM somewhere there.
When you start up your OVMF-VM, it WON'T USE YOUR GRAPHICS UNTIL WINDOWS DRIVERS KICK IN.
So it boots, shows you all the animations of windows logos and stuff, and only then fires up your GPU and starts outputting.
Basically, Schlunze in his posts request a .nsh EFI shell script to make OVMF use PT-d GPU from the very beginning, not relying on QXL(i didn't even think it's possible!).

You won't get accelerated QEMU Windows Graphics when not doing secondary passthrough.

And if i recall correctly, i915-patch actually breaks DRI, so... yeah.

I've found 3 UEFI-compatible ROMs:

Asus.GTX660Ti.2048.130208.rom
Gigabyte.GTX660Ti.2048.130107.rom
MSI.GTX660Ti.2048.120828.rom

The system does not boot with any of these one (in non-OVMF mode)
I can't find any doc about envytools (I suppose I should use nvagetbios), can you please give me the doc's link or the syntax of the command?
Thanks

Offline

#5135 2015-05-19 22:45:47

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:

I've found 3 UEFI-compatible ROMs:

Asus.GTX660Ti.2048.130208.rom
Gigabyte.GTX660Ti.2048.130107.rom
MSI.GTX660Ti.2048.120828.rom

The system does not boot with any of these one (in non-OVMF mode)
I can't find any doc about envytools (I suppose I should use nvagetbios), can you please give me the doc's link or the syntax of the command?
Thanks

nvagetbios is a part of envytools. You git clone the source, compile it as noted in readme, and use nvagetbios to extract the bios. The instructions on how to use it are output by the program itself as usual.

Well.. You've tried to use regular SeaBios with those ROM and failed? Can you show me how you attach those files to your card?.. It should be something like "romfile=..."...

OT: yay, my next post is numbered 256!

Last edited by Duelist (2015-05-19 22:47:25)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5136 2015-05-20 05:13:11

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi, did anyone encounter a situation, where passing in VGA into linux guest worked (including X), but windows one (with the same config, except things like disk path etc.) returned in the device manager "Code 12: This device cannot find enough free resources that it can use" ?

I'm trying with current Arch with libvirt, since I have only one gpu (laptop), it's possible to blindly write "virsh destroy vmname", but not to find the qemu, kill it and rebind it myself smile)

hw:
HP Elitebook 8560p, radeon 6470M

Last edited by ghormoon (2015-05-20 05:13:58)

Offline

#5137 2015-05-20 05:41:54

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

hi, did anyone encounter a situation, where passing in VGA into linux guest worked (including X), but windows one (with the same config, except things like disk path etc.) returned in the device manager "Code 12: This device cannot find enough free resources that it can use" ?

I'm trying with current Arch with libvirt, since I have only one gpu (laptop), it's possible to blindly write "virsh destroy vmname", but not to find the qemu, kill it and rebind it myself smile)

hw:
HP Elitebook 8560p, radeon 6470M

I have the same result if I pass multiple cards (discrete GPU + virtual GPU). I also had the same result when running Windows on bare metal (discrete + integrated), so I suppose is a limitation of Windows. Have you selected -vga none in the QEMU configuration?

Offline

#5138 2015-05-20 06:00:19

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, even though -vga none doesn't actually make any difference. it only stops cirrus output from showing on vnc console, but will not remove the cirrus card itself.
If I remove the cirrus card, I end up in state, where the qemu will hang and not even try to boot anything. I can either have both, cirrus + passed in gpu, or none, but having only passed in one will hang seabios sad

Offline

#5139 2015-05-20 06:27:25

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
corna wrote:

I've found 3 UEFI-compatible ROMs:

Asus.GTX660Ti.2048.130208.rom
Gigabyte.GTX660Ti.2048.130107.rom
MSI.GTX660Ti.2048.120828.rom

The system does not boot with any of these one (in non-OVMF mode)
I can't find any doc about envytools (I suppose I should use nvagetbios), can you please give me the doc's link or the syntax of the command?
Thanks

nvagetbios is a part of envytools. You git clone the source, compile it as noted in readme, and use nvagetbios to extract the bios. The instructions on how to use it are output by the program itself as usual.

Well.. You've tried to use regular SeaBios with those ROM and failed? Can you show me how you attach those files to your card?.. It should be something like "romfile=..."...

OT: yay, my next post is numbered 256!

Yes, I should have read the README more carefully ;-)
I've extracted the BIOS of my GPU with envytools and the VM (in non-OVMF mode) works fine. But my BIOS is 1MB, while the techpowerup BIOSs are about 180KB... Are them in 2 different formats?
What do you mean with "it WON'T USE YOUR GRAPHICS UNTIL WINDOWS DRIVERS KICK IN."? Where does it output before the windows driver?
This is my QEMU script

#!/bin/sh
QEMU_AUDIO_DRV=pa
QEMU_PA_SAMPLES=128
DISKIMG="/winvm/winVM_C.img"
D_DRIVE="/home/nicola/Documents/winVM_D.img"
ROMFILE="/winvm/gtx660ti_bios/envytools_Zotac.rom"
OVMF_CODE="/winvm/ovmf-x64/OVMF_CODE-pure-efi.fd"
OVMF_VARS="/winvm/ovmf-x64/OVMF_VARS-pure-efi.fd"

ulimit -l 8409088

qemu-system-x86_64 --enable-kvm -M q35 -smp 8,sockets=1,cores=4,threads=2 -m 6144 \
-soundhw hda -cpu host,kvm=off -rtc base=localtime,clock=host \
-net nic,model=virtio -net user,hostname=windowsvm \
-drive file=${DISKIMG},if=virtio -drive file=${D_DRIVE},if=virtio \
-drive if=pflash,format=raw,readonly,file=${OVMF_CODE} \
-drive if=pflash,format=raw,file=${OVMF_VARS} \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,x-vga=on,multifunction=on,romfile=${ROMFILE} \
-device vfio-pci,host=01:00.1,bus=pcie.0 -vga none $@

The non-OVMF script is identical, except for the OVMF_CODE and OVMF_VARS lines

Offline

#5140 2015-05-20 13:34:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:

Yes, I should have read the README more carefully ;-)
I've extracted the BIOS of my GPU with envytools and the VM (in non-OVMF mode) works fine. But my BIOS is 1MB, while the techpowerup BIOSs are about 180KB... Are them in 2 different formats?
What do you mean with "it WON'T USE YOUR GRAPHICS UNTIL WINDOWS DRIVERS KICK IN."? Where does it output before the windows driver?

A typical GPU ROM is 64 kilobytes, UEFI-enabled one is 128k. That's on AMD. On Nvidia cards there may be differences, try opening the downloaded files and your own VBIOS in hex editor and look. But actually, have you even bothered asking the manufacturer?
Anyway, i think both files should work, the one you've extracted and the one you've downloaded.

Where does it outputs? Well, depends on the number of GPUs you're assigning: when using windows7+OVMF, you'll have to use QXL GPU additionally. Originally OVMF did output it's text stuff(and tianocore logo) only to QXL, now it mirrors it to the assigned card. Then windows boot logo comes up and renders itself only on QXL, or, if there's no QXL present, nowhere. That's why i've warned you - there was some occasions where people thought that their VM hung, but it was loading silently with no output. It's only hanging if it eats 100% of one vCPU.

Last edited by Duelist (2015-05-20 13:39:27)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5141 2015-05-20 13:39:12

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

hi, did anyone encounter a situation, where passing in VGA into linux guest worked (including X), but windows one (with the same config, except things like disk path etc.) returned in the device manager "Code 12: This device cannot find enough free resources that it can use" ?

I'm trying with current Arch with libvirt, since I have only one gpu (laptop), it's possible to blindly write "virsh destroy vmname", but not to find the qemu, kill it and rebind it myself smile)

hw:
HP Elitebook 8560p, radeon 6470M

Surprise-surprise, you have two devices doing their shady VGA stuff, and only ONE VGA memory range! If i understand this correctly, VGA arbitration should manage this, but windows... just disables the second card, be it cirrus or assigned. Try doing OVMF-wa... you have 6470M and a notebook, OVMF will not work.
Actually, you should be able to bind one device to the VM and use it. But you have a notebook, i can't really remember anyone doing this with their laptop, and i do remember aw saying something about mobile GPUs being "not so dedicated" as their discrete counterparts. Or was it about intel integrated gpus?..
If it hangs - try looking into dmesg via SSH or into qemu logs or somewhere else, there should be something in dmesg.

Last edited by Duelist (2015-05-20 13:39:53)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5142 2015-05-20 13:45:33

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

as for the range - yes, I've read a tutorial, where they said I have to disable cirrus in the device manager and reboot. but no change, there's down facing arrow (device disabled), but it still occupies the memory range sad
as for OVMF - I've dumped the rom some time ago and it was not OVMF, that's verified. No idea if there's any way to get OVMF one which will be compatible. Likely not.
This laptop doesn't even have the intel IGP wired in at all, just this Radeon as PCI device. it works in linux, so it is possible to pass in smile there are way more problems with laptops that actually can switch between those two.

for the hang - dmesg won't be much help, but If I make seabios output debug to console, it hangs at point where seabios tries to initialize the VGA. it looks like this when it boots http://pastebin.com/t7nRAdNj (with cirrus), and like this http://pastebin.com/hBQm9dme without cirrus (hangs).

Offline

#5143 2015-05-20 16:31:14

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

as for the range - yes, I've read a tutorial, where they said I have to disable cirrus in the device manager and reboot. but no change, there's down facing arrow (device disabled), but it still occupies the memory range sad
as for OVMF - I've dumped the rom some time ago and it was not OVMF, that's verified. No idea if there's any way to get OVMF one which will be compatible. Likely not.
This laptop doesn't even have the intel IGP wired in at all, just this Radeon as PCI device. it works in linux, so it is possible to pass in smile there are way more problems with laptops that actually can switch between those two.

for the hang - dmesg won't be much help, but If I make seabios output debug to console, it hangs at point where seabios tries to initialize the VGA. it looks like this when it boots http://pastebin.com/t7nRAdNj (with cirrus), and like this http://pastebin.com/hBQm9dme without cirrus (hangs).

It hangs just after it tries to load VGA ROM. So i think it's integrated into some other weird place. Check your ROM.
Try extracting it via sysfs and attaching it as romfile=... option.

Last edited by Duelist (2015-05-20 16:35:03)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5144 2015-05-20 18:21:31

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

Duelist wrote:

Basically, Schlunze in his posts request a .nsh EFI shell script to make OVMF use PT-d GPU from the very beginning, not relying on QXL(i didn't even think it's possible!).

The point is, now i have to switch onboard graphics or PCI-E graphics in main UEFI BIOS, then i can choose in the integratetd Bootloader(CSM) or gummiboot, wich OS shoud be started (Dual Boot).

It might be useful for isolating the "second" OS from scrach too, not fiddling around with wrong OS disk numbers and increases boot time^^.

The other way is to make fix settings for second OS in main BIOS, mount the EFI partion on two Systems linux nativ win virtualized. That´s a really bad idea i think. I don´t know if EFI stuff is ro by OSses.
And IGP graphics maybe not work during boot up linux in this case, thats a no go too. On the other hand is it possible to have two EFi partitons?

Maybe there is another way without startup.nsa?

Offline

#5145 2015-05-20 19:15:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Schlunze wrote:

Hi,

Duelist wrote:

Basically, Schlunze in his posts request a .nsh EFI shell script to make OVMF use PT-d GPU from the very beginning, not relying on QXL(i didn't even think it's possible!).

The point is, now i have to switch onboard graphics or PCI-E graphics in main UEFI BIOS, then i can choose in the integratetd Bootloader(CSM) or gummiboot, wich OS shoud be started (Dual Boot).

It might be useful for isolating the "second" OS from scrach too, not fiddling around with wrong OS disk numbers and increases boot time^^.

The other way is to make fix settings for second OS in main BIOS, mount the EFI partion on two Systems linux nativ win virtualized. That´s a really bad idea i think. I don´t know if EFI stuff is ro by OSses.
And IGP graphics maybe not work during boot up linux in this case, thats a no go too. On the other hand is it possible to have two EFi partitons?

Maybe there is another way without startup.nsa?

Sorry, my mind is a bit blurry, so i'll answer only one question you've asked and i've understood:

On the other hand is it possible to have two EFi partitons?

Yes. Moreover, if i recall correctly, you can have two different boot loaders on the same EFI special partition(ESP).


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5146 2015-05-20 19:54:48

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks,

Duelist wrote:

Yes. Moreover, if i recall correctly, you can have two different boot loaders on the same EFI special partition(ESP).

I look for a way to get rid of swiching primary graphic output in main bios for dual boot reasons.

Current standart configuration:
Bios setting for primary graphic output is onboard graphics.
Linux is bootet in CSM, i can use a win7 uefi vm with vga-passthrough. No need for vga arbitration due to qemu-ovmf on host. (Linux uefi not tested right now, maybe the EFI fat fs gets corrupt if its mountet on host and vm)

Dual Boot Windows7 UEFI:
Bios setting for primary graphic output is onboard graphics.
Windows is startet with standart vga-graphic, Intel driver is not installed right now. Pcie card shows up in windows device manger say everything is fine. But I can´ t use the nvidia card in this case.
For me it looks like a Windows VGA arbitration problem.

Dual Boot Windows7 UEFI: (most desired dual boot usecase)
Bios setting for primary graphic output is PCI-Express.
Pcie card is fully useable, the integrated graphic is turned off, not useable in this case.

I don´t know a solution for this switching problem.

Maybe a startup.nsh script to manage the inital switching depending on bootloader entry. Not possible?
Maybe boot Linux CSM with pcie as primary graphic output in main bios. Primary graphics is in pci-stub for qemu - i dont know what happens in this case but i will try it.

Thank you in advance again.

Last edited by Schlunze (2015-05-20 21:29:00)

Offline

#5147 2015-05-21 18:42:37

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Weeeeeeird. I have libvirt and kraxel-org-based jenkins-git images, and it works fine there.

Can you please share that image somewhere? Thanks!!

Offline

#5148 2015-05-21 19:17:23

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Schlunze wrote:

Thanks,

Duelist wrote:

Yes. Moreover, if i recall correctly, you can have two different boot loaders on the same EFI special partition(ESP).

I look for a way to get rid of swiching primary graphic output in main bios for dual boot reasons.

...snip...

Dual Boot Windows7 UEFI: (most desired dual boot usecase)
Bios setting for primary graphic output is PCI-Express.
Pcie card is fully useable, the integrated graphic is turned off, not useable in this case.

I don´t know a solution for this switching problem.

Maybe a startup.nsh script to manage the inital switching depending on bootloader entry. Not possible?
Maybe boot Linux CSM with pcie as primary graphic output in main bios. Primary graphics is in pci-stub for qemu - i dont know what happens in this case but i will try it.

Thank you in advance again.

Whaaat...
I didn't understand a thing again. Dual Boot - is a term of having two separate Operating Systems installed simultaneously. not running them. So you boot up your machine, choose, for example, linux, and use it. To get into windows, you shutdown your linux system, start the PC again and choose windows in your bootloader(usually grub). So dual booting doesn't involve virtualization at all.

If you want to pass-through a primary card into the VM - you just do it, considering you're using OVMF. No need to think about arbitration. Linux host boot loader(UEFI or legacy bios) doesn't matter at all.
So, you select pci-e graphics as your main GPU in your firmware setup(the "bios"), boot up bare metal windows and use it. Then you reboot to your linux host with a second GPU as host output, usually it is intel integrated GPU and having that "primary" discrete card drivers blacklisted and the device itself bound to pci-stub, futher ensuring that it's not used by anything in the host apart from vfio-pci.
Then, when the card is bound to vfio-pci, your start up your VM and live happily. That's how, i think, you want to live with your system.


@paperinick:

[user@crossfire ~]$ dnf info edk2.git-ovmf-x64.noarch Using metadata from Sun May 17 15:15:34 2015 (4 days, 6:49:58 hours old) Installed Packages Name : edk2.git-ovmf-x64 Arch : noarch Epoch : 0 Version : 0 Release : 20150518.b1004.g54ae9c0 Size : 11 M Repo : @System From repo : qemu-firmware-jenkins Summary : Open Virtual Machine Firmware URL : http://sourceforge.net/apps/mediawiki/tianocore/index.php?title=EDK2 License : BSD License (no advertising) with restrictions on use and redistribution Description : EFI Development Kit II : Open Virtual Machine Firmware : 64bit version

I was too lazy to fix the paths, so i just did tar -cvz /usr/share/edk2.git/ovmf-x64/*.fd -f ovmf.tar.gz...


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5149 2015-05-21 19:38:57

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

A typical GPU ROM is 64 kilobytes, UEFI-enabled one is 128k. That's on AMD. On Nvidia cards there may be differences, try opening the downloaded files and your own VBIOS in hex editor and look. But actually, have you even bothered asking the manufacturer?
Anyway, i think both files should work, the one you've extracted and the one you've downloaded.

Where does it outputs? Well, depends on the number of GPUs you're assigning: when using windows7+OVMF, you'll have to use QXL GPU additionally. Originally OVMF did output it's text stuff(and tianocore logo) only to QXL, now it mirrors it to the assigned card. Then windows boot logo comes up and renders itself only on QXL, or, if there's no QXL present, nowhere. That's why i've warned you - there was some occasions where people thought that their VM hung, but it was loading silently with no output. It's only hanging if it eats 100% of one vCPU.

Hi
I've installed Windows in UEFI mode and I'm trying to make it work with OVMF
When I have qxl activated it prints error 12, but if I remove it I don't have any output from the discrete card. I tried with the 3 UEFI-enabled ROMS found on techpowerup and with my Zotac ROM, but the result is always the same
How can I retrieve more debug informations?

Offline

#5150 2015-05-21 20:32:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:
Duelist wrote:

A typical GPU ROM is 64 kilobytes, UEFI-enabled one is 128k. That's on AMD. On Nvidia cards there may be differences, try opening the downloaded files and your own VBIOS in hex editor and look. But actually, have you even bothered asking the manufacturer?
Anyway, i think both files should work, the one you've extracted and the one you've downloaded.

Where does it outputs? Well, depends on the number of GPUs you're assigning: when using windows7+OVMF, you'll have to use QXL GPU additionally. Originally OVMF did output it's text stuff(and tianocore logo) only to QXL, now it mirrors it to the assigned card. Then windows boot logo comes up and renders itself only on QXL, or, if there's no QXL present, nowhere. That's why i've warned you - there was some occasions where people thought that their VM hung, but it was loading silently with no output. It's only hanging if it eats 100% of one vCPU.

Hi
I've installed Windows in UEFI mode and I'm trying to make it work with OVMF
When I have qxl activated it prints error 12, but if I remove it I don't have any output from the discrete card. I tried with the 3 UEFI-enabled ROMS found on techpowerup and with my Zotac ROM, but the result is always the same
How can I retrieve more debug informations?

1. What windows do you use? Win 7? Have you followed the instructions from OVMF whitepaper? Win 8? Win 8 won't need QXL at all.
2. ISA debug console, like

-debugcon file:debug.log -global isa-debugcon.iobase=0x402 

should produce debug.log with OVMF logging.
Have you tried installing windows with QXL and then installing the drivers despite seeing code 12? Maybe QXL will fall off(windows will disable it) and the GPU will fire up nicely(like it happened on my setup).


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5151 2015-05-22 07:45:12

corna
Member
Registered: 2015-05-13
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

1. What windows do you use? Win 7? Have you followed the instructions from OVMF whitepaper? Win 8? Win 8 won't need QXL at all.
2. ISA debug console, like

-debugcon file:debug.log -global isa-debugcon.iobase=0x402 

should produce debug.log with OVMF logging.
Have you tried installing windows with QXL and then installing the drivers despite seeing code 12? Maybe QXL will fall off(windows will disable it) and the GPU will fire up nicely(like it happened on my setup).

I'm using Windows 7 (look at my previous posts). If I use -M q35 I get an error 12, while if I use -M pc-i440fx-2.2 it reports an error 43 (even if I'm using kvm=off and no hyper-v enlightments)
A simple workaround for all the DRI problems is to use Wayland: in Wayland DRI works and the current status on gnome 3.14 is quite good.
I haven't read the OVMF whitepaper, I'm reading it right now thanks

Offline

#5152 2015-05-22 11:07:36

crashtua
Member
Registered: 2015-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone.
I have following configuration:

sabertooth fx990 r2.0
fx 8320
gtx 980(vm)
gtx 750ti(host)

kvm modules options:

cat /etc/modprobe.d/kvm*
options kvm-amd npt=0
options kvm ignore_msrs=1

vm command line:

sudo qemu-system-x86_64 \
-name qwerty_nosound \
-machine pc-i440fx-2.0,accel=kvm,usb=off \
-cpu host,kvm=off \
-m 8192 \
-mem-path /dev/hugepages \
-realtime mlock=off \
-smp 8,sockets=1,cores=8,threads=1 \
-uuid d616d9cd-aa9d-564a-7999-704787e5ea6c \
-no-user-config \
-nodefaults \
-rtc base=localtime \
-usb \
-boot menu=off,strict=on \
-msg timestamp=on \
-device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=06:00.1 \
-device virtio-scsi-pci,id=scsi \
-device AC97,id=sound0,bus=pci.0,addr=0x6 \
-drive file=/home/eugene/Downloads/ru_windows_7_professional_with_sp1_vl_build_x64_dvd_u_677774.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/dev/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/home/eugene/Downloads/virtio-win-0.1.96.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/eugene/dota.img,id=disk1,format=raw,if=none -device scsi-hd,drive=disk1 \
-netdev user,id=inet \
-device e1000,netdev=inet \
-netdev tap,id=hoa,script=/usr/qemu-ifup \
-device e1000,netdev=hoa,mac=DE:AD:BE:EF:68:98 \
-nographic \
-monitor stdio

Everything works fine, but every 10-15 seconds during gaming fps drops to 0 for 1 second and then back to normal. Can someone tell how this lags can be removed?

Last edited by crashtua (2015-05-22 12:31:55)

Offline

#5153 2015-05-22 13:17:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

corna wrote:
Duelist wrote:

1. What windows do you use? Win 7? Have you followed the instructions from OVMF whitepaper? Win 8? Win 8 won't need QXL at all.
2. ISA debug console, like

-debugcon file:debug.log -global isa-debugcon.iobase=0x402 

should produce debug.log with OVMF logging.
Have you tried installing windows with QXL and then installing the drivers despite seeing code 12? Maybe QXL will fall off(windows will disable it) and the GPU will fire up nicely(like it happened on my setup).

I'm using Windows 7 (look at my previous posts). If I use -M q35 I get an error 12, while if I use -M pc-i440fx-2.2 it reports an error 43 (even if I'm using kvm=off and no hyper-v enlightments)
A simple workaround for all the DRI problems is to use Wayland: in Wayland DRI works and the current status on gnome 3.14 is quite good.
I haven't read the OVMF whitepaper, I'm reading it right now thanks

Use 440FX because q35 gives you no advantages. Breeze through the OVMF whitepaper, you should be able to find something that you've missed, because i can't see the problem right now. Also, seems like you forgot about the ISA debug console...


crashtua wrote:

Everything works fine, but every 10-15 seconds during gaming fps drops to 0 for 1 second and then back to normal. Can someone tell how this lags can be removed?

Well, first you'll need to clear up your config.

-machine pc-i440fx-2.0,accel=kvm,usb=off \

Are you sure about that line? accel=kvm is soooo "old school", that it's not needed anymore.
Turning off USB looks strange, especially having "-usb \" later on.

-mem-path /dev/hugepages \ 

That may be the problematic place, as hugepages and npt may work not-so-well on AMD platforms, try toggling that and NPT.

-realtime mlock=off \

Now THAT's a strange thing to do. Back in the day, i've forced mlock=on only to find out it's default on.
Are you sure you know why you've turned it off and what that option does?

-device AC97,id=sound0,bus=pci.0,addr=0x6 \

I can see a soundcard device, but no sound output options, do you have sound on your VM?..

-netdev user,id=inet \ -device e1000,netdev=inet \ -netdev tap,id=hoa,script=/usr/qemu-ifup \ -device e1000,netdev=hoa,mac=DE:AD:BE:EF:68:98 \

Since you're obviously playing DotA, do you really need a user-mode networking AND a tap interface simultaneously? User mode networking can be very CPU consuming.

The general advice:
Monitor your CPU load. You're using 8 cores out of 8 available ones, maybe the host system does something strange.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5154 2015-05-22 13:59:34

crashtua
Member
Registered: 2015-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

That may be the problematic place, as hugepages and npt may work not-so-well on AMD platforms, try toggling that and NPT.

You mean set kvm-amd.npt to 1 and leave -mem-path /dev/hugepages?

Duelist wrote:

Now THAT's a strange thing to do. Back in the day, i've forced mlock=on only to find out it's default on.
Are you sure you know why you've turned it off and what that option does?

No, I am not. Just grabbed some qemu execute string from internet, and it worked for me, but as you can see, not as good as I expected:)
And thanks for reply, because it is really hard to follow up this 200-page topic.

Offline

#5155 2015-05-22 14:57:17

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Thanks for the suggestion about the vmx setting and looking at the qemu command line. (beginning p. 206, end of p 205)  However, the problem persists.  Unfortunately, I didn't check the functioning of XP Mode every step of the way in going from qemu-script q35 with SeaBios to libvirt 440fx with OVMF and UEFI, so I don't know which part of the transition made it stop working.
This is the command line from /var/log/libvirt/qemu:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name Windows7 -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu SandyBridge,+vmx,kvm=off -drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/Windows7_VARS.fd,if=pflash,format=raw,unit=1 -m 8192 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 6,sockets=6,cores=1,threads=1 -uuid f63ea610-45ae-4daa-bc14-4e2204XXXXX -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/dev/sr0,if=none,id=drive-ide0-0-1,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive file=/home/me/virtio-win-0.1-100.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/dev/cryptvg2/win7,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:XX:XX:XX:XX,bus=pci.0,addr=0x3 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x8 -device vfio-pci,host=03:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device usb-host,hostbus=4,hostaddr=4,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on 

This is the old version:

QEMU_AUDIO_TIMER_PERIOD=0 QEMU_AUDIO_DRV=alsa \ sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 \ -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -smp 6,sockets=1,cores=6,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none -nographic \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \ -usbdevice tablet -usbdevice host:0803:3095 -usbdevice host:0461:4d22 \ -usbdevice host:1131:1001 -usbdevice host:2304:023f \ -drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio \ -device piix4-ide,bus=pcie.0,id=piix4-ide \ -drive file=/dev/cdrom,id=cddisk -device ide-cd,bus=piix4-ide.1,drive=cddisk \ -net vde,vlan=0 -net nic,vlan=0,macaddr=52:45:4d:XX:XX:XX,model=virtio \ -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -rtc base=localtime \ -monitor unix:/home/me/monitor,server,nowait

I got rid of the hv stuff and upgraded the Nvidia drivers after converting to OVMF.  Obviously the cpu has changed form "host" to "SandyBridge" (there was no host option when I made the new version using VMM), and some of the usb devices have been changed around.  I suspect the main difference is libvirt.

Offline

#5156 2015-05-22 15:39:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

I got rid of the hv stuff and upgraded the Nvidia drivers after converting to OVMF.  Obviously the cpu has changed form "host" to "SandyBridge" (there was no host option when I made the new version using VMM), and some of the usb devices have been changed around.  I suspect the main difference is libvirt.

libvirt is just a mechanism for managing the VM, it's still just using QEMU in the end, so if XP mode can be done via QEMU commandline, it should be possible to do it with libvirt.  To get the "host" CPU option, just manually type "host-passthrough" in the virt-manager drop-down box.  It's not listed as a select-able option, but it works.  You can also manually edit the xml to something like:

 <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='1'/> </cpu>

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5157 2015-05-22 15:43:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

crashtua wrote:

Everything works fine, but every 10-15 seconds during gaming fps drops to 0 for 1 second and then back to normal. Can someone tell how this lags can be removed?

Perhaps something to do with leaving no cores available for the host?  You have an 8-core host and you've created an 8-core guest.  How is the normal running of the host supposed to not interfere with the VM?  It's gotta get CPU time somehow.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5158 2015-05-22 16:47:15

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw Thanks, I'll give it a try.  <rant> There's only so much reconfiguration one can do.  I'm concerned that Windows will once again decide that I have changed machines and that I have 3 days to activate. The last two times the automatic version didn't work and I had to call the Microsoft number, still semiautomatic.  Sooner or later I assume that won't work either and they'll insist I buy a new licence.  Rather annoying since I've had only one installation and am merely trying to optimize the VM.  Since that takes time (witness the last 200+ pages), activation is necessary at some point. Oi <rant/>

Re libvirt, surely there's more to it than that.  Libvirt uses virsh to attach devices; I use to just use vfio-bind. Also, I tried passing the qemu command line from /var/log/libvirt to the command line in a script, and get nothing, black screen, no VM.

Last edited by mostlyharmless (2015-05-22 17:07:11)

Offline

#5159 2015-05-22 17:18:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Re libvirt, surely there's more to it than that.  Libvirt uses virsh to attach devices; I use to just use vfio-bind. Also, I tried passing the qemu command line from /var/log/libvirt to the command line in a script, and get nothing, black screen, no VM.

libvirt passes various file descriptors by number rather than letting QEMU open things on its own, it also configures networks and passes tap devices, so you can't simply cut and paste the command line to a shell prompt and expect it to work.  You can however re-parse it into something that will work by hand, I've done this many times in debugging issues.  Why do you think that virsh nodedev-detach is really all that much different from vfio-bind?  libvirt knows how to manipulate devices and drivers via sysfs too.  If anything, the vfio-bind script is the path to avoid.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5160 2015-05-22 17:33:20

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah, I removed the network stuff, I assume the rest could run; I'll do more trimming if the cpu change doesn't do it.  I don't know about virsh vs vfio-bind; I only know what I've read here, and switched to virsh accordingly when I switched to libvirt.  But, as you point out, there are subtle differences, else one would not be preferable to the other. 

Before I switched to libvirt, I tried using virsh instead of vfio-bind with my old script and got the dreaded Windows 7B error every time. Switched back to vfio-bind, it disappeared again.  So there's something Windows detects differently between the two.  I have no idea whether it's part of the current XP mode problem.

Offline

#5161 2015-05-22 18:48:13

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

crashtua wrote:
Duelist wrote:

That may be the problematic place, as hugepages and npt may work not-so-well on AMD platforms, try toggling that and NPT.

You mean set kvm-amd.npt to 1 and leave -mem-path /dev/hugepages?

Duelist wrote:

Now THAT's a strange thing to do. Back in the day, i've forced mlock=on only to find out it's default on.
Are you sure you know why you've turned it off and what that option does?

No, I am not. Just grabbed some qemu execute string from internet, and it worked for me, but as you can see, not as good as I expected:)
And thanks for reply, because it is really hard to follow up this 200-page topic.

As i've been suggested on IRC: man 3 mlock. Basically, by saying "mlock=off" you're saying "HA! Let that VM memory be swapped!". Which is a very strange thing to do, considering our system being more or less realtime. So chop off that line. Also, using VFIO pins the memory in one place, preventing it from being moved around in the system memory, so, as i understand it, it will never actually be swapped, and that option won't do anything useful.

Test the VM performance with all combinations of npt=1 or npt=0 and hugepages enabled/disabled. That's all. You might eventually get better performance with npt=1 and hugepages enabled, or it may do the opposite thing due to AMD problems, as was noted two years old in the op-post.

And regarding your CPU usage, as aw suggests... I have 4 cores, of which i'm allowing VM to use all 4 cores, and i haven't observed anything alike to your problems when neither host nor guest system is doing something very cpu intensive. When the host loads all four cores with some stuff - the VM struggles and lags, and vice versa.
A generally good idea would be to isolate some(maybe all seven, leaving host with one core) CPU cores from host usage and dedicate them to the VM. Usually this helps a lot when pursuiting latency issues. You could set those cores to NoHZ mode(or something like that, i don't quite remember) or doing cpuset.

And there's a good in-depth manual available from redhat.

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Last edited by Duelist (2015-05-22 18:48:30)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5162 2015-05-22 19:38:32

crashtua
Member
Registered: 2015-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Честно? Я хз откуда я это взял хДД

Thanks everyone for reply, have good points for optimization. With leaving 2 cores for host and npt=1+hugepages i have constant, but 60 fps in dota(0-110 before changes), so most likely I am on right track.

Offline

#5163 2015-05-22 19:39:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

crashtua wrote:
Duelist wrote:

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Честно? Я хз откуда я это взял хДД

Thanks everyone for reply, have good points for optimization. With leaving 2 cores for host and npt=1+hugepages i have constant, but 60 fps in dota(0-110 before changes), so most likely I am on right track.

60fps sounds suspiciously like an in-game limitter


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5164 2015-05-22 19:47:06

paperinick
Member
Registered: 2015-04-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I was too lazy to fix the paths, so i just did

that's good enough, many thanks! I will experiment next week and hopefully see some progress!!

Offline

#5165 2015-05-22 20:49:52

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
crashtua wrote:
Duelist wrote:

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Честно? Я хз откуда я это взял хДД

Thanks everyone for reply, have good points for optimization. With leaving 2 cores for host and npt=1+hugepages i have constant, but 60 fps in dota(0-110 before changes), so most likely I am on right track.

60fps sounds suspiciously like an in-game limitter

Assuming it's still a source-based stuffgame, there's a variable called fps_max which, well, sets the limit.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5166 2015-05-23 04:15:28

Jodaco
Member
Registered: 2014-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So in general my config has been working great for me, but I do have random lockups of the guest and host with no information in the logs whatsoever or any pattern to when they happen. I am just using qemu with command line parameters rather than libvirt. I am on an intel platform.   I was wondering if anyone has any suggestions as to what kernel debugging options or other possible flags to pass to qemu for extra information so that I might track this down.

Offline

#5167 2015-05-23 07:14:59

mqddb
Member
Registered: 2015-04-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi ALL:
     If CPU Integrated Graphics can passthrough to VM?

Offline

#5168 2015-05-23 07:15:02

crashtua
Member
Registered: 2015-05-22
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

60fps sounds suspiciously like an in-game limitter

Duelist wrote:

Assuming it's still a source-based stuffgame, there's a variable called fps_max which, well, sets the limit.

Doublechecked, no limit and no vsync enabled. But anyway, it is stable fps, even new witcher performs on stable 40, and ocasional freezes cant break normal gameplay. And older games, like tomb rider, works on constant 60(everything max, vsync enabled).

Offline

#5169 2015-05-23 18:53:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mqddb wrote:

Hi ALL:
     If CPU Integrated Graphics can passthrough to VM?

Intel, no, not yet.  AMD, maybe.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5170 2015-05-24 13:28:13

protake
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally found some time to set up everything properly a few days ago and it works pretty well. The only thing that's bugging me is the mouse/keyboard. When passing them through, of course they aren't accessible from the host any more. Is there any way to make them accessible to both the guest and host? It'd be nice for the guest to be "escapable".

Offline

#5171 2015-05-24 13:31:15

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Synergy works great for mouse and keyboard.

Offline

#5172 2015-05-24 19:59:40

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

protake wrote:

Finally found some time to set up everything properly a few days ago and it works pretty well. The only thing that's bugging me is the mouse/keyboard. When passing them through, of course they aren't accessible from the host any more. Is there any way to make them accessible to both the guest and host? It'd be nice for the guest to be "escapable".

I use libvirt's graphics window(doing win7+QXL being disabled), which looks like this:
A quick screenshot
(Due to HDD hardware migration process, i can't run the guest right now).
When the system starts, VM starts to emulate QXL graphics, and that provides me a window where keyboard and mouse input can be grabbed. To escape the guest, i must push ctrl-shift-alt which happens very rarely in normal VM use(not doing emacs stuff, for example).
So far - so good, input latency is low enough to play quake 3 arena competitively.
Before moving to libvirt i did almost the same thing, adding -device qxl to my qemu commandline and running without -nographics to have a small window where input may be grabbed.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5173 2015-05-24 21:52:37

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi, I've got into state where I actually see windows booting (the logo). after that, screen goes black. I can rdp into it and it says device works OK. maybe it remembers something wrong from the previous (cirrus) setup sad
I'll try to reinstall it, if I figure out how to make it boot from CD without pressing "any key", since usb keyboard doesn't work at that point sad

another (maybe related, maybe not) problem I have is that when shutting down the domain, I get crash on "kernel tried to execute NX-protected page - exploit attempt?", I can't even halt the host after that ...
dmesg: http://pastebin.com/c3bRQQFe

any ideas on any of those problems? smile

Offline

#5174 2015-05-24 22:30:10

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

hi, I've got into state where I actually see windows booting (the logo). after that, screen goes black. I can rdp into it and it says device works OK. maybe it remembers something wrong from the previous (cirrus) setup sad
I'll try to reinstall it, if I figure out how to make it boot from CD without pressing "any key", since usb keyboard doesn't work at that point sad

another (maybe related, maybe not) problem I have is that when shutting down the domain, I get crash on "kernel tried to execute NX-protected page - exploit attempt?", I can't even halt the host after that ...
dmesg: http://pastebin.com/c3bRQQFe

any ideas on any of those problems? smile

You're not using OVMF, right? If so...
What the mess you have with your hardware?
What devices do you have and what are you trying to pass-through?
I see that you have kernel modesetting enabled and radeon module enabled, and that just attracts problems.

 [ 9.016962] fb: switching to radeondrmfb from VESA VGA

Framebuffer thingie works via VGA, and you're trying to isolate VGA access and use it from vfio(and the VM) doing x-vga=on...

...

How the hell this thing is supposed to work? That's a notebook, it has one screen only! Are you trying to do secondary pass-through or throw a VNC connection from-guest-to-host to have a small window with windows(sic!) running 3D stuff in it? Or what?

Seems like you unbind the device from radeon and attach it to vfio-pci... but the device isn't properly returned to it's un-enabled state:

 [ 218.594123] [drm:rv770_stop_dpm [radeon]] *ERROR* Could not force DPM to low.

and something hints me that you'll have trouble with ROM, assuming 03:00.0 is actually the GPU in question(you haven't showed us lspci output yet), because...
AHAHAHAHA I AM GOING INSANE, THAT'S AN SD CONTROLLER! WHICH IS BROKEN TOO!

 [ 227.044242] vfio-pci 0000:03:00.0: Invalid ROM contents

And then even more weird cra shi things happen:

[ 227.054680] dmar: DRHD: handling fault status reg 3 [ 227.054688] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ef000 DMAR:[fault reason 06] PTE Read access is not set

THAT looks very bad, what the hell is that 00:1d.0 device? Looks like you're trying to pass-through it too.

OH WELL, just give me the LSPCI output and tell me what are you trying to do, all i see now is a mess of hardware and virtualization everywhere.
Please, list iommu_groups too.

...

[ 1150.557220] mmc0: SDHCI controller on PCI [0000:03:00.0] using DMA [ 1150.639153] vfio_pci_disable: Failed to reset device 0000:05:00.0 (-11)

Seems like somewhere there you've shutdown your VM, and...
Your system just exploded with errors:

[ 1159.834105] [drm:uvd_v1_0_start [radeon]] *ERROR* UVD not responding, trying to reset the VCPU!!! eight more lines alike [ 1169.050536] [drm:uvd_v1_0_start [radeon]] *ERROR* UVD not responding, trying to reset the VCPU!!! [ 1169.070551] [drm:uvd_v1_0_start [radeon]] *ERROR* UVD not responding, giving up!!! [ 1169.070572] [drm:evergreen_startup [radeon]] *ERROR* radeon: error initializing UVD (-1).

I have absolutely no clue what that means, but three exclamation marks and "trying to reset the VCPU" looks very serious.
But nevertheless, it tries to continue and...

[ 1170.117249] radeon 0000:01:00.0: fb0: radeondrmfb frame buffer device [ 1170.117251] radeon 0000:01:00.0: registered panic notifier [ 1170.135759] [drm] Initialized radeon 2.41.0 20080528 for 0000:01:00.0 on minor 0 [ 1170.136029] kernel tried to execute NX-protected page - exploit attempt? (uid: 0) [ 1170.136065] BUG: unable to handle kernel paging request at ffff880429522c40

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5175 2015-05-24 22:45:50

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lspci:
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b4)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b4)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b4)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation QM67 Express Chipset Family LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Seymour [Radeon HD 6400M/7400M Series]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
03:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
03:00.2 SD Host controller: JMicron Technology Corp. Standard SD Host Controller (rev 30)
04:00.0 Network controller: Intel Corporation Centrino Advanced-N 6205 [Taylor Peak] (rev 34)
05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)

virt manager xml: http://pastebin.com/6PqWB18r (note that x-vga=on is added in the qemu-wrapper)

as for the questions asked:
OVMF: no, it seems to be unsupported on my HW.

I'm passing in 00:1a.0 00:1b.0 00:1d.0 03:00.0 03:00.2 01:00.0 01:00.1 05:00.0 i.e. basically everything, that is not a pci/sata conroller or a network card.

true I've let the radeon to load, I'll try next time without it. linux vm (almost the same config) pickes it ok though smile

how is it supposed to work - to have vga and all the periphetals passed in, host will be headless.  most of the work I'll do on the vm with VGA passed in with some other VMs  around for testing, services and such.

SD controller is broken by default smile interesting though that it works in the vm big_smile

as for the crash in the end, that happens in the linux VM too, though until shutdown that works ok.

Offline

#5176 2015-05-24 23:01:11

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ghormoon:
And what about the video output? Switchable graphics always was a problem, even on windows machines.
Since you're having a separate(logitech G15) keyboard attached, i guess you have your video output routed via external display connector to the separate screen, and the notebook is actually... closed? Y' trying to assemble something only from the available parts and the PC has a broken screen or something? Why you aren't doing this thing with a normal PC?..

If host will be headless - i'd suggest you chopping the said head off before doing anything with the VM.
Don't load the radeon. Don't start Kernel ModeSetting(KMS). Disable all video output as we'll need VGA free from anything.
And drop all the unnecessary devices like the SD controller just for testing the GPU part. Then, after everything's set, you might attach all the other devices needed. Stuff like SD controllers may be not the separate device, but a part of something integrated, having their ROM and power controls hidden in depths of proprietary stuff and closed firmwares.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5177 2015-05-24 23:25:18

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

as for video output, I use the internal display. the reason I'm not doing this on desktop PC is that desktop PC is too heavy to carry daily in backpack smile
as for the HW, there is not any switchable graphics. the integrated one in the i7 cpu is not wired in at all. there's no way to turn off the radeon if you wanna have some output. not like the laptops which allow you to switch between igp and radeon.
as for the g15 usb keyboard - when I go headless, I can't use the internal one for input, since it's connected as ps/2 one.  That's next level, which involves patching qemu. so that's the last one smile

I'll retry with blacklisted radeon, kms off, also tell grub not to change modes, let's see what it does.
current state is that linux VM can both show boot messages and actually work on the passed in GPU, windows goes black after showing logo. likely at the point where the radeon driver kicked in.

Offline

#5178 2015-05-24 23:49:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

I'll retry with blacklisted radeon, kms off, also tell grub not to change modes, let's see what it does.
current state is that linux VM can both show boot messages and actually work on the passed in GPU, windows goes black after showing logo. likely at the point where the radeon driver kicked in.

Try attaching a VGA(or hdmi or whatever you have there) screen to the GPU. Maybe it gets in trouble with the integrated screen, i'm not sure how it's attached to the GPU. I guess you'd be disappointed to have a burden of a separate screen.
If you're saying that intel's GPU isn't wired anyhow at all - that looks promising, maybe you have a real, full-featured AMD GPU inside, that fact will simplify some things.

So, yeah, do the clean install with no unnecessary hardware attached and see how it goes.

If this will work - it'll be the first actual notebook we've observed passthrough running.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5179 2015-05-25 02:14:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ghormoon

Are you letting the host radeon driver claim the GPU that you're assigning?  Linux GPU drivers are really not that good at releasing devices.  What happens if you force it to be bound to pci-stub?  Secondary laptop graphics are pretty much uncharted territory, but you've got a lot of errors throughout that dmesg log, which all seem to point to radeon.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5180 2015-05-25 10:18:57

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've gotten everything working nicely with a Radeon card passed through to a Windows guest and using Intel integrated graphics for host. However, I've found that I am starting to use more USB devices in the Windows guest VM (game controllers, wheels, etc), and find it inconvenient to have to start and stop the VM each time I want to use a new/different USB device. From aw blog, it is mentioned that entire USB controlled could be passed to VM using same process (vfio-pci). But... from lsusb I find that the USB bus that a plugged in device resides on is always the same 3.0 controller bus no matter what physical USB port I plug into (motherboard has 2 USB 2.0 ports, 6 3.0 ports, and an additional 1x 2.0 and 1x 3.0 port are available through the case). lspci shows 2 EHCI 2.0 controllers and 1 XHCI 3.0 controllers, so it doesn't make sense that all USB ports would be handled by a single controller, especially the 2.0 ports. Is there something I am not understanding here? Would a separate pci-e USB card dedicated to the guest be a better option? What would be best way to get hot-pluggability of USB devices to guest?

Physical layout of motherboard for reference:
oie_One_RPCjb_V7hw.png

lspci | grep USB:

00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

lsusb:

Bus 004 Device 002: ID 8087:8001 Intel Corp. Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 002: ID 8087:8009 Intel Corp. Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 045b:0210 Hitachi, Ltd Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 007: ID 09da:9066 A4Tech Co., Ltd. F3 V-Track Gaming Mouse Bus 001 Device 006: ID 05c7:2012 Qtronix Corp. Bus 001 Device 002: ID 045b:0209 Hitachi, Ltd Bus 001 Device 021: ID 045e:028e Microsoft Corp. Xbox360 Controller Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

A4Tech and Qtronix are plugged into motherboard 2.0 port, Xbox360 controller is plugged into 3.0 port, but everything gets put on Bus 001...

$ readlink /sys/bus/usb/devices/usb1 ../../../devices/pci0000:00/0000:00:14.0/usb1 $ readlink /sys/bus/usb/devices/usb2 ../../../devices/pci0000:00/0000:00:14.0/usb2 $ readlink /sys/bus/usb/devices/usb3 ../../../devices/pci0000:00/0000:00:1a.0/usb3 $ readlink /sys/bus/usb/devices/usb4 ../../../devices/pci0000:00/0000:00:1d.0/usb4

Shouldn't the 2.0 ports at least go to Bus 003 or Bus 004?

Last edited by mutiny (2015-05-25 10:42:27)

Offline

#5181 2015-05-25 10:30:26

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Usb 2.0 ports are probably ones on motherboard, usb headers for front usb...

Offline

#5182 2015-05-25 10:38:12

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Usb 2.0 ports are probably ones on motherboard, usb headers for front usb...

The motherboard has both 2.0 and 3.0 ports, as well as headers for front/external 2.0 and 3.0, and even if I plug into either of the front USB ports they still end up on same bus/controller?

Last edited by mutiny (2015-05-25 10:41:45)

Offline

#5183 2015-05-25 10:45:49

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For me on Z77 intel board, usb 2.0 are accessible through 7 pin (or 9) headers on board, and on some revisions also 2 top ports on rear.

They work with vfio passing 00:1d.0, 00:1a.0 or 00:1f.0 (depends on revision)

Last edited by slis (2015-05-25 10:47:14)

Offline

#5184 2015-05-25 20:04:13

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mutiny wrote:

I've gotten everything working nicely with a Radeon card passed through to a Windows guest and using Intel integrated graphics for host. However, I've found that I am starting to use more USB devices in the Windows guest VM (game controllers, wheels, etc), and find it inconvenient to have to start and stop the VM each time I want to use a new/different USB device. From aw blog, it is mentioned that entire USB controlled could be passed to VM using same process (vfio-pci). But... from lsusb I find that the USB bus that a plugged in device resides on is always the same 3.0 controller bus no matter what physical USB port I plug into (motherboard has 2 USB 2.0 ports, 6 3.0 ports, and an additional 1x 2.0 and 1x 3.0 port are available through the case). lspci shows 2 EHCI 2.0 controllers and 1 XHCI 3.0 controllers, so it doesn't make sense that all USB ports would be handled by a single controller, especially the 2.0 ports. Is there something I am not understanding here? Would a separate pci-e USB card dedicated to the guest be a better option? What would be best way to get hot-pluggability of USB devices to guest?

I had this exact same problem with my ASRock Z87 Extreme 6. No matter what USB port I used (external or internal), the device would always end up behind the XHCI controller. This is how the situation looks like in lshw:

 *-usb:0 description: USB controller product: 8 Series/C220 Series Chipset Family USB xHCI vendor: Intel Corporation physical id: 14 bus info: pci@0000:00:14.0 version: 05 width: 64 bits clock: 33MHz capabilities: pm msi xhci bus_master cap_list configuration: driver=xhci_hcd latency=0 resources: irq:34 memory:f0c20000-f0c2ffff *-usbhost:0 product: xHCI Host Controller vendor: Linux 4.0.0-1-amd64 xhci-hcd physical id: 0 bus info: usb@4 logical name: usb4 version: 4.00 capabilities: usb-3.00 configuration: driver=hub slots=6 speed=5000Mbit/s *-usb description: USB hub product: AS2107 vendor: ASMedia physical id: 3 bus info: usb@4:3 version: 0.01 serial: USB2.0 Hub capabilities: usb-3.00 configuration: driver=hub maxpower=8mA slots=4 speed=5000Mbit/s *-usbhost:1 product: xHCI Host Controller vendor: Linux 4.0.0-1-amd64 xhci-hcd physical id: 1 bus info: usb@3 logical name: usb3 version: 4.00 capabilities: usb-2.00 configuration: driver=hub slots=14 speed=480Mbit/s *-usb:0 description: USB hub product: AS2107 vendor: ASMedia physical id: 3 bus info: usb@3:3 version: 0.01 serial: USB2.0 Hub capabilities: usb-2.10 configuration: driver=hub maxpower=100mA slots=4 speed=480Mbit/s *-usb:1 description: Mouse product: Powerful Receiver vendor: Itron physical id: 4 bus info: usb@3:4 version: 1.00 capabilities: usb-2.00 configuration: driver=usbhid maxpower=40mA speed=2Mbit/s *-usb:2 description: Audio device product: C-Media USB Headphone Set vendor: C-Media Electronics, Inc. physical id: d bus info: usb@3:d version: 1.00 capabilities: usb-1.10 audio-control configuration: driver=usbhid maxpower=100mA speed=12Mbit/s *-usb:1 description: USB controller product: 8 Series/C220 Series Chipset Family USB EHCI #2 vendor: Intel Corporation physical id: 1a bus info: pci@0000:00:1a.0 version: 05 width: 32 bits clock: 33MHz capabilities: pm debug ehci bus_master cap_list configuration: driver=ehci-pci latency=0 resources: irq:16 memory:f0c3c000-f0c3c3ff *-usbhost product: EHCI Host Controller vendor: Linux 4.0.0-1-amd64 ehci_hcd physical id: 1 bus info: usb@1 logical name: usb1 version: 4.00 capabilities: usb-2.00 configuration: driver=hub slots=2 speed=480Mbit/s *-usb description: USB hub vendor: Intel Corp. physical id: 1 bus info: usb@1:1 version: 0.05 capabilities: usb-2.00 configuration: driver=hub slots=6 speed=480Mbit/s *-usb:2 description: USB controller product: 8 Series/C220 Series Chipset Family USB EHCI # vendor: Intel Corporation physical id: 1d bus info: pci@0000:00:1d.0 version: 05 width: 32 bits clock: 33MHz capabilities: pm debug ehci bus_master cap_list configuration: driver=ehci-pci latency=0 resources: irq:23 memory:f0c3b000-f0c3b3ff *-usbhost product: EHCI Host Controller vendor: Linux 4.0.0-1-amd64 ehci_hcd physical id: 1 bus info: usb@2 logical name: usb2 version: 4.00 capabilities: usb-2.00 configuration: driver=hub slots=2 speed=480Mbit/s *-usb description: USB hub vendor: Intel Corp. physical id: 1 bus info: usb@2:1 version: 0.05 capabilities: usb-2.00 configuration: driver=hub slots=8 speed=480Mbit/s

Why would manufacturers implement the USB 2.0 ports behind USB 3.0 controllers (and even throw in an additional ASMedia AS2107 hub), is quite strange. How can it possibly be cheaper to do it that way, than to wire the USB 2.0 ports from the chipset?

Anyways, I solved the problem by buying (reluctantly) a cheap uPD720201 based PCI-e USB 3.0 controller off ebay and passed that trough to win7. Works great. But it really sucks having to use an extension card, when the motherboard physically already has multiple USB controllers and plenty of ports :\

Offline

#5185 2015-05-26 02:49:43

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well, I've retried linux guest without radeon on the host (added to grub defaults "intel_iommu=on nofb nomodeset modprobe.blacklist=radeon quiet" and turned off graphical output in grub)
the result is that the dmesg is not *that* bad, but still not ok.

on VM start I get: (00:1d.0 is usb controller where I have keyboard and mouse, they work in the vm)
[  125.866335] dmar: DRHD: handling fault status reg 3
[  125.866400] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ef000
               DMAR:[fault reason 06] PTE Read access is not set
[  125.992123] kvm: zapping shadow pages for mmio generation wraparound
[  132.713767] br0: port 1(vnet0) entered forwarding state
[  134.660405] kvm [585]: vcpu1 unhandled rdmsr: 0x606
[  139.884802] kvm [585]: vcpu0 unhandled rdmsr: 0x611
[  139.884885] kvm [585]: vcpu0 unhandled rdmsr: 0x639
[  139.884959] kvm [585]: vcpu0 unhandled rdmsr: 0x641
[  139.885026] kvm [585]: vcpu0 unhandled rdmsr: 0x619

on vm shutdown, nothing happens, virsh list says it's "in shutdown" (forever smile)
if I do virsh destroy vmname it hangs.
then if I do virsh list, it hangs.
I need to shoot down the qemu process manually, then virsh list works again.
any ideas what's wrong?

Offline

#5186 2015-05-26 03:33:02

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

update: I've tried windows again, from a snapshot without amd drivers.
I can boot it and have the gpu working (as generic VGA adapter), it even recognized correctly max resolution of the screen if I tried to change it (defualt was 800x600).  so far so good.
When I tried to install the driver, during "detecting hardware", I get BSOD 0xCA sad (nothing new in host dmesg at that time, boot time it gives the same errors as linux)

Last edited by ghormoon (2015-05-26 03:33:59)

Offline

#5187 2015-05-26 03:35:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

update: I've tried windows again, from a snapshot without amd drivers.
I can boot it and have the gpu working (as generic VGA adapter), it even recognized correctly max resolution of the screen if I tried to change it (defualt was 800x600).  so far so good.
When I tried to install the driver, during "detecting hardware", I get BSOD 0xCA sad

Use the driver specific to your hardware, don't go the detect driver path, even for updates via the catalyst control panel.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5188 2015-05-26 04:26:33

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm not sure what do you mean. When I select the driver on the website manually, it still gives me the catalyst installer, that goes through the "detecting graphics hardware" step and there it crashes. I can't select anything in the installer manually. should I extract only the driver and don't install the rest or what?

Offline

#5189 2015-05-26 05:56:23

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just a heads up it looks like any Nvidia drivers past 347.88 are disabling themselves again, and spitting back code 43 errors.

Offline

#5190 2015-05-26 10:32:23

toxster
Member
Registered: 2015-04-28
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys

Introducing My next project, Virtualized OSX Yosemite (10.10.3) host with passthrough Nvidia GTX 970 running at 4K 60Hz..

Got everything working except USB passthrough of mouse (only apple keyboard works for some apparent reason)

Heaven ~68fps 720p with Extreme profile, 1080p ~40fps will benchmark with my win8.1 host later.

looks promising smile

Modo: Gigabyte X99-Gaming 5
CPU: Intel 5960x
Passthrough GPU: Asus Strix NVidia 970 GTX

Last edited by toxster (2015-05-26 10:33:54)

Offline

#5191 2015-05-26 13:54:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Punkbob wrote:

Just a heads up it looks like any Nvidia drivers past 347.88 are disabling themselves again, and spitting back code 43 errors.

Running 352.86 here, no Code 43 problems.  The NVIDIA Control Panel stopped working, but I see others reporting that in non-VM applications.  It worked on 350.12.  Code 43 is a pretty generic "something is wrong" error, hypervisor detection isn't the only reason for it to get thrown.  I recently discovered that failure to plugin aux power to cards also generates a Code 43.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5192 2015-05-26 13:59:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

I'm not sure what do you mean. When I select the driver on the website manually, it still gives me the catalyst installer, that goes through the "detecting graphics hardware" step and there it crashes. I can't select anything in the installer manually. should I extract only the driver and don't install the rest or what?

Just trying to say that the "Automatically detect and install driver" code is known to cause BSODs, if you're doing the "Manually select your driver" path, then you're getting something else.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5193 2015-05-26 14:17:11

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't get where you maen to do that. on the web site, I do manually select (notebook gpu > HD series > 6xxx> win 7 x64) and it gives me 288MB catalyst. that one only gives me default/custom selection and field for path. on next (either default or custom), it goes through few steps before allowing you to select things (at least on custom, default won't let you) and it crashes before that selection just after saying "detecting graphics hardware" in the progress bar. I can't find any way to get more specific driver than by the series 6xxx. What am I missing?

Offline

#5194 2015-05-26 15:42:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

I don't get where you maen to do that. on the web site, I do manually select (notebook gpu > HD series > 6xxx> win 7 x64) and it gives me 288MB catalyst. that one only gives me default/custom selection and field for path. on next (either default or custom), it goes through few steps before allowing you to select things (at least on custom, default won't let you) and it crashes before that selection just after saying "detecting graphics hardware" in the progress bar. I can't find any way to get more specific driver than by the series 6xxx. What am I missing?

You're doing everything right, it just doesn't work somewhere else. Does dmesg say anything when the bsod happens?
And the bsod in question.. 0x000000CA you say? What are the other parameters? That's an interesting error actually. And try chopping off that 1d controller and plug in USB devices the other way, i'm not sure if it isn't causing some other weird things to happen.

Last edited by Duelist (2015-05-26 15:43:11)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5195 2015-05-26 16:15:40

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

not a single line in dmesg.
when I'll get home, I'll retry and get the missing bsod parameters.
currently 1d is only controller I can use, maybe I can not passthrough it and pass the usb devices one by one, that should be possible somehow, though I've not tried it yet smile
or maybe I can chop them off and try to install the driver over rdp? let's see smile

Offline

#5196 2015-05-26 17:09:17

lordleto
Member
Registered: 2014-11-14
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mind sharing your xml/commandline?

toxster wrote:

Hey guys

Introducing My next project, Virtualized OSX Yosemite (10.10.3) host with passthrough Nvidia GTX 970 running at 4K 60Hz..

Got everything working except USB passthrough of mouse (only apple keyboard works for some apparent reason)

Heaven ~68fps 720p with Extreme profile, 1080p ~40fps will benchmark with my win8.1 host later.

looks promising smile

Modo: Gigabyte X99-Gaming 5
CPU: Intel 5960x
Passthrough GPU: Asus Strix NVidia 970 GTX

Offline

#5197 2015-05-26 17:19:34

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Punkbob wrote:

Just a heads up it looks like any Nvidia drivers past 347.88 are disabling themselves again, and spitting back code 43 errors.

Running 352.86 here, no Code 43 problems.  The NVIDIA Control Panel stopped working, but I see others reporting that in non-VM applications.  It worked on 350.12.  Code 43 is a pretty generic "something is wrong" error, hypervisor detection isn't the only reason for it to get thrown.  I recently discovered that failure to plugin aux power to cards also generates a Code 43.

Hmm, I will look into it more then. Btw, are you running Windows 10 or 8.1, cause it might be an issue on my end as I am running my VMs with Windows 10.

Offline

#5198 2015-05-26 19:41:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Punkbob wrote:
aw wrote:
Punkbob wrote:

Just a heads up it looks like any Nvidia drivers past 347.88 are disabling themselves again, and spitting back code 43 errors.

Running 352.86 here, no Code 43 problems.  The NVIDIA Control Panel stopped working, but I see others reporting that in non-VM applications.  It worked on 350.12.  Code 43 is a pretty generic "something is wrong" error, hypervisor detection isn't the only reason for it to get thrown.  I recently discovered that failure to plugin aux power to cards also generates a Code 43.

Hmm, I will look into it more then. Btw, are you running Windows 10 or 8.1, cause it might be an issue on my end as I am running my VMs with Windows 10.

8.1, AIUI 10 has numerous issues running as a VM


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5199 2015-05-26 20:58:52

toxster
Member
Registered: 2015-04-28
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lordleto wrote:

Mind sharing your xml/commandline?

Sure, havent gotten around in attempting to forward the entire USB controller, supposedly doing so should work better.

Some comments to the XML below:

yoyo.iso is yosemite installer created with unibeast, end result is an installation that don't require any "hacked" kexts.
the direkt kernel /root/vm/boot is chameleon r2700 boot

<domain type='kvm' id='8' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>osx</name> <uuid>ddb18084-f1d7-4041-b7be-9ab0437470c6</uuid> <memory unit='KiB'>8286208</memory> <currentMemory unit='KiB'>8286208</currentMemory> <vcpu placement='static'>4</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> <kernel>/root/vm/boot</kernel> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>core2duo</model> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/cam/mac_hdd.img'/> <backingStore/> <target dev='sda' bus='sata'/> <alias name='sata0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <interface type='network'> <mac address='c8:85:50:7f:9c:bf'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='e1000-82545em'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x05ac'/> <!-- apple keyboard --> <product id='0x0250'/> <address bus='1' device='9'/> </source> <alias name='hostdev0'/> </hostdev> <memballoon model='none'> <alias name='balloon0'/> </memballoon> </devices> <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='isa-applesmc,osk=googleit!'/> <!-- read out from your mac or google it. --> <qemu:arg value='-smbios'/> <qemu:arg value='type=2'/> <qemu:arg value='-device'/> <qemu:arg value='ide-drive,bus=ide.1,drive=MacDVD'/> <qemu:arg value='-drive'/> <qemu:arg value='id=MacDVD,if=none,snapshot=on,file=/root/vm/yoyo.iso'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.1'/> <qemu:arg value='-vga'/> <qemu:arg value='none'/> </qemu:commandline> </domain>

Offline

#5200 2015-05-26 20:59:13

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ghormoon wrote:

I don't get where you maen to do that. on the web site, I do manually select (notebook gpu > HD series > 6xxx> win 7 x64) and it gives me 288MB catalyst. that one only gives me default/custom selection and field for path. on next (either default or custom), it goes through few steps before allowing you to select things (at least on custom, default won't let you) and it crashes before that selection just after saying "detecting graphics hardware" in the progress bar. I can't find any way to get more specific driver than by the series 6xxx. What am I missing?


Have you tried manually installing the driver? Once downloaded, you only extract it somewhere on your disk but do not use wizard. Then from Windows Device Manager, you right clic on the graphic card (probably with the yellow warning sign), and install driver from here by pointing to the path were the drivers has been extracted.

I used to do it this way with my old setup (Xen + AMD cards), in order to avoid BSOD.

Offline

#5201 2015-05-26 22:38:05

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
ghormoon wrote:

I don't get where you maen to do that. on the web site, I do manually select (notebook gpu > HD series > 6xxx> win 7 x64) and it gives me 288MB catalyst. that one only gives me default/custom selection and field for path. on next (either default or custom), it goes through few steps before allowing you to select things (at least on custom, default won't let you) and it crashes before that selection just after saying "detecting graphics hardware" in the progress bar. I can't find any way to get more specific driver than by the series 6xxx. What am I missing?


Have you tried manually installing the driver? Once downloaded, you only extract it somewhere on your disk but do not use wizard. Then from Windows Device Manager, you right clic on the graphic card (probably with the yellow warning sign), and install driver from here by pointing to the path were the drivers has been extracted.

I used to do it this way with my old setup (Xen + AMD cards), in order to avoid BSOD.

I doubt that this will work, since AMD Catalyst has got oh so complex in last two years or so.

...

Whoa, what a colourful glitches i have with hugepages enabled. Just awesome. It's covered in stripes. And i haven't even specified the VM to use hugepages in the xml...
Well, seems like my issue with every-15-seconds-freeze isn't related to the VM, but is caused by the game.

Last edited by Duelist (2015-05-26 23:26:55)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5202 2015-05-26 23:52:46

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems that sometimes when I shut down the VM the QEMU hangs while doing, and this appears in dmesg when force killing qemu

[ 8032.137126] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 [ 8032.144976] IP: [<ffffffff8141ff71>] vfio_device_get+0x11/0x50 [ 8032.150251] pci-stub 0000:02:00.0: claimed by stub [ 8032.150354] vfio-pci 0000:02:00.1: Relaying device request to user (#0) [ 8032.162192] PGD 0 [ 8032.164217] Oops: 0000 [#1] PREEMPT SMP [ 8032.168190] Modules linked in: udf crc_itu_t arc4 ecb md4 md5 hmac nls_utf8 cifs dns_resolver vhost_net vhost macvtap macvlan tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc cfg80211 rfkill gspca_zc3xx gspca_main videodev media snd_hda_codec_hdmi mousedev hid_logitech_hidpp hid_logitech_dj uas usb_storage snd_hda_codec_realtek snd_hda_codec_generic coretemp hwmon intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm fuse nvidia(O) nls_iso8859_1 crct10dif_pclmul crc32_pclmul nls_cp437 crc32c_intel vfat ghash_clmulni_intel iTCO_wdt fat iTCO_vendor_support mxm_wmi aesni_intel snd_usb_audio aes_x86_64 lrw gf128mul snd_usbmidi_lib snd_hda_intel snd_hda_controller snd_hda_codec snd_rawmidi e1000e snd_hda_core snd_seq_device r8169 [ 8032.239642] snd_hwdep snd_pcm evdev mii i2c_i801 mac_hid snd_timer ptp glue_helper mei_me ablk_helper drm snd cryptd sb_edac pcspkr serio_raw pps_core tpm_tis mei i2c_core soundcore psmouse lpc_ich edac_core shpchp tpm wmi processor button sch_fq_codel nfs lockd grace sunrpc fscache ext4 crc16 mbcache jbd2 hid_generic usbhid hid sr_mod cdrom sd_mod atkbd libps2 ahci libahci xhci_pci libata xhci_hcd scsi_mod usbcore usb_common i8042 serio [ 8032.278086] CPU: 3 PID: 3125 Comm: qemu-system-x86 Tainted: G O 4.1.0-1-ARCH #1 [ 8032.286506] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99 Extreme6, BIOS P1.90 04/17/2015 [ 8032.296303] task: ffff88048cf0a8c0 ti: ffff880447034000 task.ti: ffff880447034000 [ 8032.303764] RIP: 0010:[<ffffffff8141ff71>] [<ffffffff8141ff71>] vfio_device_get+0x11/0x50 [ 8032.312023] RSP: 0018:ffff880447037ae8 EFLAGS: 00010296 [ 8032.317320] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 8032.324434] RDX: ffff88047cc71c40 RSI: ffff880447037bc0 RDI: 0000000000000000 [ 8032.331549] RBP: ffff880447037af8 R08: ffff88047cc71c40 R09: ffff88047cc71c40 [ 8032.338666] R10: ffffffff810d443a R11: 0000000000003029 R12: ffff88047cc71c40 [ 8032.345780] R13: ffffffff81423090 R14: ffff88048c032c00 R15: ffff88048c032c00 [ 8032.352903] FS: 00007f4bd6a70700(0000) GS:ffff88048f2c0000(0000) knlGS:0000000000000000 [ 8032.360971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8032.366701] CR2: 0000000000000018 CR3: 000000000180b000 CR4: 00000000001407e0 [ 8032.373815] Stack: [ 8032.375824] ffff88048c065002 0000000000000000 ffff880447037b18 ffffffff8141ffcd [ 8032.383267] ffff880447037b38 ffff880447037bc0 ffff880447037b38 ffffffff8142313d [ 8032.390709] ffff880447037bd0 ffff880447037bd0 ffff880447037b58 ffffffff814230ef [ 8032.398155] Call Trace: [ 8032.400597] [<ffffffff8141ffcd>] vfio_device_get_from_dev+0x1d/0x30 [ 8032.406932] [<ffffffff8142313d>] vfio_pci_get_devs+0x3d/0x70 [ 8032.412662] [<ffffffff814230ef>] vfio_pci_walk_wrapper+0x5f/0x70 [ 8032.418742] [<ffffffff81304129>] pci_walk_bus+0x79/0xa0 [ 8032.424055] [<ffffffff81424a30>] vfio_pci_release+0x290/0x410 [ 8032.429878] [<ffffffff81423100>] ? vfio_pci_walk_wrapper+0x70/0x70 [ 8032.436127] [<ffffffff8141f9f0>] vfio_device_fops_release+0x20/0x40 [ 8032.442466] [<ffffffff811dfb6c>] __fput+0x9c/0x200 [ 8032.447337] [<ffffffff811dfd1e>] ____fput+0xe/0x10 [ 8032.452204] [<ffffffff81095bf4>] task_work_run+0xd4/0xf0 [ 8032.457597] [<ffffffff8107b593>] do_exit+0x3b3/0xba0 [ 8032.462641] [<ffffffff8107be0b>] do_group_exit+0x3b/0xb0 [ 8032.468024] [<ffffffff810877a6>] get_signal+0x246/0x670 [ 8032.473323] [<ffffffff810e6aa3>] ? hrtimer_try_to_cancel+0x93/0x120 [ 8032.479668] [<ffffffff81015577>] do_signal+0x37/0x770 [ 8032.484799] [<ffffffff810e7a54>] ? hrtimer_nanosleep+0xc4/0x1d0 [ 8032.490791] [<ffffffff810fb751>] ? SyS_futex+0x81/0x190 [ 8032.496102] [<ffffffffa061c4f7>] ? kvm_on_user_return+0x77/0x90 [kvm] [ 8032.502612] [<ffffffff81015d10>] do_notify_resume+0x60/0x80 [ 8032.508258] [<ffffffff8158e0bc>] int_signal+0x12/0x17 [ 8032.513389] Code: ed eb d8 0f 1f 80 00 00 00 00 48 89 df e8 38 fe ff ff 48 89 d8 eb cb 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 <48> 8b 7f 18 e8 16 fe ff ff b8 01 00 00 00 f0 0f c1 03 83 c0 01 [ 8032.533315] RIP [<ffffffff8141ff71>] vfio_device_get+0x11/0x50 [ 8032.539233] RSP <ffff880447037ae8> [ 8032.542713] CR2: 0000000000000018 [ 8032.546021] ---[ end trace 276b0f1585740db3 ]--- [ 8032.550625] Fixing recursive fault but reboot is needed!

Offline

#5203 2015-05-27 03:42:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Seems that sometimes when I shut down the VM the QEMU hangs while doing, and this appears in dmesg when force killing qemu

Could you please test this?  http://paste.fedoraproject.org/225883/98048143/


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5204 2015-05-27 12:22:30

jack_boss
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, maybe it's a bit late to jump on the bandwagon, but if I read the first post correctly a attempt with the intel IGP is legit if using OVMF.

In any case, I ran the OVMF test and recieved a black qemu window.
It took a bit of try and error (adding kernel paraments and changes in /etc/mkinitcpio.conf, due to my lack of seeing the obvious or due to my hardware. I'd like to share what I have...

My issue before I go any deeper with the guide is that don't get a /etc/vfio-pci.cfg file.

/boot/loader/entries/arch.conf

options root=/dev/sdb2 rw intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a vfio_iommu_type1.allow_unsafe_interrupts=1

/etc/mkinitcpio.conf

MODULES="kvm kvm_intel pci_stub"

/etc/systemd/system/vfio-bind.service
(Same as on initial post)

I ran systemctl enable vfio-bind.service.

Thank you.

Offline

#5205 2015-05-27 12:33:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Punkbob wrote:
aw wrote:

Running 352.86 here, no Code 43 problems.  The NVIDIA Control Panel stopped working, but I see others reporting that in non-VM applications.  It worked on 350.12.  Code 43 is a pretty generic "something is wrong" error, hypervisor detection isn't the only reason for it to get thrown.  I recently discovered that failure to plugin aux power to cards also generates a Code 43.

Hmm, I will look into it more then. Btw, are you running Windows 10 or 8.1, cause it might be an issue on my end as I am running my VMs with Windows 10.

8.1, AIUI 10 has numerous issues running as a VM

@Punkbob

What build of Windows 10 are you using and did you need to do anything special to get it installed?  I see one report that installing build 10041 requires at least one hyper-v extension to be enabled, which isn't going to sit well with Nvidia drivers.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5206 2015-05-27 12:41:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jack_boss wrote:

Hello, maybe it's a bit late to jump on the bandwagon, but if I read the first post correctly a attempt with the intel IGP is legit if using OVMF.

In any case, I ran the OVMF test and recieved a black qemu window.
It took a bit of try and error (adding kernel paraments and changes in /etc/mkinitcpio.conf, due to my lack of seeing the obvious or due to my hardware. I'd like to share what I have...

My issue before I go any deeper with the guide is that don't get a /etc/vfio-pci.cfg file.

Please try the howto guide in my sig, libvirt has come a long way and vfio-bind scripts should no longer be necessary.  intel_iommu=on and a pci-stub.ids= should be sufficient to simply let libvirt handle the rest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5207 2015-05-27 13:32:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jack_boss wrote:

My issue before I go any deeper with the guide is that don't get a /etc/vfio-pci.cfg file.

Ha! Well... You just create it manually, where you put something like DEVICES="device-ids,comma-separated". But that's not the recommended way of doing vfio bind anymore, it's much simpler to use libvirt.

jack_boss wrote:

/boot/loader/entries/arch.conf

options root=/dev/sdb2 rw intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a vfio_iommu_type1.allow_unsafe_interrupts=1

You must NOT enable unsafe interrupts if you're not being told to do so in the dmesg. Read aw's blog and his VFIO FAQ.

Also, he posted a 5-part story of doing the GPU passthrough the right way, containing the fresh instructions and thoughts. Very worth reading, especially knowing the fact that he actually wrote all that vfio stuff.

EDIT: huh, that aw's message wasn't there when i wrote this. Seems like my F5 key isn't working well anymore..

Last edited by Duelist (2015-05-27 13:33:49)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5208 2015-05-27 14:01:15

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I have a running Win7 (non OVMF) machine and a running Win 10 (ovmf) machine. Host is Arch Linux, with following kernel from AUR : 4.0.1-1-vfio.

I am using onboard intel as host graphics, and a GTX 970 as gaming card. Lately, I had issues with Kodi on host, and I had to switch "acceleration" from UXA to SNA.

In the end, I now need to reinstall my Win7 machine with OVMF, so that I can have DRI enabled on host. However, the Windows 7 installation hangs at the "Starting Windows" screen. Looks like some kind of "hardware" issue.

Here is my "install script" that worked with Win 10, but doesn't work with win 7 (I only replaced the "win disk" line, and the "iso file" line) :

qemu-system-x86_64 -name windaube -enable-kvm -m 16384 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \ -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:1d.0 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/guest/windaube,id=win,format=raw,if=none -device scsi-hd,drive=win \ -drive file=/home/nesousx/Isos/virtio-win-0.1-81.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -drive file=/home/nesousx/Isos/Win_7_Pro_x64.iso,id=virtiocd,if=none -device scsi-cd,drive=virtiocd \ -net nic,model=virtio,macaddr=52:54:00:7c:3c:42 -net bridge,br=br0 \ -nographic

I have tried to run it :

* with a small amount of RAM ;
* without assigning devices.

Nothing worked.

Do someone have any idea why it is not working?

Thanks in advance.

Last edited by Nesousx (2015-05-27 16:38:22)

Offline

#5209 2015-05-27 14:28:44

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried the instructions on post 4470 on page 179?  You need to use the qxl driver for the initial output...

Last edited by mostlyharmless (2015-05-27 14:29:23)

Offline

#5210 2015-05-27 14:38:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's generally a good idea to get the VM installed first, then add an assigned GPU.  I've also heard to use either qxl or vga devices for the install.  Also, AIUI Win7+OVMF+SMP is incompatible with hyper-v extensions, so you'll need to disable them even if you're not using Nvidia.

EDIT: I believe virt-manager has support for this upstream, but it was just fixed last week.  So, if you're pulling from the devel tree, hyper-v should automatically be disabled once you select OVMF and have specified a Windows 7 guest type.

Last edited by aw (2015-05-27 14:48:19)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5211 2015-05-27 15:15:27

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was able to make a Radeon HD7770 work in Win 7 X64 SP1 by first booting up with QXL graphics and then doing "device_add vfio-pci,host=01:00.0" and "device_add vfio-pci,host=01:00.1" from the Qemu Console while the system was already up and running. Device manager was showing the usual "Code 10" at this point.

The act of "hot-plugging" the GPU however, made it possible to complete the installation of Catalyst 14.12 without immediately crashing the installer (like what usually happens).

After Catalyst was installed, I was able to add the -device parameters to my qemu command-line and the guest boots up using the QXL for VGA and switches output to the vfio'd GPU as soon as desktop is ready.

This way I don't have to use 'x-vga' or any sort of additional patches for the kernel v4.0.0. Also I'm able to use the integrated Intel HD4600 graphics for my host without having to disable DRI.

Rebooting/Suspending/Hibernation all seem to work just fine with this too.

I'm also passing trough the integrated Intel HD Audio and a USB 3.0 PCI-e card. They also work without any issues. I can even do low-latency ASIO stuff with the real sound card, which is really cool.

By using virtio networking and tightvnc mirror driver with raw encoding, I get so good VNC performance between the host and guest, that I don't miss SPICE. Even 3D works smoothly over VNC now tongue And of course there is Steam in-home streaming for a little better latency for realtime 3D gaming, or simply switching the inputs from the monitor.

Offline

#5212 2015-05-27 15:58:43

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
Punkbob wrote:

Hmm, I will look into it more then. Btw, are you running Windows 10 or 8.1, cause it might be an issue on my end as I am running my VMs with Windows 10.

8.1, AIUI 10 has numerous issues running as a VM

@Punkbob

What build of Windows 10 are you using and did you need to do anything special to get it installed?  I see one report that installing build 10041 requires at least one hyper-v extension to be enabled, which isn't going to sit well with Nvidia drivers.

I am using build 9926 on my most recent VM, and I didn't change anything or enable anything different then I would with a normal Windows 10 install. When I first switched over to Windows 10 a few months ago it initially didn't have any specific display drivers from Nvidia, so I just used the ones from Windows 8.1. However once I try drivers that are specifically released for Windows 10, the screen goes black, and when I reboot it just spits the code 43 error back at me.

Edit: Let me try a later build real quick, I thought I was using one that was more recent, but I think I confused which iso I grabbed off my NAS.

Last edited by Punkbob (2015-05-27 16:03:06)

Offline

#5213 2015-05-27 15:59:53

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

Have you tried the instructions on post 4470 on page 179?  You need to use the qxl driver for the initial output...

Thanks a lot, it seems to be working perfectly.

Last edited by Nesousx (2015-05-27 17:36:44)

Offline

#5214 2015-05-27 16:35:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Cubex wrote:

Seems that sometimes when I shut down the VM the QEMU hangs while doing, and this appears in dmesg when force killing qemu

Could you please test this?  http://paste.fedoraproject.org/225883/98048143/

Here's a bit more complete patch posted upstream, I'd appreciate testing in your scenario.  Thanks.  https://lkml.org/lkml/2015/5/27/546


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5215 2015-05-27 16:39:12

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just as an update, as I haven't gotten to the point where I can try the Nvidia drivers yet, build 10074 of Windows 10 needs you to set your cpu as being a core2duo in order to boot the install environment.

Offline

#5216 2015-05-27 16:55:53

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Win 10 needs this to boot...

<feature policy='force' name='lahf_lm'/>
<feature policy='force' name='cx16'/>

Offline

#5217 2015-05-27 17:15:37

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Win 10 needs this to boot...

<feature policy='force' name='lahf_lm'/>
<feature policy='force' name='cx16'/>


Tried this, didn't work. Unless I am doing it wrong:

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <feature policy='force' name='lahf_lm'/>
    <feature policy='force' name='cx16'/>
  </cpu>

Offline

#5218 2015-05-27 17:19:27

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ah, for me works with kvm64

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>kvm64</model>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='force' name='lahf_lm'/>
    <feature policy='force' name='cx16'/>
  </cpu>

edit: problem with this is that you can't enable/disable shadow play in geforce experience for better streaming...

Last edited by slis (2015-05-27 17:24:53)

Offline

#5219 2015-05-27 17:22:35

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

ah, for me works with kvm64

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>kvm64</model>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='force' name='lahf_lm'/>
    <feature policy='force' name='cx16'/>
  </cpu>

edit: problem with this is that you can't enable/disable shadow play in geforce exeprience for better streaming...


Confirmed that this works, let me see what happens with latest Nvidia drivers.

Edit: I've seen mention that the Q35 version supposedly works out of the box, but I can't get it to see my drives in ovmf, and haven't really looked into it besides a quick test.

Edit2: This works, and the latest version of Nvidia drivers are running fine.

So the current state of Windows 10 in kvm is this:

If you want to run it as your CPU model you need to run the old 9926 build and it will only run with Nvidia Win8.1 drivers up to 347.88.

if you want the latest drivers or even just to run the later builds at all in kvm, you either need to set your CPU model to Core2duo, or use kvm64 with the additional options above.

Last edited by Punkbob (2015-05-27 18:36:58)

Offline

#5220 2015-05-27 17:36:29

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

It's generally a good idea to get the VM installed first, then add an assigned GPU.  I've also heard to use either qxl or vga devices for the install.  Also, AIUI Win7+OVMF+SMP is incompatible with hyper-v extensions, so you'll need to disable them even if you're not using Nvidia.

EDIT: I believe virt-manager has support for this upstream, but it was just fixed last week.  So, if you're pulling from the devel tree, hyper-v should automatically be disabled once you select OVMF and have specified a Windows 7 guest type.

Sounds interesting ! I'd like to give a try via virt-manager, however I can't select OVMF/UEFI as bios. Virt-manager / libvirt doesn't detect it. It is greyed out, and says that the firmware image is not installed.

I have qemu 2.3.0-2, virt-manager 1.2.0-2, libvirt 1.2.15-1 and ovmf 16229-1.

Any idea what would be the cause please? Also, what does AUI stands for, please?

Last edited by Nesousx (2015-05-27 17:39:16)

Offline

#5221 2015-05-27 17:45:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:
aw wrote:

It's generally a good idea to get the VM installed first, then add an assigned GPU.  I've also heard to use either qxl or vga devices for the install.  Also, AIUI Win7+OVMF+SMP is incompatible with hyper-v extensions, so you'll need to disable them even if you're not using Nvidia.

EDIT: I believe virt-manager has support for this upstream, but it was just fixed last week.  So, if you're pulling from the devel tree, hyper-v should automatically be disabled once you select OVMF and have specified a Windows 7 guest type.

Sounds interesting ! I'd like to give a try via virt-manager, however I can't select OVMF/UEFI as bios. Virt-manager / libvirt doesn't detect it. It is greyed out, and says that the firmware image is not installed.

I have virt-manager version 1.2.0-2, libvirt 1.2.15-1 and ovmf 16229-1.

Any idea what would be the cause please? Also, what does AUI stands for, please?

Maybe your ovmf package isn't installing the files where libvirt/virt-manager looks for them?  On my Fedora 21 system the binaries are installed in /usr/share/edk2.git/ovmf-x64/  Your virt-manager would need to be pulled form the devel tree since last Wednesday to include the fix.  AIUI = As I Understand It.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5222 2015-05-27 18:09:37

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Nesousx wrote:
aw wrote:

It's generally a good idea to get the VM installed first, then add an assigned GPU.  I've also heard to use either qxl or vga devices for the install.  Also, AIUI Win7+OVMF+SMP is incompatible with hyper-v extensions, so you'll need to disable them even if you're not using Nvidia.

EDIT: I believe virt-manager has support for this upstream, but it was just fixed last week.  So, if you're pulling from the devel tree, hyper-v should automatically be disabled once you select OVMF and have specified a Windows 7 guest type.

Sounds interesting ! I'd like to give a try via virt-manager, however I can't select OVMF/UEFI as bios. Virt-manager / libvirt doesn't detect it. It is greyed out, and says that the firmware image is not installed.

I have virt-manager version 1.2.0-2, libvirt 1.2.15-1 and ovmf 16229-1.

Any idea what would be the cause please? Also, what does AIUI stands for, please?

Maybe your ovmf package isn't installing the files where libvirt/virt-manager looks for them?  On my Fedora 21 system the binaries are installed in /usr/share/edk2.git/ovmf-x64/  Your virt-manager would need to be pulled form the devel tree since last Wednesday to include the fix.  AIUI = As I Understand It.

Ok, thanks, I'll look into it.

Everytime I saw you saying AIUI, it was next to Windows. Hence, I believed it was some kind of "special" version of Windows since I never heard that acronym. Thanks for teaching me. smile

Offline

#5223 2015-05-27 19:01:08

jack_boss
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Nesousx wrote:
aw wrote:

It's generally a good idea to get the VM installed first, then add an assigned GPU.  I've also heard to use either qxl or vga devices for the install.  Also, AIUI Win7+OVMF+SMP is incompatible with hyper-v extensions, so you'll need to disable them even if you're not using Nvidia.

EDIT: I believe virt-manager has support for this upstream, but it was just fixed last week.  So, if you're pulling from the devel tree, hyper-v should automatically be disabled once you select OVMF and have specified a Windows 7 guest type.

Sounds interesting ! I'd like to give a try via virt-manager, however I can't select OVMF/UEFI as bios. Virt-manager / libvirt doesn't detect it. It is greyed out, and says that the firmware image is not installed.

I have virt-manager version 1.2.0-2, libvirt 1.2.15-1 and ovmf 16229-1.

Any idea what would be the cause please? Also, what does AUI stands for, please?

Maybe your ovmf package isn't installing the files where libvirt/virt-manager looks for them?  On my Fedora 21 system the binaries are installed in /usr/share/edk2.git/ovmf-x64/  Your virt-manager would need to be pulled form the devel tree since last Wednesday to include the fix.  AIUI = As I Understand It.

I'm stuck at this step too, since I started following your guide.
I found out that ovmf from the arch official repository installs its files to

/usr/share/ovmf/ovmf_x64.bin

vs ovmf-svn (installed today)

/usr/share/ovmf/64/ovmf_x64.bin

I made softlinks to point to the "correct" place. Didn't work.

Last edited by jack_boss (2015-05-27 19:07:46)

Offline

#5224 2015-05-27 19:06:13

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jack_boss wrote:
aw wrote:
Nesousx wrote:

Sounds interesting ! I'd like to give a try via virt-manager, however I can't select OVMF/UEFI as bios. Virt-manager / libvirt doesn't detect it. It is greyed out, and says that the firmware image is not installed.

I have virt-manager version 1.2.0-2, libvirt 1.2.15-1 and ovmf 16229-1.

Any idea what would be the cause please? Also, what does AUI stands for, please?

Maybe your ovmf package isn't installing the files where libvirt/virt-manager looks for them?  On my Fedora 21 system the binaries are installed in /usr/share/edk2.git/ovmf-x64/  Your virt-manager would need to be pulled form the devel tree since last Wednesday to include the fix.  AIUI = As I Understand It.

I'm stuck at this step too, since I started following your guide.
I found out that ovmf from the arch official repository installs its files to

/usr/share/ovmf/ovmf_x64.bin

vs ovmf-svn (installed today)

/usr/share/ovmf/64/ovmf_x64.bin

I made softlinks to point to the "correct" place. Didn't work.


You need to follow the instructions here:

https://wiki.archlinux.org/index.php/PC … stallation

The packages on aur won't work.

Edit: And to make it appear in virtual manager add this to qemu.conf:

nvram = [
    "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd",
    "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw",
]

Now this is where I am unclear, I only included the second aarch64 line because libvirt fails to start if it isn't in there, but there isn't anything at that location. I guess qemu.conf needs it defined even if you don't use it?

Last edited by Punkbob (2015-05-27 19:15:34)

Offline

#5225 2015-05-27 19:21:23

jack_boss
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Punkbob wrote:
jack_boss wrote:
aw wrote:

Maybe your ovmf package isn't installing the files where libvirt/virt-manager looks for them?  On my Fedora 21 system the binaries are installed in /usr/share/edk2.git/ovmf-x64/  Your virt-manager would need to be pulled form the devel tree since last Wednesday to include the fix.  AIUI = As I Understand It.

I'm stuck at this step too, since I started following your guide.
I found out that ovmf from the arch official repository installs its files to

/usr/share/ovmf/ovmf_x64.bin

vs ovmf-svn (installed today)

/usr/share/ovmf/64/ovmf_x64.bin

I made softlinks to point to the "correct" place. Didn't work.


You need to follow the instructions here:

https://wiki.archlinux.org/index.php/PC … stallation

The packages on aur won't work.

Edit: And to make it appear in virtual manager add this to qemu.conf:

nvram = [
    "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd",
    "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw",
]

Now this is where I am unclear, I only included the second aarch64 line because libvirt fails to start if it isn't in there, but there isn't anything at that location. I guess qemu.conf needs it defined even if you don't use it?

How did you figure this part out?

 nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd", "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw", ]

I assumed it would appear in virt-manager without that part, I mean ovm-svn would probably work to if i were to edit qemu.conf.

Offline

#5226 2015-05-27 19:22:26

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Punkbob wrote:
jack_boss wrote:
aw wrote:

Maybe your ovmf package isn't installing the files where libvirt/virt-manager looks for them?  On my Fedora 21 system the binaries are installed in /usr/share/edk2.git/ovmf-x64/  Your virt-manager would need to be pulled form the devel tree since last Wednesday to include the fix.  AIUI = As I Understand It.

I'm stuck at this step too, since I started following your guide.
I found out that ovmf from the arch official repository installs its files to

/usr/share/ovmf/ovmf_x64.bin

vs ovmf-svn (installed today)

/usr/share/ovmf/64/ovmf_x64.bin

I made softlinks to point to the "correct" place. Didn't work.


You need to follow the instructions here:

https://wiki.archlinux.org/index.php/PC … stallation

The packages on aur won't work.

Edit: And to make it appear in virtual manager add this to qemu.conf:

nvram = [
    "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd",
    "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw",
]

Now this is where I am unclear, I only included the second aarch64 line because libvirt fails to start if it isn't in there, but there isn't anything at that location. I guess qemu.conf needs it defined even if you don't use it?

This is the right answer, in general. In more detail:

* The AUR builds are probably not okay -- I assume they don't package the separate (split) OVMF_CODE.fd (firmware binary) and OVMF_VARS.fd (varstore template) files. Gerd's packages do.

* You should edit the nvram stanza in /etc/libvirt/qemu.conf as described above, and restart libvirtd. This is documented in the OVMF whitepaper, section  "Installation of OVMF guests with virt-manager and virt-install".

* Unless you give me the exact error message from libvirt, I won't know why it rejected the single-entry nvram stanza. (Refer to /var/log/messages or systemctl status libvirtd.) My guess is that you may have left a stray comma at the end of that sole entry. (OTOH, as Alex points out, your example now has a stray comma after the second entry, and apparently libvirt doesn't complain... So please provide the exact error message.)

Offline

#5227 2015-05-27 19:31:50

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so it looks like my issue was that I was commenting out the line in the middle of the configuration block with a #. If I remove the line completely libvirtd restarts fine. My fault for being lazy.

This is what systemctl spits back at me with the # in the config block.

May 27 15:27:09 Desktop libvirtd[2264]: libvirt version: 1.2.15
May 27 15:27:09 Desktop libvirtd[2264]: internal error: Failed to find path for dmidecode binary
May 27 15:27:09 Desktop libvirtd[2264]: configuration file syntax error: /etc/libvirt/qemu.conf:516: expecting a value
May 27 15:27:09 Desktop libvirtd[2264]: Initialization of QEMU state driver failed: configuration file syntax error: /etc/libvirt/qemu.conf:516: expecting a value
May 27 15:27:09 Desktop libvirtd[2264]: Driver state initialization failed

Edit: And yeah the aur builds of ovmf just create one binary, so using gerd's package is the correct option to get it working with virt-manager.

Last edited by Punkbob (2015-05-27 19:33:26)

Offline

#5228 2015-05-27 20:15:05

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@lersek thanks a lot this is actually working :

1. Getting ovmf from Gerf Hoffman's repo ;
2. Editing qemu.conf.

I now can select OVMF from virt-manager GUI. Not sure about hyper-V extensions though. I will probably have to turn them off manually. I'll keep you posted.

Offline

#5229 2015-05-28 01:46:46

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

update on the laptop experiment:
when I've tried to get more info on the BSOD, it just installed the driver fine. I've not changed anything since the BSOD, no idea why it didn't crash now then ...
though I'm still in state I was in previous install (the one where I installed drivers when cirrus was primary) - it boots, shows logo ans "starting windows", then screen goes black and while after that I see the sound it does when finished booting and asks for password.
I don't see anything interesting in event log, nothing in dmesg at that time.
interesting thing is that windows lists the card as 7400M series (after installing amd driver). no idea if that's correct or not, I've not installed windows on this machine in some time, so I don't have where to check, if that's correct. (6400 and 7400 are quite simmilar, linux lists it as 6400m/7400m)

the problem with not shutting down is still present, any ideas on that?

Offline

#5230 2015-05-28 06:16:58

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to use cpu-passthrough mode with that 2 extra flags to make windows 10 boot/work with "real" cpu id?

Offline

#5231 2015-05-28 16:56:22

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Is it possible to use cpu-passthrough mode with that 2 extra flags to make windows 10 boot/work with "real" cpu id?

It's a problem with kvm *itself* so it makes more sense to report a bug with upstream.

Edit: I am debugging it now, the error I see spit back at me in the log files is:

kvm [5953]: vcpu0 unhandled wrmsr: 0x38d data 0

Which when I search there are references on the kvm mailing list to processor functionality that the OS is trying to address, and kvm not having the extra code to properly use those features.

I am going to build a test box and turn serial debugging on inside windows, and go from there to see what it complains about.

Last edited by Punkbob (2015-05-28 17:09:27)

Offline

#5232 2015-05-28 17:09:50

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

AIUI Win7+OVMF+SMP is incompatible with hyper-v extensions, so you'll need to disable them even if you're not using Nvidia.

Errr... I've had this working some time ago, but... now the manually-added hyper-v enlightenments are gone from my domain xml... I should try adding them manually. Oh well. Yeah, not it hangs up on boot logo.

ghormoon wrote:

though I'm still in state I was in previous install (the one where I installed drivers when cirrus was primary) - it boots, shows logo ans "starting windows", then screen goes black and while after that I see the sound it does when finished booting and asks for password.

Can you plug a screen into some of that GPU's video outputs? Like, an HDMI port or VGA or whatever you have there? Chances are the GPU doesn't find where to output and starts working as a 3D accelerator. Maybe you'll need to research your laptop deeper.

Punkbob wrote:

I am going to build a test box and turn serial debugging on inside windows, and go from there to see what it complains about.

Am i misunderstanding something, or wouldn't you achieve the same using qemu's built-in GDB server?

Last edited by Duelist (2015-05-28 18:57:31)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5233 2015-05-28 19:59:21

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist

From what I understand if I want to see what is going on inside windows I need to enable serial debugging and then connect to that from another windows machine as windows uses a special format. I also need to look at kvm to see what calls are being made to it, but I was starting from the inside out.

If I am wrong please feel free to correct me, as it'd save me some time configuring windbg.

Offline

#5234 2015-05-29 02:17:14

Punkbob
Member
Registered: 2014-10-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So completely useless, but if you enable debug mode in Windows it will boot on haswell just fine.

It appears that it's an issue with setting a value on IA32_FIXED_CTR_CTRL which relates to CPU performance monitoring on processors Nethalhem or later.

A VMware article mentions things that you have to do to enable it in esx, might be something that kvm needs to address.
http://kb.vmware.com/selfservice/micros … Id=2030221

Edit: Or I could be barking up the completely wrong tree, still need to debug the windows kernel.

Edit2: Solved it or at least got it to boot?

So I did 2 things, first I set the CPU to "host-passthrough" and then ran "echo 1 > /sys/module/kvm/parameters/ignore_msrs"

It basically tells kvm to not send a fault when it can't access or set a msr value on the host CPU. The host-passthrough part is most likely not required, but something I tried that at least gave me a different msr error(vcpu0 ignored rdmsr: 0x639) and should also enable you to use Nvidia game-streaming as it will report your exact CPU.

Last edited by Punkbob (2015-05-29 03:35:43)

Offline

#5235 2015-05-29 03:15:44

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nice, I will try it later today after work.

Offline

#5236 2015-05-29 03:19:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Punkbob wrote:

Edit2: Solved it or at least got it to boot?

So I did 2 things, first I set the CPU to "host-passthrough" and then ran "echo 1 > /sys/module/kvm/parameters/ignore_msrs"

It basically tells kvm to not send a fault when it can't access or set a msr value on the host CPU. The host-passthrough part is most likely not required, but something I tried that at least gave me a different msr error(vcpu0 ignored rdmsr: 0x639) and should enable you to use Nvidia game-streaming.

This is also necessary to make the Passmark Performance Test work, so I've typically got it enabled already FWIW


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5237 2015-05-29 04:40:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also, when using CPU-Z software, dmesg screams with unhandled rdmsr errors, i guess cpu-z tries to determine the frequency and a multiplier by accessing those MSRs, and since we have our CPUs virtual, it fails too.
I don't think that this detail is very worth noting, but let it be.

BTW:
What do you guys recommend for storage subsystem?
I've added a small, 8GB raw file image located on old OCZ SSD to my VM, and connected it via virtio-blk-pci.

I've tried using iothreads(which are related to x-data-plane, i guess) with virtio-blk-pci but i have huge host CPU usage in benchmarks and small freezes in some weird games when loading heavy parts of the scene(the game is poorly written, but it's almost 8 years old). ATTO disk benchmark also reports a decrease in linear write performance starting somewhere at 256k sector size.

That compares poorly with virtio-scsi with four queues enabled, it provides me a significant increase in linear write performance, but the freezes aren't gone and there's a huge host CPU usage while benchmarking(well, it's pushing 200-300MB/s).

I remember aw said that he has x-data-plane enabled, but... it's either i've missed something in libvirt's domain xml or virtio-scsi is better.
And also, seems like cache=none performs better than cache=directsync when not using iothreads - redhat's optimization guide doesn't mention the second caching method or iothreads, maybe it's already outdated?..

But i definitely should determine a good method of measuring disk performance, as ATTO disk benchmark doesn't count IOPS.

Last edited by Duelist (2015-05-29 05:07:46)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5238 2015-05-29 12:48:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I remember aw said that he has x-data-plane enabled

Nope, I use virtio-scsi w/ multiqueue


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5239 2015-05-31 02:17:43

ceri
Member
Registered: 2013-10-12
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it worth getting the i7-4790K over the i5-4690K for the extra vCPUs (hyperthreading)? What kind of benefits would I see?

My use case is simultaneous gaming on the VM, with moderate multitasking on the host (say 30-50 firefox tabs, etc).

Offline

#5240 2015-05-31 03:59:21

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Now kernel doesn't panic or throw any error, but QEMU (stable) still hangs at shutting down, I will try the git version tomorrow

EDIT: Noticed that there is another version for upstream, I will test that one, sorry for confusion

Last edited by Cubex (2015-06-01 01:24:48)

Offline

#5241 2015-05-31 21:56:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ceri wrote:

Is it worth getting the i7-4790K over the i5-4690K for the extra vCPUs (hyperthreading)? What kind of benefits would I see?

My use case is simultaneous gaming on the VM, with moderate multitasking on the host (say 30-50 firefox tabs, etc).

30-50 firefox tabs

LOL!
FQYhiJS.png
And that's only the QEMU group. As you can see, firefox is more memory-hungry.

And now seriously:
I have 4 cores in my cheap Athlon X4 750K, and i'm assigning all the cores to the guest. Yeah, when either host or guest is loaded - the other system will stutter, lag and suffer from CPU starvation. But i'm very rarely that "productive".

You'll want additional cores to use cpu pinning.
So, for example, i could've cut three cores from the host and dedicate(pin) them to the VM. That would give the guest a very good CPU performance(almost native/baremetal) and what is more important - reduce the latencies and that would theoretically fix the audio problems. But i don't like to leave my host with one core only, because... i feel sad for it.

So basically, more cores - more performance, obviously, and if getting an i7 isn't much a financial burden for you - it would be great to use it.

BUT! Beware of VT-d, as some(4770K for example) K-series CPUs have disabled VT-d. That feature is crucial for the passthrough to work.

Also, you could get a Xeon E3-series, like aw does(read the first part of the guide on his blog), or, better yet - take a Xeon E5. That'd be an ultimate choice, but ultimately pricey.

Last edited by Duelist (2015-05-31 21:57:32)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5242 2015-05-31 23:40:36

DelusionalLogic
Member
Registered: 2013-05-07
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

-Snip-
I have 4 cores in my cheap Athlon X4 750K, and i'm assigning all the cores to the guest. Yeah, when either host or guest is loaded - the other system will stutter, lag and suffer from CPU starvation. But i'm very rarely that "productive".

You'll want additional cores to use cpu pinning.
So, for example, i could've cut three cores from the host and dedicate(pin) them to the VM. That would give the guest a very good CPU performance(almost native/baremetal) and what is more important - reduce the latencies and that would theoretically fix the audio problems. But i don't like to leave my host with one core only, because... i feel sad for it.

So basically, more cores - more performance, obviously, and if getting an i7 isn't much a financial burden for you - it would be great to use it.
-Snip-

How would i go about dedicating some cores to the VM? And could it be done dynamically? (while the system and VM is running). I already have the VCPU cores pinned, but the cores are still being used on the host.

I'm having some problems, in the beginning it was with USB sound cards (it made the whole VM stutter when it was running). Now i'm using a virtual soundcard and the whole system is just slightly slow.

Offline

#5243 2015-06-01 01:36:52

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

How would i go about dedicating some cores to the VM? And could it be done dynamically? (while the system and VM is running). I already have the VCPU cores pinned, but the cores are still being used on the host.

I'm having some problems, in the beginning it was with USB sound cards (it made the whole VM stutter when it was running). Now i'm using a virtual soundcard and the whole system is just slightly slow.

There's a number of various ways of doing so, but... i haven't used any of these:

cpuset
taskset
Libvirt's way - similar to cpuset maybe?..

And there was a way of using isolcpus and switching the cores offline for host(NoHZ mode?) and only bringing them up for the guest system, but i can't find more info... But yeah, somewhere in the thread someone did that.

Dynamically... Well, dynamic allocation ruins the latencies, as i understand it. Using cpuset does exactly that you want - it only makes the process working on the specified core, without forcing all other process from that core.


I'm not so familiar with all this stuff, because i don't have "enough" CPU cores for experiments.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5244 2015-06-01 02:12:03

ceri
Member
Registered: 2013-10-12
Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

*SNIP*
And that's only the QEMU group. As you can see, firefox is more memory-hungry.

And now seriously:
I have 4 cores in my cheap Athlon X4 750K, and i'm assigning all the cores to the guest. Yeah, when either host or guest is loaded - the other system will stutter, lag and suffer from CPU starvation. But i'm very rarely that "productive".

You'll want additional cores to use cpu pinning.
So, for example, i could've cut three cores from the host and dedicate(pin) them to the VM. That would give the guest a very good CPU performance(almost native/baremetal) and what is more important - reduce the latencies and that would theoretically fix the audio problems. But i don't like to leave my host with one core only, because... i feel sad for it.

So basically, more cores - more performance, obviously, and if getting an i7 isn't much a financial burden for you - it would be great to use it..

Haha point taken.

(As a side note, you should consider using tree style tabs.)

I'm planning on reusing my old ram (16GB) and using the savings to get an i7-4790K which does support VT-d: http://ark.intel.com/compare/80810,80811,80807,80806

Given that this has 4 real cores and 8 threads, would I still have to worry about CPU pinning to avoid stuttering? If so, I think i would just allocate 50% to each side.

Offline

#5245 2015-06-01 03:45:10

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Both patches works for fixing the null pointer at shutdown, thanks.

EDIT:

"dmar: DRHD: handling fault status reg 2"
is caused by USB3 controller in MB when using qemu git version, but in stable works

Last edited by Cubex (2015-06-01 04:05:47)

Offline

#5246 2015-06-01 16:39:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guys, some time ago i was told that UEFI GOP can't go beyond 1024x768 without device-specific drivers.

How does gigabyte Z87X-UD3H output 1920x1080 in UEFI setup? Intel-specific hacks?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5247 2015-06-02 21:03:11

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all,
I've gone through some of the posts around pages 26-29 concerning running vga passthrough as a non-root user. Although I used `chown` on /dev/vfio for my user, I still got the following error:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga none #--- it returns this: -smp 4,sockets=1,cores=4,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga noneqemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f400232e9d0, 0x0, 0xc0000, 0x7f3fa0000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f400232e9d0, 0xc0000, 0x20000, 0x7f3fee600000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f400232e9d0, 0x100000, 0x3ff00000, 0x7f3fa0100000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to setup container for group 15 qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

The same command runs fine as root. I have also chown'ed /dev/hugepages in addition -- to no avail. Does anyone have an answer to this problem? I apologize if an answer was earlier in the 200+ pages that I couldn't search well.

Offline

#5248 2015-06-02 21:07:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noctlos wrote:

Hey all,
I've gone through some of the posts around pages 26-29 concerning running vga passthrough as a non-root user. Although I used `chown` on /dev/vfio for my user, I still got the following error:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga none #--- it returns this: -smp 4,sockets=1,cores=4,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga noneqemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f400232e9d0, 0x0, 0xc0000, 0x7f3fa0000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f400232e9d0, 0xc0000, 0x20000, 0x7f3fee600000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio_dma_map(0x7f400232e9d0, 0x100000, 0x3ff00000, 0x7f3fa0100000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to setup container for group 15 qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

The same command runs fine as root. I have also chown'ed /dev/hugepages in addition -- to no avail. Does anyone have an answer to this problem? I apologize if an answer was earlier in the 200+ pages that I couldn't search well.

In order to assign a PCI device to a VM, all of the guest memory needs to be pinned (ie. locked) into host memory and mapped through the IOMMU.  A normal user only has the ability to lock 64KB of memory (see ulimit).  Your VM is probably bigger than that, therefore you need to increase the locked memory limit for the user to run the VM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5249 2015-06-02 22:16:48

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

In order to assign a PCI device to a VM, all of the guest memory needs to be pinned (ie. locked) into host memory and mapped through the IOMMU.  A normal user only has the ability to lock 64KB of memory (see ulimit).  Your VM is probably bigger than that, therefore you need to increase the locked memory limit for the user to run the VM.

Thanks for the tip! How do I increase the locked memory limit?

Offline

#5250 2015-06-02 22:19:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noctlos wrote:
aw wrote:

In order to assign a PCI device to a VM, all of the guest memory needs to be pinned (ie. locked) into host memory and mapped through the IOMMU.  A normal user only has the ability to lock 64KB of memory (see ulimit).  Your VM is probably bigger than that, therefore you need to increase the locked memory limit for the user to run the VM.

Thanks for the tip! How do I increase the locked memory limit?

/etc/security/limits.d/ (see limits.conf it the parent directory)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5251 2015-06-03 13:57:36

goeo_
Member
Registered: 2015-06-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi. I'm trying to do this, but I keep getting a code 43 error.

I have a laptop with an intel i7 4900mq cpu and a nvidia gtx 770m gpu. I'm assigning the nvidia gpu to the vm (running windows 8.1) and using the igpu (intel hd 4600) on the host. Here's my vm's xml file:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>goeosvm</name> <uuid>fb039381-3dbf-4bb6-b8cd-83a602b141db</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>8</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='4' threads='2'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdb4'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/goeosvm.img'/> <target dev='hdc' bus='ide'/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:3a:8b:fc'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes' listen='127.0.0.1' keymap='tr'> <listen type='address' address='127.0.0.1'/> <image compression='off'/> </graphics> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom bar='on' file='/home/goeo_/Documents/MSI.GTX770M.3072.130324.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/> </qemu:commandline> </domain>

What am I doing wrong?

Offline

#5252 2015-06-03 14:09:11

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
noctlos wrote:
aw wrote:

In order to assign a PCI device to a VM, all of the guest memory needs to be pinned (ie. locked) into host memory and mapped through the IOMMU.  A normal user only has the ability to lock 64KB of memory (see ulimit).  Your VM is probably bigger than that, therefore you need to increase the locked memory limit for the user to run the VM.

Thanks for the tip! How do I increase the locked memory limit?

/etc/security/limits.d/ (see limits.conf it the parent directory)

I created a file at /etc/security/limits.d/50-qemu.conf

@domain hard as 10000000

I also did the following:

#chown noctlos:noctlos /dev/vfio/15 #chmod 666 /dev/vfio/vfio #chown -R noctlos:noctlos /dev/hugepages

unfortunately, when I go to run qemu, it again gives me:

qemu-system-x86_64: -device vfio-pci,host=03:00.0: vfio_dma_map(0x7f69d1c20890, 0x0, 0x80000000, 0x7f6718000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0: vfio_dma_map(0x7f69d1c20890, 0x100000000, 0x200000000, 0x7f6798000000) = -12 (Cannot allocate memory) qemu-system-x86_64: -device vfio-pci,host=03:00.0: vfio: memory listener initialization failed for container qemu-system-x86_64: -device vfio-pci,host=03:00.0: vfio: failed to setup container for group 15 qemu-system-x86_64: -device vfio-pci,host=03:00.0: vfio: failed to get group 15 qemu-system-x86_64: -device vfio-pci,host=03:00.0: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:00.0: Device 'vfio-pci' could not be initialized

And for reference:

% ulimit -a :( -t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 8192 -c: core file size (blocks) unlimited -m: resident set size (kbytes) unlimited -u: processes 63576 -n: file descriptors 4096 -l: locked-in-memory size (kbytes) 64 -v: address space (kbytes) unlimited -x: file locks unlimited -i: pending signals 63576 -q: bytes in POSIX msg queues 819200 -e: max nice 20 -r: max rt priority 0 -N 15: unlimited

Last edited by noctlos (2015-06-03 14:10:37)

Offline

#5253 2015-06-03 14:16:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@noctlos

You're not setting the locked memory limit


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5254 2015-06-03 14:18:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@goeo_

You're expecting discrete graphics in a laptop to behave the same as a discrete card in a desktop.  You might have a whole new set of issues to tackle for optimus.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5255 2015-06-03 14:44:51

goeo_
Member
Registered: 2015-06-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@goeo_

You're expecting discrete graphics in a laptop to behave the same as a discrete card in a desktop.  You might have a whole new set of issues to tackle for optimus.

i thought optimus was software only? what kind of different behaviors should i expect? has this never been done before?

Offline

#5256 2015-06-03 14:47:54

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well, at least it somehow arbiters the display, it usually copies contents of the video buffers back and forth, I'd kinda fear what it will do if there will be different OSes using them ... though you can try smile)

Offline

#5257 2015-06-03 15:10:41

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@noctlos

You're not setting the locked memory limit

Yup. Instead of:

@group hard as <limit>

I needed to use something like

@group hard memlock <limit> @group soft memlock <limit>

Along with chown'ing the appropriate files mentioned previously, I can start my VM as user. Woohoo!

Now, if only I could figure out a way to get the sound to work!

I'm using the qemu option

 -soundhw hda 

with QEMU_AUDIO_DRV=pa. I don't get any errors when I run it as user as some have, but the sound quality is as bad as ever. Did anyone solve this?

EDIT: Nevermind, the sound works okay when I increase QEMU_PA_SAMPLES to the default of 4096.

Last edited by noctlos (2015-06-03 15:15:30)

Offline

#5258 2015-06-03 16:33:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

goeo_ wrote:

has this never been done before?

Exactly
BTW, i've observed very bad behaviour of 770m in some other(i think it was lenovo) notebook, breaking lspci with unknown headers.
So i'd totally not expect it to work. Ask ghormoon, he has an old laptop with a more "true" dedicated card, and that's even a radeon, and... there's still a huge fail.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5259 2015-06-03 23:31:55

avi9526
Member
Registered: 2015-05-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I tried to use GPU passthrough with libvirt, qemu-kvm on ubuntu, but there was a problem related to PCI-e in guest OS was downgraded to version 1.1 instead of 2.0 (when run win on bare metal), does this problem appears with latest software versions on arch?

Offline

#5260 2015-06-04 00:33:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

avi9526 wrote:

Hi, I tried to use GPU passthrough with libvirt, qemu-kvm on ubuntu, but there was a problem related to PCI-e in guest OS was downgraded to version 1.1 instead of 2.0 (when run win on bare metal), does this problem appears with latest software versions on arch?

Devices will often reduce the link rate to save power when not under load, this is why tools like gpu-z provide a render load to see the correct link rate.  You also don't want to fully trust what the guest thinks of the link rate, look at the host device with lspci.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5261 2015-06-04 01:12:51

PrinzipDesSees
Member
Registered: 2015-06-04
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can pass a second gpu through just fine with a naked qemu line, like exactly in the OP. But when I try to punch this into a libvirt xml, it won't work.
Even with xmls others posted as working.
The log says: that the monitoring socket could not be created and that vfio-pci could not be initialised because it gets access to /dev/vfio/16 denied.
Well, I have made everything root for testing in /etc/libvirt/libvirtd.conf and /etc/libvirt/qemu.conf.
I chmodded /dev/vfio/16 /dev/vfio/17 /dev/vfio/7 /dev/vfio/vfio to 666.
I set selinux to disabled.
I have allowed the device cgroup in the config files.
I tried with brute force on a testing kernel without support for cgroups, selinux, capabilities and every security feature I could find ripped out or set to allow. Still access denied.
Inbefore the failing are warnings that the virtual instance is "tained" with high privileges (o rly?) and custom argv. Yeah sure thats right. Does that taint prevent starting?
How can I give qemu the x-vga=on parameter without having the extra <qemu:commandline> segment in the xml? Would make life a lot easier.
I was browsing the libvirt source code some time but I wasnt entirely sure where the allowed extra arguments are passed to qemu.
I will paste actual configs, logs and versions later, because the target computer is offline and I am typing from another guys internet.

Offline

#5262 2015-06-04 01:40:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@PrinzipDesSees

See part 5 of the series in my blog (sig) for how to use wrapper scripts to apply x-vga=on.  Using qemu:args is likely the problem, there are very few cases where those are necessary or advised.  You should not need to touch libvirtd.conf or qemu.conf.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5263 2015-06-04 02:34:15

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

update to the laptop experiment:
I've fiddled a bit more with drivers, ended up on latest beta ones anyway ...

it works except that it turns the display off in windows for some strange reason.
in linux, everything is ok, extending to external display too.
in windows, laptop screen goes black, though it actually sees it in the OS, detects resolutions and such. external display works fine. somehow usable if I disable the internal one so it won't extend desktop to it. windows rated it 5.3 and 5.9, which is aprox. what it had bare metal (I don't remember exactly). I'll do some game tests someday smile

what I need to fix now, is that the VMs don't shutdown properly. thinking of trying another distro (more stable kvm and qemu), did anyone succed vfio passthrough eg. on debian?
also next thing to do is to modify qemu for ps/2 passthrough, write some management scripts and I'm good to go! smile
I actually can somehow live with that the internal screen doesn't work in windows, though it's not preffered variant (can't play games except at home, but I don't have that much time for gaming anyway tongue)

Offline

#5264 2015-06-04 02:35:35

Duodecillian
Member
Registered: 2012-05-28
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Say I wanted to run this to use primarily for virtual reality applications. Would the latency inherent in virtualization be a deal breaker? Has anyone tried?

Offline

#5265 2015-06-04 08:17:42

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there! Long time no see! My System is working really well. However I am still at Linux 3.9.

Is it safe to upgrade to 4.04? I am using the regular Linux kernel, NOT a patched one.

Offline

#5266 2015-06-04 14:17:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

Hi there! Long time no see! My System is working really well. However I am still at Linux 3.9.

Is it safe to upgrade to 4.04? I am using the regular Linux kernel, NOT a patched one.

You're really the only one that can answer this, but I sure hope the answer is yes.  That's certainly the goal, to improve things without breaking existing users.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5267 2015-06-04 14:30:34

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duodecillian wrote:

Say I wanted to run this to use primarily for virtual reality applications. Would the latency inherent in virtualization be a deal breaker? Has anyone tried?

Direct device assignment is generally a way to avoid the latency inherent in virtualization.  For the most part, the VM has direct access to the hardware.  The exceptions to that are interrupts, which are bounced through the host on current hardware, PCI config space, which is filtered by the hypervisor for virtualization, and any regions that cannot map directly to the guest, such as quirks for mmio mirrors of config space, io-port regions, or regions smaller than the processor page size.  There are also configuration options important to reducing latency, such as MSI tuning, vCPU pinning and proper balancing of CPU resources between host and guest, and potentially even IRQ pinning in the host.  If you have a processor that supports Intel APICv, the latter may allow interrupts to be injected to the guest without a vmexit.  We'll have to wait for software and hardware to support Intel Posted Interrupts to avoid the host in the interrupt path altogether.  People here are showing results within a few percentage points of bare metal in the best cases, so I would naively assume that the same is possible for VR applications.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5268 2015-06-04 15:11:14

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

awesome, thank you very much. I tried it and nothing broke! So I'm up to date.

I have another question that's bugging me for months.

Whenever I turn on my pc, first automatically archlinux starts and after that through virt-manager Windows 8.1.

How about turning it off? Is there a single command I can use to first shut down safely my virtualized Windows and after that the host?

Because If I simply use "poweroff" in terminal I don't think it's done properly, right??

Offline

#5269 2015-06-04 15:27:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

awesome, thank you very much. I tried it and nothing broke! So I'm up to date.

I have another question that's bugging me for months.

Whenever I turn on my pc, first automatically archlinux starts and after that through virt-manager Windows 8.1.

How about turning it off? Is there a single command I can use to first shut down safely my virtualized Windows and after that the host?

Because If I simply use "poweroff" in terminal I don't think it's done properly, right??

On Fedora there's an /etc/sysconfig/libvirt-guests file that includes such things as:

# action taken on host boot # - start all guests which were running on shutdown are started on boot # regardless on their autostart settings # - ignore libvirt-guests init script won't start any guest on boot, however, # guests marked as autostart will still be automatically started by # libvirtd #ON_BOOT=start # Number of seconds to wait between each guest start. Set to 0 to allow # parallel startup. #START_DELAY=0 # action taken on host shutdown # - suspend all running guests are suspended using virsh managedsave # - shutdown all running guests are asked to shutdown. Please be careful with # this settings since there is no way to distinguish between a # guest which is stuck or ignores shutdown requests and a guest # which just needs a long time to shutdown. When setting # ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a # value suitable for your guests. #ON_SHUTDOWN=suspend # If set to non-zero, shutdown will suspend guests concurrently. Number of # guests on shutdown at any time will not exceed number set in this variable. #PARALLEL_SHUTDOWN=0 # Number of seconds we're willing to wait for a guest to shut down. If parallel # shutdown is enabled, this timeout applies as a timeout for shutting down all # guests on a single URI defined in the variable URIS. If this is 0, then there # is no time out (use with caution, as guests might not respond to a shutdown # request). The default value is 300 seconds (5 minutes). #SHUTDOWN_TIMEOUT=300

I assume this works in conjunction with the systemd libvirt-guests.service.  For an assigned device VM I'd generally suggest shutdown vs suspend, unless you know suspend works with your devices.  I also notice that Windows only wakes up the first time sending a shutdown command, so there may be some configuration in the guest necessary to make it shutdown on the first attempt.  There's also libvirt xml options to configure whether suspend is available to the guest.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5270 2015-06-04 16:26:27

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

that's great. I will try this.

Thank you very much Alex! Always a help. Awesome!

Offline

#5271 2015-06-05 14:15:39

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a updated site/post where I can find the newest _working_ nvidia driver version and a todo list to get it working?

Offline

#5272 2015-06-05 14:31:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rommy wrote:

Is there a updated site/post where I can find the newest _working_ nvidia driver version and a todo list to get it working?

There are no non-working versions.  Follow the guide series in my blog below for what I think is the current best setup.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5273 2015-06-05 19:22:17

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Having issues with this, see my thread here: https://bbs.archlinux.org/viewtopic.php … 6#p1534516. Some sort of driver conflict, I think.

Offline

#5274 2015-06-05 19:41:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

Having issues with this, see my thread here: https://bbs.archlinux.org/viewtopic.php … 6#p1534516. Some sort of driver conflict, I think.

There's a perfectly good guide that I'd recommend in the link in my sig, I'd strongly recommend it over any of this vfio-bind script nonsense.  Your picture is way too blurry to figure out anything meaningful from it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5275 2015-06-05 19:58:58

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Should I start from Part 3 and proceed from there? It will be a tad difficult since I cannot boot into a graphical environment, I think i just need to remove pci-stub from the initramfs, and boot without loading the systemd service and kernel module (pci_stub).

Last edited by garnerlogan65 (2015-06-05 20:00:04)

Offline

#5276 2015-06-05 20:10:59

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

Should I start from Part 3 and proceed from there? It will be a tad difficult since I cannot boot into a graphical environment, I think i just need to remove pci-stub from the initramfs, and boot without loading the systemd service and kernel module (pci_stub).

You could have skimmed parts 1 & 2 in the time you've been waiting for a reply...


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5277 2015-06-05 20:25:53

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That's what I'm about to do. I got my system working again, by undoing all of my changes. Hopefully your method works.

Last edited by garnerlogan65 (2015-06-05 20:46:29)

Offline

#5278 2015-06-05 20:43:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

That's what I'm about to do. I got my system working again, by undoing all of my changes. Hopefully your method works.

EDIT: i have an i7 4960X/Asus Rampage IV Extreme, do I need the ACS patch?

Read part 1 again


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5279 2015-06-05 20:50:29

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems like it should work fine. I'll let you know what happens.

Offline

#5280 2015-06-05 20:56:31

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm on part 3. A little confused, how do I get this: options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:fffffff, I know how to get the vendor IDs but I'm confused on how to get this.

Offline

#5281 2015-06-05 21:03:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

I'm on part 3. A little confused, how do I get this: options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:fffffff, I know how to get the vendor IDs but I'm confused on how to get this.

If pci-stub is a module you can run modinfo pci-stub, otherwise you can look at the code and find this:

Initial PCI IDs to add to the stub driver, format is vendor:device[:subvendor[:subdevice[:class[:class_mask]]]] and multiple comma separated entries can be specified

So vendor:device are the minimum required.  ffffffff is the kernel definition for PCI_ANY_ID, so it matches anything.  So we match any device ID, any subsystem vendor ID and device ID for AMD or Nvidia, then we go on to filter a little further with class codes for multimedia for VGA devices.  You should not need to do anything but possibly use this directly.  If you have two identical 780s with only one to be used by the guest, you probably want to concentrate on the initramfs scripts using the driver_override option.  If they're not identical (different subsystem vendors), then pci-stub is an option.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5282 2015-06-05 21:06:04

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

They are identical. So don't use pci_stub? I still need to use vfio-pci with ids right? I don't understand all the different number types.

EDIT: Arch doesn't have 4.1 yet, so I guess I have to make do with pci-stub.

Last edited by garnerlogan65 (2015-06-05 21:35:28)

Offline

#5283 2015-06-05 22:40:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

They are identical. So don't use pci_stub? I still need to use vfio-pci with ids right? I don't understand all the different number types.

EDIT: Arch doesn't have 4.1 yet, so I guess I have to make do with pci-stub.

Start here:

Another issue that users encounter when sequestering devices is what to do when there are multiple devices with the same vendor:device ID and some are intended to be used for the host...

That does not rely on anything new in 4.1, the driver_override interface has been available for some time.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5284 2015-06-06 00:29:11

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Followed the rest of part 3. Here are the contents of /etc/dracut.conf.d/local.conf (http://pastebin.com/B7TcbCeF), and my /etc/modprobe.d/local.conf (http://pastebin.com/1qCFMacS), I had to make the vfio script in the /etc/ folder since I do not have an /sbin directory for some reason. Obviously as your instructions state dracut creates "initramfs-uname -r.img," am I supposed to replace the other initramfs in my bootloader with this one? Also, upon generating the new initramfs, dracut threw some errors "Failed to install module vfio" and it also failed to install vfio_iommu_type1. Did I do something wrong here?

Offline

#5285 2015-06-06 01:00:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

Followed the rest of part 3. Here are the contents of /etc/dracut.conf.d/local.conf (http://pastebin.com/B7TcbCeF), and my /etc/modprobe.d/local.conf (http://pastebin.com/1qCFMacS), I had to make the vfio script in the /etc/ folder since I do not have an /sbin directory for some reason. Obviously as your instructions state dracut creates "initramfs-uname -r.img," am I supposed to replace the other initramfs in my bootloader with this one? Also, upon generating the new initramfs, dracut threw some errors "Failed to install module vfio" and it also failed to install vfio_iommu_type1. Did I do something wrong here?

Yes, you're building an initramfs to replace the original.  Is vfio built as a module or static?  (grep VFIO /boot/config-`uname -r`)  I generally wouldn't recommend it static, we'll need to find some other way to get the script to run in the initramfs if it's not a module.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5286 2015-06-06 02:58:20

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Doing "grep VFIO" on the normal initramfs, returns no output. It is built as a module (modinfo vfio), didn't realize you were the author smile I literally thought the image was supposed to be called "initramfs-uname -r", I have since fixed it. Should I try booting with it?

Offline

#5287 2015-06-06 17:31:50

Rommy
Member
Registered: 2014-10-17
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

There are no non-working versions.  Follow the guide series in my blog below for what I think is the current best setup.

Thank you very much :-)

Offline

#5288 2015-06-06 18:06:17

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

In the first post of my recent how-to series I talk about the hardware I use, not that I'd recommend it, but to talk about why it works for my config - http://vfio.blogspot.com/2015/05/vfio-g … dware.html

I won't be as generous as Duelist, skip AMD platforms.  They've admittedly given up on the single core performance race, you don't know what you're getting on an APU system for isolation, and their latest performance/workstation class chipset is quickly coming up on its four year anniversary since launch.
. . .

If you really want to assign a disk controller to a VM, you're into risky territory, mostly because the cheap ones are all crap.  If you try to assign an integrated controller, the boot ROM for it may be integrated into the system BIOS, so you'll have no way to boot the VM unless you can extract it... and hope it supports UEFI if you're going the OVMF route with the VM.  You may have better luck with a discrete HBA, but you're still in the unknown/crap space.  Personally I find virtio-scsi w/ multi-queue and a fast backing setup to meet my needs.

Sorry for slight off-topic; I'm planning to build a kvm VM running large ZFS filesystem; this filesystem works best when it has direct access to backing storage. The idea is to use vfio and pass an LSI HBA (either build into SuperMicro motherboard, or a discrete one) to VM, and attach all ZFS disks to it. Guest would be using mpt2sas driver. However for boot and root filesystem of guest I would use regular block device with virtio driver, rather than passed-through HBA. I suppose in this case there should be no need to worry about BIOS of HBA (no boot device) and my only worry should about tuning the machine for ZFS? BTW, apparently only little effort is required to use ZFS under kvm, e.g. https://github.com/zfsonlinux/zfs/issue … -106493033


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5289 2015-06-06 20:29:56

emacsomancer
Member
Registered: 2014-09-20
Posts: 103

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My relevant computer specs:

I'm considering getting either another GTX 960 or else a GTX 970 (of some sort). (Or else getting a 720, 730, 750 for use with the Linux host, and move the GTX 960 for use by Windows]. For the purpose of running games in a Windows VM within an Arch host.

As far as I can tell all of my parts of VT-d capable/enabled. (I can enable VT-d in my motherboard bios, I noticed).   

Before I buy another GPU I wanted to check to make sure it is likely I would be able to get GPU passthrough to a Windows VM working with this setup. (Any concerns with GTX 960 or 970? or recommendations?)

[I'm assuming there's no easy way to dynamically switch from NVIDIA to integrated video to free up my NVIDIA card when I want to run Windows.]

Last edited by emacsomancer (2015-06-06 21:04:45)

Offline

#5290 2015-06-06 20:55:01

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

... UEFI GOP can't go beyond 1024x768

This is false in general.

without device-specific drivers.

This is self-contradictory :)

How does gigabyte Z87X-UD3H output 1920x1080 in UEFI setup? Intel-specific hacks?

GOP is shorthand for GRAPHICS_OUTPUT_PROTOCOL. It is an interface (an abstraction) that is defined by the UEFI specification. UEFI fimware video drivers produce it. It can flexibly express any mode list that the actual driver decides to expose, based on video RAM and any other internal sw & hw factors. So this is why the first quote is false.

The second quote makes no sense because GOPs are exposed by, and are the standard interface of, device-specific UEFI video drivers. A UEFI GOP does not exist without a device-specific UEFI driver; at worst you can say that the actual driver is low quality or generic, because, while it recognizes the video card, it cannot utilize its full capabilities.

One nice bit about the GOP design is that the runtime OS (a UEFI OS) can inherit the linear framebuffer characteristics (start address, size, pixel format, resolution etc; just writing from memory, not checking the spec) from the GOP, and continue using the video mode that has been last selected while in UEFI. There won't be any acceleration, but you can have a nice (fixed) color depth and (fixed) resolution in the UEFI runtime OS without device-specific drivers. (In order to select a different video mode, you have to reboot into UEFI, or install the OS-level device-specific video driver.)

The three occurrences of the word "inherit" in the OVMF whitepaper all relate to this matter.

Offline

#5291 2015-06-07 11:30:51

WildyLion
Member
Registered: 2015-06-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi.

I'm kinda new here, but I managed to set up KVM VGA passthrough using VFIO with Windows 8.1 as a guest. The thing is, my VM seems to have some timing issues despite all the hyper-v enlightenments turned on.
E.g. I'm able to play Witcher 3, but with post-processing settings turned down a notch from what I'm usually playing on bare metal at. I'm passing through a Xonar U7 audio card via USB controller passthrough and the sound seems to click sometimes, indicating something is wrong with the timing.
I'm also passing a USB hub containing my mouse, trackball and keyboard to the same guest using a hardware ATEN KVM device that switches the USB ports to the needed root hub.

My configuration is:

1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html AMD FX-8350
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html ASUS M5A99FX PRO R 2.0
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html 16 GB RAM
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Sapphire R9 290 Vapor-X as a passthrough card (Slot 3), hosts a Dell 2415 via DP
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html ATI Radeon HD 4650 as a boot VGA (Slot 1), hosts both of my monitors (Dell 2415 and 2412)
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Boot options: pci-stub.ids=1002:67be,1002:aac8 iommu=pt iommu=1
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Using libvirt 1.2.15, qemu 2.3.0 and stock kernel (4.0.4-2 at present)
1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html VM config is here: http://pastebin.ca/3020403

Could anybody please give me a hint on where to look next?

Edit: Maybe I should reinstall everything yet again and use Q35? Will that solve my issues?

Last edited by WildyLion (2015-06-07 11:35:23)

Offline

#5292 2015-06-07 13:14:10

jurie
Member
Registered: 2015-06-07
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there,

I'm using ASRock Z97 Extreme6 and a Intel i5-4690K with an old GTX 670.
I followed the steps from https://www.youtube.com/watch?v=Tq345_v2-Ew  but cant see to get it working?
My setup/ config is here http://pastebin.com/kp3ES6Sn

The virtual machine boots, but I cant see any output on my GTX 670 card?
Should I just enable i915 VGA arbiter patch as a module "i915" ?

Could someone please point me in the right direction please?
Thanks

Offline

#5293 2015-06-07 16:39:31

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am having issues setting up sound in my Guest VM. Guest is running under Win 10, here is the "launcher":

qemu-system-x86_64 -name winten -enable-kvm -m 16384 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS.fd \ -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:1d.0 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/guest/win10,id=win,format=raw,if=none -device scsi-hd,drive=win \ -drive file=/dev/guest/win7,id=data,format=raw,if=none -device scsi-hd,drive=data \ -net nic,model=virtio,macaddr=52:54:00:7c:3c:42 -net bridge,br=br0 \ -soundhw hda \ -nographic

Everything seems OK. However, sounds is "lagging" / distorted. I have tried several stuff from first post, like :

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

.. changing sample size from 32 to 8192.. didn't help.

Please also note that I see the following message while running the launch script :

pulseaudio: pa_context_connect() failed pulseaudio: Reason: Connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver

And :

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64...

Also changing buffer size and period size from several values.

I always end up in lagging audio.

I remember having this issue months ago (on a Win 7 host), then I fixed it, but I can't recall what I did. Maybe it was something about using pulse system wide. But I am not willing to do it again unless I am really sure it comes from here.

Lately I have been using  a USB sound card shared from Host to Guest and it worked fine. However, I had issue in the host, since I had to reselect sound card every time I plug it out / back in, which bored me.

If anyone has any insight on how to debug this, it would be gladly appreciated. Otherwise, it might be better to find a fix (such as this : http://alsa.opensrc.org/Udev, I believe) for the USB sound card on host, but this will be off topic.

Thanks a lot in advance

Last edited by Nesousx (2015-06-07 16:39:53)

Offline

#5294 2015-06-07 17:09:04

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello! smile
I have successfully gotten VGA-passthrough with qemu working on my second monitor with my 970gtx on my Sr-2 classified.
Currently I am having troubles getting qemu to recognize a boot device, I have searched for MANY hours, so perhaps someone could point me in the right direction? Perhaps I am sleepy and missed something.
Currently using this vm script. I have left the disk and cd devices undefined here, as I have no clue what is wrong and would like to start from a clear slate. QEMU boots fine on my second card, and passthrough is working.

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

#Startup Windows 7 Guest with VGA-Passthrough
qemu-system-x86_64 -name win7 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /home/coyote/seabios/out/bios.bin \
-device vfio-pci,host=04:00.0,x-vga=on -device vfio-pci,host=04:00.1 \
-vga none \

exit 0

Offline

#5295 2015-06-07 17:18:08

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

Hello! smile
I have successfully gotten VGA-passthrough with qemu working on my second monitor with my 970gtx on my Sr-2 classified.
Currently I am having troubles getting qemu to recognize a boot device, I have searched for MANY hours, so perhaps someone could point me in the right direction? Perhaps I am sleepy and missed something.
Currently using this vm script. I have left the disk and cd devices undefined here, as I have no clue what is wrong and would like to start from a clear slate. QEMU boots fine on my second card, and passthrough is working.

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

#Startup Windows 7 Guest with VGA-Passthrough
qemu-system-x86_64 -name win7 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /home/coyote/seabios/out/bios.bin \
-device vfio-pci,host=04:00.0,x-vga=on -device vfio-pci,host=04:00.1 \
-vga none \

exit 0

Hi,

You don't seem to be using any "hard drive" in your "starter script". You should probably have a line like :

-drive file=/dev/guest/win10,id=win,format=raw,if=none -device scsi-hd,drive=win

Have you checked Aw's Blog (http://vfio.blogspot.fr/), especially the 5 parts tutorial on VFIO GPU? Plenty of advices there.

You might also want to re-read complety first post.

EDIT : you said, that you left the CD and HDD devices undefined. But I believe we actually want to see the full config file in order to help you.

Last edited by Nesousx (2015-06-07 17:20:36)

Offline

#5296 2015-06-07 17:20:11

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

"I have left the disk and cd devices undefined here, as I have no clue what is wrong"

Thanks for reply but I said this already tongue

I will check that link, thank you, other suggestions would be great!

Offline

#5297 2015-06-07 17:20:56

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

"I have left the disk and cd devices undefined here, as I have no clue what is wrong"

Thanks for reply but I said this already tongue

I will check that link, thank you, other suggestions would be great!

Check my edit.

Offline

#5298 2015-06-07 17:31:04

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, well what is a very common way to set this up? Could I use an iso made from a windows cd?

Currently I have created a windows iso from the windows 7 ultimate disk. and prepared a raw windows img using the dd command.
do I need to specify a bus device? I have recieved no errors other than no bootable device found...

/home/coyote/windows7.iso
/home/coyote/win7.img
/home/coyote/virtio.iso "windows drivers"

Many thanks in advance.

Offline

#5299 2015-06-07 17:40:46

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

Ok, well what is a very common way to set this up? Could I use an iso made from a windows cd?

Currently I have created a windows iso from the windows 7 ultimate disk. and prepared a raw windows img using the dd command.
do I need to specify a bus device? I have recieved no errors other than no bootable device found...

/home/coyote/windows7.iso
/home/coyote/win7.img
/home/coyote/virtio.iso "windows drivers"

Many thanks in advance.

What you are posting doesn't look anything like it was suggested by OP and I have no idea how you are using this in your script.

Here is a working config, try to modify it to your needs (drives / disks part) :

qemu-system-x86_64 -name winten -enable-kvm -m 16384 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS.fd \ -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 -device vfio-pci,host=00:1d.0 \ -vga none \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/guest/win10,id=win,format=raw,if=none -device scsi-hd,drive=win \ -drive file=/dev/guest/win7,id=data,format=raw,if=none -device scsi-hd,drive=data \ -drive file=/home/nesousx/Isos/virtio-win-0.1-81.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/home/nesousx/Isos/Win_7_Pro_x64.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -net nic,model=virtio,macaddr=52:54:00:7c:3c:42 -net bridge,br=br0 \ -soundhw hda \ -nographic

First 2 drives statements are "system" and "data" drives, they are LVM volumes. Then the 2 others are iso files (one for Win 7, one for the virtio drivers). Also, if you are using a "just created" iso, make sure it is a working and really boots.

Edit : try this from your original script :

#Startup Windows 7 Guest with VGA-Passthrough qemu-system-x86_64 -name win7 -enable-kvm -m 1024 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /home/coyote/seabios/out/bios.bin \ -device vfio-pci,host=04:00.0,x-vga=on -device vfio-pci,host=04:00.1 \ -drive file=/home/coyote/win7.img,id=win,format=raw,if=none -device scsi-hd,drive=win \ -drive file=/home/coyote/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/home/coyote/windows7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \ -vga none

Last edited by Nesousx (2015-06-07 21:39:40)

Offline

#5300 2015-06-08 00:45:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

The second quote makes no sense because GOPs are exposed by, and are the standard interface of, device-specific UEFI video drivers. A UEFI GOP does not exist without a device-specific UEFI driver; at worst you can say that the actual driver is low quality or generic, because, while it recognizes the video card, it cannot utilize its full capabilities.

Who writes those UEFI drivers? Firmware-engineers(ASUS, Gigabyte, guys who produce the motherboards, etc.), vendors of the devices(EVGA, Sapphire, guys that don't produce motherboards etc.) or AMD/Intel themselves? Where they are stored? Are they hidden somewhere in the firmware(UEFI), or in the device ROM? If it's an intel iGPU, where that ROM with the device driver is located, as i can't recall intel GPUs being listed as a regular PCI device?
And..
Why does emulated QXL can't go beyond 1024x768... even with an extended amount of VRAM given?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5301 2015-06-08 07:21:23

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for tips Nesousx I will give it a shot when I am off work this evening. This forum post is amazing smile Respect for everyone who put in the time here.

Offline

#5302 2015-06-08 10:03:46

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
lersek wrote:

The second quote makes no sense because GOPs are exposed by, and are the standard interface of, device-specific UEFI video drivers. A UEFI GOP does not exist without a device-specific UEFI driver; at worst you can say that the actual driver is low quality or generic, because, while it recognizes the video card, it cannot utilize its full capabilities.

Who writes those UEFI drivers? Firmware-engineers(ASUS, Gigabyte, guys who produce the motherboards, etc.), vendors of the devices(EVGA, Sapphire, guys that don't produce motherboards etc.) or AMD/Intel themselves? Where they are stored? Are they hidden somewhere in the firmware(UEFI), or in the device ROM? If it's an intel iGPU, where that ROM with the device driver is located, as i can't recall intel GPUs being listed as a regular PCI device?

From a UEFI perspective, it doesn't matter where the UEFI driver binary comes from. It can be part of the system flash chip (soldered to the motherboard), it can be part of the PCI oprom (in case of a discrete graphics card), but you can also load it from a FAT32 filesystem (eg. USB pendrive or EFI system partition), from the UEFI shell, or with a Driver#### variable.

In practice, for discrete graphics card, "PCI oprom" is the only sensible answer -- ie. you get the driver when you buy the card. For integrated graphics cards, I think it's usually part of the system firmware.

And..
Why does emulated QXL can't go beyond 1024x768... even with an extended amount of VRAM given?

The UEFI GOP installed for QXL can very well go beyond 1024x768. You can see the supported resolutions here:

https://github.com/tianocore/edk2/blob/ … ize.c#L211

I use QXL resolutions higher than 1024x768 in my guests all the time.

I think you are conflating two things here. The 1024x768 resolution limit is in place for *Windows 7* only, and that's because Windows 7 is *not* a UEFI operating system when it comes to its builtin video driver. Windows 7 will boot on a UEFI platform, yes, even without a CSM, but its builtin video driver is broken in the sense that it will *only* look for the legacy video bios services (the VBE, VESA BIOS Extansion). It won't even look for a preconfigured framebuffer, it *insists* on the legacy BIOS video services (Int10h). A fully conformant UEFI operating system is not permitted to do this, and this is also why you can run Windows 7 on physical UEFI platforms only if you enter the UEFI setup utility during boot, and enable the CSM or "VBIOS" support or whatever the vendor in question calls it.

We disliked Windows 7's dependency on a full-blown CSM, and worked around it in OVMF. Namely, OVMF's video driver provides, fully independently of the multi-resolution UEFI GOP, a very limited VESA BIOS Extension too, even without incorporating the CSM. We call this the "Int10h shim". It is a shim because it provides a minimal set of VBE services, which are expected to be interpreted *only* by the Windows 7 real mode emulator. It is written in assembly, and it is placed in memory, and provides only the services, so that Windows 7 can be installed.

https://github.com/tianocore/edk2/blob/ … beShim.asm
https://github.com/tianocore/edk2/blob/ … VbeShim.sh
https://github.com/tianocore/edk2/blob/ … /VbeShim.c
https://github.com/tianocore/edk2/blob/ … /VbeShim.h

Summary: the UEFI GOP that OVMF provides for QXL has a nice list of resolutions, dependent on QXL vram size, and it is available to *actually* UEFI compliant OSes, like many Linux versions and Windows 8 and later. Windows 7 is not such an operating system, it cannot use the GOP, it insists on legacy video BIOS services, hence OVMF fakes the absolute minimum of legacy video BIOS services just for Windows 7's sake, even if you build OVMF without a CSM. This "fake" is called the Int10h shim, and the resolution limitation pertains only to the Int10h shim, not the UEFI GOP.

Offline

#5303 2015-06-08 14:24:18

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@lersek: Well, that clears up a lot...
And how do i change the display resolution initiated by OVMF? Some EFI shell command, i guess?
I just remember the time i've tried to install windows8 - it inherited that 1024x768 mode from OVMF, and there was a huge fail, because it couldn't raise the resolution and refused to launch internet exploder due to low resolution, and there was no way of downloading the proper drivers without using some external storage.

And what about... the installation disc? Windows7 seems to work fine while you're installing it(so i guess it inherits the videomode from OVMF, and works with GOP), but when the installation is done and it's time to show that colourful 4-piece logo animation - it hangs. Maybe there are multiple videodrivers, and we can swap them?

...hmm, it's possible to load the EFI driver from a fat32 partition.. I guess the file should be .efi, and i must load it manually from the UEFI-shell.. Since i have a modified ROM for my asus cards which has an almost 3rd party EFI-driver embedded in it, maybe i wouldn't need it.. Still haven't flashed it physically, heh.

Last edited by Duelist (2015-06-08 14:34:07)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5304 2015-06-08 18:10:03

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Many thanks to Nesousx first for his awesome fix.
I was able to successfully boot windows today, with networking, usb keyboard and mouse. I installed the Nvidia 970gtx drivers and have some white lines running across the screen. Anything I can do to fix this?

Offline

#5305 2015-06-08 20:32:14

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

@lersek: Well, that clears up a lot...
And how do i change the display resolution initiated by OVMF? Some EFI shell command, i guess?

IIRC this has been discussed before. It's certainly described in the OVMF whitepaper, subsection "Platform Driver". Press ESC at the TianoCore splash screen, Navigate to Device Manager | OVMF Platform Configuration, select the preferred resolution from the drop-down list, commit changes, then reboot. (You can reboot by: (a) forcing it from the qemu monitor or virt-manager, (b), going to the EFI shell and issuing "reset -c", (c) going to Boot Maintenance Manager and selecting Reset System.)

I just remember the time i've tried to install windows8 - it inherited that 1024x768 mode from OVMF, and there was a huge fail, because it couldn't raise the resolution and refused to launch internet exploder due to low resolution, and there was no way of downloading the proper drivers without using some external storage.

Correct. That's exactly the case when you change the resolution in the firmware as I described at the top.

And what about... the installation disc? Windows7 seems to work fine while you're installing it (so i guess it inherits the videomode from OVMF, and works with GOP)

As long as the UEFI boot loader of Windows 7 (which is a UEFI application) is running, it uses the GOP natively. It could even switch resolutions if it wanted to.

but when the installation is done and it's time to show that colourful 4-piece logo animation - it hangs.

That's when the handover to the runtime OS happens, and the builtin kernel mode video driver takes control. It does not hang (it should not, anyway).

Instead, on a BIOS machine (or a UEFI+CSM machine), you'd see a 640x480, 16 color, planar VGA mode. (The most basic VGA mode.) With some messages.

This planar VGA mode was way too complex to implement in the Int10h shim, so we return "success" for the request, but do nothing. (Search the ASM file linked earlier for "SetModeLegacy".)

Two of us tried to implement this mode, but it's not really possible, because Windows 7 massages the legacy VGA CRTC registers manually, and seemed to interfere with whatever we tried to do. So, you get a blank screen, but it does not last long, and these screens are not interactive. At some point the builtin kernel mode video driver decides to look for VBE modes (better resolution, true color), and then it finds the "fake" 1024x768x32 mode, and asks the shim to switch to it. Which we do, and then you get the first green (or blue) screen (dependent on Windows Server 2008 R2 vs. Windows 7), with the mouse etc.

It might take a few tens of seconds for this screen to show up, especially if you have the guest on a spindle disk.

Offline

#5306 2015-06-09 00:48:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

As long as the UEFI boot loader of Windows 7 (which is a UEFI application) is running, it uses the GOP natively. It could even switch resolutions if it wanted to.

lersek wrote:

but when the installation is done and it's time to show that colourful 4-piece logo animation - it hangs.

That's when the handover to the runtime OS happens, and the builtin kernel mode video driver takes control. It does not hang (it should not, anyway).

So there's two possible ways of "patching" windows 7:
1. Changing the bootloader to something else. As i feel it(yep, i have a strange feeling not based on any research) the windows setup and the windows core don't differ much, and it may be possible to swap them. Also, maybe there's some weird third-party-influenced way of loading the windows, but that's way beyond our scope... and win10 is coming, so win7 is becoming obsolete in some years.
2. Changing the "builtin kernel mode video driver", as you've called it. That should be somewhat more simple, as i remember some OEM reinstall images for certain notebooks which had a system image with GPU drivers incorporated into them.

Any suggestions? Maybe i'll take a look on that, but i'm just somewhat busy with the life, books and installing gentoo into a prefix to install gentoo on rasp.pi...


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5307 2015-06-10 17:13:33

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi. I've finally changed my guest GPU - temporarily to GTX 960, from Radeon 4850. Coming from Radeon I knew that it wouldn't work without problems - but I can't overcome them at this moment. I got dreaded 'Code 43' on Nvidia GPU. I've disabled hyperv extensions and set kvm hidden state to off. I'm now checking if kernel update to linux-vfio could help with anything, but it would be great if you could give me some tips.

I use Nvidia 9800 GTX+ with nvidia blob for host, have i3-550 CPU and AsRock Z77 Extreme4. All kvm-related things enabled (as I was running it properly with Radeon working on machine). I have 8 GiB total and 3 passed for machine.

960 GPU plus it's audio adapter are in pci-stubs and added to machine config.

<domain type='kvm'> <name>wintendo</name> <uuid>bf6bda0e-2664-4bab-998a-aac4dc255c83</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>3</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='3' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pcid'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='smx'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='tm2'/> <feature policy='require' name='ss'/> <feature policy='require' name='pbe'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/50.raw'/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/d2.img'/> <target dev='vde' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:17:65:a1'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x0719'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

Last edited by dRaiser (2015-06-10 18:11:50)

Offline

#5308 2015-06-10 18:31:10

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Hi. I've finally changed my guest GPU - temporarily to GTX 960, from Radeon 4850. Coming from Radeon I knew that it wouldn't work without problems - but I can't overcome them at this moment. I got dreaded 'Code 43' on Nvidia GPU. I've disabled hyperv extensions and set kvm hidden state to off. I'm now checking if kernel update to linux-vfio could help with anything, but it would be great if you could give me some tips.

I use Nvidia 9800 GTX+ with nvidia blob for host, have i3-550 CPU and AsRock Z77 Extreme4. All kvm-related things enabled (as I was running it properly with Radeon working on machine). I have 8 GiB total and 3 passed for machine.

960 GPU plus it's audio adapter are in pci-stubs and added to machine config.

<domain type='kvm'> <name>wintendo</name> <uuid>bf6bda0e-2664-4bab-998a-aac4dc255c83</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>3</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='3' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pcid'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='smx'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='tm2'/> <feature policy='require' name='ss'/> <feature policy='require' name='pbe'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/50.raw'/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/d2.img'/> <target dev='vde' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:17:65:a1'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x0719'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

Try force-disabling hyper-V enlightenments, maybe they're enabled by default now. That would be strange, however.

Also, you could try swapping the host and the guest cards(to see if it's not the host nvidia-guest nvidia related conflict, but a card-related problem), and maybe try using nouveau with 9800GTX+(aka GTS250 btw, you can reflash it that way for lulz)
Oh, and i think you should use host-passthrough CPU for the true cpu id being shown for GeForce Experience App to work right. That'll cut all those feature requests stuff you have there now.
Also you have Cirrus VGA enabled, ensure that you really need it(QXL is recommended for Windows7, not cirrus).

Last edited by Duelist (2015-06-10 18:34:48)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5309 2015-06-10 18:42:44

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
dRaiser wrote:

Hi. I've finally changed my guest GPU - temporarily to GTX 960, from Radeon 4850. Coming from Radeon I knew that it wouldn't work without problems - but I can't overcome them at this moment. I got dreaded 'Code 43' on Nvidia GPU. I've disabled hyperv extensions and set kvm hidden state to off. I'm now checking if kernel update to linux-vfio could help with anything, but it would be great if you could give me some tips.

I use Nvidia 9800 GTX+ with nvidia blob for host, have i3-550 CPU and AsRock Z77 Extreme4. All kvm-related things enabled (as I was running it properly with Radeon working on machine). I have 8 GiB total and 3 passed for machine.

960 GPU plus it's audio adapter are in pci-stubs and added to machine config.

<domain type='kvm'> <name>wintendo</name> <uuid>bf6bda0e-2664-4bab-998a-aac4dc255c83</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>3</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <topology sockets='1' cores='3' threads='1'/> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pcid'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='smx'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='tm2'/> <feature policy='require' name='ss'/> <feature policy='require' name='pbe'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/50.raw'/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/d2.img'/> <target dev='vde' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:17:65:a1'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x045e'/> <product id='0x0719'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

Try force-disabling hyper-V enlightenments, maybe they're enabled by default now. That would be strange, however.

I've tried both versions (completely deleted and off) without luck.

Also, you could try swapping the host and the guest cards(to see if it's not the host nvidia-guest nvidia related conflict, but a card-related problem), and maybe try using nouveau with 9800GTX+(aka GTS250 btw, you can reflash it that way for lulz)

Oh, and i think you should use host-passthrough CPU for the true cpu id being shown for GeForce Experience App to work right. That'll cut all those feature requests stuff you have there now.

Thanks, I didn't know that. However it won't help resolve main issue, am I right?

Also you have Cirrus VGA enabled, ensure that you really need it(QXL is recommended for Windows7, not cirrus).

I've just switched between both without any difference. It's feasible for testing and didn't caused any problems before, but I'll try disabling completely.

Last edited by dRaiser (2015-06-10 18:43:30)

Offline

#5310 2015-06-10 18:59:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dRaiser

Emulated primary graphics + secondary nvidia is not how GeForce works, nvidia should be your only graphics.  The mode you're trying works fine for Quadro.  Also, since you have host and guest nvidia with the host nvidia proprietary drivers, double check that you're successfully preventing the host nvidia driver from binding to resources on the guest card.  Nvidia does not support unbinding their driver from the device and is known to not release resources.  There's often dmesg errors when launching the vm if this happens.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5311 2015-06-10 19:44:48

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@dRaiser

Emulated primary graphics + secondary nvidia is not how GeForce works, nvidia should be your only graphics.  The mode you're trying works fine for Quadro.  Also, since you have host and guest nvidia with the host nvidia proprietary drivers, double check that you're successfully preventing the host nvidia driver from binding to resources on the guest card.  Nvidia does not support unbinding their driver from the device and is known to not release resources.  There's often dmesg errors when launching the vm if this happens.


Ok, I've removed emulated graphics, but now can't see anything at all (so I don't know any windows errors and such).

I have pci-stubs in kernel line and vfio-bind configured. No dmesg errors after starting up machine, but it's not continuing booting properly, so I suspect it can't use card.

From lspci --nk I see this lines:

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3201] Kernel driver in use: vfio-pci Kernel modules: nouveau, nvidia

I think it's correct, but I'll ask for confirmation anyway.

Quick test:

sudo qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -vga none

also works properly. So I think that 960 should be accessible for VM. Maybe I'll struggle and try run new Windows install...

Edit:

.. but it won't work when machines don't get video output at all. I wonder what's wrong. When I have emulated video enabled, then 960's discovered correctly (I've installed driver this way without problem).
I've switched host driver to nouveau and to newer kelner. Nothing changed.

Last edited by dRaiser (2015-06-10 21:51:16)

Offline

#5312 2015-06-10 22:53:26

supplicant0x90
Member
Registered: 2015-06-10
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[aw] et al -

Your guides have been hugely valuable to me. I can't thank you enough for taking the time to share your research with us.

I've gotten everything else working through those (and days of doc review), but I can't figure out how get my (Windows 8.1) guest to instantiate with a vmx flag on the vcpu (to be clear, I mean for nested virtualization; the host already has it enabled and working fine).

How do you enable VT-x in the guest through virt-manager or virsh? I tried doing

host-passthrough

on the cpu, since the host supports it, but the vmx flag doesn't come through. I also tried passing a

<feature policy='force' name='vmx'>

argument in the cpu settings of the config using "virsh edit $vm" (along with other options to confirm it works), and some of the other cpu flags do pop as expected, but vmx does not.

Is there some special setting I've overlooked? I see 'man qemu' says you can call it using 'iommu=on|off' as a qemu-system-x86_64 argument, but I can't find anything else in the libvirt, virsh, or virt-manager docs.

Does it matter whether the guest uses BIOS or UEFI?

Y4Dq.png

Apps on the guest confirm it's not a display issue:

Y4Dw.png

Offline

#5313 2015-06-10 22:54:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

.. but it won't work when machines don't get video output at all. I wonder what's wrong. When I have emulated video enabled, then 960's discovered correctly (I've installed driver this way without problem).
I've switched host driver to nouveau and to newer kelner. Nothing changed.

That's sad, i wanted to suggest you to grep vgaarb in dmesg to see if there's any problems with that.

And..
You're not using OVMF, are you?
AW descriped some qemu wrapper script to append x-vga option to the device bound to vfio-pci and used by libvirt. libvirt will never allow x-vga to be on. So you'll need a wrapper script.

Or you could use ovmf to avoid VGA problems at all. You don't need to have any support of UEFI in your hardware except the GPU, and your 9XX geforce surely does support UEFI.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5314 2015-06-10 22:57:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Quick test:

sudo qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -vga none

also works properly. So I think that 960 should be accessible for VM. Maybe I'll struggle and try run new Windows install...

Edit:

.. but it won't work when machines don't get video output at all. I wonder what's wrong. When I have emulated video enabled, then 960's discovered correctly (I've installed driver this way without problem).
I've switched host driver to nouveau and to newer kelner. Nothing changed.

You're using x-vga=on here, but making no attempt to do so in your xml.  See part 5 of my how-to for a wrapper script around your qemu binary to add x-vga.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5315 2015-06-10 22:59:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supplicant0x90 wrote:

I've gotten everything else working through those (and days of doc review), but I can't figure out how get my (Windows 8.1) guest to instantiate with a vmx flag on the vcpu (to be clear, I mean for nested virtualization; the host already has it enabled and working fine).

Are you loading kvm-intel with nested=1?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5316 2015-06-10 22:59:53

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supplicant0x90 wrote:

How do you enable VT-x in the guest through virt-manager or virsh? I tried doing

host-passthrough

on the cpu, since the host supports it, but the vmx flag doesn't come through. I also tried passing a

<feature policy='force' name='vmx'>

argument in the cpu settings of the config using "virsh edit $vm" (along with other options to confirm it works), and some of the other cpu flags do pop as expected, but vmx does not.

I don't know about intel, but on AMD we have

options kvm-amd npt=1 nested=0

module options.

[user@crossfire ~]$ modinfo kvm-intel filename: /lib/modules/4.0.4-202.fc21.x86_64/kernel/arch/x86/kvm/kvm-intel.ko.xz license: GPL author: Qumranet alias: cpu:type:x86,ven*fam*mod*:feature:*0085* depends: kvm intree: Y vermagic: 4.0.4-202.fc21.x86_64 SMP mod_unload signer: Fedora kernel signing key sig_key: EC:B8:61:B3:8D:2D:A2:6D:43:B0:15:C1:0B:16:34:C7:33:4F:DD:D3 sig_hashalgo: sha256 parm: vpid:bool parm: flexpriority:bool parm: ept:bool parm: unrestricted_guest:bool parm: eptad:bool parm: emulate_invalid_guest_state:bool parm: vmm_exclusive:bool parm: fasteoi:bool parm: enable_apicv:bool parm: enable_shadow_vmcs:bool parm: nested:bool parm: pml:bool parm: ple_gap:int parm: ple_window:int parm: ple_window_grow:int parm: ple_window_shrink:int parm: ple_window_max:int

As you can see, you have a module option called nested too! ensure that you've enabled that.


@aw, you should put a notice somewhere on your blog saying that you're the one(almost?) who wrote vfio-pci, i have a feeling people massively miss that point.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5317 2015-06-10 23:00:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, this is getting awkward. Second-in-second. Dude. That's insane.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5318 2015-06-10 23:04:09

supplicant0x90
Member
Registered: 2015-06-10
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw/Duelist:

Are you loading kvm-intel with nested=1?

No. I didn't see that anywhere. I'll research it from here and report back. Thank you!

Last edited by supplicant0x90 (2015-06-10 23:05:19)

Offline

#5319 2015-06-11 01:06:52

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, aw and Duelist.

I couldn't utilize aw's wrapper as virsh edit shouts about wrong binary, so I've made simple check with qemu-commandline parameters (I know, it's ugly, but that was just to figure out..)

<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,x-vga=on'/> </qemu:commandline>

I could launch machine with it, but no deal. Still black screen and windows doesn't even boot (I can't hear startup). So I assume x-vga doesn't actually help.

I was going to try OVMF as Duelist suggested, but it won't boot to my prepared earlier disk... Do I need to redo install if I want to use it? Sigh. If we can't find any other cure, I'll probably try with new OVMF install then...

--

One more patient test later: after few minutes of waiting Windows loads, I've got startup sound and *device connected* sounds, but no screen image. So near, so far...

--

Success! Worked when added also "multifunction" to qemu arg line. Tommorow I'll post whole xml again for future. smile

Last edited by dRaiser (2015-06-11 01:49:55)

Offline

#5320 2015-06-11 01:54:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Thanks, aw and Duelist.

I couldn't utilize aw's wrapper as virsh edit shouts about wrong binary, so I've made simple check with qemu-commandline parameters (I know, it's ugly, but that was just to figure out..)

<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,x-vga=on'/> </qemu:commandline>

NNNNNNNNNNNOOOOOOOOOOOOOO!!!!!!!!!!!!!!!!!

EDIT: did you chmod the wrapper to make it executable?

Last edited by aw (2015-06-11 01:56:39)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5321 2015-06-11 02:00:41

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
dRaiser wrote:

Thanks, aw and Duelist.

I couldn't utilize aw's wrapper as virsh edit shouts about wrong binary, so I've made simple check with qemu-commandline parameters (I know, it's ugly, but that was just to figure out..)

<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,x-vga=on'/> </qemu:commandline>

NNNNNNNNNNNOOOOOOOOOOOOOO!!!!!!!!!!!!!!!!!

EDIT: did you chmod the wrapper to make it executable?

Sorry sad ...but it works...

Yes, I did chmod. I've also corrected path to qemu in its content (as it wasn't valid location for qemu on my system), I was able to run it by hand by console, but virsh validation couldn't accept it as correct binary. I'll post specific error later if you'd like.

Offline

#5322 2015-06-11 03:34:07

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just got around to trying out the Hawaii bus reset patch -- it works wonderfully.  Huge thanks for getting that in!
Now I can finally do what I set out to do when building this system: consolidate multiple systems into one. 

I had to disable AppArmor in libvirt, though.  The AppArmor "helper" suddenly decided to act completely un-helpful.
It complained that it was "unable to load profile" ("no such file or directory"), without telling me WHAT "file or directory" it can't find. 
It also couldn't seem to comprehend the split images, and denied write access to the "vars" file.  (Right now, I'm back to using a single image.)

If you pass through a raw disk via virtio, do TRIM commands get passed through, as well?
I found an article about hole-punching via TRIM on virtio-scsi with file-backed drives, but I'd be using virtio and a raw disk.

For the moment, I'm giving the virtual machine the ASMedia SATA controller, and using a hack to make it bootable..
I have a small IDE boot partition with rEFInd installed, so rEFInd can load AHCI.efi (a driver packaged with the "XPC" bootloader).

Something else I've noticed: if I try to use my Xonar U7 (USB Audio Class 2.0) on the passed-through XHCI controller, the entire audio stack hangs after a while.
I have to unplug the sound card and plug it back in again.  For now, I'm passing through the onboard Realtek audio and using the USB sound card on the host.

Last edited by DanaGoyette (2015-06-11 03:37:49)

Offline

#5323 2015-06-11 08:54:29

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DanaGoyette wrote:

Something else I've noticed: if I try to use my Xonar U7 (USB Audio Class 2.0) on the passed-through XHCI controller, the entire audio stack hangs after a while.
I have to unplug the sound card and plug it back in again.  For now, I'm passing through the onboard Realtek audio and using the USB sound card on the host.

Hi,

Just so you know I had issues for months with the Xonar U7. IIRC I also had issues with this card I posted about in this thread, but can't really remember what. Finally, the "clicking" noise of the U7 when you plug / unplug it made me crazy.

I sold it and bought a cheap USB card, everything works as expected. A part from the fact that I have to "lock" the USB card in the host.. but this is udev related as far as understand it, and will happen with any USB card.

Edit : page 138, post #3437. Now I remember my issue. Sometimes, totally randomly the sound will stutter and make everything lags for a 1 sec or 2 when the card was used via USB. Then when I emulated the audio while using USB on host I always had a very small delay before sounds occurs. This delay didn't exist when I emulated sound via the onboard (or others USB) cards.

Last edited by Nesousx (2015-06-11 09:00:47)

Offline

#5324 2015-06-11 08:55:46

mqddb
Member
Registered: 2015-04-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear ALL:
     intel IGD is not supported YET. But, I asked software engineer of KVMGT project, they say intel IGD can be passthrough to VM. I want know why intel IGD can't supported, Is this problem can be fix?

Offline

#5325 2015-06-11 13:16:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mqddb wrote:

Dear ALL:
     intel IGD is not supported YET. But, I asked software engineer of KVMGT project, they say intel IGD can be passthrough to VM. I want know why intel IGD can't supported, Is this problem can be fix?

You can assign it, but it won't work until Intel releases a driver that doesn't depend on passing through random op-regions, registers on other devices, and modifying device IDs on other components in the VM.  IGD is not confined to the device at 00:02.0, it's smeared across the chipset.  Ask them when that driver is coming.  Even then, I believe the current plan requires some modifications to the guest BIOS.  It's also not clear if that driver will require VGA access and therefore permanently lock VGA arbitration for the rest of the system or support UEFI.  Even if it were to support UEFI, I don't think current Intel IGD hardware can fully disable claiming VGA transactions without disabling parts of the device, which might still require permanently locking arbitration for the duration of the VM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5326 2015-06-12 02:59:05

PrinzipDesSees
Member
Registered: 2015-06-04
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another thing:

My host has no X installed.
Can I pass the output of a virtual machine to a second display?  As in completely not passing through any GPU but just having a virtual GPU throw the image on monitor 2 without installing X.

Offline

#5327 2015-06-12 03:19:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PrinzipDesSees wrote:

Another thing:

My host has no X installed.
Can I pass the output of a virtual machine to a second display?  As in completely not passing through any GPU but just having a virtual GPU throw the image on monitor 2 without installing X.

You want to make QXL output on some hardware screen, right?

You could make the host headless(without X or any other video output) by passing through your main GPU inside some VM and having no other GPU left for host.

Last edited by Duelist (2015-06-12 03:20:15)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5328 2015-06-12 09:25:03

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I installed ovmf-svn from aur and now have the following files:

[michael@stingray ~]$ find /usr/share/ovmf/ /usr/share/ovmf/ /usr/share/ovmf/ia32 /usr/share/ovmf/ia32/ovmf_vars_ia32.bin /usr/share/ovmf/ia32/ovmf_code_ia32.bin /usr/share/ovmf/ia32/ovmf_ia32.bin /usr/share/ovmf/x64 /usr/share/ovmf/x64/ovmf_vars_x64.bin /usr/share/ovmf/x64/ovmf_code_x64.bin /usr/share/ovmf/x64/ovmf_x64.bin

Trying to set up a new VM with virt-manager complains that libvirt did not detect any UEFI Bios, so i can't switch the firmware to UEFI. For which files is libvirt/virtmanager looking? Probably for some *.fd instead of .bin?

Offline

#5329 2015-06-12 09:55:05

PrinzipDesSees
Member
Registered: 2015-06-04
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You want to make QXL output on some hardware screen, right?

You could make the host headless(without X or any other video output) by passing through your main GPU inside some VM and having no other GPU left for host.


Yes.

That seems excessive, cant I make it use just the monitor?

Offline

#5330 2015-06-12 11:44:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[facepalm]

EDIT: QEMU does support SDL and SDL does not necessarily require X.  But, if you're not doing VGA/GPU device assignment, it's not relevant to this thread.

Last edited by aw (2015-06-12 14:09:59)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5331 2015-06-13 17:47:43

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Vga passthrough working with gtx 970 on sr-2 classified and dual xeon 5680s. The only issue I have noticed is a freeze on first boot of a new  game. I reboot qemu / wondows and the issue is gone and game loads. Any ideas? Also sound stutters slightly.

Thanks in advance.

Offline

#5332 2015-06-13 20:06:46

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Hi,

I installed ovmf-svn from aur and now have the following files:

[michael@stingray ~]$ find /usr/share/ovmf/ /usr/share/ovmf/ /usr/share/ovmf/ia32 /usr/share/ovmf/ia32/ovmf_vars_ia32.bin /usr/share/ovmf/ia32/ovmf_code_ia32.bin /usr/share/ovmf/ia32/ovmf_ia32.bin /usr/share/ovmf/x64 /usr/share/ovmf/x64/ovmf_vars_x64.bin /usr/share/ovmf/x64/ovmf_code_x64.bin /usr/share/ovmf/x64/ovmf_x64.bin

Trying to set up a new VM with virt-manager complains that libvirt did not detect any UEFI Bios, so i can't switch the firmware to UEFI. For which files is libvirt/virtmanager looking? Probably for some *.fd instead of .bin?

The old way

The right way:
/etc/libvirt/qemu.conf
# Location of master nvram file
#
# When a domain is configured to use UEFI instead of standard
# BIOS it may use a separate storage for UEFI variables. If
# that's the case libvirt creates the variable store per domain
# using this master file as image. Each UEFI firmware can,
# however, have different variables store. Therefore the nvram is
# a list of strings when a single item is in form of:
#   ${PATH_TO_UEFI_FW}:${PATH_TO_UEFI_VARS}.
# Later, when libvirt creates per domain variable store, this
# list is searched for the master image.
nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

BUT! As noted in aw's blog:

If UEFI is not available, your libvirt and virt manager tools may be too old.

So the above lines are needed only for old versions.


supertrampx wrote:

Vga passthrough working with gtx 970 on sr-2 classified and dual xeon 5680s. The only issue I have noticed is a freeze on first boot of a new  game. I reboot qemu / wondows and the issue is gone and game loads. Any ideas? Also sound stutters slightly.

Thanks in advance.

Maybe your "game" tries to detect your GPU and as we know - all "auto-detecting" software like driver-detect stuff generate BSODs or some other weird behaviour. A strange example is GeForce Experience not allowing you to capture video if you don't have a "real" CPUID.
A workaround would be to use host-passthrough CPU model and ignore_msrs=1 in kvm module options for some weird software to work right.
You can see CPU-z query the CPU by doing rdmsr calls and dmesg tells you that that msrs are unknown.
Another reason for that would be... Maan, dual xeon? Socket 771, am i right? I'd crash the first time i'd see that system, not just a game. In other words - you have a very, very, very WEIRD config from VM's view.

@aw, does your face hurt?.. If so - what was there before the edit?..

Last edited by Duelist (2015-06-13 20:11:30)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5333 2015-06-13 20:31:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

Vga passthrough working with gtx 970 on sr-2 classified and dual xeon 5680s. The only issue I have noticed is a freeze on first boot of a new  game. I reboot qemu / wondows and the issue is gone and game loads. Any ideas? Also sound stutters slightly.

Thanks in advance.

http://vfio.blogspot.com/2014/09/vfio-i … ndows.html


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5334 2015-06-14 08:25:58

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Awesome thanks guys for the leads. tongue yes this setup is weird and fast "wink", and to make it even more fun, my pci-e ports are behind an Nvidia NF200 controller.
"Socket 771, am i right?"  actually socket : 1366 tongue
So from what I understand, specify a cpu in my vm file so that special software calls wont detect that its in a vm and cause a BSOD?
I am unsure of host-passthrough cpu model and what you mean.


Current VM Config
----------------------------------------------------------------------------------------------------------------------------------------
#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

#Startup Windows 7 Guest with VGA-Passthrough
qemu-system-x86_64 -name win7 -enable-kvm -m 12000 -cpu host,kvm=off \
-smp 6,sockets=1,cores=4,threads=2 \
-rtc base=localtime -realtime mlock=off \
-bios /home/coyote/seabios/out/bios.bin \
-device vfio-pci,host=04:00.0,x-vga=on -device vfio-pci,host=04:00.1 \
-usb -usbdevice host:046d:c068 -usbdevice host:046d:c22d -usbdevice host:047f:c010 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/coyote/win7.img,id=win,format=raw,if=none -device scsi-hd,drive=win \
-drive file=/home/coyote/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/coyote/windows7.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-vga none
exit 0

----------------------------------------------------------------------------------------------------------------------------------------------
Info on my machine

Motherboard : http://www.evga.com/articles/00537/
Processors : http://ark.intel.com/products/47916/Int … -Intel-QPI
RAM : 24 GB ecc 1600 "would be using 48 but if you fill all slots, the ram is forced to run at 1300"
Graphics : http://us.msi.com/product/vga/GTX-970-G … o-overview "run this card in passthrough to windows kvm"
Graphics Host : An old GTX 260 with Nouveau drivers
HD : PNY SSD 250 GB and Crucial c300 250GB
6x  1TB Hard drives for backup in hot swap bays
1x Dell 30 inch ultra sharp for windows
1x Samsung 24 inch HD monitor Linux
OS openSUSE Tumbleweed 4.0.4-3
OS passthrough Windows 7

Last edited by supertrampx (2015-06-14 08:41:52)

Offline

#5335 2015-06-14 08:27:06

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If I can get this running, I may try with 4 graphics cards, as I have the slots for it.

I am also getting some warnings when I run the vm

WARNING: Image format was not specified for '/home/coyote/virtio.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

WARNING: Image format was not specified for '/home/coyote/windows7.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

Last edited by supertrampx (2015-06-14 08:38:02)

Offline

#5336 2015-06-14 08:31:27

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you wish here are some project photos tongue Little bragging here sorry smile But I am proud and it has been a rocking project!

http://s560.photobucket.com/user/joecay … ibrary/sr2

Offline

#5337 2015-06-14 09:05:55

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to this thread, for making this possible smile

Here is my project:
6 hosts in lan for gaming
gigabyte z77, i7 2600, 20gb ram, 120gb ssd, 3tb sata, 2x gtx680 = 12 players/seats
Debian jessie with 3.19 kernel, custom win7 for guests. Running steam instead explorer, with everything locked down hard.

https://www.youtube.com/watch?v=YoLLHKzXbC8

It's been running for 6 months, No problems so far, everything is remote managed. And hosts happen to be on 10user 100mbps local wifi network, so they all can remote play over steam in-home streaming.

Offline

#5338 2015-06-14 12:16:29

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
TripleSpeeder wrote:

Hi,

I installed ovmf-svn from aur and now have the following files:

[michael@stingray ~]$ find /usr/share/ovmf/ /usr/share/ovmf/ /usr/share/ovmf/ia32 /usr/share/ovmf/ia32/ovmf_vars_ia32.bin /usr/share/ovmf/ia32/ovmf_code_ia32.bin /usr/share/ovmf/ia32/ovmf_ia32.bin /usr/share/ovmf/x64 /usr/share/ovmf/x64/ovmf_vars_x64.bin /usr/share/ovmf/x64/ovmf_code_x64.bin /usr/share/ovmf/x64/ovmf_x64.bin

Trying to set up a new VM with virt-manager complains that libvirt did not detect any UEFI Bios, so i can't switch the firmware to UEFI. For which files is libvirt/virtmanager looking? Probably for some *.fd instead of .bin?

The right way:
/etc/libvirt/qemu.conf
# Location of master nvram file
#
# When a domain is configured to use UEFI instead of standard
# BIOS it may use a separate storage for UEFI variables. If
# that's the case libvirt creates the variable store per domain
# using this master file as image. Each UEFI firmware can,
# however, have different variables store. Therefore the nvram is
# a list of strings when a single item is in form of:
#   ${PATH_TO_UEFI_FW}:${PATH_TO_UEFI_VARS}.
# Later, when libvirt creates per domain variable store, this
# list is searched for the master image.
nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

Hmm, but these are .fd files, while ovmf-svn builds only .bin files as shown above. I suppose they're not interchangeable, or are they?

Offline

#5339 2015-06-14 14:19:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

If I can get this running, I may try with 4 graphics cards, as I have the slots for it.

I am also getting some warnings when I run the vm

WARNING: Image format was not specified for '/home/coyote/virtio.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

WARNING: Image format was not specified for '/home/coyote/windows7.iso' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.

The warnings tell that you should specify the format of your images.

-drive file=/home/coyote/win7.img,id=win,format=raw,if=none -device scsi-hd,drive=win \ -drive file=/home/coyote/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/home/coyote/windows7.iso,id=isocd,if=none -device scsi-cd,drive=isocd

You've only specified the format for win7.img. So it should be like this:

-drive file=/home/coyote/virtio.iso,id=virtiocd,,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -drive file=/home/coyote/windows7.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd

And you're using cpu host model already, so try following aw's advice and make everything use MSI(interrupts, not the vendor).

MultiGPU setup... SLI on nvidia hasn't been observed working yet. I've seen(and using) only CrossFire multiGPU configs. Maybe slis have a different opinion on this.
I've suggested one guy to use HyperSLI to trick nvidia's driver, but it didn't work.


slis wrote:

Thanks to this thread, for making this possible smile

Here is my project:
6 hosts in lan for gaming
gigabyte z77, i7 2600, 20gb ram, 120gb ssd, 3tb sata, 2x gtx680 = 12 players/seats
Debian jessie with 3.19 kernel, custom win7 for guests. Running steam instead explorer, with everything locked down hard.

https://www.youtube.com/watch?v=YoLLHKzXbC8

It's been running for 6 months, No problems so far, everything is remote managed. And hosts happen to be on 10user 100mbps local wifi network, so they all can remote play over steam in-home streaming.

Yeah, that's the right usecase for that tech. But you have two 680s... How does it work? One gpu per VM? Quadrified them and split into many VMs? Or what?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5340 2015-06-14 16:22:40

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah it's one gpu per vm, splitting doesn't work, Quadrified (to grid k2) them just for hyper-v and latest driver.

Offline

#5341 2015-06-14 18:32:33

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Yeah it's one gpu per vm, splitting doesn't work, Quadrified (to grid k2) them just for hyper-v and latest driver.

So how does it gets 12 players?..


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5342 2015-06-14 22:25:12

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
slis wrote:

Yeah it's one gpu per vm, splitting doesn't work, Quadrified (to grid k2) them just for hyper-v and latest driver.

So how does it gets 12 players?..

6 hosts with 2x 680, 2vms per host, one gpu each smile

Last edited by slis (2015-06-14 22:26:42)

Offline

#5343 2015-06-14 23:34:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
Duelist wrote:
slis wrote:

Yeah it's one gpu per vm, splitting doesn't work, Quadrified (to grid k2) them just for hyper-v and latest driver.

So how does it gets 12 players?..

6 hosts with 2x 680, 2vms per host, one gpu each smile

Aww, come on! That's not so fun. But fun anyway.

Why it doesn't split? It'd be extra awesome to do four VMs per one GPU, making 8 guests out of one host.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5344 2015-06-14 23:36:16

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah it would be nice, but I tried with citrix xen and it wouldn't work, passing worked.

Offline

#5345 2015-06-15 09:51:52

mqddb
Member
Registered: 2015-04-24
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Punkbob wrote:
aw wrote:
aw wrote:

8.1, AIUI 10 has numerous issues running as a VM

@Punkbob

What build of Windows 10 are you using and did you need to do anything special to get it installed?  I see one report that installing build 10041 requires at least one hyper-v extension to be enabled, which isn't going to sit well with Nvidia drivers.

I am using build 9926 on my most recent VM, and I didn't change anything or enable anything different then I would with a normal Windows 10 install. When I first switched over to Windows 10 a few months ago it initially didn't have any specific display drivers from Nvidia, so I just used the ones from Windows 8.1. However once I try drivers that are specifically released for Windows 10, the screen goes black, and when I reboot it just spits the code 43 error back at me.

Edit: Let me try a later build real quick, I thought I was using one that was more recent, but I think I confused which iso I grabbed off my NAS.

Hi All:
    Is there an method to solve this code 43 issue? I also face this boring problem.

Offline

#5346 2015-06-15 11:11:54

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For me Windows10_InsiderPreview_x64_EN-US_10074.iso, is working fine with 352.84 on gtx680.

Offline

#5347 2015-06-15 11:39:18

WildyLion
Member
Registered: 2015-06-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

Could anybody advise me on how to solve subtle timing issues with a 8.1 guest?
When I'm playing music in foobar, the EQ would stop for a second and the sound will stutter, and when I'm playing games, I still get sudden FPS dropouts.

My hardware
* FX-8350 CPU
* M5A99FX PRO 2.0 M/B
* Sapphire Vapor-X Radeon R9 290 as passthrough GPU (VFIO mode)
* USB controller with k/b, mouse and Xonar U7 passed to the VM via PCI passthrough
* I switch k/b and mouse between the motherboard USB ports with an ATEN CS22U KVM

What I've already done:

* Using i440FX vChipset at the moment (with OVMF)
* kvm-amd nested pages disabled
* Hyper-V extensions enabled for the VM at install time
* Host CPU (FX-8350) cpufreq set to performance
* Hugepages memory backing enabled, with hugetlbfs
* 6 VCPUs pinned to 6 real cores (0-5 respectively)

Offline

#5348 2015-06-15 16:53:22

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi smile

So my X99-Deluxe exploded out of sudden and took my 5930k with it ! sad

Been a month and I finally replaced it with a X99-E WS + 5820k .

Everything is working fine , but I couldn't get VFIO to bind to my NVIDIA using modprobe.d . Binding my devices using the vfio-bind script is fine .

Modules are loaded early in initramfs , and "options vfio-pci ids=" are specified correctly . My other devices get bound correctly using this method , only NVIDIA cards don't .

Offline

#5349 2015-06-15 17:28:52

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

If vfio-pci is a module, you need to do something to make sure it gets loaded before the native driver has already claimed the device.  Using the modprobe.d softdep option might be a way to handle this, or you can use the techniques I provide in part 3 of my howto series


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5350 2015-06-15 18:46:57

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Thanks , but I am blacklisting nouveau too . In fact when I issue lspci -k , there is no driver claiming the devices at all . Not the native drivers nor vfio-pci . Changing from universal NVIDIA's PCI IDs to GPU's specific doesn't solve the issue either .

Offline

#5351 2015-06-15 19:59:33

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

What happens when you use /sys/bus/pci/drivers_probe when in that state where lspci -k shows no driver?  ex:  echo 0000:01:00.0 > /sys/bus/pci/drivers_probe

I assume you're actually using a v4.1-rc kernel since the ids= option was only introduced post v4.0, right?  What does dmesg show?  dmesg | grep -i vfio


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5352 2015-06-15 20:40:48

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When using :

echo 0000:09:00.0 > /sys/bus/pci/drivers_probe

Nothing happens , no errors are thrown , no dmesg messages , and there is still no drivers attached to the GPU .

I am using 4.1-rc7 compiled last night .

Thanks smile

Last edited by Denso (2015-06-15 20:41:01)

Offline

#5353 2015-06-15 20:44:27

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

When using :

echo 0000:09:00.0 > /sys/bus/pci/drivers_probe

Nothing happens , no errors are thrown , no dmesg messages , and there is still no drivers attached to the GPU .

I am using 4.1-rc7 compiled last night .

Thanks smile

I was really looking to see if vfio-pci was accepting your ids options via dmesg, not just errors from the above command


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5354 2015-06-15 20:48:19

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg | grep -i vfio :

[Mon Jun 15 23:36:14 2015] VFIO - User Level meta-driver version: 0.3 [Mon Jun 15 23:36:14 2015] vfio_pci: add [8086:8d20[ffff:ffff]] class 0x000000/00000000 [Mon Jun 15 23:36:14 2015] vfio_pci: add [8086:15a0[ffff:ffff]] class 0x000000/00000000 [Mon Jun 15 23:36:14 2015] vfio_pci: add [1912:0015[ffff:ffff]] class 0x000000/00000000

these are all the devices I'm passing through minus the GPU/HDMI Audio (NVIDIA) devices . What is confusing is that they get bound to vfio-pci just fine using the vfio-bind script from OP .

Offline

#5355 2015-06-15 21:21:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

dmesg | grep -i vfio :

[Mon Jun 15 23:36:14 2015] VFIO - User Level meta-driver version: 0.3 [Mon Jun 15 23:36:14 2015] vfio_pci: add [8086:8d20[ffff:ffff]] class 0x000000/00000000 [Mon Jun 15 23:36:14 2015] vfio_pci: add [8086:15a0[ffff:ffff]] class 0x000000/00000000 [Mon Jun 15 23:36:14 2015] vfio_pci: add [1912:0015[ffff:ffff]] class 0x000000/00000000

these are all the devices I'm passing through minus the GPU/HDMI Audio (NVIDIA) devices . What is confusing is that they get bound to vfio-pci just fine using the vfio-bind script from OP .

We're only getting an Intel audio device, Intel NIC, and Renesas USB3 controller via vfio-pci.ids... why would it claim the nvidia devices?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5356 2015-06-15 21:42:31

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

... why would it claim the nvidia devices?

Errr ... Because I told it to do so ? wink

Here is what modprobe.d looks like for me :

options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff,1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,8086:8d20,8086:15a0,1912:0015

You see , it claims all the devices except both of my NVIDIA GPUs .

Last edited by Denso (2015-06-15 21:43:01)

Offline

#5357 2015-06-15 21:50:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
aw wrote:

... why would it claim the nvidia devices?

Errr ... Because I told it to do so ? wink

Here is what modprobe.d looks like for me :

options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff,1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,8086:8d20,8086:15a0,1912:0015

You see , it claims all the devices except both of my NVIDIA GPUs .

Your dmesg says those first 4 entries aren't taking effect.  Do they work by themselves?  What if you add your entries at the beginning?  Others have found that you can use multiple lines.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5358 2015-06-15 22:07:40

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Your dmesg says those first 4 entries aren't taking effect.  Do they work by themselves?  What if you add your entries at the beginning?  Others have found that you can use multiple lines.

GPUs don't get bound unless I use the vfio-bind script on them . Specifying them in modprobe.d has no effect what so ever . I also tried breaking them to multiple lines , but the result is the same :

options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=8086:8d20,8086:15a0,1912:0015

vfio-pci ids= claims all the specified devices successfully except for the NVIDIA devices .

Last edited by Denso (2015-06-15 22:08:22)

Offline

#5359 2015-06-15 22:19:58

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

Are you rebuilding your initramfs between each of these changes?  What shows up in dmesg if you run 'modprobe -r vfio-pci; modprobe vfio-pci' with modprobe.d as in the previous post?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5360 2015-06-16 03:05:26

Myranti
Member
Registered: 2010-03-04
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Up front, I'll admit I haven't kept up to date on this thread since about page 184, but I was curious about something regarding AMD FX CPU series (at least relevant to the octocore models) and how it pertains to IOMMU grouping.

My assumption from what I vaguely remember reading is that those with the FX cpu's don't need to use the ACS override patch and can use an unpatched kernel without having to worry about IOMMU grouping issues. Do I understand this correctly or is my recollection incorrect? Or is there more to it that I need to read up on?

Last edited by Myranti (2015-06-16 03:06:06)

Offline

#5361 2015-06-16 03:56:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There are all sorts of ways to use an unpatched kernel, read my blog for examples.  There's no magic bullet in AMD FX CPUs, other than perhaps the chipsets for them are so old that we've already got quirks for them.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5362 2015-06-16 04:11:49

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

WildyLion wrote:

Hi guys,

Could anybody advise me on how to solve subtle timing issues with a 8.1 guest?
When I'm playing music in foobar, the EQ would stop for a second and the sound will stutter, and when I'm playing games, I still get sudden FPS dropouts.

My hardware
* FX-8350 CPU
* M5A99FX PRO 2.0 M/B
* Sapphire Vapor-X Radeon R9 290 as passthrough GPU (VFIO mode)
* USB controller with k/b, mouse and Xonar U7 passed to the VM via PCI passthrough
* I switch k/b and mouse between the motherboard USB ports with an ATEN CS22U KVM

What I've already done:

* Using i440FX vChipset at the moment (with OVMF)
* kvm-amd nested pages disabled
* Hyper-V extensions enabled for the VM at install time
* Host CPU (FX-8350) cpufreq set to performance
* Hugepages memory backing enabled, with hugetlbfs
* 6 VCPUs pinned to 6 real cores (0-5 respectively)

Do you mean that you're pinning all 6 vCPUs to the cpuset 0-5 or are you pinning each individual vCPU to a pCPU within the set 0-5, ie. vCPU0->pCPU0, vCPU1->pCPU1, etc?  The latter is recommended for optimal locality.  You don't mention if you're using virtio for disk and network within the VM, both are highly recommended and will help to improve overall VM performance.  Either as a test or for very VM specific tuning, you can try using isolcpus= on the host to isolate the pCPU running vCPU from the general scheduler.  I believe running the host with nohz=off has also been suggested.  Using iommu=pt can also avoid IOMMU overhead for host devices and may help marginally.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5363 2015-06-16 17:27:09

WildyLion
Member
Registered: 2015-06-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Do you mean that you're pinning all 6 vCPUs to the cpuset 0-5 or are you pinning each individual vCPU to a pCPU within the set 0-5, ie. vCPU0->pCPU0, vCPU1->pCPU1, etc?  The latter is recommended for optimal locality.  You don't mention if you're using virtio for disk and network within the VM, both are highly recommended and will help to improve overall VM performance.  Either as a test or for very VM specific tuning, you can try using isolcpus= on the host to isolate the pCPU running vCPU from the general scheduler.  I believe running the host with nohz=off has also been suggested.  Using iommu=pt can also avoid IOMMU overhead for host devices and may help marginally.

Hi there.

Turns out these were not timing issues, it's my audio card (Xonar U7) that was stuttering all along.
I was able to achieve perfect sound with using my monitor's audio output (via HDMI). Though I still get minor FPS dropouts in Witcher 3 this way, it's completely playable now.

I'd really appreciate any advice on how to a) either make my U7 work under the hypervisor or b) get PulseAudio sound working with libvirt (my PA is in user mode right now).

As for your questions, yes, I've configured vCPUs pinned to pCPUs and I'm using virtio for everything (though currently my disk image is in qcow2 format, that's slowing things down considerably with NTFS on top of it).

Offline

#5364 2015-06-17 01:36:56

PrinzipDesSees
Member
Registered: 2015-06-04
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

I tried the method with your wrapper script for passing x-vga but the only thing happening is that libvirt will hang until force restarted.
Yes I changed the paths to the right ones.
In between it does not seem anything to happen, not even the xml gets parsed.

Last edited by PrinzipDesSees (2015-06-17 01:39:07)

Offline

#5365 2015-06-17 01:47:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK folks, trying to guess what might be wrong with your set is really not fun.  If you want some help, document exactly what you're doing.  Prolific use of pastebin is encouraged.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5366 2015-06-17 01:54:54

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

OK folks, trying to guess what might be wrong with your set is really not fun.  If you want some help, document exactly what you're doing.  Prolific use of pastebin is encouraged.

Since I may need to post asking assistance soon if I can't get past my current hurdle, do you have a format/template in mind we should follow?  Not that one size fits all, but I'd not want to be leaving out information and i figure for certain problems there is at least a base level. (I've been trying to catch up, but I've been busy for the last 150 pages and might have missed changes along the way)

Offline

#5367 2015-06-17 02:03:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
aw wrote:

OK folks, trying to guess what might be wrong with your set is really not fun.  If you want some help, document exactly what you're doing.  Prolific use of pastebin is encouraged.

Since I may need to post asking assistance soon if I can't get past my current hurdle, do you have a format/template in mind we should follow?  Not that one size fits all, but I'd not want to be leaving out information and i figure for certain problems there is at least a base level. (I've been trying to catch up, but I've been busy for the last 150 pages and might have missed changes along the way)

Just use common sense, this isn't a bug reporting system, but use the same techniques to report enough information that someone can spot an error or reproduce the problem.  If you're trying to use a libvirt wrapper script, show the wrapper script and the xml.  [Sorry PrinzipDesSees, I'm not picking on you, we're getting a lot of this]  If you've changed libvirt.conf, qemu.conf or anything else related to the VM, report it.  Look for errors in dmesg or libvirt log files.  The more interest you take in solving your own problems, the more interest I'm going to have in helping you.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5368 2015-06-17 04:09:50

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Just use common sense, this isn't a bug reporting system, but use the same techniques to report enough information that someone can spot an error or reproduce the problem.

Reasonable enough.  Just wanted to check in case I was going to forget anything.  Plus, posting this will give me a reference if i get some slow time at work to search the interwebs for errors I'm getting...

Based off of the original post, starting with a stock install, pci-stub in the boot string was used to isolate the cards and the suggested vfio-bind script used to put them on the vfio-pci driver (if I don't use the script, they are on pci-stub and qemu complains.. mentioning that because I thought I saw that this wasn't supposed to be required anymore).

I am attempting for a headless host and the guest taking over the sole video card (so commands are run over ssh on the host)

system info:
Mobo: Asus Sabertooth 990FX/Gen3 R2.0 AM3+ AMD 990FX
Video: Asus HD7770-DC-1GD5-V2 Radeon HD 7770 GHz Edition 1GB

# uname -a Linux pandora 4.0.5-1-ARCH #1 SMP PREEMPT Sat Jun 6 18:37:49 CEST 2015 x86_64 GNU/Linux
# cat /proc/cmdline initrd=\initramfs-linux.img root=/dev/sda2 rw quiet splash video=efifb:off pci-stub.ids=1002:683d,1002:aab0
# lspci -nnk ... 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d] Subsystem: ASUSTeK Computer Inc. Device [1043:0429] Kernel driver in use: vfio-pci Kernel modules: radeon 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] Subsystem: ASUSTeK Computer Inc. Device [1043:aab0] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ...

Testing is done with one of two commands

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off\ -machine type=pc,accel=kvm \ -smp 2,sockets=1,cores=2,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga none -nographic \ -usb -usbdevice host:045e:0768 -usbdevice host:3842:2410 \ -cdrom /root/archbang-150516-i686.iso

or

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host \ -smp 1,sockets=1,cores=1,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga none -nographic -display stdio\ -device virtio-scsi-pci,id=scsi \ -drive file=/root/arch_guest.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -usb -usbdevice host:045e:0768 -usbdevice host:3842:2410

(the two usbdevices should be my mouse and keyboard.  removing them doesn't improve anything but definitely disables any keyboard support i had.)

Both guests boot and seem to get to the login prompt and then hang.   arch_guest.img hangs at the login prompt and won't take any input at all.   archbang.iso gets to the message saying that it's reached target Graphical Interface (occasionally not quite that far) and hangs, but i can switch to consoles 2-6 and log in (though i can't launch X from there because it's busy doing it's thing).

from an archbang run...
I assume that this in my guest's dmesg log is a good sign that the passthrough worked:

[ 0.290104] vgaarb: setting as boot device: PCI:0000:00:03.0 [ 0.290104] vgaarb: device added: PCI:0000:00:03.0,decodes=io+mem,owns=io+mem,locks=none [ 0.290104] vgaarb: loaded [ 0.290104] vgaarb: bridge control possible 0000:00:03.0 [ 0.290152] PCI: Using ACPI for IRQ routing [ 0.290152] PCI: pci_cache_line_size set to 64 bytes ... [ 6.487464] radeon 0000:00:03.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used) [ 6.487467] radeon 0000:00:03.0: GTT: 1024M 0x0000000040000000 - 0x000000007FFFFFFF

For a minute I was thinking it was simply that I needed to figure out the correct combination of  "-nographics -display xxxx" or whatever, but I am getting this trace soon before archbang hangs and that makes me think there is a problem with my set up that i'm missing.

[ 6.633752] CPU: 1 PID: 437 Comm: systemd-udevd Not tainted 4.0.2-1-ARCH #1 [ 6.633752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150422_083828-anatol 04/01/2014 [ 6.633752] task: f7353fc0 ti: f738a000 task.ti: f738a000 [ 6.633752] EIP: 0060:[<f8670cf1>] EFLAGS: 00010286 CPU: 1 [ 6.633752] EIP is at drm_pcie_get_speed_cap_mask+0x31/0xe0 [drm] [ 6.633752] EAX: f5dd6800 EBX: f738bb18 ECX: 00000000 EDX: f738bb18 [ 6.633752] ESI: 00000000 EDI: ee60c000 EBP: f738bab4 ESP: f738ba8c [ 6.633752] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 6.633752] CR0: 80050033 CR2: 00000020 CR3: 351bd000 CR4: 000007d0 [ 6.633752] Stack: [ 6.633752] c133b044 f738ba98 c13402f2 000080d0 00004a7c f8b62618 ee4f0000 8e6555bd [ 6.633752] 00000000 ee4f0000 f738bb60 f8b62639 c133c5ed f7332a0c ee439700 ee4f0000 [ 6.633752] f738bae0 f846d000 00000000 ee60c000 f8bc1e19 f738baf0 c1349d7d 00000000 [ 6.633752] Call Trace: [ 6.633752] [<c133b044>] ? get_device+0x14/0x30 [ 6.633752] [<c13402f2>] ? klist_class_dev_get+0x12/0x20 [ 6.633752] [<f8b62618>] ? si_dpm_init+0x38/0x11a0 [radeon] [ 6.633752] [<f8b62639>] si_dpm_init+0x59/0x11a0 [radeon] [ 6.633752] [<c133c5ed>] ? device_add+0x16d/0x5f0 [ 6.633752] [<c1349d7d>] ? pm_runtime_init+0xcd/0xe0 [ 6.633752] [<c133ca87>] ? device_register+0x17/0x20 [ 6.633752] [<f846b135>] ? hwmon_device_register_with_groups.part.0+0xa5/0x100 [hwmon] [ 6.633752] [<f846b1e1>] ? hwmon_device_register_with_groups+0x51/0x60 [hwmon] [ 6.633752] [<f8af6317>] ? radeon_hwmon_init+0x57/0x90 [radeon] [ 6.633752] [<f8af69aa>] radeon_pm_init+0x49a/0x860 [radeon] [ 6.633752] [<c1351426>] ? request_firmware+0x36/0x40 [ 6.633752] [<f8b279b3>] si_init+0x263/0xb30 [radeon] [ 6.633752] [<c1338c69>] ? vga_switcheroo_register_client+0x39/0x50 [ 6.633752] [<f8a8f267>] radeon_device_init+0x957/0xaf0 [radeon] [ 6.633752] [<f8a8d4e0>] ? cail_reg_read+0x70/0x70 [radeon] [ 6.633752] [<f8a9173a>] radeon_driver_load_kms+0x8a/0x200 [radeon] [ 6.633752] [<f866ec7e>] drm_dev_register+0x8e/0xd0 [drm] [ 6.633752] [<f86712e9>] drm_get_pci_dev+0xa9/0x1b0 [drm] [ 6.633752] [<c1170564>] ? kmem_cache_alloc_trace+0x1c4/0x200 [ 6.633752] [<f8a8d34e>] ? radeon_pci_probe+0x6e/0xa0 [radeon] [ 6.633752] [<f8a8d35c>] radeon_pci_probe+0x7c/0xa0 [radeon] [ 6.633752] [<c128f34f>] pci_device_probe+0x6f/0xd0 [ 6.633752] [<c11ed945>] ? sysfs_create_link+0x25/0x50 [ 6.633752] [<c133f403>] driver_probe_device+0x93/0x3c0 [ 6.633752] [<c133f7e9>] __driver_attach+0x79/0x80 [ 6.633752] [<c133f770>] ? __device_attach+0x40/0x40 [ 6.633752] [<c133d7e7>] bus_for_each_dev+0x57/0xa0 [ 6.633752] [<c133eeee>] driver_attach+0x1e/0x20 [ 6.633752] [<c133f770>] ? __device_attach+0x40/0x40 [ 6.633752] [<c133eb37>] bus_add_driver+0x157/0x240 [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<c133ffbd>] driver_register+0x5d/0xf0 [ 6.633752] [<c128eba3>] __pci_register_driver+0x33/0x40 [ 6.633752] [<f86714cd>] drm_pci_init+0xdd/0x100 [drm] [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<f86d4092>] radeon_init+0x92/0xa7 [radeon] [ 6.633752] [<c100047a>] do_one_initcall+0xaa/0x200 [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<c112cd9d>] ? free_pages_prepare+0x19d/0x2e0 [ 6.633752] [<c1170f65>] ? kfree+0x135/0x140 [ 6.633752] [<c115d238>] ? __vunmap+0xb8/0xf0 [ 6.633752] [<c117041e>] ? kmem_cache_alloc_trace+0x7e/0x200 [ 6.633752] [<c1498f95>] ? do_init_module+0x21/0x198 [ 6.633752] [<c1498f95>] ? do_init_module+0x21/0x198 [ 6.633752] [<c1498fc4>] do_init_module+0x50/0x198 [ 6.633752] [<c10d2883>] load_module+0x1df3/0x23b0 [ 6.633752] [<c10749ee>] ? finish_task_switch+0x4e/0xe0 [ 6.633752] [<c10d2f4f>] SyS_init_module+0x10f/0x170 [ 6.633752] [<c149e357>] sysenter_do_call+0x12/0x12 [ 6.633752] Code: ec 20 3e 8d 74 26 00 c7 02 00 00 00 00 8b 80 04 01 00 00 89 d3 65 8b 0d 14 00 00 00 89 4d f4 31 c9 85 c0 74 16 8b 40 08 8b 70 1c <0f> b7 46 20 66 3d 06 11 74 06 66 3d 66 11 75 18 b8 ea ff ff ff [ 6.633752] EIP: [<f8670cf1>] drm_pcie_get_speed_cap_mask+0x31/0xe0 [drm] SS:ESP 0068:f738ba8c [ 6.633752] CR2: 0000000000000020 [ 6.686348] ---[ end trace 7a3848033a378f35 ]---

Full dmesg: http://pastebin.com/RiNrY9sW

I'm looking back over the thread from the beginning to see if I can find anything that might help... i seem to recall some stuff regarding about reseting the card before/after launch qemu (not the eject device talk), but maybe I made that up or maybe it won't help.

Last edited by Blind Tree Frog (2015-06-17 04:11:54)

Offline

#5369 2015-06-17 15:47:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

Based off of the original post, starting with a stock install, pci-stub in the boot string was used to isolate the cards and the suggested vfio-bind script used to put them on the vfio-pci driver (if I don't use the script, they are on pci-stub and qemu complains.. mentioning that because I thought I saw that this wasn't supposed to be required anymore).

I don't advocate the vfio-bind script, especially the version that blindly binds any device in a group to vfio-pci regardless of the type.  vfio-pci doesn't know how to handle bridges and can actually disable them, preventing access to the downstream devices that you're actually trying to use.  Newer kernels will not allow binding vfio-pci to bridges.

That said, if you're not using libvirt, you do need to bind the endpoints you intend to assign to vfio-pci.  Perhaps your confusion on this point is that libvirt will do it for you, so for a libvirt-based setup, it's sufficient to get the GPU bound to pci-stub and leave the rest to libvirt.

If I were creating a qemu commandline setup, I'd still use virsh nodedev-deatch for binding to vfio-pci rather than the vfio-bind script.

I am attempting for a headless host and the guest taking over the sole video card (so commands are run over ssh on the host)

FWIW, the kernel handles the boot VGA ROM differently than secondary ROMs.  You actually get the shadow copy that might be modified during the execution of the ROM code.  So, somehow getting a copy of the real, untouched ROM and passing it as a file would be encouraged for this scenario.

system info:
Mobo: Asus Sabertooth 990FX/Gen3 R2.0 AM3+ AMD 990FX
Video: Asus HD7770-DC-1GD5-V2 Radeon HD 7770 GHz Edition 1GB

# uname -a Linux pandora 4.0.5-1-ARCH #1 SMP PREEMPT Sat Jun 6 18:37:49 CEST 2015 x86_64 GNU/Linux
# cat /proc/cmdline initrd=\initramfs-linux.img root=/dev/sda2 rw quiet splash video=efifb:off pci-stub.ids=1002:683d,1002:aab0
# lspci -nnk ... 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d] Subsystem: ASUSTeK Computer Inc. Device [1043:0429] Kernel driver in use: vfio-pci Kernel modules: radeon 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0] Subsystem: ASUSTeK Computer Inc. Device [1043:aab0] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ...

Testing is done with one of two commands

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off\ -machine type=pc,accel=kvm \ -smp 2,sockets=1,cores=2,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga none -nographic \ -usb -usbdevice host:045e:0768 -usbdevice host:3842:2410 \ -cdrom /root/archbang-150516-i686.iso

or

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host \ -smp 1,sockets=1,cores=1,threads=1 \ -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1 \ -vga none -nographic -display stdio\ -device virtio-scsi-pci,id=scsi \ -drive file=/root/arch_guest.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -usb -usbdevice host:045e:0768 -usbdevice host:3842:2410

(the two usbdevices should be my mouse and keyboard.  removing them doesn't improve anything but definitely disables any keyboard support i had.)

Both guests boot and seem to get to the login prompt and then hang.   arch_guest.img hangs at the login prompt and won't take any input at all.   archbang.iso gets to the message saying that it's reached target Graphical Interface (occasionally not quite that far) and hangs, but i can switch to consoles 2-6 and log in (though i can't launch X from there because it's busy doing it's thing).

from an archbang run...
I assume that this in my guest's dmesg log is a good sign that the passthrough worked:

[ 0.290104] vgaarb: setting as boot device: PCI:0000:00:03.0 [ 0.290104] vgaarb: device added: PCI:0000:00:03.0,decodes=io+mem,owns=io+mem,locks=none [ 0.290104] vgaarb: loaded [ 0.290104] vgaarb: bridge control possible 0000:00:03.0 [ 0.290152] PCI: Using ACPI for IRQ routing [ 0.290152] PCI: pci_cache_line_size set to 64 bytes ... [ 6.487464] radeon 0000:00:03.0: VRAM: 1024M 0x0000000000000000 - 0x000000003FFFFFFF (1024M used) [ 6.487467] radeon 0000:00:03.0: GTT: 1024M 0x0000000040000000 - 0x000000007FFFFFFF

For a minute I was thinking it was simply that I needed to figure out the correct combination of  "-nographics -display xxxx" or whatever, but I am getting this trace soon before archbang hangs and that makes me think there is a problem with my set up that i'm missing.

[ 6.633752] CPU: 1 PID: 437 Comm: systemd-udevd Not tainted 4.0.2-1-ARCH #1 [ 6.633752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150422_083828-anatol 04/01/2014 [ 6.633752] task: f7353fc0 ti: f738a000 task.ti: f738a000 [ 6.633752] EIP: 0060:[<f8670cf1>] EFLAGS: 00010286 CPU: 1 [ 6.633752] EIP is at drm_pcie_get_speed_cap_mask+0x31/0xe0 [drm] [ 6.633752] EAX: f5dd6800 EBX: f738bb18 ECX: 00000000 EDX: f738bb18 [ 6.633752] ESI: 00000000 EDI: ee60c000 EBP: f738bab4 ESP: f738ba8c [ 6.633752] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 6.633752] CR0: 80050033 CR2: 00000020 CR3: 351bd000 CR4: 000007d0 [ 6.633752] Stack: [ 6.633752] c133b044 f738ba98 c13402f2 000080d0 00004a7c f8b62618 ee4f0000 8e6555bd [ 6.633752] 00000000 ee4f0000 f738bb60 f8b62639 c133c5ed f7332a0c ee439700 ee4f0000 [ 6.633752] f738bae0 f846d000 00000000 ee60c000 f8bc1e19 f738baf0 c1349d7d 00000000 [ 6.633752] Call Trace: [ 6.633752] [<c133b044>] ? get_device+0x14/0x30 [ 6.633752] [<c13402f2>] ? klist_class_dev_get+0x12/0x20 [ 6.633752] [<f8b62618>] ? si_dpm_init+0x38/0x11a0 [radeon] [ 6.633752] [<f8b62639>] si_dpm_init+0x59/0x11a0 [radeon] [ 6.633752] [<c133c5ed>] ? device_add+0x16d/0x5f0 [ 6.633752] [<c1349d7d>] ? pm_runtime_init+0xcd/0xe0 [ 6.633752] [<c133ca87>] ? device_register+0x17/0x20 [ 6.633752] [<f846b135>] ? hwmon_device_register_with_groups.part.0+0xa5/0x100 [hwmon] [ 6.633752] [<f846b1e1>] ? hwmon_device_register_with_groups+0x51/0x60 [hwmon] [ 6.633752] [<f8af6317>] ? radeon_hwmon_init+0x57/0x90 [radeon] [ 6.633752] [<f8af69aa>] radeon_pm_init+0x49a/0x860 [radeon] [ 6.633752] [<c1351426>] ? request_firmware+0x36/0x40 [ 6.633752] [<f8b279b3>] si_init+0x263/0xb30 [radeon] [ 6.633752] [<c1338c69>] ? vga_switcheroo_register_client+0x39/0x50 [ 6.633752] [<f8a8f267>] radeon_device_init+0x957/0xaf0 [radeon] [ 6.633752] [<f8a8d4e0>] ? cail_reg_read+0x70/0x70 [radeon] [ 6.633752] [<f8a9173a>] radeon_driver_load_kms+0x8a/0x200 [radeon] [ 6.633752] [<f866ec7e>] drm_dev_register+0x8e/0xd0 [drm] [ 6.633752] [<f86712e9>] drm_get_pci_dev+0xa9/0x1b0 [drm] [ 6.633752] [<c1170564>] ? kmem_cache_alloc_trace+0x1c4/0x200 [ 6.633752] [<f8a8d34e>] ? radeon_pci_probe+0x6e/0xa0 [radeon] [ 6.633752] [<f8a8d35c>] radeon_pci_probe+0x7c/0xa0 [radeon] [ 6.633752] [<c128f34f>] pci_device_probe+0x6f/0xd0 [ 6.633752] [<c11ed945>] ? sysfs_create_link+0x25/0x50 [ 6.633752] [<c133f403>] driver_probe_device+0x93/0x3c0 [ 6.633752] [<c133f7e9>] __driver_attach+0x79/0x80 [ 6.633752] [<c133f770>] ? __device_attach+0x40/0x40 [ 6.633752] [<c133d7e7>] bus_for_each_dev+0x57/0xa0 [ 6.633752] [<c133eeee>] driver_attach+0x1e/0x20 [ 6.633752] [<c133f770>] ? __device_attach+0x40/0x40 [ 6.633752] [<c133eb37>] bus_add_driver+0x157/0x240 [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<c133ffbd>] driver_register+0x5d/0xf0 [ 6.633752] [<c128eba3>] __pci_register_driver+0x33/0x40 [ 6.633752] [<f86714cd>] drm_pci_init+0xdd/0x100 [drm] [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<f86d4092>] radeon_init+0x92/0xa7 [radeon] [ 6.633752] [<c100047a>] do_one_initcall+0xaa/0x200 [ 6.633752] [<f86d4000>] ? 0xf86d4000 [ 6.633752] [<c112cd9d>] ? free_pages_prepare+0x19d/0x2e0 [ 6.633752] [<c1170f65>] ? kfree+0x135/0x140 [ 6.633752] [<c115d238>] ? __vunmap+0xb8/0xf0 [ 6.633752] [<c117041e>] ? kmem_cache_alloc_trace+0x7e/0x200 [ 6.633752] [<c1498f95>] ? do_init_module+0x21/0x198 [ 6.633752] [<c1498f95>] ? do_init_module+0x21/0x198 [ 6.633752] [<c1498fc4>] do_init_module+0x50/0x198 [ 6.633752] [<c10d2883>] load_module+0x1df3/0x23b0 [ 6.633752] [<c10749ee>] ? finish_task_switch+0x4e/0xe0 [ 6.633752] [<c10d2f4f>] SyS_init_module+0x10f/0x170 [ 6.633752] [<c149e357>] sysenter_do_call+0x12/0x12 [ 6.633752] Code: ec 20 3e 8d 74 26 00 c7 02 00 00 00 00 8b 80 04 01 00 00 89 d3 65 8b 0d 14 00 00 00 89 4d f4 31 c9 85 c0 74 16 8b 40 08 8b 70 1c <0f> b7 46 20 66 3d 06 11 74 06 66 3d 66 11 75 18 b8 ea ff ff ff [ 6.633752] EIP: [<f8670cf1>] drm_pcie_get_speed_cap_mask+0x31/0xe0 [drm] SS:ESP 0068:f738ba8c [ 6.633752] CR2: 0000000000000020 [ 6.686348] ---[ end trace 7a3848033a378f35 ]---

Full dmesg: http://pastebin.com/RiNrY9sW

I'm looking back over the thread from the beginning to see if I can find anything that might help... i seem to recall some stuff regarding about reseting the card before/after launch qemu (not the eject device talk), but maybe I made that up or maybe it won't help.

You're assigning an AMD card to a 440FX VM running Linux.  That's pretty much the one case where I recommend Q35.  I've posted patches for this, but I lost track of whether they've all been accepted.  The problem is that the radeon driver blindly assumes that a downstream port exists above the device because that's the way it appears on real hardware.  It then goes trying to read link speed info on a device that doesn't exist and you get the above oops.  You need to switch to Q35, add a root port and put the GPU below the root port.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5370 2015-06-17 16:03:53

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Blind Tree Frog wrote:

Based off of the original post, starting with a stock install, pci-stub in the boot string was used to isolate the cards and the suggested vfio-bind script used to put them on the vfio-pci driver (if I don't use the script, they are on pci-stub and qemu complains.. mentioning that because I thought I saw that this wasn't supposed to be required anymore).

I don't advocate the vfio-bind script, especially the version that blindly binds any device in a group to vfio-pci regardless of the type.  vfio-pci doesn't know how to handle bridges and can actually disable them, preventing access to the downstream devices that you're actually trying to use.  Newer kernels will not allow binding vfio-pci to bridges.

That said, if you're not using libvirt, you do need to bind the endpoints you intend to assign to vfio-pci.  Perhaps your confusion on this point is that libvirt will do it for you, so for a libvirt-based setup, it's sufficient to get the GPU bound to pci-stub and leave the rest to libvirt.

If I were creating a qemu commandline setup, I'd still use virsh nodedev-deatch for binding to vfio-pci rather than the vfio-bind script.

Switching to virsh was on the list once I got things running.   Now that you mention it, I do remember reading your stuff that it was libvirt that handled the vfio-pci stuff automatically.

aw wrote:
btf wrote:

I am attempting for a headless host and the guest taking over the sole video card (so commands are run over ssh on the host)

FWIW, the kernel handles the boot VGA ROM differently than secondary ROMs.  You actually get the shadow copy that might be modified during the execution of the ROM code.  So, somehow getting a copy of the real, untouched ROM and passing it as a file would be encouraged for this scenario.

I was hoping to avoid that, but I feared I might have to look into that at some point.

aw wrote:

You're assigning an AMD card to a 440FX VM running Linux.  That's pretty much the one case where I recommend Q35.  I've posted patches for this, but I lost track of whether they've all been accepted.  The problem is that the radeon driver blindly assumes that a downstream port exists above the device because that's the way it appears on real hardware.  It then goes trying to read link speed info on a device that doesn't exist and you get the above oops.  You need to switch to Q35, add a root port and put the GPU below the root port.

Ah, that's what the trace is complaining about.  A cursory search was leading me to similar errors about a root device not being available, but I wasn't sure if that applied here or not since my PCI root was in a different group.

Thanks.

edit:
assuming that I am looking at the correct commits in qemu, I believe I see similar commits, but they are old enough that I'd expect that (Apr 13) so I'm guessing that either this specific issue didn't get hit, or not all patches were accepted.

Last edited by Blind Tree Frog (2015-06-17 18:12:13)

Offline

#5371 2015-06-17 17:05:37

novist
Member
Registered: 2014-03-14
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone actually tried to get gpu passthrough working on laptop with nvidia optimus? I wonder if its even possible. If it were would be good incentive to get such laptop.

Offline

#5372 2015-06-17 20:03:01

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

Has anyone actually tried to get gpu passthrough working on laptop with nvidia optimus? I wonder if its even possible. If it were would be good incentive to get such laptop.

I've observed very weird 3-state GPU on some lenovo laptop:
The Nvidia GPU was full on when it's used.
The Nvidia GPU is in power-save mode, just like unused ones on a "real" PC when it was used.
The Nvidia GPU is in some uninitialized state, where lspci fails to read the info, complaining about some "bad headers", when the GPU wasn't used since system boot.

And the screen switching problem, as some notebooks have their auxiliary video output routed only to intel, only to nvidia or to both via some weird switch.


Has anyone tried? I can't recall. If it's even possible? There was a guy without optimus, who tried to pass a signle and only GPU to the VM leaving the host headless, and he failed. So i highly doubt that it's possible in theory. It all depends on the vendor of the laptop, and what their firmware engineers were smoking.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5373 2015-06-17 20:19:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You also have to wonder how well the thermal controls work for an integrated environment like a laptop.  On a discrete desktop card, you know the GPU fan speed is controlled through the GPU device driver, the rest of the PC knows nothing about it.  On a laptop there's often a more holistic approach to thermal management.  If I was trying one, I'd be very careful until I gained some confidence that the response to thermal load was working.

The only experience I have is a pre-optimus laptop that I could never get past Code 43, probably because it's tied to a model specific, custom nvidia driver (ie. it doesn't work with downloads direct from nvidia even on bare metal).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5374 2015-06-17 20:52:05

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You also have to wonder how well the thermal controls work for an integrated environment like a laptop.  On a discrete desktop card, you know the GPU fan speed is controlled through the GPU device driver, the rest of the PC knows nothing about it.  On a laptop there's often a more holistic approach to thermal management.  If I was trying one, I'd be very careful until I gained some confidence that the response to thermal load was working.

Usually laptops have one fan to cool them all. So i guess if you can rule the fan manually, set it to full power and stress-test the hardware.

In the worst case(if the vendor is a total dick) - you can always resolder the fan to sata power connector or some other known good voltage. And use a cooling station.

BUT. That fan might not be enough, as i've observed some cheap and 2-3 years old dells overheating from running a windows calculator calculating "1000000!". The VM will use the hardware more intensely than running stuff bare-metal.

Last edited by Duelist (2015-06-17 20:52:13)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5375 2015-06-17 21:43:09

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex, not sure if you got my email or not. VFIO is still not working properly, still missing some files. i know that VFIO is built as a module, as evidenced by running "modinfo vfio". Not sure how to proceed.

Offline

#5376 2015-06-17 22:03:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@garnerlogan65

See post #5365 and #5367.  I'm not interested in guessing the details of the problem, repeating request that were not followed, or holding hands through reboots.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5377 2015-06-17 23:13:17

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

FWIW, the kernel handles the boot VGA ROM differently than secondary ROMs.  You actually get the shadow copy that might be modified during the execution of the ROM code.  So, somehow getting a copy of the real, untouched ROM and passing it as a file would be encouraged for this scenario.

aw wrote:

You're assigning an AMD card to a 440FX VM running Linux.  That's pretty much the one case where I recommend Q35.  I've posted patches for this, but I lost track of whether they've all been accepted.  The problem is that the radeon driver blindly assumes that a downstream port exists above the device because that's the way it appears on real hardware.  It then goes trying to read link speed info on a device that doesn't exist and you get the above oops.  You need to switch to Q35, add a root port and put the GPU below the root port.

And now it seems to be working.  I downloaded the ROM from a site upstream but using it didn't work, so I just dropped that and I finally managed to log into a guest system ("yay!!").

qemu command:

#!/bin/bash # archbang guest qemu-system-x86_64 \ -enable-kvm -M q35 -m 6144 \ -smp 5,sockets=1,cores=5,threads=1 -cpu host,kvm=off \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=fakeRoot.1 \ -device vfio-pci,host=03:00.0,bus=fakeRoot.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=03:00.1,bus=fakeRoot.1,addr=00.1 \ -cdrom /root/archbang-150516-i686.iso \ -usb -usbdevice host:045e:0768 -usbdevice host:3842:2410 \ -rtc base=localtime \ -vga none -display none \ $@

There are two minor hiccups I've noticed thus far:
1) the arch guest I previously set up has an ethernet device, but didn't automatically grab a IP address.  The archbang one did however, so I'm assuming a reinstall will fix it.
2) Previously the keyboard was supported at the boot loader screen, now it doesn't seem to be. May be a non-issue in the end but I figured I should write down for my later tinkering.

Anyhow, now to get sound, networking, and a handful of other things set up (passing the bluray player directly to the vm for one) and then toss it into virsh.  Thanks for your help, aw; I owe you a beer if you are ever in the RTP area.

Offline

#5378 2015-06-18 17:39:07

garnerlogan65
Member
Registered: 2014-11-22
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw, I already gave you the information you asked for (whether vfio was static or not), and I'm not sure what else to do from here, if I knew what to do I would have done it, I can't really do anything with vfio files missing. The last thing I tried to do was near the end of part 3 of your guide, and install options to dracut and generating an initramfs, which did not work. It said that it was unable to install vfio and such. Not sure how to proceed from here.

Offline

#5379 2015-06-18 19:22:22

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

garnerlogan65 wrote:

aw, I already gave you the information you asked for (whether vfio was static or not), and I'm not sure what else to do from here, if I knew what to do I would have done it, I can't really do anything with vfio files missing. The last thing I tried to do was near the end of part 3 of your guide, and install options to dracut and generating an initramfs, which did not work. It said that it was unable to install vfio and such. Not sure how to proceed from here.

If your having difficulties with a certain part, there are other guides out there that are not made specefically for VFIO, but should give you a hint into what your doing wrong and how you can correct it.

Also, don't forget to do you 'man' page pushups.  Like:

man grep

Offline

#5380 2015-06-18 21:28:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@garnerlogan65

Here's a trivial example:

garnerlogan65 wrote:

I can't really do anything with vfio files missing. The last thing I tried to do was near the end of part 3 of your guide, and install options to dracut and generating an initramfs, which did not work. It said that it was unable to install vfio and such. Not sure how to proceed from here.

What vfio files are missing?  What exactly are you trying to do "near the end of part 3 of your guide"?  What exactly didn't work generating an initramfs?  Give the error message, screen log, anything!  If you can't bother to provide enough background on what you've done, what you're trying to do, what you're expecting and what actually happened, then I can't be bothered to spend time on your problems.  Sorry.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5381 2015-06-19 22:11:22

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Denso

Are you rebuilding your initramfs between each of these changes?  What shows up in dmesg if you run 'modprobe -r vfio-pci; modprobe vfio-pci' with modprobe.d as in the previous post?

Yes . Whenever I do changes to initramfs config file , I rebuild it .

modprobe -r vfio-pci modprobe: FATAL: Module vfio_pci is in use.

Of course , VFIO is in use already (being bound to Intel Audio , Intel NIC for my VM) . It just won't get bound to GPUs using modprobe.d .

Offline

#5382 2015-06-19 22:26:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

Well obviously you can't remove the module while a VM is running and using it...  I don't know what to tell you, I've got a system running with exactly this right now:

pci-stub.ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff,6549:ffffffff,11c1:5811

It works exactly like it's supposed to, claiming all the Nvidia VGA and audio devices, plus all Teradici devices, and then a random firewire controller that shares a group with a NIC.  Double check everything.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5383 2015-06-19 23:49:42

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Umm , I'm not using pci-stub.ids .

I'm using "options vfio-pci ids =" inside a modprobe.d conf file (Kernel 4.1-rc7) .

I'm skipping pci-stub altogether and binding the devices directly to vfio . It works for the Intel Audio , Intel NIC and a PCI-E USB3 controller . Just not the GPUs .

I think I'll compile a 4.1-rc1 and see if it would work . It used to work on that version like a charm .

I'll keep you posted ! smile

Offline

#5384 2015-06-20 04:14:18

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

vfio-pci and pci-stub use pretty much the same code for the ids option, the difference is that pci-stub is usually builtin and vfio-pci is usually a module.  So, if you get different results, it probably has something to do with that.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5385 2015-06-20 11:57:29

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Even 4.0-rc1 didn't work , which was working perfectly fine before I install this motherboard (X99-E WS) . Perhaps it is because it uses PLX chips ? I read a previous post that vfio-bind can bind bridges while modprobe.d can't right ?

Here is the tree :

root ~ lspci -t -+-[0000:ff]-+-0b.0 | +-0b.1 | +-0b.2 | +-0c.0 | +-0c.1 | +-0c.2 | +-0c.3 | +-0c.4 | +-0c.5 | +-0f.0 | +-0f.1 | +-0f.4 | +-0f.5 | +-0f.6 | +-10.0 | +-10.1 | +-10.5 | +-10.6 | +-10.7 | +-12.0 | +-12.1 | +-13.0 | +-13.1 | +-13.2 | +-13.3 | +-13.4 | +-13.5 | +-13.6 | +-13.7 | +-14.0 | +-14.1 | +-14.2 | +-14.3 | +-14.6 | +-14.7 | +-15.0 | +-15.1 | +-15.2 | +-15.3 | +-16.0 | +-16.6 | +-16.7 | +-17.0 | +-17.4 | +-17.5 | +-17.6 | +-17.7 | +-1e.0 | +-1e.1 | +-1e.2 | +-1e.3 | +-1e.4 | +-1f.0 | \-1f.2 \-[0000:00]-+-00.0 +-01.0-[01]-- +-01.1-[02]-- +-02.0-[07-11]----00.0-[08-11]--+-08.0-[0b]----00.0 | +-09.0-[0c-11]----00.0-[0d-11]--+-01.0-[0e]----00.0 | | +-05.0-[0f]----00.0 | | +-07.0-[10]----00.0 | | \-09.0-[11]----00.0 | +-10.0-[09]--+-00.0 | | \-00.1 | \-11.0-[0a]-- +-03.0-[03-06]----00.0-[04-06]--+-08.0-[06]-- | \-10.0-[05]--+-00.0 | \-00.1 +-05.0 +-05.1 +-05.2 +-05.4 +-11.0 +-11.4 +-14.0 +-16.0 +-19.0 +-1a.0 +-1b.0 +-1c.0-[12]-- +-1c.3-[13-16]----00.0-[14-16]--+-01.0-[15]----00.0 | \-05.0-[16]----00.0 +-1c.4-[17]----00.0 +-1c.6-[18]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3
root ~ lspci 00:00.0 Host bridge: Intel Corporation Xeon E5 v3/Core i7 DMI2 (rev 02) 00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) 00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) 00:05.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02) 00:05.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Hot Plug (rev 02) 00:05.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02) 00:05.4 PIC: Intel Corporation Xeon E5 v3/Core i7 I/O APIC (rev 02) 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05) 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05) 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) 00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05) 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-LM (rev 05) 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05) 00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05) 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) 00:1c.3 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5) 00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5) 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05) 03:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 04:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 04:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 05:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) 05:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1) 07:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 08:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 08:09.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 08:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 08:11.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 09:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2) 09:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1) 0b:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) 0c:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0d:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0d:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0d:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0d:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 0e:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 13:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 14:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 14:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 15:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 16:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 17:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 18:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller ff:0b.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0b.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0b.2 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0c.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02) ff:0f.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02) ff:0f.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:0f.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:0f.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:10.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02) ff:10.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02) ff:10.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:10.6 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:10.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:12.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02) ff:12.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02) ff:13.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02) ff:13.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02) ff:13.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02) ff:13.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02) ff:14.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02) ff:14.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02) ff:14.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02) ff:14.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02) ff:14.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:14.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:15.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control (rev 02) ff:15.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control (rev 02) ff:15.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers (rev 02) ff:15.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers (rev 02) ff:16.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02) ff:16.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02) ff:16.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02) ff:17.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02) ff:17.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:1e.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02) ff:1f.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02)

I'm sorry for bothering you this much Alex sad

Offline

#5386 2015-06-20 14:29:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It was a mistake for vfio-bind to try to bind bridges and a bug in vfio-pci to allow bridges.  The latter has since been corrected in vfio-pci.  That means that if you use vfio-bind, you may end up with bridges without a driver since vfio-pci won't let them bind.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5387 2015-06-20 14:48:12

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ aw :

After using vfio-bind and starting all of my VMs , checking all of the bridges' drivers in use , they are all using "pcieport" . So I guess I'm fine for now smile

Let's see if I can hunt and fix the modprobe.d ids issue .

Offline

#5388 2015-06-20 18:30:14

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have checked MSI enable on my GTx 970. In linux it reports MSI: enable-    in windows when I view the connection type pci, i see the card after driver install using msi.


I am getting this error occasionally after a freeze and vm crashes, usually when I am installing, or accessing the graphics card.

qemu-system-x86_64: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

any thoughts or ideas?

Last edited by supertrampx (2015-06-20 18:35:05)

Offline

#5389 2015-06-20 19:19:36

DanaGoyette
Member
Registered: 2014-01-03
Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The latest time I tried to reset my radeon via suspend (because the workaround seemed to have failed), I got this message when trying to start the VM after resume:
Error starting domain: Requested operation is not valid: PCI device 0000:08:00.0 is not assignable

Syslog showed the following message:
libvirtd[2036]: internal error: Device 0000:08:00.0 is behind a switch lacking ACS and cannot be assigned

That's coming from libvirt, not the kernel.  Oddly, it didn't complain about the same device before suspend.
I know I can just set relaxed_acs_check=1 in libvirtd.conf, but it would be good to find out why it suddenly decides to complain.

lspci -tv (with 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_212.html topic_213.html topic_214.html topic_215.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html denoting devices I'm forwarding to this VM):

-[0000:00]-+-00.0 Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller +-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device aac8 +-02.0 Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 +-16.3 Intel Corporation 8 Series/C220 Series Chipset Family KT Controller +-19.0 Intel Corporation Ethernet Connection I217-LM +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 *-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller *-1c.0-[02]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller +-1c.1-[03-0a]----00.0-[04-0a]--+-01.0-[05]-- | +-04.0-[06]-- | +-05.0-[07]-- | *-07.0-[08]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller | \-09.0-[09-0a]----00.0-[0a]----00.0 Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] *-1c.3-[0b]----00.0 Intel Corporation I210 Gigabit Network Connection +-1c.4-[0c-44]-- +-1d.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 +-1f.0 Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller +-1f.2 Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] +-1f.3 Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller \-1f.6 Intel Corporation 8 Series Chipset Family Thermal Management Controller

lspci -nnv for the path involving that device, after suspend/resume:

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06) Subsystem: Super Micro Computer Inc Device [15d9:0818] Flags: bus master, fast devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c <?> Kernel driver in use: ie31200_edac 00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=03, subordinate=0a, sec-latency=0 Memory behind bridge: ee800000-eebfffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0818] Capabilities: [a0] Power Management version 3 Kernel driver in use: pcieport 03:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at eeb00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=03, secondary=04, subordinate=0a, sec-latency=0 Memory behind bridge: ee800000-eeafffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting <?> Capabilities: [148] Virtual Channel Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 04:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=04, secondary=08, subordinate=08, sec-latency=0 Memory behind bridge: eea00000-eeafffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 08:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) (prog-if 30 [XHCI]) Subsystem: Super Micro Computer Inc Device [15d9:0818] Flags: bus master, fast devsel, latency 0, IRQ 16 Memory at eea00000 (64-bit, non-prefetchable) [size=8K] Capabilities: [50] Power Management version 3 Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Capabilities: [90] MSI-X: Enable+ Count=8 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Latency Tolerance Reporting Kernel driver in use: xhci_hcd

Offline

#5390 2015-06-20 19:23:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

I have checked MSI enable on my GTx 970. In linux it reports MSI: enable-    in windows when I view the connection type pci, i see the card after driver install using msi.


I am getting this error occasionally after a freeze and vm crashes, usually when I am installing, or accessing the graphics card.

qemu-system-x86_64: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

any thoughts or ideas?

You got a fatal AER error.  Are you overclocking?  You can boot the host with pci=noaer to disable it, but that's no guarantee that the you're not still getting bus errors.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5391 2015-06-21 01:50:24

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi ...

Anyone else is having trouble with their beidged networking ? Both of my VMs are now unreachable , and they can't reach the outer network .

Commands I use to set the bridge up :

ip link add name br0 type bridge ip link set dev br0 up ip link set dev eth0 promisc on ip link set dev eth0 up ip link set dev eth0 master br0 ip link set dev br0 up ip addr add 10.0.1.240/24 broadcast 10.0.1.255 dev br0 ip route add default via 10.0.1.1

The part of code to start the VM (the net bit) :

-netdev bridge,id=br0 -device virtio-net-pci,netdev=br0,id=nic0,mac=B8:A7:2A:6E:09:B2

Even using the legacy method doesn't work :

-net nic,model=virtio,macaddr=B8:A7:2A:6E:09:B2 -net bridge,br=br0

Windows guests can't identify the network or connect to the outside world , I even set the adapter's IP/Subnet/Gateway/DNS manually (Yes , I know what I'm doing) .

Dmesg doesn't show anything not familier :

[Sun Jun 21 02:53:51 2015] br0: port 1(eth0) entered forwarding state [Sun Jun 21 02:53:51 2015] br0: port 1(eth0) entered forwarding state [Sun Jun 21 02:54:06 2015] br0: port 1(eth0) entered forwarding state [Sun Jun 21 02:56:03 2015] device tap0 entered promiscuous mode [Sun Jun 21 02:56:03 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 02:56:03 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 02:56:07 2015] kvm: zapping shadow pages for mmio generation wraparound [Sun Jun 21 02:56:18 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 02:57:12 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 02:57:12 2015] device tap0 left promiscuous mode [Sun Jun 21 02:57:12 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 03:03:04 2015] device tap0 entered promiscuous mode [Sun Jun 21 03:03:04 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:03:04 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:03:08 2015] kvm: zapping shadow pages for mmio generation wraparound [Sun Jun 21 03:03:19 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:04:33 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 03:04:33 2015] device tap0 left promiscuous mode [Sun Jun 21 03:04:33 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 03:15:35 2015] device tap0 entered promiscuous mode [Sun Jun 21 03:15:35 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:15:35 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:15:39 2015] kvm: zapping shadow pages for mmio generation wraparound [Sun Jun 21 03:15:50 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:16:37 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 03:16:37 2015] device tap0 left promiscuous mode [Sun Jun 21 03:16:37 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 03:48:07 2015] device tap0 entered promiscuous mode [Sun Jun 21 03:48:07 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:48:07 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 03:48:12 2015] kvm: zapping shadow pages for mmio generation wraparound [Sun Jun 21 03:48:22 2015] br0: port 2(tap0) entered forwarding state [Sun Jun 21 04:33:41 2015] br0: port 2(tap0) entered disabled state [Sun Jun 21 04:33:41 2015] device tap0 left promiscuous mode [Sun Jun 21 04:33:41 2015] br0: port 2(tap0) entered disabled state

Note that the "disabled state" messages are when I actually shutdown the VM to test , so it doesn't go to the disabled state out of spite .

Kernel is 4.1-rc7 with Qemu-git compiled 7 hours ago . (It didn't work with the pervious version compiled a week ago either) .

Any light ?

Last edited by Denso (2015-06-21 01:54:17)

Offline

#5392 2015-06-21 02:10:56

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

what if you manually create the tap? did you try? I start my vm with
-netdev tap,id=netdev0,ifname=vmtap10,script=no,downscript=no
-device virtio-net-pci,netdev=netdev0,id=net0,mac=xx:xx:xx:xx:xx:xx,bus=pci.2,addr=0x1
the tap has to exist of course smile

Offline

#5393 2015-06-21 02:30:18

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ghormoon :

Thanks , I've never tried to create taps manually because QEMU was doing it automatically . And I still prefer to set up one bridge and let QEMU create as many taps as it needs when I run new VMs .

Regards smile

Offline

#5394 2015-06-21 02:32:02

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it's ok to make it automatically if it works and suits you. the point was to test out, where the problem is, if the creation, or something entirely different smile

Offline

#5395 2015-06-21 02:45:29

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ ghormoon :

I tried what you suggested just now , and the result is the same .

I can't believe that just changing the motherboard would result in this much hassle . -_-

EDIT :

THe first thing I did after installing this motherboard was compiling Qemu + OVMF + Kernel using GCC 5.1 which was in the stable repos for Arch at that time . Could it be the reason for these unusual issues I'm having thus far ?!

Last edited by Denso (2015-06-21 07:01:37)

Offline

#5396 2015-06-21 21:08:32

scoobydog
Member
Registered: 2014-04-14
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,

after some experiences with my socket 2011 CPU and x79 chipset I checked the new 1150 Socket CPU with a z97 chipset, and it works with the i915 patch and seabios and without patching only with OVMF on Debian 8 Jessie.

I was so happy that I create some Tutorials for all how want to start with this.

my system is build on:
Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz socket 1150
Gigabyte GA-Z97X-UD3H-BK
16GB DDR3 Memory
512GB SSD
NVIDIA GeForce GTX 780 (ASUS GTX780-DC2OC-3GD5) for passthrugh

Debian 8.1 Jessie

Linux  3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u1), Copyright (c) 2003-2008 Fabrice Bellard
ovmf    0~20150106.5c2d456b-1 from SID repository

English tutorial:
Tutorial Debian 8, passthrough you physical video card to your virtual Windows7 with KVM and OVMF

German Tutorial:
Tutorial: Debian 8, mit KVM und OVMF die Physikalische Grafikkarte im virtuellen Windows7 benutzen

for Kernel Patching
HOWTO Debian Jessie 8 Kernel patch i915 VGA arbiter


kvm-exp.blogspot.de

Last edited by scoobydog (2015-06-21 21:44:28)

Offline

#5397 2015-06-21 22:18:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

EDIT :

THe first thing I did after installing this motherboard was compiling Qemu + OVMF + Kernel using GCC 5.1 which was in the stable repos for Arch at that time . Could it be the reason for these unusual issues I'm having thus far ?!

The gentooist within me speaks: "Look for the build log, luke".


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5398 2015-06-21 23:16:10

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

scoobydog wrote:

hi,

after some experiences with my socket 2011 CPU and x79 chipset I checked the new 1150 Socket CPU with a z97 chipset, and it works with the i915 patch and seabios and without patching only with OVMF on Debian 8 Jessie.

I was so happy that I create some Tutorials for all how want to start with this.

my system is build on:
Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz socket 1150
Gigabyte GA-Z97X-UD3H-BK
16GB DDR3 Memory
512GB SSD
NVIDIA GeForce GTX 780 (ASUS GTX780-DC2OC-3GD5) for passthrugh

Debian 8.1 Jessie

Linux  3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u1), Copyright (c) 2003-2008 Fabrice Bellard
ovmf    0~20150106.5c2d456b-1 from SID repository

English tutorial:
Tutorial Debian 8, passthrough you physical video card to your virtual Windows7 with KVM and OVMF

German Tutorial:
Tutorial: Debian 8, mit KVM und OVMF die Physikalische Grafikkarte im virtuellen Windows7 benutzen

for Kernel Patching
HOWTO Debian Jessie 8 Kernel patch i915 VGA arbiter


kvm-exp.blogspot.de

Hi ! Thank you for spending the time writing those posts ! smile

@Duelist :

I am certain that if there were problems with the building proccess , the whole thing would throw and error and ask if you would like to start from the begining .

Anyone here compiled these packages using GCC 5.1 ?

Offline

#5399 2015-06-22 00:52:41

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@Duelist :

I am certain that if there were problems with the building proccess , the whole thing would throw and error and ask if you would like to start from the begining .

Gentooist in me speaks again: "Nope. If you're missing some dependencies, make could just say something "oh, you don't have that software, well, i'll just drop it's support out completely" and continue building. Moreover, in gentoo there's a feature called USE-flags to specifically tell the toolchain that you want some select features."

And also, you can check your iptables.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5400 2015-06-22 02:05:52

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Gentooist in me speaks again: "Nope. If you're missing some dependencies, make could just say something "oh, you don't have that software, well, i'll just drop it's support out completely" and continue building. Moreover, in gentoo there's a feature called USE-flags to specifically tell the toolchain that you want some select features."

And also, you can check your iptables.

Well , I'll build em packages again using yaourt . How can I save the build log for examining it later ?

Oh , and I don't use iptaples or any other firewalls for the time being , so its not the cause for this issue .

Cheers !

Offline

#5401 2015-06-22 06:33:46

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
Duelist wrote:

Gentooist in me speaks again: "Nope. If you're missing some dependencies, make could just say something "oh, you don't have that software, well, i'll just drop it's support out completely" and continue building. Moreover, in gentoo there's a feature called USE-flags to specifically tell the toolchain that you want some select features."

And also, you can check your iptables.

Well , I'll build em packages again using yaourt . How can I save the build log for examining it later ?

Oh , and I don't use iptaples or any other firewalls for the time being , so its not the cause for this issue .

Cheers !

As you might have guessed, i'm not so familiar with arch's build system or whatever you've used(gentoo uses ebuilds where it's almost always just make $MAKE_OPTS, LFS uses plain make and make install, fedora uses rpmbuild which uses, if i remember right some special environment for make install call), i can't help you with extracting the build log. It's just a general advice to examine the build log when you change nothing but the compiler's version and something breaks.

If you haven't configured your kernel manually, you certainly do have iptables(or it's alternatives) and netfilter installed, so it doesn't hurt to check or even flush them all and see if it works.
"iptables --flush -t filter" should work, if i remember right.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5402 2015-06-22 11:55:04

supertrampx
Member
Registered: 2015-06-07
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
supertrampx wrote:

I have checked MSI enable on my GTx 970. In linux it reports MSI: enable-    in windows when I view the connection type pci, i see the card after driver install using msi.


I am getting this error occasionally after a freeze and vm crashes, usually when I am installing, or accessing the graphics card.

qemu-system-x86_64: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

any thoughts or ideas?

You got a fatal AER error.  Are you overclocking?  You can boot the host with pci=noaer to disable it, but that's no guarantee that the you're not still getting bus errors.




I am not overclocking. I will try the recommended pci=noaer tag.

also quick note, I found this response by another SR-2 Classified user on another forum perhaps it is of relevance?

"
Long answer: There is a bug in the NF200 PCIe bridges that will prevent you from allocating more than about 2.5GB of RAM (possibly less, depending on the hardware you have in the machine) to each VM.

It can be worked around if you are using the latest bleeding edge QEMU (used by both Xen and KVM). I've been running such a system for years, long before the required QEMU feature was added, but I had to bodge a patch onto hvmloader to make it mark all the low RAM between 1GB and 4GB as reserved to make it work. Otherwise when the VM accesses the RAM that overlaps with the PCI IOMEM region (aperture) it will trample the real IOMEM region, which will crash the machine, and if it is the disk controller's IOMEM it is trampling, it can corrupt your data, too."

Tips on how to reserve this ram, or patch the kernel to reserve it would be excellent!
Thanks in advanced, a fix on this would be awesome!
Passthrough works great, but freezes often due to these AER bugs

Last edited by supertrampx (2015-06-22 11:57:12)

Offline

#5403 2015-06-22 12:11:13

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

Long answer: There is a bug in the NF200 PCIe bridges that will prevent you from allocating more than about 2.5GB of RAM (possibly less, depending on the hardware you have in the machine) to each VM.

I'm going off memory of something that I noted but didn't look into too closely, but I think I did see this  occurring before I switched to Q35... that would explain why the available memory never looked right.

Edit:
self clarification if nothing else.... not saying that I have a nf200 pcie, just saying that that sounded similar to what I remember seeing.  So it's at least calming to me to see that it could have been related to the machine type I was using and misconfigured PCI root stuff.  Perhaps it was something else entirely.

Last edited by Blind Tree Frog (2015-06-22 14:37:40)

Offline

#5404 2015-06-22 12:12:30

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Denso wrote:

@Duelist :

I am certain that if there were problems with the building proccess , the whole thing would throw and error and ask if you would like to start from the begining .

Gentooist in me speaks again: "Nope. If you're missing some dependencies, make could just say something "oh, you don't have that software, well, i'll just drop it's support out completely" and continue building. Moreover, in gentoo there's a feature called USE-flags to specifically tell the toolchain that you want some select features."

Assuming the Makefile was written with such support. (as a general statement... i didn't pay attention to what we are building right now)

Offline

#5405 2015-06-22 13:39:21

nlamember
Member
Registered: 2015-06-22
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, Could anybody help me with my problem? Here is the situation: 1. Compute Node [Ubuntu 14.04.02, KVM, Kilo] with Nvidia Tesla K40 I passed it by using this method: https://wiki.openstack.org/wiki/Pci_passthrough I download all drivers and CUDA libraries + I compiled all sample files succesfully. However when I run them they run but in the end the do not finish sad. For example run log of deviceQuery:
----------------------------------------------
deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s) Device 0: "Tesla K40m"
//INFO ABOUT IT
Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery,
CUDA Driver = CUDART,
CUDA Driver Version = 7.0,
CUDA Runtime Version = 7.0,
NumDevs = 1,
Device0 = Tesla K40m
Result = PASS
|
--------------------------------------
And then it just hangs, only option is to ctrl+c. Moreover I installed everything on host too and there it finished sucessfully without any problems. Any help will be appreciated.

dmesg on VM says only:
[ 1475.225692] nvidia 0000:00:08.0: irq 51 for MSI/MSI-X
dmesg on host:
kernel: [ 2897.503162] vfio-pci 0000:02:00.0: irq 324 for MSI/MSI-X

Offline

#5406 2015-06-22 14:11:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've heard some news about XenGT(or KVMGT or whatever) in linux 4.1, but lkml doesn't hint me on details.
Can anyone give me a brief info about it's state? Does anybody knows about it?

Oh, and also, since we all need some very robust input to our VMs, there's a new way of doing this: virtio-input. If i understand correctly, it's inside 4.1 release now.

Last edited by Duelist (2015-06-22 14:16:25)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5407 2015-06-22 14:23:09

PrinzipDesSees
Member
Registered: 2015-06-04
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally got the x-vga working with a wrapper script, it was the stupidest ninja typo ever.

Many thanks @aw for all the information in the vfio blogspot and to OP and aw with info and explanation in this thread! Finally everything works like a charm.

@Denso

<bla>
And here in the gentoo handbook in english:
https://wiki.gentoo.org/wiki/Handbook:A … g_features

PORT_LOGDIR saves the entire logs into the specified path
The options with elog allow for filtering specific logclasses and further things.
</bla>

oops right, yaourt is from arch, not gentoo, I have mistaken it with the other package manager in gentoo. Ignore this.


EDIT:

@Duelist
That seems very interesting, I'll post here if I find something specific about it.

Here something on the xen mailinglist, seems they havent got it into the official kernel yet but there is a link to multiple github repos containing preview code.
They have also done a demo patch for using the thing on qemu. It seems however that this only works for intel graphics currently.
http://lists.xen.org/archives/html/xen- … 01848.html

They renamed the thing to Intel GVT
https://01.org/xen/blogs/wangbo85/2015/ … e-q12015-0

Last edited by PrinzipDesSees (2015-06-22 14:39:06)

Offline

#5408 2015-06-22 14:29:27

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PrinzipDesSees wrote:

@Denso

And here in the gentoo handbook in english:
https://wiki.gentoo.org/wiki/Handbook:A … g_features

PORT_LOGDIR saves the entire logs into the specified path
The options with elog allow for filtering specific logclasses and further things.

Dude, i'm certainly sure he uses arch, so your gentoo-specific advice is a bit off.
A gentooist in me spoke because the first thing you do when something breaks in gentoo is reading the build log. Same thing for LFS, so considering the fact he compiled his QEMU with a cutting-edge fresh GCC and stumbled into problems - maybe build log would be useful to check.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5409 2015-06-22 14:34:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

supertrampx wrote:

Long answer: There is a bug in the NF200 PCIe bridges that will prevent you from allocating more than about 2.5GB of RAM (possibly less, depending on the hardware you have in the machine) to each VM.

It can be worked around if you are using the latest bleeding edge QEMU (used by both Xen and KVM). I've been running such a system for years, long before the required QEMU feature was added, but I had to bodge a patch onto hvmloader to make it mark all the low RAM between 1GB and 4GB as reserved to make it work. Otherwise when the VM accesses the RAM that overlaps with the PCI IOMEM region (aperture) it will trample the real IOMEM region, which will crash the machine, and if it is the disk controller's IOMEM it is trampling, it can corrupt your data, too."

Tips on how to reserve this ram, or patch the kernel to reserve it would be excellent!
Thanks in advanced, a fix on this would be awesome!
Passthrough works great, but freezes often due to these AER bugs

This sounds like exactly what ACS and IOMMU groups are meant to protect.  Researching the NF200, it looks like it's specifically designed to optimize peer-to-peer traffic *without* forwarding it upstream and therefore without IOMMU interaction.  Can I assume that you need to use the ACS override patch to even assign devices behind the NF200 to separate VMs?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5410 2015-06-22 17:00:04

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally got VGA passthrough working stably with 4.1.0 kernel, VFI-PCI, OVMF ROM and manually loaded VGA ROM with EFI support. Thanks to aw's amazing blogspot posts smile

One issue, however, remains:
I'm trying to passthrough a X-Fi (PCI-E) sound card, since I just can't get sound passed through to the on-board sound card linux uses...
So I added the sound card (06:00.0 - alone in an IOMMU group) to the vfio-pci module and added the device in the virt-manager GUI (just like I do with the graphics card). After the OS (Win 8.1) installation and first boot the sound works fine with the built-in drivers. However, after installing the official X-Fi drivers for Win 8.1, my system seems to run into trouble handling the IRQ ("irq 18: nobody cared [...] Disabling IRQ #18"). The Windows loading screen then takes 2 minutes after the IRQ issue comes up (on first boot it takes more like 5 secs...).

Here is some (cropped to the essential info) output describing the problem and determining where the IRQ belongs to:

dmesg > http://ix.io/jeY (see the block [Jun22 16:33] at the very bottom)
/proc/interrupts > http://ix.io/jf0 (IRQ #18 handles the vfio-pci interrupt for the sound card, which is 06:00.1)
lspci -v > http://ix.io/jeZ (this are all the devices registered to IRQ #18)

Rebooting the host and reloading the vfio-pci kernel module don't produce different behavior, neither does uninstalling the X-Fi drivers again and/or installing the ones from windows update. I also reinstalled Win 8.1 like 4 times now, and everything happens exactly the same way. I don't really get what's going on there... :\

Last edited by CharlieBra7o (2015-06-22 17:34:58)

Offline

#5411 2015-06-22 17:15:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CharlieBra7o wrote:

Finally got VGA passthrough working stably with 4.1.0 kernel, VFI-PCI, OVMF ROM and manually loaded VGA ROM with EFI support. Thanks to aw's amazing blogspot posts smile

One issue, however, remains:
I'm trying to passthrough a X-Fi (PCI-E) sound card, since I just can't get sound passed through to the on-board sound card linux uses...
So I added the sound card (06:00.0 - alone in an IOMMU group) to the vfio-pci module and added the device in the virt-manager GUI (just like I do with the graphics card). After the OS (Win 8.1) installation and first boot the sound works fine with the built-in drivers. However, after installing the official X-Fi drivers for Win 8.1, my system seems to run into trouble handling the IRQ ("irq 18: nobody cared [...] Disabling IRQ #18"). The Windows loading screen then takes 2 minutes after the IRQ issue comes up (on first boot it takes more like 5 secs...).

Here is some (cropped to the essential info) output describing the problem and determining where the IRQ belongs to:

dmesg > http://ix.io/jeY (see the block [Jun22 16:33] at the very bottom)
/proc/interrupts > http://ix.io/jf0 (IRQ #18 handles the vfio-pci interrupt for the sound card, which is 06:00.1)
lspci -nnk > http://ix.io/jeZ (this are all the devices )

Rebooting the host and reloading the vfio-pci kernel module don't produce different behavior, neither does uninstalling the X-Fi drivers again and/or installing the ones from windows update. I also reinstalled Win 8.1 like 4 times now, and everything happens exactly the same way. I don't really get what's going on there... :\

Probably broken INTx masking on the sound card.  You can try using the nointxmask=1 option to vfio-pci, which can be kind of difficult to manage because you'll need to get exclusive interrupts for everything, or you could try something like this to just require an exclusive interrupt for the sound card:

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index c6dc1df..e0e6cb3 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3051,6 +3051,7 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_CHELSIO, 0x0030, quirk_broken_intx_masking); DECLARE_PCI_FIXUP_HEADER(0x1814, 0x0601, /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup Ralink RT2800 802.11n PCI */ quirk_broken_intx_masking); +DECLARE_PCI_FIXUP_HEADER(0x1102, 0x000b, quirk_broken_intx_masking); /1 /bin /boot /cdrom /dev /etc /home /initrd.img /initrd.img.old /lib /lib32 /lib64 /libx32 /lost+found /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /ubiquity-apt-clone /usr /var /vmlinuz /vmlinuz.old /win81-base.qcow2 /win81-base.qcow2.md5 /win81-writable-amd.qcow2 /win81-writable-nvidia.qcow2 /win81-writable-nvidia.qcow2.backup 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_212.html topic_213.html topic_214.html topic_215.html topic_216.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Realtek RTL8169 PCI Gigabit Ethernet Controller (rev 10) 1.sh topic_100.html topic_101.html topic_102.html topic_103.html topic_104.html topic_105.html topic_106.html topic_107.html topic_108.html topic_109.html topic_10.html topic_110.html topic_111.html topic_112.html topic_113.html topic_114.html topic_115.html topic_116.html topic_117.html topic_118.html topic_119.html topic_11.html topic_120.html topic_121.html topic_122.html topic_123.html topic_124.html topic_125.html topic_126.html topic_127.html topic_128.html topic_129.html topic_12.html topic_130.html topic_131.html topic_132.html topic_133.html topic_134.html topic_135.html topic_136.html topic_137.html topic_138.html topic_139.html topic_13.html topic_140.html topic_141.html topic_142.html topic_143.html topic_144.html topic_145.html topic_146.html topic_147.html topic_148.html topic_149.html topic_14.html topic_150.html topic_151.html topic_152.html topic_153.html topic_154.html topic_155.html topic_156.html topic_157.html topic_158.html topic_159.html topic_15.html topic_160.html topic_161.html topic_162.html topic_163.html topic_164.html topic_165.html topic_166.html topic_167.html topic_168.html topic_169.html topic_16.html topic_170.html topic_171.html topic_172.html topic_173.html topic_174.html topic_175.html topic_176.html topic_177.html topic_178.html topic_179.html topic_17.html topic_180.html topic_181.html topic_182.html topic_183.html topic_184.html topic_185.html topic_186.html topic_187.html topic_188.html topic_189.html topic_18.html topic_190.html topic_191.html topic_192.html topic_193.html topic_194.html topic_195.html topic_196.html topic_197.html topic_198.html topic_199.html topic_19.html topic_1.html topic_200.html topic_201.html topic_202.html topic_203.html topic_204.html topic_205.html topic_206.html topic_207.html topic_208.html topic_209.html topic_20.html topic_210.html topic_211.html topic_212.html topic_213.html topic_214.html topic_215.html topic_216.html topic_21.html topic_22.html topic_23.html topic_24.html topic_25.html topic_26.html topic_27.html topic_28.html topic_29.html topic_2.html topic_30.html topic_31.html topic_32.html topic_33.html topic_34.html topic_35.html topic_36.html topic_37.html topic_38.html topic_39.html topic_3.html topic_40.html topic_41.html topic_42.html topic_43.html topic_44.html topic_45.html topic_46.html topic_47.html topic_48.html topic_49.html topic_4.html topic_50.html topic_51.html topic_52.html topic_53.html topic_54.html topic_55.html topic_56.html topic_57.html topic_58.html topic_59.html topic_5.html topic_60.html topic_61.html topic_62.html topic_63.html topic_64.html topic_65.html topic_66.html topic_67.html topic_68.html topic_69.html topic_6.html topic_70.html topic_71.html topic_72.html topic_73.html topic_74.html topic_75.html topic_76.html topic_77.html topic_78.html topic_79.html topic_7.html topic_80.html topic_81.html topic_82.html topic_83.html topic_84.html topic_85.html topic_86.html topic_87.html topic_88.html topic_89.html topic_8.html topic_90.html topic_91.html topic_92.html topic_93.html topic_94.html topic_95.html topic_96.html topic_97.html topic_98.html topic_99.html topic_9.html Subsystem: Realtek RTL8169/8110 Family PCI Gigabit Ethernet NIC

(the -nn part of you lspci didn't seem to take, so double check that vendor:device ID).

You might also read the article on my blog about getting devices to use MSI, that's usually the better option for something as latency sensitive as audio anyway.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5412 2015-06-22 17:33:41

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

oops right, yaourt is from arch, not gentoo, I have mistaken it with the other package manager in gentoo. Ignore this.

No problem man , thanks for spending the time looking for this ! smile

Dude, i'm certainly sure he uses arch

For me , Arch is the only Linux out there . smile

Anyway , I'll try downgrading to GCC 4.9 and see if it would fix those issues . I'll keep you posted !

Edit : Sigh , isn't there a "Legacy" repo for Arch ? wink

Edit 2 :

So I finally captured the build log (GCC 5.1) , it threw a lot of warnings :

nbd.c: In function 'nbd_trip': nbd.c:1321:15: warning: 'request.len' may be used uninitialized in this function [-Wmaybe-uninitialized] ret = blk_read(exp->blk, ^ nbd.c:1385:54: warning: 'request.from' may be used uninitialized in this function [-Wmaybe-uninitialized] ret = blk_co_discard(exp->blk, (request.from + exp->dev_offset) ^ nbd.c:1291:18: warning: 'request.handle' may be used uninitialized in this function [-Wmaybe-uninitialized] reply.handle = request.handle; ^ nbd.c:1354:26: warning: 'request.type' may be used uninitialized in this function [-Wmaybe-uninitialized] if (request.type & NBD_CMD_FLAG_FUA) { ^ block/vmdk.c: In function 'vmdk_open_desc_file.isra.12': block/vmdk.c:848:39: warning: 'extent' may be used uninitialized in this function [-Wmaybe-uninitialized] extent->flat_start_offset = flat_offset << 9; ^ block/vmdk.c:776:17: note: 'extent' was declared here VmdkExtent *extent; ^ block/vmdk.c: In function 'vmdk_open_vmdk4': block/vmdk.c:701:24: warning: 'extent' may be used uninitialized in this function [-Wmaybe-uninitialized] extent->has_marker = le32_to_cpu(header.flags) & VMDK4_FLAG_MARKER; ^ block/vmdk.c: In function 'vmdk_open_sparse': block/vmdk.c:513:9: warning: 'extent' may be used uninitialized in this function [-Wmaybe-uninitialized] ret = vmdk_init_tables(bs, extent, errp); ^ block/vmdk.c:492:17: note: 'extent' was declared here VmdkExtent *extent; ^ hw/input/adb.c: In function 'adb_poll': hw/input/adb.c:81:23: warning: array subscript is above array bounds [-Warray-bounds] d = s->devices[i]; ^ hw/usb/hcd-ohci.c: In function 'ohci_service_ed_list.part.6': hw/usb/hcd-ohci.c:414:26: warning: array subscript is above array bounds [-Warray-bounds] if ((ohci->rhport[i].ctrl & OHCI_PORT_PES) == 0) { ^ hw/usb/hcd-ohci.c:414:26: warning: array subscript is above array bounds [-Warray-bounds] if ((ohci->rhport[i].ctrl & OHCI_PORT_PES) == 0) { ^ slirp/mbuf.c: In function 'm_cat': slirp/mbuf.c:136:1: warning: assuming signed overflow does not occur when assuming that (X + c) < X is always false [-Wstrict-overflow] m_cat(struct mbuf *m, struct mbuf *n) ^ slirp/mbuf.c:158:11: warning: assuming signed overflow does not occur when assuming that (X + c) < X is always false [-Wstrict-overflow] if(m->m_size>size) return; ^ slirp/socket.c: In function 'soread': slirp/socket.c:193:5: warning: 'n' may be used uninitialized in this function [-Wmaybe-uninitialized] if (n == 2 && nn == iov[0].iov_len) { ^ In file included from /usr/lib/glib-2.0/include/glibconfig.h:9:0, from /usr/include/glib-2.0/glib/gtypes.h:32, from /usr/include/glib-2.0/glib/galloca.h:32, from /usr/include/glib-2.0/glib.h:30, from /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/include/glib-compat.h:19, from /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/include/qemu-common.h:43, from slirp/socket.c:8: slirp/socket.c: In function 'soreadbuf': /usr/include/glib-2.0/glib/gmacros.h:240:39: warning: 'iov.iov_len' may be used uninitialized in this function [-Wmaybe-uninitialized] #define MIN(a, b) (((a) < (b)) ? (a) : (b)) ^ slirp/socket.c:215:15: note: 'iov.iov_len' was declared here struct iovec iov[2]; ^ slirp/socket.c: In function 'sorecvoob': slirp/socket.c:193:5: warning: 'n' may be used uninitialized in this function [-Wmaybe-uninitialized] if (n == 2 && nn == iov[0].iov_len) { ^ slirp/socket.c:153:6: note: 'n' was declared here int n, nn; ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c: In function 'get_real_device': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c:649:44: warning: 'id' may be used uninitialized in this function [-Wmaybe-uninitialized] pci_dev->dev.config[1] = (id & 0xff00) >> 8; ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c: In function 'assign_failed_examine': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c:771:12: warning: 'device_id' may be used uninitialized in this function [-Wmaybe-uninitialized] return g_strdup_printf( ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c:771:12: warning: 'vendor_id' may be used uninitialized in this function [-Wmaybe-uninitialized] In file included from /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-m68k/translate.c:23:0: /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-m68k/translate.c: In function 'disas_tas': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/tcg/tcg-op.h:58:5: warning: 'addr' may be used uninitialized in this function [-Wmaybe-uninitialized] tcg_gen_op2(&tcg_ctx, opc, GET_TCGV_I32(a1), GET_TCGV_I32(a2)); ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-m68k/translate.c:1507:10: note: 'addr' was declared here TCGv addr; ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-sh4/helper.c: In function 'superh_cpu_get_phys_page_debug': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-sh4/helper.c:527:12: warning: 'physical' may be used uninitialized in this function [-Wmaybe-uninitialized] return physical; ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-sh4/helper.c: In function 'superh_cpu_get_phys_page_debug': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-sh4/helper.c:527:12: warning: 'physical' may be used uninitialized in this function [-Wmaybe-uninitialized] return physical; ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c: In function 'get_real_device': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c:649:44: warning: 'id' may be used uninitialized in this function [-Wmaybe-uninitialized] pci_dev->dev.config[1] = (id & 0xff00) >> 8; ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c: In function 'assign_failed_examine': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c:771:12: warning: 'device_id' may be used uninitialized in this function [-Wmaybe-uninitialized] return g_strdup_printf( ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/hw/i386/kvm/pci-assign.c:771:12: warning: 'vendor_id' may be used uninitialized in this function [-Wmaybe-uninitialized] In file included from /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-m68k/translate.c:23:0: /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-m68k/translate.c: In function 'disas_tas': /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/tcg/tcg-op.h:58:5: warning: 'addr' may be used uninitialized in this function [-Wmaybe-uninitialized] tcg_gen_op2(&tcg_ctx, opc, GET_TCGV_I32(a1), GET_TCGV_I32(a2)); ^ /tmp/yaourt-tmp-root/aur-qemu-git/src/qemu/target-m68k/translate.c:1507:10: note: 'addr' was declared here TCGv addr; ^ libtool: warning: remember to run 'libtool --finish /usr/lib' libtool: warning: 'libcacard.la' has not been installed in '/usr/lib'

Anyone can make any sense of these ? and wether they're related to the bridged networking issues I'm having ?

Thanks !

Last edited by Denso (2015-06-22 17:52:02)

Offline

#5413 2015-06-22 22:45:29

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

(the -nn part of you lspci didn't seem to take, so double check that vendor:device ID).

Actually it was just -v. I just mistyped it coz I typed "-nnk" about 1000 times the last few days wink Edited my post accordingly...

aw wrote:

You might also read the article on my blog about getting devices to use MSI, that's usually the better option for something as latency sensitive as audio anyway.

I just tried MSI. The card supports it (according to lspci -v), and it gets enabled (/proc/interrupts and devmgmt.msc tell me that) after I set the DWORD in the registry. But with MSI there is still no sound... Even trying to (re)install the sound drivers with MSI enabled doesn't work - the problem is, that every driver I install (tried 3 different ones), deletes or sets the MSI DWORD to 0, so INTx is used again, resulting in no sound... Is there a trick to this?

The weirdest thing, however, is, that the sound works fine right after the installation (when MSI is turned OFF). Doesn't make any sense to me at all...

Offline

#5414 2015-06-22 22:50:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@CharlieBra7o

Maybe the card doesn't really work with MSI either, interrupts are surprisingly difficult for hardware to get right.  I'd pursue the broken intx masking path if MSI doesn't immediately work.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5415 2015-06-23 02:16:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone !

I removed the qemu-git package and installed the pre-compiled package from the Extra repository , and everything is working like it should !

So it surely has something to do with GCC 5 .

Regards .

Offline

#5416 2015-06-23 10:41:50

Volle
Member
Registered: 2013-12-05
Posts: 37

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

thanks for all this info in this thread but before i start to try this out i've got one question:
Is there a way to still render the guest in the qemu window?
I'm asking since i want to use this on my notebook without an additional screen connected.

Greetings
Chris

Offline

#5417 2015-06-23 12:02:17

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Volle wrote:

Hey guys,

thanks for all this info in this thread but before i start to try this out i've got one question:
Is there a way to still render the guest in the qemu window?
I'm asking since i want to use this on my notebook without an additional screen connected.

Greetings
Chris

Have multiple profiles for the same machine and you should get that (ie: same hard drive, different launch parameters).  That said, I'm fairly certain that Windows will not appreciate you doing this at all (since it doesn't like the hardware changing from under it).  Linux may not either, but I don't know offhand how much it will complain.

Offline

#5418 2015-06-23 12:17:40

Volle
Member
Registered: 2013-12-05
Posts: 37

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

Have multiple profiles for the same machine and you should get that (ie: same hard drive, different launch parameters).  That said, I'm fairly certain that Windows will not appreciate you doing this at all (since it doesn't like the hardware changing from under it).  Linux may not either, but I don't know offhand how much it will complain.

I think you didn't get me right tongue
What i want to do is that i never use a screen and only use the vm in a window ...
But i was reading Alex Williamson's blog (RTFM) and found my answer: no...

I think i misread this forum post (post #10 in https://forums.virtualbox.org/viewtopic … 7&t=64298)
I know its about virtualbox and not qemu but they should be similar.

So no vgpu passthrough for me :<

Thanks anyways

Chris

Offline

#5419 2015-06-23 12:27:54

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Volle wrote:

I think you didn't get me right tongue
What i want to do is that i never use a screen and only use the vm in a window ...
But i was reading Alex Williamson's blog (RTFM) and found my answer: no...

I think i misread this forum post (post #10 in https://forums.virtualbox.org/viewtopic … 7&t=64298)
I know its about virtualbox and not qemu but they should be similar.

So no vgpu passthrough for me :<

There is a reason that I didn't get you... that makes no sense in context of this discussion tongue


Yeah, no you just want to run QEMU (or Virtualbox, or VMWare) normally and ignore the passthrough stuff.   Technically if you had a second video card I guess you could run passthrough on it and then use VNC to push it to a window on your host, but then you are back to the host's graphics and lose any advantage of the guest having his own card.

Offline

#5420 2015-06-23 13:04:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

Technically if you had a second video card I guess you could run passthrough on it and then use VNC to push it to a window on your host, but then you are back to the host's graphics and lose any advantage of the guest having his own card.

That's not at all true, using an assigned GPU is a huge advantage for guest graphics, even if using something like a TightVNC server in the guest to connect.  Local connections via VNC also get the benefit that we have many Gbps of throughput using the internal bridge.  There are several videos on youtube using this model where games run a quite playable frame rates.

If your guest is Win8+, it's also possible to use both an assigned GPU and emulated graphics.  You can get 3D rendering into the emulated window, but the performance is significantly slower in this model.  The better approach is to use the emulated graphics for boot, disable it for the guest OS, and connect via guest VNC once running.  If you can't use that emulated + assigned GPU model, then you lose access to the VM until the guest VNC server is running.  This can work too, but if Windows ever wants to go into recovery mode, you'd need to remove the assigned GPU and add an emulated VGA to interact.

The more troubling problem in a laptop is that the secondary graphics in a hybrid laptop is rarely as isolated as a discrete GPU on a desktop.  As far as I'm aware, few, if any, have made it work.  The GPU ROM is often embedded in ACPI, so likely needs to be extracted by doing a dump, disassemble, and encoding of the ROM table.  Even then, mystery Code 43s seem to plaque dual graphics laptop users.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5421 2015-06-23 13:08:27

Volle
Member
Registered: 2013-12-05
Posts: 37

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

There is a reason that I didn't get you... that makes no sense in context of this discussion tongue

Yeah, no you just want to run QEMU (or Virtualbox, or VMWare) normally and ignore the passthrough stuff.   Technically if you had a second video card I guess you could run passthrough on it and then use VNC to push it to a window on your host, but then you are back to the host's graphics and lose any advantage of the guest having his own card.

I've got a notebook with an Intel HD chip and an Nvidia 940M
Host: arch using the Intel HD chip rendering the desktop
Guest: win8.1 using the Nvidia 940M

My intention was running a game on the Guest (since there is 'no' game support in linux yet)

[edit]
ninja'd by aw.
Yes this exactly what i want to do!
I'll give it a try (like you explained) maybe i can get it to work ....

thanks

Last edited by Volle (2015-06-23 13:11:57)

Offline

#5422 2015-06-23 13:12:20

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Blind Tree Frog wrote:

Technically if you had a second video card I guess you could run passthrough on it and then use VNC to push it to a window on your host, but then you are back to the host's graphics and lose any advantage of the guest having his own card.

That's not at all true, using an assigned GPU is a huge advantage for guest graphics, even if using something like a TightVNC server in the guest to connect.  Local connections via VNC also get the benefit that we have many Gbps of throughput using the internal bridge.  There are several videos on youtube using this model where games run a quite playable frame rates.

yes, "any advantage" was the wrong choice of words.  The guest still gets hardware acceleration, but there is a performance hit from the VNC interface between machines

Offline

#5423 2015-06-23 14:44:47

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did something happen to kraxel.org RPM git repo? My DNF seems to fail to update. I haven't investigated that yet.

Cannot download edk2/edk2.git-ovmf-x64-0-20150623.b1070.g37fe82e.noarch.rpm

Hmm, the rpm is in place, it's named differently:....b1071/g3cd2484
Seems like a successful build occured when i've ran dnf update.

Last edited by Duelist (2015-06-23 15:03:27)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5424 2015-06-23 15:02:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Did something happen to kraxel.org RPM git repo? My DNF seems to fail to update. I haven't investigated that yet.

Hmm.. Seems like jenkins feels itself bad - there's a jenkins spec, there's an ovmf build target, but

Cannot download edk2/edk2.git-ovmf-x64-0-20150623.b1070.g37fe82e.noarch.rpm

I'm not so familiar with jenkins, but something is broken.

Your repo data is out of date https://www.kraxel.org/repos/jenkins/edk2/


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5425 2015-06-23 15:04:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:

Did something happen to kraxel.org RPM git repo? My DNF seems to fail to update. I haven't investigated that yet.

Hmm.. Seems like jenkins feels itself bad - there's a jenkins spec, there's an ovmf build target, but

Cannot download edk2/edk2.git-ovmf-x64-0-20150623.b1070.g37fe82e.noarch.rpm

I'm not so familiar with jenkins, but something is broken.

Your repo data is out of date https://www.kraxel.org/repos/jenkins/edk2/

Yeah, i was right, i was trying to update the minute new build have appeared.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5426 2015-06-24 04:42:16

rb3
Member
Registered: 2012-09-17
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, I'm trying to set up the vfio passthrough on my mid-2014 rMBP, as I have documented here. I'm getting stuck at the "Testing if its working out" section of the original post on this thread. When I execute the indicated command (modified to provide my ROM file, which I'm not sure supports EFI), I get a QEMU window with a prompt, and no output in the connected monitor. Can anyone steer me in the right direction to get this to work? Thanks!

Offline

#5427 2015-06-24 04:45:26

ghormoon
Member
Registered: 2015-05-20
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

can you SSH to the guest to check if the device is listed there? if yes, get lspci -v to compare with host, if it looks ok or not.
EDIT: if you mean qemu with prompt as login prompt, you don't even need ssh.
but that brings me to another question - I assume the graphics are switchable,  not dedicated one for display, one for external one. you'd need to switch to the passed in graphics to see anything.

Last edited by ghormoon (2015-06-24 04:48:35)

Offline

#5428 2015-06-24 12:51:10

CharlieBra7o
Member
Registered: 2014-03-01
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@CharlieBra7o

Maybe the card doesn't really work with MSI either, interrupts are surprisingly difficult for hardware to get right.  I'd pursue the broken intx masking path if MSI doesn't immediately work.

Quick update on the X-Fi issue:

While trying the nointmask=1 option, I found out, that for some reason my modprobe.d conf file doesnt even get loaded on boot... Nevertheless, with nointmask=1 enabled, the VM can't bind the PCI-E device, so I discarded that option and tried the MSI option again..:

I found out, that once this "IRQ n: nobody cared [...] Disabling IRQ #n" error occurs (ie. when I install some sort of official X-Fi drivers, there is no way (reboot/reinstall) of getting sound again, until the host gets rebooted. Then, after a host reboot and a Windows reinstall, the X-Fi works again without MSI. Without installing any official X-Fi drivers and using Windows' built-in "HDA" drivers, which give me sound, and with MSISupported=0 I can then reboot the VM and no IRQ issues come up. Trying MSISupported=1 successfully enables MSI and the sound still works after rebooting the VM.

The problem with the X-Fi is, that it has this so-called "FlexiJack" input, which defaults to "Line-In" and can only be set to "Microphone" with the official X-Fi drivers (at least as far as my research goes). So without the official drivers, there is no way of using a Microphone except connecting a front panel via the "HD audio" pin connectors on the sound card with front panel headphones/microphone connectors - which can be used right away, without prior configuration. My front panel, unfortunally, introduces a lot of noise into the signal, which makes me crazy... Nevertheless, I'm now gonna stick to this somewhat 'dirty' workaround...

Offline

#5429 2015-06-24 15:36:09

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've not had these issues with my Sound Blaster Z. Windows won't install any default drivers. But once I install the Creative Drivers it works. That was me using Windows 7.

Offline

#5430 2015-06-25 15:24:40

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just a quick update in case anyone has the same problem:  I had a working Seabios qemu command line q35 Windows 7 which eventually I converted over to OVMF/libvirt/440fx, no reinstall, though I had to "reauthenticate" with Microsoft a couple of times.  Everything worked fine except for XP mode which quit with a cryptic "Unable to start server message" (see 5122-5125)

Windows Virtual PC Cannot start Windows Virtual PC Host Process. Check the System event log for more details Server execution failed [OK] The Event log had: The server {567DUXXX1-4678-A214-61DXXXXXXXXX} did not register with DCOM within the required timeout

I noticed that since the change to OVMF etc, Windows now thinks I have a Xeon E312x instead of a i7-4930k (which is the actual hardware). I had tried changing to "copy host configuration" and incorporated the other changes suggested by aw, but had no luck with XP mode and when I tried changing to cpu host got a bluescreen crash.  I then noticed that the default configuration in libvirt was 6 sockets, 1 core and 1 thread, and changed it manually to 1 socket, 6 cores and 1 thread.  Figured a i7 system should only have one socket wink   Voila, no bluescreen with cpu host: Windows still thinks I have a Xeon, Device Manager is unchanged with 6 cpus, but now XP mode works again. Performance seems the same regardless.

Last edited by mostlyharmless (2015-06-25 15:26:43)

Offline

#5431 2015-06-25 16:25:42

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

XP mode

You need to enable "Nested Virtualization" :

$ nano /etc/modprobe.d/nested.conf options kvm_intel nested=1 options kvm_intel enable_shadow_vmcs=1

reboot and test XP Mode smile

Edit : Oops , didn't realize you got it working already !

Last edited by Denso (2015-06-25 16:26:52)

Offline

#5432 2015-06-25 17:00:34

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, yeah, I already had those things enabled, furthermore, XP mode had worked before on the old setup, so it couldn't have been a host configuration problem, it had to be a VM configuration problem, logically speaking....

Offline

#5433 2015-06-26 01:47:40

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mostlyharmless wrote:

I noticed that since the change to OVMF etc, Windows now thinks I have a Xeon E312x instead of a i7-4930k (which is the actual hardware).

 <cpu mode='host-passthrough'> <topology sockets='SOCKETS' cores='CORES PER SOCKET' threads='THREADS PER CORE'/> </cpu>

So it'd be 1-4-2 for a full, real 4930k since you have hyper-threading.
Or 1-3-2 for your 6 vCPU config.

But the main thing is - host-passthrough cpu mode, which fixed my CPUID to the real one.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5434 2015-06-26 08:30:15

Peetipablo
Member
Registered: 2015-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've been banging my head against a wall trying to get this to work for a few days with no real success.  I'm able to get a VM up and running, but when I try to turn vga off and pass my graphics card to the VM I don't get any signal at all.  If anyone could take a quick glance at my setup and point me in the right direction, I'd be grateful.

CPU: Intel i5 4690k
GPU: AMD r9 290
Mobo: Asrock Z97 Extreme 4
Memory: 16GB 1600Mhz
Monitor1: Asus 144hz, Displayport ------> IGPU, DVI-D ----------> Discrete GPU
Monitor 2:  Random 60hz, HDMI -----> IGPU

Trying to use my IGP as a host display adapter, and pass through my r9 290 to windows 8.1 VM.

Vfio-bind script:

#!/bin/bash #!/bin/sh DEVS="0000:01:00.0 0000:01:00.1" for DEV in $DEVS; do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override done modprobe -i vfio-pci

Vfio-bind script is working correctly (I believe):

[peetipablo@donbot ~]$ tree /sys/bus/pci/devices/0000\:01\:00.0/driver /sys/bus/pci/devices/0000:01:00.0/driver ├── 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 ├── 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 ├── bind ├── module -> ../../../../module/vfio_pci ├── new_id ├── remove_id ├── uevent └── unbind

pci-stub enabled in /etc/mkinitcpio.conf, stub.ids set in /etc/default/grub and bootloader/kernel updated, will attach output if requested.

Starting up in either virt-manager, or from console it seems that the VM is booted and running fine, just zero video output.

/var/log/libvirt/qemu/win8.1.log:

2015-06-26 08:03:16.529+0000: starting up libvirt version: 1.2.16, qemu version: 2.3.0 LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name win8.1 -S -mac$ qemu: terminating on signal 15 from pid 2107 2015-06-26 08:08:44.561+0000: shutting down

n2ZbjUH.png

Relavent journalctl:

journalctl -b: Jun 26 04:26:23 donbot libvirtd[2107]: failed to create directory '/var/lib/libvirt/images': File exists Jun 26 04:26:25 donbot NetworkManager[608]: <info> (vnet0): carrier is OFF Jun 26 04:26:25 donbot NetworkManager[608]: <info> (vnet0): new Tun device (driver: 'unknown' ifindex: 6) Jun 26 04:26:25 donbot NetworkManager[608]: <info> (vnet0): exported as /org/freedesktop/NetworkManager/Devices/5 Jun 26 04:26:25 donbot NetworkManager[608]: <info> (virbr0): bridge port vnet0 was attached Jun 26 04:26:25 donbot NetworkManager[608]: <info> (vnet0): enslaved to virbr0 Jun 26 04:26:25 donbot NetworkManager[608]: <error> [1435307185.502845] [devices/nm-device.c:2086] nm_device_generate_connection(): (v Jun 26 04:26:25 donbot kernel: device vnet0 entered promiscuous mode Jun 26 04:26:25 donbot NetworkManager[608]: <info> (vnet0): link connected Jun 26 04:26:25 donbot kernel: virbr0: port 2(vnet0) entered listening state Jun 26 04:26:25 donbot kernel: virbr0: port 2(vnet0) entered listening state Jun 26 04:26:25 donbot dbus[612]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop. Jun 26 04:26:25 donbot systemd[1]: Starting Virtual Machine and Container Registration Service... Jun 26 04:26:25 donbot dbus[612]: [system] Successfully activated service 'org.freedesktop.machine1' Jun 26 04:26:25 donbot systemd[1]: Started Virtual Machine and Container Registration Service. Jun 26 04:26:25 donbot systemd-machined[2906]: New machine qemu-win8.1. Jun 26 04:26:25 donbot systemd[1]: Started Virtual Machine qemu-win8.1. Jun 26 04:26:25 donbot systemd[1]: Starting Virtual Machine qemu-win8.1. Jun 26 04:26:25 donbot kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270 Jun 26 04:26:25 donbot kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0 Jun 26 04:26:27 donbot ntpd[1053]: bind(23) AF_INET6 fe80::fc54:ff:feab:ec15%6#123 flags 0x11 failed: Cannot assign requested address Jun 26 04:26:27 donbot ntpd[1053]: unable to create socket on vnet0 (10) for fe80::fc54:ff:feab:ec15%6#123 Jun 26 04:26:27 donbot ntpd[1053]: failed to init interface for address fe80::fc54:ff:feab:ec15%6 Jun 26 04:26:27 donbot avahi-daemon[2477]: Registering new address record for fe80::fc54:ff:feab:ec15 on vnet0.*. Jun 26 04:26:27 donbot kernel: virbr0: port 2(vnet0) entered learning state Jun 26 04:26:28 donbot sudo[2877]: pam_unix(sudo:session): session closed for user root Jun 26 04:26:29 donbot ntpd[1053]: Listen normally on 11 vnet0 [fe80::fc54:ff:feab:ec15%6]:123 Jun 26 04:26:29 donbot ntpd[1053]: new interface(s) found: waking up resolver Jun 26 04:26:29 donbot NetworkManager[608]: <info> (virbr0): link connected Jun 26 04:26:29 donbot kernel: virbr0: topology change detected, propagating Jun 26 04:26:29 donbot kernel: virbr0: port 2(vnet0) entered forwarding state

Offline

#5435 2015-06-26 10:10:16

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Peetipablo wrote:

So I've been banging my head against a wall trying to get this to work for a few days with no real success.  I'm able to get a VM up and running, but when I try to turn vga off and pass my graphics card to the VM I don't get any signal at all.  If anyone could take a quick glance at my setup and point me in the right direction, I'd be grateful.

CPU: Intel i5 4690k
GPU: AMD r9 290
Mobo: Asrock Z97 Extreme 4
Memory: 16GB 1600Mhz
Monitor1: Asus 144hz, Displayport ------> IGPU, DVI-D ----------> Discrete GPU
Monitor 2:  Random 60hz, HDMI -----> IGPU

Trying to use my IGP as a host display adapter, and pass through my r9 290 to windows 8.1 VM.


Well.. That vfio-bind script is not needed when libvirt is used.

Here lies a fresh and working guide on how to set it up. The information on that blog is 100% accurate, as it's written by the author of vfio module.
There may be some details with i915(for IGPU) driver and 9-series chipsets, but i'm not familiar with them. It'd be nice if you provide your distro and kernel versions.

Last edited by Duelist (2015-06-26 10:20:55)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5436 2015-06-26 12:44:40

Peetipablo
Member
Registered: 2015-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm running latest updated  arch, and have been trying both the linked linux-vfio kernel on page 1 of this thread and linux-mainline.  I actually was referencing that blog last night. I was under the impression that the i915 fix wasn't necessary if booting with UEFI.

Offline

#5437 2015-06-26 13:14:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's a little bit unclear to me what you're doing with pci-stub versus that vfio bind script.  As I document in my howto series, there are ways to use scripts like that in your initramfs to avoid pci-stub, but not generally in combination with pci-stub.  You are apparently getting the device bound to vfio-pci in the end though.  Your paste of the libvirt log is truncating the qemu commandline, so there's really nothing to see there.

The i915 patch is only required for VGA-mode, it should not be necessary for OVMF.  Did you check whether the ROM on your assigned GPU supports UEFI?

Since the GPU is connected at the processor root port, the 9-series PCH root port quirks are also not relevant.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5438 2015-06-26 13:38:46

Peetipablo
Member
Registered: 2015-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Alex,

Thanks for the reply.  I was reading your how-to series last night and was (attempting) try different things, the vfio-bind script is from your writeup.

Apologies, but I did not check to ensure my GPU has a UEFI-supporting ROM.  To be honest, I just assumed since it was a newer card it would support UEFI.  I'm not really sure how to check.

Edit:

it looks like reference 290 cards have only legacy bios roms.  I'm attempting to find and flash a UEFI rom so we'll see if that makes a difference.  Thanks again for the help, much appreciated.

Edit2:

I flashed a UEFI rom on my card and changed a few things around.  I removed pci-stub and the vfio-pci script to bind, and followed the guide to load vfio-pci directly in the kernel.  Still using linux-mainline 4.1 kernel.

/etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt" GRUB_PRELOAD_MODULES="vfio-pci part_gpt part_msdos"

/etc/mkinitcpio.conf:

MODULES="nls_cp437 xhci_hcd vfio vfio_iommu_type1 vfio_pci vfio_virqfd" FILES="/etc/modprobe.d/radeon_blacklist.conf /etc/modprobe.d/vfio-options.conf"

/etc/modprobe.d/radeon_blacklist.conf:

blacklist radeon

/etc/modprobe.d/vfio-options.conf:

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

lsmod:

[peetipablo@donbot ~]$ lsmod|less [peetipablo@donbot ~]$ lsmod|grep vfio vfio_pci 36864 0 vfio_virqfd 16384 1 vfio_pci vfio_iommu_type1 20480 0 vfio 24576 2 vfio_iommu_type1,vfio_pci

vfio_pci bound:

[peetipablo@donbot ~]$ sudo tree /sys/bus/pci/devices/0000\:01\:00.0/driver [sudo] password for peetipablo: /sys/bus/pci/devices/0000:01:00.0/driver ├── 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 ├── 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 ├── bind ├── module -> ../../../../module/vfio_pci ├── new_id ├── remove_id ├── uevent └── unbind

Before I flashed UEFI bios I just wouldn't get any output on my DVI(dedicated GPU) with 01:00.0 and 01:00.1 bound to the VM, now I don't get any output AND my system freezes and I have to hard reboot lol  It seems I'm making progress in the wrong direction?

Anyway, if anyone has any idea or could point me in the right direction it'd be extremely appreciated.  I'm extremely new to Arch, I jumped in headfirst for the first time and built my system last weekend and I've learned so much in the short week I've been using it.  I feel like I've spent dozens of hours trying to get this to work and I'm not sure what else to do. sad

Last edited by Peetipablo (2015-06-26 18:51:27)

Offline

#5439 2015-06-26 21:01:36

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Peetipablo

I'm not sure what you were doing with pci-stub and vfio-pci previously, but it seemed like you ended up with the device attached to vfio-pci.  If that's true, then the biggest change is the GPU ROM, so that's what I would be suspicious of first.  Can you flash it back to the original ROM and get the original problem?  Did the ROM come from the GPU card vendor?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5440 2015-06-26 22:14:38

Peetipablo
Member
Registered: 2015-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW,

I flashed the backup I made of the rom, and am not freezing anymore.  Still no output from discreet GPU video.  The ROM was from MSI (gpu vendor).

Offline

#5441 2015-06-26 22:19:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guess you're stuck with VGA-mode then


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5442 2015-06-26 22:32:54

Peetipablo
Member
Registered: 2015-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guess you're right smile I just tried VGA mode and I was able to pass the screen to DVI-D.

Thanks a bunch for your help, I'm sure you get a million people asking for help w/ this setup every day.

Offline

#5443 2015-06-27 03:55:50

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any update on the modprobe.d vfio-pci ids not loaded on boot ?

I'm starting to suspect that compiling the kernel with GCC 5.1 is causing that , just like compiling the qemu package using GCC 5.1 caused all the bridging issues in the past week for me .

Let's hope that Arch would push 4.1 to the Core repo. soon smile

Offline

#5444 2015-06-27 09:26:16

walkindude
Member
Registered: 2012-03-04
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

windows 10 build 10130
all running well on sabertooth 990fx r2.0, fx 8320, 8 gb ram, r9 270
except from time to time I got freezes (vm and host also).. but sometimes, if I'm listening for the music on host sound continues or loops (like 50/50)
in /var/log/messages I got this

Jun 27 03:35:50 localhost kernel: [ 4038.876277] general protection fault: 0000 [#1] SMP Jun 27 03:35:50 localhost kernel: [ 4038.877219] Modules linked in: hidp vhost_net vhost macvtap macvlan ecb tun act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_nat ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set xt_NFLOG nfnetlink_log xt_LOG nf_log_ipv4 nf_log_common nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_defrag_ipv6 xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_pkttype xt_physdev br_netfilter bridge stp llc xt_owner xt_NFQUEUE xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink rfcomm fuse ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter arc4 mousedev joydev btusb bluetooth hid_generic psmouse serio_raw atkbd libps2 ath9k_htc evdev led_class ath9k_common ath9k_hw ath kvm_amd mac80211 amdkfd amd_iommu_v2 kvm radeon crct10dif_pclmul crc32_pclmul crc32c_intel cfg80211 ghash_clmulni_intel snd_hda_codec_realtek aesni_intel aes_x86_64 glue_helper snd_hda_codec_generic lrw gf128mul ablk_helper cryptd rfkill snd_hda_codec_hdmi i2c_algo_bit pcspkr fam15h_power snd_hda_intel drm_kms_helper k10temp snd_hda_controller usbhid ttm r8169 hid snd_hda_codec drm snd_hwdep snd_pcm sp5100_tco mii snd_timer i2c_piix4 snd i2c_core soundcore shpchp tpm_infineon tpm_tis i8042 serio tpm acpi_cpufreq processor button sch_fq_codel nls_iso8859_1 nls_cp437 vfat fat usblp it87 hwmon_vid hwmon nfsd nfs auth_rpcgss oid_registry nfs_acl lockd grace sunrpc fscache ip_tables x_tables ext4 crc16 mbcache jbd2 dm_mod sd_mod ohci_pci ahci xhci_pci ohci_hcd ehci_pci libahci xhci_hcd ehci_hcd libata usbcore usb_common scsi_mod Jun 27 03:35:50 localhost kernel: [ 4038.892624] CPU: 3 PID: 3715 Comm: qemu-system-x86 Tainted: G A 4.0.6-1-uksm-ck #1 Jun 27 03:35:50 localhost kernel: [ 4038.894141] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2501 04/08/2014 Jun 27 03:35:50 localhost kernel: [ 4038.895671] task: ffff8801eee5ee40 ti: ffff8801e736c000 task.ti: ffff8801e736c000 Jun 27 03:35:50 localhost kernel: [ 4038.897240] RIP: 0010:[<ffffffff8107138d>] [<ffffffff8107138d>] finish_wait+0x48/0x64 Jun 27 03:35:50 localhost kernel: [ 4038.898807] RSP: 0018:ffff8801e736fcb8 EFLAGS: 00010046 Jun 27 03:35:50 localhost kernel: [ 4038.900365] RAX: 0000000000000292 RBX: ffff8801e736fce8 RCX: 08ca6ab8cb26e9a3 Jun 27 03:35:50 localhost kernel: [ 4038.901924] RDX: a0369728737bf719 RSI: 0000000000000292 RDI: ffff8801e7370080 Jun 27 03:35:50 localhost kernel: [ 4038.903521] RBP: ffff8801e736fcd8 R08: 0000000000000292 R09: ffff88044ecca060 Jun 27 03:35:50 localhost kernel: [ 4038.905098] R10: 0000000000000000 R11: 000003ac41530809 R12: ffff8801e7370080 Jun 27 03:35:50 localhost kernel: [ 4038.906690] R13: ffff8801e736fd00 R14: 000003ac13433b48 R15: 0000000000000000 Jun 27 03:35:50 localhost kernel: [ 4038.908312] FS: 00007f91014f2700(0000) GS:ffff88044ecc0000(0000) knlGS:00000000ec2b6b40 Jun 27 03:35:50 localhost kernel: [ 4038.909911] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 27 03:35:50 localhost kernel: [ 4038.911512] CR2: 0000000000000000 CR3: 00000002370b7000 CR4: 00000000000407e0 Jun 27 03:35:50 localhost kernel: [ 4038.913149] Stack: Jun 27 03:35:50 localhost kernel: [ 4038.914759] ffff8801eee5ee40 ffff8801e7370000 0000000000000001 ffff8801e7370080 Jun 27 03:35:50 localhost kernel: [ 4038.916406] ffff8801e736fd38 ffffffffa0653e15 0000000000000000 ffff8801eee5ee40 Jun 27 03:35:50 localhost kernel: [ 4038.918069] ffffffff8107174e ffff8801e736fd00 ffff8801e736fd00 00000000f45e99a5 Jun 27 03:35:50 localhost kernel: [ 4038.919722] Call Trace: Jun 27 03:35:50 localhost kernel: [ 4038.921373] [<ffffffffa0653e15>] kvm_vcpu_block+0xea/0x13f [kvm] Jun 27 03:35:50 localhost kernel: [ 4038.923064] [<ffffffff8107174e>] ? wait_woken+0x76/0x76 Jun 27 03:35:50 localhost kernel: [ 4038.924731] [<ffffffffa0755ff3>] ? svm_scale_tsc+0x49/0x49 [kvm_amd] Jun 27 03:35:50 localhost kernel: [ 4038.926417] [<ffffffffa0667f24>] kvm_arch_vcpu_ioctl_run+0x730/0xfa6 [kvm] Jun 27 03:35:50 localhost kernel: [ 4038.928127] [<ffffffffa0662abe>] ? kvm_arch_vcpu_load+0x13e/0x148 [kvm] Jun 27 03:35:50 localhost kernel: [ 4038.929818] [<ffffffffa0654c3f>] kvm_vcpu_ioctl+0x185/0x51a [kvm] Jun 27 03:35:50 localhost kernel: [ 4038.931517] [<ffffffff810985c3>] ? do_futex+0xe2/0x843 Jun 27 03:35:50 localhost kernel: [ 4038.933188] [<ffffffff810b3632>] ? acct_update_integrals+0x1c/0x1e Jun 27 03:35:50 localhost kernel: [ 4038.934878] [<ffffffff81211c96>] ? __next_cpu+0x1a/0x29 Jun 27 03:35:50 localhost kernel: [ 4038.936584] [<ffffffff8106c670>] ? check_smt_siblings+0x3f/0xa4 Jun 27 03:35:50 localhost kernel: [ 4038.938283] [<ffffffff81159f9e>] do_vfs_ioctl+0x3a0/0x42a Jun 27 03:35:50 localhost kernel: [ 4038.940008] [<ffffffff8115a082>] SyS_ioctl+0x5a/0x7f Jun 27 03:35:50 localhost kernel: [ 4038.941700] [<ffffffff8141e649>] system_call_fastpath+0x12/0x17 Jun 27 03:35:50 localhost kernel: [ 4038.943388] Code: 00 00 4c 8d 6e 18 4c 3b 6e 18 75 06 4c 3b 6e 20 74 2d 48 89 f3 49 89 fc e8 4c cb 3a 00 48 8b 4b 18 48 89 c6 48 8b 53 20 4c 89 e7 <48> 89 51 08 48 89 0a 4c 89 6b 18 4c 89 6b 20 e8 ab cc 3a 00 58 Jun 27 03:35:50 localhost kernel: [ 4038.948810] RSP <ffff8801e736fcb8> Jun 27 03:35:50 localhost kernel: [ 4038.958710] ---[ end trace 74a2ffa0673c39e8 ]---

tried with bfs and cfs, voluntary preemtion, 1000 hz and 300 hz, problem remains

Offline

#5445 2015-06-27 13:33:10

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 03:05.0 Multimedia audio controller: Ensoniq 5880B [AudioPCI] (rev 02) 04:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1) 04:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1) 05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) 07:00.0 SATA controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

You might say something like "that guy is outright insane", but..
The third card is coming somewhere on monday. Yeah.
UV2lvIi.jpg
(sorry for the big image)

So the riser works. The motherboard is F2A55 by asus, so there's only 2 PCI slots left free. Push it to the limit.
So far crossfire on two cards works, built-in audio is passed-through to the VM, hardware mixering still WIP, but everything works despite common sense.

After the third card the next step to insanity would be buying something with pci-e 3.0 and an APU to swap that GT610 with a fourth card or maybe try passing through the APU's graphics to achieve so called Dual Graphics, a special type of XDMA crossfire.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5446 2015-06-27 16:26:39

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist :

I wish there were high-end single-slot GPUs . I want to virtualize all of the house's PCs , but I have no PCI-E slots remaining , because both of my GPUs are dual-slot .

Oh and I wish there were PCI-E lanes splitters or something . That would allow you to split 16x into 2 8x . and 8x into 2 4x .

Is that a Fractal Design Core 1000 ? That case used to huse my gaming rig before I virtualize it . smile

Last edited by Denso (2015-06-27 16:30:11)

Offline

#5447 2015-06-27 16:51:44

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist

FYI Tried 1-3-2 and everything still works, the same. I've got host-passthrough, but Windows still thinks I have a Xeon.  I assume that's a Windows 7/OVMF/UEFI thing.  Not that it matters.

Last edited by mostlyharmless (2015-06-27 16:52:17)

Offline

#5448 2015-06-27 18:02:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

Quadro K4200 has pretty impressive specs for a single slot card, but at a huge premium.  On the plus side, they actually support VM usage models on it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5449 2015-06-27 19:00:12

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Aren't Quadros optimized for rendering and workstation workloads rather than gaming ?

Hmmm , $600 for a used one seems reasonable enough ! Now let's see how it handles gaming ... Off to find some reviews smile

Oh , it seems that the newer Quadros are dual-slot only . (5000 & 6000) .

Last edited by Denso (2015-06-27 19:07:12)

Offline

#5450 2015-06-27 19:40:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

Aren't Quadros optimized for rendering and workstation workloads rather than gaming ?

Hmmm , $600 for a used one seems reasonable enough ! Now let's see how it handles gaming ... Off to find some reviews smile

Oh , it seems that the newer Quadros are dual-slot only . (5000 & 6000) .

@Denso

The 4th digit of the Quadro numbers are equivalent to the 2nd digit of Geforce numbers, bigger means more cores.  The 3rd digit represents the generation.  So a K4200 is newer than a K5000, but all of the Quadros in the Kx200 generation got a big bump in core count, so a K4200 may be more equivalent in performance to a K5000 than to a K4000.  IOW, K5000 and K6000 aren't newer than K4200, they're the higher core count versions of the previous generation.

For gaming, I don't know how the Geforce drivers compare to the Quadro drivers.  I wouldn't expect to see much difference.  Quadros typically aren't clocked as high as Geforce, so aim your sights a little lower for equivalent generation and core count.  You might also lose Geforce Experience, automatic game tuning and the other features it provides.

EDIT: If you're expecting to find a single slot Titan, it's not going to happen.  Even Quadro goes to dual-slot after the K4xxx size, but they pack more into a single slot than anything else.

Last edited by aw (2015-06-27 19:41:50)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5451 2015-06-27 22:54:43

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Thanks for taking the time to provide such information . You rock ! smile

And yes , only K4s are single-slot sad

Maybe Pascal would allow them to cram more power into the single-slot spec.

Offline

#5452 2015-06-28 19:34:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@Duelist :

I wish there were high-end single-slot GPUs . I want to virtualize all of the house's PCs , but I have no PCI-E slots remaining , because both of my GPUs are dual-slot .

Oh and I wish there were PCI-E lanes splitters or something . That would allow you to split 16x into 2 8x . and 8x into 2 4x .

Is that a Fractal Design Core 1000 ? That case used to huse my gaming rig before I virtualize it . smile

http://www.cirrascale.com/blog/wp-conte … _Riser.jpg
If i only had pci-e 3.0, i'd stuff my pc with cards like a real mad man.

The case is.. wait for it... GMC High Five why-the-hell-i-bought-it-in-the-first-place edition.
I'm dreaming of some horizontal-mount case, but all i see is some server 4U cases, some CM HAF XB and some other company that made horizontal or "flipped"(io ports facing down or up) mounts for the motherboard. Because, obviously, all my GPUs are passively cooled, they would really be cooler if air convection worked the right way.

Well.. Risers will help you with your pci-e slots problem.
Buy something that has an insane amount of pci-e 16x slots and a bunch of regular risers. And a bunch of that cirrascale pieces of awesome, and... push the CPU's limit on pci-e lanes. My CPU supports up to 24 lanes, but there's no such motherboard.
And remember about the power supply - i'm pushing my not so fresh 550 FSP to it's limit(if i will overclock my cpu and try to burn something on the dvd writer(which i occasionally have) while mining/bruteforcing something on the radeons and play quake on linux host, it'll rack up a total of 600W). Gladly, i have a soldering iron, an oscilloscope and a bunch of stuff to repair most of the electrical damage that may happen. I hope.

BTW, the "splitter", afaik, is called a "bridge". And yea, it really does split. Examine that cirrascale thingie, it's interesting.


http://vr-zone.com/uploads/14361/P1020933.jpg?4285f0
That(or something else with less hilarious design). With that cirrascale things. 7x4=28 GPUs total. That would be insanely awesome.
But what if we go deeper and put that "riser" into it's clone, making 7 ports out of one?
I guess ACS and bridges and stuff would go batshit insane, but heck it'd be awesome. The PC would look like some nuclear power plant, especially if the GPUs would be something Fiji-based with that huge water coolers.
Oh well, sweet insane dreams and fantasies.


...whoa, that PLX PEX 8780 is 80-Lane, 20-Port PCI Express Gen 3 (8 GT/s) Switch. 80-lane. 20 port. Just wire it out right. I guess aw will shatter my dreams by saying that this switch doesn't do ACS or something else.



-- mod edit: read the Forum Etiquette and only post thumbnails http://wiki.archlinux.org/index.php/For … s_and_Code [jwr] --

Last edited by Duelist (2015-06-28 19:46:55)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5453 2015-06-28 20:07:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

...whoa, that PLX PEX 8780 is 80-Lane, 20-Port PCI Express Gen 3 (8 GT/s) Switch. 80-lane. 20 port. Just wire it out right. I guess aw will shatter my dreams by saying that this switch doesn't do ACS or something else.

From the product brief:

 PEX8780 Key Features
o Standards Compliant
- PCI Express Base Specification, r3.0
(compatible w/ PCIe r1.0a/1.1 & 2.0)
- PCI Power Management Spec, r1.2
- Microsoft Windows Logo Compliant
- Supports Access Control Services
- Dynamic link-width control
- Dynamic SerDes speed control

Kudos to PLX

Edit: GRID K1 & K2 use PLX PEX 8747 and have proper ACS isolation.

Last edited by aw (2015-06-28 20:13:01)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5454 2015-06-29 00:11:14

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Duelist :

That's one nice post man !! My ASUS X99-E WS has 2 PLX 8747 bridges (7 Slots) and I'm filling them up already .

Why don't you get a case like mine ? wink

http://i.imgur.com/imkMJy9.jpg

The SAS backplanes are good quality (for mine anyway) , but the overall plastic parts are cheap and prone to crack . You can see in the image , the power button's cover is gone and one of the HDD trays is cracked .

@aw : PLX bridges are good for VFIO and passthrough , but some motherboards makers screw it up somehow , like my Z77 from ASRock . Used to crash hard whenever I launch the VM . Now with my current ASUS X99-E WS everything behind the bridges works like a charm as if its tied directly to the CPU's Root .

Anyway , I've finally purchased an SSD to store my RAW VM images (created using truncate) , what is the best Filesystem to use for that SSD ?



-- mod edit: read the Forum Etiquette and only post thumbnails http://wiki.archlinux.org/index.php/For … s_and_Code [jwr] --

Offline

#5455 2015-06-29 04:10:37

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That was actually something that I was curious about. I have 2 Samsung 840 Pros in a mdadm RAID 0, but I swear it hasn't really improved performance any, if anything I notice lag in games. Such as loading textures, or jumping around to view different players on a map, it takes a split second for things to load.

Ive had a stable setup for some time now, but need to work on performance tuning.

Offline

#5456 2015-06-29 04:32:12

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@The_Moves :

I am certain that putting my VMs files on this new SSD (850 EVO with XFS on top of it) didn't improve performance over my old 5 7200rpm MD RAID 0 . If any , there is a little improvement on Windows boot times , but launching applications (ESPECIALLY FIREFOX !!!) remains without a noticable improvement .

Congrats on reaching the stable milestone !

Last edited by Denso (2015-06-29 04:46:29)

Offline

#5457 2015-06-29 10:44:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

That was actually something that I was curious about. I have 2 Samsung 840 Pros in a mdadm RAID 0, but I swear it hasn't really improved performance any, if anything I notice lag in games. Such as loading textures, or jumping around to view different players on a map, it takes a split second for things to load.

Ive had a stable setup for some time now, but need to work on performance tuning.

Damn! I notice that effects in most games, no matter if they're running from a raw image relying on a system(ext4) OCZ Agility 3 SSD, or a raw block device being the latest(FZEX) WD Black HDD.
ATTO disk benchmark shows no problems, 200MBps read-write on the SSD and specification-correct(180?.. i don't remember) speeds on the HDD, but the lag is still there.

I've tried every combination of virtio disk systems: virtio-blk-pci, virtio-scsi, native and write-through and whatever else cache modes, multiqueue... Nothing helped much. Maybe CPU pinning helps to address that issues?

The memory shouldn't be a problem - aw says it's pinned in place and isn't swappable or moved in any way while the devices are passed through.

Last edited by Duelist (2015-06-29 10:44:59)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5458 2015-06-29 13:03:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iommu=pt might help improve the latency of host disk and therefore the guest disk image stored on that disk.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5459 2015-06-29 15:16:39

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw, I read on your blog that you would need a wrapper script to do q35 on libvirt with windows. how does that work out?

Offline

#5460 2015-06-29 15:24:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@noctlos

Something like this:

#!/bin/sh exec qemu-kvm \ `echo "\$@" | \ sed 's|i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e|i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=pcie.1|g' | \ sed 's|bus=pci.2,addr=0x4|bus=pcie.1,addr=0x0.0|g'`

It could use some work, this is just something I crudely hacked together.  The idea is to key on the i82801b11-bridge that libvirt adds to also add a root port, then move the gpu, which happens to be pci.2/0x4 in my setup to pcie.1/0x0.0.  Improved libvirt support for q35 is coming, soon.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5461 2015-06-29 16:21:36

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Nothing helped much. Maybe CPU pinning helps to address that issues?

I have pinned CPUs now, two cores w/ two threads on my X5660 and still experience the issue. I was thinking of adding another core/thread, but from reading your response that may not help. Plus while watching NMON (with option L and c enables) it didn't appear that I was CPU bound anyways. I'm sure there are better tests with this though. Also was considering updating the Kernel as well, from 3.18.7 to 4. Still running Fedora 21

I will try AWs suggestion.

Last edited by The_Moves (2015-06-29 16:22:31)

Offline

#5462 2015-06-29 16:55:24

noctlos
Member
Registered: 2013-08-26
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I see. Since qemu-kvm is not part of my path, i would rectify that as `qemu-system-x86_64 -enable-kvm`, right? Also, how do I get libvirt to use that wrapper script?

Offline

#5463 2015-06-29 16:58:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noctlos wrote:

@aw
I see. Since qemu-kvm is not part of my path, i would rectify that as `qemu-system-x86_64 -enable-kvm`, right? Also, how do I get libvirt to use that wrapper script?

Change qemu-kvm to qemu-system-x86_64, but do not start adding random options.  You're modifying how libvirt is calling QEMU, not creating a full command line.  virsh edit the domain and update the <emulator> tag to point to the wrapper script.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5464 2015-06-29 17:10:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
Duelist wrote:

Nothing helped much. Maybe CPU pinning helps to address that issues?

I have pinned CPUs now, two cores w/ two threads on my X5660 and still experience the issue. I was thinking of adding another core/thread, but from reading your response that may not help. Plus while watching NMON (with option L and c enables) it didn't appear that I was CPU bound anyways. I'm sure there are better tests with this though. Also was considering updating the Kernel as well, from 3.18.7 to 4. Still running Fedora 21

I will try AWs suggestion.

Seems like you have enough cores to spare: try doing isolcpus to fully isolate a CPU from everything except the VM. Maybe THAT will help.

Because, @aw, i have iommu=pt and it doesn't help much.


Oh, and btw...

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 03:05.0 Multimedia audio controller: Ensoniq 5880B [AudioPCI] (rev 02) 04:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1) 04:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1) 05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09) 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

You know what this means;)
...

/sys/kernel/iommu_groups/8/devices: 0000:00:15.0 0000:00:15.2 0000:04:00.0 0000:05:00.0 0000:07:00.0 0000:00:15.1 0000:00:15.3 0000:04:00.1 0000:06:00.0 0000:07:00.1

D-OH! How could i forget...
Alright, port change, port change!

Oh no, we're going down! 04:00.0 is in that group too! Noooooo!

00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 2) 00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 3)
-[0000:00]-+-00.0 +-00.2 +-02.0-[01]--+-00.0 | \-00.1 +-04.0-[02]--+-00.0 | \-00.1 +-11.0 +-12.0 +-12.2 +-13.0 +-13.2 +-14.0 +-14.2 +-14.3 +-14.4-[03]----05.0 +-15.0-[04]--+-00.0 | \-00.1 +-15.1-[05]----00.0 +-15.2-[06]----00.0 +-15.3-[07]--+-00.0 | \-00.1 ...

Yeaaaaah, those are not processor root ports.
Seems like i must make the host headless(detach the GT610) and networkless(use a pci or usb network card) to be able to passthrough the full group...
I guess there's no real way of shuffling the iommu groups without changing the motherboard, right?

Alright, multiply the weirdness:

 <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </hostdev>

Note the driver name: sorry, aw, but i'll use pci-assign for select devices.

The VM booted, but i observe some weird things happening, the main card restarts the driver. Seems like they're doing their handshake mating dance...

After a dozen of screen blinks, they've settled down.
http://i.imgur.com/Mhr3izX.png
Sorry for the russian locale, but it's a habit..
(I don't know wtf is that mirage driver)
The offline device is QXL.
Well, now the fun part: GPU-Z, some benchmarks and maybe a crossfire.

http://i.imgur.com/i4JfRdt.png
CCC is overwhelmed(it was with only two cards) by awesomeness, and refuses to launch:D

CrossFireX is online. It is enabled and shown by GPU-Z 0.8.2 that it has 2 GPUs. Out of three.
It turned on much faster than usual though...

The thing is - i don't do drbridges. My cards are the first with GCN and XDMA crossfire, that's why it works with two cards in the first place.
I remember that AMD Support told me that i will need an even number of cards.
But there's a tri-way crossfire on some 7XXX cards, and there's configs even without the bridges. Something is amiss.
And i can't really determine which two cards are being used.

After i turned off crossfire, GPU-z crashed the driver, but it was reset by the system, so i was able to see three GPUs with some missing info. After a second launch of GPU-z, the driver hung and the system hung too. After a force power off, the VM no longer boots, eating 100% CPU. Time to reboot the host.

After a host reboot... The second card went offline, the third card works. Crossfire doesn't work, but i'm able to use it's video outputs. So it works as a separate device, but crossfire is unavailable... Hmm, what could be a reason.. Maybe the bus width?

PCI-e 2.0 x1 = 5Gbps = 500 MBps. A crossfire bridge is capable of 900 MBps.
So that's why XDMA crossfire was introduced - pci express bandwidth was big enough to pump all the crossfire stuff through itself, and everything AMD needed was enabling some way of DMA communication between devices.
If i'd have pci-e 3.0, even on 1x i'd have ~984 MBps as stated here.
Oh well. Is there a way of simulating higher bus throughput? I honestly don't believe that it really fills all that 900 MBps and it may be a simple software block.

So i guess the final result is semi-win: i can run three VMs simultaneously, if i have three spare screens. Or run a double-crossfire and single-card VMs, needing only two screens to spare.
Crossfire doesn't work with the third card if the second is offline. I wonder what will happen if i will remove the second card from the VM and try doing crossfire, but something hints me that fail will be the result.
Surprisingly though, that alone third card is self-contained - i can run a VM only from that card, BUT with some buts:
1. one VM launch per host boot. Legacy pci-assign doesn't do any resets right.
2. No VGA at all! Since i have it broken - i use OVMF, so it works.
3. It doesn't fit in the case.
All those problems may be fixed by pci-e 3.0 with more processor root ports and just more slots - i have no pci-e slots left free on my F2A55. Also, pci-e 3.0 alone would make crossfire through 1x possible, i think.

I'll photograph that madness tomorrow, for lulz, That was one hell of a day.



-- mod edit: read the Forum Etiquette and only post thumbnails http://wiki.archlinux.org/index.php/For … s_and_Code [jwr] --

Last edited by Duelist (2015-06-29 21:02:53)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5465 2015-06-29 21:05:21

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

Debian user here, but I decided to drop by and tell about my experience with Qemu + KVM and passthrough since all the relevant information is here. Thanks to everyone involved in creating the tools and documentation for doing this, it has been immensely useful.

I've set up my system with Debian as the host for work and general desktop usage and a Windows 7 guest as my toy OS for games. The system:

- Custom kernel 4.0.5 with all the necessary KVM and VFIO features enabled
- Qemu 2.1.2 (I later compiled a build of 2.3.0)

- Intel DP67BG. Needs the ACS patch and allow_unsafe_interrupts.
- i7 2600

I actually got those back in 2011 with the intention of using Xen but didn't, partly because of laziness and partly because I had Nvidia cards which meant trouble without Quadro hacks. Better late than never. smile

- Titan X for Windows, a GT430 for Linux

I initially passed through the GT430 to make sure everything works. It did, though the VM would only start once per boot. Suspending and resuming between VM starts sufficed. The Titan X, however, does not have that problem. I can stop and start it as many times as I want without fail!

- Delock KVM switch for USB devices. It functions as a USB hub as well so I need just one connection for each OS. There are three separate USB controllers on the DP67BG so I pass through one of them to Windows.
- Audioengine D1, a USB DAC for audio. Emulated sound through Qemu was completely distorted and stuttery no matter what settings I tried so I needed to pass through audio. I connected the D1 to the KVM switch as well. That did NOT work well. In addition to short dropouts the audio would occasionally completely cut off for about two seconds before catching up again. Very annoying. I connected it directly to a passed through USB port and the total cutoffs have not occurred since (for the past day, knock on wood). I suppose putting a time-sensitive USB device behind a hub on a VM is asking for trouble. I'll have to figure out a hassle-free way to play audio from either OS to one set of headphones. Manually plugging/unplugging cables is the last resort.
- A raw format disk image on an ext4 formatted SSD partition used as a VirtIO drive. As I debugged my skipping audio, I noticed disk IO causes rather high amounts of DPC latency in the guest. Games produce noticeable hiccups whenever they are loading stuff. I'm planning on passing through the second SATA controller (Marvell) on the motherboard so I can forget about optimizing IO. Its only connector is an eSata in the back panel so I'll have to route the cable back in the case. smile

Speaking of DPC latency in the guest, I'm getting oddly inconsistent readings. Sometimes the baseline is ~1000 µs, sometimes ~2000 µs. I can't figure out why. Setting the cpu governor to performance on the host halves this to either 500 or 1000 µs. Any ideas to what's causing this or if the values are higher than they should be?

Another question regarding vCpu = pCpu pinning which I don't quite understand: I have so far run into CPU bottlenecks in some games that didn't exist on a bare metal OS. Let's say I want to provide all the CPU performance to the guest should it need it. The i7 2600 is a hyper-threaded quad-core so I use -smp 8,sockets=1,cores=4,threads=2. Since I won't use both systems at the very same time, I don't care if either one of can hog all the resources. In fact it would be desirable. Do I benefit anything from, say, using taskset 0xFFFFFFFF which according to the man page means "all processors"? I have noticed that the CPU usage seems to lack affinity on the guest, with all 8 vCPUs often showing some usage. On a bare metal Windows, the CPU usage was much more concentrated, with one core or two cores typically maxed out and others partially loaded. I do not know if this is a symptom of anything or related to performance issues at all. I'd appreciate any insight on this.

Offline

#5466 2015-06-29 22:31:54

biocyberman
Member
Registered: 2015-06-29
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello geeky KVM users!
I am stuck while trying to set up passthrough for my compute GPU NVIDIA cards (Tesla k20c). The cards do not have ports to plug monitor cables on. They are only meant for computation. My host is "uname -a: Linux server 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux" (I know Ubuntu is not Archlinux, but it seems if you know Archlinux you know everthing big_smile). My guest is Windows 7, 64bit. Below is my startup script. Withouth the line that enables GPU passthrough (containing vfio-pci), I can boot and run the Windows guest successfully.  But when I enable the passthrough, qemu crashes with error shown at the end of this post. Any help is much appreciated.

#!/bin/bash configfile=/etc/vfio-pci1.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo "Binding vendor: $vendor and device: $device" echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } echo Loading vfio... sudo modprobe vfio-pci echo Start binding devices... cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done echo Done binding devices echo Starting virtual machine ... sudo qemu-system-x86_64 -enable-kvm -M q35 -m $((1024*32)) -cpu host \ -smp cpus=2,sockets=2,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga std \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,multifunction=off,x-vga=off,host=02:00.0,bus=root.1,addr=00.0 \ -drive file=/data/win7kvm.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/data/Windows7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on \ -runas kvmuser echo Done! exit 0

qmeu crash message when enable vfio-pci passthough for GPU:

qemu-system-x86_64: vfio_dma_map(0x7ff529f9ecb0, 0xc0000, 0x8000, 0x7fed080c0000) = -12 (Cannot allocate memory) qemu: hardware error: vfio: DMA mapping failed, unable to continue CPU #0: EAX=80000033 EBX=80000090 ECX=00000033 EDX=00000cfd ESI=00000001 EDI=00000090 EBP=00000097 ESP=00006f94 EIP=ffff1e1c EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA] SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] FS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] GS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA] LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy GDT= 000f6f98 00000037 IDT= 000f6fd6 00000000 CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80 FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000 XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000 CPU #1: EAX=00000000 EBX=00000000 ECX=00000000 EDX=000206d7 ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000 0000ffff 00008b00 GDT= 00000000 0000ffff IDT= 00000000 0000ffff CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000000 FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80 FPR0=0000000000000000 0000 FPR1=0000000000000000 0000 FPR2=0000000000000000 0000 FPR3=0000000000000000 0000 FPR4=0000000000000000 0000 FPR5=0000000000000000 0000 FPR6=0000000000000000 0000 FPR7=0000000000000000 0000 XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000 XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000 XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000 XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

Last edited by biocyberman (2015-06-29 22:33:55)

Offline

#5467 2015-06-30 06:47:27

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I have a question to everybody in the house :

Using rsync to backup the VMs' hdd images while the VMs are running causes the host to crash hard .

Any way to backup while the VMs are running ? It has to be from within the host not the guest + it has to be rsync for its inplace feature .

Offline

#5468 2015-06-30 07:45:34

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

So I have a question to everybody in the house :

Using rsync to backup the VMs' hdd images while the VMs are running causes the host to crash hard .

Any way to backup while the VMs are running ? It has to be from within the host not the guest + it has to be rsync for its inplace feature .

Surely there are many ways of doing this, but I wonder how could you possibly be using rsync for this. Does hits mean that you have mounted single block device both in guest and in host? If so, then no surprise that one of them is crashing. FWIW, I am using snapshot feature from ZFS and use zvols as block devices attached to virtual machines. Also I have installed inside guest qemu-ga from stable virtio-win and use it to freeze guest filesystem right before taking a snapshot.

Last edited by Bronek (2015-06-30 09:44:05)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5469 2015-06-30 08:01:44

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso, please don't hijack threads: https://wiki.archlinux.org/index.php/Fo … _hijacking


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5470 2015-06-30 10:44:49

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Im using OVMF and passing through onboard P9X79 PRO "ASMedia USB3.0 controller":

When keyboard is attached to the controller, the VM gets a hang for 2-3 minutes before booting Windows 8.1 (on POST) and the USB controller no longer works in the VM. When I physically disconnect the keyboard BEFORE starting the VM it works normally, I can attach it later when OS boots (but need to disconnect it again if I reboot OS).

In other words, it seems like OVMF doesnt like the controller+keyboard attached to it while booting?

Can I somehow disable USB or keyboard support before booting?

What could be the problem? Is there a known fix I cant seem to find out? Tried alot of combinations already... Please help!

EDIT:
Using UEFI minimal Fedora 22 with everything latest and greatest (virt-preview)

Last edited by devianceluka (2015-06-30 10:46:14)

Offline

#5471 2015-06-30 15:50:46

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Im using OVMF and passing through onboard P9X79 PRO "ASMedia USB3.0 controller":

When keyboard is attached to the controller, the VM gets a hang for 2-3 minutes before booting Windows 8.1 (on POST) and the USB controller no longer works in the VM. When I physically disconnect the keyboard BEFORE starting the VM it works normally, I can attach it later when OS boots (but need to disconnect it again if I reboot OS).

I had the same issue with my ASMedia USB3 Controllers. I would be able to boot the VM with Keyboard and Mouse connected, but they would randomly just stop working. Only a reboot of the host would fix them. Instead, I have started passing through my onboard USB 2.0 Controllers and had no such stability issues. I would stay away from the ASMEDIA USB3 Controllers

Last edited by The_Moves (2015-06-30 19:53:57)

Offline

#5472 2015-06-30 19:11:00

sunmast
Member
Registered: 2014-12-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
nbhs wrote:

Yes deveject is used to automate the ejecting, but you can eject the card from windows just like you would eject a pendrive.

EDIT: make sure ejecting is done on the vm first run.

deveject works perfect, the vm switches from the cirrus card to the radeon on bootup and back und shutdown big_smile
I'm still a bit unclear on the virtio stuff though, I downloaded the drivers, but how do I install them (srsly, windows is confusing…)?

I'm using OVMF to pass through a EVGA GTX 960 card, and experiencing the reset issue too (VM cannot be restarted without host restart).
I searched deveject on Google, but there are multiple projects called deveject, and it seems they are all for USB devices.

Could you give the link to deveject you used?

Also is there any way to reset the card from host side? E.g. power off/on the card from some Linux command?

Thanks!

Offline

#5473 2015-06-30 19:52:10

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
The_Moves wrote:
Duelist wrote:

Nothing helped much. Maybe CPU pinning helps to address that issues?

I have pinned CPUs now, two cores w/ two threads on my X5660 and still experience the issue. I was thinking of adding another core/thread, but from reading your response that may not help. Plus while watching NMON (with option L and c enables) it didn't appear that I was CPU bound anyways. I'm sure there are better tests with this though. Also was considering updating the Kernel as well, from 3.18.7 to 4. Still running Fedora 21

I will try AWs suggestion.

Seems like you have enough cores to spare: try doing isolcpus to fully isolate a CPU from everything except the VM. Maybe THAT will help.

Because, @aw, i have iommu=pt and it doesn't help much.

I tried the CPU Pinning, it may have helped a little, however I do not think I am fully setup correcty. Here is a snipit from my VM Config as well as my /etc/default/grub:

# GamingMachine.xml: <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='8'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='9'/> </cputune> # Using the above, I setup the below CPUs for isolation # Snipit from GRUB_CMDLINE line in /etc/default/grub: isolcpus=2,3,8,9

I re-read the OP and it brought me here: http://www.linux-kvm.com/content/tip-ru … cific-cpus
Using the commands there, I noticed that my VM was not in fact running on the cores I specified:

# Original Setup [root@kvmhost1 ~]# taskset -p 1214 pid 1214's current affinity mask: fff [root@kvmhost1 ~]# taskset -c -p 1214 pid 1214's current affinity list: 0-11 # Correcting the afinity [root@kvmhost1 ~]# taskset -p -c 2,3,8,9 1214 pid 1214's current affinity list: 0-11 pid 1214's new affinity list: 2,3,8,9 # Final [root@kvmhost1 ~]# taskset -c -p 1214 pid 1214's current affinity list: 2,3,8,9 [root@kvmhost1 ~]# taskset -p 1214 pid 1214's current affinity mask: 30c

I have yet to test to above tweak as I found this while on lunch break. Why isn't the <cputune> section in my VM config not setting the affinity for the PID which represents my VM?

As for iommu=pt, I started receiving strange jerkiness in games, so I have removed that flag. Maybe some other kernel flags are conflicting with it, or maybe i'm missing something. It makes sense that iommu=pt should help performance as it eliminates overhead - which i read from here: https://bugzilla.redhat.com/show_bug.cgi?id=1201503

Here is my full GRUB_CMDLINE line from /etc/default/grub:

GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora-server/root rd.lvm.lv=fedora-server/swap rd.driver.blacklist=nouveau kvm-intel.nested=1 intel_iommu=on isolcpus=2,3,8,9 pci-stub.ids=8086:3a34,8086:3a35,8086:3a36,8086:3a3a,1102:0012,1103:0611,1b4b:91a4,1b21:1042,1000:0072,10de:1187,10de:0e0a,8086:10d3,8086105e"

Offline

#5474 2015-06-30 19:58:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

I tried the CPU Pinning, it may have helped a little, however I do not think I am fully setup correcty. Here is a snipit from my VM Config as well as my /etc/default/grub:

# GamingMachine.xml: <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='8'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='9'/> </cputune> # Using the above, I setup the below CPUs for isolation # Snipit from GRUB_CMDLINE line in /etc/default/grub: isolcpus=2,3,8,9

Without knowing the host cpu or guest cpu topology, this is pretty meaningless.

I re-read the OP and it brought me here: http://www.linux-kvm.com/content/tip-ru … cific-cpus
Using the commands there, I noticed that my VM was not in fact running on the cores I specified:

# Original Setup [root@kvmhost1 ~]# taskset -p 1214 pid 1214's current affinity mask: fff [root@kvmhost1 ~]# taskset -c -p 1214 pid 1214's current affinity list: 0-11 # Correcting the afinity [root@kvmhost1 ~]# taskset -p -c 2,3,8,9 1214 pid 1214's current affinity list: 0-11 pid 1214's new affinity list: 2,3,8,9 # Final [root@kvmhost1 ~]# taskset -c -p 1214 pid 1214's current affinity list: 2,3,8,9 [root@kvmhost1 ~]# taskset -p 1214 pid 1214's current affinity mask: 30c

I have yet to test to above tweak as I found this while on lunch break. Why isn't the <cputune> section in my VM config not setting the affinity for the PID which represents my VM?

You're messing with the cpu affinity of the qemu process, not the vCPUs.  There's an <emulatorpin> control for this https://libvirt.org/formatdomain.html#elementsCPUTuning


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5475 2015-06-30 20:08:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think isolcpus should really help in a way of isolating host CPUs fully and dedicating them to the guest, so no IRQs are able to steal the guest's CPU time.
Maybe i misunderstand it.

Last edited by Duelist (2015-06-30 20:09:06)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5476 2015-06-30 20:20:03

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Without knowing the host cpu or guest cpu topology, this is pretty meaningless

My processor is an X5660, and I believe I have it setup correct with 2 Cores and 2 Thread. My question is, are the core id the same between 'isolcpus' and my 'vm config'? Is 2 in one the same 2 in the other?

Here is the core thread layout for my Xeon X5660:

[root@kvmhost1 ~]# egrep "^processor|^core id" /proc/cpuinfo processor : 0 core id : 0 processor : 1 core id : 1 processor : 2 core id : 2 processor : 3 core id : 8 processor : 4 core id : 9 processor : 5 core id : 10 processor : 6 core id : 0 processor : 7 core id : 1 processor : 8 core id : 2 processor : 9 core id : 8 processor : 10 core id : 9 processor : 11 core id : 10 [root@kvmhost1 ~]#
aw wrote:

You're messing with the cpu affinity of the qemu process, not the vCPUs.  There's an <emulatorpin> control for this https://libvirt.org/formatdomain.html#elementsCPUTuning

I will look into this.

Offline

#5477 2015-06-30 21:06:41

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek :

I'm using raw disk images (.img) for my VMs . No block devices or zvols . I'm just looking for a way to copy those images to a backup location but qemu always crashes along with the host .

@jasonwryan :

How is my question about qemu VMs disk images unrelated to this qemu thread ?

Offline

#5478 2015-06-30 21:42:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

How is my question about qemu VMs disk images unrelated to this qemu thread ?

How is a question about snapshotting a running VM disk image unrelated to graphics assignment?  Virtualization is a very broad topic, not every question about running or managing a QEMU VM is relevant to this particular thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5479 2015-06-30 22:28:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, there's positive news about my project and negative.
The positive news are:
Three GPUs actually WORK! Separately.
So i can mine my whatever-coins in a virtual machine. A strange thing to do, but combined with openstack that could be a base for some virtual private miner(VPS, but for mining) service.. ...althrough it'd be as pointless as /dev/null as a service.
But also i can crack some passwords! oclHashCat is able to utilize all three GPUs at their full power, so cracking NTLM hashes from filthy wind0wz is now 33% faster, heh.
I can attach a whooping number of screens to the VM: Nine. There was a version of HD7750 with 6 mini-dp ports, but it's cost was nowhere near my setup.
That 9 screens would work with AMD EyeFinity. I've actually tried doing three screens(one fullhd 24" and two square 1280x1024 17") on two GPUs, and i've ran out of video memory quickly, 2gb cards would be much nicer. AFAIK, the video memory does stack when using XDMA crossfire.
But i've distracted from the topic, back to qemu again.
Crossfire works with two cards only even if there's only one card running using vfio-pci and the others are one pci-assign from xen, which is considered legacy.
I've managed to find a way of determining which cards are crossfire-bound - the unbound card still has it's video outputs working, the paired one shares the display connectors with the master.
Also, i am really able to run two or three VMs in parallel, so it's three "seats" per one case. Well, almost, as the third card does not fit inside.
http://i.imgur.com/PN6XNaq.jpg
Again, sorry for the big image.
Negative news:
Crossfire with 3 cards totally doesn't work in my setup.
Maybe the culprit is pci-e bandwidth, as pci-e 2.0 x1 is not enough to overwhelm that physical crossfire bridge.
Or maybe the AMD support was right, and crossfire requires an even number of cards to work. I hope i'll find it out someday, but most likely not.

Now the third card goes to the sponsor(he paid for half of the card) and i continue to run crossfire with two cards, hoping to sell them one day and buy a GTX 750 KalmX. Or...
Hmm, what if i'll find a PLX-based switch which does ACS and stuff and presents three cards as if they were all connected to the proccesor's root port?..
Not only there would be a somewhat bandwidth increase(maybe), but it would enable me to use vfio-pci for all three cards eliminating the reset problems.
Sad thing that pci-e switches are rare an expensive, especially in my location. If i could only borrow it for some time...
The alternate way of proceeding on my way to tri-way crossfire in a VM would be using something on A85X/88X with three pci-e 16x slots(they are hopefully routed as processor-root-ports).
Or running windows bare-metal! That's a thing to try.



-- mod edit: read the Forum Etiquette and only post thumbnails http://wiki.archlinux.org/index.php/For … s_and_Code [jwr] --

Last edited by Duelist (2015-06-30 22:33:09)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5480 2015-07-01 00:06:44

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Dang , I forgot the thread's title completely  : (

@Duelist :

I guess you can run three AMD gpus on crossfire . I think they call it TriFire but don't quote me on that . I've never purchased one since the ATI days .
PCI-E expansion chasis are really expensive . Magma sells a 3 slot chassis for over $2000 . And they even have a 10 slots chassis for an unknown price , probably would be over $7000 . You're better off building more virtualization hosts with regular hardware .

Last edited by Denso (2015-07-01 00:10:36)

Offline

#5481 2015-07-01 11:45:56

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My ASMedia USB 3.0 is working flawlessly (from 1 page before) with OVMF-pure-efi, but not with OVMF-with-csm and QEMU. Can I do something about it anywhere in OVMF-with-csm (because the GPU used with ASMedia is not GOP capable) to load EFI for USB controllers?

EDIT:
On QEMU I get it to work almost flawlessly with Legacy USB 3.0 disabled in BIOS + pcie_aspm=off. After 10-30 reboots of the OS/VM I get DMAR error on host and only the reboot of the host solves it. It looks like the USB controller gets into a power saving mode or whatever and it doesnt unstick/reset anymore and only a host reboot solves it. Can it be fixed with any command/patch? To reset manually?

Last edited by devianceluka (2015-07-01 12:18:04)

Offline

#5482 2015-07-01 12:21:33

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Crossfire with 3 cards on my setup doesn't work even on bare metal.
But i've been able to record some benchmarks:
Unigine Heaven Benchmark 4.0: D3D11, quality medium, resolution 1920x1080 fullscreen(basic preset with full resolution).
Bare metal: 654 single card(100%), 1244 crossfire(190% performance with crossfire)
VM:630 single card(96%), 10131087 crossfire(160%173% crossfire from single VM card, 81%87% from bare metal performance).
I ran only one run, so i think some CPU activity on host did interfere with crossfire VM run.
There's also some stuttering occuring in the crossfire VM, i think about turning iommu=pt to iommu=on and maybe running PCAP or something to capture that and try to align it with host CPU and/or disk usage.
Sadly, toggling iommu=pt to iommu=on didn't help much. There's still stuttering present when unigine shows a new area or scene. Especially this is seen in the end of scene #19, where the camera pans to the city, and in the beginning of scene #22, where there's a fade in in the city.
If anyone is able to reproduce that lags, please tell.

Now i'll try to setup a second VM with the third card only.

Last edited by Duelist (2015-07-01 12:57:53)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5483 2015-07-01 14:55:31

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Sadly, toggling iommu=pt to iommu=on didn't help much. There's still stuttering present when unigine shows a new area or scene. Especially this is seen in the end of scene #19, where the camera pans to the city, and in the beginning of scene #22, where there's a fade in in the city.
If anyone is able to reproduce that lags, please tell.

Now i'll try to setup a second VM with the third card only.

This same issue happens to me. For instance, in World of Tanks, gliding over different tanks with the cursor causes the screen to freeze for a split second then jump ahead. In GTA5, some textures take a while to load when traveling at a high rate of speed, but this may be normal. I also noticed the issue on previous Windows 10 builds when swapping between the ICONS. As I play different games, i might be able to reproduce the issue.

Another issue I noticed was with GTA4, when I would load it, it would just crash the host to crash.

I think it might be good to look at different performance recording statistics from the host end, like iostat and vmstat.

With Windows 10, when opening up Task Manager, I also noticed that Windows knows it is a VM. I thought 'kvm=off' was supposed to fix that, but maybe it is just with NVIDIA Drivers?

Offline

#5484 2015-07-01 16:06:09

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Why nobody wants to help?

Offline

#5485 2015-07-01 16:38:09

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Why nobody wants to help?

Why are you using 'OVMF-with-csm'?

Offline

#5486 2015-07-01 17:09:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Why nobody wants to help?

If you insist:
reset issues are tough. And they require quirks for particular devices.
BUT. Why do you p-t a USB controller, if there's a device passthrough, which gives almost 100% performance and is trouble-free?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5487 2015-07-01 17:47:42

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry for taking so long, took a nap after nerd rage... I'm already trying for 4 days straight, 12h minimum each day, trying ALOT of combinations of BIOS and linux stuff and rebooting.

The_Moves wrote:
devianceluka wrote:

Why nobody wants to help?

Why are you using 'OVMF-with-csm'?

Because the GPU used with ASMedia does not support GOP (R7 240).

OVMF-pure-efi - works flawlessly (okay in 100 reboots, it didnt recognize it 1 time - its no problem)
SeaBIOS - works half the time
OVMF-with-csm - doesnt work, before booting Windows logo and after OVMF logo there's " _ " for 1-2 minutes and then it boots and doesnt recognize it in windows. If I disconnect the keyboard before starting VM and connecting back when windows boots, it works "flawlessly"

The real question is why wouldnt it boot with csm and only with pure-efi? Tried to remove all uhci/xhci/usb devices in XML, thought it might be in a dilemma which to choose and it doesnt fix it.

The only explanation from me would be that because SeaBIOS sometimes works sometimes doesnt (legacy ROM), OVMF-with-csm is using legacy ROM from USB controller also then... Where and how to change it to use EFI ROM for everything except GPU?

Duelist wrote:
devianceluka wrote:

Why nobody wants to help?

If you insist:
reset issues are tough. And they require quirks for particular devices.
BUT. Why do you p-t a USB controller, if there's a device passthrough, which gives almost 100% performance and is trouble-free?

It is not 100% identical. It is bad for gaming. It is software and CPU accelerated. Would you flash your Android with an emulated USB controller? The question is only time when a trouble will arise and something will not be compatible... I have 4 GPU and 4 USB, so...

----

Also one observation made by me... Sometimes SeaBIOS when it doesnt work, it in my opinion"locks" the controller and even after host reboot it doesnt work. Only solution is pcie_aspm=off to pcie_aspm=force and it works after next host reboot (policy powersave ; performance & default do not work ever).

----

Before someone says it is a BIOS/firmware bug... It MAY be, but it is FIXABLE (by me), but needs a human hand. There 100% must be a software solution/fix. As I'm saying I get like 3-4 lines on host: "dmar dma write request device fault addr". Also if EFI is working, there should atleast be a setting where one can force (like in real MB UEFI BIOS) to force EFI on devices (CSM) if the devices have working EFI ROMs atleast.

----

Host:
ASUS P9X79 PRO
i7-3820
16GB DDR3
256GB 850 PRO + 128GB 850 EVO
2x 2TB WD Black
2x R9 290
2x R7 240
4.0.6-300.fc22.x86_64 (virt-preview - everything latest!)
GRUB: intel_iommu=on pci-stub.ids=1b4b:9130,1b21:0612,1002:6613,1002:aab0,1002:6798,1002:aaa0,8086:1d26,8086:1d2d kvm-intel.nested=1 iommu=pt pcie_aspm=off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1

----

Almost forgot. If I remove ASMedia USB 3.0 controller from pci-stub, it increases the chances of working in SeaBIOS and surviving reboots by ALOT....

Last edited by devianceluka (2015-07-01 18:22:41)

Offline

#5488 2015-07-01 18:48:02

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ aw

Did you found any reelevant info for the VT-d support in Chipset discussion? I was waiting for it. With the huge amount of contradicting info going around from both Intel and manufacturers, I still want a final word on that, and you are close enough to the Intel guys to get it (And by final word, I mean someone that can also explain why the documentation incosistency).
https://bbs.archlinux.org/viewtopic.php … 9#p1523059

I'm still pondering how reelevant it is to get a Chipset that Intel claims that has VT-d support, in case I have to make or recommend a new build for this usage. Yes, I know that you can get this working on Desktop, yet I want to know if Server parts have a definite advantage which is not Firmware related.


Also, I'm aware that LGA 1155/1150 parts does not support ACS for the PCIe Controller of the Processor, so all PCIe slots that are connected to it should fall into the same IOMMU Group. However, assuming there is a Motherboard that adds a PLX chip (Typically you see it in some high end Motherboards that claims SLI/Crossfire in 16x/16x mode instead of 8x/8x), can the PLX chip itself isolate the PCIe Cards below it even if there is no isolation between Processor PCIe Controller and the PLX chip?
What about the new trend of PCIe SSDs supporting NVMe? Aren't they PCI devices? Could I do something like, do PCI Passthrough of a PCIe SSD to a VM, then using the OVMF Firmware and QEMU NVMe support to boot from it? I'm also curious on how SATA Express and M.2 devices looks like from this point of view (Do they appear in lspci?), since they use PCIe Lanes and also SATA, which could come from two different places (SATA from Chipset, but PCIe Lanes can come from either the same Chipset or from the Processor itself). I expect these two to be rather hard to work with.


Duelist wrote:

I've heard some news about XenGT(or KVMGT or whatever) in linux 4.1, but lkml doesn't hint me on details.
Can anyone give me a brief info about it's state? Does anybody knows about it?

According to Phoronix there was something related to XenGT that was included in 4.1:
http://www.phoronix.com/scan.php?page=n … -DRM-First

It talks about "client side" and "guest". I believe than it means that the Linux 4.1 Intel i915 Driver is XenGT/KVMGT ready from inside a VM, but it still doesn't mention anything about host side (Since it needs support from both the Linux kernel, QEMU, SeaBIOS, and for Xen, Xen itself. Not sure if the KVM component also needs a hand or is included in the Linux kernel part). So basically, I take that you don't need a special build of the Drivers to use the virtualized GPU from inside a VM, but you would still need to set up the host.

Offline

#5489 2015-07-01 21:37:39

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@devianceluka:
Things to try also:
Flashing the motherboard's firmware. Maybe there are updates of that asmedia(i suppose it's soldered to the board just like mine) controller's firmware too.
Digging into the GPU in question - I have R7 250E and i've got my UEFI support working by doing weird and strange things which i completely understand now. Maybe you'll be able to pull that trick too?
Modifying the firmware itself would be almost impossible, sadly. But, i think, you can feed the firmware with any EFI driver for a device just by putting them on a right filesystem(fat32 of some sort). And it will load. Sadly, USB controllers have nothing to do with pci option roms.

P.S.

Would you flash your Android

I'd flash it with a flashbang grenade. Considering all this mess we have on x86 platforms, i don't even want to know anything about android and vendors and hardware and driver blobs and stuff. I don't want to loose my last remains of sanity. So i'm stuck with a buttoney half chinese flip-phone and a dead sony err.. with some soldering iron modifications. I don't have much social life to rely on all those funky whatsapp-viber-facebook-twitter stuff, i don't even get enough e-mails.
If you need 100% insurance that everything will work - boot the windows baremetal if your operation is not available on linux host. If you have security concerns about that, unplug everything unneeded from the machine beforehand. If you set it up right, that is an easy task to do.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5490 2015-07-01 21:42:31

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What are you saying about UEFI GPU trickery? Dont understand. You hacked it or what? Got a link?
Where to search for asmedia fw update? The MB BIOS is latest...

Offline

#5491 2015-07-02 14:47:35

iH8c0ff33
Member
Registered: 2015-07-02
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,
I'm trying to passtrough an Nvidia GTX 770.
I used the AUR package linux-vfio and booting with "i915.enable_hd_vgaarb=1 intel_iommu=on" but i keep getting this error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 1                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
so i searched google for this error and found another boot parameter for ACS override but the error hasn't changed so much
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/13: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
I don't know what to do please help me
Thanks

Offline

#5492 2015-07-02 14:57:55

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

@devianceluka:
Things to try also:
Flashing the motherboard's firmware. Maybe there are updates of that asmedia(i suppose it's soldered to the board just like mine) controller's firmware too.
Digging into the GPU in question - I have R7 250E and i've got my UEFI support working by doing weird and strange things which i completely understand now. Maybe you'll be able to pull that trick too?
Modifying the firmware itself would be almost impossible, sadly. But, i think, you can feed the firmware with any EFI driver for a device just by putting them on a right filesystem(fat32 of some sort). And it will load. Sadly, USB controllers have nothing to do with pci option roms.

P.S.

Would you flash your Android

I'd flash it with a flashbang grenade. Considering all this mess we have on x86 platforms, i don't even want to know anything about android and vendors and hardware and driver blobs and stuff. I don't want to loose my last remains of sanity. So i'm stuck with a buttoney half chinese flip-phone and a dead sony err.. with some soldering iron modifications. I don't have much social life to rely on all those funky whatsapp-viber-facebook-twitter stuff, i don't even get enough e-mails.
If you need 100% insurance that everything will work - boot the windows baremetal if your operation is not available on linux host. If you have security concerns about that, unplug everything unneeded from the machine beforehand. If you set it up right, that is an easy task to do.

Are you going to let me in the dust waiting forever with your half completed answer? Can you elaborate more on GPU GOP trickery?

Offline

#5493 2015-07-02 15:59:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

devianceluka wrote:

Are you going to let me in the dust waiting forever with your half completed answer? Can you elaborate more on GPU GOP trickery?

Patience, my friend. I understand your passion, as i have adopted the same feeling some time ago. But sometimes i have to allocate some time on my RL problems. You'd be pretty shocked to know the details, so i won't share them.

I don't know anything about asmedia, but i suggest you digging in the other direction.
(I'd give you a link to my earlier messages about that, but i don't remember when i wrote them so..)
So. The idea is simple. My vendor(ASUS) supplied an update of GPU FW which enabled GOP and that image contained an .efi device driver.
BUT IT WAS BROKEN! So i went and googled for a supplement.
And i've found two things:
1. In the early days of UEFI and GOP(2008-2010, i think) AMD gave a presentation about GOP support being universal for all their 6 and 7 series GPUs. 8 and R5-7-9 stuff didn't exist that day.
2. Some mac fanatics compiled a ROM file with a efi driver from some 6XXX card. So that hinted me that i can take my broken firmware and replace the GOP part with the working one, taken from some 6-series(i think it was called barton or barts..) card.

That i did. I don't remember how to do that properly, so i'm not able to give you step-by-step instructions.
What you need to do: take my UEFI-capable ROM which i made for myself, use EDK2 tool called... called...
damn i forgot. Anyway, try using that to extract the .efi that really works and then you need to compress it back into your own "legacy-only" ROM made for your card exclusively. Append that romfile to your device in a VM(so no need to flash anything!) and hope it will work.

There are tools in EDK2 for that tasks, i just forgot how they were named and where are they.

I'll try lurking through my old messages to find the details, but considering your inpatience..

https://bbs.archlinux.org/viewtopic.php … 1#p1513281
Here. The message with the details and links and stuff. Modifying GPU ROM is easier that USB controllers'.
...just found out that that message contains a link to my previous message about that. LOL.

Last edited by Duelist (2015-07-02 18:41:58)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5494 2015-07-02 16:34:36

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
devianceluka wrote:

Are you going to let me in the dust waiting forever with your half completed answer? Can you elaborate more on GPU GOP trickery?

Patience, my friend. I understand your passion, as i have adopted the same feeling some time ago. But sometimes i have to allocate some time on my RL problems. You'd be pretty shocked to know the details, so i won't share them.

I don't know anything about asmedia, but i suggest you digging in the other direction.
(I'd give you a link to my earlier messages about that, but i don't remember when i wrote them so..)
So. The idea is simple. My vendor(ASUS) supplied an update of GPU FW which enabled GOP and that image contained an .efi device driver.
BUT IT WAS BROKEN! So i went and googled for a supplement.
And i've found two things:
1. In the early days of UEFI and GOP(2008-2010, i think) AMD gave a presentation about GOP support being universal for all their 6 and 7 series GPUs. 8 and R5-7-9 stuff didn't exist that day.
2. Some mac fanatics compiled a ROM file with a efi driver from some 6XXX card. So that hinted me that i can take my broken firmware and replace the GOP part with the working one, taken from some 6-series(i think it was called barton or barts..) card.

That i did. I don't remember how to do that properly, so i'm not able to give you step-by-step instructions.
What you need to do: take my UEFI-capable ROM which i made for myself, use EDK2 tool called... called...
damn i forgot. Anyway, try using that to extract the .efi that really works and then you need to compress it back into your own "legacy-only" ROM made for your card exclusively. Append that romfile to your device in a VM(so no need to flash anything!) and hope it will work.

There are tools in EDK2 for that tasks, i just forgot how they were named and where are they.

I'll try lurking through my old messages to find the details, but considering your inpatience..

https://bbs.archlinux.org/viewtopic.php … 1#p1513281
Here. The message with the details and links and stuff. Modifying GPU ROM is easier that USB controllers'.

Thanks man! I will look into it as that's my only option...

Offline

#5495 2015-07-02 16:50:16

iH8c0ff33
Member
Registered: 2015-07-02
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iH8c0ff33 wrote:

Hi everyone,
I'm trying to passtrough an Nvidia GTX 770.
I used the AUR package linux-vfio and booting with "i915.enable_hd_vgaarb=1 intel_iommu=on" but i keep getting this error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 1                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
so i searched google for this error and found another boot parameter for ACS override but the error hasn't changed so much
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/13: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
I don't know what to do please help me
Thanks

Ok I think i solved the probled. I tried not to use the Systemd service and running vfio-bind many times then it worked O.o
Thanks anyway

Offline

#5496 2015-07-02 22:15:25

sunmast
Member
Registered: 2014-12-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I'm trying to pass through a EVGA GTX 960 card with OVMF method (because I need to use Intel IGP on the host), but I'm hitting the VM unable to reboot problem (unable to reboot VM unless reboot host first, otherwise host freezes). It seems the card is not reset correctly between reboots.

Searching the thread it seems one workaround is to "eject" the card before Windows shutdown, but does this work for OVMF method? In Windows I don't see the card is removable. I've tried deveject and devcon (from WDK), they both don't work. I've also tried to disable the card before shutdown, it doesn't solve the problem.

I'm wondering if there is any command from host side can help? Like reset the card each time when qemu exits?

Thanks in advance!

Offline

#5497 2015-07-03 12:30:46

maelask3
Member
Registered: 2015-07-03
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using Libvirt 1.2.17 and QEMU 2.3, Windows 8.1 setup BSODs (SYSTEM_SERVICE_EXCEPTION) no matter if the machine is i440fx or q35, or if the graphics card is passed through or not when it gets to Copying files.
Windows 10 setup doesn't even boot.
Edit : Worth mentioning : using OVMF with a Radeon R9 270X (Gigabyte OC)

Last edited by maelask3 (2015-07-03 12:31:09)

Offline

#5498 2015-07-03 15:52:52

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 AMD-Vi: mmio-addr: 00000000feb20000 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:00.2 AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 01:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 01:00.1 AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 AMD-Vi: DEV_SELECT devid: 02:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:12.2 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:13.2 AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 AMD-Vi: DEV_RANGE_END devid: 06:1f.7 AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 AMD-Vi: DEV_SELECT devid: 00:15.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 07:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 07:00.1 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:16.2 AMD-Vi: DEV_SPECIAL(IOAPIC[0]) devid: 00:14.0 AMD-Vi: DEV_SPECIAL(HPET[0]) devid: 00:14.0 AMD-Vi: DEV_SPECIAL(IOAPIC[255]) devid: 00:00.1 [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found AMD-Vi: Disabling interrupt remapping

I'm assuming my table is broken?

I'm not exactly sure how to interpert the iommu dump data.

EDIT:  May help

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) [1002:5a14] (rev 02) 00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) [1002:5a23] 00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) [1002:5a16] 00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) [1002:5a18] 00:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E) [1002:5a19] 00:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F) [1002:5a1a] 00:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port G) [1002:5a1b] 00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40) 00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397] 00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396] 00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397] 00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396] 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42) 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40) 00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40) 00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399] 00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0] 00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397] 00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396] 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600] 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601] 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602] 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603] 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604] 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605] 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] RV630 XT [Radeon HD 2600 XT] [1002:9588] 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV630 HDMI Audio [Radeon HD 2600 Series] [1002:aa08] 02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06) 03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042] 04:00.0 SATA controller [0106]: JMicron Technology Corp. JMB362 SATA Controller [197b:2362] (rev 10) 05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042] 06:06.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788] 07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] [1002:679a] 07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]

Last edited by Zeorymer (2015-07-03 15:58:11)

Offline

#5499 2015-07-03 17:03:43

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:

Mvd6M1B.png
I'm assuming my table is broken?

Try

ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1

mapping ioapic directly. Adjust accordingly, if i recall correctly - that should work for 990FX...

BTW, what versions of everything do you use? I observe fixes for this being added somewhere in 2012. AW may correct me.

Last edited by Duelist (2015-07-03 17:13:45)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5500 2015-07-03 17:35:22

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Zeorymer wrote:

http://i.imgur.com/Mvd6M1B.png
I'm assuming my table is broken?

Try

ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1

mapping ioapic directly. Adjust accordingly, if i recall correctly - that should work for 990FX...

BTW, what versions of everything do you use? I observe fixes for this being added somewhere in 2012. AW may correct me.

Even though it complains about ioapic[9] and ioapic[10]?  Then again, iommu is not my expertise.

Kernel is -ck with required options enabled for passthrough.

Everything else was in Arch repos, though I haven't really gotten that far thanks to Asus not knowing how to make a bios correctly.

Offline

#5501 2015-07-03 17:45:40

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:

Even though it complains about ioapic[9] and ioapic[10]?  Then again, iommu is not my expertise.

Kernel is -ck with required options enabled for passthrough.

Everything else was in Arch repos, though I haven't really gotten that far thanks to Asus not knowing how to make a bios correctly.

I said adjust accordingly.. I'd try mapping ioapic[10] to 00:14.0 and ioapic[9] to 00:01.0. Maybe vice-versa... I can't tell for sure.
Yeah, 00:14.0 to ioapic[9], 00:01.0 to ioapic[10]. That should do the thing.

Tell the real versions.

Last edited by Duelist (2015-07-03 17:48:18)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5502 2015-07-03 18:56:59

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Zeorymer wrote:

Even though it complains about ioapic[9] and ioapic[10]?  Then again, iommu is not my expertise.

Kernel is -ck with required options enabled for passthrough.

Everything else was in Arch repos, though I haven't really gotten that far thanks to Asus not knowing how to make a bios correctly.

I said adjust accordingly.. I'd try mapping ioapic[10] to 00:14.0 and ioapic[9] to 00:01.0. Maybe vice-versa... I can't tell for sure.
Yeah, 00:14.0 to ioapic[9], 00:01.0 to ioapic[10]. That should do the thing.

Tell the real versions.

Whell booting with that caused a panic...I'm going to assume first that I messed something up.

http://pastebin.com/3AgVU6KB

pastebin because of the size of the log.

It seems to be related to the 7950 I'm trying to pass.  I doubt the card itself, it was working yesterday in Windows (before I nuked it)

Kernel version is at the top of the log.
extra/qemu 2.3.0-3
community/libvirt 1.2.17-1
ovmf-fishman-git  <-SVN version was marked out of date, if this version will be a problem, I can change it.

Anything else?

Offline

#5503 2015-07-05 01:49:22

MonopolyMan
Member
Registered: 2015-07-05
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've had Windows 8.1 running for quite some time now in a VM and almost everything is working fine. However, I still feel like I'm missing something as far a CPU tweaking goes.

I have a Xeon with 8 threads (4 cores + HT). I launch with the qemu arg "cpu -host" and use libvirt to set the topology to 3 cores 6 thread. Then I use cpuset to create a cpuset and move all tasks (and kernel threads) to cpu 0-1. Then I have my vcpus pinned to CPUs 2-7. I've also enabled the Hyper-V enlightenments. However, I still feel like I'm getting questionable CPU performance. The most notable example is Borderlands: PS, which has really poor performance in the VM. I am passing through a 270X and play a lot of GPU intesive games just fine, but BL gives me issues like no other game.

Do anyone have any tips on how I could further optimize my CPU performance?

Offline

#5504 2015-07-05 04:02:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@MonopolyMan

What host kernel?  A while back we added lazy debug register handling, primarily due to BL2 use of debug registers.

EDIT: kernel 3.15 includes lazy debug register handling

Last edited by aw (2015-07-05 13:25:07)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5505 2015-07-05 08:17:20

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, my VDIs are working almost fine, they are stable and the performance is good. There are two small problems though

  • keyboard is not available during initial phase of Windows startup (selections in text mode, before Windows logo)

  • quite often during startup (animated Windows logo) the virtual machine will enter "paused" state and there is nothing to indicate why. The only way to recover is "virsh destroy", rollback its system volume (* because of above) and then start again

Here is current virsh definition of one of the affected VDIs.

<domain type='kvm' id='3'> <name>gdynia</name> <uuid>7a76b37e-3604-4ab1-84ef-69de09123ea3</uuid> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <hugepages> <page size='2048' unit='KiB'/> </hugepages> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> <vcpupin vcpu='4' cpuset='14'/> <vcpupin vcpu='5' cpuset='15'/> <vcpupin vcpu='6' cpuset='16'/> <vcpupin vcpu='7' cpuset='17'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type> <loader type='rom'>/usr/share/qemu/bios.bin</loader> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='4096'/> </hyperv> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='2'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64.xvga03.sh</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/zvol/zdata/vdis/gdynia'/> <backingStore/> <target dev='vda' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/zvol/zdata/protected'/> <backingStore/> <target dev='vdc' bus='virtio'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/disk/by-partuuid/c5a05f68-9f2c-4ad6-96c5-a5fc85d73d7c'/> <backingStore/> <target dev='vdd' bus='virtio'/> <alias name='virtio-disk3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/root/virtio-win.iso'/> <backingStore/> <target dev='hdd' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:01:34:56:e7'/> <source bridge='br0'/> <target dev='vnet1'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/gdynia.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x0853'/> <product id='0x0124'/> <address bus='2' device='3'/> </source> <alias name='hostdev0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x84' slot='0x00' function='0x0'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain>

and some information about the host

root@gdansk ~ # uname -a Linux gdansk 4.0.7-1-ARCH #1 SMP PREEMPT Sat Jul 4 15:22:43 BST 2015 x86_64 GNU/Linux root@gdansk ~ # qemu-system-x86_64 --version QEMU emulator version 2.2.1, Copyright (c) 2003-2008 Fabrice Bellard root@gdansk ~ # virsh --version 1.2.16 root@gdansk ~ # cat /proc/cmdline BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw spl.spl_hostid=0xa8c03302 console=ttyS0,115200N8R nomodeset udev.children-max=32 edac_core.edac_mc_panic_on_ue=1 nohz=off intel_iommu=on iommu=pt pci-stub.ids=8086:1d20,1002:6707,1002:aa80,1002:67b0,1002:aac8,104c:8241,1033:0194 isolcpus=2,3,4,5,6,7,8,9,10,11,14,15,16,17,18,19,20,21,22,23 initrd=../initramfs-linux.img root@gdansk ~ # zgrep VFIO /proc/config.gz CONFIG_VFIO_IOMMU_TYPE1=m CONFIG_VFIO=m CONFIG_VFIO_PCI=m CONFIG_VFIO_PCI_VGA=y CONFIG_VFIO_PCI_MMAP=y CONFIG_VFIO_PCI_INTX=y CONFIG_KVM_VFIO=y root@gdansk ~ # free -gwt :( total used free shared buffers cache available Mem: 125 51 58 0 6 9 65 Swap: 159 0 159 Total: 285 51 218 root@gdansk /etc/sysctl.d # cat 80-hugepages.conf # Reserve this many 2MB pages for virtual machine vm.nr_hugepages = 25000 root@gdansk ~ # grep "processor" /proc/cpuinfo | wc -l 24 root@gdansk ~ # cat /proc/cpuinfo | tail -28 processor : 23 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz stepping : 4 microcode : 0x427 cpu MHz : 1268.617 cache size : 15360 KB physical id : 1 siblings : 12 core id : 5 cpu cores : 6 apicid : 43 initial apicid : 43 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt bugs : bogomips : 5204.72 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: root@gdansk ~ # cat /usr/bin/qemu-system-x86_64.xvga03.sh #!/bin/sh exec nice --adjustment=-5 /usr/bin/qemu-system-x86_64 `echo "$@" | sed 's/vfio-pci,host=03:00.0/vfio-pci,host=03:00.0,x-vga=on/g'` root@gdansk ~ # lspci -vnn -s 0000:03:00 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman LE GL [FirePro V5900] [1002:6707] (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0b06] Flags: bus master, fast devsel, latency 0, IRQ 128 Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at dfe20000 (64-bit, non-prefetchable) [size=128K] I/O ports at 7000 [size=256] Expansion ROM at dfe00000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: radeon 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80] Flags: bus master, fast devsel, latency 0, IRQ 10 Memory at dfe40000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Kernel driver in use: pci-stub Kernel modules: snd_hda_intel

Last edited by Bronek (2015-07-05 08:22:29)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5506 2015-07-05 18:49:30

MonopolyMan
Member
Registered: 2015-07-05
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@MonopolyMan

What host kernel?  A while back we added lazy debug register handling, primarily due to BL2 use of debug registers.

EDIT: kernel 3.15 includes lazy debug register handling

I just installed 3.16 and that seemed to fix it! It made both Dirty Bomb and Borderlands go from completely unplayable to 60+ FPS.

Thank you very much!

Offline

#5507 2015-07-05 19:12:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:

Anything else?


Well... try swapping BDFs in the commandline, making ioapic[9] 00:00.1 and vice versa.
Also you can try swapping the host and the guest cards, placing your HD7950 into 01:00.0(first pci-e slot).
BTW, the card itself should be fine, because, afair, 270x is rebadged HD7950, and as you can see there and along the thread, people have no problems using it.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5508 2015-07-05 19:16:21

MonopolyMan
Member
Registered: 2015-07-05
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One other thing. For sound, I am using -soundhw hda. This works really well for output, but I have an odd issue with input. For some reason, my microphone is delayed by about 20-25 second on the guest. However, there is no issues on the host. I can't seem to find any reason why this might be happening. Does anyone have any susgestions?


Note: I'm using:   
<qemu:env name='QEMU_PA_SAMPLES' value='4096'/>
<qemu:env name='QEMU_AUDIO_DRV' value='pa'/>

Offline

#5509 2015-07-06 22:46:42

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Zeorymer wrote:

Anything else?

Well... try swapping BDFs in the commandline, making ioapic[9] 00:00.1 and vice versa.

Tried that, froze on "booting kernel"
I noticed a line appear while booting that doesn't seem to get logged:

/init/ line 1 = ivrs_ioapic[9]=00:14.0 : Command not found /init/ line 1 = ivrs_ioapic[10]=00:00.1 : Command not found

or something like that.

It seems to work anyway as the table error doesn't show

Also you can try swapping the host and the guest cards, placing your HD7950 into 01:00.0(first pci-e slot).

I attempted to do this as i was setting up the system, but it wants to diosplay everything out the first card in the system, even if the monitors are in the 2600XT in the lower slots.  Any way I can force it to?

BTW, the card itself should be fine, because, afair, 270x is rebadged HD7950, and as you can see there and along the thread, people have no problems using it.

The R9 270X = 7870
The R9 280 = 7950

However there are quite a few 7950s on the list being passed sucessfully.


Additionally, I built the linux-vfio package to see if it made any difference, system booted with the same timeout messages and locked up when I tried the vfio-bind command.

Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:02 JK-Desktop kernel: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=07:00.0 address=0x00000002451e49e0] Jul 06 18:30:02 JK-Desktop kernel: BUG: unable to handle kernel paging request at ffffffffffffc039 Jul 06 18:30:02 JK-Desktop kernel: IP: [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: PGD 180e067 PUD 1810067 PMD 0 Jul 06 18:30:02 JK-Desktop kernel: Oops: 0000 [#1] PREEMPT SMP Jul 06 18:30:02 JK-Desktop kernel: Modules linked in: nls_utf8 isofs vfio_pci vfio_iommu_type1 vfio it87 hwmon_vid eeepc_wmi asus_wmi sparse_keymap led_class rfkill video kvm_amd kvm fuse crct10dif_pclmu Jul 06 18:30:02 JK-Desktop kernel: hid_logitech ff_memless hid_generic hid_microsoft usbhid hid uas usb_storage atkbd libps2 ahci libahci libata ohci_pci xhci_pci ohci_hcd ehci_pci xhci_hcd ehci_hcd scs Jul 06 18:30:02 JK-Desktop kernel: CPU: 4 PID: 720 Comm: cinnamon Not tainted 4.0.7-2-vfio #1 Jul 06 18:30:02 JK-Desktop kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO, BIOS 1604 10/16/2012 Jul 06 18:30:02 JK-Desktop kernel: task: ffff88023db2b2a0 ti: ffff8800a4144000 task.ti: ffff8800a4144000 Jul 06 18:30:02 JK-Desktop kernel: RIP: 0010:[<ffffffff81576b77>] [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: RSP: 0018:ffff8800a4147a08 EFLAGS: 00010292 Jul 06 18:30:02 JK-Desktop kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00000000c0000100 Jul 06 18:30:02 JK-Desktop kernel: RDX: ffff8800a4147fd8 RSI: ffff88023db2b2a0 RDI: ffff88024ed13e40 Jul 06 18:30:02 JK-Desktop kernel: RBP: ffff8800a4147a18 R08: ffff8800a4144000 R09: ffff88024ed13e70 Jul 06 18:30:02 JK-Desktop kernel: R10: 0000000000000020 R11: 0000000000000004 R12: 000000000006224f Jul 06 18:30:02 JK-Desktop kernel: R13: ffff8800a4147b80 R14: ffff8800a4147bec R15: 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: FS: 00007f532c8c39c0(0000) GS:ffff88024ed00000(0000) knlGS:0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 06 18:30:02 JK-Desktop kernel: CR2: ffffffffffffc039 CR3: 00000000a4ca2000 CR4: 00000000000407e0 Jul 06 18:30:02 JK-Desktop kernel: Stack: Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147bec 0000000000000000 ffff8800a4147ab8 ffffffff815798fc Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147a28 0000000000000000 0000000000000000 000000a9a6b041b6 Jul 06 18:30:02 JK-Desktop kernel: 000000a9a6aa1f67 ffffffff810e0bd0 ffff88024ed0e408 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: Call Trace: Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff815798fc>] schedule_hrtimeout_range_clock.part.6+0xfc/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff810e0bd0>] ? __run_hrtimer+0x250/0x250 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579891>] ? schedule_hrtimeout_range_clock.part.6+0x91/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579939>] schedule_hrtimeout_range_clock+0x19/0x50 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579983>] schedule_hrtimeout_range+0x13/0x20 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ec8f4>] poll_schedule_timeout+0x44/0x70 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee111>] do_sys_poll+0x531/0x600 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee2e1>] SyS_poll+0x71/0x130 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff8157a889>] system_call_fastpath+0x12/0x17 Jul 06 18:30:02 JK-Desktop kernel: Code: b8 00 00 48 89 e5 53 48 83 ec 08 48 8b 10 48 85 d2 74 0a 48 83 b8 a8 07 00 00 00 74 27 65 48 8b 1c 25 60 b8 00 00 e8 99 f5 ff ff <48> 8b 83 38 c0 ff ff a8 08 75 f Jul 06 18:30:02 JK-Desktop kernel: RIP [<ffffffff81576b77>] schedule+0x37/0x90

Offline

#5510 2015-07-07 01:51:54

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:

I attempted to do this as i was setting up the system, but it wants to diosplay everything out the first card in the system, even if the monitors are in the 2600XT in the lower slots.  Any way I can force it to?

There's three ways of changing this:
1. Making the host use UEFI too and ignore VGA. That way you'll see UEFI setup menu on all video outputs, but the GPU will be free of any extra burden like VGA.
2. On some Gigabyte motherboards it's possible to select the primary GPU device, to which all VGA stuff will be directed.
3. Live with it, but the VM should use OVMF. Currently my host system is legacy-booted with all VGA stuff routed to 01:00.0(so if i will push ctrl-alt-f1 while VM is off, i'll see a console on the screen), but when the system loads, my nvidia driver kicks in, loads the 04:00.0 GT610 card, and 01:00.0 is not used by host system(as i have my radeon blacklisted, but it's not prone to bind itself when not asked)...
Oh wait, you don't even need OVMF, if x-vga=on works correctly on your platform. For me it doesn't, so i use OVMF.
Anyway, i have the following lines in my xorg.conf line, maybe you'll like it too:

Section "Device" Identifier "Device0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "GeForce GT 610" BusID "PCI:04:00:00" EndSection

As you can see, i specify the device be it's BDF address, so if i would have multiple nvidia GPUs, normally only 04:00.0 would work.

Zeorymer wrote:

The R9 270X = 7870
The R9 280 = 7950

However there are quite a few 7950s on the list being passed sucessfully.

The card's just rare, i guess. Anyway, i don't remember them being any special, just another radeon southern(right?) islands GPU with reset problems that were fixed some time ago.

Zeorymer wrote:
Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:02 JK-Desktop kernel: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=07:00.0 address=0x00000002451e49e0] Jul 06 18:30:02 JK-Desktop kernel: BUG: unable to handle kernel paging request at ffffffffffffc039 Jul 06 18:30:02 JK-Desktop kernel: IP: [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: PGD 180e067 PUD 1810067 PMD 0 Jul 06 18:30:02 JK-Desktop kernel: Oops: 0000 [#1] PREEMPT SMP Jul 06 18:30:02 JK-Desktop kernel: Modules linked in: nls_utf8 isofs vfio_pci vfio_iommu_type1 vfio it87 hwmon_vid eeepc_wmi asus_wmi sparse_keymap led_class rfkill video kvm_amd kvm fuse crct10dif_pclmu Jul 06 18:30:02 JK-Desktop kernel: hid_logitech ff_memless hid_generic hid_microsoft usbhid hid uas usb_storage atkbd libps2 ahci libahci libata ohci_pci xhci_pci ohci_hcd ehci_pci xhci_hcd ehci_hcd scs Jul 06 18:30:02 JK-Desktop kernel: CPU: 4 PID: 720 Comm: cinnamon Not tainted 4.0.7-2-vfio #1 Jul 06 18:30:02 JK-Desktop kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO, BIOS 1604 10/16/2012 Jul 06 18:30:02 JK-Desktop kernel: task: ffff88023db2b2a0 ti: ffff8800a4144000 task.ti: ffff8800a4144000 Jul 06 18:30:02 JK-Desktop kernel: RIP: 0010:[<ffffffff81576b77>] [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: RSP: 0018:ffff8800a4147a08 EFLAGS: 00010292 Jul 06 18:30:02 JK-Desktop kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00000000c0000100 Jul 06 18:30:02 JK-Desktop kernel: RDX: ffff8800a4147fd8 RSI: ffff88023db2b2a0 RDI: ffff88024ed13e40 Jul 06 18:30:02 JK-Desktop kernel: RBP: ffff8800a4147a18 R08: ffff8800a4144000 R09: ffff88024ed13e70 Jul 06 18:30:02 JK-Desktop kernel: R10: 0000000000000020 R11: 0000000000000004 R12: 000000000006224f Jul 06 18:30:02 JK-Desktop kernel: R13: ffff8800a4147b80 R14: ffff8800a4147bec R15: 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: FS: 00007f532c8c39c0(0000) GS:ffff88024ed00000(0000) knlGS:0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 06 18:30:02 JK-Desktop kernel: CR2: ffffffffffffc039 CR3: 00000000a4ca2000 CR4: 00000000000407e0 Jul 06 18:30:02 JK-Desktop kernel: Stack: Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147bec 0000000000000000 ffff8800a4147ab8 ffffffff815798fc Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147a28 0000000000000000 0000000000000000 000000a9a6b041b6 Jul 06 18:30:02 JK-Desktop kernel: 000000a9a6aa1f67 ffffffff810e0bd0 ffff88024ed0e408 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: Call Trace: Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff815798fc>] schedule_hrtimeout_range_clock.part.6+0xfc/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff810e0bd0>] ? __run_hrtimer+0x250/0x250 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579891>] ? schedule_hrtimeout_range_clock.part.6+0x91/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579939>] schedule_hrtimeout_range_clock+0x19/0x50 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579983>] schedule_hrtimeout_range+0x13/0x20 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ec8f4>] poll_schedule_timeout+0x44/0x70 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee111>] do_sys_poll+0x531/0x600 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee2e1>] SyS_poll+0x71/0x130 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff8157a889>] system_call_fastpath+0x12/0x17 Jul 06 18:30:02 JK-Desktop kernel: Code: b8 00 00 48 89 e5 53 48 83 ec 08 48 8b 10 48 85 d2 74 0a 48 83 b8 a8 07 00 00 00 74 27 65 48 8b 1c 25 60 b8 00 00 e8 99 f5 ff ff <48> 8b 83 38 c0 ff ff a8 08 75 f Jul 06 18:30:02 JK-Desktop kernel: RIP [<ffffffff81576b77>] schedule+0x37/0x90

Actually, this looks very bad. I think only aw or someone his level can give you a proper advice. Strange thing those ioapic remapping commands didn't work.

Last edited by Duelist (2015-07-07 01:57:35)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5511 2015-07-07 04:26:44

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Actually, this looks very bad. I think only aw or someone his level can give you a proper advice. Strange thing those ioapic remapping commands didn't work.

So, in simple terms, I'm screwed because Asus messed up something.

I guess the easy way of fixing this is to replace the board with a known working one, which is hard to do with minimal funds.

Could power problems cause this?  I've had suspicions that my PSU is on it's way out for a bit now because of some strange behavior with Windows and that both my previous 8320 and the 7950 had to be RMA'd last year (roughly around the same time as well).


My sensors show this, but I don't know how accurate these sensors are.

it8721-isa-0290 Adapter: ISA adapter in0: +2.81 V (min = +2.63 V, max = +2.75 V) ALARM in1: +2.87 V (min = +2.84 V, max = +3.05 V) in2: +0.80 V (min = +0.00 V, max = +2.72 V) +3.3V: +3.31 V (min = +5.90 V, max = +2.74 V) ALARM in4: +2.27 V (min = +2.95 V, max = +1.50 V) ALARM in5: +2.50 V (min = +1.06 V, max = +2.35 V) ALARM in6: +1.54 V (min = +0.56 V, max = +2.53 V) 3VSB: +5.40 V (min = +3.02 V, max = +6.10 V) Vbat: +3.38 V fan1: 944 RPM (min = 20 RPM) fan2: 0 RPM (min = 291 RPM) ALARM fan3: 0 RPM (min = 66 RPM) ALARM temp1: +30.0°C (low = +97.0°C, high = -18.0°C) ALARM sensor = thermistor temp2: +27.0°C (low = -56.0°C, high = +94.0°C) sensor = thermistor temp3: -128.0°C (low = -1.0°C, high = +127.0°C) sensor = disabled intrusion0: OK

I do see some obvious garbage, like fans 2 and 3, that have nothing in them.

Last edited by Zeorymer (2015-07-07 04:27:34)

Offline

#5512 2015-07-07 07:48:05

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys
I've been working at this all day, but no matter what I do, IOMMU doesn't seem to ever enable, at the kernel level. My processor and mobo both (supposedly) have vt-d support, but dmesg | grep iommu only returns the boot parameters themselves; almost as if they are being completely ignored. My boot options are

intel_iommu=on,igfx_on,pass-through i915.enable_hd_vgaarb=1 pci-stub.ids=10de:13c0,10de:0fbb

And my build is:
ASUS Maximus VII Ranger
Intel i7 4970K (ark.intel does claim this K edition had VT-d support)
Intel HD 4600 iGPU (arch host)
Nvidia GTX 980 (passthrough)

I am so lost, google has been no help.

Update:
Made a bit of a mistake, dmesg | grep IOMMU (as opposed to iommu) returns [0.000000] Intel-IOMMU: enabled, and nothing else. /sys/kernel/iommu_groups is completely empty.

Last edited by morat (2015-07-07 08:38:34)

Offline

#5513 2015-07-07 14:19:47

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:

Could power problems cause this?  I've had suspicions that my PSU is on it's way out for a bit now because of some strange behavior with Windows and that both my previous 8320 and the 7950 had to be RMA'd last year (roughly around the same time as well).

Use the PSU calculator, like outervision's one, and compare calculations with your PSU.
Do not trust software voltmeters, use a physical one and push it into molex(IDE power) or pci-e power connector to find out 12V voltage.

...the motherboards for AM3+ are kinda cheap, however i understand your concern. Try joining #kvm and ask there. I feel an easy fix somewhere near.

And i've forgot, did you update your bios? It'd be useful to try every revision supporting IOMMU.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5514 2015-07-07 14:22:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

Hi guys
I've been working at this all day, but no matter what I do, IOMMU doesn't seem to ever enable, at the kernel level. My processor and mobo both (supposedly) have vt-d support, but dmesg | grep iommu only returns the boot parameters themselves; almost as if they are being completely ignored. My boot options are

intel_iommu=on,igfx_on,pass-through i915.enable_hd_vgaarb=1 pci-stub.ids=10de:13c0,10de:0fbb

And my build is:
ASUS Maximus VII Ranger
Intel i7 4970K (ark.intel does claim this K edition had VT-d support)
Intel HD 4600 iGPU (arch host)
Nvidia GTX 980 (passthrough)

I am so lost, google has been no help.

Update:
Made a bit of a mistake, dmesg | grep IOMMU (as opposed to iommu) returns [0.000000] Intel-IOMMU: enabled, and nothing else. /sys/kernel/iommu_groups is completely empty.

What does lsmod | grep vfio says?
It should be like

vfio_iommu_type1 20480 1 vfio_pci 36864 4 vfio 24576 9 vfio_iommu_type1,vfio_pci

but only when vfio is used. Try loading it manually via modprobe.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5515 2015-07-07 14:35:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
morat wrote:

Hi guys
I've been working at this all day, but no matter what I do, IOMMU doesn't seem to ever enable, at the kernel level. My processor and mobo both (supposedly) have vt-d support, but dmesg | grep iommu only returns the boot parameters themselves; almost as if they are being completely ignored. My boot options are

intel_iommu=on,igfx_on,pass-through i915.enable_hd_vgaarb=1 pci-stub.ids=10de:13c0,10de:0fbb

And my build is:
ASUS Maximus VII Ranger
Intel i7 4970K (ark.intel does claim this K edition had VT-d support)
Intel HD 4600 iGPU (arch host)
Nvidia GTX 980 (passthrough)

I am so lost, google has been no help.

Update:
Made a bit of a mistake, dmesg | grep IOMMU (as opposed to iommu) returns [0.000000] Intel-IOMMU: enabled, and nothing else. /sys/kernel/iommu_groups is completely empty.

What does lsmod | grep vfio says?
It should be like

vfio_iommu_type1 20480 1 vfio_pci 36864 4 vfio 24576 9 vfio_iommu_type1,vfio_pci

but only when vfio is used. Try loading it manually via modprobe.

vfio is irrelevant until the iommu gets squared away.  Is VT-d enabled in the BIOS?  Do you have a DMAR line like this in dmesg?

ACPI: DMAR 0x00000000DAFDB000 0000B8 (v01 INTEL SNB 00000001 INTL 00000001)

It's always a good idea to pastebin dmesg even if you don't know what you're looking for.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5516 2015-07-07 18:05:16

WildyLion
Member
Registered: 2015-06-07
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there.

I wonder if there's a way to use VFIO-PCI passthrough under Libvirt user session (qemu://session)?
It's been a huge success so far, but I'm using it via Libvirt system session, but it's keeping me from using PulseAudio, so far I keep getting the message that 'PCI device <ID> is not assignable>, is there any way around that?

Offline

#5517 2015-07-07 18:53:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

WildyLion wrote:

Hi there.

I wonder if there's a way to use VFIO-PCI passthrough under Libvirt user session (qemu://session)?
It's been a huge success so far, but I'm using it via Libvirt system session, but it's keeping me from using PulseAudio, so far I keep getting the message that 'PCI device <ID> is not assignable>, is there any way around that?

Not currently


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5518 2015-07-08 00:19:45

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

What does lsmod | grep vfio says?
It should be like

vfio_iommu_type1 20480 1 vfio_pci 36864 4 vfio 24576 9 vfio_iommu_type1,vfio_pci

but only when vfio is used. Try loading it manually via modprobe.

I have loaded vfio, and it does appear

aw wrote:

vfio is irrelevant until the iommu gets squared away.  Is VT-d enabled in the BIOS?  Do you have a DMAR line like this in dmesg?

ACPI: DMAR 0x00000000DAFDB000 0000B8 (v01 INTEL SNB 00000001 INTL 00000001)

It's always a good idea to pastebin dmesg even if you don't know what you're looking for.

Something has screwed my iGPU, and I can't get any output from it anymore; so I reset the CMOS, and now IOMMU seems to work. That would be the end of this, but now I cant use the onboard graphics as the display for arch, so can't bind the gpu, as its the only other output. Unfortunately I already have set it up to use the gpu. How can I roll back the changes I made from the OP?

Update:
I can use vga to get output from arch, so here is the dmesg

Last edited by morat (2015-07-08 00:56:41)

Offline

#5519 2015-07-08 01:25:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[ 0.000000] Command line: initrd=\initramfs-linux.img iommu=force,calgary intel_iommu=on root=PARTUUID=dc06f381-717e-4143-8e0e-a2ee69b1649c rw i915.enable_hd_vgaarb=1 pci-stub.ids=10de:13c0,10de:0fbb

So we're just making up kernel boot option now, right? (calgary)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5520 2015-07-08 14:07:24

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
[ 0.000000] Command line: initrd=\initramfs-linux.img iommu=force,calgary intel_iommu=on root=PARTUUID=dc06f381-717e-4143-8e0e-a2ee69b1649c rw i915.enable_hd_vgaarb=1 pci-stub.ids=10de:13c0,10de:0fbb

So we're just making up kernel boot option now, right? (calgary)

We are? I thought Calgary was the option to select Intel support.

Offline

#5521 2015-07-08 15:50:21

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:
[ 0.000000] Command line: initrd=\initramfs-linux.img iommu=force,calgary intel_iommu=on root=PARTUUID=dc06f381-717e-4143-8e0e-a2ee69b1649c rw i915.enable_hd_vgaarb=1 pci-stub.ids=10de:13c0,10de:0fbb

So we're just making up kernel boot option now, right? (calgary)

We are? I thought Calgary was the option to select Intel support.

Ok, even though it's not documented in kenel-parameters.txt calgary is a valid option there... but you don't have a calgary system, I hopedefinitely.

config CALGARY_IOMMU bool "IBM Calgary IOMMU support" select SWIOTLB depends on X86_64 && PCI ---help--- Support for hardware IOMMUs in IBM's xSeries x366 and x460 systems. Needed to run systems with more than 3GB of memory properly with 32-bit PCI devices that do not support DAC (Double Address Cycle). Calgary also supports bus level isolation, where all DMAs pass through the IOMMU. This prevents them from going anywhere except their intended destination. This catches hard-to-find kernel bugs and mis-behaving drivers and devices that do not use the DMA-API properly to set up their DMA buffers. The IOMMU can be turned off at boot time with the iommu=off parameter. Normally the kernel will make the right choice by itself. If unsure, say Y.

Last edited by aw (2015-07-08 16:27:38)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5522 2015-07-09 01:16:02

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Ok, even though it's not documented in kenel-parameters.txt calgary is a valid option there... but you don't have a calgary system, I hopedefinitely.

config CALGARY_IOMMU bool "IBM Calgary IOMMU support" select SWIOTLB depends on X86_64 && PCI ---help--- Support for hardware IOMMUs in IBM's xSeries x366 and x460 systems. Needed to run systems with more than 3GB of memory properly with 32-bit PCI devices that do not support DAC (Double Address Cycle). Calgary also supports bus level isolation, where all DMAs pass through the IOMMU. This prevents them from going anywhere except their intended destination. This catches hard-to-find kernel bugs and mis-behaving drivers and devices that do not use the DMA-API properly to set up their DMA buffers. The IOMMU can be turned off at boot time with the iommu=off parameter. Normally the kernel will make the right choice by itself. If unsure, say Y.

Ah, right, OK. Well that's good to know, but IOMMU is already functioning properly, as per my last post. It looks like the dvi port on the mobo is busted, for reasons unknown, VGA and HDMI work fine, but running the vfio-bind script on the GPU crashes the computer, and seems to completely halt the CPU (motherboard opcode). Any ideas?

Last edited by morat (2015-07-09 01:17:32)

Offline

#5523 2015-07-09 03:04:05

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

Ah, right, OK. Well that's good to know, but IOMMU is already functioning properly, as per my last post. It looks like the dvi port on the mobo is busted, for reasons unknown, VGA and HDMI work fine, but running the vfio-bind script on the GPU crashes the computer, and seems to completely halt the CPU (motherboard opcode). Any ideas?

IOMMU doesn't work right, and vfio-bind script sucks.
A fresh guide to follow.

@nbhs: mark the op-post as somewhat outdated and post a huge link to aw's blog, that'll help a lot, thanks!


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5524 2015-07-09 06:30:04

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

IOMMU doesn't work right, and vfio-bind script sucks.
A fresh guide to follow.

@nbhs: mark the op-post as somewhat outdated and post a huge link to aw's blog, that'll help a lot, thanks!

Excellent, thanks; and even moreso to @aw. I'll let you know if it works.
Will I need to rollback what I did from OP's guide, and if so, how?

Offline

#5525 2015-07-09 13:23:59

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

Excellent, thanks; and even moreso to @aw. I'll let you know if it works.
Will I need to rollback what I did from OP's guide, and if so, how?

Depending on where you've stopped. Disable the vfio-bind systemd service, delete its' remains, use a recent qemu and kernel - and you should be good to go for following aw's guide.

Gratitude is always welcome, but try reading modinfo vfio, he wrote that...

Last edited by Duelist (2015-07-09 14:41:17)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5526 2015-07-09 15:43:47

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone .

Sorry if this is slightly off-topic but is there any real performance or stability gains when using Libvirt over regular command line script to define and launch the VM ?

Offline

#5527 2015-07-09 16:09:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

libvirt is a VM management interface.  It facilitates many things about managing VMs, but it's not some magic wrapper that turbo boosts the VM.  It still emits a QEMU command line in the end.  If you want to keep using vfio-bind and manual scripts, then please also keep all the bugs that come along with them.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5528 2015-07-09 16:34:13

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

then please also keep all the bugs that come along with them.

Thank Alex . I saw that libvirt emits a longer QEMU command line too . I think using "options vfio-pci ids=" with a QEMU command script is the best solution for me . (I find it easier to edit a simple shell script that editing a libvirt XML) .

Thank you !

Offline

#5529 2015-07-10 01:03:54

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After following @aw's guide, libvirt returns:

Error starting domain: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

Looking through my journal since today's boot, I have a loooot of errors. At the time I try to run the VM, I get messages similar to this for every IOMMU device (where 0000:01:00.1 is my GPU:

libvirtd[5956]: internal error: Failed to find parent device for 0000:01:00.0 libvirtd[5956]: Failed to open config space file '/sys/bus/pci/devices/0000:00:00.0/config': Permission denied

Dmesg here
Something something wit's end?

Offline

#5530 2015-07-10 01:13:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

After following @aw's guide, libvirt returns:

Error starting domain: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

Looking through my journal since today's boot, I have a loooot of errors. At the time I try to run the VM, I get messages similar to this for every IOMMU device (where 0000:01:00.1 is my GPU:

libvirtd[5956]: internal error: Failed to find parent device for 0000:01:00.0 libvirtd[5956]: Failed to open config space file '/sys/bus/pci/devices/0000:00:00.0/config': Permission denied

Dmesg here
Something something wit's end?

session mode libvirt does not support device assignment, you need to use system mode libvirt (ie. run it as root).  Also, your attempt to load vfio-pci early:

rd.driver.pre=vfio-pci

Isn't working:

[ 8.591790] nouveau 0000:01:00.0: enabling device (0004 -> 0007) [ 8.591932] nouveau [ DEVICE][0000:01:00.0] BOOT0 : 0x124000a1 [ 8.591934] nouveau [ DEVICE][0000:01:00.0] Chipset: GM204 (NV124) [ 8.591934] nouveau [ DEVICE][0000:01:00.0] Family : NV110 [ 8.646855] nouveau [ VBIOS][0000:01:00.0] using image from PROM [ 8.646990] nouveau [ VBIOS][0000:01:00.0] BIT signature found [ 8.646992] nouveau [ VBIOS][0000:01:00.0] version 84.04.31.00.c9 [ 8.647160] nouveau [ DEVINIT][0000:01:00.0] adaptor not initialised [ 8.658974] nouveau [ PMC][0000:01:00.0] MSI interrupts enabled [ 8.658994] nouveau [ PFB][0000:01:00.0] RAM type: GDDR5 [ 8.658995] nouveau [ PFB][0000:01:00.0] RAM size: 4096 MiB [ 8.658995] nouveau [ PFB][0000:01:00.0] ZCOMP: 0 tags

You'll need to figure out the equivalent trick for Arch, rd.driver.pre works with dracut-based initramfs images, like Fedora.  You may want to start with a simple pci-stub.ids option and move on to something more complicated later.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5531 2015-07-10 02:16:28

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, some progress has been made...
When running the VM, libvirtd causes a kernel panic. After reading it, it seems like it might be caused by a bug in libvirt, but I'm no expert, so I'll defer to you. Dmesg here

Offline

#5532 2015-07-10 02:24:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You fat fingered your pci-stub.ids entry and didn't verify that it worked correctly.  You have:

pci-stub.ids=10de:13x0,10de:0fbb#i915.enable_hd_vgaarb=1

You want:

pci-stub.ids=10de:13c0,10de:0fbb i915.enable_hd_vgaarb=1

And then double check it worked with:

lspci -nnks 1:

Which should show the in-use driver is pci-stub.  Not sure why your libvirt install is so unhappy.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5533 2015-07-10 02:29:57

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

You fat fingered your pci-stub.ids entry and didn't verify that it worked correctly.  You have:

pci-stub.ids=10de:13x0,10de:0fbb#i915.enable_hd_vgaarb=1

You want:

pci-stub.ids=10de:13c0,10de:0fbb i915.enable_hd_vgaarb=1

And then double check it worked with:

lspci -nnks 1:

Which should show the in-use driver is pci-stub.  Not sure why your libvirt install is so unhappy.

Commenting out the vga fix was intentional, I have yet to patch the kernel, but that x.... Time to move on from the 2008 apple-supplied rubber dome keyboard, I think, causes more than a few mistakes.

Update:
I dont get a kernel panic any more, which is nice; but now libvirt spits this out:

Error starting domain: internal error: early end of file from monitor: possible problem: 2015-07-10T12:55:29.897699Z qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1433, in startup self._backend.create() File "/usr/lib/python2.7/site-packages/libvirt.py", line 1029, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: early end of file from monitor: possible problem: 2015-07-10T12:55:29.897699Z qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory

Hugepages have been set up as per your guide, @aw. Why break the trend?
It's so close, I can almost taste it...

Last edited by morat (2015-07-10 03:01:22)

Offline

#5534 2015-07-10 03:04:56

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg says you have 2048 2M pages, for 4G.  What's the memory size of the guest?  You could also start simple and add hugepages once you get things working...

EDIT:  Also, you have a GTX980, which I'd fully expect to support UEFI.  Why are you considering the i915/VGA route?

Last edited by aw (2015-07-10 03:06:48)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5535 2015-07-10 03:35:16

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

dmesg says you have 2048 2M pages, for 4G.  What's the memory size of the guest?  You could also start simple and add hugepages once you get things working...

EDIT:  Also, you have a GTX980, which I'd fully expect to support UEFI.  Why are you considering the i915/VGA route?

I need to use the IGP for Arch, as I don't have another discrete card. EDIT: I just realized how inane that is, I'm using the IGP right now without hassle; looks like I don't need i915. That being said, my onboard DVI is AWOL, and I have no idea why, in Arch, BIOS/UEFI, and the physical Windows.

It all seems to work now, or at least... start. I can't get networking going, libvirt lists my two physical interfaces, and an empty bridge. None of the three give the guest connectivity. I also can't mount a driver cd to the guest, virsh says hotplug isn't supported, and virt-manager claims it works, but the guest doesn't see the drive.

Last edited by morat (2015-07-10 03:37:07)

Offline

#5536 2015-07-10 03:36:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:

dmesg says you have 2048 2M pages, for 4G.  What's the memory size of the guest?  You could also start simple and add hugepages once you get things working...

EDIT:  Also, you have a GTX980, which I'd fully expect to support UEFI.  Why are you considering the i915/VGA route?

I need to use the IGP for Arch, as I don't have another discrete card.

Umm... you can still use IGD with the UEFI solution


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5537 2015-07-10 03:40:24

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Zeorymer wrote:
Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:02 JK-Desktop kernel: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=07:00.0 address=0x00000002451e49e0] Jul 06 18:30:02 JK-Desktop kernel: BUG: unable to handle kernel paging request at ffffffffffffc039 Jul 06 18:30:02 JK-Desktop kernel: IP: [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: PGD 180e067 PUD 1810067 PMD 0 Jul 06 18:30:02 JK-Desktop kernel: Oops: 0000 [#1] PREEMPT SMP Jul 06 18:30:02 JK-Desktop kernel: Modules linked in: nls_utf8 isofs vfio_pci vfio_iommu_type1 vfio it87 hwmon_vid eeepc_wmi asus_wmi sparse_keymap led_class rfkill video kvm_amd kvm fuse crct10dif_pclmu Jul 06 18:30:02 JK-Desktop kernel: hid_logitech ff_memless hid_generic hid_microsoft usbhid hid uas usb_storage atkbd libps2 ahci libahci libata ohci_pci xhci_pci ohci_hcd ehci_pci xhci_hcd ehci_hcd scs Jul 06 18:30:02 JK-Desktop kernel: CPU: 4 PID: 720 Comm: cinnamon Not tainted 4.0.7-2-vfio #1 Jul 06 18:30:02 JK-Desktop kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO, BIOS 1604 10/16/2012 Jul 06 18:30:02 JK-Desktop kernel: task: ffff88023db2b2a0 ti: ffff8800a4144000 task.ti: ffff8800a4144000 Jul 06 18:30:02 JK-Desktop kernel: RIP: 0010:[<ffffffff81576b77>] [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: RSP: 0018:ffff8800a4147a08 EFLAGS: 00010292 Jul 06 18:30:02 JK-Desktop kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00000000c0000100 Jul 06 18:30:02 JK-Desktop kernel: RDX: ffff8800a4147fd8 RSI: ffff88023db2b2a0 RDI: ffff88024ed13e40 Jul 06 18:30:02 JK-Desktop kernel: RBP: ffff8800a4147a18 R08: ffff8800a4144000 R09: ffff88024ed13e70 Jul 06 18:30:02 JK-Desktop kernel: R10: 0000000000000020 R11: 0000000000000004 R12: 000000000006224f Jul 06 18:30:02 JK-Desktop kernel: R13: ffff8800a4147b80 R14: ffff8800a4147bec R15: 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: FS: 00007f532c8c39c0(0000) GS:ffff88024ed00000(0000) knlGS:0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 06 18:30:02 JK-Desktop kernel: CR2: ffffffffffffc039 CR3: 00000000a4ca2000 CR4: 00000000000407e0 Jul 06 18:30:02 JK-Desktop kernel: Stack: Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147bec 0000000000000000 ffff8800a4147ab8 ffffffff815798fc Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147a28 0000000000000000 0000000000000000 000000a9a6b041b6 Jul 06 18:30:02 JK-Desktop kernel: 000000a9a6aa1f67 ffffffff810e0bd0 ffff88024ed0e408 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: Call Trace: Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff815798fc>] schedule_hrtimeout_range_clock.part.6+0xfc/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff810e0bd0>] ? __run_hrtimer+0x250/0x250 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579891>] ? schedule_hrtimeout_range_clock.part.6+0x91/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579939>] schedule_hrtimeout_range_clock+0x19/0x50 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579983>] schedule_hrtimeout_range+0x13/0x20 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ec8f4>] poll_schedule_timeout+0x44/0x70 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee111>] do_sys_poll+0x531/0x600 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee2e1>] SyS_poll+0x71/0x130 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff8157a889>] system_call_fastpath+0x12/0x17 Jul 06 18:30:02 JK-Desktop kernel: Code: b8 00 00 48 89 e5 53 48 83 ec 08 48 8b 10 48 85 d2 74 0a 48 83 b8 a8 07 00 00 00 74 27 65 48 8b 1c 25 60 b8 00 00 e8 99 f5 ff ff <48> 8b 83 38 c0 ff ff a8 08 75 f Jul 06 18:30:02 JK-Desktop kernel: RIP [<ffffffff81576b77>] schedule+0x37/0x90

Actually, this looks very bad. I think only aw or someone his level can give you a proper advice. Strange thing those ioapic remapping commands didn't work.

Since aw is in the thread, may I have your opinion on my troubles?  That is, if you have any idea.

Offline

#5538 2015-07-10 04:31:24

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great, everything works; even the PCI wifi card. Oh, except the Code 43.

I'm using a GTX 980, is it too new for the workarounds? Was all this for nothing?

Offline

#5539 2015-07-10 12:53:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:
Duelist wrote:
Zeorymer wrote:
Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:01 JK-Desktop kernel: AMD-Vi: Completion-Wait loop timed out Jul 06 18:30:02 JK-Desktop kernel: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=07:00.0 address=0x00000002451e49e0] Jul 06 18:30:02 JK-Desktop kernel: BUG: unable to handle kernel paging request at ffffffffffffc039 Jul 06 18:30:02 JK-Desktop kernel: IP: [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: PGD 180e067 PUD 1810067 PMD 0 Jul 06 18:30:02 JK-Desktop kernel: Oops: 0000 [#1] PREEMPT SMP Jul 06 18:30:02 JK-Desktop kernel: Modules linked in: nls_utf8 isofs vfio_pci vfio_iommu_type1 vfio it87 hwmon_vid eeepc_wmi asus_wmi sparse_keymap led_class rfkill video kvm_amd kvm fuse crct10dif_pclmu Jul 06 18:30:02 JK-Desktop kernel: hid_logitech ff_memless hid_generic hid_microsoft usbhid hid uas usb_storage atkbd libps2 ahci libahci libata ohci_pci xhci_pci ohci_hcd ehci_pci xhci_hcd ehci_hcd scs Jul 06 18:30:02 JK-Desktop kernel: CPU: 4 PID: 720 Comm: cinnamon Not tainted 4.0.7-2-vfio #1 Jul 06 18:30:02 JK-Desktop kernel: Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO, BIOS 1604 10/16/2012 Jul 06 18:30:02 JK-Desktop kernel: task: ffff88023db2b2a0 ti: ffff8800a4144000 task.ti: ffff8800a4144000 Jul 06 18:30:02 JK-Desktop kernel: RIP: 0010:[<ffffffff81576b77>] [<ffffffff81576b77>] schedule+0x37/0x90 Jul 06 18:30:02 JK-Desktop kernel: RSP: 0018:ffff8800a4147a08 EFLAGS: 00010292 Jul 06 18:30:02 JK-Desktop kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00000000c0000100 Jul 06 18:30:02 JK-Desktop kernel: RDX: ffff8800a4147fd8 RSI: ffff88023db2b2a0 RDI: ffff88024ed13e40 Jul 06 18:30:02 JK-Desktop kernel: RBP: ffff8800a4147a18 R08: ffff8800a4144000 R09: ffff88024ed13e70 Jul 06 18:30:02 JK-Desktop kernel: R10: 0000000000000020 R11: 0000000000000004 R12: 000000000006224f Jul 06 18:30:02 JK-Desktop kernel: R13: ffff8800a4147b80 R14: ffff8800a4147bec R15: 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: FS: 00007f532c8c39c0(0000) GS:ffff88024ed00000(0000) knlGS:0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 06 18:30:02 JK-Desktop kernel: CR2: ffffffffffffc039 CR3: 00000000a4ca2000 CR4: 00000000000407e0 Jul 06 18:30:02 JK-Desktop kernel: Stack: Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147bec 0000000000000000 ffff8800a4147ab8 ffffffff815798fc Jul 06 18:30:02 JK-Desktop kernel: ffff8800a4147a28 0000000000000000 0000000000000000 000000a9a6b041b6 Jul 06 18:30:02 JK-Desktop kernel: 000000a9a6aa1f67 ffffffff810e0bd0 ffff88024ed0e408 0000000000000000 Jul 06 18:30:02 JK-Desktop kernel: Call Trace: Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff815798fc>] schedule_hrtimeout_range_clock.part.6+0xfc/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff810e0bd0>] ? __run_hrtimer+0x250/0x250 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579891>] ? schedule_hrtimeout_range_clock.part.6+0x91/0x120 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579939>] schedule_hrtimeout_range_clock+0x19/0x50 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff81579983>] schedule_hrtimeout_range+0x13/0x20 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ec8f4>] poll_schedule_timeout+0x44/0x70 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee111>] do_sys_poll+0x531/0x600 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811eca70>] ? poll_select_copy_remaining+0x150/0x150 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff811ee2e1>] SyS_poll+0x71/0x130 Jul 06 18:30:02 JK-Desktop kernel: [<ffffffff8157a889>] system_call_fastpath+0x12/0x17 Jul 06 18:30:02 JK-Desktop kernel: Code: b8 00 00 48 89 e5 53 48 83 ec 08 48 8b 10 48 85 d2 74 0a 48 83 b8 a8 07 00 00 00 74 27 65 48 8b 1c 25 60 b8 00 00 e8 99 f5 ff ff <48> 8b 83 38 c0 ff ff a8 08 75 f Jul 06 18:30:02 JK-Desktop kernel: RIP [<ffffffff81576b77>] schedule+0x37/0x90

Actually, this looks very bad. I think only aw or someone his level can give you a proper advice. Strange thing those ioapic remapping commands didn't work.

Since aw is in the thread, may I have your opinion on my troubles?  That is, if you have any idea.

Have you tried disabling hugepages for the IOMMU yet? (modprobe.d: options vfio_iommu_type1 disable_hugepages=1)  What's the processor?  This seems like a common problem on APUs but seems to work fine on non-APU processors.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5540 2015-07-10 12:55:20

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

Great, everything works; even the PCI wifi card. Oh, except the Code 43.

I'm using a GTX 980, is it too new for the workarounds? Was all this for nothing?

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5541 2015-07-10 16:12:25

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Have you tried disabling hugepages for the IOMMU yet? (modprobe.d: options vfio_iommu_type1 disable_hugepages=1)  What's the processor?  This seems like a common problem on APUs but seems to work fine on non-APU processors.

He has a family 15h CPU with

00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]

alike device, so i think it's an FX platform(not APU, APUs don't work without their specific southbridges) and the initial problem was

[Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found AMD-Vi: Disabling interrupt remapping

I've suggested mapping ioapic directly via commandline, like

ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1

but that(after address adjustments) moved him to IOTLB_INV_TIMEOUT error.

The iommu dump he posted is:

AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 AMD-Vi: mmio-addr: 00000000feb20000 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:00.2 AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 01:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 01:00.1 AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 AMD-Vi: DEV_SELECT devid: 02:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:12.2 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:13.2 AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 AMD-Vi: DEV_RANGE_END devid: 06:1f.7 AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 AMD-Vi: DEV_SELECT devid: 00:15.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 07:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 07:00.1 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:16.2 AMD-Vi: DEV_SPECIAL(IOAPIC[0]) devid: 00:14.0 AMD-Vi: DEV_SPECIAL(HPET[0]) devid: 00:14.0 AMD-Vi: DEV_SPECIAL(IOAPIC[255]) devid: 00:00.1

Maybe you can suggest something else?

Last edited by Duelist (2015-07-10 16:13:39)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5542 2015-07-10 16:14:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist

Suggestion: Working motherboard


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5543 2015-07-10 16:23:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Duelist

Suggestion: Working motherboard

Crap. Is his IVRS table messed up so bad?

@Zeorymer:
Can you tell us an exact model of your motherboard? I know that it's ASUS, but which one?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5544 2015-07-10 16:24:41

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

@Duelist

Suggestion: Working motherboard

Crap. Is his IVRS table messed up so bad?

Dunno, but it's beyond the scope of this thread.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5545 2015-07-10 16:35:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Duelist wrote:
aw wrote:

@Duelist

Suggestion: Working motherboard

Crap. Is his IVRS table messed up so bad?

Dunno, but it's beyond the scope of this thread.

I understand your tiredness of supporting countless people, but if i'll come into #kvm and ask everyone, including you, about that..:D


http://www.gossamer-threads.com/lists/xen/devel/274857
https://bbs.archlinux.org/viewtopic.php?id=163102

Suddenly, i've cleared my vision and other google results started to seem fitting.

Last edited by Duelist (2015-07-10 16:40:47)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5546 2015-07-10 16:35:45

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw, What kernel are you currently running? I just updated to FC22 and stock 4.0.6-300 kernel without issues so far. Do you stay with Rawhide or within the FC releases for kernel packages? Or just compile your own :-)

Offline

#5547 2015-07-10 16:42:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
aw wrote:

Have you tried disabling hugepages for the IOMMU yet? (modprobe.d: options vfio_iommu_type1 disable_hugepages=1)  What's the processor?  This seems like a common problem on APUs but seems to work fine on non-APU processors.

He has a family 15h CPU with

00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]

alike device, so i think it's an FX platform(not APU, APUs don't work without their specific southbridges) and the initial problem was

[Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found AMD-Vi: Disabling interrupt remapping

I've suggested mapping ioapic directly via commandline, like

ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1

but that(after address adjustments) moved him to IOTLB_INV_TIMEOUT error.

The iommu dump he posted is:

AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 AMD-Vi: mmio-addr: 00000000feb20000 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:00.2 AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 01:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 01:00.1 AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 AMD-Vi: DEV_SELECT devid: 02:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:12.2 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:13.2 AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 AMD-Vi: DEV_ALIAS_RANGE devid: 06:00.0 flags: 00 devid_to: 00:14.4 AMD-Vi: DEV_RANGE_END devid: 06:1f.7 AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 AMD-Vi: DEV_SELECT devid: 00:15.0 flags: 00 AMD-Vi: DEV_SELECT_RANGE_START devid: 07:00.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 07:00.1 AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 AMD-Vi: DEV_RANGE_END devid: 00:16.2 AMD-Vi: DEV_SPECIAL(IOAPIC[0]) devid: 00:14.0 AMD-Vi: DEV_SPECIAL(HPET[0]) devid: 00:14.0 AMD-Vi: DEV_SPECIAL(IOAPIC[255]) devid: 00:00.1

Maybe you can suggest something else?

ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1

Last edited by nbhs (2015-07-10 16:43:04)

Offline

#5548 2015-07-10 16:47:33

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1
Zeorymer wrote:
/init/ line 1 = ivrs_ioapic[9]=00:14.0 : Command not found /init/ line 1 = ivrs_ioapic[10]=00:00.1 : Command not found

or something like that.

Wait, how could i miss that part of his message?>_<
Sorry for bothering guys.

@Zeorymer:
Are you sure you've put that command line switches into the right... place?
Cause if you're running kernel newer than 3.19(where there's a patch for that) - that line is your only solution apart from aw's ultimate one.

Last edited by Duelist (2015-07-10 16:48:37)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5549 2015-07-10 17:55:46

Zeorymer
Member
Registered: 2015-07-03
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
nbhs wrote:
ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1
Zeorymer wrote:
/init/ line 1 = ivrs_ioapic[9]=00:14.0 : Command not found /init/ line 1 = ivrs_ioapic[10]=00:00.1 : Command not found

or something like that.

Wait, how could i miss that part of his message?>_<
Sorry for bothering guys.

@Zeorymer:
Are you sure you've put that command line switches into the right... place?
Cause if you're running kernel newer than 3.19(where there's a patch for that) - that line is your only solution apart from aw's ultimate one.

Kernel was posted several posts ago: 4.0.7
Processor is an 8350.

GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Arch" GRUB_CMDLINE_LINUX_DEFAULT="iommu=debug,verbose amd_iommu_dump=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" GRUB_CMDLINE_LINUX="" # # Preload both GPT and MBR modules so that they are not missed GRUB_PRELOAD_MODULES="part_gpt part_msdos" # Uncomment to enable Hidden Menu, and optionally hide the timeout count #GRUB_HIDDEN_TIMEOUT=5 #GRUB_HIDDEN_TIMEOUT_QUIET=true # Uncomment to use basic console GRUB_TERMINAL_INPUT=console # Uncomment to disable graphical terminal #GRUB_TERMINAL_OUTPUT=console # The resolution used on graphical terminal # note that you can use only modes which your graphic card supports via VBE # you can see them in real GRUB with the command `vbeinfo' GRUB_GFXMODE=auto # Uncomment to allow the kernel use the same resolution used by grub GRUB_GFXPAYLOAD_LINUX=keep # Uncomment if you want GRUB to pass to the Linux kernel the old parameter # format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx" #GRUB_DISABLE_LINUX_UUID=true # Uncomment to disable generation of recovery mode menu entries #GRUB_DISABLE_RECOVERY=true # Uncomment and set to the desired menu colors. Used by normal and wallpaper # modes only. Entries specified as foreground/background. GRUB_COLOR_NORMAL="light-blue/black" GRUB_COLOR_HIGHLIGHT="light-cyan/blue" # Uncomment one of them for the gfx desired, a image background or a gfxtheme #GRUB_BACKGROUND="/path/to/wallpaper" #GRUB_THEME="/path/to/gfxtheme" # Uncomment to get a beep at GRUB start #GRUB_INIT_TUNE="480 440 1" #GRUB_SAVEDEFAULT="true"

This is where I put it, is this wrong?  I know there are other things in there that I recently added to see if I can get other information (first one doesn't seem to do anything new).

What I find odd id that I get the command not found messages but I don't get the IVRS messages being mapped wrong...which leads me to believe that the kernel THINKS the command is being passed(?)

So...a new board is the only solution?  Damn you Asus.

Last edited by Zeorymer (2015-07-10 17:59:26)

Offline

#5550 2015-07-10 18:17:38

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Zeorymer

You can file a bug or post to the iommu mailing list

@The_Moves

Whatever I need for the situation.  Often rc kernels for development and testing, latest Fedora kernel otherwise.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5551 2015-07-10 19:47:32

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Zeorymer wrote:
Duelist wrote:
nbhs wrote:
ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1
Zeorymer wrote:
/init/ line 1 = ivrs_ioapic[9]=00:14.0 : Command not found /init/ line 1 = ivrs_ioapic[10]=00:00.1 : Command not found

or something like that.

Wait, how could i miss that part of his message?>_<
Sorry for bothering guys.

@Zeorymer:
Are you sure you've put that command line switches into the right... place?
Cause if you're running kernel newer than 3.19(where there's a patch for that) - that line is your only solution apart from aw's ultimate one.

Kernel was posted several posts ago: 4.0.7
Processor is an 8350.

GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="Arch" GRUB_CMDLINE_LINUX_DEFAULT="iommu=debug,verbose amd_iommu_dump=1 ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" GRUB_CMDLINE_LINUX="" # # Preload both GPT and MBR modules so that they are not missed GRUB_PRELOAD_MODULES="part_gpt part_msdos" # Uncomment to enable Hidden Menu, and optionally hide the timeout count #GRUB_HIDDEN_TIMEOUT=5 #GRUB_HIDDEN_TIMEOUT_QUIET=true # Uncomment to use basic console GRUB_TERMINAL_INPUT=console # Uncomment to disable graphical terminal #GRUB_TERMINAL_OUTPUT=console # The resolution used on graphical terminal # note that you can use only modes which your graphic card supports via VBE # you can see them in real GRUB with the command `vbeinfo' GRUB_GFXMODE=auto # Uncomment to allow the kernel use the same resolution used by grub GRUB_GFXPAYLOAD_LINUX=keep # Uncomment if you want GRUB to pass to the Linux kernel the old parameter # format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx" #GRUB_DISABLE_LINUX_UUID=true # Uncomment to disable generation of recovery mode menu entries #GRUB_DISABLE_RECOVERY=true # Uncomment and set to the desired menu colors. Used by normal and wallpaper # modes only. Entries specified as foreground/background. GRUB_COLOR_NORMAL="light-blue/black" GRUB_COLOR_HIGHLIGHT="light-cyan/blue" # Uncomment one of them for the gfx desired, a image background or a gfxtheme #GRUB_BACKGROUND="/path/to/wallpaper" #GRUB_THEME="/path/to/gfxtheme" # Uncomment to get a beep at GRUB start #GRUB_INIT_TUNE="480 440 1" #GRUB_SAVEDEFAULT="true"

This is where I put it, is this wrong?  I know there are other things in there that I recently added to see if I can get other information (first one doesn't seem to do anything new).

What I find odd id that I get the command not found messages but I don't get the IVRS messages being mapped wrong...which leads me to believe that the kernel THINKS the command is being passed(?)

So...a new board is the only solution?  Damn you Asus.

/init/ line 1 = ivrs_ioapic[9]=00:14.0 : Command not found /init/ line 1 = ivrs_ioapic[10]=00:00.1 : Command not found

Those lines mean nothing, i get them too, the kernel arguments get passed you can check by doing cat /proc/cmdline, it has something to do with the mknitcpio script parsing the kernel parameters, but its nothing to worry about

Offline

#5552 2015-07-10 21:28:40

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I hit a brick wall trying to add SATA passthrough to my setup. Passing through the seconday onboard controller, a Marvell 6121 if I remember correctly, locked everything up, and I mean everything. Even the hardware reset button did nothing. WTF? Obviously I could not get any error messages out of this. I purchased a PCIe SATA addon card. That one's a Marvell 91xx. The thought of using something else than Marvell crossed my mind but I couldn't find anything reasonable at the moment.

I connected one of my SSDs to the device. My grub is actually on that SSD but the drive is not needed afterwards and my system boots fine with the controller pci-stubbed. Of course I wouldn't be able to install anything on it without breaking my host OS but this is just a proof of concept to see if it works before I rearrange things.

Now I don't get the hard crash but the SATA passthrough still does not work. For one thing, SeaBIOS takes long, maybe almost a minute before anything else than the SeaBIOS version string appears every time I pass through the devices on the addon card. When it gets going, I can't get any bootable devices, not even the virtualised ones defined with the -drive option. No install media or disk images show up. Enabling the boot menu shows only the virtualised floppy drive, PXE boot and such. I tried starting a very minimalistic setup to rule out everything else but the result is the same.

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1

where 7:00.0 is the SATA device and 7:00.1 is an IDE device on the same card. They share the same IOMMU group so I'm passing both of them. The install medium is an untouched Windows 7 SP1 image. I also tried installing the guest to a virtualised image and adding the SATA controller on the next boot. Everything gone again. I cannot understand why even the virtualised drives disappear. Am I doing something obviously wrong? As for providing debug output, I haven't found anything to provide. Nothing appears in dmesg when things go haywire. I initially got "nobody cared about IRQ" messages but moving the SATA card to another PCIe slot with its own IRQ fixed those. I see libvirt is the common way nowadays. I tried the SeaBIOS way with virt-manager but it behaved exactly the same as with using qemu directly.

I also tried using OVMF instead of SeaBIOS but I had even less success with that. The Windows installer always froze at the animated Win logo. I tried with an old OVMF version (r15214) and the latest build from https://www.kraxel.org/repos/jenkins/edk2/. My very minimalistic VM, without even any drives attached:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -drive file=OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=OVMF_VARS-pure-efi.fd,if=pflash,format=raw,unit=1

Interestingly, passing through the SATA controller does not hinder the boot when using OVMF but I cannot see if it would actually work since I can't get anywhere in installing the guest.
So frustrating when everything else than the SATA matter is falling into place so nicely.

Offline

#5553 2015-07-10 21:33:40

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

impulse_255 wrote:

I also tried using OVMF instead of SeaBIOS but I had even less success with that. The Windows installer always froze at the animated Win logo. I tried with an old OVMF version (r15214) and the latest build from https://www.kraxel.org/repos/jenkins/edk2/. My very minimalistic VM, without even any drives attached:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -drive file=OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=OVMF_VARS-pure-efi.fd,if=pflash,format=raw,unit=1

Interestingly, passing through the SATA controller does not hinder the boot when using OVMF but I cannot see if it would actually work since I can't get anywhere in installing the guest.
So frustrating when everything else than the SATA matter is falling into place so nicely.

i dont think OVMF is able to boot from a sata controller right now, i posted a patched binary like 50 pages back, theres also ovmf-fishman-git on aur

Last edited by nbhs (2015-07-10 21:34:49)

Offline

#5554 2015-07-10 22:29:31

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
impulse_255 wrote:

I also tried using OVMF instead of SeaBIOS but I had even less success with that. The Windows installer always froze at the animated Win logo. I tried with an old OVMF version (r15214) and the latest build from https://www.kraxel.org/repos/jenkins/edk2/. My very minimalistic VM, without even any drives attached:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 -cdrom windows_7.iso -drive file=OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=OVMF_VARS-pure-efi.fd,if=pflash,format=raw,unit=1

Interestingly, passing through the SATA controller does not hinder the boot when using OVMF but I cannot see if it would actually work since I can't get anywhere in installing the guest.
So frustrating when everything else than the SATA matter is falling into place so nicely.

i dont think OVMF is able to boot from a sata controller right now, i posted a patched binary like 50 pages back, theres also ovmf-fishman-git on aur

Good to know, thank you. I also just noticed Alex (thank you as well for all the materials on VFIO and passthrough) has said Windows 8+ is needed for OVMF, my bad. From what I have read, the 64-bit version of 7 should work with UEFI so I thought I would be fine.

Offline

#5555 2015-07-10 23:03:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@impulse_255

It is possible to use OVMF with win7, I haven't gone back to add that blog post yet.  Seems like I ran into libvirt issues that made me back-burner it for a while.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5556 2015-07-11 00:30:20

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@impulse_255

It is possible to use OVMF with win7, I haven't gone back to add that blog post yet.  Seems like I ran into libvirt issues that made me back-burner it for a while.

Off-topic:
#5555
Could you please explain what...
"made me back-burner it" meant? My foolish mind only finds associations with butthurt.
On-topic:
Win7 with OVMF requires some special hacks - you should add QXL graphics device for the duration of install, otherwise you won't be able to see anything.
This is covered in OVMF whitepaper.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5557 2015-07-11 00:32:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5558 2015-07-11 00:42:14

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well...
Seems like it's the different cultures - if not the same size, all the big burners on the stove are located farther away from the edge. I guess it has to do something with fire safety. And usually that burners are used for some long-term powerful cooking, like a 10L of some soup. So yeah, it kinda makes sense.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5559 2015-07-11 01:18:27

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
morat wrote:

Great, everything works; even the PCI wifi card. Oh, except the Code 43.

I'm using a GTX 980, is it too new for the workarounds? Was all this for nothing?

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.

Yeah, I did, to the letter. Guest is running 344.11 drivers, connected via DVI; no output. Code 43 at all times, and Windows doesnt recognize it without drivers, not even as a generic display device. All hyper-v extentions, the clock entry, the lot, have been removed from the xml config. Is there anything else that can trigger it?

Offline

#5560 2015-07-11 02:54:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:
morat wrote:

Great, everything works; even the PCI wifi card. Oh, except the Code 43.

I'm using a GTX 980, is it too new for the workarounds? Was all this for nothing?

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.

Yeah, I did, to the letter. Guest is running 344.11 drivers, connected via DVI; no output. Code 43 at all times, and Windows doesnt recognize it without drivers, not even as a generic display device. All hyper-v extentions, the clock entry, the lot, have been removed from the xml config. Is there anything else that can trigger it?

"Removed"?
Correct me if i'm wrong, but check again, i think some of them were default on.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5561 2015-07-11 03:05:48

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
morat wrote:
aw wrote:

Sorry, no data, no help.  I'm not going to guess whether you followed the instructions to edit the domain xml to remove the hyper-v extensions.

Yeah, I did, to the letter. Guest is running 344.11 drivers, connected via DVI; no output. Code 43 at all times, and Windows doesnt recognize it without drivers, not even as a generic display device. All hyper-v extentions, the clock entry, the lot, have been removed from the xml config. Is there anything else that can trigger it?

"Removed"?
Correct me if i'm wrong, but check again, i think some of them were default on.

As in, I removed them.

Offline

#5562 2015-07-11 03:07:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Don't tell us what you did, show us


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5563 2015-07-11 03:19:03

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I get home I'll upload the domain XML, is there anything else you need to see?

Offline

#5564 2015-07-11 14:41:15

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OT

I guess it has to do something with fire safety. And usually that burners are used for some long-term powerful cooking, like a 10L of some soup. So yeah, it kinda makes sense.

Don't any of you cook? Or do you just sit at the computer all day until food magically appears courtesy of some long neglected spouse? :roll eyes:

Last edited by mostlyharmless (2015-07-11 14:41:44)

Offline

#5565 2015-07-11 22:57:30

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgraded to 4.2-rc1 :

All of the VMs with GPUs passed through to them suffer from no output on their respective screens , checking htop , these VMs are stuck on 100% CPU load on their threads .

dmesg shows no errors or warnings . QEMU command line doesn't return any errors or warnings either .

Reverting back to 4.1-rc7 , and everything is working like it should . (No config or script files changes were done , just upgrading & downgrading the kernel) .

Last edited by Denso (2015-07-11 22:59:26)

Offline

#5566 2015-07-11 23:39:54

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success at last! Though not without lots of setbacks and cursing at my computer.

With OVMF and Windows 8 I managed to pass through my add-on Marvell controller. However, with the latest OVMF build I could not install Windows. At the point the installer starts copying files to the new system partition, I got a BSOD (or is it a frowny face these days?) with a KMODE_EXCEPTION_NOT_HANDLED error code. This only occurred when running on OVMF, not on SeaBIOS. In UEFI mode, Windows also seems to be VERY pissy about showing disk drives that are not exactly the way it wants (GPT and such), even with drive images that should be empty. I recall it was just as terrible when I tried without success to install Win 7 in UEFI mode on this hardware back in 2011 so no surprises there.

I dug up the SATA patched OVMF binaries as hinted by nbhs. With them, the installer finished without problems! I have not tried installing on a passed-through drive or booting from one yet. The system drive is on VirtIO and the drive on the passed controller is a secondary one that I added after installation. During the first boot with the controller passed through I got a BSOD related to viostor (VirtIO). It also froze my host, but at least the reset button still worked wink. Next boot went smoothly. Don't know what went wrong there.

Even if this is as far as I'll manage to get, it should already be helpful if the games are installed on the passed drive and load their stuff from there. On a side note, the Marvell controller's performance is shoddy. I get 180MB/90MB R/W sequential. Googling revealed that it seems to be on par with what can be expected. Access times are down to 0.3-0.5 ms though, which is nice. I have not tried games on the passed drive yet but I'm hoping the IO related stuttering isn't there anymore. Lots of other optimisations such as cpu pinning still to do, anyway.

Offline

#5567 2015-07-12 05:04:39

sunmast
Member
Registered: 2014-12-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to run OSX with OVMF method?

Offline

#5568 2015-07-12 06:16:45

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@impulse_255

It is possible to use OVMF with win7, I haven't gone back to add that blog post yet.  Seems like I ran into libvirt issues that made me back-burner it for a while.

Yes, like others here mentioned all disks reconized by win7uefi need gpt. For dual boot you need to add legacy ide drivers in win registry if virtio drivers are used in libvirt/qemu.
On my machine sata controller-passtrough never worked the expected way, also with marvell-sata-controller. Others got it running with Asrock Z97 Extreme6. One time i´ve seen the disk during win7 ovmf installation, after installation reboot, it also hung up. Maybe it was the graphic-card like mentioned.

I took the route without sata-controller passthrough since i´ve read in several posts, performance is comparable and system-stability without is better.
Don´t know exactly if this is right.

Good luck @Impulse_255

Offline

#5569 2015-07-12 09:05:25

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fkl8JlE.jpg
Aww, going from Catalyst 14.12 to Catalyst 15.7 on Win7 wasn't as easy as I had hoped sad

Offline

#5570 2015-07-12 17:43:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Upgraded to 4.2-rc1 :

All of the VMs with GPUs passed through to them suffer from no output on their respective screens , checking htop , these VMs are stuck on 100% CPU load on their threads .

dmesg shows no errors or warnings . QEMU command line doesn't return any errors or warnings either .

Reverting back to 4.1-rc7 , and everything is working like it should . (No config or script files changes were done , just upgrading & downgrading the kernel) .

Thanks for the report!  I've bisected it back to this commit.

In my testing, the VM is not hung, but it does take over 90 seconds to get the OVMF splash screen (Seabios is less affected but still noticeable).  I've reported the problem upstream.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5571 2015-07-12 17:55:14

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Thanks ! Actually it takes around 5 minutes for me , and the VM operates in Slow motion mode . (When a key stroke is sent the actual response happens 5 seconds later) .

Hopefully it will be fixed for -rc2 or 3 .

Offline

#5572 2015-07-12 17:58:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@aw :

Thanks ! Actually it takes around 5 minutes for me , and the VM operates in Slow motion mode . (When a key stroke is sent the actual response happens 5 seconds later) .

Hopefully it will be fixed for -rc2 or 3 .

The only issue I see is pre-boot, once the guest OS is running, performance is normal.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5573 2015-07-12 20:29:09

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sunmast wrote:

Is it possible to run OSX with OVMF method?

Yes. I have a proof-picture, if you're curious.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5574 2015-07-12 20:31:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://lists.nongnu.org/archive/html/qe … 00021.html
Isn't that's just what VFIO was made for? I mean, Virtual Function...

ALSO:
https://support.microsoft.com/en-us/kb/314082/en-us
Finally found an article for those of you who experiment with various hard drive controllers, like virtio, bare metal and stuff.

Last edited by Duelist (2015-07-12 20:40:34)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5575 2015-07-12 21:21:22

Peetipablo
Member
Registered: 2015-06-26
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So my VM is pretty stable and games are playable, however I'm having some serious problems with audio.

Passing through host audio with -soundhw hda gives me verybad DPC latency, spiking up to 8000 microseconds regularly and never drops below 1000.  Running the process using round robin scheduler as reccomended on page 6 works helps, DPC latency is stable at 1000 and never spiking however whenever I quit the VM, qemu process redlines all my processor cores and my computer becomes very unresponsive unless I am able to tab to a terminal and killall qemu-system-x86_64.

I also have a USB razer kraken headset, however passing it using -usbdevice there is much audio cracking and nothing I try reduces that at all.

I've tried using both alsa and pulse for audio and neither work better than the other... anyone have any suggestions on how I can reduce my DPC latency to normal values?

Edit:

Specifications:
Processor - Intel i5 4690k
Memory - 16GB
VM GPU - Radeon R9 290
Host GPU - Intel iGPU
Guest OS - Windows 8.1 64bit

qemu startup script:

#!/bin/bash #Start synergy client systemctl start synergyc@peetipablo #Screen setup xrandr --output DP1 --off xrandr --output HDMI1 --off xrandr --output HDMI3 --primary --auto sleep 1 #start QEMU emulation QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -name win8 -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Haswell,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -bios OVMF.fd -m 8192 -mem-path /dev/hugepages -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 \ -localtime -nographic \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/OVMF.fd \ -drive if=pflash,format=raw,file=/usr/share/ovmf/OVMF_VARS-pure-efi.fd \ -device virtio-scsi-pci,id=scsi \ -device vfio-pci,host=01:00.0,multifunction=on \ -device vfio-pci,host=01:00.1 \ -drive file=/dev/mapper/sdb1_crypt,cache=none,id=hdd,format=raw,if=none -device scsi-hd,drive=hdd \ -drive file=/home/peetipablo/VMs/virtio-win-0.1.96.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \ -device virtio-net,netdev=net0,mac=DE:AD:BE:EF:EB:B7 -netdev tap,ifname=tap0,id=net0 \ -soundhw hda \ -vga none #Restore screen xrandr --output HDMI1 --off xrandr --output DP1 --primary --mode 1920x1080 --rate 144 --left-of HDMI3 xrandr --output HDMI3 --auto --right-of DP1 #Stop synergy client systemctl stop synergyc@peetipablo

oRMFMae.jpg

Last edited by Peetipablo (2015-07-12 21:29:35)

Offline

#5576 2015-07-13 02:12:33

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Mouse response is normal , but the key strokes are delayed by ~5 seconds .

@Duelist :

"I would like to be able to provide access to the PCIe devices to the VMs while simultaneously allowing the host OS to still access the PCIe devices."

Last edited by Denso (2015-07-13 02:13:54)

Offline

#5577 2015-07-13 04:32:10

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

Last edited by morat (2015-07-13 05:29:20)

Offline

#5578 2015-07-13 05:34:54

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@Duelist :

"I would like to be able to provide access to the PCIe devices to the VMs while simultaneously allowing the host OS to still access the PCIe devices."

As far as I know, that isn't possible, and even if it were, it would be unwise; data corruption etc. GPUs, network cards etc. are built to handle multiple outputs, but there's a reason a PCIe card can only fit in one slot.

Offline

#5579 2015-07-13 07:36:19

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I currently have a working setup using Q35 + SeaBIOS, but am trying to change over to using i440fx + OVMF. The VBIOS for the card I'm attempting to pass-through does not appear to have an EFI section according to rom-parser, but the manufacturer has provided an updated version that does.

I've gotten a working VM without the card passed through, but adding the card and specifying the updated VBIOS with a <rom file="..." /> segment appears to just hang after boot -- there's no output on the physical monitor attached to the card, the VM consumes an absolutely constant amount of CPU time, and it never comes up on the network. Remove the card from the VM, and it boots just fine and I can reach it over VNC.

Any idea on how to troubleshoot further? Do I need to try flashing the updated vbios to the card? If so, how do I do this under Linux? I've tried atiflash, but it just returns 'Adapter not found' when I ask it to list current adapters...

Offline

#5580 2015-07-13 07:38:14

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

The UEFI shell didn't recognize your Windows 8.1 ISO because it's in UDF format, not iso9660 -- I had this issue initially myself.

Offline

#5581 2015-07-13 08:15:21

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:

So I currently have a working setup using Q35 + SeaBIOS, but am trying to change over to using i440fx + OVMF. The VBIOS for the card I'm attempting to pass-through does not appear to have an EFI section according to rom-parser, but the manufacturer has provided an updated version that does.

I've gotten a working VM without the card passed through, but adding the card and specifying the updated VBIOS with a <rom file="..." /> segment appears to just hang after boot -- there's no output on the physical monitor attached to the card, the VM consumes an absolutely constant amount of CPU time, and it never comes up on the network. Remove the card from the VM, and it boots just fine and I can reach it over VNC.

Any idea on how to troubleshoot further? Do I need to try flashing the updated vbios to the card? If so, how do I do this under Linux? I've tried atiflash, but it just returns 'Adapter not found' when I ask it to list current adapters...

I've had that problem and solved it.
If you have an ASUS HD7750 Silent - i can give you a working ROM.
You don't need to flash anything, just append as romfile option.


Hmmm.. finally a thing to put in my signature.

Last edited by Duelist (2015-07-13 08:17:54)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5582 2015-07-13 09:11:09

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:
morat wrote:

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

The UEFI shell didn't recognize your Windows 8.1 ISO because it's in UDF format, not iso9660 -- I had this issue initially myself.

I got hopeful for a sec, but supposedly not:

[lain@navi .vm]$ file Windows7.iso Windows7.iso: ISO 9660 CD-ROM filesystem data 'GSP1RMCPRXFREO_EN_DVD' (bootable) [lain@navi .vm]$ file Windows81.iso Windows81.iso: ISO 9660 CD-ROM filesystem data 'IRM_CCSA_X64FRE_EN-US_DV5' (bootable)

Offline

#5583 2015-07-13 09:14:10

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I've had that problem and solved it.
If you have an ASUS HD7750 Silent - i can give you a working ROM.
You don't need to flash anything, just append as romfile option.


Hmmm.. finally a thing to put in my signature.


That was a fairly confusing series of links, but I ended up following the stuff on the insanelymac link you gave and was able to create a version of my current VBIOS with the EFI GOP added, and rom-parser reports it contains a valid EFI ROM signature... I still get the same issues when trying to boot, however sad.

Offline

#5584 2015-07-13 09:28:43

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
Nazfellun wrote:
morat wrote:

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

The UEFI shell didn't recognize your Windows 8.1 ISO because it's in UDF format, not iso9660 -- I had this issue initially myself.

I got hopeful for a sec, but supposedly not:

[lain@navi .vm]$ file Windows7.iso Windows7.iso: ISO 9660 CD-ROM filesystem data 'GSP1RMCPRXFREO_EN_DVD' (bootable) [lain@navi .vm]$ file Windows81.iso Windows81.iso: ISO 9660 CD-ROM filesystem data 'IRM_CCSA_X64FRE_EN-US_DV5' (bootable)

Get the 'disktype' utility and run it on the images, confirm that they have an El Torito boot record entry that points to a FAT file system, relevant section of the output will look something like this if it does:

 El Torito boot record, catalog at 22 Bootable non-emulated image, starts at 2828, preloads 4 KiB Platform 0x00 (x86), System Type 0x00 (Empty) Bootable non-emulated image, starts at 2830, preloads 512 bytes Platform 0xEF (EFI), System Type 0x00 (Empty) Windows / MS-DOS boot loader FAT12 file system (hints score 5 of 5) Volume size 1.390 MiB (1457664 bytes, 2847 clusters of 512 bytes) Volume name "EFISECTOR"

If it doesn't... get another image, I guess?

Offline

#5585 2015-07-13 12:43:50

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

You didn't hide kvm, this needs to appear within the <features> secions:

 <kvm> <hidden state='on'/> </kvm>

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5586 2015-07-13 13:01:36

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:
Duelist wrote:

I've had that problem and solved it.
If you have an ASUS HD7750 Silent - i can give you a working ROM.
You don't need to flash anything, just append as romfile option.


Hmmm.. finally a thing to put in my signature.


That was a fairly confusing series of links, but I ended up following the stuff on the insanelymac link you gave and was able to create a version of my current VBIOS with the EFI GOP added, and rom-parser reports it contains a valid EFI ROM signature... I still get the same issues when trying to boot, however sad.

Scratch that, I built a git copy of ovmf and the VBIOS I built is working using that. Installing drivers now... will report back on whether or not this works out smile. Thanks for your help either way, Duelist.

Offline

#5587 2015-07-13 13:07:16

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
morat wrote:

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

You didn't hide kvm, this needs to appear within the <features> secions:

 <kvm> <hidden state='on'/> </kvm>

Fixed.
Still code 43.

Offline

#5588 2015-07-13 13:50:47

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strangely, although I am getting output on the physical card it isn't listed in device manager. Despite this, the AMD Catalyst installer detects the graphics hardware and tries (unsuccessfully) to install it. It gets up to installing the actual driver, the screen flickers once, then the physical display loses signal, the VM goes offline and ramps up to a constant 100% CPU usage with no disk activity. Happens using both the modified copy of my existing VBIOS and the vendor-provided updated one.

My hardware:
Motherboard: Gigabyte 990FXA-UD3
CPU: FX-8350
Host GPU: Gigabyte GTX 750 Ti OC
Guest GPU: Gigabyte R9 290 (initial reference cooler release)

Offline

#5589 2015-07-13 14:32:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:
morat wrote:

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

You didn't hide kvm, this needs to appear within the <features> secions:

 <kvm> <hidden state='on'/> </kvm>

Fixed.
Still code 43.

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5590 2015-07-13 14:52:17

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
morat wrote:
aw wrote:

You didn't hide kvm, this needs to appear within the <features> secions:

 <kvm> <hidden state='on'/> </kvm>

Fixed.
Still code 43.

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

Edit:
@aw, is any of this likely to be caused by running Win 7 / SeaBIOS? My Win 8 image lacks a FAT boot for UEFI, meaning OVMF was a no go; do I need to source a valid one, or find a way to fix this image?

Last edited by morat (2015-07-13 15:00:20)

Offline

#5591 2015-07-13 14:58:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:
morat wrote:

Fixed.
Still code 43.

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

Last edited by aw (2015-07-13 14:58:31)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5592 2015-07-13 15:13:15

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
morat wrote:
aw wrote:

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

Offline

#5593 2015-07-13 15:23:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:
morat wrote:

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

As always, no data help.  I can't fix your script without seeing it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5594 2015-07-13 15:26:24

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
morat wrote:
aw wrote:

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

As always, no data help.  I can't fix your script without seeing it.

It's your script, instructions followed to the letter.
EDIT:
I've removed the other gpu option, and changed qemu-kvm to qemu-system-x86_64:

 #!/bin/sh exec qemu-system-x86_64 \ `echo "\$@" | \ sed 's|01:00.0|01:00.0,x-vga=on|g'`

Last edited by morat (2015-07-13 15:37:15)

Offline

#5595 2015-07-13 15:37:43

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
aw wrote:
morat wrote:

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

As always, no data help.  I can't fix your script without seeing it.

It's your script, instructions followed to the letter.

Shall we talk about last time you followed the instructions "to the letter" and failed to include kvm state hidden?  If you can't be bothered to show your work, then I can't be bothered to help.  Sorry.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5596 2015-07-13 15:39:28

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Always seconds between. If it isn't too much trouble, could you be bothered to read my work?

Offline

#5597 2015-07-13 15:40:44

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Did you: "Be sure to chmod 755 the file before proceeding."


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5598 2015-07-13 15:42:49

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

# ls -lvm.vga -rwxr-xr-x 1 root root 94 Jul 14 01:35 /usr/libexec/qemu-kvm.vga

Yes, without permissions it returns a permissions error.

Offline

#5599 2015-07-13 15:46:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Make the first like

#!/bin/sh -x

And run it by hand as:

/usr/libexec/qemu-kvm.vga -help

What does it output?


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5600 2015-07-13 15:51:51

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Make the first like

#!/bin/sh -x

And run it by hand as:

/usr/libexec/qemu-kvm.vga -help

What does it output?

Outputs the full qemu-system-x86_64 help, but virsh still fails.

Offline

#5601 2015-07-13 15:53:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Perhaps it's a PATH issue then since you've taken the liberty to not call the qemu binary with the full path.

EDIT: Remove the -x for libvirt usage

Last edited by aw (2015-07-13 15:54:37)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5602 2015-07-13 15:55:28

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Maybe, but probably not. exec /sbin/qemu-system-x86_64 still fails.
EDIT:
Removing the -x did not change anything.

Last edited by morat (2015-07-13 15:56:29)

Offline

#5603 2015-07-13 15:57:57

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Maybe the wrapper script should also be in /sbin, mine lives in /usr/libexec because that's where my qemu-kvm binary lives


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5604 2015-07-13 16:00:35

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Nope. Looks like we're running out of options hmm
EDIT:
Diff-ing the two outputs (qemu-kvm.vga & qemu-system-x86_64) shows that they are identical, so it seems irrational that virsh still throws an error.

Last edited by morat (2015-07-13 16:04:27)

Offline

#5605 2015-07-13 16:04:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Change /bin/sh to /bin/bash?

Use full paths for echo and sed?

What distro are you on?  Maybe apparmor or similar is preventing execution.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5606 2015-07-13 16:07:33

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, it works. Good.
Seems unusual that it should require full paths, but w/e, I'm happy. Onto the next problem, whatever it will be.

EDIT:
I haven't yet built and switched to the patched kernel (which btw, at 4.2-rc1 is missing most of the i915 files targeted by your patch), and that is something that can definitely wait till morning. Good hustle, thanks for the help.

Last edited by morat (2015-07-13 16:09:39)

Offline

#5607 2015-07-13 16:13:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Perhaps you could post your working script and reference the distro to help those that come after you.

I would suggest not using 4.2-rc1, there's a reason rc kernels are rc.  Use a released kernel.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5608 2015-07-13 16:17:56

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64.vga:

#! /bin/bash exec /sbin/qemu-system-x86_64 \ `/bin/echo "\$@" | \ /bin/sed 's|01:00.0|01:00.0,x-vga=on|g'`

Working on Arch 4.0.7-2-ARCH

@aw
I'll just rebuild my current kernel then, seems like the safest route.

Offline

#5609 2015-07-13 16:39:17

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:

Strangely, although I am getting output on the physical card it isn't listed in device manager. Despite this, the AMD Catalyst installer detects the graphics hardware and tries (unsuccessfully) to install it. It gets up to installing the actual driver, the screen flickers once, then the physical display loses signal, the VM goes offline and ramps up to a constant 100% CPU usage with no disk activity. Happens using both the modified copy of my existing VBIOS and the vendor-provided updated one.

My hardware:
Motherboard: Gigabyte 990FXA-UD3
CPU: FX-8350
Host GPU: Gigabyte GTX 750 Ti OC
Guest GPU: Gigabyte R9 290 (initial reference cooler release)

I'll write a guide some day.
What guest OS and what device manager(devmgmt.msc?) you've used? I find it really odd that you can't see the card in there.
Also, which ROM did you bisect and took .efi part from? I remember finding 3 or 4 variants of GOP driver, maybe you could try different ones.


UPD: Catalyst 15.7 installed successfully, without any problems. Crossfire works as before.

Last edited by Duelist (2015-07-13 17:32:41)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5610 2015-07-13 21:11:23

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another status update:

First of all, I use Witcher 3 as my benchmark. It loads roughly four CPUs and goes between being CPU and GPU limited about 50/50 on my (bare) hardware in the open game areas. I figure it makes a good way to check for CPU performance issues. I've had stuttering and nasty sound dropouts and I thought it happened when the game was doing IO. The game now runs from a passed through SATA controller's drive and the stuttering is still the same. Seems I was too quick to place blame.

I moved my setup on top of libvirt and gained a ghastly amount of qemu command line arguments which I don't know about and don't dare touch in case I break it. Why does it do so much extra stuff when the XML is really bare-bones?
Anyway, I pinned my CPUs and confirmed that qemu threads were only appearing on the selected CPUs (hit "1" when running top). I ended up giving Windows three cores and their hyperthreads so 6 vCpus. That did not help much either. Then I tried booting my host with isolcpus=2-7 so host processes only run on the first core and its hyperthread. Seems to work beautifully! I only ran it for a very brief time but the constant stutters were all gone. GPU usage and FPS were also up a little, suggesting it's less CPU bottlenecked now, though that was a very unscientific quick observation.

I had trouble understanding what exactly was involved in doing cpu pinning since so many different things and use cases have been documented. I'll take what worked and spell it out in case someone else needs help with it. This is for a HT quad-core with 8 logical CPUs. Adjust the CPU division to your hardware and needs.

On an existing libvirt setup, put this in the xml file:

<domain> ... <vcpu placement='static'>6</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='7'/> <emulatorpin cpuset='0'/> <iothreadpin iothread='1' cpuset='1'/> </cputune> ... </domain>

What this does is give the guest 6 vCpus and place the guest threads on cores 2-7. (Could be 0-5 instead as well, shouldn't matter. I opted to keep the first two for the host). I'm not sure if the emulatorpin and iothreadpin settings are necessary but they basically move qemu's internal tasks and IO handling to the specified CPUs. Read more: https://libvirt.org/formatdomain.html
This alone won't do much good because the host will still schedule its threads on the same CPUs. Adding the isolcpus=2-7 makes CPUs 2 through 7 unavailable for regular userspace processes. Some kernel tasks and interrupts still run on them but AFAIK it should be far less intrusive. The isolated CPUs are only used explicitly such as with cpuset. Kind of sucks when you want to use the CPUs on your host. I haven't looked up if the isolation could be done dynamically but that sounds like an ideal way.

Edit: Another random observation: using OVMF solves the problem of USB passthrough devices not working until the guest os has booted. At least I think so. Windows 8 has different error recovery menus though so I'm not sure if it loads USB drivers there. Legacy BIOS doesn't play nice with USB apparently. Same thing happens on my hardware BIOS if I disable its legacy USB hack mode.

Last edited by impulse_255 (2015-07-13 21:42:56)

Offline

#5611 2015-07-14 03:08:26

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Nazfellun wrote:

Strangely, although I am getting output on the physical card it isn't listed in device manager. Despite this, the AMD Catalyst installer detects the graphics hardware and tries (unsuccessfully) to install it. It gets up to installing the actual driver, the screen flickers once, then the physical display loses signal, the VM goes offline and ramps up to a constant 100% CPU usage with no disk activity. Happens using both the modified copy of my existing VBIOS and the vendor-provided updated one.

My hardware:
Motherboard: Gigabyte 990FXA-UD3
CPU: FX-8350
Host GPU: Gigabyte GTX 750 Ti OC
Guest GPU: Gigabyte R9 290 (initial reference cooler release)

I'll write a guide some day.
What guest OS and what device manager(devmgmt.msc?) you've used? I find it really odd that you can't see the card in there.
Also, which ROM did you bisect and took .efi part from? I remember finding 3 or 4 variants of GOP driver, maybe you could try different ones.


UPD: Catalyst 15.7 installed successfully, without any problems. Crossfire works as before.

After further messing about, the card is showing up in device manager using the 'Microsoft Basic Display Adapter' driver. It doesn't list any errors, but Catalyst still won't install (tried 14.12 which was what I ran under SeaBIOS, also tried 15.7). Guest OS is Windows 8.1.

I didn't end up bisecting a driver for the EFI section, as the linked UEFIRomExtract repo only appears to have a binary for Macs, and I have no idea how to compile it from source. As such, I just used the GOP provided in the first version of the AMD-UEFI-GOP-MAKER tool, which I think is from a HD6450.

Offline

#5612 2015-07-14 09:31:57

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

UPD: Catalyst 15.7 installed successfully, without any problems. Crossfire works as before.

Yeah I just got the Catalyst 15.7 working on my HD7770 as well. I just needed to boot Win7 without the GPU passthrough and then hot-plug the GPU via Qemu console (device_add vfio-pci....). After that, the installer works.

I also figured out a solution to the xHCI/EHCI USB port mapping problem that someone else noted a few pages ago. The problem was that on some Intel based systems, all the physical USB ports found on the motherboard would get mapped behind the single Intel xHCI controller. Even when the chipset contains EHCI controllers that show up in lspci. The solution that I finally came up with, is to directly access the PCI config space of the xHCI controller and change the Port Mapping Mask register. The register simply has a bit for each physical port, selecting if it should show up behind the EHCI controller or the xHCI controller. So for example, with my ASrock board I ended up using: setpci -s 00:14.0 0xd0.W=2037, where 00:14.0 is the address of the xHCI controller, 0xd0 is the address of the port mapping mask register and 0x2037 is the value that maps the physical USB2.0 ports behind EHCI controllers. Now I'm able to passthrough one of the Intel EHCI controllers to my VM and gain 4 x real USB2.0 ports.

Offline

#5613 2015-07-14 12:50:58

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fld wrote:

I also figured out a solution to the xHCI/EHCI USB port mapping problem that someone else noted a few pages ago. The problem was that on some Intel based systems, all the physical USB ports found on the motherboard would get mapped behind the single Intel xHCI controller. Even when the chipset contains EHCI controllers that show up in lspci. The solution that I finally came up with, is to directly access the PCI config space of the xHCI controller and change the Port Mapping Mask register. The register simply has a bit for each physical port, selecting if it should show up behind the EHCI controller or the xHCI controller. So for example, with my ASrock board I ended up using: setpci -s 00:14.0 0xd0.W=2037, where 00:14.0 is the address of the xHCI controller, 0xd0 is the address of the port mapping mask register and 0x2037 is the value that maps the physical USB2.0 ports behind EHCI controllers. Now I'm able to passthrough one of the Intel EHCI controllers to my VM and gain 4 x real USB2.0 ports.

I am having this exact issue with my X99 platform . Is there a detailed guide or a tutorial that you can redirect me to ?

Offline

#5614 2015-07-14 13:47:03

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:

I didn't end up bisecting a driver for the EFI section, as the linked UEFIRomExtract repo only appears to have a binary for Macs, and I have no idea how to compile it from source. As such, I just used the GOP provided in the first version of the AMD-UEFI-GOP-MAKER tool, which I think is from a HD6450.

The thing is - that tools are from EDK2. I've found EFIRom somewhere there some time ago, but i've lost it. Anyway, what's the device IDs of that basic adapter?
I think something is broken deeply.

fld wrote:

Yeah I just got the Catalyst 15.7 working on my HD7770 as well. I just needed to boot Win7 without the GPU passthrough and then hot-plug the GPU via Qemu console (device_add vfio-pci....). After that, the installer works.

I dunno, i have two HD7750s attached, i've ran the installer, it finished without error and asked for a reboot. After a guest reboot, everything was perfect, except the crossfire turned itself off. With a flick of a switch, all was done.

Last edited by Duelist (2015-07-14 13:49:42)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5615 2015-07-14 17:07:58

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
fld wrote:

I also figured out a solution to the xHCI/EHCI USB port mapping problem that someone else noted a few pages ago. The problem was that on some Intel based systems, all the physical USB ports found on the motherboard would get mapped behind the single Intel xHCI controller. Even when the chipset contains EHCI controllers that show up in lspci. The solution that I finally came up with, is to directly access the PCI config space of the xHCI controller and change the Port Mapping Mask register. The register simply has a bit for each physical port, selecting if it should show up behind the EHCI controller or the xHCI controller. So for example, with my ASrock board I ended up using: setpci -s 00:14.0 0xd0.W=2037, where 00:14.0 is the address of the xHCI controller, 0xd0 is the address of the port mapping mask register and 0x2037 is the value that maps the physical USB2.0 ports behind EHCI controllers. Now I'm able to passthrough one of the Intel EHCI controllers to my VM and gain 4 x real USB2.0 ports.

I am having this exact issue with my X99 platform . Is there a detailed guide or a tutorial that you can redirect me to ?

I'm afraid there are no guides or tutorials for this. I just used the X87 PCH datasheet to figure out the correct register and then experimented with different values, until I had figured out the mapping between the bitmask and all the physical ports.

So you would need to start with the X99 PCH datasheet at around page 650 where it gets into xHCI port routing.

Duelist wrote:
fld wrote:

Yeah I just got the Catalyst 15.7 working on my HD7770 as well. I just needed to boot Win7 without the GPU passthrough and then hot-plug the GPU via Qemu console (device_add vfio-pci....). After that, the installer works.

I dunno, i have two HD7750s attached, i've ran the installer, it finished without error and asked for a reboot. After a guest reboot, everything was perfect, except the crossfire turned itself off. With a flick of a switch, all was done.

I'm running without x-vga, so maybe that's why.

Offline

#5616 2015-07-14 19:31:38

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm afraid there are no guides or tutorials for this. I just used the X87 PCH datasheet to figure out the correct register and then experimented with different values, until I had figured out the mapping between the bitmask and all the physical ports.

So you would need to start with the X99 PCH datasheet at around page 650 where it gets into xHCI port routing.

Thanks ! It seems that I can only seperate it into 2 controllers only . I'm better off using my PCI-E USB3 card then smile

Offline

#5617 2015-07-14 20:05:17

sunmast
Member
Registered: 2014-12-16
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
sunmast wrote:

Is it possible to run OSX with OVMF method?

Yes. I have a proof-picture, if you're curious.

Hi Duelist, would you please share your qemu command line to run OSX with OVMF? Are you using Chameleon or Clover?

Thanks a lot!

Offline

#5618 2015-07-14 20:27:47

Nektarios
Member
Registered: 2013-02-19
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to completely disable a graphics card (nvidia or ati) when not in use? And when I mean completely, I mean close to 0 power usage, no fan usage etc.

I want this so I can freely shut down the vga card that I use for the VM passthrough when the VM is not running.

Offline

#5619 2015-07-14 21:09:28

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nektarios wrote:

Is there a way to completely disable a graphics card (nvidia or ati) when not in use? And when I mean completely, I mean close to 0 power usage, no fan usage etc.

I want this so I can freely shut down the vga card that I use for the VM passthrough when the VM is not running.

Sure, buy an expensive server that supports physical PCI hotplug.  Most systems do not provide power control for PCI slots, nor is there any way to manipulate the hardware to disable slot power.  If you have a hotplug capable expresscard slot (ie. laptop - I haven't found one for desktop yet) and you're willing to take a significant performance penalty (5GT/s x1 PCIe), then something like the EXP GDC external video card slot will work (often with some commandline trickery).  You might also be surprised how little power some of the newer cards draw when idle.  I haven't tried my GTX750 in the external slot yet, but some smaller cards show less than 5W when unassigned.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5620 2015-07-14 22:46:51

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sunmast wrote:
Duelist wrote:
sunmast wrote:

Is it possible to run OSX with OVMF method?

Yes. I have a proof-picture, if you're curious.

Hi Duelist, would you please share your qemu command line to run OSX with OVMF? Are you using Chameleon or Clover?

Thanks a lot!

I can't. I've seen the picture of that. But only the picture.

fld wrote:

I'm running without x-vga, so maybe that's why.

ಠ_ಠ
I am too, since i needed to patch my VBIOS to get GOP... I use OVMF, OVMF doesn't require x-vga.
If you run OVMF too with a factory-flashed GPU ROM - could you send me the file or at least the .efi part?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5621 2015-07-15 02:31:33

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Does vfio-pci ids REQUIRE pci-stub to be loaded ?

I'm asking because I've been having a hard time getting vfio-pci ids to be loaded at boot , and now that I added pci-stub module before vfio-pci in the boot modules chain IT WORKS ! And every device get bound to VFIO directly and correctly .

Please note that pci-stub is not built statically in Arch .

Offline

#5622 2015-07-15 02:33:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

No, vfio-pci has no dependency on pci-stub.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5623 2015-07-15 02:39:53

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Well , that is strange . "options vfio-pci ids=" didn't work on both of my NVIDIAs until I added pci-stub to the top of the boot chain . Anyway it is working like a charm now and I can finally get rid of the vfio-bind script .

Offline

#5624 2015-07-15 08:07:11

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@ Well , that is strange . "options vfio-pci ids=" didn't work on both of my NVIDIAs until I added pci-stub to the top of the boot chain . Anyway it is working like a charm now and I can finally get rid of the vfio-bind script .

@Denso could it be that, when pci-stub is not loaded, noveau driver loaded/probed your card thus preventing vfio-pci from working correctly, and loading pci-stub prevents this from happening (as is its purpose)?


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5625 2015-07-15 09:56:54

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Nazfellun wrote:

I didn't end up bisecting a driver for the EFI section, as the linked UEFIRomExtract repo only appears to have a binary for Macs, and I have no idea how to compile it from source. As such, I just used the GOP provided in the first version of the AMD-UEFI-GOP-MAKER tool, which I think is from a HD6450.

The thing is - that tools are from EDK2. I've found EFIRom somewhere there some time ago, but i've lost it. Anyway, what's the device IDs of that basic adapter?
I think something is broken deeply.


The listed device & vendor ID match up to the ones reported by lspci -nn on the host, and the PCI bus/slot listed for the device matches the libvirt configuration.

Last edited by Nazfellun (2015-07-15 09:57:27)

Offline

#5626 2015-07-15 11:59:10

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, how hard and/or unrealistic would be to run program with second GPU (bound to vfio) on host, when it's not used by any VM? I'm thinking of approach similar to bumblebe. Is it possible at all? I assume it would need to detach GPU from vfio, run app using it (not leaving host X server), attach to vfio again.

I was planning to set up second machine (aside from Windows one) with Linux, but it's giving me hell. With nearly same config as Windows one. Installed everything, it's using dedicated Nvidia 960, but with low res and disabled OpenGL on Nouveau and with "nvidia gpu at pci:0:8:0 is not supported by NVIDIA driver" with Nvidia binary. I'd like to run some GPU-heavy apps on this dedicated GPU.

I was posting my config some time ago, but just as reminder. It's running perfectly with Windows guest.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>wintendo</name> <uuid>bf6bda0e-2664-4bab-998a-aac4dc255c83</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='off'/> <vapic state='off'/> <spinlocks state='off'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/50.raw'/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/d2.img'/> <target dev='vde' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:17:65:a1'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,multifunction=on,x-vga=on'/> </qemu:commandline> </domain>

Offline

#5627 2015-07-15 13:22:54

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
Denso wrote:

@ Well , that is strange . "options vfio-pci ids=" didn't work on both of my NVIDIAs until I added pci-stub to the top of the boot chain . Anyway it is working like a charm now and I can finally get rid of the vfio-bind script .

@Denso could it be that, when pci-stub is not loaded, noveau driver loaded/probed your card thus preventing vfio-pci from working correctly, and loading pci-stub prevents this from happening (as is its purpose)?

That's not possible . I'm not using pci-stub in any shape or form . Not in the boot loader config files or any other place , I just added it to the modules to be loaded automatically at boot .

I am blacklisting nouveau from the begining so it is not the issue , and when I do lspci -k , there is no driver attached to these cards at all .

I believe Alex said that vfio-pci ids have similar code to pci-stub ids . Maybe vfio-pci "thinks" that pci-stub is ALWAYS built statically , which is not the case on Arch .

Offline

#5628 2015-07-15 15:45:59

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I upgraded my video card to nvidia gtx 970 which pushed me to upgrade the drivers -- now getting error code 43.

I tried -cpu host,hv_time,kvm=off but looks like I need to add more hv_ options, just not sure what all needs to be added.

I don't use libvirt so does anyone mind posting your kvm command line -cpu parameters?

Thanks.

Offline

#5629 2015-07-15 15:47:29

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

tuxuser wrote:

I upgraded my video card to nvidia gtx 970 which pushed me to upgrade the drivers -- now getting error code 43.

I tried -cpu host,hv_time,kvm=off but looks like I need to add more hv_ options, just not sure what all needs to be added.

I don't use libvirt so does anyone mind posting your kvm command line -cpu parameters?

Thanks.

Remove hv_ option, not add


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5630 2015-07-15 16:05:47

tuxuser
Member
Registered: 2014-08-23
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
tuxuser wrote:

I upgraded my video card to nvidia gtx 970 which pushed me to upgrade the drivers -- now getting error code 43.

I tried -cpu host,hv_time,kvm=off but looks like I need to add more hv_ options, just not sure what all needs to be added.

I don't use libvirt so does anyone mind posting your kvm command line -cpu parameters?

Thanks.

Remove hv_ option, not add

I meant to type hv_time=off above.   My assumption had been, per recent discussions, that one these options might be defaulting to on when not specified.  I'll recheck this tonight starting with only -cpu host,kvm=off and moving onward

These are the options I know of hv_time hv_relaxed hv_vapic hv_spinlocks (though i don't know of a proper value for hv_spinlocks so i assume this would be disabled by default)

Offline

#5631 2015-07-16 04:18:34

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Windows 10 + OVMF + virtio-blk-pci is a no-go .

It always throws a BSOD in the installation stage (kmode_exception_not_handled) .

I've tried using 3 different builds (10041 , 10074 , 10240) all BSOD at the same spot with the same error .

dmesg throws tons of these :

[Thu Jul 16 07:10:04 2015] kvm_get_msr_common: 5728 callbacks suppressed [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm_get_msr_common: 9839 callbacks suppressed [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm_get_msr_common: 6502 callbacks suppressed [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm_get_msr_common: 36388 callbacks suppressed [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309

QEMU command :

qemu-system-x86_64 -name winbeta -nographic \ -enable-kvm -m 2G -cpu host,kvm=off -smp sockets=1,cores=2,threads=1 \ -vga std -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/SSD/ovmf_winbeta.bin \ -drive file=/SSD/Win_Beta.img,cache=none,if=none,format=raw,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/SSD/WinBeta.iso,id=iso_install,if=none,format=raw \ -device scsi-cd,drive=iso_install \ -cdrom /SSD/virtio.iso \ -netdev bridge,id=br0 -device virtio-net-pci,netdev=br0,id=nic0,mac=B8:A7:2A:6E:05:B2 \ -vnc :1 \ -localtime \ -usbdevice tablet \ -monitor unix:/tmp/vm_winbeta,server,nowait &

Anyone using Windows 10 ?

Last edited by Denso (2015-07-16 04:30:21)

Offline

#5632 2015-07-16 07:34:48

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For windows 10 use
echo 1 > /sys/module/kvm/parameters/ignore_msrs

Offline

#5633 2015-07-16 09:17:48

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

For windows 10 use
echo 1 > /sys/module/kvm/parameters/ignore_msrs

Hi Slis . Unfortunately it is already set to ignore msrs in modprobe.d conf file :

options kvm ignore_msrs=1

I had to make sure it is indeed set to ignore msrs :

root ~ cat /sys/module/kvm/parameters/ignore_msrs Y

Still no go sad

Thanks smile

Last edited by Denso (2015-07-16 09:17:57)

Offline

#5634 2015-07-16 13:10:12

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Windows 10 + OVMF + virtio-blk-pci is a no-go .

It always throws a BSOD in the installation stage (kmode_exception_not_handled) .

I've tried using 3 different builds (10041 , 10074 , 10240) all BSOD at the same spot with the same error .

dmesg throws tons of these :

[Thu Jul 16 07:10:04 2015] kvm_get_msr_common: 5728 callbacks suppressed [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:04 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm_get_msr_common: 9839 callbacks suppressed [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:09 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm_get_msr_common: 6502 callbacks suppressed [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:14 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm_get_msr_common: 36388 callbacks suppressed [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30a [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu0 ignored rdmsr: 0x30b [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309 [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30a [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x30b [Thu Jul 16 07:10:19 2015] kvm [31145]: vcpu1 ignored rdmsr: 0x309

QEMU command :

qemu-system-x86_64 -name winbeta -nographic \ -enable-kvm -m 2G -cpu host,kvm=off -smp sockets=1,cores=2,threads=1 \ -vga std -nodefconfig \ -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \ -drive if=pflash,format=raw,file=/SSD/ovmf_winbeta.bin \ -drive file=/SSD/Win_Beta.img,cache=none,if=none,format=raw,id=drive0,aio=native \ -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \ -device virtio-scsi-pci,id=scsi \ -drive file=/SSD/WinBeta.iso,id=iso_install,if=none,format=raw \ -device scsi-cd,drive=iso_install \ -cdrom /SSD/virtio.iso \ -netdev bridge,id=br0 -device virtio-net-pci,netdev=br0,id=nic0,mac=B8:A7:2A:6E:05:B2 \ -vnc :1 \ -localtime \ -usbdevice tablet \ -monitor unix:/tmp/vm_winbeta,server,nowait &

Anyone using Windows 10 ?

I get the same issue in my Win8.1 install; any time I try to use OVMF, I get a kmode_etc. bsod. Using libvirt and qemu both have this issue.

Offline

#5635 2015-07-16 13:15:40

evujumenuk
Member
Registered: 2013-07-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:

Borderlands folks, I think this is the problem you're trying to describe:

https://dl.dropboxusercontent.com/u/198 … emetal.jpg

https://dl.dropboxusercontent.com/u/198 … ds2-vm.jpg

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

https://dl.dropboxusercontent.com/u/198 … vm-new.jpg

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

So, what is the best current alternative to hv-time? Can't really use that with Nvidia drivers.

/USE PMTIMER?

Offline

#5636 2015-07-16 14:00:44

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dRaiser wrote:

Hi, how hard and/or unrealistic would be to run program with second GPU (bound to vfio) on host, when it's not used by any VM? I'm thinking of approach similar to bumblebe. Is it possible at all? I assume it would need to detach GPU from vfio, run app using it (not leaving host X server), attach to vfio again.

I was planning to set up second machine (aside from Windows one) with Linux, but it's giving me hell. With nearly same config as Windows one. Installed everything, it's using dedicated Nvidia 960, but with low res and disabled OpenGL on Nouveau and with "nvidia gpu at pci:0:8:0 is not supported by NVIDIA driver" with Nvidia binary. I'd like to run some GPU-heavy apps on this dedicated GPU.

I was posting my config some time ago, but just as reminder. It's running perfectly with Windows guest.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>wintendo</name> <uuid>bf6bda0e-2664-4bab-998a-aac4dc255c83</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='off'/> <vapic state='off'/> <spinlocks state='off'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/sbin/qemu-system-x86_64</emulator> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/50.raw'/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/d2/d2.img'/> <target dev='vde' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:17:65:a1'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0016'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,multifunction=on,x-vga=on'/> </qemu:commandline> </domain>

I've tried VM with Arch instead of Linux Mint and it's fine on Nouveau - still no luck with nvidia (latest is 352). Is Nvidia blocking it also for Linux guests? I'm going to try some older versions.

Offline

#5637 2015-07-16 15:27:29

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah, they also block it linux guests...
Can someone with 980 try soft mod (with fake id patch on qemu) to quadro m5000 and try latest geforce driver with hyper-v on?
For me 680 works with latest geforce drivers and hyper-v, as grid k2 with faked id in qemu.
And about enabling NVFBC, without geforce experience, you can use GRID SDK, there is EnableNVFBC.exe that does same thing as switching shadowplay on/off, to enable better encoder for streaming.

Offline

#5638 2015-07-16 16:00:34

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

Windows 10 + OVMF + virtio-blk-pci is a no-go .
...
Anyone using Windows 10 ?

I have been using Windows 10 for many months now, and just last night updated to the latest 10240 Build (RTM Supposidlt). I don't know what virtio-blk-pci is though? Is it like mapping a disk to the VM? I use Raw images.

slis wrote:

Can someone with 980 try soft mod (with fake id patch on qemu) to quadro m5000 and try latest geforce driver with hyper-v on?

Slis, can you please explain or link to where you can change the PCI IDs via Qemu? I have a GTX760 that i'd like to soft-mod via Qemu. I also have a GTX465 which I soft-modded via rom hacks into a Quadro 5k. It worked fine with ESXi but the performance isn't near my GTX760 for gaming... I would love to have my 760 be represented as a Quadro so I can enable the HyperV Flags - assuming it will help performance.

Also, I have been using the latest Windows 10 NVIDIA Drivers provided by Windows Update without issue.

Offline

#5639 2015-07-16 16:26:42

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

I have been using Windows 10 for many months now, and just last night updated to the latest 10240 Build (RTM Supposidlt). I don't know what virtio-blk-pci is though? Is it like mapping a disk to the VM? I use Raw images.

virtio-blk-pci is a paravirtualized disk controller , and you attach your RAW image to it . its supposed to give better performance than other controllers in QEMU .

What alternatives should I use that would give good performance ?

Last edited by Denso (2015-07-16 16:28:30)

Offline

#5640 2015-07-16 16:57:07

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
Denso wrote:

Windows 10 + OVMF + virtio-blk-pci is a no-go .
...
Anyone using Windows 10 ?

I have been using Windows 10 for many months now, and just last night updated to the latest 10240 Build (RTM Supposidlt). I don't know what virtio-blk-pci is though? Is it like mapping a disk to the VM? I use Raw images.

slis wrote:

Can someone with 980 try soft mod (with fake id patch on qemu) to quadro m5000 and try latest geforce driver with hyper-v on?

Slis, can you please explain or link to where you can change the PCI IDs via Qemu? I have a GTX760 that i'd like to soft-mod via Qemu. I also have a GTX465 which I soft-modded via rom hacks into a Quadro 5k. It worked fine with ESXi but the performance isn't near my GTX760 for gaming... I would love to have my 760 be represented as a Quadro so I can enable the HyperV Flags - assuming it will help performance.

Also, I have been using the latest Windows 10 NVIDIA Drivers provided by Windows Update without issue.

Patch is there...
https://bbs.archlinux.org/viewtopic.php … 0#p1475170

Offline

#5641 2015-07-16 18:21:53

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@The_Moves :

Can you share your Windows 10 QEMU command line ?

Many thanks !

Offline

#5642 2015-07-16 18:27:55

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:
The_Moves wrote:

I have been using Windows 10 for many months now, and just last night updated to the latest 10240 Build (RTM Supposidlt). I don't know what virtio-blk-pci is though? Is it like mapping a disk to the VM? I use Raw images.

virtio-blk-pci is a paravirtualized disk controller , and you attach your RAW image to it . its supposed to give better performance than other controllers in QEMU .

What alternatives should I use that would give good performance ?

I use virtio-blk-scsi. It gave me best results in atto disk benchmark...


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5643 2015-07-16 18:38:12

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist :

I'm not entirely sure that it is the virtio-blk-pci controller that is the culprit . But I'm sure that these happen when that BSOD occur :

[Thu Jul 16 20:28:22 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:37 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:37 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:37 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:14 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:19 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:19 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:19 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:33 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:38 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639

Unfortuanetly , setting "options kvm ignore_msrs=1" didn't solve anything .

Offline

#5644 2015-07-16 19:10:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@Duelist :

I'm not entirely sure that it is the virtio-blk-pci controller that is the culprit . But I'm sure that these happen when that BSOD occur :

[Thu Jul 16 20:28:22 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:22 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:27 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:32 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:37 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:37 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:37 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:43 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:48 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:53 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:28:58 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:03 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:08 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:13 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:14 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:18 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:19 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:19 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:19 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:23 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:24 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:28 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:29 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:33 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:33 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:34 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:38 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:38 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:39 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:44 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:49 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:54 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm_get_msr_common: 92 callbacks suppressed [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:29:59 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm_get_msr_common: 91 callbacks suppressed [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639 [Thu Jul 16 20:30:04 2015] kvm [674]: vcpu0 ignored rdmsr: 0x639

Unfortuanetly , setting "options kvm ignore_msrs=1" didn't solve anything .

All it says is win10 asked CPU's MSR number 0x639, and the CPU didn't respond.
Try determining what is this MSR and why windows would want to read it. There's no one-line fix, i suppose.


BTW:
Catalyst 15.7 broke on the second boot of the VM. It blanks the screen after greeting the user, and it refuses to delete itself - the installer is just doing nothing.
I don't know what caused this, system updates(i don't recall any important) or the catalyst itself. Beware.
UPD:
What the hell... Alright, my catalyst broke, i've booted into safe mode and turned off my GPUs in device manager, booted with QXL and uninstalled catalyst via windows control panel...
I'm 100% sure it said "remove all versions"...
VM reboot later, the GPU fires up, device manager shows HD7750(BOTH!), aero works...
3D WORKS! Crossfire is offline, though. There are some amd driver processes in taskmanager left.
I've UNINSTALLED THE DRIVERS.
What did just happen?
protip: uninstall 14.12 before installing 15.7
protip2: do NOT install 15.7. It breaks my VM, making even the host system crash(it just shuts itself down, like in the dark old ages).

Last edited by Duelist (2015-07-16 20:29:41)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5645 2015-07-16 19:32:49

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@The_Moves :

Can you share your Windows 10 QEMU command line ?

Many thanks !

I'm actually using libvirt/virsh to manage my VMs, here is my XML:

http://pastebin.com/3kKMEq0F

Offline

#5646 2015-07-16 20:04:11

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist & @The_Moves :

Thank you both , I guess I'll wait for Windows 10 to be GA and see what would happen then smile

Update :

Tried Seabios , it works like butter with Windows 10 . It is OVMF-only issue .

Last edited by Denso (2015-07-16 20:48:27)

Offline

#5647 2015-07-16 22:53:56

MonopolyMan
Member
Registered: 2015-07-05
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So all of a sudden my VM stopped working (same script and xml has worked before). I'm getting "vfio: error: device 0000:01:00.0 is already attached" when I try to start the VM.


The script I use:

#!/bin/bash xrandr --output VGA1 --off set -x modprobe vfio-pci if [ -e /sys/bus/pci/devices/0000:01:00.0/driver ]; then echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind fi echo 1002 6810 > /sys/bus/pci/drivers/vfio-pci/new_id if [ -e /sys/bus/pci/devices/0000:01:00.1/driver ]; then echo 0000:01:00.1 > /sys/bus/pci/devices/0000:01:00.1/driver/unbind fi echo 1002 AAB0 > /sys/bus/pci/drivers/vfio-pci/new_id killall synergy killall synergyc sudo synergyc 192.168.1.14:24800 virsh start gamingvm set +x sleep 5 xrandr --output HDMI3 --off xrandr --output HDMI3 --mode 1600x900 exit 0

The error I get when trying to run:

error: Failed to start domain gamingvm error: internal error: early end of file from monitor: possible problem: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error: device 0000:01:00.0 is already attached qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Offline

#5648 2015-07-16 23:01:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MonopolyMan wrote:

So all of a sudden my VM stopped working (same script and xml has worked before). I'm getting "vfio: error: device 0000:01:00.0 is already attached" when I try to start the VM.

Have you tried turning it off and on again?  The libvirt log (/var/log/libvirt/qemu/gamingvm.log) might provide some clues.  Are you sure the XML doesn't list the device twice, maybe once via <hostdev> and again via <qemu:args>?  That's as far as I'll guess without logs or xml.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5649 2015-07-16 23:19:41

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Denso wrote:

@Duelist & @The_Moves :

Thank you both , I guess I'll wait for Windows 10 to be GA and see what would happen then smile

Update :

Tried Seabios , it works like butter with Windows 10 . It is OVMF-only issue .

I have the same issue with almost the same configuration. I tried uninstalling the nvidia driver from Windows 10, and now I get no issues with code error 43 and I have a full 1080P screen resolution.
I am now running stock Windows 10 nvidia driver on build 9926. (not Nvidia driver, but the Windows driver).

Last edited by trimm (2015-07-16 23:21:16)

Offline

#5650 2015-07-16 23:27:38

MonopolyMan
Member
Registered: 2015-07-05
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
MonopolyMan wrote:

So all of a sudden my VM stopped working (same script and xml has worked before). I'm getting "vfio: error: device 0000:01:00.0 is already attached" when I try to start the VM.

Have you tried turning it off and on again?  The libvirt log (/var/log/libvirt/qemu/gamingvm.log) might provide some clues.  Are you sure the XML doesn't list the device twice, maybe once via <hostdev> and again via <qemu:args>?  That's as far as I'll guess without logs or xml.

The logs didn't provide any more information and I triple checked the XML before posting. Oddly enough, i just redefined the VM and it worked again. I didn't even change anything...

Regardless, thanks for your help.

Offline

#5651 2015-07-16 23:45:07

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems like OVMF is storming today:
I am unable not only to run windows 7 with AMD, but i am trying to just install it without any attached devices - and it crashes and crashes and crashes and hangs with 4 cores 100% usage.

...i should plan a new build, this one is starting to drive me nuts.

Last edited by Duelist (2015-07-16 23:45:51)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5652 2015-07-17 04:21:27

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:
The_Moves wrote:
Denso wrote:

Windows 10 + OVMF + virtio-blk-pci is a no-go .
...
Anyone using Windows 10 ?

I have been using Windows 10 for many months now, and just last night updated to the latest 10240 Build (RTM Supposidlt). I don't know what virtio-blk-pci is though? Is it like mapping a disk to the VM? I use Raw images.

slis wrote:

Can someone with 980 try soft mod (with fake id patch on qemu) to quadro m5000 and try latest geforce driver with hyper-v on?

Slis, can you please explain or link to where you can change the PCI IDs via Qemu? I have a GTX760 that i'd like to soft-mod via Qemu. I also have a GTX465 which I soft-modded via rom hacks into a Quadro 5k. It worked fine with ESXi but the performance isn't near my GTX760 for gaming... I would love to have my 760 be represented as a Quadro so I can enable the HyperV Flags - assuming it will help performance.

Also, I have been using the latest Windows 10 NVIDIA Drivers provided by Windows Update without issue.

Patch is there...
https://bbs.archlinux.org/viewtopic.php … 0#p1475170

Is this patch still upstream? What versions of QEMU does it work with? I currently have qemu-2.3.0-5.fc22.x86_64 installed

Last edited by The_Moves (2015-07-17 04:21:36)

Offline

#5653 2015-07-17 06:29:02

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Can someone with 980 try soft mod (with fake id patch on qemu) to quadro m5000 and try latest geforce driver with hyper-v on?

I've got a 980, can you tell me what I need to do to try the soft-mod?

Offline

#5654 2015-07-17 07:00:11

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That patch is aw's improvisation, i don't think it will be upstream ever....

Well you can patch qemu with this patch https://bbs.archlinux.org/viewtopic.php … 0#p1475170

and change card id-s to quadro m5000, I guess its  10de:13f0, then try latest geforce driver with hyper-v on.

Last edited by slis (2015-07-17 07:01:14)

Offline

#5655 2015-07-17 07:15:33

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Don't you also need a rom file to pass along with the spoofed pciid? Unfortunately Maxwell Quadro rom dumps don't seem to be available anywhere. They are not in Techpowerup's VGA bios database either. I'm very interested in trying to masquerade my Titan X as M6000.

Regarding OVMF problems, I think it's something to do with the latest versions. I got a BSOD in Windows 8 even with any configuration (no passthrough, just virtualised everything). An old build, in my case the sata patched binary, worked fine.

Offline

#5656 2015-07-17 07:17:55

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah, forgot about bios ... it should show up soon... you can try with no rom or with default one, it may work....

Offline

#5657 2015-07-17 07:41:31

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Yeah, forgot about bios ... it should show up soon... you can try with no rom or with default one, it may work....

It won't, telling the VM that the hardware ID is different from what the GPU claims will only cause more problems.
@impulse_255
Do you know the last version of OVMF that worked for you? I think it's probably the best shot I have at the moment, besides surgery on the GPU.

Offline

#5658 2015-07-17 07:45:44

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well I just tried my 680 as grid k2 without grid k2 romfile, and it works....

Offline

#5659 2015-07-17 07:54:50

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:
slis wrote:

Yeah, forgot about bios ... it should show up soon... you can try with no rom or with default one, it may work....

It won't, telling the VM that the hardware ID is different from what the GPU claims will only cause more problems.
@impulse_255
Do you know the last version of OVMF that worked for you? I think it's probably the best shot I have at the moment, besides surgery on the GPU.

I'm using the one linked from this post: https://bbs.archlinux.org/viewtopic.php … 8#p1518188

I have not tried any other versions yet. I used SeaBIOS before and switched to OVMF a week ago. Sadly, the sata boot feature isn't working, probably because of the crappy Marvell controller.

Offline

#5660 2015-07-17 11:25:06

dRaiser
Member
From: Poland
Registered: 2013-05-20
Posts: 49
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've discovered that after launching (and powering off) Linux guest, I cannot later start Windows guest, need to restart host. Well, so long with Linux guest at this moment then.

Edit: It was different issue: machine was hanging when powering because it had problem to take integrated sound device. Pulseaudio kill is helpful here.

Last edited by dRaiser (2015-07-18 03:40:44)

Offline

#5661 2015-07-17 12:56:22

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Quadro cards operate as a secondary display, the GPU ROM isn't used.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5662 2015-07-17 14:40:47

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I recently started using a M-Audio 1010lt PCI soundcard (via a ASMedia ASM1083 PCIe to PCI bridge) for my host audio. But ever since, I've been getting this strange DMAR flood in my dmesg whenever the sound card is being used:

[64721.567981] dmar: DRHD: handling fault status reg 3 [64721.567998] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [64721.568080] dmar: DRHD: handling fault status reg 3 [64721.568082] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [64721.716747] dmar: DRHD: handling fault status reg 3 [64721.716751] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set ...

As I was originally only using just "intel_iommu=on,igfx_off", I thought adding the "iommu=pt" would help to get rid of this problem, because it is supposed to disable the DMAR. However, it would appear that adding the "iommu=pt" changed nothing.

I haven't noticed any other ill effects than my dmesg being flooded. I was hoping perhaps aw knows something about this?

The interesting bits from dmesg:

[ 0.000000] Linux version 4.0.0-2-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.3 (Debian 4.9.3-1) ) #1 SMP Debian 4.0.8-1 (2015-07-11) [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.0.0-2-amd64 root=/dev/mapper/main-lvroot ro consoleblank=0 iommu=pt intel_iommu=on,igfx_off [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable [ 0.000000] BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000059000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006ea58fff] usable [ 0.000000] BIOS-e820: [mem 0x000000006ea59000-0x000000006ea5ffff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000006ea60000-0x000000006f672fff] usable [ 0.000000] BIOS-e820: [mem 0x000000006f673000-0x000000006fb7cfff] reserved [ 0.000000] BIOS-e820: [mem 0x000000006fb7d000-0x000000008be94fff] usable [ 0.000000] BIOS-e820: [mem 0x000000008be95000-0x000000008c09afff] reserved [ 0.000000] BIOS-e820: [mem 0x000000008c09b000-0x000000008c0dbfff] usable [ 0.000000] BIOS-e820: [mem 0x000000008c0dc000-0x000000008c187fff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000008c188000-0x000000008cf84fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000008cf85000-0x000000008cffefff] type 20 [ 0.000000] BIOS-e820: [mem 0x000000008cfff000-0x000000008cffffff] usable [ 0.000000] BIOS-e820: [mem 0x000000008f800000-0x00000000cf9fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000042f5fffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] efi: EFI v2.31 by American Megatrends [ 0.000000] efi: ACPI=0x8c164000 ACPI 2.0=0x8c164000 SMBIOS=0x8cf83498 [ 0.000000] efi: mem00: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000000000000-0x0000000000058000) (0MB) [ 0.000000] efi: mem01: [Reserved | | | | | |WB|WT|WC|UC] range=[0x0000000000058000-0x0000000000059000) (0MB) [ 0.000000] efi: mem02: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000000059000-0x000000000009e000) (0MB) [ 0.000000] efi: mem03: [Reserved | | | | | |WB|WT|WC|UC] range=[0x000000000009e000-0x00000000000a0000) (0MB) [ 0.000000] efi: mem04: [Loader Data | | | | | |WB|WT|WC|UC] range=[0x0000000000100000-0x0000000000f6f000) (14MB) [ 0.000000] efi: mem05: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000000f6f000-0x00000000359c4000) (842MB) [ 0.000000] efi: mem06: [Loader Data | | | | | |WB|WT|WC|UC] range=[0x00000000359c4000-0x0000000036cda000) (19MB) [ 0.000000] efi: mem07: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000036cda000-0x0000000049afd000) (302MB) [ 0.000000] efi: mem08: [Loader Data | | | | | |WB|WT|WC|UC] range=[0x0000000049afd000-0x000000006bc9c000) (545MB) [ 0.000000] efi: mem09: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x000000006bc9c000-0x000000006ea3b000) (45MB) [ 0.000000] efi: mem10: [Loader Code | | | | | |WB|WT|WC|UC] range=[0x000000006ea3b000-0x000000006ea59000) (0MB) [ 0.000000] efi: mem11: [ACPI Memory NVS | | | | | |WB|WT|WC|UC] range=[0x000000006ea59000-0x000000006ea60000) (0MB) [ 0.000000] efi: mem12: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x000000006ea60000-0x000000006ebb5000) (1MB) [ 0.000000] efi: mem13: [Boot Code | | | | | |WB|WT|WC|UC] range=[0x000000006ebb5000-0x000000006f648000) (10MB) [ 0.000000] efi: mem14: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x000000006f648000-0x000000006f658000) (0MB) [ 0.000000] efi: mem15: [Boot Code | | | | | |WB|WT|WC|UC] range=[0x000000006f658000-0x000000006f66a000) (0MB) [ 0.000000] efi: mem16: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x000000006f66a000-0x000000006f673000) (0MB) [ 0.000000] efi: mem17: [Runtime Data |RUN| | | | |WB|WT|WC|UC] range=[0x000000006f673000-0x000000006fb7d000) (5MB) [ 0.000000] efi: mem18: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x000000006fb7d000-0x000000006fb8c000) (0MB) [ 0.000000] efi: mem19: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x000000006fb8c000-0x00000000885b4000) (394MB) [ 0.000000] efi: mem20: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x00000000885b4000-0x000000008894c000) (3MB) [ 0.000000] efi: mem21: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x000000008894c000-0x0000000088a41000) (0MB) [ 0.000000] efi: mem22: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088a41000-0x0000000088a62000) (0MB) [ 0.000000] efi: mem23: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088a62000-0x0000000088a65000) (0MB) [ 0.000000] efi: mem24: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088a65000-0x0000000088a6d000) (0MB) [ 0.000000] efi: mem25: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088a6d000-0x0000000088a7a000) (0MB) [ 0.000000] efi: mem26: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088a7a000-0x0000000088a8e000) (0MB) [ 0.000000] efi: mem27: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088a8e000-0x0000000088a92000) (0MB) [ 0.000000] efi: mem28: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088a92000-0x0000000088a99000) (0MB) [ 0.000000] efi: mem29: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088a99000-0x0000000088a9a000) (0MB) [ 0.000000] efi: mem30: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088a9a000-0x0000000088c2a000) (1MB) [ 0.000000] efi: mem31: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088c2a000-0x0000000088c2f000) (0MB) [ 0.000000] efi: mem32: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088c2f000-0x0000000088dcf000) (1MB) [ 0.000000] efi: mem33: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088dcf000-0x0000000088dd1000) (0MB) [ 0.000000] efi: mem34: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088dd1000-0x0000000088dd8000) (0MB) [ 0.000000] efi: mem35: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088dd8000-0x0000000088ddb000) (0MB) [ 0.000000] efi: mem36: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088ddb000-0x0000000088de5000) (0MB) [ 0.000000] efi: mem37: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088de5000-0x0000000088de8000) (0MB) [ 0.000000] efi: mem38: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088de8000-0x0000000088dfb000) (0MB) [ 0.000000] efi: mem39: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000088dfb000-0x0000000088dfc000) (0MB) [ 0.000000] efi: mem40: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x0000000088dfc000-0x000000008acdb000) (30MB) [ 0.000000] efi: mem41: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x000000008acdb000-0x000000008bba0000) (14MB) [ 0.000000] efi: mem42: [Boot Code | | | | | |WB|WT|WC|UC] range=[0x000000008bba0000-0x000000008be95000) (2MB) [ 0.000000] efi: mem43: [Reserved | | | | | |WB|WT|WC|UC] range=[0x000000008be95000-0x000000008bf0d000) (0MB) [ 0.000000] efi: mem44: [Reserved | | | | | |WB|WT|WC|UC] range=[0x000000008bf0d000-0x000000008c09b000) (1MB) [ 0.000000] efi: mem45: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x000000008c09b000-0x000000008c0d2000) (0MB) [ 0.000000] efi: mem46: [Loader Data | | | | | |WB|WT|WC|UC] range=[0x000000008c0d2000-0x000000008c0dc000) (0MB) [ 0.000000] efi: mem47: [ACPI Memory NVS | | | | | |WB|WT|WC|UC] range=[0x000000008c0dc000-0x000000008c178000) (0MB) [ 0.000000] efi: mem48: [ACPI Memory NVS | | | | | |WB|WT|WC|UC] range=[0x000000008c178000-0x000000008c17a000) (0MB) [ 0.000000] efi: mem49: [ACPI Memory NVS | | | | | |WB|WT|WC|UC] range=[0x000000008c17a000-0x000000008c188000) (0MB) [ 0.000000] efi: mem50: [Runtime Data |RUN| | | | |WB|WT|WC|UC] range=[0x000000008c188000-0x000000008ce7e000) (12MB) [ 0.000000] efi: mem51: [Runtime Data |RUN| | | | |WB|WT|WC|UC] range=[0x000000008ce7e000-0x000000008cee6000) (0MB) [ 0.000000] efi: mem52: [Runtime Data |RUN| | | | |WB|WT|WC|UC] range=[0x000000008cee6000-0x000000008cee8000) (0MB) [ 0.000000] efi: mem53: [Runtime Data |RUN| | | | |WB|WT|WC|UC] range=[0x000000008cee8000-0x000000008cf85000) (0MB) [ 0.000000] efi: mem54: [Runtime Code |RUN| | | | |WB|WT|WC|UC] range=[0x000000008cf85000-0x000000008cf9d000) (0MB) [ 0.000000] efi: mem55: [Runtime Code |RUN| | | | |WB|WT|WC|UC] range=[0x000000008cf9d000-0x000000008cfff000) (0MB) [ 0.000000] efi: mem56: [Boot Data | | | | | |WB|WT|WC|UC] range=[0x000000008cfff000-0x000000008d000000) (0MB) [ 0.000000] efi: mem57: [Conventional Memory| | | | | |WB|WT|WC|UC] range=[0x0000000100000000-0x000000042f600000) (13046MB) [ 0.000000] efi: mem58: [Reserved | | | | | | | | | ] range=[0x000000008f800000-0x00000000cfa00000) (1026MB) [ 0.000000] efi: mem59: [Memory Mapped I/O |RUN| | | | | | | |UC] range=[0x00000000f8000000-0x00000000fc000000) (64MB) [ 0.000000] efi: mem60: [Memory Mapped I/O |RUN| | | | | | | |UC] range=[0x00000000fec00000-0x00000000fec01000) (0MB) [ 0.000000] efi: mem61: [Memory Mapped I/O |RUN| | | | | | | |UC] range=[0x00000000fed00000-0x00000000fed04000) (0MB) [ 0.000000] efi: mem62: [Memory Mapped I/O |RUN| | | | | | | |UC] range=[0x00000000fed1c000-0x00000000fed20000) (0MB) [ 0.000000] efi: mem63: [Memory Mapped I/O |RUN| | | | | | | |UC] range=[0x00000000fee00000-0x00000000fee01000) (0MB) [ 0.000000] efi: mem64: [Memory Mapped I/O |RUN| | | | | | | |UC] range=[0x00000000ff000000-0x0000000100000000) (16MB) [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x000000008C164000 000024 (v02 ALASKA) [ 0.000000] ACPI: XSDT 0x000000008C164088 00008C (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FACP 0x000000008C16EC08 00010C (v05 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: DSDT 0x000000008C1641A8 00AA59 (v02 ALASKA A M I 00000240 INTL 20091112) [ 0.000000] ACPI: FACS 0x000000008C186080 000040 [ 0.000000] ACPI: APIC 0x000000008C16ED18 000092 (v03 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: FPDT 0x000000008C16EDB0 000044 (v01 ALASKA A M I 01072009 AMI 00010013) [ 0.000000] ACPI: ASF! 0x000000008C16EDF8 0000A5 (v32 INTEL HCG 00000001 TFSM 000F4240) [ 0.000000] ACPI: SSDT 0x000000008C16EEA0 000539 (v01 PmRef Cpu0Ist 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 0x000000008C16F3E0 000AD8 (v01 PmRef CpuPm 00003000 INTL 20051117) [ 0.000000] ACPI: SSDT 0x000000008C16FEB8 0001C7 (v01 PmRef LakeTiny 00003000 INTL 20051117) [ 0.000000] ACPI: MCFG 0x000000008C170080 00003C (v01 ALASKA A M I 01072009 MSFT 00000097) [ 0.000000] ACPI: HPET 0x000000008C1700C0 000038 (v01 ALASKA A M I 01072009 AMI. 00000005) [ 0.000000] ACPI: SSDT 0x000000008C1700F8 00036D (v01 SataRe SataTabl 00001000 INTL 20091112) [ 0.000000] ACPI: SSDT 0x000000008C170468 003493 (v01 SaSsdt SaSsdt 00003000 INTL 20091112) [ 0.000000] ACPI: AAFT 0x000000008C173900 000235 (v01 ALASKA OEMAAFT 01072009 MSFT 00000097) [ 0.000000] ACPI: DMAR 0x000000008C173B38 0000B8 (v01 INTEL HSW 00000001 INTL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.020185] dmar: Host address width 39 [ 0.020188] dmar: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.020195] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a [ 0.020198] dmar: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.020202] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da [ 0.020204] dmar: RMRR base: 0x0000008c032000 end: 0x0000008c03efff [ 0.020207] dmar: RMRR base: 0x000000cf800000 end: 0x000000cf9fffff [ 0.020373] dmar: DRHD: handling fault status reg 3 [ 0.020376] dmar: DMAR:[DMA Read] Request device [00:02.0] fault addr 8ff50000 DMAR:[fault reason 06] PTE Read access is not set [ 0.020381] Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.020389] Enabled IRQ remapping in x2apic mode [ 0.020390] x2apic enabled [ 0.020395] Switched APIC routing to cluster x2apic. [ 0.324675] pci 0000:07:00.0: [1b21:1080] type 01 class 0x060401 [ 0.325104] pci 0000:07:00.0: System wakeup disabled by ACPI [ 0.325200] pci 0000:08:01.0: [1412:1712] type 00 class 0x040100 [ 0.325225] pci 0000:08:01.0: reg 0x10: [io 0xa040-0xa05f] [ 0.325237] pci 0000:08:01.0: reg 0x14: [io 0xa070-0xa07f] [ 0.325248] pci 0000:08:01.0: reg 0x18: [io 0xa060-0xa06f] [ 0.325259] pci 0000:08:01.0: reg 0x1c: [io 0xa000-0xa03f] [ 0.325344] pci 0000:08:01.0: supports D2 [ 0.325433] pci 0000:07:00.0: PCI bridge to [bus 08] (subtractive decode) [ 0.325441] pci 0000:07:00.0: bridge window [io 0xa000-0xafff] [ 0.325452] pci 0000:07:00.0: bridge window [io 0xa000-0xafff] (subtractive decode) [ 0.325452] pci 0000:07:00.0: bridge window [io 0x0000-0x0cf7 window] (subtractive decode) [ 0.325453] pci 0000:07:00.0: bridge window [io 0x0d00-0xffff window] (subtractive decode) [ 0.325454] pci 0000:07:00.0: bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode) [ 0.325455] pci 0000:07:00.0: bridge window [mem 0x000c0000-0x000c3fff window] (subtractive decode) [ 0.325456] pci 0000:07:00.0: bridge window [mem 0x000c4000-0x000c7fff window] (subtractive decode) [ 0.325457] pci 0000:07:00.0: bridge window [mem 0x000c8000-0x000cbfff window] (subtractive decode) [ 0.325458] pci 0000:07:00.0: bridge window [mem 0x000cc000-0x000cffff window] (subtractive decode) [ 0.325459] pci 0000:07:00.0: bridge window [mem 0x000d0000-0x000d3fff window] (subtractive decode) [ 0.325459] pci 0000:07:00.0: bridge window [mem 0x000d4000-0x000d7fff window] (subtractive decode) [ 0.325460] pci 0000:07:00.0: bridge window [mem 0x000d8000-0x000dbfff window] (subtractive decode) [ 0.325461] pci 0000:07:00.0: bridge window [mem 0x000dc000-0x000dffff window] (subtractive decode) [ 0.325462] pci 0000:07:00.0: bridge window [mem 0x000e0000-0x000e3fff window] (subtractive decode) [ 0.325463] pci 0000:07:00.0: bridge window [mem 0x000e4000-0x000e7fff window] (subtractive decode) [ 0.325464] pci 0000:07:00.0: bridge window [mem 0x000e8000-0x000ebfff window] (subtractive decode) [ 0.325465] pci 0000:07:00.0: bridge window [mem 0x000ec000-0x000effff window] (subtractive decode) [ 0.325466] pci 0000:07:00.0: bridge window [mem 0xcfa00000-0xfeafffff window] (subtractive decode) [ 0.597344] DMAR: No ATSR found [ 0.597406] IOMMU: dmar1 using Queued invalidation [ 0.597421] IOMMU: hardware identity mapping for device 0000:00:00.0 [ 0.597423] IOMMU: hardware identity mapping for device 0000:00:01.0 [ 0.597426] IOMMU: hardware identity mapping for device 0000:00:03.0 [ 0.597428] IOMMU: hardware identity mapping for device 0000:00:14.0 [ 0.597430] IOMMU: hardware identity mapping for device 0000:00:16.0 [ 0.597432] IOMMU: hardware identity mapping for device 0000:00:19.0 [ 0.597434] IOMMU: hardware identity mapping for device 0000:00:1a.0 [ 0.597436] IOMMU: hardware identity mapping for device 0000:00:1b.0 [ 0.597438] IOMMU: hardware identity mapping for device 0000:00:1c.0 [ 0.597440] IOMMU: hardware identity mapping for device 0000:00:1c.2 [ 0.597442] IOMMU: hardware identity mapping for device 0000:00:1c.3 [ 0.597444] IOMMU: hardware identity mapping for device 0000:00:1c.4 [ 0.597446] IOMMU: hardware identity mapping for device 0000:00:1c.5 [ 0.597448] IOMMU: hardware identity mapping for device 0000:00:1c.6 [ 0.597450] IOMMU: hardware identity mapping for device 0000:00:1d.0 [ 0.597452] IOMMU: hardware identity mapping for device 0000:00:1f.0 [ 0.597455] IOMMU: hardware identity mapping for device 0000:00:1f.2 [ 0.597457] IOMMU: hardware identity mapping for device 0000:00:1f.3 [ 0.597462] IOMMU: hardware identity mapping for device 0000:01:00.0 [ 0.597464] IOMMU: hardware identity mapping for device 0000:01:00.1 [ 0.597469] IOMMU: hardware identity mapping for device 0000:03:00.0 [ 0.597475] IOMMU: hardware identity mapping for device 0000:04:00.0 [ 0.597480] IOMMU: hardware identity mapping for device 0000:05:00.0 [ 0.597485] IOMMU: hardware identity mapping for device 0000:06:00.0 [ 0.597487] IOMMU: Setting RMRR: [ 0.597489] Ignoring identity map for HW passthrough device 0000:00:14.0 [0x8c032000 - 0x8c03efff] [ 0.597492] Ignoring identity map for HW passthrough device 0000:00:1a.0 [0x8c032000 - 0x8c03efff] [ 0.597495] Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x8c032000 - 0x8c03efff] [ 0.597498] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.597499] Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.597505] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O [ 34.346230] dmar: DRHD: handling fault status reg 3 [ 34.346244] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.494973] dmar: DRHD: handling fault status reg 3 [ 34.494976] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.643650] dmar: DRHD: handling fault status reg 3 [ 34.643653] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.643738] dmar: DRHD: handling fault status reg 3 [ 34.643740] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.792415] dmar: DRHD: handling fault status reg 3 [ 34.792423] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.792532] dmar: DRHD: handling fault status reg 3 [ 34.792534] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.941181] dmar: DRHD: handling fault status reg 3 [ 34.941185] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 34.941269] dmar: DRHD: handling fault status reg 3 [ 34.941272] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [ 35.089969] dmar: DRHD: handling fault status reg 3 [ 35.089983] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 ...

lsgroups.sh

### Group 0 ### 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) ### Group 1 ### 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] ### Group 2 ### 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) ### Group 3 ### 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) ### Group 4 ### 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) ### Group 5 ### 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) ### Group 6 ### 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) ### Group 7 ### 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) ### Group 8 ### 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) ### Group 9 ### 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) ### Group 10 ### 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) ### Group 11 ### 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) ### Group 12 ### 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5) ### Group 13 ### 00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5) 07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) 08:01.0 Multimedia audio controller: VIA Technologies Inc. ICE1712 [Envy24] PCI Multi-Channel I/O Controller (rev 02) ### Group 14 ### 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) ### Group 15 ### 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) ### Group 16 ### 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 17 ### 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) ### Group 18 ### 05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) ### Group 19 ### 06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

lspci -vv:

07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=07, secondary=08, subordinate=08, sec-latency=32 I/O behind bridge: 0000a000-0000afff Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard 08:01.0 Multimedia audio controller: VIA Technologies Inc. ICE1712 [Envy24] PCI Multi-Channel I/O Controller (rev 02) Subsystem: VIA Technologies Inc. M-Audio Delta 1010LT Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 32 Interrupt: pin A routed to IRQ 19 Region 0: I/O ports at a040 [size=32] Region 1: I/O ports at a070 [size=16] Region 2: I/O ports at a060 [size=16] Region 3: I/O ports at a000 [size=64] Capabilities: [80] Power Management version 1 Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: snd_ice1712

Offline

#5663 2015-07-17 15:03:16

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Quadro cards operate as a secondary display, the GPU ROM isn't used.


Interesting. Does that mean that they do not work as a primary VGA at all, or that the ROM isn't needed when the card is secondary?

Offline

#5664 2015-07-17 15:16:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

impulse_255 wrote:
aw wrote:

Quadro cards operate as a secondary display, the GPU ROM isn't used.


Interesting. Does that mean that they do not work as a primary VGA at all, or that the ROM isn't needed when the card is secondary?

Both.  Nvidia does not support the primary graphics use case with Quadro (nor GeForce, but nothing VM related is supported on GeForce) and in secondary mode, with the Nvidia graphics drivers, the ROM is not used.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5665 2015-07-17 15:30:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fld wrote:

I recently started using a M-Audio 1010lt PCI soundcard (via a ASMedia ASM1083 PCIe to PCI bridge) for my host audio. But ever since, I've been getting this strange DMAR flood in my dmesg whenever the sound card is being used:

[64721.567981] dmar: DRHD: handling fault status reg 3 [64721.567998] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [64721.568080] dmar: DRHD: handling fault status reg 3 [64721.568082] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set [64721.716747] dmar: DRHD: handling fault status reg 3 [64721.716751] dmar: DMAR:[DMA Read] Request device [08:00.0] fault addr 10000000 DMAR:[fault reason 06] PTE Read access is not set ...

As I was originally only using just "intel_iommu=on,igfx_off", I thought adding the "iommu=pt" would help to get rid of this problem, because it is supposed to disable the DMAR. However, it would appear that adding the "iommu=pt" changed nothing.

I haven't noticed any other ill effects than my dmesg being flooded. I was hoping perhaps aw knows something about this?

So you have a crappy Via sound card behind a crappy ASMedia bridge and it doesn't work with the IOMMU... quelle surprise.  Actually it just seems like the audio card is buggy, it appears to be doing a DMA read at address 256MB, probably to force any previous DMA writes to memory, ie. a flush.  Is this address consistent between boots?  I'm guessing it is.  The hardware designers probably thought "hey, there's always memory at 256MB, we can just read from that for a DMA flush, what could go wrong".  Then along comes the IOMMU that restricts DMA from the device to only the regions the driver has mapped for the device and you get these sorts of errors.  Maybe the driver could try to map a buffer at 256MB, but it apparently doesn't.  All in all, the audio card isn't fit for use behind an IOMMU.

Now, why doesn't iommu=pt magically work?  Well, it's a 32bit (or less) device and in order to support a 32bit device on a system with potentially more than 4G of RAM we'd need to have bounce buffers.  But wait, we have an IOMMU, why would we also want bounce buffers?  So even though you requested passhrough mode, the sound card got demoted (you chopped this part out of the dmesg) and it's running with normal translation.

Get a better sound card.

EDIT: Oh, BTW in an interesting twist of technology, the sound card would probably work perfectly assigned to a VM because guest physical memory address 256MB will be DMA read-able.

Last edited by aw (2015-07-17 15:35:43)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5666 2015-07-17 19:29:39

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
impulse_255 wrote:
aw wrote:

Quadro cards operate as a secondary display, the GPU ROM isn't used.


Interesting. Does that mean that they do not work as a primary VGA at all, or that the ROM isn't needed when the card is secondary?

Both.  Nvidia does not support the primary graphics use case with Quadro (nor GeForce, but nothing VM related is supported on GeForce) and in secondary mode, with the Nvidia graphics drivers, the ROM is not used.


Okay, thanks!

I'll be back home in a couple of days. Then I will try spoofing my card as an M6000 and tell how it went.

Offline

#5667 2015-07-18 04:02:41

sofloantonio
Member
Registered: 2015-07-18
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone got this working with an a10-5800k or similar?

I have the a10 and a gtx750ti. I have a few potentially stupid questions.

So the aim is to have the host os (arch) be using the integrated graphics of the a10, and the windows installation to use the gtx 750. In bios, there is a setting which can switch between whether the pci graphics card is used by default or if the integrated graphics are used. Does this need to be changed? or does it have no effect?

also, im using two hdmi cords, both connected to a monitor which accepts two different inputs. one hdmi is connected to the graphics card, the other to the motherboard's hdmi slot. is this correct?

thanks

Offline

#5668 2015-07-18 09:52:23

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

So you have a crappy Via sound card behind a crappy ASMedia bridge and it doesn't work with the IOMMU... quelle surprise.  Actually it just seems like the audio card is buggy, it appears to be doing a DMA read at address 256MB, probably to force any previous DMA writes to memory, ie. a flush.

Sounds very plausible.

aw wrote:

Is this address consistent between boots?  I'm guessing it is.

Right you are.

aw wrote:

The hardware designers probably thought "hey, there's always memory at 256MB, we can just read from that for a DMA flush, what could go wrong".  Then along comes the IOMMU that restricts DMA from the device to only the regions the driver has mapped for the device and you get these sorts of errors.  Maybe the driver could try to map a buffer at 256MB, but it apparently doesn't.

So the the hardware doesn't define a memory BAR at 256M, even though it tries to read it. Could it be possible to hack the snd_ice1712 driver to reserve that memory? Or is it too late to try and "fix" it in the driver?

aw wrote:

All in all, the audio card isn't fit for use behind an IOMMU.

Strange that the card still does appear to "fully work". The act of doing a DMA Read at a fault address must still be achieving the flush?

aw wrote:

Now, why doesn't iommu=pt magically work?  Well, it's a 32bit (or less) device and in order to support a 32bit device on a system with potentially more than 4G of RAM we'd need to have bounce buffers.  But wait, we have an IOMMU, why would we also want bounce buffers?

Then passthrough will only work for devices that support 64 bit addressing?

aw wrote:

  So even though you requested passhrough mode, the sound card got demoted (you chopped this part out of the dmesg) and it's running with normal translation.

Yeah, sorry about cutting away important stuff, I was trying to make it easier to read. Here is the full dmesg: http://paste.debian.net/hidden/f5743abd/

aw wrote:

Get a better sound card.

Yeah, perhaps I'll get one of those simple USB sound cards. I'm guessing they are fairly usable by now. Only reason I'm still using the VIA sound card is because it can directly drive my headphones really nicely without any sort of background hiss or noise.

But in the meanwhile, I'm thinking ill just figure out a way for syslog to suppress those DMAR messages. Since those messages filling up my log seem to be the only bad thing that is happening.

Offline

#5669 2015-07-18 13:06:13

evujumenuk
Member
Registered: 2013-07-02
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evujumenuk wrote:
aw wrote:
aw wrote:

Borderlands folks, I think this is the problem you're trying to describe:

https://dl.dropboxusercontent.com/u/198 … emetal.jpg

https://dl.dropboxusercontent.com/u/198 … ds2-vm.jpg

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

https://dl.dropboxusercontent.com/u/198 … vm-new.jpg

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

So, what is the best current alternative to hv-time? Can't really use that with Nvidia drivers.

/USE PMTIMER?

I just thought of something.

Could one enable Invariant TSC on the -cpu flag and get a clock as good as hypervclock? The catch would be that savevm (and, by extension, migration) would not work. But I suspect that is already the case with VFIO.

Offline

#5670 2015-07-18 15:50:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fld wrote:

So the the hardware doesn't define a memory BAR at 256M, even though it tries to read it. Could it be possible to hack the snd_ice1712 driver to reserve that memory? Or is it too late to try and "fix" it in the driver?

This isn't related to BARs, this is how the device accesses memory.  It should be possible to hack the driver to create a coherent mapping at 256MB, it appears that dma_declare_coherent_memory() would be the mechanism to do such a thing.  I also note this driver sets a coherent DMA mask of 28 bits, ie. 256MB, which is a bit at odds with declaring coherent memory starting at 256MB and an indication of just how outdated this hardware is.

Strange that the card still does appear to "fully work". The act of doing a DMA Read at a fault address must still be achieving the flush?

Potentially

Then passthrough will only work for devices that support 64 bit addressing?

64bit is not strictly required, but the device must be capable of DMA'ing to any physical memory in the system.  However, conventional PCI devices like this are a special case.  Only PCIe devices use a unique requester ID.  For conventional PCI devices, the PCIe-to-PCI bridge provides the requester ID, this is why the error reports 08:00.0 rather than 08:01.0 because that's the requester ID for the entire secondary side of the bridge.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5671 2015-07-19 03:01:59

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, everything works now, which is nice; everything remaining is a matter of convenience. My problem was eventually solved by using the OVMF binaries linked by @impulse_255, so there must be an issue in the newest version included in the edk.
I do have a couple of questions though:
1. I'm using a seperate physical interface for networking, a PCI wifi card, with the host using onboard Ethernet; but this means for slower and higher-latency transfer between the two, what is the simplest host-to-guest networking, without external connectivity? I thought it would be NAT, but documentation seems to suggest that it is for guest to guest connectivity. Bridging doesn't seem to make h-t-g networking a possibility, so how can I do it? EDIT: I'm terrible at reading.
2. Is there a better alternative than Synergy for gaming? Hitting the edges is a problem, and it seems to handle direct pointer captures very badly, so besides Steam Inhouse Streaming, is there a viable way to play D3D games?

Thanks for all your help getting me this far, and if anyone with similar issues to mine needs any help, I'd be more than happy.

Last edited by morat (2015-07-19 03:02:59)

Offline

#5672 2015-07-19 04:33:42

wmarler
Member
From: Denver, CO
Registered: 2015-04-18
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey forum,

I'm running into a wall. I'm using virt-manager to prepare a VM for VGA passthrough. When a VM is configured to use UEFI firmware, Windows will not install, but fails on "Copying Windows files (0%)". In the case of Win8.1/10, Windows throws up the frowny-face blue-screen with a generic rebooting message and reboots, and Win 7 freezes with a black screen. On reboot & trying to install Windows again, the disk has been partitioned (4 partitions with Win 8.1/10, 3 with 7), but the main/non-system partition cannot be installed to without formatting. The details reads "... This hard disk space is formatted with an unsupported version of the NTFS file system." The disk can be re-formatted and selected as the install target, but doing so results in the same reboot/black screen. Creating/formatting partitions manually results in the crash/black screen too. I've tried many, many iterations. I finally created 2 VM's identical to each other save for uuid, LVM storage location, and NIC MAC address, gave one the UEFI firmware ("win10b") and the other the BIOS firmware ("win10a"), and installed the BIOS one just fine.

My conclusion: there's something up with the UEFI firmware and/or my motherboard. I have no insight into what, nor how to get any insight. Watching 'journalctl -f' during the crash shows nothing, and I don't know how to get Windows to tell me anything.

I am using the UEFI firmware as provided by the Gerd Hoffman RPM repository. The version is edk2.git-ovmf-x64.noarch 0-20150716.b1120.g387e7c1
I'm on Fedora 22, kernel 4.0.8-300, with an i5-4960 CPU, and an ASRock Z87 PRO3 motherboard (no successes and 1 failure reported in the spreadsheet; disheartening, but not conclusive). The motherboard BIOS is up to date, according to the motherboard update utility.

A dumpxml of the working VM is here (win10a.xml).
A dumpxml of the non-working VM is here (win10b.xml).

The diff between these two dumps is here:

% diff win10a.xml win10b.xml 2,4c2,4 < <name>win10a</name> < <uuid>b80ea20a-5157-4afd-87f9-63b1e2a5b1bf</uuid> < <description>BIOS Firmware</description> --- > <name>win10b</name> > <uuid>c8c14fba-ecdd-4d91-ae7c-a9f52497d1b2</uuid> > <description>UEFI Firmware</description> 9a10 > <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> 40c41 < <source dev='/dev/vgmain/win10_1'/> --- > <source dev='/dev/vgmain/win10_2'/> 86c87 < <mac address='52:54:00:e7:67:f7'/> --- > <mac address='52:54:00:c8:1b:45'/>

The issue is easily reproducible. 

Any suggestions other than "buy a known-good motherboard" or "use the VGA arbitration patch"? Those are obvious next steps, but neither are solutions I'm particularly interested in without exhausting all of my options.

TIA

Last edited by wmarler (2015-07-19 04:47:43)

Offline

#5673 2015-07-19 04:43:52

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@wmarler, I just had this exact issue:
Use this set of binaries, as linked by @impulse_255
I'd say this will probably work for you, as yours was my exact issue.
Win 7 will not work, it seems to have UEFI trouble.

Also, please code tag your diffs, and other code.

Last edited by morat (2015-07-19 04:45:24)

Offline

#5674 2015-07-19 05:20:15

wmarler
Member
From: Denver, CO
Registered: 2015-04-18
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

@wmarler, I just had this exact issue:
Use this set of binaries, as linked by @impulse_255
I'd say this will probably work for you, as yours was my exact issue.

Thanks, that worked. And was very easy :-D.

morat wrote:

Win 7 will not work, it seems to have UEFI trouble.

Bummer ... that's what I have a license for atm.

Offline

#5675 2015-07-19 06:48:57

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

2. Is there a better alternative than Synergy for gaming? Hitting the edges is a problem, and it seems to handle direct pointer captures very badly, so besides Steam Inhouse Streaming, is there a viable way to play D3D games?

Thanks for all your help getting me this far, and if anyone with similar issues to mine needs any help, I'd be more than happy.

If what you're referring to by "seems to handle direct pointer captures very badly" is what I think, ensure you have 'relativeMouseMoves = true' in your server config, bind a button to lockCursorToScreen(toggle), and then lock the cursor to the vm screen when you're playing, should solve that issue and also your edge problem.

Offline

#5676 2015-07-19 07:42:47

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:
morat wrote:

2. Is there a better alternative than Synergy for gaming? Hitting the edges is a problem, and it seems to handle direct pointer captures very badly, so besides Steam Inhouse Streaming, is there a viable way to play D3D games?

Thanks for all your help getting me this far, and if anyone with similar issues to mine needs any help, I'd be more than happy.

If what you're referring to by "seems to handle direct pointer captures very badly" is what I think, ensure you have 'relativeMouseMoves = true' in your server config, bind a button to lockCursorToScreen(toggle), and then lock the cursor to the vm screen when you're playing, should solve that issue and also your edge problem.

Worked like a charm, thanks!

Michael Corleone wrote:

Just when I thought I was out, they pull me back in.

The most luck I've had with audio so far is Windows 8 recognising a "High Definition Audio Controller", which believes it's outputting sound, but I hear nothing. I have used vnc_allow_host_audio = 1, and tried setting the qemu user to my own, but that only causes permission issues with the pci. libvirt is set to use ich9.

Edit: I should also point out that I'm not using, nor have I been able to test, the nVidia HDMI sound, as the financial situ at the moment prohibits good monitors.

Last edited by morat (2015-07-19 08:41:17)

Offline

#5677 2015-07-19 12:32:49

Schlunze
Member
Registered: 2013-10-03
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morat wrote:

The most luck I've had with audio so far is Windows 8 recognising a "High Definition Audio Controller", which believes it's outputting sound, but I hear nothing. I have used vnc_allow_host_audio = 1, and tried setting the qemu user to my own, but that only causes permission issues with the pci. libvirt is set to use ich9.

Maybe you use libvirt / virtmanager and the guest window is closed then you get no sound output.

Offline

#5678 2015-07-19 21:09:33

Col_Panic
Member
Registered: 2015-07-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys! Long time reader here, first time poster. I hope someone can help me out!
I've had passthrough working for a while now, on my old GPU; I'm running kernel 4.1.2, and binding the GPU to vfio on boot.
I'm also loading the modules:
amd_iommu
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
I don't have any other special configuration on either my host or guest.

After switching out my primary GPU for the old KVM one, and adding a brand shiny new secondary GPU, my setup is now:

CPU: AMD FX8350
MOBO: Asus M5A97 LE R2.0 (with latest BIOS)
GPU 1: Radeon HD 7770
GPU 2: Radeon R9 270x

Passthrough still works for the 7770 if I use that one as secondary instead, but trying to pass through my new GPU doesn't seem to work. My Windows guest starts, then my host locks up as the guest tries to load the GPU.
This is what it dumps out:

[ 645.924773] qemu-system-x86: sending ioctl 5326 to a partition! [ 645.924784] qemu-system-x86: sending ioctl 80200204 to a partition! [ 648.526720] vfio-pci 0000:00:13.2: enabling device (0000 -> 0002) [ 648.558718] vfio_cap_init: 0000:00:13.2 hiding cap 0xa [ 648.674350] AMD-Vi: Completion-Wait loop timed out [ 648.787568] AMD-Vi: Completion-Wait loop timed out [ 648.900662] AMD-Vi: Completion-Wait loop timed out [ 648.936352] AMD-Vi: Completion-Wait loop timed out [ 649.012717] AMD-Vi: Completion-Wait loop timed out [ 649.123771] AMD-Vi: Completion-Wait loop timed out [ 649.234736] AMD-Vi: Completion-Wait loop timed out [ 649.349347] AMD-Vi: Completion-Wait loop timed out [ 649.464152] AMD-Vi: Completion-Wait loop timed out [ 649.478740] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003) [ 649.478995] vfio_ecap_init: 0000:05:00.0 hiding ecap 0x19@0x270 [ 649.479008] vfio_ecap_init: 0000:05:00.0 hiding ecap 0x1b@0x2d0 [ 649.498706] vfio-pci 0000:05:00.1: enabling device (0000 -> 0002) [ 649.678811] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f810] [ 649.678823] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f830] [ 649.678829] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f850] [ 650.797361] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f8a0] [ 650.808317] AMD-Vi: Completion-Wait loop timed out [ 650.808320] ------------[ cut here ]------------ [ 650.808326] WARNING: CPU: 6 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 650.808328] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 650.808364] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 650.808379] CPU: 6 PID: 3851 Comm: qemu-system-x86 Not tainted 4.1.2-040102-generic #201507101335 [ 650.808380] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 650.808382] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 650.808385] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 650.808387] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 650.808389] Call Trace: [ 650.808394] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 650.808398] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 650.808400] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 650.808401] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 650.808404] [<ffffffff81013673>] ? __switch_to+0x1e3/0x580 [ 650.808406] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 650.808409] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 650.808411] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 650.808417] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 650.808421] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 650.808437] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 650.808446] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 650.808452] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 650.808455] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 650.808459] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 650.808461] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 650.808463] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 650.808465] ---[ end trace 29b4fb9fbe5211cb ]--- [ 650.926817] AMD-Vi: Completion-Wait loop timed out [ 650.926826] ------------[ cut here ]------------ [ 650.926841] WARNING: CPU: 5 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 650.926844] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 650.926923] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 650.926957] CPU: 5 PID: 3851 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 650.926961] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 650.926965] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 650.926971] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 650.926976] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 650.926982] Call Trace: [ 650.926991] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 650.926998] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 650.927003] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 650.927007] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 650.927014] [<ffffffff81013673>] ? __switch_to+0x1e3/0x580 [ 650.927019] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 650.927025] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 650.927029] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 650.927041] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 650.927053] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 650.927083] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 650.927105] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 650.927119] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 650.927127] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 650.927133] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 650.927138] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 650.927144] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 650.927149] ---[ end trace 29b4fb9fbe5211cc ]--- [ 651.058621] AMD-Vi: Completion-Wait loop timed out [ 651.058626] ------------[ cut here ]------------ [ 651.058632] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 651.058633] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 651.058669] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 651.058684] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 651.058686] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 651.058687] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 651.058689] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 651.058691] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 651.058694] Call Trace: [ 651.058699] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 651.058702] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 651.058704] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 651.058705] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 651.058707] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 651.058710] [<ffffffff8118fb1e>] ? compound_unlock_irqrestore+0xe/0x20 [ 651.058713] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 651.058715] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 651.058720] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 651.058725] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 651.058742] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 651.058750] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 651.058756] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 651.058784] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 651.058788] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 651.058800] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 651.058818] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 651.058829] ---[ end trace 29b4fb9fbe5211cd ]--- [ 651.183912] AMD-Vi: Completion-Wait loop timed out [ 651.183917] ------------[ cut here ]------------ [ 651.183922] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 651.183924] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 651.183956] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 651.183970] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 651.183971] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 651.183973] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 651.183975] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 651.183977] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 651.183979] Call Trace: [ 651.183984] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 651.183987] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 651.183989] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 651.183990] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 651.183993] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 651.183996] [<ffffffff81190156>] ? put_compound_page+0xd6/0x1b0 [ 651.183999] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 651.184000] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 651.184006] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 651.184010] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 651.184027] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 651.184035] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 651.184041] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 651.184045] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 651.184047] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 651.184050] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 651.184052] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 651.184054] ---[ end trace 29b4fb9fbe5211ce ]--- [ 651.311628] AMD-Vi: Completion-Wait loop timed out [ 651.311633] ------------[ cut here ]------------ [ 651.311640] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 651.311641] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 651.311682] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 651.311699] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 651.311701] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 651.311703] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 651.311706] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 651.311708] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 651.311711] Call Trace: [ 651.311716] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 651.311720] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 651.311722] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 651.311724] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 651.311727] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 651.311730] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 651.311732] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 651.311739] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 651.311744] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 651.311762] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 651.311772] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 651.311779] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 651.311783] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 651.311786] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 651.311789] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 651.311792] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 651.311794] ---[ end trace 29b4fb9fbe5211cf ]--- [ 651.437788] AMD-Vi: Completion-Wait loop timed out [ 651.437793] ------------[ cut here ]------------ [ 651.437798] WARNING: CPU: 5 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 651.437799] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 651.437837] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 651.437853] CPU: 5 PID: 3851 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 651.437854] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 651.437856] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 651.437858] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 651.437860] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 651.437862] Call Trace: [ 651.437867] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 651.437870] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 651.437872] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 651.437873] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 651.437876] [<ffffffff81013673>] ? __switch_to+0x1e3/0x580 [ 651.437878] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 651.437881] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 651.437883] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 651.437888] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 651.437893] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 651.437909] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 651.437917] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 651.437923] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 651.437927] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 651.437930] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 651.437932] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 651.437935] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 651.437936] ---[ end trace 29b4fb9fbe5211d0 ]--- [ 651.562996] AMD-Vi: Completion-Wait loop timed out [ 651.563001] ------------[ cut here ]------------ [ 651.563007] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170() [ 651.563008] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 651.563045] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 651.563061] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 651.563062] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 651.563064] ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000 [ 651.563066] 0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb [ 651.563068] ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff [ 651.563070] Call Trace: [ 651.563075] [<ffffffff817d076c>] dump_stack+0x45/0x57 [ 651.563078] [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0 [ 651.563080] [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20 [ 651.563081] [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170 [ 651.563083] [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140 [ 651.563087] [<ffffffff8118fb1e>] ? compound_unlock_irqrestore+0xe/0x20 [ 651.563090] [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0 [ 651.563091] [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80 [ 651.563097] [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1] [ 651.563101] [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1] [ 651.563117] [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] [ 651.563126] [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm] [ 651.563132] [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] [ 651.563135] [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510 [ 651.563138] [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450 [ 651.563140] [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0 [ 651.563143] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 651.563145] ---[ end trace 29b4fb9fbe5211d1 ]--- [ 651.654821] BUG: unable to handle kernel NULL pointer dereference at 0000000000000049 [ 651.654855] IP: [<ffffffff810f745e>] drop_futex_key_refs.isra.14+0xe/0x70 [ 651.654882] PGD 0 [ 651.654890] Oops: 0000 [#1] SMP [ 651.654903] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore [ 651.655166] kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi [ 651.655244] CPU: 4 PID: 3848 Comm: qemu-system-x86 Tainted: G W 4.1.2-040102-generic #201507101335 [ 651.655275] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015 [ 651.655305] task: ffff88048e2764a0 ti: ffff8804bc08c000 task.ti: ffff8804bc08c000 [ 651.655350] RIP: 0010:[<ffffffff810f745e>] [<ffffffff810f745e>] drop_futex_key_refs.isra.14+0xe/0x70 [ 651.655380] RSP: 0018:ffff8804bc08fcc8 EFLAGS: 00010286 [ 651.655396] RAX: ffff88048e2764a0 RBX: 0000000000000001 RCX: 00000000c0000100 [ 651.655419] RDX: ffff8804bc090000 RSI: 0000000000000049 RDI: ffff8805313eda00 [ 651.655440] RBP: ffff8804bc08fcc8 R08: ffff8804bc08c000 R09: 0000000000000000 [ 651.655461] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000 [ 651.655482] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88048e2764a0 [ 651.655504] FS: 00007f24f8e3cb00(0000) GS:ffff88054ed00000(0000) knlGS:0000000000000000 [ 651.655528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 651.655545] CR2: 0000000000000049 CR3: 00000004bc371000 CR4: 00000000000407e0 [ 651.655566] Stack: [ 651.655574] ffff8804bc08fe28 ffffffff810f9122 ffff8804bc11bc00 ffffffff00000019 [ 651.655598] 0000000000000000 ffff8804bc08fd60 0000000000000002 000055989a1a1ac0 [ 651.655622] dead000000200200 ffffc90003567780 0000000000000064 ffff8804bc08fd20 [ 651.655646] Call Trace: [ 651.655657] [<ffffffff810f9122>] futex_wait+0x1a2/0x280 [ 651.655676] [<ffffffff8120d160>] ? poll_select_copy_remaining+0x130/0x130 [ 651.655698] [<ffffffff8120d160>] ? poll_select_copy_remaining+0x130/0x130 [ 651.655719] [<ffffffff810fa4ee>] do_futex+0xde/0x5f0 [ 651.655736] [<ffffffff8101e2b9>] ? read_tsc+0x9/0x10 [ 651.655753] [<ffffffff810ebd8e>] ? ktime_get_ts64+0x4e/0xf0 [ 651.655771] [<ffffffff8120d131>] ? poll_select_copy_remaining+0x101/0x130 [ 651.655793] [<ffffffff810faa71>] SyS_futex+0x71/0x150 [ 651.655811] [<ffffffff817d8272>] system_call_fastpath+0x16/0x75 [ 651.655830] Code: c3 0f 1f 40 00 55 48 89 e5 0f 1f 40 00 e8 0b cb 11 00 5d f3 c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 85 ff 48 89 e5 74 3a <8b> 06 83 e0 03 83 f8 01 74 10 83 f8 02 75 06 f0 ff 4f 4c 74 15 [ 651.655921] RIP [<ffffffff810f745e>] drop_futex_key_refs.isra.14+0xe/0x70 [ 651.655943] RSP <ffff8804bc08fcc8> [ 651.655953] CR2: 0000000000000049 [ 651.660818] ---[ end trace 29b4fb9fbe5211d2 ]--- [ 651.916078] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f920] [ 651.916087] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f940] [ 653.034592] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fdb0] [ 653.034601] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f9d0] [ 653.034605] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f9f0] [ 653.034609] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa10] [ 653.034613] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa30] [ 653.034616] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa50] [ 653.034619] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa70] [ 653.034622] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa90] [ 653.034626] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fab0] [ 653.034629] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fad0] [ 653.034632] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2faf0] [ 653.034635] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb10] [ 653.034639] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb30] [ 653.034642] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb50] [ 653.034645] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb70] [ 653.034648] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb90] [ 653.034651] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fbb0] [ 653.034654] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fbd0] [ 653.034658] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fbf0] [ 653.034661] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc10] [ 653.034664] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc30] [ 653.034667] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc50] [ 653.034670] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc70] [ 653.034674] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc90] [ 653.034677] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fcb0] [ 653.034680] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fcd0] [ 653.034683] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fcf0] [ 653.034686] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd10] [ 653.034689] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd30] [ 653.034693] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd50] [ 653.034696] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd70]

...followed by a lot more IOTLB_INV_TIMEOUT messages.

I also get a set of "AMD-Vi: Completion-Wait loop timed out" messages on boot if I bind the 270x card (otherwise, it's the same as the other card), but afterwards, lspci shows the card properly bound to vfio.

I'm not sure what to do here, Has anyone seen this issue before, who might be able to point me in the right direction?

Offline

#5679 2015-07-19 21:17:52

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Col_Panic
I was getting inexplicable kernel panics for the longest time, before it was pointed out to me that one of the PCI ids was mistyped. Check and double check, and if it still hangs, pastebin your dmesg and command line/domain xml; and perhaps someone wiser than I could lend a hand.

Offline

#5680 2015-07-19 21:20:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Col_Panic

modprobe.d:
options vfio_iommu_type1 disable_hugepages=1

That's my default answer for anyone getting strange AMD-Vi problems.

BTW, amd_iommu cannot be built as a module.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5681 2015-07-19 21:21:37

morat
Member
Registered: 2015-07-07
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Schlunze wrote:
morat wrote:

The most luck I've had with audio so far is Windows 8 recognising a "High Definition Audio Controller", which believes it's outputting sound, but I hear nothing. I have used vnc_allow_host_audio = 1, and tried setting the qemu user to my own, but that only causes permission issues with the pci. libvirt is set to use ich9.

Maybe you use libvirt / virtmanager and the guest window is closed then you get no sound output.

So then, is it impossible to have sound passed through the host without a libvirt VNC session?

Offline

#5682 2015-07-19 22:05:17

Col_Panic
Member
Registered: 2015-07-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Col_Panic

modprobe.d:
options vfio_iommu_type1 disable_hugepages=1

That's my default answer for anyone getting strange AMD-Vi problems.

BTW, amd_iommu cannot be built as a module.

Thanks! This seems to have gotten me somewhere,
I'm still getting the same iommu_unmap, etc messages, followed by IOTLB_INV_TIMEOUT. But instead of crashing the host after IOTLB_INV_TIMEOUT, it's constantly looping back to the first error.
Passthrough isn't working yet, but at least one problem is solved!

Offline

#5683 2015-07-19 22:34:59

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No success whatsoever with the pciid spoofing. I attempted to spoof my Titan X as an M6000 for the purpose of using Hyper-V enlightenments. I patched Qemu 2.3.0 with the patch posted earlier. It needed some modifications to work with 2.3.0. The file was moved to vfio/pci.c instead of misc/vfio.c and one of the structs was now VFIOPCIDevice instead of VFIODevice.

Once I started Qemu with the new pci id, the VM GPU would not show a picture. I let it boot the OS but it would remain blank. Of course I did a sanity check to see if it would work with the alteration enabled but with the original pci id. That worked. However, passing any rom with the romfile option produced a black screen, even with the actual rom dumped with GPU-Z in guest Windows. In summary:

1) altered PCIIDs to M6000 (x-vid=0x10DE,x-did=0x17f0,x-ss-vid=0x10DE,x-ss-did=0x1129) => black screen
2) altered PCIIDs to M6000, supplied original rom => black screen
3) altered PCIIDS to original (x-vid=0x10DE,x-did=0x17c2,x-ss-vid=0x10DE,x-ss-did=0x1132) => works
4) altered PCIIDS to original, supplied original rom => black screen

#4 puzzles me the most. Another thing worth mentioning is that I got a BSOD when I tried to boot without any trickery after attempting all of the above. After a host reboot, I could boot the guest again, suggesting that the GPU was left in a broken state before the reboot. I don't know if supplying an M6000 rom, which I don't have, would do any good since even the card's own rom gives me a black screen. I wonder if Nvidia has added more obstacles to the cards.

Could someone else with a Maxwell card try this?

EDIT: For clarification, I also tried #4 after a host reboot to make sure the GPU was not left in a messed up state. Black screen. I forgot to mention that I briefly tried SeaBIOS instead of OVMF with a more minimalistic qemu command line and #1 but I still couldn't get a picture out of the GPU.

Here's an example of my configuration for the GPU passthrough. I removed the hostdev configuration and replaced it with a custom qemu command line configuration:

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,id=hostdevvga,bus=pci.0,addr=0x8,x-vid=0x10DE,x-did=0x17c2,x-ss-vid=0x10DE,x-ss-did=0x1132,romfile=GM200.rom'/> </qemu:commandline>

where the values are according to #4.

Last edited by impulse_255 (2015-07-19 22:54:09)

Offline

#5684 2015-07-20 06:27:28

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thx impulse_255 for trying it out....

This is how I start my vm as grid k2....

<qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,x-vid=0x10DE,x-did=0x11BF,x-ss-vid=0x10DE,x-ss-did=0x0965''/>

Offline

#5685 2015-07-20 10:02:47

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, I got it working. First of all, the rom dump I took from Windows was different to one I took from the host with the GPU in an uninitialised state (resumed from S3 suspend and had not started the VM after that). Once I repeated #4 with that dump, it worked.

Next I read the eevblog forum posts on hardware pci id modding (euphemism for fiddling with your pricey graphics cards with a soldering iron) and rom flashing. Damn, we have it easy. Everything can be safely done in software. smile I took the working original rom and changed the pci ids with a hex editor. VGA roms have a rudimentary checksum which can be fooled by changing some non-critical plaintext strings present in the rom to compensate for any relevant changes: http://www.eevblog.com/forum/chat/hacki … #msg208495

It works! In installed the Quadro driver and enabled hyper-v enlightenments. I did not run benchmarks yet but I can't see any significant difference, though. Crysis Warhead which runs noticeably worse than on bare metal is still as cpu limited as ever. Well, it was an exiting journey anyway, lol.

Offline

#5686 2015-07-20 10:44:17

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nice, does it work with geforce driver? not quadro one? but I guess its same... now i can go maxwell haha smile
edit: you got i7 2600? about crysis... did you try run different cpu config like cores=4/8 threads=1? and what about cpu governor is it set to performance? I'm using it like this http://justpaste.it/mgsj



edit2: maybe you should post it to eevlog, so that guys with vt-d don't have to solder big_smile

Last edited by slis (2015-07-20 11:50:42)

Offline

#5687 2015-07-20 11:51:37

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The quadro driver is a separate package. After booting with the changes, Windows was in standard vga mode and I had to install the other driver. It seems to work just the same, including the nv control panel.

I have tried various cpu settings such as cores=4,threads=2 which will give the guest all the throughput it can get. Performance governor too. Now I use cores=3,threads=2 and keep host processes on the remaining core to keep latency and stutter down. I'm not that concerned with how Crysis runs. I noticed its framerate issues and I have since used it as a benchmark for a game that does not play nicely in a VM (for me at least). It would be good to iron out such exceptional issues. Other games such as Witcher 3 run just fine with everything cranked up.

BTW, a disadvantage of having a card appear as Quadro: MSI Afterburner no longer provides voltage controls for overclocking. sad Maybe it could be fooled somehow as well. It's the exact same card and bios except for the ids after all. But that does not belong here.

Offline

#5688 2015-07-20 11:56:33

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For me using threads=1 with cores=4/8 instead threads=2 cores=2/4 is much faster in cpu dependent games / dota2.

Offline

#5689 2015-07-20 12:03:07

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

For me using threads=1 with cores=4/8 instead threads=2 cores=2/4 is much faster in cpu dependent games / dota2.

Hmm. Someone on reddit claimed the exact opposite. Do you know if Windows actually treats logical (HT) processors differently than physical or does threads=1 vs. 2 change hypervisor behaviour?

Offline

#5690 2015-07-20 12:07:18

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, I don't know, you should try it, with smth similar to what I posted on that pasteit link, real cores are 0-3.

Offline

#5691 2015-07-20 12:24:33

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Well, I don't know, you should try it, with smth similar to what I posted on that pasteit link, real cores are 0-3.


All right, thanks. I'll try that.

Offline

#5692 2015-07-20 20:57:32

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wonder is it possible to pass 2nd GPU or two GPUs to same VM without ACS patch?
I just trying to pass HD6950 (work alone just fine) and GTX260 into same VM, but it's just don't work.
When I try to only pass GTX260 (that use 2nd PCIe port) it's fail with "KVM internal error. Suberror: 1".

Is that would work if I'm going to apply ACS? And if so is same patch would work for 4.1 kernel?

PS: Also a bit off-topic question, but anyway. Is there way to make QEMU emulate USB mouse with more than 3 buttons? I already using "-device usb-mouse", but on host I have mouse with 7 buttons while QEMU only grabbing 3 first.

Last edited by SXX (2015-07-20 20:57:45)

Online

#5693 2015-07-20 21:14:36

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nngh. The Quadro solution isn't problem free either. If hyper-v enlightenments are enabled, the driver does work but runs in some weird crippled mode. There are some missing configuration options such as display rotation, digital audio and custom resolutions. It also would not let me use refresh rates over 85 Hz which is very annoying since I could go up to 144 Hz. Now I'm torn between making the most out of my monitor in less demanding games or getting some extra performance in more demanding ones. To make matters worse I can't simply switch them on or off between VM restarts because in Quadro mode and hyper-v disabled fullscreen games show a blank screen. WTF? Switching between "graphics cards" needs a driver reinstall every time. I'm starting to believe nvidia's explanation that it was indeed incompetence, not malice, that broke hyper-v.

As a bonus, I ran GTA 5's built-in benchmark with Quadro mod + HV and Titan X without HV. The difference is small but there were annoying stutters that occurred without HV. It could be placebo, too. smile Here are the results:

HV enabled:
Frames Per Second (Higher is better) Min, Max, Avg
Pass 0, 39.046333, 92.392410, 58.477955
Pass 1, 20.063923, 80.518539, 48.624252
Pass 2, 36.416340, 88.159317, 61.158989
Pass 3, 35.666138, 93.538368, 69.098633
Pass 4, 15.514916, 103.371994, 58.484459

HV disabled:
Frames Per Second (Higher is better) Min, Max, Avg
Pass 0, 24.502665, 69.663994, 57.547050
Pass 1, 27.271063, 61.661015, 47.173367
Pass 2, 36.337238, 79.242561, 59.791538
Pass 3, 24.399113, 84.966293, 67.116913
Pass 4, 15.372684, 90.449654, 55.312733

I think I'll stick without HV at least for now because of the myriad issues.

Offline

#5694 2015-07-20 23:29:20

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

impulse_255 wrote:

No success whatsoever with the pciid spoofing. I attempted to spoof my Titan X as an M6000 for the purpose of using Hyper-V enlightenments. I patched Qemu 2.3.0 with the patch posted earlier. It needed some modifications to work with 2.3.0. The file was moved to vfio/pci.c instead of misc/vfio.c and one of the structs was now VFIOPCIDevice instead of VFIODevice.

Impulse,

Do you think you could link your updated patch for the stock pci.c in QEMU 2.3.0? I've tried modifying the patch but i haven't been able to get it to work

Last edited by The_Moves (2015-07-21 00:20:06)

Offline

#5695 2015-07-21 01:47:02

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I might have gotten it, but now i'm getting this:

XML document failed to validate against schema: Unable to validate doc against/usr/share/libvirt/schemas/domain.rng
Element domain has extra content: qemu:command line

EDIT: Never mind, I must not have gotten it patched correctly...more research from my end on how to patch packages. I was able to get a Kernel patched, but this...

Last edited by The_Moves (2015-07-21 02:27:14)

Offline

#5696 2015-07-21 02:03:43

Col_Panic
Member
Registered: 2015-07-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I'm still having no luck getting past my problem for this GPU. I've been experimenting with different boot options, and I'm still stuck at where I was after aw's suggestion; instead of locking the host, I'm just getting an endless loop of iommu page flushes, and a guest that complains if I try to destroy it.
I thought I was getting somewhere with nofullflush, when I was able to shut down the guest, but it turns out that was a fluke.
I came across the same symptoms on the KVM dev list, and it was concluded in that thread that it was probably a BIOS problem, rather than a driver issue.
Looks like it's time to upgrade my discount-store MOBO sad

Offline

#5697 2015-07-21 05:00:05

slis
Member
Registered: 2014-06-02
Posts: 125

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@the_moves, you need to change first line of xml to make it take qemu args,..
edit:

<domain type='kvm' id='5' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> 

@impulse_255 thx for that, it shows good  boost of min/ max fps...  well if its detected like quadro it would show nv control panel with less options then as geforce, but you maybe could use nview for resolution and stuff

Last edited by slis (2015-07-21 05:03:14)

Offline

#5698 2015-07-21 08:47:00

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@The_Moves: Here is the updated spoofing patch for Qemu 2.3.0: http://pastebin.com/vFpN06LX

@slis: The control panel shows all the options when hyper-v is disabled. Only when hyper-v is enabled, stuff goes missing and refresh rates are limited, so it doesn't seem like it's meant to be some quadro specific limitation. In fact, there is a quadro specific "workstation" category which also goes missing with HV. I have only so far tried with all four hv_* options enabled or none at all but I'm assuming it works by detecting the Microsoft Hv string just like the Geforce driver.

I really wonder what makes Nvidia drivers so allergic to it.

EDIT: But of course if you don't need 100+ Hz refresh rates or all the dials and buttons, you probably won't have any problems. Even G-Sync works in the crippled mode. I'm just being my obsessive perfectionist self. sad

Last edited by impulse_255 (2015-07-21 08:51:04)

Offline

#5699 2015-07-21 14:42:46

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

Just wonder is it possible to pass 2nd GPU or two GPUs to same VM without ACS patch?
I just trying to pass HD6950 (work alone just fine) and GTX260 into same VM, but it's just don't work.
When I try to only pass GTX260 (that use 2nd PCIe port) it's fail with "KVM internal error. Suberror: 1".

Is that would work if I'm going to apply ACS? And if so is same patch would work for 4.1 kernel?

PS: Also a bit off-topic question, but anyway. Is there way to make QEMU emulate USB mouse with more than 3 buttons? I already using "-device usb-mouse", but on host I have mouse with 7 buttons while QEMU only grabbing 3 first.

Get second mouse for host (wireless for example) and passthrough main one
or
passthrough your mouse to VM and use Synergy (host as client, VM as server).

Offline

#5700 2015-07-21 15:39:55

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
SXX wrote:

Just wonder is it possible to pass 2nd GPU or two GPUs to same VM without ACS patch?
I just trying to pass HD6950 (work alone just fine) and GTX260 into same VM, but it's just don't work.
When I try to only pass GTX260 (that use 2nd PCIe port) it's fail with "KVM internal error. Suberror: 1".

Is that would work if I'm going to apply ACS? And if so is same patch would work for 4.1 kernel?

PS: Also a bit off-topic question, but anyway. Is there way to make QEMU emulate USB mouse with more than 3 buttons? I already using "-device usb-mouse", but on host I have mouse with 7 buttons while QEMU only grabbing 3 first.

Get second mouse for host (wireless for example) and passthrough main one
or
passthrough your mouse to VM and use Synergy (host as client, VM as server).

No need for a second mouse and kb, a kvm switch (the other kvm) is handy if a software solution is not good enough.

Offline

#5701 2015-07-21 15:50:54

Col_Panic
Member
Registered: 2015-07-19
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I'm not quite ready to give up yet.
Since it looks to me like my [kvm/machine/driver] is trying to map and unmap the same memory region over and over again when it tries to attach the Radeon card to the host, is it possible that multiple devices are competing for the same memory space, or it's otherwise inaccessible? Maybe there's a way to tell VFIO to set the DMA mappings for this device at an early stage of the Linux boot process?
Or maybe I just have no idea what I'm talking about, and should give up already?

Offline

#5702 2015-07-21 15:54:26

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Col_Panic

in fact, the majority of the IOMMU mappings for the VM are setup prior to an execution of the VM, it's only a small set of legacy ranges and memory mapped I/O regions that change run-time.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5703 2015-07-22 05:09:17

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@impulse_255

Have you run into the KMODE_EXCEPTION_NOT_HANDLED error?

I've modified my GPU rom to have the Quadro K5000 (from a GTX 760) Device ID as well as the appropriate checksum. However, when I pass the romfile, all i get is a black screen... When I add a Virtual Console to the VM and view the boot through virt-manager, I see the above mentioned error. When I just change the device IDs and not pass a ROM, I am able to get Windows 10 setup to start, but when formatting the disk it shows the above error. Booting off a Windows 7 disk, all I get is a Windows 7 pulsating windows icon.

For some reason now, i can't even get my VM to boot with the previous config without hacking the device ID. I have power cycled my system, but I haven't pulled the GPU, maybe something bad is being stored on the GPU?

Offline

#5704 2015-07-22 09:02:25

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@The_Moves :

I had this exact BSOD with this exact error .

As soon as I switched to an older version of OVMF (15000~ something) , it was solved and I could install Windows 10 without a BSOD .

Offline

#5705 2015-07-22 09:16:33

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@The_Moves:

Did you modify your own rom or did you download it from somewhere? I tried a 3rd party rom for my card and got a black screen. Likewise a rom dump made in the guest from an initialised card would not work. The only way that worked was this incantation after a power cycle/suspend:

# echo 1 > /sys/bus/pci/devices/<pciid>/rom; # cat /sys/bus/pci/devices/<pciid>/rom > vbios.rom; # echo 0 > /sys/bus/pci/devices/<pciid>/rom

Also, changing IDs/roms without a power cycle gave me KMODE_EXCEPTION_NOT_HANDLED BSODs.

Did you have a working, installed guest before? Like said, the new OVMF builds throw the same BSOD at boot or installation, namely at the copying files step. Windows 7 also won't work with OVMF without some magic but I gave up with that.

Offline

#5706 2015-07-22 14:21:32

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso and @impulse_255

I will have to look at downgrading my qemu to what I was at before, I believe 2.1 or so. Impulse, what version are you running currently? You guys described exactly what I was experiencing. I believe the ROM file I was using was a dump from my card, but i'll boot onto my FreeDOS USB and grab a new copy. I followed updating the Device ID and Checksum in the ROM, but maybe I should do the Subsystem ID as well - I only discovered late last night that the x-ss-vid was for subsystem...

I've had a working Windows 10 build with OVMF for months now , only got the issue when I was when I started messing with the IDs for the card and used the latest version of Qemu to do it. It was my gaming system and was working on performance tweaking, trying to get as many FPS as possible.

I've seen this before, but what does it do? How does echoing 1 to a file change things?

# echo 1 > /sys/bus/pci/devices/<pciid>/rom; # cat /sys/bus/pci/devices/<pciid>/rom > vbios.rom; # echo 0 > /sys/bus/pci/devices/<pciid>/rom

@impulse_255, by changing IDs you mean changing the x-vid and x-ss-vid in the qemu line?

Offline

#5707 2015-07-22 14:37:31

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, I ended up flashing the updated vbios from gigabyte, which has a valid UEFI rom signature. This worked fine and still runs as normal in my seabios setup on the same system, but I get the same issue as before when trying to install catalyst -- it gets up to installing the driver proper, the system momentarily freezes up and then reboots during installing the actual driver. I do get (low-resolution) output on the physical card up until the VM freezes and reboots.

I've pasted my vm config here. Is there anything I might be missing on the host side of the setup for ovmf to work?

Offline

#5708 2015-07-22 17:10:48

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

@Denso and @impulse_255

I will have to look at downgrading my qemu to what I was at before, I believe 2.1 or so. Impulse, what version are you running currently? You guys described exactly what I was experiencing. I believe the ROM file I was using was a dump from my card, but i'll boot onto my FreeDOS USB and grab a new copy. I followed updating the Device ID and Checksum in the ROM, but maybe I should do the Subsystem ID as well - I only discovered late last night that the x-ss-vid was for subsystem...

I've had a working Windows 10 build with OVMF for months now , only got the issue when I was when I started messing with the IDs for the card and used the latest version of Qemu to do it. It was my gaming system and was working on performance tweaking, trying to get as many FPS as possible.

Oh. :( I can't say what's wrong there. Power cycling has fixed all the black screen/BSOD messes I've caused so far, and there shouldn't be any lasting hardware changes involved. The installer issues sound exactly like the OVMF issue. I once accidentally used a fresh, non-working version with my already installed Windows and it crashed all the same, so the issue is not limited to just the installer. Are you still using the same OVMF as before?

I'm currently using Qemu 2.3.0 and the old sata patched OVMF build whose build date or number I don't know.

The_Moves wrote:

I've seen this before, but what does it do? How does echoing 1 to a file change things?

Frankly, I don't know what it exactly does. It's some kernel interface that presents itself as a file.

The_Moves wrote:

@impulse_255, by changing IDs you mean changing the x-vid and x-ss-vid in the qemu line?

Yes. To clarify: if I booted the guest with one set of IDs, shut it down and tried to boot it again without a power cycle (or S3 suspend, which I use) with another set of IDs, Windows would BSOD. This happened even if the IDs and rom would otherwise work when I stick to using one at a time. BTW, did you modify the subsystem ID in the rom as well? I searched & replaced all I could find: 2 instances of PCI ID and one of SS ID. edit: couldn't read

Last edited by impulse_255 (2015-07-22 17:19:08)

Offline

#5709 2015-07-22 17:16:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BTW, I reported the Win8.1/OVMF install issue to Laszlo who bisected and found the commit that introduced it.  There's already a patch on their list to fix it, but the ongoing sorceforge outage is delaying it being committed and new builds including it.  Stick with a pre-July build until things get moving again.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5710 2015-07-22 17:20:52

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

BTW, I reported the Win8.1/OVMF install issue to Laszlo who bisected and found the commit that introduced it.  There's already a patch on their list to fix it, but the ongoing sorceforge outage is delaying it being committed and new builds including it.  Stick with a pre-July build until things get moving again.

Great, thanks!

Offline

#5711 2015-07-23 01:02:11

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed stock Qemu 2.1 from Source RPMs and am still getting the 'KMODE_EXCEPTION_NOT_HANDLED' error. When I remove libvirt and qemu, are there other files I should remove manually? I'm almost to the point of rebuilding from stock Fedora 21 or Fedora 22. I had no issues until I upgraded from 21 to 22, but 21 is going out of support soon.

Offline

#5712 2015-07-23 01:18:14

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

BTW, I reported the Win8.1/OVMF install issue to Laszlo who bisected and found the commit that introduced it.  There's already a patch on their list to fix it, but the ongoing sorceforge outage is delaying it being committed and new builds including it.  Stick with a pre-July build until things get moving again.

Oh, you mean OVMF itself. The OVMF firmware from EDK2 has an issue...so my Qemu/Libvirt reinstalling really wasn't the issue...ugh. I need to read closer.

Edit: I got finally got it working, but am still unable to pass a romfile or the multifunction attribute (not sure what it does). Windows does see the card as a K5000. In my game World of Tanks, normally when I would hover over different icons for tanks in the Garage, the screen would freeze for a split second. Overall though, there really isn't an improvement in FPS from what I can see. Loading into maps has improved by 50%. But it wouldn't hurt to do some actual benchmarks.

 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.0,addr=0x0f,addr=0x0f,bus=pci.0,x-vid=0x10DE,x-did=0x11BA,x-ss-vid=0x10DE,x-ss-did=0x0965,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=04:00.1,addr=0x10,bus=pci.0'/> </qemu:commandline>

Now for some reason, my Start Menu doesn't work, nor does Edge. It might just be a downlevel version of OVMF, libvirt, or qemu.

Edit 2: I was able to get my Start Menu and Edge to work with a reset of Windows itself.

Last edited by The_Moves (2015-07-23 23:15:38)

Offline

#5713 2015-07-23 09:02:20

jack_boss
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I need help; before I go buy a KM switch which are not cheap considering I have to import it.. etc.

I need to passthrough a USB Host controller for mouse and keyboard, becasue if I want to pass through a USB Host device it has to be attached. I went with the following and it has its own iommu group:

USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0

lspci -n -s 00:1d gave me 8086:8ca6 which I added to my kernel command line.
The other vendor:product id's are for my GTX770

options root=UUID=1aea1286-102b-444d-9f30-494305255937 rw elevator=noop intel_iommu=on iommu=pt pci-stub.ids=10de:1184,10de:0e0a,8086:8ca6 hugepages=2048

A USB Controller was already present (I've tried USB2 and USB3, same results)
Finally in virt-manager I clicked Add Hardware >PCI Host Device> USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

In the VM (Windows) I can see drivers being loaded and installed. The thing is, it doesnt matter in which port I plug in my mouse (removed from host/tried wireless usb mouse dongle, pen drive) nothing is listed or moves (just in case i tried every port)

Please help.

Just if somebody needs to see this:

00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 Subsystem: ASRock Incorporation Device 8ca6 Kernel driver in use: vfio-pci Kernel modules: ehci_pci

lsmod | grep -E "virt|kvm|iommu"


vfio_iommu_type1 20480 1 vfio 24576 8 vfio_iommu_type1,vfio_pci kvm_intel 155648 4 kvm 442368 1 kvm_intel virtio_blk 20480 0 virtio_scsi 20480 0 scsi_mod 151552 5 uas,usb_storage,libata,sd_mod,virtio_scsi virtio_net 32768 0 virtio_ring 20480 3 virtio_blk,virtio_net,virtio_scsi virtio 16384 3 virtio_blk,virtio_net,virtio_scsi

Offline

#5714 2015-07-23 10:21:51

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jack_boss wrote:

Hello, I need help; before I go buy a KM switch which are not cheap considering I have to import it.. etc.

I need to passthrough a USB Host controller for mouse and keyboard, becasue if I want to pass through a USB Host device it has to be attached. I went with the following and it has its own iommu group:

USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0

lspci -n -s 00:1d gave me 8086:8ca6 which I added to my kernel command line.
The other vendor:product id's are for my GTX770

options root=UUID=1aea1286-102b-444d-9f30-494305255937 rw elevator=noop intel_iommu=on iommu=pt pci-stub.ids=10de:1184,10de:0e0a,8086:8ca6 hugepages=2048

A USB Controller was already present (I've tried USB2 and USB3, same results)
Finally in virt-manager I clicked Add Hardware >PCI Host Device> USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

In the VM (Windows) I can see drivers being loaded and installed. The thing is, it doesnt matter in which port I plug in my mouse (removed from host/tried wireless usb mouse dongle, pen drive) nothing is listed or moves (just in case i tried every port)

Please help.

Just if somebody needs to see this:

00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 Subsystem: ASRock Incorporation Device 8ca6 Kernel driver in use: vfio-pci Kernel modules: ehci_pci

lsmod | grep -E "virt|kvm|iommu"


vfio_iommu_type1 20480 1 vfio 24576 8 vfio_iommu_type1,vfio_pci kvm_intel 155648 4 kvm 442368 1 kvm_intel virtio_blk 20480 0 virtio_scsi 20480 0 scsi_mod 151552 5 uas,usb_storage,libata,sd_mod,virtio_scsi virtio_net 32768 0 virtio_ring 20480 3 virtio_blk,virtio_net,virtio_scsi virtio 16384 3 virtio_blk,virtio_net,virtio_scsi

I think that on some Intel chipsets, the kernel maps all your physical USB ports behind the Intel xHCI USB controller by default and leaves the EHCI ones unused. http://lxr.free-electrons.com/source/dr … rks.c#L850. I worked around it by manually changing the Port Routing Mask Register to map physical ports behind the EHCI controllers. The register can simply be changed on the fly.

Offline

#5715 2015-07-23 11:01:48

jack_boss
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think that on some Intel chipsets, the kernel maps all your physical USB ports behind the Intel xHCI USB controller by default and leaves the EHCI ones unused. http://lxr.free-electrons.com/source/dr … rks.c#L850. I worked around it by manually changing the Port Routing Mask Register to map physical ports behind the EHCI controllers. The register can simply be changed on the fly.

Might be worth a try, if I just knew how to do it... first google attempt didn't really help either.

Offline

#5716 2015-07-23 12:17:21

fld
Member
Registered: 2015-02-06
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jack_boss wrote:

I think that on some Intel chipsets, the kernel maps all your physical USB ports behind the Intel xHCI USB controller by default and leaves the EHCI ones unused. http://lxr.free-electrons.com/source/dr … rks.c#L850. I worked around it by manually changing the Port Routing Mask Register to map physical ports behind the EHCI controllers. The register can simply be changed on the fly.

Might be worth a try, if I just knew how to do it... first google attempt didn't really help either.

You want to access the port routing register in the xhci controllers pci config space. Personally I ended up using "setpci -s 00:14.0 0xd0.W=2037" in my /etc/rc.local file on the Intel 8 Series PCH. Intel has pretty decent documentation available for the chipsets.

Offline

#5717 2015-07-23 20:37:09

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Someone 2 pages ago asked about APU and it's GPU. His questions remain unanswered ever since.
Well, he was right, but there's some caveats left:
1. disable hugepages. They trash everything.
2. do not try to p-t APU's GPU, p-t only a dedicated one. The integrated one is... integrated into the CPU, so there's a sophisticated power management system present, which doesn't work with only the GPU p-t'd.(i've learned that only by examining amd's BKDG, so no practice experience) As such, even though CrossFireX works, dual graphics will not.
3. x-vga generally doesn't work on these platforms. Use OVMF, which is quite unstable now.
4. some bandwidth issues present on some motherboards. For example, i am crazy enough to run four GPUs on an FM2 system(i have an athlon 750k), and the only pci-e slot routed fully 16x is the first one, the others are 8x(and if there are other devices present in 1x connectors, it works as x4).
5. this platform has good device isolation capabilities, no ACS patch needed. When stuffed with GPUs, there are three IOMMU groups: the first GPU, the second, and the other stuff you have on PCI bus, so you'll have to use xen's pci-assign to use those devices.
6. lspci shows that the integrated GPU is near the root, so there must be no problems with device isolation, you may use it as a host device.
7. the setup switch, AFAIK, routes only VGA. GOP(UEFI) gets mirrored to all available devices and GPUs, making that switch useless. Gigabyte motherboards can change the primary VGA device on any platform, which is a nice feature for our objectives.
8. you might want to try running proprietary drivers on your host GPU. There was some problems in the past, i haven't heard about them very long. Maybe Device ID stanza in xorg.conf helps to sort out that "autobinding" problem.

I hope i've answered most questions about FM2 platforms.
Conclusion: FM2 is not recommended for device pass-through, but you can do a dozen of work-arounds to make it work.


BTW: Win7+OVMF worked for me until july. OVMF whitepaper tells us to add an additional QXL emulated GPU to be able to see something until the GPU fires up properly. Win7 likes VBE, which is emulated only on QXL and not present on real UEFI systems.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5718 2015-07-23 23:14:37

JHeaton
Member
From: United Kingdom
Registered: 2009-05-16
Posts: 158

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all. So I set this up over the past couple of days, thanks in huge part to the posters in this thread and the various linked resources. I'm running two GTX 970 cards in my PC and passing them both through to the virtual machine; these are recognised by Windows, I have them showing in Device Manager, the latest nVidia drivers are installed, however I seem unable to use SLI; the nVidia control panel simply does not offer me the option, whereas normally in Windows it does so straight away. Has anyone so far managed to get SLI working using this kind of setup?

Thanks!

Offline

#5719 2015-07-24 01:10:21

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@JHeaton - I believe people have gotten CrossFire to work, but not that I remember reading. Have you tried setting it up to there the SLI doesn't use the bridge, but instead the interconnect through the PCIe bus? What motherboard are you running? What generation PCIe?

Offline

#5720 2015-07-24 01:13:01

JHeaton
Member
From: United Kingdom
Registered: 2009-05-16
Posts: 158

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

@JHeaton - I believe people have gotten CrossFire to work, but not that I remember reading. Have you tried setting it up to there the SLI doesn't use the bridge, but instead the interconnect through the PCIe bus? What motherboard are you running? What generation PCIe?

Thanks for the reply. I actually wasn't aware you could even do that, do you have information on how I would go about it? Motherboard is Asrock Z97 Extreme6 and PCIe 3.0.

Offline

#5721 2015-07-24 04:25:44

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've only heard about it. Don't actually utilize SLI.

Last edited by The_Moves (2015-07-24 04:26:29)

Offline

#5722 2015-07-24 14:02:20

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Get second mouse for host (wireless for example) and passthrough main one
or
passthrough your mouse to VM and use Synergy (host as client, VM as server).

Sadly none of this work for me because I have multiple VMs at the same time where all of them must have mouse working in games. Still trying to figure out solution, may be going to try add some non-standard mouse support into QEMU.

Also I'm managed to pass GTX 260 successfully, but only without "x-vga" and active "-vga qxl". It's end up that my issue with it was caused by fact that GTX 260 won't reset properly on VM reset/poweroff so it's remain in some sort of unusable state after first usage. So I bypass it by booting VM with emulated GPU and then just disable/enable GTX 260 via Device Manager: it's end up working just fine.

Now I have other question. I heard that virtio driver must support "read only" mode with filesystem changes written into separate file and QCOW must support that. What I want is to share disk with Steam games between two VMs so both may access it without breakage. Is there anyone who successfully used read-only mode / snapshots with Windows guests?

Last edited by SXX (2015-07-24 14:03:11)

Online

#5723 2015-07-24 14:26:14

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@SXX

It's not very read-only if the filesystem can be changed, is it?  qemu-img can create multiple qcow2 images, all using the same backing store file, so writes are added to each leaf image, not the base image.  I suspect you do not want a read-only image exposed to each VM, because games generally like to apply updates.  There's a QEMU -snapshot option which stores disk writes in memory and throws them away on VM shutdown, but I wouldn't recommend it for this case since any long running VM will quickly consume all your RAM.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5724 2015-07-24 14:34:39

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

JHeaton wrote:

Hi all. So I set this up over the past couple of days, thanks in huge part to the posters in this thread and the various linked resources. I'm running two GTX 970 cards in my PC and passing them both through to the virtual machine; these are recognised by Windows, I have them showing in Device Manager, the latest nVidia drivers are installed, however I seem unable to use SLI; the nVidia control panel simply does not offer me the option, whereas normally in Windows it does so straight away. Has anyone so far managed to get SLI working using this kind of setup?

Thanks!

I am the one with the crossfire working. My cards use XDMA crossfire(nvidia doesn't have that technology), which utilizes PCI-e bandwidth instead of an external bridge.
So. Nvidia are some tight asses, and prohibit SLI running on non-SLI certified motherboards.
There was a project called HyperSLI to enable SLI on any motherboard. Generally it requires the machine to have virtualization support(meaning nested virtualization for us), BUT i've red a story where some sick folks ran two GeForce 6600 in SLI on a P3 motherboard(obviously lacking any virtualization), however they had a performance drawback instead of a boost.
I remember someone tried to use a more modern version of HyperSLI, but he failed. You can try too, but i doubt that it will work.

I don't know about 2-chip(SLI on one PCB) solutions working in a virtual machine. Maybe they do work.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5725 2015-07-24 15:02:28

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First of all thanks for an answer! This is super helpful because all information I can find on internet it's RHEL bugtracker where I suppose everyone talking about Linux guests and bugs related to VMs that's filesystem aware drive is read-only.

aw wrote:

It's not very read-only if the filesystem can be changed, is it?

The point it's to have them read-only on host. Reason for this is fact I don't want to apply any changes to system drive because I don't really need any Windows or driver updates and image itself stored on small SSD in compressed state. When Windows using drive it's start to become bigger and bigger so I wish to keep system drive image completely unchanged until I decide otherwise.

aw wrote:

I suspect you do not want a read-only image exposed to each VM, because games generally like to apply updates.

It's won't be running like that all the time because I only need it when some friends come to play with me.
So when I need to apply updates I'll just run it like normal VM with writetable drives.

aw wrote:

qemu-img can create multiple qcow2 images, all using the same backing store file, so writes are added to each leaf image, not the base image.

It's not exactly what I need, but may be hanby for system drive. Is it possible to use that with existing QCOW image?
I already have Windows guest QCOW image so I want clone into two with little changes.

aw wrote:

There's a QEMU -snapshot option which stores disk writes in memory and throws them away on VM shutdown, but I wouldn't recommend it for this case since any long running VM will quickly consume all your RAM.

It's actually exactly what I need. Any idea if it's going to work for physical HDD partition?

-drive file=/dev/disk/by-uuid/0AD949C94647B394,format=raw,if=virtio

Also according to wiki "any changes made to the virtual machine while it is running are written to temporary". Would be interesting if I can write temporary changes on my HDD RAID.

Currently my setup looks like that:
- Host system and two Windows QCOW images (13GB each) on small SSD (60GB).
- RAID of two 250GB HDDs for data.
- 1TB HDD with NTFS partition for VM games
So I with it's possible to use that NTFS partition in both VMs because it's would take several days to move it into QCOW and not sure what performance would be.

Last edited by SXX (2015-07-24 15:23:47)

Online

#5726 2015-07-24 15:49:40

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@SXX

Yes, you can make new qcow2 images with your existing as a backing store.  I think it even works with raw images, so your disk might also work as well.  Also, don't come crying to me when -snapshot either uses all your ram or fills your tmpfs and starts giving write errors, I warned you.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5727 2015-07-24 16:17:55

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Thanks for information! I'll try to setup once I have time it and report back.
Don't think that write errors anything like an issue as long as it's only affect running VM and not actual disk image. :-)

PS: BTW to change where is snapshot going "TMPDIR=/home/tmp" can be used, by default it's write into /var/tmp.

Last edited by SXX (2015-07-24 16:35:02)

Online

#5728 2015-07-24 22:50:19

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So you were right @aw! It's working with raw drive partitions just fine and IO performance with QCOW2 image is better than with "-snapshot" (e.g Google Chrome on guest start to freeze with that option when downloading file). I'd put my current configuration on gist.

Sadly VM with Nvidia have inconsistent startup success rate (e.g sometimes only load after it's started once with Linux VM) and looks like it's may cause host freeze on restart or 2nd startup. Though it's may be easily hardware failure due to overheating or who know what else (GPU is 7 years old). Anyway I successfully run Unigine Heaven on host and both VMs simultaneously.

Thanks again! Now I just need to find out how to do CPU pinning without using libvirt.

Online

#5729 2015-07-25 07:44:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

Sadly VM with Nvidia have inconsistent startup success rate (e.g sometimes only load after it's started once with Linux VM) and looks like it's may cause host freeze on restart or 2nd startup. Though it's may be easily hardware failure due to overheating or who know what else (GPU is 7 years old). Anyway I successfully run Unigine Heaven on host and both VMs simultaneously.

It's too early to roar with success.

I've had that sort of problem when i ran SeaBIOS on an AMD platform - it worked just sometimes. Try disabling hugepages maybe?

Last edited by Duelist (2015-07-25 09:07:09)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5730 2015-07-25 12:31:23

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Try disabling hugepages maybe?

I don't use huge pages because those never improved VM performance a lot for my use cases. BTW I only applied ACS because never needed any other patches for one GPU passthrough, can this be reason of my issues with Nvidia?

UPD: Found a way to have consistent startup success:

  1. Run Kubuntu livecd with nouveau till display activated.

  2. Now shutdown it or just kill QEMU.
    Display remain active with some image freezed on it.

  3. Boot into Windows with QXL.

  4. Go to Control Panel -> Device Manager.

  5. Disable and enable device.
    At that point cursor on freeze image goes into left top corner of screen and monitor lose signal.
    Though Windows still don't see monitor, GPU not working.

  6. Now disable and enable it once again.
    Monitor connected to Nvidia now active! It's working!

Though there is still weird issue with fan speed control. E.g if Nouveaus started with high speed fan then even after driver become working Windows VM don't decrease fan speed. Though if started with low speed than VM can control speed just fine.

UDP2: Looks like there issue with clocks too, sadly. Though one more disable / enable cycle fixes it.

Last edited by SXX (2015-07-25 15:59:51)

Online

#5731 2015-07-26 10:01:56

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The latest OVMF dated today works for me with Windows 8.1 again.

Edit: not after all. My existing install booted up but as I was doing some experiments with a new test install, the installer still crashed when it started copying files.

Last edited by impulse_255 (2015-07-26 14:50:06)

Offline

#5732 2015-07-26 12:47:29

Denso
Member
Registered: 2014-08-30
Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@impulse_255 :

I approve that I can finally install Windows 10 . The code was compiled 30 minutes ago .

Offline

#5733 2015-07-26 17:54:26

kantras
Member
Registered: 2013-06-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using the ovmf-git AUR package, I was just able to install Windows 8 Pro earlier this morning - new install, compiled from about 6 hours ago

Last edited by kantras (2015-07-26 17:55:10)

Offline

#5734 2015-07-26 18:09:53

devianceluka
Member
Registered: 2014-05-19
Posts: 43

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OVMF boot from physical SATA disk possible yet (SATA controller passthrough)?

Offline

#5735 2015-07-26 21:08:22

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

With the latest OVMF builds, i am unable to install windows 7.
The installer successfully does it's job, reboots the VM and the system freezes without the logo showing with 4 cores on 100%.
When force resetting the machine, next boot shows me a loading logo which freezes in the middle.
I am not passing through any devices, the only GPU is the emulated QXL...
Seems like it's time to sing along to that song, where "all men play on ten".

There's a message for us.
Anyone had same experience?

Last edited by Duelist (2015-07-26 21:43:45)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5736 2015-07-26 23:06:29

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

BTW, I reported the Win8.1/OVMF install issue to Laszlo who bisected and found the commit that introduced it.  There's already a patch on their list to fix it, but the ongoing sorceforge outage is delaying it being committed and new builds including it.  Stick with a pre-July build until things get moving again.

Right.

Here's some more background.

The bug breaks the UEFI variable runtime services when going from physical addressing to virtual addressing, at SetVirtualAddressMap(). The variable service drivers were rewritten recently (there had been two separate variable drivers, one for secure boot enabled platforms, and another for secure boot-less platforms -- those were unified now). During this, commits a6811666 (SVN 17758) and fa0737a8 (SVN 17765) introduced the bug. Date: 2015-Jul-01.

The rest of the timeline (mostly in UTC+02:00, ie. CEST):

  • July 16th (approx): sourceforge.net blows up. The most recent commit in our SVN repo at this point is 18032. The last SVN commit shoveled over into our git mirror corresponds to SVN rev 18030. (The mirroring daemon runs periodically.)

  • July 21st: Alex reports the installation regression to me at 1 AM approx. Patches for the bug are (independently) posted to the list at 11 AM. http://thread.gmane.org/gmane.comp.bios.edk2.devel/133 . At 2-3 PM I read Alex's  email, bisect the bug, and recall the patches from the list I saw just a few hours earlier. I test them, and they work. The patches cannot be committed because SVN is down.

  • July 23rd: SVN has been down for a week. We cannot exist without a repo any longer, not just due to the above bug(fixes), but also because other patches have been accumulating. Jordan sets up a temporary git repo (a fork of our well-known svn-to-git mirror) on github, so that we can queue up the ready-to-merge (reviewed) patches in the SVN interim. See http://thread.gmane.org/gmane.comp.bios.edk2.devel/227 . First of all, Jordan queues the two patches from the list that made it to SVN (as revs 18031 and 18032) but never to the git mirror.

  • July 23rd still: I round up the bulk of pending reviewed patches for Jordan in a personal branch, and Jordan merges them to the temporary git repo (fast forward merge), on top of 18032. These new patches include my work that ports the SMBIOS client code to ARM, and refactor the original x86 code a bit. They also include the variable driver bugfix noted above.

  • July 25th: Jordan goes on and picks up three more patches from the list that have become ready for merging

  • July 26th (today -- well, it's past midnight, so "yesterday"): the SVN repo comes back from the dead. Surprise: the most recent revision it knows about is 18014. So, in a hurry, Jordan commits patches from the git repo back to SVN (revisions 18015..18030), which thereby qualifies as restoration from backup, and commits 18031..18032 from the tempoary queue (which is still restoral from backup, theoretically).

  • July 26th still: because SVN is back, with new patches no less, SVN revisions 18031..18032 are mirrored to our semi-official git mirror. Then Gerd's Jenkins daemon performs a git pull (at the commit corresponding to SVN rev 18032: b59e5ad), and successfully builds build#1121. This is what you guys have now, and it contains the bug.

  • July 26th still: Jordan flushes the 31 new queued patches to SVN. The SVN revision number we thus reach is 18063. The temporary git queue is practically discontinued. The new SVN commits percolate to the git mirror as well. The bugfix you guys would need corresponds to SVN revs 18054 (git e43525ee) and 18055 (git f18b2162).

  • July 26th still: Jenkins build 1122 (which would correspond to SVN rev 18063 / git 88656abf1b) fails, beause my SMBIOS refactoring (SVN r18040 / git 164cf403) conflicts with some downstream-only patches we have in Gerd's build. I get an automatic email about the build failure, but I'm not surprised. Namely, I keep those same downstream patches in my local tree, so I have already adapted them. I send them to Gerd so he can resume the Jenkins build. Unfortunately, Gerd seems to have entered a phase of "not reading email for a while" already -- my email misses him and the build remains stuck at build#1121 / SVN rev 18032 / git b59e5ad, leaving you guys without the necessary fix.

Offline

#5737 2015-07-26 23:18:23

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Until the Jenkins build gets sorted out, you guys can try this build: http://people.redhat.com/~lersek/ovmf-r … -4530eb75/ . It is based on 18063 and has a bunch of local patches (my usual env setup and some of my work in progress, but nothing that should interfere with your use case). No RPM, just raw binaries, but you all ought to know how to point the domain XML / libvirt's qemu.conf to them at this point. smile

Offline

#5738 2015-07-26 23:20:49

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:

Here's some more background.

Whoa, what a mess you've had there.
So, I guess, the conclusion for the majority of us is to wait until Gerd Hoffman wakes up and resumes the auto builds, right?
That would be one or two days, I think, not a problem.
I'm kind of surprised that you had the patch ready in 11 hours, but couldn't deliver it for almost a week due to sourceforge storming.It's still storming, right?

UPD:

 <loader readonly='yes' type='pflash'>/mnt/hdd/qemu/OVMF_CODE.fd</loader> <nvram>/mnt/hdd/qemu/OVMF_VARS.fd</nvram>


I guess that would be the lines to use the files you've provided, but i'm having some troubles with permissions...

Last edited by Duelist (2015-07-26 23:50:25)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5739 2015-07-26 23:41:37

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

I'm kind of surprised that you had the patch ready in 11 hours, but couldn't deliver it for almost a week due to sourceforge storming.It's still storming, right?

For us at least, it's okay, I'd think. http://sourceforge.net/blog/

Duelist wrote:

UPD:

 <loader readonly='yes' type='pflash'>/mnt/hdd/qemu/OVMF_CODE.fd</loader> <nvram>/mnt/hdd/qemu/OVMF_VARS.fd</nvram>

I guess that would be the lines to use the files you've provided, but i'm having some troubles with permissions...

First, the above XML fragment is not entirely correct. The loader element looks okay, but the nvram element should not be updated -- you should continue using the virtual machine's private varstore. (The OVMF_CODE.fd binary that I provided should be compatible with it.) ... Actually, the loader element is the only one you have to care about, if you already have a prior VM. If you insist, the varstore *template* (which OVMF_VARS.fd is) can be specified in the "template" attribute of the nvram element.

Libvirt's qemu.conf should be modified, and libvirt restarted, if you intend to create new VMs with the files I provided. That config file is where you stick them both. So: loader is okay, and revert nvram to what it used to be earlier.

Then, regarding permissions -- qemu runs with unprivileged UID and GID, plus in an SELinux "sandbox", if you run Fedora / RHEL and launch qemu from libvirt. So for now the easiest would be probably if you just copied these new files under

/usr/share/edk2.git/ovmf-x64

(which is a directory that belongs to Gerd's edk2.git-ovmf-x64 package), and then ran:

cd /usr/share/edk2.git/ovmf-x64 chown root:root OVMF_*.fd chmod 644 OVMF_*.fd restorecon -FvvR .

Offline

#5740 2015-07-26 23:47:15

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
Duelist wrote:

I'm kind of surprised that you had the patch ready in 11 hours, but couldn't deliver it for almost a week due to sourceforge storming.It's still storming, right?

For us at least, it's okay, I'd think. http://sourceforge.net/blog/

Duelist wrote:

UPD:

 <loader readonly='yes' type='pflash'>/mnt/hdd/qemu/OVMF_CODE.fd</loader> <nvram>/mnt/hdd/qemu/OVMF_VARS.fd</nvram>

I guess that would be the lines to use the files you've provided, but i'm having some troubles with permissions...

First, the above XML fragment is not entirely correct. The loader element looks okay, but the nvram element should not be updated -- you should continue using the virtual machine's private varstore. (The OVMF_CODE.fd binary that I provided should be compatible with it.) ... Actually, the loader element is the only one you have to care about, if you already have a prior VM. If you insist, the varstore *template* (which OVMF_VARS.fd is) can be specified in the "template" attribute of the nvram element.

Libvirt's qemu.conf should be modified, and libvirt restarted, if you intend to create new VMs with the files I provided. That config file is where you stick them both. So: loader is okay, and revert nvram to what it used to be earlier.

Then, regarding permissions -- qemu runs with unprivileged UID and GID, plus in an SELinux "sandbox", if you run Fedora / RHEL and launch qemu from libvirt. So for now the easiest would be probably if you just copied these new files under

/usr/share/edk2.git/ovmf-x64

(which is a directory that belongs to Gerd's edk2.git-ovmf-x64 package), and then ran:

cd /usr/share/edk2.git/ovmf-x64 chown root:root OVMF_*.fd chmod 644 OVMF_*.fd restorecon -FvvR .

You are right in every sentence, i've concluded the same. VARS are not needed, it's easiest to just copy the files over, which i did.

However, that didn't resolve my windows7 installation freeze issue.(win 10 release is in two or three days, so i don't care much) I don't know for others. The time(and they) will tell.


BTW, it's awesome to read SF's blog with all those "system_xxx online" posts almost every day. That voice in my head keeps saying stuff like "Battlecruiser operational".

Last edited by Duelist (2015-07-26 23:52:35)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5741 2015-07-27 01:12:06

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So as I expected problem with Nvidia GTX 260 VM startup was caused by VGA arbitration and it's caused KVM error with x-vga=on too. It's still really weird because AMD GPU only had temporary gamma corruption issue, but looks like Nvidia Windows driver fail to take control over GPU when it's not initialized on boot properly.

Would be great if that eventually will be solved without EFI needed (GTX 260 doesn't support it), but for now it's will work for me as when I need two VMs I don't really care about 3D acceleration on host. I'll just reboot with different options when I need two VMs or just wouldn't worry about fan noise. Thanks to everyone who helped me!

Online

#5742 2015-07-27 01:39:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@SXX

Maybe time for new hardware?  GTX 260 may have been pretty sweet 6 years ago... just sayin'


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5743 2015-07-27 02:54:04

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No doubt about that. Though this is just what I got for free because I don't really want to buy any Nvidia products. Unfortunately still need it for humble open source game development and testing.

Later on if AMD finally going to drop Catalyst kernel blob I'll buy Fury X or something.

...

BTW is there anyone using Windows with access time update disabled? Any downside? I just think it's should decrease QCOW2 growth quite a lot. Oops, looks like windows don't enable it by default.

Last edited by SXX (2015-07-27 04:36:52)

Online

#5744 2015-07-27 09:58:10

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

However, that didn't resolve my windows7 installation freeze issue.(win 10 release is in two or three days, so i don't care much) I don't know for others. The time(and they) will tell.

Can you link an earlier comment of yours about this issue, with more details? (I assume you described it earlier.)

Also, Gerd has incorporated my updated downstream-only smbios patches, and the Jenkins build is back to normal -- I just got an email about build # 1125 succeeding; I believe the RPMs too should be soon refreshed at the usual location.

Offline

#5745 2015-07-27 10:41:44

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
Duelist wrote:

However, that didn't resolve my windows7 installation freeze issue.(win 10 release is in two or three days, so i don't care much) I don't know for others. The time(and they) will tell.

Can you link an earlier comment of yours about this issue, with more details? (I assume you described it earlier.)

Also, Gerd has incorporated my updated downstream-only smbios patches, and the Jenkins build is back to normal -- I just got an email about build # 1125 succeeding; I believe the RPMs too should be soon refreshed at the usual location.

Strange, it's still B1121...
11:50 UTC - B1125 is in place.

...
Ouch it hurts.
Is there a legitimate way of attaching an ISA debug console to a libvirt domain(VM)? I tried to pass the regular options as qemu:commandline, but it says that there's no such option...
Googling that problem leads me to my own messages about that debug feature.

Last edited by Duelist (2015-07-27 15:10:19)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5746 2015-07-27 17:31:51

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone know if it's possible to boot (with gpu passthrough of course) only using coreboot for linux guests and getting rid of both seabios and ovmf?
One of the reasons I'm asking is because ovmf loads really slow.

Offline

#5747 2015-07-27 17:45:41

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:

Does anyone know if it's possible to boot (with gpu passthrough of course) only using coreboot for linux guests and getting rid of both seabios and ovmf?
One of the reasons I'm asking is because ovmf loads really slow.

http://www.coreboot.org/QEMU

Offline

#5748 2015-07-27 18:50:09

winie
Member
Registered: 2014-03-01
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:
winie wrote:

Does anyone know if it's possible to boot (with gpu passthrough of course) only using coreboot for linux guests and getting rid of both seabios and ovmf?
One of the reasons I'm asking is because ovmf loads really slow.

http://www.coreboot.org/QEMU

I mean how to get it working with vga passthrough

Offline

#5749 2015-07-27 23:22:25

The_Moves
Member
Registered: 2015-01-06
Posts: 58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You might try posting on their IRC channel: http://www.coreboot.org/IRC

Offline

#5750 2015-07-27 23:38:48

zir_blazer
Member
Registered: 2013-12-12
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

winie wrote:
The_Moves wrote:
winie wrote:

Does anyone know if it's possible to boot (with gpu passthrough of course) only using coreboot for linux guests and getting rid of both seabios and ovmf?
One of the reasons I'm asking is because ovmf loads really slow.

http://www.coreboot.org/QEMU

I mean how to get it working with vga passthrough

You shouldn't even be able to do anything close to that, since Coreboot doesn't work in a standalone fashion. Its purpose is to initialize the Hardware components, THEN use SeaBIOS or OVMF for BIOS/UEFI support, which do the actual booting and presents the devices, ACPI, etc, to the OS. You can't really omit them.

A different matter is using Xen to make a paravirtualized VM. Since Linux is fully concious of the fact that it is a PV VM, these can skip having a Firmware. But that isn't reelevant to this Thread.

Last edited by zir_blazer (2015-07-27 23:39:30)

Offline

#5751 2015-07-28 04:39:39

jack_boss
Member
Registered: 2014-09-09
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fld wrote:
jack_boss wrote:

I think that on some Intel chipsets, the kernel maps all your physical USB ports behind the Intel xHCI USB controller by default and leaves the EHCI ones unused. http://lxr.free-electrons.com/source/dr … rks.c#L850. I worked around it by manually changing the Port Routing Mask Register to map physical ports behind the EHCI controllers. The register can simply be changed on the fly.

Might be worth a try, if I just knew how to do it... first google attempt didn't really help either.

You want to access the port routing register in the xhci controllers pci config space. Personally I ended up using "setpci -s 00:14.0 0xd0.W=2037" in my /etc/rc.local file on the Intel 8 Series PCH. Intel has pretty decent documentation available for the chipsets.

I understand your helping, but I don't seem to know what I'm looking for. The manpage for setpci didn't help either.
I dont know where to get the folloing missing credentials.

-s [[[[<domain>]:]<bus>]:][<slot>][.[<func>]]

If this is the controller in question which I dont want xHCI to handle:

00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

00 is the domain
1d is the bus
0 is slut
function ?

Motherboard: Z97 Extreme4

Last edited by jack_boss (2015-07-28 04:41:15)

Offline

#5752 2015-07-28 07:08:20

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Should normal PCI devices be able to be passed through just like PCI-express devices? I have a motherboard with 2 PCI slots behind a PCIe-to-PCI bridge, and am interested in passing through a PCI audio card (non PCIe) to a Windows 8.1 guest. On attempt of booting the guest with the PCI card passed, I get a BSOD with "DRIVER_IRQL_NOT_LESS_OR_EQUAL (vioscsi.sys), and with the card removed from startup command, the guest boots up completely fine with other PCIe devices passed through. Perhaps my setup or command is not sane?

exec qemu-system-x86_64 \ -machine type=pc,accel=kvm,kernel_irqchip=on \ -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \ -smp cores=2,threads=1,sockets=1 \ -m 4G \ -device vfio-pci,host=01:00.0,multifunction=on,romfile=XFX7870EFI.rom \ -device vfio-pci,host=01:00.1,multifunction=on \ -device vfio-pci,host=03:00.0 \ -device vfio-pci,host=05:00.0 \ -device vfio-pci,host=06:00.0 \ -device virtio-scsi-pci,id=scsi \ -device scsi-block,drive=winhdd \ -name Windows8.1 \ -drive file=OVMF_CODE.fd,if=pflash,format=raw,readonly \ -drive file=OVMF_VARS_win8.fd,if=pflash,format=raw \ -drive file=/dev/sdc,format=raw,if=none,cache=none,id=winhdd \ -display none \ -vga none \ -net none \ -serial none \ -parallel none \ -enable-kvm \ -daemonize \ -rtc base=localtime \ -nodefaults

If "-device vfio-pci,host=05:00.0 \" is removed from the command, guest boots successfully.

lspci:

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d0) 00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0) 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] 03:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 10) 04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41) 05:00.0 Multimedia audio controller: C-Media Electronics Inc CMI8738/CMI8768 PCI Audio (rev 10) 06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 12)

Thanks for any help!

Offline

#5753 2015-07-28 11:15:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jack_boss wrote:

If this is the controller in question which I dont want xHCI to handle:

00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1

00 is the domain
1d is the bus
0 is slut
function ?

Motherboard: Z97 Extreme4

...

[user@crossfire ~]$ ls /sys/bus/pci/devices/ 0000:00:00.0 0000:00:11.0 0000:00:13.2 0000:00:14.4 0000:00:16.0 0000:00:18.2 0000:01:00.0 0000:03:05.0 0000:06:00.0 0000:00:00.2 0000:00:12.0 0000:00:14.0 0000:00:15.0 0000:00:16.2 0000:00:18.3 0000:01:00.1 0000:04:00.0 0000:00:02.0 0000:00:12.2 0000:00:14.2 0000:00:15.1 0000:00:18.0 0000:00:18.4 0000:02:00.0 0000:04:00.1 0000:00:04.0 0000:00:13.0 0000:00:14.3 0000:00:15.2 0000:00:18.1 0000:00:18.5 0000:02:00.1 0000:05:00.0

Where:

0000: 01: 00. 1 DOMAIN:BUS:DEVICE.FUNCTION

As you are on the localhost, it's fine to assume that domain is always 0000, so this is optional, and you can specify only BDF combo without the domain.
Details. I haven't observed a large enough system to implement this.

mutiny wrote:

Should normal PCI devices be able to be passed through just like PCI-express devices? I have a motherboard with 2 PCI slots behind a PCIe-to-PCI bridge, and am interested in passing through a PCI audio card (non PCIe) to a Windows 8.1 guest. On attempt of booting the guest with the PCI card passed, I get a BSOD with "DRIVER_IRQL_NOT_LESS_OR_EQUAL (vioscsi.sys), and with the card removed from startup command, the guest boots up completely fine with other PCIe devices passed through. Perhaps my setup or command is not sane?

The main requirement for a device to be passed through is to have a valid IOMMU group. Period.
If there's a device you want in the same group with the devices you want to leave for the host - you'll have to p-t the whole group with all it's devices.
OR
use old, legacy, deprecated, xen pci-assign.
It works as vfio for regular PCI devices, and also doesn't require iommu groups to be valid. In fact, that's the only way i got my third GPU passed-through.

The problem you have usually happens when there's a malfunctioning windows driver. So the device is passed-through fine(check dmesg for errors btw), it's the drivers(or the windows kernel) who's glitching. As you can guess by the error message, check virtio-scsi driver(update it maybe?) and also check for the right IRQLs set for the sound card you've p-t'd. They really like interrupts.

I am passing my embedded on the motherboard

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 01)

via pci-assign and i haven't observed any problems so far.

And what's that "kernel_irqchip=on" you have there? You sure you need it and you know how it works?

P.S.
Does anyone know what the hell is that book i've linked under "details"?..

Last edited by Duelist (2015-07-28 11:44:54)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5754 2015-07-28 13:02:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mutiny

Conventional PCI devices often don't support or have broken support for interrupt masking.  You can try using the vfio-pci module option nointxmask=1 to mask it at the APIC rather than the device, but this imposes the requirement that the device uses an exclusive interrupt.  If you find the interrupt shared with other drivers, you'll need to unbind those conflicting devices from host drivers.  If you have multiple conventional PCI devices behind the same PCIe-to-PCI bridge, the IOMMU cannot distinguish between them and therefore they are necessarily grouped together.  The deprecated pci-assign driver (which has nothing to do with xen) will allow assigning such device separately, but it's much, much worse than ignoring the lack of isolation between PCIe devices and not recommended imho.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5755 2015-07-28 14:15:28

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Offline

#5756 2015-07-28 18:27:15

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:

Well, I ended up flashing the updated vbios from gigabyte, which has a valid UEFI rom signature. This worked fine and still runs as normal in my seabios setup on the same system, but I get the same issue as before when trying to install catalyst -- it gets up to installing the driver proper, the system momentarily freezes up and then reboots during installing the actual driver. I do get (low-resolution) output on the physical card up until the VM freezes and reboots.

I've pasted my vm config here. Is there anything I might be missing on the host side of the setup for ovmf to work?

Update: I've managed to get OVMF to work for my setup. For some reason, I had to remove a few settings from both my 'features' and 'clock' elements (see below) to get through driver installation, but I was able to add them back in afterwards with no apparent issues resulting. Not certain specifically which were causing the issue, and currently not too interested in trying more installations to confirm which it was.

Stuff removed from <features>:

 <apic/> <pae/>

Stuff removed from <clock>:

 <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/>

On another note, I also setup another vm using the same configuration but running Windows 10. This also worked, but there are multiple issues with the graphics there (strange output problems when going fullscreen, performance issues, other stuff). Given it works fine on Windows 8 and the Windows 10 drivers for my card are currently only a preview, it may very well just be a driver issue.

Offline

#5757 2015-07-28 19:33:06

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nazfellun wrote:

Stuff removed from <features>:

 <apic/> <pae/>

I bet on PAE. I hardly find a need for it on a x86_64 system.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5758 2015-07-28 20:50:49

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Nazfellun wrote:

Stuff removed from <features>:

 <apic/> <pae/>

I bet on PAE. I hardly find a need for it on a x86_64 system.

Probably. It was only there in the first place as libvirt adds it to my new VMs by default.

Offline

#5759 2015-07-28 22:14:50

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dumb question of the day:
Is there a tool in EDK2 which does the opposite of what EfiRom does? I mean, decompresses the EFI device driver from the ROM? I've found only a EFI shell program source.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5760 2015-07-29 15:28:23

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi, I also have a question: when I use OpenHardwareMonitor in my guest Windows 8.1 the Clocks section is empty. What can I do? Is that also the case for you guys?

here is a picture of what it looks like:


http://i.imgur.com/6rJMV7X.jpg

moderator edit [ewaller] Converted image tags for over sized image to url link.

Last edited by ewaller (2015-07-29 15:47:52)

Offline

#5761 2015-07-29 19:34:30

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Is there a legitimate way of attaching an ISA debug console to a libvirt domain(VM)? I tried to pass the regular options as qemu:commandline, but it says that there's no such option...
Googling that problem leads me to my own messages about that debug feature.

I'm sure you did the right thing -- almost. The bit that you probably missed is that the "commandline" and "arg" elements come from a different XML namespace than the rest of the elements in the domain XML. (They are defined by an XML Schema that targets the "http://libvirt.org/schemas/domain/qemu/1.0" namespace -- BTW do not click this link, it's just a string.)

Therefore, you have to declare that XML namespace with a namespace prefix, and qualify the "commandline" and "arg" elements with this namespace prefix. This namespace / prefix association can occur in any element that (recursively) contains these elements; or, even, you could just set the default namespace with attributes of the "arg" and "commandline" elements themselves (but that would take a lot more typing). See:

https://en.wikipedia.org/wiki/XML_names … eclaration

In practice though, the namespace is usually declared in the root element of the XML document.

Anyway, enough of the abstract discussion, here's what you need:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> ... stuff ... <qemu:commandline> <qemu:arg value='-global'/> <qemu:arg value='isa-debugcon.iobase=0x402'/> <qemu:arg value='-debugcon'/> <qemu:arg value='file:/tmp/GUEST_NAME.log'/> </qemu:commandline> </domain>

In the above,

qemu

is the namespace prefix being declared,

http://libvirt.org/schemas/domain/qemu/1.0

is the namespace URI being mapped to the

qemu

namespace prefix, and

qemu:commandline

and

qemu:arg

are the elements that you need.

(The "qemu" prefix is incidental, it could be replaced with any other prefix; the point is that the same prefix must be used in the namespace declaration and later for qualifying "commandline" and "arg". The "qemu" prefix is just a "mediator"; what matters is that "commandline" and "arg" come from the namespace "http://libvirt.org/schemas/domain/qemu/1.0".)

Offline

#5762 2015-07-29 19:38:14

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

hi, I also have a question: when I use OpenHardwareMonitor in my guest Windows 8.1 the Clocks section is empty. What can I do? Is that also the case for you guys?

here is a picture of what it looks like:


http://i.imgur.com/6rJMV7X.jpg

moderator edit [ewaller] Converted image tags for over sized image to url link.

It is almost impossible.
The data OpenHWMon shows you is from either PCI Config Space of some processor functions(00:18.3 for amd for example) or SMBus.
You can't p-t that processor functions since that'd most likely would break the host system, and you can't organise an SMBus access for the VM.
Moreover, why would you want to provide some low-level access to the hardware? What's the point of running a VM then?

If you want to overclock - do it either linux-way or run native windows to get stable settings.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5763 2015-07-29 20:14:24

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

all I want is that my front screen of my Chassis (LEET) shows me the temperature and Mhz of the CPU. For monitoring purposes. Nothing else. I didn't think it is this complicated. But I have to find a way, this is annoying, because it is the first thing you see when you look at my HTPC case.

Offline

#5764 2015-07-29 20:16:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

all I want is that my front screen of my Chassis (LEET) shows me the temperature and Mhz of the CPU. For monitoring purposes. Nothing else. I didn't think it is this complicated. But I have to find a way, this is annoying, because it is the first thing you see when you look at my HTPC case.

This is not a good candidate to be driven from inside the VM.  It's exceptionally complicated.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5765 2015-07-29 20:20:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

all I want is that my front screen of my Chassis (LEET) shows me the temperature and Mhz of the CPU. For monitoring purposes. Nothing else. I didn't think it is this complicated. But I have to find a way, this is annoying, because it is the first thing you see when you look at my HTPC case.

Drive it from the host system. Your VM and host system share the same CPU...
Oh, you want a device driver for your screen and it isn't in the kernel yet? Well...


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5766 2015-07-29 20:23:55

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok thanks guys for the clarification. I wish there'd be a way. At least I know I can't do anything about it and that every one of you have the same problem.

Offline

#5767 2015-07-29 20:26:24

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
4kGamer wrote:

all I want is that my front screen of my Chassis (LEET) shows me the temperature and Mhz of the CPU. For monitoring purposes. Nothing else. I didn't think it is this complicated. But I have to find a way, this is annoying, because it is the first thing you see when you look at my HTPC case.

Drive it from the host system. Your VM and host system share the same CPU...
Oh, you want a device driver for your screen and it isn't in the kernel yet? Well...

How can I do it from the host system???

Offline

#5768 2015-07-29 21:13:31

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:
Duelist wrote:
4kGamer wrote:

all I want is that my front screen of my Chassis (LEET) shows me the temperature and Mhz of the CPU. For monitoring purposes. Nothing else. I didn't think it is this complicated. But I have to find a way, this is annoying, because it is the first thing you see when you look at my HTPC case.

Drive it from the host system. Your VM and host system share the same CPU...
Oh, you want a device driver for your screen and it isn't in the kernel yet? Well...

How can I do it from the host system???

Depends on the display.
Even though it's beyond the scope of this thread, i'll try to help:
How does it work under windows? Is the screen plugged into USB?  Or maybe it's some other connection?
How does it look like? Can you link some photos maybe?

UPD:
http://ishankarve.blogspot.com/2010/10/ … 06-on.html
Futher googling shows that it is possible and should be rather simple. Have fun.

Last edited by Duelist (2015-07-29 21:19:43)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5769 2015-07-29 21:24:05

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems like it's just a USB screen, but detaching it from the VM, which I assume is running HTPC software, would remove some functionality.  I'd probably go the route of a plugin for the software that can monitor a remote system and a daemon in the host that provides data.  Anyway, this is all totally unrelated to this thread and should be moved elsewhere.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5770 2015-07-30 15:19:02

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, it is connected through internal usb. I will open up a new thread, just here a picture of what it currently looks like:
http://i.imgur.com/tRdqxAT.jpg

and this is how it should look: http://www.steigerdynamics.com/products … ngineering

I will try the linked tutorial, but I can see myself failing at it.

thanks anyways!

Offline

#5771 2015-07-30 18:54:43

fabrizziop
Member
Registered: 2015-07-30
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an FX-8350 + GA990FXA-UD3, with 270x as passthrough GPU and 5770 as linux GPU. I want to upgrade my 270x to something faster. Does the r9 290/390 or the r9 285/380 work correctly, without reset issues?. Thanks.

Offline

#5772 2015-07-30 20:42:19

Kartoffel
Member
Registered: 2015-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have a question regarding audio in QEMU.

When I run my script while I'm using KDE5 I get no audio issues.

sudo QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 qemu-system-x86_64 -enable-kvm -m 8G -cpu host,kvm=off \ -nographic \ -smp 4,sockets=1,cores=4,threads=1 \ -mem-path /dev/hugepages \ -device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 \ -vga none \ -soundhw hda \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk \ -net nic -net bridge,br=br0 \ -device vfio-pci,host=00:1d.0

When I run it while using a window manager such as awesome I get this error and subsequently a lot of crackling in Windows 7 x64:

ALSA lib pcm_dmix.c:1024:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DAC alsa: Failed to open `default': alsa: Reason: Device or resource busy ALSA lib pcm_dmix.c:1024:(snd_pcm_dmix_open) unable to open slave alsa: Could not initialize DACalsa: Failed to open `default': alsa: Reason: Device or resource busy audio: Failed to create voice `dac'

What could be the cause of this?
Any help is greatly appreciated.

Offline

#5773 2015-07-30 21:49:48

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kartoffel wrote:

Hi, I have a question regarding audio in QEMU.

When I run my script while I'm using KDE5 I get no audio issues.

sudo QEMU_AUDIO_DRV=alsa QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 qemu-system-x86_64 -enable-kvm -m 8G -cpu ...snip..

..cut..
What could be the cause of this?
Any help is greatly appreciated.

The lack of pulse audio, maybe?(AFAIR, it works on top of alsa) Or a different user running alsa?

fabrizziop wrote:

I have an FX-8350 + GA990FXA-UD3, with 270x as passthrough GPU and 5770 as linux GPU. I want to upgrade my 270x to something faster. Does the r9 290/390 or the r9 285/380 work correctly, without reset issues?. Thanks.

The list.

AFAIR:
1. Nobody tested AMD 3XX-series GPUs yet.
2. The reset issues were resolved by some kernel patch, which went upstream some months ago.

I'd go with the 390 or 380 or whatever which has HBM. The first serious technology advancement since GCN.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5774 2015-07-30 22:46:50

Kartoffel
Member
Registered: 2015-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

The lack of pulse audio, maybe?(AFAIR, it works on top of alsa) Or a different user running alsa?

Thanks for your help.
Running pulseaudio within awesome no longer displays the previous error, but now there's a new one.

alsa: Requested buffer size 512 was rejected, using 1881 alsa: Requested period size 170 was rejected, using 940

Again, this is something that does not happen within KDE5.
Thanks again.

Offline

#5775 2015-07-31 01:03:42

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kartoffel wrote:
Duelist wrote:

The lack of pulse audio, maybe?(AFAIR, it works on top of alsa) Or a different user running alsa?

Thanks for your help.
Running pulseaudio within awesome no longer displays the previous error, but now there's a new one.

alsa: Requested buffer size 512 was rejected, using 1881 alsa: Requested period size 170 was rejected, using 940

Again, this is something that does not happen within KDE5.
Thanks again.

Does it work that way? If so - ignore that lines. You can get rid of those lines by deleting

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170

And if you don't care much about using pulse audio or not - you can and should adapt QEMU to use it:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa

P.S.
I feel sad about advising people to use pulse audio, it's too complex and over-abstract for me and my uses.

Last edited by Duelist (2015-07-31 01:03:59)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5776 2015-07-31 06:37:54

Kartoffel
Member
Registered: 2015-07-30
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

Does it work that way? If so - ignore that lines. You can get rid of those lines by deleting

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170

When it rejects those settings the audio doesn't sound properly.

Duelist wrote:

And if you don't care much about using pulse audio or not - you can and should adapt QEMU to use it:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa

I have used those pa settings previously within KDE5 and they produced static quite frequently.
In addition, using those pa settings give an error within awesome:

pulseaudio: pa_context_connect() failed pulseaudio: Reason: connection refused pulseaudio: Failed to initialize PA contextaudio: Could not init 'pa' audio driver

From what I've gathered, it may be caused by running pa in single user mode and not system mode.
But because those settings give static, I believe there is no reason to pursue this further.
Any clue as to why it rejects the alsa settings?
Thanks.

Offline

#5777 2015-07-31 09:29:33

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fabrizziop wrote:

I have an FX-8350 + GA990FXA-UD3, with 270x as passthrough GPU and 5770 as linux GPU. I want to upgrade my 270x to something faster. Does the r9 290/390 or the r9 285/380 work correctly, without reset issues?. Thanks.

My Sapphire Tri-X R9 290X works fine with no issues at all (if only this card was a little bit smaller ...), with VM Windows 7 and regular Catalyst drivers. No gotchas, it just works. I am not using UEFI boot, so cannot say how this card would behave with OVMF. Also, I am using workstation class motherboard SuperMicro X9DA7, no idea how yours would work with this card.

I would advice against R9 285. I was unable to make FirePro W7100 work with pass-through and it uses the same chip Tonga PRO. Of course the reason for this could be specific to W7100 (ASIC or firmware), not to the chip. But on the other hand ...

Last edited by Bronek (2015-07-31 09:32:45)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5778 2015-07-31 10:00:38

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm not having much luck installing Windows7 Pro using OVMF at the moment.

I've got a fully working Windows8.1 Enterprise Machine with OVMF that I've been using for months and months, but I would like to try Windows10 as a free upgrade which you can't do from an enterprise version so I am currently trying to get Windows7 Pro working.

I'm using libvirt XML files to start the VM, I've assigned the QXL vga adapter, as well as my AMD Radeon R290 card, I can boot from the Windows7 ISO which I see on the QXL adapter via VNC that it shows "Windows is loading files" progress then it goes to "Starting Windows" and just crashes.

I can see that there have been issues reported with the OVMF builds so I've tried multiple builds:

The latest from https://www.kraxel.org/repos/jenkins/edk2/ (31st July)
15214
16229
17542
And finally the custom one from lersek based on 18063  http://people.redhat.com/~lersek/ovmf-r … -4530eb75/

All of them do the same, just appear blank/crashed after "Starting Windows" before you get to the installation screen.

I've also tried dropping in to the EFI Shell and manually starting the Windows7 EFI boot application as per https://technet.microsoft.com/en-gb/lib … s.10).aspx

Any one have any ideas?

Software Versions:

qemu-2-4-0-rc3
libvirt version: 1.2.15

Hardware:

Intel i7-4770s
Jetway NF9J-Q87
AMD Radeon R290

Here's my current XML file:

<domain type='kvm'> <name>windows7</name> <uuid>7b222825-fc7d-4a66-a72c-5875063752d2</uuid> <memory unit='KiB'>5291456</memory> <currentMemory unit='KiB'>5291456</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader readonly='yes' type='pflash'>/home/root/ovmf-builds/15214/OVMF.fd</loader> <boot dev='cdrom'/> </os> <features> <acpi/> <apic/> <pae/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'/> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/qemu-2-4-0-rc3/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' /> <source file='/virtualguests/windows7/windows7-c-drive.qcow2'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/root/virtualguestisos/windows7/Windows_7_64-bit_Professional_x64.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/root/virtualguestisos/virtio-isos/virtio-win-0.1-74.iso'/> <target dev='hdd' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root' /> <interface type='bridge'> <mac address='52:54:00:12:34:76'/> <source bridge='br0'/> <target dev='tap8'/> <model type='e1000'/> <alias name='virtio'/> <rom bar='off'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52e'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0719'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1415'/> <product id='0x2000'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0fcf'/> <product id='0x1009'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0' /> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> <graphics type='vnc' port='5903' autoport='no' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> </devices> </domain>

Thanks

Graham

Offline

#5779 2015-07-31 11:57:11

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've just managed to progress further but without using libvirt, just plain qemu command line. I can now see the Windows7 pro Installation Screen in VNC via QXL.

Have also gone back to 16229, as it looks like 15214 doesnt work for Windows7. http://ubuntuforums.org/showthread.php?t=2262351

I did try 16229 in libvirt but I couldn't get it working.

I'll post my progress after installation.

Command line used:

/usr/local/qemu-2-4-0-rc3/bin/qemu-system-x86_64 \ -enable-kvm \ -m 4096 \ -cpu host,kvm=off \ -drive if=pflash,format=raw,readonly,file=/home/root/ovmf-builds/16229/usr/share/ovmf/ovmf_x64.bin \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -drive file=/virtualguests/windows7/windows7-c-drive.qcow2,if=none,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 \ -drive file=/home/root/virtualguestisos/windows7/Windows_7_64-bit_Professional_x64.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw \ -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 \ -drive file=/home/root/virtualguestisos/virtio-isos/virtio-win-0.1-74.iso,if=none,id=drive-ide0-1-1,readonly=on,format=raw \ -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 \ -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x3 \ -device usb-host,hostbus=1,hostaddr=5,id=hostdev0 \ -device usb-host,hostbus=1,hostaddr=11,id=hostdev1 -device usb-host,hostbus=1,hostaddr=3,id=hostdev2 \ -device usb-host,hostbus=1,hostaddr=9,id=hostdev3 -device vfio-pci,host=01:00.0,id=hostdev4,bus=pci.0,multifunction=on,addr=0x8 -device vfio-pci,host=01:00.1,id=hostdev5,bus=pci.0,addr=0x9 \ -vnc 0.0.0.0:3

Last edited by gneville (2015-07-31 12:00:10)

Offline

#5780 2015-07-31 12:56:29

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Windows7 Pro Installed worked and finished.

After first reboot of installation it dropped back to EFI Shell, had to manually start from HDD:

FS1:
\EFI\Boot\bootx64.efi

After "Completing Installation" it again dropped back EFI Shell, again manually started EFI from HDD:

FS1:
\EFI\Boot\bootx64.efi

Now progressed in to setup of Username and computer name and Product Key and finally Win7 desktop

Downloaded AMD Catalyst 15.7.1 and installed and rebooted now got Windows7 desktop on AMD HDMI Port.

Offline

#5781 2015-07-31 13:04:22

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5782 2015-07-31 13:25:10

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Worth trying the OVMF files from here : https://www.archlinux.org/packages/extra/any/ovmf/

The issue I've got now is on each boot I have to manually start the EFI boot in the EFI Shell. Hoping that after I get to Upgrade Win7 Pro to Win10 Pro and then finally to a clean install I won't need to worry.

Duelist wrote:

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.

Offline

#5783 2015-07-31 15:42:45

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gneville wrote:

Worth trying the OVMF files from here : https://www.archlinux.org/packages/extra/any/ovmf/

The issue I've got now is on each boot I have to manually start the EFI boot in the EFI Shell. Hoping that after I get to Upgrade Win7 Pro to Win10 Pro and then finally to a clean install I won't need to worry.

Duelist wrote:

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.

Your issue is just a matter of configuration - specify the boot order, and you should be fine.

Although i do have pacman installed(i am one curious bastard), i use OVMF from git and i have fedora.
Could you tell me the build number which works for you?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5784 2015-07-31 19:14:57

mutiny
Member
Registered: 2015-02-15
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

The main requirement for a device to be passed through is to have a valid IOMMU group. Period.
If there's a device you want in the same group with the devices you want to leave for the host - you'll have to p-t the whole group with all it's devices.
OR
use old, legacy, deprecated, xen pci-assign.
It works as vfio for regular PCI devices, and also doesn't require iommu groups to be valid. In fact, that's the only way i got my third GPU passed-through.

The problem you have usually happens when there's a malfunctioning windows driver. So the device is passed-through fine(check dmesg for errors btw), it's the drivers(or the windows kernel) who's glitching. As you can guess by the error message, check virtio-scsi driver(update it maybe?) and also check for the right IRQLs set for the sound card you've p-t'd. They really like interrupts.

I am passing my embedded on the motherboard

00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 01)

via pci-assign and i haven't observed any problems so far.

And what's that "kernel_irqchip=on" you have there? You sure you need it and you know how it works?

Thank you for the response. It seems to be a Windows issues then. kernel-irqchip=on is enabled by default for qemu I believe, not sure why I explicitly added it but I don't think it makes any difference.

aw wrote:

@mutiny

Conventional PCI devices often don't support or have broken support for interrupt masking.  You can try using the vfio-pci module option nointxmask=1 to mask it at the APIC rather than the device, but this imposes the requirement that the device uses an exclusive interrupt.  If you find the interrupt shared with other drivers, you'll need to unbind those conflicting devices from host drivers.  If you have multiple conventional PCI devices behind the same PCIe-to-PCI bridge, the IOMMU cannot distinguish between them and therefore they are necessarily grouped together.  The deprecated pci-assign driver (which has nothing to do with xen) will allow assigning such device separately, but it's much, much worse than ignoring the lack of isolation between PCIe devices and not recommended imho.

Thank you very much aw. I will try with these options and see what happens. It is the only conventional PCI device I have installed at the moment, the other PCI slot is not populated.

Offline

#5785 2015-07-31 20:23:18

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist
The build - ovmf 16229-1 - is directly here: https://www.archlinux.org/packages/extra/any/ovmf/


I've now successfully upgraded the Win7 Pro Guest to Windows10 Pro and all works with OVMF. I've grabbed my new Win10 Pro product key and I've successfully performed a clean build of Windows10 Pro using OVFM as well. AMD Catalyst 15.7.1 installed and working fine with Win10. No reset issues either.



Duelist wrote:
gneville wrote:

Worth trying the OVMF files from here : https://www.archlinux.org/packages/extra/any/ovmf/

The issue I've got now is on each boot I have to manually start the EFI boot in the EFI Shell. Hoping that after I get to Upgrade Win7 Pro to Win10 Pro and then finally to a clean install I won't need to worry.

Duelist wrote:

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.

Your issue is just a matter of configuration - specify the boot order, and you should be fine.

Although i do have pacman installed(i am one curious bastard), i use OVMF from git and i have fedora.
Could you tell me the build number which works for you?

Offline

#5786 2015-07-31 23:45:28

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can't stop the progress...

Oh well. The time has come.
I have successfully installed Windows 10.
And damn it kind of frightened me when it didn't finish downloading the catalyst driver setup, but already installed the drivers.
Creepy stuff.
There's a good thing in all this modern hell: the screensaver settings menu remains the same for the damn 15 years. Yeah, that one where you cohoose the screensaver, with the funky drawn CRT-screen on it. It's the same as in Windows XP or W2K. I remember what a huge progress it was when that CRT screen changed it's design compared to 9x systems...

Anyway, Catalyst installed, crossfire launched, OVMF works. QXL is not needed.
AMD site hints me that my cards support DirectX 12.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5787 2015-08-01 12:50:47

dragonreborn
Member
Registered: 2014-08-11
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all,

I really need your help.
For months everything has been worked perfectly with VGA passthrough.
I have made a new Gentoo install as well and could enable the VGA passthrough there too.
Then something must have happened, after a few weeks of no-gaming, i tried to play some games, but after around 10 mins of gaming it became laggier and laggier, after one minute i got blank screen, and the following error message in terminal:

qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

If i try to launch qemu again, ill get the following:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: Error: Failed to setup INTx fd: Device or resource busy qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

dmesg:

[ 3933.246342] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (ehci_hcd:usb1) [ 3933.267313] vfio-pci 0000:01:00.0: Refused to change power state, currently in D3 [ 3933.977098] vfio-pci 0000:01:00.0: timed out waiting for pending transaction; performing function level reset anyway

qemu command:

sudo qemu-system-x86_64 -enable-kvm -m 8196 -cpu host,kvm=off -smp 4 -device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 -vga none -hda /mnt/harddrives/ssd/win8.qcow2 -usb -usbdevice host:1532:0032 -usbdevice host:1532:0109 -soundhw hda

I need to reboot the PC, then i can use qemu again (for 10 mins).
Noone seem to share this error with me on google.

Since i had backup from old Arch installation, where everything worked perfectly, i have restored everything, problem still persists.
I tried to downgrade apps, new install, fiddle with kernel, still no luck.

Getting desperate here, from time to time i like to play but dont want to switch back to windows ><
Please hep me if u can, and let me know if u need some more infos.
Thanks in advance.

Last edited by dragonreborn (2015-08-01 12:51:40)

Offline

#5788 2015-08-01 14:43:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dragonreborn

There's a disable_idle_d3=1 option to vfio-pci in v4.1 that you might want to try.  If D3 state is this broken for your device, it should probably be blacklisted in the kernel.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5789 2015-08-01 16:25:06

dragonreborn
Member
Registered: 2014-08-11
Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@dragonreborn

There's a disable_idle_d3=1 option to vfio-pci in v4.1 that you might want to try.  If D3 state is this broken for your device, it should probably be blacklisted in the kernel.


My bad.
Naturally after a couple of weeks trying to figure it out, and then writing on this forum, I figured it out.
It's most probably a hardware fault, fans aren't working on vga at all. Caused the card to overheat and stop working in 10 minutes.
Tried out the 2nd bios as well, no luck, no spin up at boot either.
I'll try a native windows out with the Evga software, otherwise back to RMA. Thanks for the help nevertheless!

Offline

#5790 2015-08-01 21:27:28

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.

Just two quick notes.

First, I verified Windows 7 right now. I built OVMF from edk2 SVN r18122 (git 7669f734). Both the installer ISO ("en_windows_7_professional_with_sp1_x64_dvd_u_676939.iso") and my preexistent guest boot to the GUI just fine, using QXL.

Second, I don't understand why you said "i failed to attach the debug console" again. I explained in excruciating detail how you should edit the domain XML, here: https://bbs.archlinux.org/viewtopic.php … 2#p1548842 . You didn't react to that comment, so I guess you either missed it, or didn't read it far enough for the actual XML snippet.

Offline

#5791 2015-08-01 21:37:27

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey Guys!

I have been trying to get GPU passthrough to work, but cannot seem to find out where the problem lies, since I do not get any errors.

I have primarily used these guides as sources:
http://ubuntuforums.org/showthread.php?t=2262280
https://www.pugetsystems.com/labs/artic … 4-KVM-585/
https://bbs.archlinux.org/viewtopic.php?id=162768

Specs:
Lubuntu 14.04 with patched 3.18.0 kernel
GTX 750TI
i5 4690K (VT-d enabled)
Gigabyte Z97 D3H
TV with multiple outputs (VGA, HDMI)

The IGPU runs over VGA, whereas the GPU is connected via HDMI.
I use the IGPU for the host system, while attempting to pass the GTX 750TI through to the Guest.
Both are connected to one single TV.
When I run the script, a qemu window pops up saying "compat_monitor0 console Qemu 2.1.2 monitor - type help
When I switch to the HDMI output of the TV, to which the CPU is connected, nothing appears. However, my GPU usage ramps up to 25%, hence something must be going on.
http://s18.postimg.org/9u9hg6v61/qemu.png

When I assign one of the downloaded gpu roms in the script, the colors become fuzzy and inverted (which cannot be seen on the screen), while nothing appears on the HDMI output and the cpu usage only momentarily rises, while dropping shortly after:
http://postimg.org/image/4txnlf85x/

Prepwork:
.)I compiled, patched and installed kernel 3.18.0 with ACS and i915: http://ubuntuforums.org/showthread.php?t=2262280
.)Downloaded a rom file for my GPU (GTX 750 TI): http://www.techpowerup.com/vgabios/inde … =&memSize=
.)Upgraded Qemu to version 2.1.2 via this ppa:jacob/virtualisation
.)Installed all necessary packages as suggested by this guide: https://help.ubuntu.com/community/KVM/Installation


1.)Grub

GRUB_DEFAULT=0 GRUB_HIDDEN_TIMEOUT=0 GRUB_HIDDEN_TIMEOUT_QUIET=true GRUB_TIMEOUT=10 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream i915.enable_hd_vgaarb=1" GRUB_CMDLINE_LINUX=""

2.)Nvidia IDs

lspci -nn | grep NVIDIA 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2) 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)

3.)/etc/initramfs-tools/modules

pci_stub ids=10de:1380,10de:0fbc

4.)/etc/vfio-pci.cfg

0000:01:00.0 0000:01:00.1

5.)Here my script (Here without a rom file assigned, this can be seen in the screen above)

#!/bin/bash configfile=/etc/vfio-pci.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=1 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/home/jackson/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -drive file=/home/jackson/Downloads/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \ -boot menu=on exit 0 

Please help me find out what's wrong.

Thank you!

Last edited by morphological (2015-08-02 06:52:18)

Offline

#5792 2015-08-01 21:39:25

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5793 2015-08-02 05:57:07

wmarler
Member
From: Denver, CO
Registered: 2015-04-18
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morphological

I have no experience with getting VGA passthrough working on Lubuntu, but I did get it working with both Fedora 22 and Arch, with both Windows 7 and Windows 10 technical preview, with the same video card & very similar GPU (i5-4690. "K" versions usually do not support VT-d but according to ARK yours does, so that's good). You should use the UEFI method and do not patch your kernel with the i915 VGA arbiter patch; your video card supports it. Also, you do not need to download a ROM for the card. Be advised, you will need to use this version of the OVMF binaries. Without it, the install will start but fail (with no messages... very maddening).

If you have access to/a license for Windows 10, I recommend using it over 7. For me at least, it performed much better. I can't speak to Windows 8/8.1.

You didn't mention using Alex Williamson's blog as a guide. You should read it. This forum is way too long, and the blog condenses what you need to know down into a few good posts. I linked you at part 3; you should read & follow it and part 4. The comments are helpful too; there are not many to sift through.

Offline

#5794 2015-08-02 09:20:30

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can confirm also that edk2 SVN r18123, which I got from here https://www.kraxel.org/repos/jenkins/edk2/ , works for me when Installing and Running Windows7

However I can only get it working still via qemu command line.

@lersek have you managed to get Windows7 working via libvirt? If so could you provide a copy of your XML file?

Edit: I've managed to get it working now with libvirt, I had to take away some of the hyper-v settings:


 <features> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <clock> <timer name='hypervclock' present='yes'/> </clock>

Edit2: Speifying the nvram file, got me past the issue where I'd always have to manually boot the UEFI bootloader e.g. FS1: \EFI\Boot\bootx64.fd. I had to change the boot order in the Tiano Core BIOS and have it.

Here's my XML file in case anyone wants to refer to it:

<domain type='kvm'> <name>windows7</name> <uuid>7b222825-fc7d-4a66-a72c-5875063752d2</uuid> <memory unit='KiB'>5291456</memory> <currentMemory unit='KiB'>5291456</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <loader type='pflash' readonly='yes'>/home/root/ovmf-builds/ovmf-jenkins/ovmf-x64-02082015/usr/share/edk2.git/ovmf-x64-18123/OVMF-pure-efi.fd</loader> <nvram>/home/root/ovmf-builds/ovmf-jenkins/ovmf-x64-02082015/usr/share/edk2.git/ovmf-x64-18123/OVMF_VARS-pure-efi.fd</nvram> <boot dev='cdrom'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'/> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/local/qemu-2-4-0-rc3/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' /> <source file='/virtualguests/windows7/windows7-c-drive.qcow2'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/root/virtualguestisos/windows7/Windows_7_64-bit_Professional_x64.iso'/> <target dev='hdc' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/root/virtualguestisos/virtio-isos/virtio-win-0.1-74.iso'/> <target dev='hdd' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='1' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root' /> <interface type='bridge'> <mac address='52:54:00:12:34:76'/> <source bridge='br0'/> <target dev='tap8'/> <model type='e1000'/> <alias name='virtio'/> <rom bar='off'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52e'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0719'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0fcf'/> <product id='0x1009'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0' /> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> <graphics type='vnc' port='5903' autoport='no' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> </video> </devices> </domain>

Thanks

lersek wrote:
Duelist wrote:

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.

Just two quick notes.

First, I verified Windows 7 right now. I built OVMF from edk2 SVN r18122 (git 7669f734). Both the installer ISO ("en_windows_7_professional_with_sp1_x64_dvd_u_676939.iso") and my preexistent guest boot to the GUI just fine, using QXL.

Second, I don't understand why you said "i failed to attach the debug console" again. I explained in excruciating detail how you should edit the domain XML, here: https://bbs.archlinux.org/viewtopic.php … 2#p1548842 . You didn't react to that comment, so I guess you either missed it, or didn't read it far enough for the actual XML snippet.

Last edited by gneville (2015-08-02 13:13:43)

Offline

#5795 2015-08-02 19:13:47

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
Duelist wrote:

@gneville:
I have problems installing windows7 on OVMF since... 18 of july.
I don't know what causes it, as i failed to attach the debug console.
Glad to see you've managed to finish it up via commandline.
It just hangs when the setup-from-hard-drive starts to boot up.

Just two quick notes.

First, I verified Windows 7 right now. I built OVMF from edk2 SVN r18122 (git 7669f734). Both the installer ISO ("en_windows_7_professional_with_sp1_x64_dvd_u_676939.iso") and my preexistent guest boot to the GUI just fine, using QXL.

Second, I don't understand why you said "i failed to attach the debug console" again. I explained in excruciating detail how you should edit the domain XML, here: https://bbs.archlinux.org/viewtopic.php … 2#p1548842 . You didn't react to that comment, so I guess you either missed it, or didn't read it far enough for the actual XML snippet.

Thanks, i've missed that post because i've posted an answer to someone else's post the moment you've posted. Thanks!
Strangely though, i remember specifying the namespace, and libvirt didn't object on the syntax, however i didn't check for the actual QEMU command line produced. Will test again soon.

gneville wrote:

Edit2: Speifying the nvram file, got me past the issue where I'd always have to manually boot the UEFI bootloader e.g. FS1: \EFI\Boot\bootx64.fd. I had to change the boot order in the Tiano Core BIOS and have it.

Well, usually libvirt makes a copy of the OVMF_VARS file, where all the variale stuff are stored. Yeah, boot order is there too. If the loader is specified there, OVMF doesn't do it's "scan for all loaders" routine and boots the next available device, which is PXE, i guess. I might be wrong in that sentence.
I don't know why your VM didn't make that copy.

---

dragonreborn wrote:

It's most probably a hardware fault, fans aren't working on vga at all. Caused the card to overheat and stop working in 10 minutes.

I wasn't swift enough to tell that this is really an overheating case.

jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

How about us ignoring that rule for another two years please? I run fedora, and all my messages belong to this particular topic because that thread is the only thread on the net with that much of activity and info. There are lersek(who's the developer of OVMF) and aw(who's the (redhat i guess) developer of vfio).
Most of the questions are not distro-specific, anyway. If the arch packages differ from what others are using, it's only either a more fresh version of an upstream package, or with some very rare and specific patches like OVMF with sata booting.

Last edited by Duelist (2015-08-02 19:40:38)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5796 2015-08-02 20:31:04

impulse_255
Member
Registered: 2015-06-29
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I switched to Windows 10. I found a comment on Alex's blog a while ago that suggested that on Win 10 Nvidia's driver only does Hyper-V checks when the system reboots.

Peculiarly, it is so. I've been running the VM with HV enlightenments on and it keeps working after several shutdown/start cycles. Only when the guest is rebooted or started up after a crash, does the driver crap out with code 43 (edit: and during driver installation, I would assume). Likewise when in code 43 state, the guest must be rebooted when HV is off to fix it. Afterwards HV can be enabled again. The same behaviour happens with a quadro. I mentioned earlier that the quadro driver also detects HV but instead of refusing to work it runs in some kind of degraded mode (tldr: refresh rates are capped and some other functionality is missing in control panel).

Now I can get 144 Hz output to my monitor without sacrificing HV. Since I'm avoiding the HV checks anyway, I'm running with Geforce pci ids instead of Quadro's so overclocking tools work to their full extent as well.

Edit 2: I wonder if this is because of Fast Boot? Win 8 has it too, but I can't remember if I had it enabled. It's worth noting that libvirt seems to disable hibernation by default which prevents Fast Boot from working.

Last edited by impulse_255 (2015-08-02 20:35:46)

Offline

#5797 2015-08-02 21:20:29

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

How about us ignoring that rule for another two years please? I run fedora, and all my messages belong to this particular topic because that thread is the only thread on the net with that much of activity and info. There are lersek(who's the developer of OVMF) and aw(who's the (redhat i guess) developer of vfio).
Most of the questions are not distro-specific, anyway. If the arch packages differ from what others are using, it's only either a more fresh version of an upstream package, or with some very rare and specific patches like OVMF with sata booting.

No. There are support communities for those distributions; these boards are for Arch.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5798 2015-08-02 21:24:47

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:
Duelist wrote:
jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

How about us ignoring that rule for another two years please? I run fedora, and all my messages belong to this particular topic because that thread is the only thread on the net with that much of activity and info. There are lersek(who's the developer of OVMF) and aw(who's the (redhat i guess) developer of vfio).
Most of the questions are not distro-specific, anyway. If the arch packages differ from what others are using, it's only either a more fresh version of an upstream package, or with some very rare and specific patches like OVMF with sata booting.

No. There are support communities for those distributions; these boards are for Arch.

Ok, seeya.  I thought this board was open to productive, distro agnostic discussions, but if non-Arch users are going to get turned away, you'll need to find another expert.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5799 2015-08-02 22:43:33

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

What? Why? we've been using this forum for 2 years now for everything and everyone related to gpu passthrough, so why now? make no sense

Last edited by nbhs (2015-08-02 22:43:57)

Offline

#5800 2015-08-02 22:51:28

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

What? Why? we've been using this forum for 2 years now for everything and everyone related to gpu passthrough, so why now? make no sense

We have the policy for a reason. This thread is no different to the rest of the boards; so, yes, it does make sense, it is entirely consistent with the overall approach.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5801 2015-08-03 00:26:13

holovati
Member
From: Denmark
Registered: 2014-03-04
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
Duelist wrote:

Is there a legitimate way of attaching an ISA debug console to a libvirt domain(VM)? I tried to pass the regular options as qemu:commandline, but it says that there's no such option...
Googling that problem leads me to my own messages about that debug feature.

I'm sure you did the right thing -- almost. The bit that you probably missed is that the "commandline" and "arg" elements come from a different XML namespace than the rest of the elements in the domain XML. (They are defined by an XML Schema that targets the "http://libvirt.org/schemas/domain/qemu/1.0" namespace -- BTW do not click this link, it's just a string.)

Therefore, you have to declare that XML namespace with a namespace prefix, and qualify the "commandline" and "arg" elements with this namespace prefix. This namespace / prefix association can occur in any element that (recursively) contains these elements; or, even, you could just set the default namespace with attributes of the "arg" and "commandline" elements themselves (but that would take a lot more typing). See:

https://en.wikipedia.org/wiki/XML_names … eclaration

In practice though, the namespace is usually declared in the root element of the XML document.

Anyway, enough of the abstract discussion, here's what you need:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> ... stuff ... <qemu:commandline> <qemu:arg value='-global'/> <qemu:arg value='isa-debugcon.iobase=0x402'/> <qemu:arg value='-debugcon'/> <qemu:arg value='file:/tmp/GUEST_NAME.log'/> </qemu:commandline> </domain>

In the above,

qemu

is the namespace prefix being declared,

http://libvirt.org/schemas/domain/qemu/1.0

is the namespace URI being mapped to the

qemu

namespace prefix, and

qemu:commandline

and

qemu:arg

are the elements that you need.

(The "qemu" prefix is incidental, it could be replaced with any other prefix; the point is that the same prefix must be used in the namespace declaration and later for qualifying "commandline" and "arg". The "qemu" prefix is just a "mediator"; what matters is that "commandline" and "arg" come from the namespace "http://libvirt.org/schemas/domain/qemu/1.0".)

The namespace link seems dead.

Offline

#5802 2015-08-03 02:25:33

mostlyharmless
Member
Registered: 2014-01-16
Posts: 50

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well if after 233 pages the Powers feel that the resident experts including the OP have violated the Arch Way, I guess that confirms everything I've heard about the Arch community. Too bad. Perhaps I should just go back to Slackware, not that anyone really cares about that here.  Sheesh.

Offline

#5803 2015-08-03 06:32:46

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

KVM passthorugh seems to work fine on Lubuntu 14.4 but I seem to get better performance when I run it from arch.

Last edited by morphological (2015-08-03 18:59:38)

Offline

#5804 2015-08-03 10:10:24

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

Is there a way around using the grub.cfg, while getting the passthrough working?

Depends on the bootloader you use. There's a very rude way of including the kernel command line into the kernel itself. Different platforms have it different. RaspberryPi, a common example of embedded linux, has a /boot/cmdline.txt file. But that's waay beyond of this topic.


jasonwryan wrote:

No. There are support communities for those distributions; these boards are for Arch.

Alright, alright! I have installed pacman again.

[user@crossfire ~]$ pacman -V .--. Pacman v4.1.2 - libalpm v8.0.2 / _.-' .-. .-. .-. Copyright (C) 2006-2013 Pacman Development Team \ '-. '-' '-' '-' Copyright (C) 2002-2006 Judd Vinet '--' This program may be freely redistributed under the terms of the GNU General Public License. [user@crossfire ~]$ cat /etc/fedora-release Fedora release 22 (Twenty Two)

See?

Last edited by Duelist (2015-08-03 10:14:31)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5805 2015-08-03 12:19:02

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
morphological wrote:

Is there a way around using the grub.cfg, while getting the passthrough working?

Depends on the bootloader you use. There's a very rude way of including the kernel command line into the kernel itself. Different platforms have it different. RaspberryPi, a common example of embedded linux, has a /boot/cmdline.txt file. But that's waay beyond of this topic.

I use arch and lubuntu 14.04 as my main systems.
Could you provide some links to documentation or a general topic designation this matter corresponds to?

Thanks!

Offline

#5806 2015-08-03 14:35:47

fushfush
Member
Registered: 2015-08-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey Guys.

My current situations is this:

I got an W520 with an i7-2720QM and Nvidia Quadro 1000m. I want to virtualize with Debian as host and Windows as guest. Therefore i want to assign my Nvidia GPU to the guest.
The reason for this choice is because I'm an mechanical engineering student, and that requires alot of windows-only programs..

I got stuck at Nvidias error 43 problem.

I haved used the 377.88 Driver, with and without kvm as hidden. I have deleted all hypervisor tags.. It still reports error 43 .. 

This is my config file over my virtual windows machine:

<domain type='kvm'> <name>win7</name> <uuid>5ebc2f74-7bbd-482d-99f4-660155187f8d</uuid> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>6291456</currentMemory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> <kvm> <hidden state='on'/> </kvm> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sda3'/> <target dev='vda' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:69:ea:30'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

Can you help me with something i can go on with? I'm pretty stuck at this point.

Thanks!

Offline

#5807 2015-08-03 16:30:39

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fushfush wrote:

Hey Guys.

My current situations is this:

I got an W520 with an i7-2720QM and Nvidia Quadro 1000m. I want to virtualize with Debian as host and Windows as guest. Therefore i want to assign my Nvidia GPU to the guest.
The reason for this choice is because I'm an mechanical engineering student, and that requires alot of windows-only programs..
Thanks!

Assuming this is a laptop, there was no one, who got it working properly AFAIR. So you're pretty much stuck with windows. However, i find it strange you have a quadro in there...
I can't give you an advice, but i can link you the spreadsheet of hardware.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5808 2015-08-03 18:11:02

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
jasonwryan wrote:

No. There are support communities for those distributions; these boards are for Arch.

Alright, alright! I have installed pacman again.

[user@crossfire ~]$ pacman -V .--. Pacman v4.1.2 - libalpm v8.0.2 / _.-' .-. .-. .-. Copyright (C) 2006-2013 Pacman Development Team \ '-. '-' '-' '-' Copyright (C) 2002-2006 Judd Vinet '--' This program may be freely redistributed under the terms of the GNU General Public License. [user@crossfire ~]$ cat /etc/fedora-release Fedora release 22 (Twenty Two)

See?

I see you aren't running Arch and you are going out of your way to flout the Forum Etiquette and antagonise the staff...


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5809 2015-08-03 18:57:44

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey there.

I have got the Passthrough working on my GTX 750TI and 4690K  with Windows 10.
Performance is ok and overclocking of the GPU works, but I am noticing one little problem.

Coretemp and CPUZ are both reporting weird values, or none at all.
My CPU is at 4.5GHZ, yet 4GHZ are displayed.
http://s16.postimg.org/uaw7ohahv/Untitled.png

I have seen this occurring on other systems, but couldnt find out how to fix it.

nbhs also seems to have a CPUZ CPU Frequency output of 4GHZ, which is probably incorrect.
http://i.imgur.com/PtTM4BD.jpg

Last edited by morphological (2015-08-03 19:03:16)

Offline

#5810 2015-08-03 19:47:44

shade
Banned

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

Hey there.

I have got the Passthrough working on my GTX 750TI and 4690K  with Windows 10.
Performance is ok and overclocking of the GPU works, but I am noticing one little problem.

Coretemp and CPUZ are both reporting weird values, or none at all.
My CPU is at 4.5GHZ, yet 4GHZ are displayed.
http://s16.postimg.org/uaw7ohahv/Untitled.png

I have seen this occurring on other systems, but couldnt find out how to fix it.

nbhs also seems to have a CPUZ CPU Frequency output of 4GHZ, which is probably incorrect.
http://i.imgur.com/PtTM4BD.jpg

Been there.
We can't get the temperatures right for two years on modern AMD platforms(TCTL!), and you're asking for frequencies inside the VM.

#5811 2015-08-04 03:24:22

wmarler
Member
From: Denver, CO
Registered: 2015-04-18
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's the SOW with regards to passthrough of USB audio devices? That was by far the easiest way I've found to get sound working without lag, but a) it doesn't mux audio from the guest and the host, and (more importantly) b) the audio is "crackly." I'm not sure where to pull debug data from to submit a bug report nor ... really where to submit a bug report anyway.

Also ... I'm open to suggestions! I think I saw one person with successful audio by "buying a different USB audio device," which I'm not opposed to, just want to try a few more things first.

I haven't really had good luck with PulseAudio (mostly because I'm not sure how to verify that the environment variables I'm exporting are being accepted by the vm when it starts; managing this machine using virt-manager).

Offline

#5812 2015-08-04 06:53:46

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am wondering if the UEFI GPU Passthrough method via OVMF would grant me more GPU and CPU performance in the VM, than the legacy variant with i915 and arbiter patch.
Are there any reports on that subject?

Last edited by morphological (2015-08-04 07:13:28)

Offline

#5813 2015-08-04 07:30:59

fushfush
Member
Registered: 2015-08-03
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
fushfush wrote:

Hey Guys.

My current situations is this:

I got an W520 with an i7-2720QM and Nvidia Quadro 1000m. I want to virtualize with Debian as host and Windows as guest. Therefore i want to assign my Nvidia GPU to the guest.
The reason for this choice is because I'm an mechanical engineering student, and that requires alot of windows-only programs..
Thanks!

Assuming this is a laptop, there was no one, who got it working properly AFAIR. So you're pretty much stuck with windows. However, i find it strange you have a quadro in there...
I can't give you an advice, but i can link you the spreadsheet of hardware.

Thanks for for spreadsheet, and yes its a Quadro. Its a workstation laptop designed for CAD/FEM-work.. You're right about posting on an wrong forum, but arch seems to have much more knowledge about this topic then debian .. Another reason to try Arch.

Offline

#5814 2015-08-04 11:51:58

lersek
Member
Registered: 2015-03-15
Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

holovati wrote:
lersek wrote:

(They are defined by an XML Schema that targets the "http://libvirt.org/schemas/domain/qemu/1.0" namespace -- BTW do not click this link, it's just a string.)

The namespace link seems dead.

Oh geez, did you read what I wrote at all? I trimmed the quote now, to add more emphasis. The namespace URI is *just a string*, not something to retrieve from the network. This is explained in the wikipedia articles that I had linked, and I had also mentioned it in my comment; see above.

Offline

#5815 2015-08-04 14:55:53

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fushfush wrote:

Thanks for for spreadsheet, and yes its a Quadro. Its a workstation laptop designed for CAD/FEM-work.. You're right about posting on an wrong forum, but arch seems to have much more knowledge about this topic then debian .. Another reason to try Arch.

well I am afraid not for much longer. The main author of this functionality in both kernel and qemu is now gone from this thread, since he happens to be a RedHat developer and local moderators here told in no uncertain terms that this forum should be used for Arch related issues only.

Last edited by Bronek (2015-08-04 14:57:02)


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5816 2015-08-04 21:15:26

Samsonite
Banned

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

I am wondering if the UEFI GPU Passthrough method via OVMF would grant me more GPU and CPU performance in the VM, than the legacy variant with i915 and arbiter patch.
Are there any reports on that subject?

Hmm, from what i remember - there was no viable difference in Unigine Heaven 4.0 benchmark numbers.
Usually people come to OVMF when x-vga doesn't work, so that makes it difficult to test side-by-side.
Take the opportunity to be the first! I suppoose there may be slight CPU difference, but otherwise the numbers should be close.

lersek wrote:
holovati wrote:
lersek wrote:

(They are defined by an XML Schema that targets the "http://libvirt.org/schemas/domain/qemu/1.0" namespace -- BTW do not click this link, it's just a string.)

The namespace link seems dead.

Oh geez, did you read what I wrote at all? I trimmed the quote now, to add more emphasis. The namespace URI is *just a string*, not something to retrieve from the network. This is explained in the wikipedia articles that I had linked, and I had also mentioned it in my comment; see above.

Don't stress out, i see that guy for the first time there.
I've been successful in attaching the debug console, however i haven't compiled a test platform yet.
There seem to be some hardware failure of some sort on my machine, as windows 10 install with the latest OVMF build BSODs with some kernel-level errors. The nested virtualization toggle, suddenly, makes some difference, as with it enabled it crashes more often. Also, not only the setup, but the passed-through devices suddenly remain code-10'ed on some builds.
Is there some patching in process, like someone rewriting SMBus logic maybe or something?

OLD, maybe even pre-sourceforge-war builds work like a charm...

Last edited by Samsonite (2015-08-04 21:16:52)

#5817 2015-08-04 23:54:36

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

I see you aren't running Arch[...]

Dear Jason Wryan,
I hope you're joking, aren't you? I've got the deepest respect for this community and also your work you invest in moderation (and administration?) of this forum. I know there's the "Arch Linux distribution support ONLY" rule, but there's also the "Respect other distributions and operating systems" rule. I think it's correct to say with many problems, especially beginner's problems, that a mix of users from different linux distributions might complicate the solution and a separation is often usefull and yes, this forum here has archlinux in its domain name, logo and everywhere. Acknowledged.
BUT: this thread's topic is a power-user, enthusiast topic, involving several aspects of a linux system. Kernel modules, -parameters, lot of different software packets etc. There can't be a topic with this name in each distribution forum, because there aren't enough people with interest and knowledge. Instead, because the lack of another single forum that fits the whole package of this topic and nonetheless because Arch linux has been predestined for this kind of experiments in the beginning (quite current kernel, AUR), this forum HAPPEND to be the place where this thread EMERGED, many arch users joined the discussion more than two years ago and still in 2013, the first cross-distribution users joined, too. Some of them had questions, but some joined with deep knowledge, partially even involved in development, helping both arch- and non-arch-users.
If you strip the non-arch-part of this thread, you might kill not all, but most of its usefulness.

PLEASE rethink your decision of strict observation of this thread and banning non-arch-users. Because of this thread no harm was done besides violation of one rule of the etiquette and bruising some egos. Sure, violating the etiquette is more than nothing, but on the other hand this thread was very, very useful and unique. There's no guarantee that AW, Duelist and other power-contributors would return/stay, but we can at least try.

If you want to close this thread I'd hope we could find a way to revive it somewhere else, but apart from the question if you'd export all posts for us, I'd no idea where. I or someone else could host a specific forum, but I have my doubts enough users would find it/register/follow. Google for KVM VGA and if you resist clicking "kvm switches", you're directed here.

P.S: You can't ban me because for not being an arch user, because I am. Longterm. But you're free to ban me for having offended you in any way. Gonna make it easy by reporting your comments in this thread, propably to yourself big_smile , but, well, I really don't understand the sudden and non-comprehensible events, more than two years after the creation of this helpful and productive thread.

Regards, an irritated arch user

Offline

#5818 2015-08-05 00:08:09

ewaller
Forum Moderator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 10,547

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Allow me to clarify a few things.  First, this topic has been discussed by the moderation team.  It is not just one of us taking action.
We do welcome everyone on the forums, regardless of whether they do, or do not use Arch Linux.  At issue is that these are support forums for Arch Linux, and it is our goal to have them be the canonical reference for Arch.  We have nothing against other distributions and we wish them well, but having specific support information that is appropriate for non Arch distributions that is not applicable to Arch is inconsistent with our published policies.

There has been no talk about closing or deleting this thread amongst the moderators.

https://wiki.archlinux.org/index.php/Fo … pport_ONLY
https://wiki.archlinux.org/index.php/Fo … _the_staff

Can we try to pull this back on track as it applies to Arch Linux?

Edit:  mauorrizze, I did see your report, but not until after I posted this message.

Last edited by ewaller (2015-08-05 00:11:19)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Like you, I have no idea what you are doing, but I am pretty sure it is wrong...Jasonwryan
----
How to Ask Questions the Smart Way

Offline

#5819 2015-08-05 10:44:52

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I have a working vga passthrough setup and I'd like to switch from gummiboot to grub. My "stupid" Asrock motheboard has a button that "clears" the bios on the I/O shield. If I press it while trying to a plug a cable it'll erase the bios config, and I have to reinstall gummibit via a LiveCD... which is boring !

My setup is : IGD for host, and an Nvidia GTX card for my guest VM.

I installed grub and it works as expected : it loads my os with the required kernel. However,

However, as soon as I add the  "intel_iommu=on" to GRUB_CMDLINE_LINUX in /etc/default/grub and regenerate config file, then my system stops booting right after grub. I have a weird screen similar to this one : http://lists.linuxfoundation.org/piperm … t-0001.jpg. The problem seems to be like this guy here: http://lists.linuxfoundation.org/piperm … 08805.html

I tried each of the following :

GRUB_CMDLINE_LINUX="intel_iommu=on, igfx_off" GRUB_CMDLINE_LINUX="intel_iommu=on, igfx_on" GRUB_CMDLINE_LINUX="intel_iommu=on, igfx_off vfio_iommu_type msi=1 pci-stub.ids=10de:13c2,10de:0fbb,00:1d.0"

Nothing worked.

Right now it works fine with gummiboot and the following "boot line", in  /boot/loader/entries/vfio.conf :

options root=/dev/sda2 rw intel_iommu=on,igfx_off vfio_iommu_type msi=1 pci-stub.ids=10de:13c2,10de:0fbb,00:1d.0

I'd glad to hear any thought on this. Thanks in advance.

Last edited by Nesousx (2015-08-05 12:26:58)

Offline

#5820 2015-08-05 14:16:15

Ecliptual
Guest

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

Hi everyone,

I have a working vga passthrough setup and I'd like to switch from gummiboot to grub. My "stupid" Asrock motheboard has a button that "clears" the bios on the I/O shield. If I press it while trying to a plug a cable it'll erase the bios config, and I have to reinstall gummibit via a LiveCD... which is boring !

My setup is : IGD for host, and an Nvidia GTX card for my guest VM.

I installed grub and it works as expected : it loads my os with the required kernel. However,

However, as soon as I add the  "intel_iommu=on" to GRUB_CMDLINE_LINUX in /etc/default/grub and regenerate config file, then my system stops booting right after grub. I have a weird screen similar to this one : http://lists.linuxfoundation.org/piperm … t-0001.jpg. The problem seems to be like this guy here: http://lists.linuxfoundation.org/piperm … 08805.html

I tried each of the following :

GRUB_CMDLINE_LINUX="intel_iommu=on, igfx_off" GRUB_CMDLINE_LINUX="intel_iommu=on, igfx_on" GRUB_CMDLINE_LINUX="intel_iommu=on, igfx_off vfio_iommu_type msi=1 pci-stub.ids=10de:13c2,10de:0fbb,00:1d.0"

Nothing worked.

Right now it works fine with gummiboot and the following "boot line", in  /boot/loader/entries/vfio.conf :

options root=/dev/sda2 rw intel_iommu=on,igfx_off vfio_iommu_type msi=1 pci-stub.ids=10de:13c2,10de:0fbb,00:1d.0

I'd glad to hear any thought on this. Thanks in advance.

First of all, "vfio_iommu_type" isn't a valid parameter to begin with.
Make sure you don't need the i915 vgaarb patch.
Which screen outputs this mess? The IGD one or GPU one?
Seems like you've got it working only on EFI-boot without CSM, am i right?
Also, you say it works "fine" with the gummiboot and igfx_off - you're using the system as a headless host?


ewaller wrote:

but having specific support information that is appropriate for non Arch distributions that is not applicable to Arch is inconsistent with our published policies.

The only distro-specific information is, obviously, kernel patches and boot loaders. I may not remember everything, but one of the things arch users have different is OVMF git builds - there's an AUR package for that, afair. Most of distro-specific information is easily adapted for arch.

ewaller wrote:

There has been no talk about closing or deleting this thread amongst the moderators.
https://wiki.archlinux.org/index.php/Fo … pport_ONLY
https://wiki.archlinux.org/index.php/Fo … _the_staff
Can we try to pull this back on track as it applies to Arch Linux?

Nice to see that this wasn't one man's decision.
I've thought about collaborating with nbhs in order to close & move this thread to another forum, but i think no one would like this.
I don't know about aw, but I am staying here. Even if i'll have to hide from you all thoroughly.
For everyone.
I just don't understand the idea of saying "arch only" to non-arch helpers. What i should do to follow that policy? Install arch? I mean, i've helped(or tried to help) to arch, gentoo, fedora and ubuntu users, to me they're no different. Moreover, most people don't even specify their distro at all...

Last edited by Ecliptual (2015-08-05 14:17:08)

#5821 2015-08-05 14:35:14

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your reply Ecliptual.

I thought that with latest kernels, (can't remember which version, 4.1 I believe) which is the one I am using iirc, I no longer needed arbiter patch. Because indeed, it looks like the problem is coming from here since I am using IGD on host and with a "head" setup, so it is the IGD that messes.

I am booting full EFI as far as I understand it, but I have no clue about the CSM thing.

Finally, I'll look on the vfio_iommu_tpe thing and remove it if it is not valid.

Offline

#5822 2015-08-05 14:43:20

Ecliptual
Guest

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

I am booting full EFI as far as I understand it, but I have no clue about the CSM thing.

That's easy to check.
0. The UEFI setup menu should be clear to understand, try checking there.
1. MBR entries won't be there. So if you don't have an ESP - you won't be able to boot. You're using gummiboot, so i assume you do have a valid ESP with the EFI loader in there.
2. I think /dev/vga_arbiter doesn't exist on systems booted without VGA present.
3. check lspci for these VGA memory and port IO ranges. They may not be present on a pure efi system.

 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at fe300000 (64-bit, non-prefetchable) [size=256K] Region 4: I/O ports at e000 [size=256]

Actually, you spark my interest...

#5823 2015-08-05 22:10:26

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ecliptual wrote:
Nesousx wrote:

I am booting full EFI as far as I understand it, but I have no clue about the CSM thing.

That's easy to check.
0. The UEFI setup menu should be clear to understand, try checking there.
1. MBR entries won't be there. So if you don't have an ESP - you won't be able to boot. You're using gummiboot, so i assume you do have a valid ESP with the EFI loader in there.
2. I think /dev/vga_arbiter doesn't exist on systems booted without VGA present.
3. check lspci for these VGA memory and port IO ranges. They may not be present on a pure efi system.

 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at fe300000 (64-bit, non-prefetchable) [size=256K] Region 4: I/O ports at e000 [size=256]

Actually, you spark my interest...

I'll be more than happy to answer your questions, however I need more explanation.

0. Not sure where to find this thing, I checked my bios in order to see if I could access some kind of UEFI shell, didn't find anything.
1. I also assume I have a valid ESP, I remember doing it when installing Arch for the first time. I have a fat boot partition with a /boot/EFI folder mounted there. I believe this is what you are talking about.
2. Does this help ?:

ls -l /dev/vga_arbiter crw------- 1 root root 10, 63 6 août 00:01 /dev/vga_arbiter

3.

lspci 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) 00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) 00:1c.6 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) 03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) 04:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10) 05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) (prog-if 00 [VGA controller]) Subsystem: ASRock Incorporation Device 0152 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 37 Region 0: Memory at f7400000 (64-bit, non-prefetchable) [size=4M] Region 2: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at f000 [size=64] Expansion ROM at <unassigned> [disabled] Capabilities: <access denied> Kernel driver in use: i915 Kernel modules: i915
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. Device 8508 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 11 Region 0: Memory at f6000000 (32-bit, non-prefetchable) [disabled] [size=16M] Region 1: Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M] Region 3: Memory at f0000000 (64-bit, prefetchable) [disabled] [size=32M] Region 5: I/O ports at e000 [disabled] [size=128] Expansion ROM at f7000000 [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nouveau

Offline

#5824 2015-08-05 22:37:04

FlatronEZ
Banned
Registered: 2015-08-05
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I am new to this community and hope I don't do anything completely wrong here wink

So my problem is the "standard" Nvidia Error Code 43 problem.

First my setup:

Ubuntu 15.04 3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015 QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3) i7 4790k Nvidia GTX 750 Ti (Palit) (in main PCIe slot) MB: Gigabyte Z97X Gaming 7

I installed Windows 10 (64bit) as a guest operating system using virt-manager. The installation process worked fine and the OS is running. When I pass my Nvidia GTX 750 Ti to the guest and install the latest Nvidia Driver then the card get's recognized correctly but gives me error code 43 after the reboot.

So what I tried:

Adding "intel_iommu=on igfx_off" to my boot paramters aswell as manually adding "-kvm=off" to my virsh config with:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> [...] <qemu:commandline> <qemu:arg value='-kvm=off'/> </qemu:commandline>

which gave me this error: "qemu-system-x86_64: -kvm=off: invalid option"

Okay so for now I dont know what to try next... What can I try to get my card running?

Here is my virt-manager config:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows-10</name> <uuid>c7f056b0-fd21-4cfe-a050-dd23cdc40ef6</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static' current='1'>8</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='dtes64'/> <feature policy='require' name='invpcid'/> <feature policy='require' name='vmx'/> <feature policy='require' name='erms'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='smep'/> <feature policy='require' name='pbe'/> <feature policy='require' name='est'/> <feature policy='require' name='monitor'/> <feature policy='require' name='abm'/> <feature policy='require' name='tm'/> <feature policy='require' name='acpi'/> <feature policy='require' name='fma'/> <feature policy='require' name='osxsave'/> <feature policy='require' name='ht'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='fsgsbase'/> <feature policy='require' name='f16c'/> <feature policy='require' name='ds'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='tm2'/> <feature policy='require' name='avx2'/> <feature policy='require' name='ss'/> <feature policy='require' name='bmi1'/> <feature policy='require' name='bmi2'/> <feature policy='require' name='pcid'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='movbe'/> <feature policy='require' name='rdrand'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/kvm-spice</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/Windows-10.qcow2'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='usb' index='0' model='nec-xhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:56:38:04'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='spice' autoport='yes'/> <sound model='ich6'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <redirdev bus='usb' type='spicevmc'> </redirdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-kvm=off'/> </qemu:commandline> </domain>

Let me know if you need more information!

Offline

#5825 2015-08-05 22:54:31

WorMzy
Forum Moderator
From: England
Registered: 2010-06-16
Posts: 3,492

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FlatronEZ wrote:

Hello,

I am new to this community and hope I don't do anything completely wrong here wink
First my setup:

Ubuntu 15.04 3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015 QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3) i7 4790k Nvidia GTX 750 Ti (Palit) (in main PCIe slot) MB: Gigabyte Z97X Gaming 7

Please ask on the Ubuntu support forums, we only support Arch here.

https://wiki.archlinux.org/index.php/Fo … pport_ONLY


Sakura:-
Mobo: ASUS P8Z77-V PRO // Processor: Intel Core i7-3770K 3.4GHz // GFX: nVidia GeForce GTX 970 Ti // RAM: 16GB (4x 4GB) Corsair DDR3 (@ 1333MHz) // Storage: 1x 3TB Seagate SATAII 4x 1TB Samsung SATAII, 2x 120GB Corsair SSD

Offline

#5826 2015-08-05 22:58:22

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@FlatronEZ, for Code 43, look here : http://vfio.blogspot.fr/2015/05/vfio-gp … first.html, and search for :

 <kvm> <hidden state='on'/> </kvm>

and hypervclock.

Offline

#5827 2015-08-06 00:19:08

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

WorMzy wrote:
FlatronEZ wrote:

Hello,

I am new to this community and hope I don't do anything completely wrong here wink
First my setup:

Ubuntu 15.04 3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015 QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3) i7 4790k Nvidia GTX 750 Ti (Palit) (in main PCIe slot) MB: Gigabyte Z97X Gaming 7

Please ask on the Ubuntu support forums, we only support Arch here.

https://wiki.archlinux.org/index.php/Fo … pport_ONLY

You realize that question had no dependency whatsoever on the distro, right?  And banned?  For that?  Was that Duelist trolling or was that really ban-worthy?  Not gaining any friends here.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#5828 2015-08-06 00:26:10

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
WorMzy wrote:
FlatronEZ wrote:

Hello,

I am new to this community and hope I don't do anything completely wrong here wink
First my setup:

Ubuntu 15.04 3.19.0-26-generic #27-Ubuntu SMP Tue Jul 28 18:27:31 UTC 2015 QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3) i7 4790k Nvidia GTX 750 Ti (Palit) (in main PCIe slot) MB: Gigabyte Z97X Gaming 7

Please ask on the Ubuntu support forums, we only support Arch here.

https://wiki.archlinux.org/index.php/Fo … pport_ONLY

You realize that question had no dependency whatsoever on the distro, right?  And banned?  For that?  Was that Duelist trolling or was that really ban-worthy?  Not gaining any friends here.

Banned because the mod made it clear that these boards are for the support of Arch, not Ubuntu: Flatron ignored that polite pointer to our Etiquette.
Duelist was banned for provoking the staff and then creating three more accounts to continue to post; again, flouting the rules.

As for making friends, that isn't high on my list of priorities. Joining this community and participating here is contingent on following the rules. If members can't--or won't--abide by those rules then the thread will be closed.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5829 2015-08-06 02:00:41

Ecliptual
Guest

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

If members can't--or won't--abide by those rules then the thread will be closed.

Alright then. If this endangers or jeopardizes the whole thread here - okay. I will leave, if my presence is so strictly not allowed here. I know you can track my activity there easily.

But there will be other users coming in from the... google(or other search engines) results. And not everyone will be using arch.
Sadly, that's how it is happening now.
I respect your actions. I see you all really have gathered and discussed the problems. You have the final word after all.
Do you have the technical possibility to put an advice somewhere on this thread's page, where everyone will see it, that non-arch distributions support is not provided?
That would be very helpful for those, who may accidentally come here to ask.

P.S.
Disband this account, again. That's a nice enough nickname to be used by someone else.

Last edited by Ecliptual (2015-08-06 02:01:37)

#5830 2015-08-06 02:14:07

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ecliptual wrote:
jasonwryan wrote:

If members can't--or won't--abide by those rules then the thread will be closed.

Alright then. If this endangers or jeopardizes the whole thread here - okay. I will leave, if my presence is so strictly not allowed here. I know you can track my activity there easily.

This isn't about you. If you had adopted this sort of helpful approach earlier, much of this drama could have been avoided. I'll delete this account, but if you are prepared to abide by the rules, your original account can be unbanned. Email me if you want this to happen.

Do you have the technical possibility to put an advice somewhere on this thread's page, where everyone will see it, that non-arch distributions support is not provided?
That would be very helpful for those, who may accidentally come here to ask.

There is no point pasting the Etiquette into a 200 page thread. Everyone here is expected to have read it and to agree to follow it.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5831 2015-08-06 07:34:01

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

If you had adopted this sort of helpful approach earlier,

Just my quick thoughts... Duellist has been helping in this thread for several months or even a year and invested some time in this, probably comparable to the commitment of your moderators. To my knowledge he started being a little more offensive after being told that this help was unwanted by the authorities.

jasonwryan wrote:

much of this drama could have been avoided.

Most if not all the drama could have been avoided if you wouln't have let this thread grow to one of the most resourceful places about this topic (including questions and help from non-archers) and start enforcing your arch-only-rule delayed by two years.

jasonwryan wrote:

Everyone here is expected to have read it and to agree to follow it.

Of course.

Offline

#5832 2015-08-06 07:46:41

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mauorrizze wrote:
jasonwryan wrote:

much of this drama could have been avoided.

Most if not all the drama could have been avoided if you wouln't have let this thread grow to one of the most resourceful places about this topic (including questions and help from non-archers) and start enforcing your arch-only-rule delayed by two years.

I'm really getting tired of this. The fact that no-one from the staff has paid any attention to this thread for two years is completely irrelevant: that is not a license to believe that you are free to behave as you will and have earned some sort of special dispensation from the rules others have to abide by.

The boards are busy and the mod team is small. The only reason I came in here was because some muppet using Ubuntu posted in one of the high volume boards and I mistakenly merged it here before I realised it had nothing to do with Arch.

So, let me be very clear, for the last time. There is nothing about this thread or the people that post in it that is special: you aren't exempt from the rules that govern these boards and they will be applied equally to all. If you have a problem with that, one of the mod team can show you the door.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5833 2015-08-06 08:10:02

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using the i915 and arbiter patch method.
I am trying to enable HyperV as suggested by nbhs: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html

However, I cannot seem to find the xml config file for my virtual machine.
/etc/libvirt/qemu  only contains a folder named networks, with xml files specific to that.

I also could not extract the xml file from my existing virtual machine.
Some sites are discouraging Nvidia card owners from applying these hyper-v settings, as they could cause instability. Are there any Nvidia owners here who have tried it?

I set it up using this this script:

 #!/bin/bash configfile=/etc/vfio-pci.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=4 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/media/jackson/2cf57f0f/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -usb -usbdevice host:045e:0745 \ -boot menu=on exit 0

Last edited by morphological (2015-08-06 10:26:25)

Offline

#5834 2015-08-06 11:15:36

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

Using the i915 and arbiter patch method.
I am trying to enable HyperV as suggested by nbhs: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html

However, I cannot seem to find the xml config file for my virtual machine.
/etc/libvirt/qemu  only contains a folder named networks, with xml files specific to that.

I also could not extract the xml file from my existing virtual machine.
Some sites are discouraging Nvidia card owners from applying these hyper-v settings, as they could cause instability. Are there any Nvidia owners here who have tried it?

I set it up using this this script:

 #!/bin/bash configfile=/etc/vfio-pci.cfg vfiobind() { dev="$1" vendor=$(cat /sys/bus/pci/devices/$dev/vendor) device=$(cat /sys/bus/pci/devices/$dev/device) if [ -e /sys/bus/pci/devices/$dev/driver ]; then echo $dev > /sys/bus/pci/devices/$dev/driver/unbind fi echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id } modprobe vfio-pci cat $configfile | while read line;do echo $line | grep ^# >/dev/null 2>&1 && continue vfiobind $line done sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \ -smp 4,sockets=1,cores=4,threads=4 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \ -drive file=/media/jackson/2cf57f0f/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \ -usb -usbdevice host:045e:0745 \ -boot menu=on exit 0

They do not cause instabilities. The nvidia drivers shut themselves off when they see you're trying to do the passthrough thing with a geforce card.
That guide tells you to add

hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time 

to your -cpu flag in commandline.
So it'll be like this:

-cpu kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time 

For additional info, read aw's blog


Now for the distro battery:
Seems like it's really against the local law to ask for support of other distributives(even though most of the questions are distro-agnostic), so i've come up with a shaky compromise.
Statistics(that big spreadsheet) shows that there's a majority of arch users, and then come debian, ubuntu, gentoo, fedora and mint.
My plan is simple: i'll create an universal and fresh op-post and a thread on each distribution's board. So when people come from google(or other search engines), they will be directed to their distro's forum into an appropriate thread.
I do have doubts about this working, but well, what else can i do.
Now just stand by while i'm trying to write this all up.

EDIT:
OP-post alpha(or beta). Not even close to finish. Feel free to comment. I knew i didn't include the link...

Last edited by Duelist (2015-08-06 16:24:26)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5835 2015-08-06 12:58:27

morphological
Member
Registered: 2015-08-01
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you very much for your quick reply!

I believe I am missing the necessary packages to use this -cpu option from the commandline.
Is this a command specific to qemu or virsh etc.?
On google I only see people using it in conjunction with their VM scripts, not from the commandline.

-cpu kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time 

As in terms of the other issue.
In order to be able to participate, I made a fresh install of arch and got the passthrough working.
Then I applied everything I had learned to my Lubuntu installation.:)

It's amazing how knowledgeable everybody is here, and I think its only fair, that if you are consulting an arch thread, that you also actually use the environment it is intended for.

Last edited by morphological (2015-08-06 12:58:45)

Offline

#5836 2015-08-06 16:27:07

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

morphological wrote:

Thank you very much for your quick reply!

I believe I am missing the necessary packages to use this -cpu option from the commandline.
Is this a command specific to qemu or virsh etc.?

relaxed/hv_relaxed. Available in libvirt 1.0.0+ (commit) and qemu 1.1+ (commit). This bit disables a Windows sanity check that commonly results in a BSOD when the VM is running on a heavily loaded host (example bugs here, here, and here). Sounds similar to the Linux kernel option no_timer_check, which is automatically enabled when Linux is running on KVM.
    vapic/hv_vapic. Available in libvirt 1.1.0+ (commit) and qemu 1.1+ (commit).
    spinlocks/hv_spinlocks. Available in libvirt 1.1.0+ (commit) and qemu 1.1+ (commit)
    hypervclock/hv_time. Available in libvirt 1.2.2+ (commit) and qemu 2.0+ (commit). Sounds similar to kvmclock, a paravirt time source which is used when Linux is running on KVM.

Note the QEMU versions. We had 1.6.2 in fedora 20, and that was literally two years ago. I honestly believe every distro now has a recent version of QEMU.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#5837 2015-08-06 17:08:43

avi9526
Member
Registered: 2015-05-15
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is that possible to unbind the one and only GPU (radeon) from host OS, then use it for guest and after guest shutdown bind back to host (and maybe restart user session)?

Offline

#5838 2015-08-06 18:02:23

MrChefur
Member
Registered: 2015-08-06
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't get the kernel to patch with the i915 patch, should I be using the older version of the kernel?
Currently I'm using kernel 4.1.4.
I run the command

patch -p1 i915_2.patch

and it returns

(Stripping trailing CRs from patch; use --binary to disable.) patching file drivers/gpu/drm/i915/i915_dma.c Reversed (or previously applied) patch detected! Skipping patch. 2 out of 2 hunks ignored -- savings rejects to file drivers/gpu/drm/i915/i915_dma.c.rej (Stripping trailing CRs from patch; use --binary to disable.) can't find file to patch at input line 56 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------------------------- |diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h |index ec82f6b..f3908f6 00644 |--- a/drivers/gpu/drm/i915/i915_drv.h |+++ b/drivers/gpu/drm/i915/i915_drv.h ----------------------------------------- File to patch:

I'm using the patch from Here

EDIT (again):
I redownloaded the kernel and patch, extracted the kernel and applied the patch using the same command as before and now it's sitting on a blank line with the cursor just refreshing (I forget the term). Is it actually installing?
It's been going for half an hour now...

EDIT (again again):
I would really appreciate any help I can get here. The terminal doesn't do anything (I left it for 8+ hours on a high end system and nothing happened).
I read all the documentation I can find and I'm still lost.

Last edited by MrChefur (2015-08-07 06:23:40)

Offline

#5839 2015-08-06 18:38:26

4kGamer
Banned
Registered: 2014-10-29
Posts: 82

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't believe what I am reading here ...are you kidding me? Do you have ANY idea how valuable aw and nbhs and others have been for numerous people worldwide??? I have seen this very thread being linked in various other forums as the number one source for getting this complicated matter done. I certainly wouldn't have done it without them. No chance.

Please Jasonwryan Go AWAY and never come back here.

Last edited by 4kGamer (2015-08-06 18:51:38)

Offline

#5840 2015-08-06 19:06:13

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4kGamer wrote:

I can't believe what I am reading here ...are you kidding me? Do you have ANY idea how valuable aw and nbhs and others have been for numerous people worldwide??? I have seen this very thread being linked in various other forums as the number one source for getting this complicated matter done. I certainly wouldn't have done it without them. No chance.

No-one cares what you think: https://wiki.archlinux.org/index.php/Fo … _the_staff


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5841 2015-08-06 20:38:34

c00ter
Banned
From: Alaskan in Washington State
Registered: 2014-08-28
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:
nbhs wrote:
jasonwryan wrote:

Please ask on your distro's boards, we only support Arch: https://wiki.archlinux.org/index.php/Fo … pport_ONLY

What? Why? we've been using this forum for 2 years now for everything and everyone related to gpu passthrough, so why now? make no sense

We have the policy for a reason. This thread is no different to the rest of the boards; so, yes, it does make sense, it is entirely consistent with the overall approach.

Hence me old ma's saying: "A foolish consistency is the hobgoblin of little minds."


UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn

Offline

#5842 2015-08-06 20:49:41

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone else looking for a ban?


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5843 2015-08-06 22:05:58

Nesousx
Member
Registered: 2012-03-27
Posts: 46
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

Anyone else looking for a ban?

Yes sir, give me one please.

I don't see any harm in what we are doing in here helping other people, "distro agnostic". Linux and open source is also about being open minded, imho. What I am reading here, is totally unbelievable. KVM VGA Passthrough is a topic many of us are comfortable with (or even more for a few of us), if we can't share that knowledge on those boards, then there is no point being here.

Last edited by Nesousx (2015-08-06 22:13:24)

Offline

#5844 2015-08-06 22:10:56

mauorrizze
Member
Registered: 2014-08-22
Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

avi9526 wrote:

Is that possible to unbind the one and only GPU (radeon) from host OS, then use it for guest and after guest shutdown bind back to host (and maybe restart user session)?

That's very unlikely, graphic cards are a lot more complicated than USB devices or some controllers. The card you wanna pass through must not have been initialized by the Mainboards BIOS/UEFI before, so it's even hard to get a headless host setup working. My advise: add a cheap graphics card, insert it in the 1x oder 4x PCIe slot, not the faster ones connected to the CPU, to avoid the need of an ACS patch. Then convince your BIOS to boot with the small GPU and you can pass the other one using aw's guide.

Offline

#5845 2015-08-06 22:14:04

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nesousx wrote:

I don't see any harm in what we are doing in here...

None of the people moaning here contribute outside this thread or, I would guess, to Arch at all, so--to be clear--your thoughts about how the boards are run are irrelevant.


The next person that posts a whine in here about our policy can thank themselves for having the thread closed. You want to share knowledge about KVM passthrough, then stick to that.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline

#5846 2015-08-07 09:10:12

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jasonwryan wrote:

Anyone else looking for a ban?

Yes, me too please. I do use Arch as my favourite distro, but I also happen to use other distros as well, and I am not going to abide by the order to discriminate their users. You are welcome to interpret this as a declaration of disobedience and ban me to pre-empt any post of mine which might (or might not) benefit users of other distros. Have it your way, if that is what you want.


Sorry about  "KVM VGA-Passthrough ..." thread sad Anyone interested in continuing this discussion elsewhere, please contact me privately. I might be able to find it a new home.

Offline

#5847 2015-08-07 09:11:39

jasonwryan
Forum & Wiki Admin
From: .nz
Registered: 2009-05-09
Posts: 15,669
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Bronek wrote:
jasonwryan wrote:

Anyone else looking for a ban?

Yes, me too please. I do use Arch as my favourite distro, but I also happen to use other distros as well, and I am not going to abide by the order to discriminate their users. You are welcome to interpret this as a declaration of disobedience and ban me to pre-empt any post of mine which might (or might not) benefit users of other distros. Have it your way, if that is what you want.

jasonwryan wrote:

The next person that posts a whine in here about our policy can thank themselves for having the thread closed. You want to share knowledge about KVM passthrough, then stick to that.

And... we have a winner!



Closing.


Arch + dwm   •   Mercurial repos  •   Github

Registered Linux User #482438

Offline